ML20215M250
| ML20215M250 | |
| Person / Time | |
|---|---|
| Site: | Arkansas Nuclear  |
| Issue date: | 06/22/1987 |
| From: | Cooley R, Mccrory S NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML20215M239 | List: |
| References | |
| 50-313-OL-87-02, 50-313-OL-87-2, NUDOCS 8706260327 | |
| Download: ML20215M250 (6) | |
Text
.
1
'I ARKANSAS NUCLEAR ONE UNIT 1 REQUALIFICATION EXAMINAT, IONS 50-313/0L-87-02 Docket No: 50-313 License No.: DPR-51 Licensee: Arkansas Power & Light Company P.O. Box 551 Little. Rock, AR 72203 Requalification Audit conducted at Arkansas Nuclear One Unit 1 l
n
/
p r
Auditor:
uh
-6 M,
~ Sf L< McC'ro~ry, Lead Exam r
Date /
Approved by: J
/
b4/
d
)
/
Fb/ A? Copey, Section phief Dafe
/'
MO s
Summary Facility licensee administration of regularly scheduled requalification examinations was observed by an NRC examiner on April 9-10, 1987..
- The written examinations administered on April-31and 10,1987, to five (5)
Reactor Operators', and seven (7) Senior Reactor 0perators were. evaluated and parallel graded by the audit examiner.
The audit exa.niner observed the.
conduct of operating examinations for six-'(6) Reactor _0perators'and six (6)
Senior Reactor.0perators. The audit examiner conducted concurrent evaluation' of operator performance for one operator or senior operator in each crew given an operating examination..The audit examiner concurred with the results of the'
~
facility licensee evaluation on all of these examinations.
V
1 l
2 Report Details i
1.
Examination Results SRO Requalification R0 Requalification Total Pass Fail Total Pass Fail %
written -
7 7
0 100 5
5 0
100 1
operating 6
6 0
100 6
6 0
100 2.-
Audit Examiner S. L. McCrory 3.
Requalification Program Evaluation NRC conducted a Pilot Requalification Program Audit of trie AN0 Unit 1 Requalification Program during the week of April 6, 1987, in accordance with the W. T. Russell memorandum titled, " Alternate Approach to Requalification Evaluations," dated May 22, 1986 (copy attached).
This audit consisted of the following steps:
o Review the facility developed requalification written examinations to determine compliance with the facility requalification program and NRC examiner standards as they apply to requalification examinations (NUREG-1021, ES-601),
o Observe the administra. tion of and co-grade a representative sample j
of the facility administered written examinations.
o Observe the administration of and co-grade a representative sample of the facility administered operating examinations.
a.
Comments on Written Examination The facility developed Reactor Operator (RO) and Senior Reactor Operator (SR0) written examinations were reviewed prior to-administration for compliance with the ANO Unit 1.requalification program and NRC guidance on requalification examinations, NUREG-1021, ES-601.
Examination weaknesses, as further explained below, threatened examination validity in two areas: 1)examinationsecurity, and 2) discrimination level (exam. difficulty).
Additional weaknesses in question format are also noted below.
(1) Written examinations are administered weekly over a period of about 6 consecutive weeks.
Examinations prepared for weeks 1 and 2 were reviewed and the examinations for week 2 contained
3 duplication of questions from week 1 (15% duplication in the R0 examination and 24% duplication in the SR0 examination)
Further, the written exam was administered in a regular classroom which did not have dedicated restroom facilities, and i
examinees were permitted to go to the lunch room on the first floor.
Although examiners were allowed to be out of the exam room only one at a time, their movements outside of the exam room were unmonitored.
This is inconsistent with standard NRC practice when conducting written examinations.
(2) All exams reviewed covered an adequately broad scope of systems and topics.
However, the degree of difficulty of most questions was low.
There was very little attempt to evaluate higher cognitive levels of understanding of system operation or plant interactions and responses.
In general, this quality of written would not be accepted by the NRC from one of its contractors without significant revision.
(3) Partial credit points were not clearly identified in most of the question keys.
This is of particular concern on short answer explanation type questions which require several elements for a full credit answer.
Breaking down partial credit helps assure consistent grading and allows emphasis to be placed on the more important aspects of the question.
A selected sample of ANO Unit 1 written examinations was independently graded by AN0 and NRC (co-graded) then compared against each other.
No significant problems were encountered during this review.
b.
Comments on the Simulator Examinations One R0, and two SR0 facility administered operating examinations were observed and co-evaluated using independently gathered notes.
Additionally, six R0 and six SR0 facility administered operating examinations were reviewed.
In general, the scenarios were fairly linear.
Events were not complicated nor compounded. There were no common mode failures (as are frequently seen in the daily reports) which impair or disable multiple trains / channels of safety and safety-related equipment.
During the administration of the operating exams, the scenarios functioned well for the most part. However, there were no rotations of operators during the operating exam. This precluded the opportunity to adequately evaluate in more than one position.
Since there is no further evaluation required prior to assigning an operator to a station other than that in which he was observed during the operating exam, it is essential to rotate operators during the annual operating exam to allow proper evaluation.
4 i
Operator evaluation in the operating (simulator) examination was I
based almost exclusively on observed performance only.
Very few questions were asked of the operators during the scenarios, and there was no following discussion of plant operations or administrative requirements.
Content areas normally covered on NRC requalification examination, but overlooked by this approach include:
For SR0s For R0s Jumper Log Emergency response Tagging procedures Pre-startup checklists Key control Radiation protection Tech. Spec. changes P& ids Fuel handling & storage
{
Operations supervisors and managers are routinely used to augment the training department evaluation team for the annual operating exam.
i i
While this is desirable to promote close interaction between the training and operations department, it could be detrimental to proper f
operator evaluation during the annual operating exam.
These non-training department evaluators are not given formalized training in i
evaluation skills nor is any sort of certification conducted as is required for NRC examiners.
a The facility licensee examiners collaborated on a team evaluation of j
the operating crew after each examination set.
While this evaluation did not directly input to the individual operator j
evaluations, it did cause evaluators to focus on the aspects of 1
performance which are addressed on the form.
The form is poorly i
worded and does not focus on performance essential to operational safety.
Further, it can cause erroneous evaluations to be made about operator performance and distracts an evaluator from observing
~
safety significant actions.
At the time of the audit the facility did not have provisions to
)
ensure that all senior licensed operators were evaluated in a supervisory position during the annual operating exam.
This l
requirement is contained in the latest revision to 10 CFR Part 55, I
but is not effective until May 26, 1987.
j c.
Exit Meeting Summary i
At the conclusion of the audit visit, the examiner met with the members of the facility licensee staff to discuss the results of the audit. The following personnel lwere present for the exit meeting:
1 1
1
j 5
NRC UTILITY S. McCrory J. Vandergrift E. Force E. Wentz i
M. Goad I
D. Smith The short comings of the written and operating exams previously identified in this report were discussed with the facility training management. As a result of these discussions the facility licensee committed to take immediate action on the following:
.(1) Review the written exams for the remaining weeks of the requalification cycle and reduce the incidence of question duplication on consecutive weeks.
(2) Review the answer keys and designate partial credit elements in short answer and essay type responses.
(3) Prepare additional operating exam scenarios and rotate panel operators during the annual operating exam.
(4) Ensure that all senior licensed operators were evaluated in at least one supervisory position during the annual operating exam.
d.
General Comments In addition to those areas committed to in the exit, the facility licensee should develop questions for use in requalification examinations which require more than basic recall knowledge or memorization.
It is apparent that the training staff for ANO Unit 1 is dedicated to developing and maintaining an effective requalification program.
The 4
professionalism and responsiveness of the training staff to NRC 1
concerns are evidence of their desire to ensure licensed operator proficiency through a comprehensive and rigorous requalification program.
, i 6
i e.
Requalification Program Evaluation Report l
Facility:
Arkansas Nuclear One Unit 1 Examiner:
S. L. McCrory i
Dates of Evaluation:
4/9-10/87 i
Areas Evaluated:
X Written Oral X Simulator l
Written Examination 1.
Evaluation of Examination:
MARGINAL
- 2.
Evaluation of Facility Examination Grading:
SATISFACTORY Oral Examination I
1.
Overall Evaluation:
SATISFACTORY i
2.
Number Observed:
3 Number Conducted:
- Independently evaluated during facility operating examination Overall Program Evaluation 1
Satisfactory:
X Marginal:
Unsatisfactory:
REASON FOR EVALUATION Written examination lacked sufficient difficulty for proper discrimination between safe and unsafe performance. Question duplication on consecutive examinations jeopardized examination security.
Submitted:
Forwarded:
Approved:
/
c~.
l//
~l/'
$Vma yj l
Branch Chief 4xafntnbr g,g Section Chief 7
,. K..T.., _ ~. _.2 py.
3 2.. -
._' - "".f_J.'.7
.....e DISTRIBUTION:
DMB OLB R.F.
BBoger W 27 g JHannon TSzymanski WRussell RKeller, RI MEMORANDUM FOR:
Richard W. Starostecki, Director JMunro, RII Division of Reactor Projects. RI TBurdick, RIII RCooleycRIL Albert F. Gibson, Director JElin RV Division of Reactor Safety, RII JZwollnski Pleech Carl J. Paperiello, Director SVarga Division of Reactor Safety, RIII DFischer DMuller Eric H. Johnson, Director GRivenbark Division of Reactor Safety BJYoungblood and Projects, RIV P0'Connor TAlexion Dennis F. Kirsch, Director VNoonan Division of Reactor Safety L01shan and Projects, RV FROM:
William T. Russell, Director Division of Human Factors Technology, NRR
SUBJECT:
ALTERNATE APPROACH TO REQUALIFICATION EVALUATIONS In our conference call of April 30, 1986, we agreed that alternate approaches to the NRC evaluation of requalification programs, including pilot testing, should be assessed to conserve NRC resources. A list of facilities that have agreed to voluntarily participate in pilot testing.is provided in Each facility requalification program is scheduled for evaluation this sumer. We request that each Region make arrangements with the noted facilities so that the (pilot. tested) program evaluations are completed prior to the end of September.
During October we will discuss and evaluate the results of the pilot testing. Any revision to requalification evaluation policy will be promulgated early in FY-87.
Guidance on the type and level of NRC participation in these pilot tests is provided in Enclosure 2.
Our objective is to determine whether this method of NRC participation in facility requalification examinations can provide us with sufficient information from which to judge the quality of the facility requalification program. For budget purposes, the pilot tests will be considered as FY-1986 requalification program evaluations..
F j-g
'l I o"'c' >
a = "')
DATEh wac ronu m no,soinacu o2do OFFICIAL RECORD COPY
- u.s. opo m>-4ao.:
e
- ..o I
ENCLOSURE 2 NRC participation in the pilot test will be limited to prior NRC 1
review / approval of the written requalification exam developed by the facility training staff, parallel grading of this exam, and prior review, observation and independent evaluation by an NRC examiner of the operating' test conducted by.the facility staff.
In general, the level of the NRC participation will 4
be commensurate with the review required by the Examiner Standards for NRC contractor developed examinations.
i The facility written exam should follow the guidance contained in.ES-601
. (Rev. 2) with respect to exam format, content, and length.
The review of.
this exam should emphasize depth of knowledge required, comprehensiveness of i
the exam with respect to broad coverage of systems and procedures, and the relationship with the facility requalification program learning objectives.
Parallel grading of the written examination'should be conducted on'20% of the licensed operators examinations developed by the facility staff. The grading can be a random selection perfonned concurrently with the facility staff grading or can be the selection of a h'igh, medium, and low score after the facility staff has graded the exams.
The review of grading should emphasize the: awarding of partial credit, consistency of scores, compliance with the answer key, and equivalency to the NRC grade. (+ 10% per category).
Evaluation of the operating exam should emphasize depth and breadth of-questioning and comprehensiveness of simulator scenarios. An independent operating evaluation should be performed by the NRC for one candidate per I
..... operating test crew observed. NRC procedures (ES-302) for documenting this candidate's performance are to be used.
Comparison of NRC and utility evaluations.(i.e., parallel grading) will be used to determine if candidate weaknesses are properly evaluated and documented.
i J
Additionally a subjective evaluation of the utility administered operating test should be performed using procedures similar those for NRC examiner certification (ES-105) for each utility examiner observed.
The utility requalification examination results, including their' overall requalification program evaluation should be subjectively evaluated using ES-601 criteria as a guide. Utility evaluation of examination weaknesses,
should be reviewed to determine the adequacy of corrective action for programmatic weaknesses in the facility training program.
e 6