ML20216B210
| ML20216B210 | |
| Person / Time | |
|---|---|
| Site: | Fort Saint Vrain  |
| Issue date: | 06/16/1987 |
| From: | Cooley R, Pellet J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML20216B153 | List: |
| References | |
| 50-267-OL-87-01, 50-267-OL-87-1, NUDOCS 8706300041 | |
| Download: ML20216B210 (6) | |
Text
' f l
REQUALIFICATION PROGRAM AUDIT AND OPERATOR LICENSE EXAMINATION REPORT l
Licensee: Public Service Company of Colorado P. O. Box 840 Denver, Colorado 80201 i
Docket: 50-267 License No: DPR-34 Operator License and Requalification Program Audit examinations at Fort St.
Vrain (FSV).
Chief Examiner: l@ [o -6' E 7 John L. Pellet, Examiner Date Signed Approved By: h. - ~
ulptfA. Cooley,SectpnChief Date Signed Summary:
NRC administered one Senior Reactor Operator license examination at FSV during the week of May 11, 1987. The license candidate'successfully completed his examination and has been issued the appropriate license.
NRC conducted a Pilot Requalification Program Audit _at FSV during the weeks of ;
May 11 and 18, 1987. Problems were identified in written examination prepara-tion and operating examination administration and evaluation. The Fort St.
Vrain Requalification Program is now evaluated as margirsal.
8706300041 870616 PDR ADOCK 05000267 V PDR
t 2
l i
REPORT DETAILS I
- 1. PERSONS EXAMINED License Examinations: TYPE: SR0 TOTAL 3 PASS: 1-100% 1-100% i FAIL: 0-0% 0-0% j
- 2. EXAMINERS l l
J. Pellet, NRC (Chief Examiner) ;
J. Whittemore, NRC j t~
3.. EXAMINATION REPORT.
Individual performance results are not included in this report because these reports are placed in the NRC Public Document Room,
- a. EXAMINATION REVIEW COMMENT RESOLUTION 1
Because the single license examination administered was a retake "
that involved only an _ operating examination, no written examination I was administered, and no comments received. '
- b. EXIT MEETING
SUMMARY
Because of the small number of license examinations administered, and the unavailability of any results for license or requalification efforts, the facility determined that no exit meeting would be held.
- c. GENERIC COMMENTS No generic comments were . developed during review of the single license examination administered. The following comments were generated during review of the facility written and operating requalification examinations.
(1) Knowledge on design and operation, especially malfunctions, of.
the nuclear instrumentation systems was weak.
(2) Understanding of operation, especially malfunctions, of the Overall Plant Control system was weak.
(3) The ability of non Shift Supervisor SR0 licensees to perform the administrative duties of the Shift Supervisor was weak.
3-I (4) There was' confusion about the relationship between the Bypass Flash Tank, the Condensate system, and the Decay Heat Removal system.
(5) When classifying events per the RERP, SR0s frequently stopped at the first classification that fit rather than assuring that l the most conservative had been identified.
- d. REQUALIFICATION PROGRAM EVALUATION NRC conducted a Pilot Requalification Program Audit of the FSV ,
Requalification Program during the weeks of May 11 and 18, 1987, in i accordance with the W. T. Russell memorandum titled, " Alternate '
Approach to Requalification Evaluations," dated May 22, 1986 (copy attached). This audit consisted of the following steps:
o Review the facility developed requalification written examina-tions to determine compliance with the facility requalification program and NRC examiner ification examinations standards (NUREG-1021, as. they) apply to requal-ES-601 o Observe the administration of and co-grade a representative sample of the facility administered written examinations.
o Observe the administration of and co-grade a representative sample of the facility administered operating examinations.
(1) The facility developed Reactor Operator (RO), Senior Reactor -
Operator (SRO), and Senior Reactor Operator Limited to Fuel -i Handling (SSLO) written examinations were reviewed prior to administration for compliance with the Fort St. Vrain 3 Requalification program and NRC guidance on requalification j examinations, NUREG-1021 ES-601. No problems were observed in l the SSLO examinations. But, the R0 and SR0 examinations were found to be unacceptable for administration as written. Prob-lems with the examinations, as further explained below, threat- ,
ened examination validity in three areas: 1) examination security, 2) question construction, and 3) discrimination level (exam difficulty).
I (a) FSV administers written requalification examinations daily for at least 5 consecutive days. Per the FSV requal-ification procedure (TP-LR), the FSV training. staff-'
developed two R0 and two SR0 written examinations, valued -
at about 80 points each. .NRC agrees that 320 points of questions is sufficient to permit generating examinations for 7 days without excessive potential for compromise or loss of examination . security. However,-the four FSV-1
,t l
4 examinations initially reviewed were not independent.
When duplicate questions were eliminated, less than 200 points of questions were used to generate the four exami-nations. NRC conciders this an unacceptably small sample from which to generate what are essentially fourteen examinations. Moreover, while four examinations were developed by the FSV training staff as required to meet step 4.3.2.4 in TP4R, the FSV requalification procedure, NRC considers that step to require development of at least four independent examinations.
(b) 'Several questions reviewed were found to-be invalid due to poor item construction. The major deficiency was poor distractor construction in multiple choice questions.
Individual problems included: distractor lengths were much different from that of the correct answer, the correct answer contained key words from the question, and given a list from which to choose the item that did not belong, the proper choice was separated from all others. Addi-tionally, the key for short answer questions frequently did not show adequate partial credit breakdown to assure consistent grading (c) The discrimination level, or difficulty, of several questions was found to be inappropriate to discriminate ,
between safe and unsafe operator licensees. Several l questions reviewed required only a trivial memorization of i t'asic nuclear theory, e.g., define Keff or subcritical multiplication. NRC requires that requalification exami-nations be operationally oriented, and that basic theoret-ical knowledge should be-tested as it applies to plant l operations. The examinations placed great emphasis on '
memorization or recall, with substantially less emphasis on interpretation and application (problem solving).
(2) A selected sample of FSV written examinations were independent-ly gradec' by FSV and NRC (co-graded) then compared against each other. No significant problems.were encountered during this ,
review. Section scores were almost exclusively within one l point raw score per section (5%), with overall scores within 2.5% in all cases except one where the difference was 3.8%.
Two individuals failed the facility graded written examination in a single category, but none failed using NRC grading. For these two failing cases, the FSV grades were 2.5% and 3% below NRC grades, which represents about one half point raw score.
Written examination administration was observed for one set of examinations, with no violation of facility or NRC requirements with respect to administration, except examination master copies were not available during the examination.
l s
.c 5
(3) One SSLO, one R0, and three SRO facility administered operating examinations were~ observed and co-evaluated using independently gathered notes. Additionally, eight R0 and thirteen SR0 facility administered operating examinations were reviewed.
The SSLO requalification program does not at this time adminis-ter an operating examination. . Implementation of SSLO operating examinations on an annual basis will occur as a result of recent changes to 10CFR55.
R0-and SR0 examination documentation was acceptable in both content and format. Based on the reviewed and observed R0 and SR0' operating examinations, methodology, and format of the facility administered operating examinations is acceptable.
Examination content was found to be acceptable, but somewhat abbreviated. compared to an NRC administered requalification operating examination. Content areas normally covered on NRC requalification examination, but routinely not covered on the facility examinations observed and reviewed include:
For SR0's For R0's Jumper Log Safeguards systems 4 Tagging procedures Nuclear instrumentation Key control Pre-startup checklists ,
Tech. Spec. changes Radiation protection P&ID's Fuel handling & storage ')
i When comparing operating examination evaluations, only one problem was encountered related to evaluation of one SR0 l examination. The facility evaluated the individual in question as marginal but satisfactory. While the NRC examiner agreed with this evaluation, discussion of the weaknesses of this individual indicated to the examiner that the facility examiners were unable to divorce their knowledge of the individual in question, and his day to day job performance, l from the performance demonstrated during the operating i examination. NRC considers such actions necessary in that only demonstrated performance during an examination may be considered in evaluating an individual's performance.
l i
L
I i
1 l
6 1 OVERALL REQUALIFICATION PROGRAM EVALUATION 1 Facility: Fort St. Vrain l Examiners: J. Pellet, J. Whittemore, ,
Dates of Evaluation: May 11-14 and May 19, 1987 l Areas Evaluated: XX Written xx Oral simulator l l
Written Examination Evaluation of NRC Examination Results (if given): Not Applicable Evaluation of FSV Examination and-Results: Marginal Oral Examination
- 1. Overall' Evaluation: Marginal
- 2. Number Observed: 5 Number Conducted: **
- Independently evaluated during facility operating examination Overall Program Evaluation Satisfactory: Marginal: XX Unsatisfactory:
REASON FOR EVALUATION
- Written examination unacceptable as originally generated ***
Submitted: Forwarded: Approved:
00 % 1 Examiner Section Chief Branch Chief
R . c .,
g-- '*"
W J, .A . . _ . __ _
_. - me- w masuwasw o -
- 1.. . :;. Ja - .. . m g r.?-n. :.,:gi-+ ;~, r if DISTRIBUTION:
e pyg.
OLB R.F.
'BBoger JHannon-W 2 7 96 TSzymanski WRussell. '!
RKeller, RI -]
MEMORANDUM FOR: Richard W. Starostecki, Director JMunro, RII i Division of Reactor Projects, RI TBurdick, RIII -!
' 'RCooleynSih Albert F. Gibson, Director ,
JElin RV-Division of' Reactor Safety, RII JZwollnski PLeech Carl .J. Paperiello, Director SVarga ,
Division of Reactor Safety, RIII DFischer I
DMull er.' .
Eric H., Johnson, Director GRivenbark-Division of Reactor Safety BJYoungblood.
and Projects, RIV- - P0'Connor TAlexion Dennis F. Kirsch, Director: VNoonan Division of Reactor Safety L01shan and Projects, RV i FROM:- William T. Russell, Director'- .
Division of Human Factors Technology, NRR ,
SUBJECT:
ALTERNATE APPROACH TO REQUALIFICATION EVALUATIONS ~
l 1
In our conference call of April 30. 1986,.we agree'd'that alternate approach'es, to the NRC evaluation of requalification' programs.fincluding pilot: testing . !
should be assessed to conserve NRC resources. A list oftfacilities that have
.,.- agreed to voluntarily participate.in pilot testing'is provided:in. -
Enclosure 1. Each: facility' requalification program is scheduled for.- :)
evaluation this summer. We request that'each Re the noted facilities so that the (pilot tested) gion programmake evaluations arrangements are~with completed prior to the'end of September. .During October welwilll. discuss ande evaluate.the results of the pilot testing. Any revision'to;requalification evaluation policy will'be promulgated early in FY-87E Guidance on the type and level of NRC. participation in these pilot tests is !
provided in Enclosure 2. Our objective-is to determine whether this method. .
of NRC participation.in facility requalification examinations can provideLus i with sufficient information. from. which to judge the . quality. of the facility: ..
requalification prog ram.- For budget ^ purposes, the pilot; tests;will be; ,
considered as FY-19E6 requalification program' evaluations. ,
, N.%
I %
(
\.
-F
..................... ..................... . . . . . . . . . . . ; . . . . . .- 1
""a"'> ................... ..................... ..................... .....................
..................... ............l........ ..................
==>
.. 1.................
. .. ..................... ..................... . . . . . . . . . . ............ ...... . . . . . .. . . . . . . . .R. . . .......1..q wxc reau m uomoiwacu ono : OFFICIAL RECORD COPYJ :
- u.s.o o am-me.:
M:M=n=&&1.4 -
Multiple Addressees 2? M Please evtact Bruce Boger, FTS 492-4868, with your final schedule and any-needs for additional guidance.
Or6ginal signed W William T. Russell, Director Division of Human Factors Technology, NRR-
Enclosures:
As stated cc: J. Sniezek H. Denton D. Eisenhut R. Bernero T. Novak-F. Miraglis J. Griffin (NUMARCWG#11) i v.
DgB25/MEMOTOREG. LIV.DIR.
wice > 0.L.6U5. .. . T . . .O. L.B. .:.DH F.T.
.u m mr)
B80 9 b ; th...... W sI.
I
....g............. ....
su > 5./. .v.. ./.8. 6. . ..5..
. . . . .. .8.. 6. . . . . ......................
ac ronu ma no,somacw o24o OFFICIAL RECORD. COPY -* u.s.m ymoo.24r
. s ,
1 ENCLOSURE 1 J REGION FACILITY COMMENTS i
I Pilgrim Written Only Salem Operating Test Only II Hatch None i i
III Callaway None i IV Wolf Creek None V Byron 1 No RV plant scheduled for requal during summer.
i 4
i l
'k l
i t
+
,\ L *
',s . .,
N N- -
.0' 2
\j ;
~%
.L ls .,
- \ l 'S f .. b^< ,
,A % -' * - .
4
.. .s. .
ENCLOSURE 2
' NRC participation' in the pilot test will be limited to prior NRC review / approval of the written requalification exam developed by the facility training staff, parallel grading of this exam, and prior review, observation and independent evaluation by an NRC examiner of the operating test conducted
.by the facility staff. In general, the level of the NRC participation will be commensurate with the review required by the Examiner Standards for NRC contractor developed' examinations.
'The facility written exam should follow the guidance contained in ES-601 (Rev. 2) with respect to exam format, content, and length. The review of-this'. exam should emphasize depth of knowledge required, comprehensiveness of.
the exam with respect'to broad coverage of systems and procedures, and the '!
relationship with the facility requalification' program learning objectives.
Parallel grading of the written examination should be conducted on 20% of the
.' licensed operators examinations developed by the facility-staff. The grading can be a random selection performed concurrently with the facility staff {
grading or can be the selection of a h'igh, medium, and low score after the f
facility staff has graded the exams. The review of grading should emphasize the awarding of partial credit, consistency of. scores, compliance with the answer key, and equivalency to the NRC grade (+ 10% per category). ]
'l Evaluation of the operating exam should emphasize depth'and breadth of i questioning and. comprehensiveness of simulator scenarios. Anind$ pendent 1
operating evaluation.should be performed by the NRC for one candidate per
)
V a, .
..~..
operating ~ test crew observed. 'NRC procedures (ES-302) for documenting this-
- candidate's performanc.e~are to be used. Comparison of NRC and utility evaluations',(i.e., parallel grading'). will be used to determine if candidate weaknesses are properly evaluated and ' documented.
Additionally a: subjective evaluation of the util.ity administered operating
. test should be performed using procedures similar those' for. NRC examiner certification '(ES-105) for.each utility examiner observed.
The utility requalification examination results including their' overall.
requalification program evaluation should be subjectively evaluated using-ES-601 criteria as a guide. Utility evaluation of. examination weaknesses should be reviewed to determine the adequacy of corrective action for.
programmatic weaknesses in the facility training program.
> n 5
s
.I Document Name:
FSV RPT Requestor's ID:
OP3 i Author's Name:
jpellet t Vr duditrptOL87-01 j l
Ybhg0
%7 i 1
l i
I I
1 l
l i
i i
i b
r.