ML20207B366
| ML20207B366 | |
| Person / Time | |
|---|---|
| Site: | Prairie Island  |
| Issue date: | 07/22/1988 |
| From: | Wright G NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | Larson C NORTHERN STATES POWER CO. |
| References | |
| NUDOCS 8808020312 | |
| Download: ML20207B366 (78) | |
Text
,..
Docket No. 50-282 Docket No. 50-306 Northern States Power Company ATTN:
Mr. C. E. Larson Vice President, Nuclear Generation 414 Nicollet Mall Minneapolis, MN 55401
Dear Mr. Larson:
SUBJECT:
REQUALIFICATION PROGRAM EVALUATION In a telephone conversation between Mr. Ted Amundson, of your staff and Mr. Paul Sunderland, arrangements were made for an evaluation of the requalification program at the Prairie Island Nuclear Generating Plant.
The evaluation visit is scheduled for the week of October 24, 1988.
For this visit, the NRC examiner will administer NRC prepared written examinations and operating tests.
The NRC examiners will visit the site approximately 30 days in advance to meet with the appropriate facility personnel and operators to review the schedule and the process for these examinations.
For the examiner to adequately prepare for this visit, it will be necessary for th %cility to furnish the approved items listed in Enclosure 1, "Reference.laterial Requirements." Mr. Amundson has been advised of our reference material requirements and where they are to be sent.
Since this is early in tne new requalification scheme, the number of items required will be determined by the NRC's judgement of a good faith effort by the facility.
The NRC reserves the right to declare a facility training program unsatisfactory and to postpone NRC administered requalification examinations if the facility ge"erJted materials are inadequat0 for examination preparation.
Enforcement accion may be considered if necessary to bring facility generated material to the level of quality for examination preparation.
Additionally, it is requested that a licensed SR0 from both the facility's Operations Department and Training Department be designated as the facil'ty representatives for these examinations.
These individuals must not be scheduled for an NRC-administered examination durig this visit, or participate as an instructor once selected.
Also, the reviewers will be required to certify that, as a result of their review, no portion of the examination has been knowhgly nomproMised.
The facility management is responsible for providing adequate space and accommodations to properly develop and conduct the examinations.
"Administration of Requalification Examinations," describes our requirements for developing and conducting the examinations.
Mr. Amundson has also been I l 8808C20312 880722 i M PDR ADOCK 0"000282 p ff e
s eDe
k s.
a 3 aJ,u Northern States Power Company 2
informed of these requirements. Also, a facility operations management representative should observe the simulation facility examination process at the site., contains the "NRC Rules and Guidance for Examinees" that will be in effect during the administration of the written examination.
The facility management is responsible for ensuring that all operators are aware of these rules.
This request for information was approved by the Office of Management and Budget under Clearance No. 3150-0101, which expires May 31, 1989.
Conments on burden and duplication may be directed to the Office of Management and Budget, Reports Management Room 3208, New Executive Office Building, Washington, D.C., 20503.
Thank you for your consideration in this,aatter.
If you have any questions on the evaluation process, please contact Paul Sunderland ([312] 790-5771) or Doug Shepard ([312] 790-5684).
Sincerely,
- [ m IL('[
~;6 Geoffrey C. Wright, Chief Operations Branch
Enclosures:
1.
Reference Material Requirements 2.
Administration of Requalification Exa.nina ti ons 3.
hRC Rules and Guidance for Examinees 4
ES 601 Rev 5 c
/ enclosures:
. Watzl, Plant Manager
. 0/DCB (RIDS)
Licensing fee Managr.ent Branch Resident Inspector, RIII Prairie Island Resident laspector, Rlli Monticello John W ferman, Ph.D.,
Nuclear Engineer, MPCA State Liaison Officer State of Minnesota D. Dilanni, NRR T. Amundson, Training Supervisor cc w/o enclosure:
D. Dilanni, Project Manager, NRR J. Hannon, Branch Chief, OLB B. Burgess, Section Chief, DRP I
RllIn Rill RIII.
Sunderf'and/imsbuy)dich (5 m
(
J'm Wri ht
,(,, e pw pa h
ENCLOSURE 1 REFERENCE MATERIAL REQUIREMENTS For the written examination, the following items must be provided to the NRC 60 days prior to the examination date:
1.
Proposed 90 and SRO requalification examination test items.
(A minimum of 350 per section of the examination.)
Since the written examination is open reference, examination items must meet the following:
a.
It-ms that require only memorization or recall are not permitted; b.
Items should require that the examinee comprehend, interpret, integrate, or apply available infomation; l
c.
Items should contain situations, aspects, or conditions that do l
not duplicate lesson plans or references; and l
d.
Items should require examinees to locate and use of references.
The written examination will be composed of two sections, each designed to be completed in 1-1/2 hours. Each section will be separate.
Section A will be administered on a static simulator; Section B will be administered in a classroom setting.
Section A is designed to evaluate the operator's knowledge of plant systems, integrated plant operations, and instrumentation and controls.
In addition, recognition of Technical Specification LCOs and the operator's ability to diagnose postulated events should be evaluated.
Section B of the written examination is designed to evaluate the ability of the operator to analyze a given set of conditions and detemine the proper procedural and/or administrative guidance.
2.
All reference material and objectives for the proposed test items.
l For the simulation facility, the following items must be provided to the 1
l NRC 60 days prior to the examination date:
A minimum of 15 scenarios The scenarios should sample areas such as LERs. emergency and abnormal procedures, and design and procedural changes that exercise the crew's ability to use facility procedures in accident prevention and mitigation.
The scenarios should evaluate each crew member as appropriate to his/her license, and shall exercise their abilities in the use of Emerger.cy Operating Procedures, Technical Specifications, and the Emergency Plan. The scenario's net time (not including time spent on briefings, setup or simulation facility problems) should average 50 minutes, based upon real time perfomance.
l 2
For the plant walkthrough examination, the follawing items must be provided to the NRC 60 days prior to the examination date:
a.
A list of systems (approximately 12) and topics appropriate to the plant walkthrouch examination that were covered during the requalificationcycle. All reference material required to i
support an examination on these topics should be provided.
J b.
Seventy-five (75) j9b performance measures These performance measures should be both inplant and in control room operator functions, that are required for the safe operation of the facility.
Titey shall include acceptable performance criteria.
l c.
If additional reference material is r(quired for examiner i
preparation, it will be requested during the pre-examination site visit.
i t
i l
i i
l l
1 l
i I
i
)
l 1
ENCLOSURE 2 ADMINISTRATION OF REQUALIFICATION EXAMINATIONS 1.
Twenty percent of the facility licensed operators shall be selected for evaluation. Normally, the crew currently in the requalification cycle will be selected. A random sample without replacement will be used to preclude a satisfactory operator from being subject to re-examination by the NRC during the term of the license. The sample will include other shif ts made up of licensed personnel who are not routinely perfonning shift duties.
2.
The simulator and a simulator operator (s) will be provided for examination development. The date(s) and duration of time needed to develop the examinations will be agreed upon by the chief examiner and the facility.
3.
The reference material used in the simulator will be reviewed by the chief examiner, no material will be made available that is solely used for training.
4.
A single room shall be provided for completing Section B of the written examination. The location of this room and supporting rest room facilities shall be such as to prevent contact with all other facility and/or contractor personnel during the duration of the examination.
5.
Minimum spacing is required to ensure examination integrity as determined by the chief examine". Minimum spacing should be one examiner per table, with a 3-feet space between tables.
No wall charts, models, and/or other training materials shall be present in the examination room.
6.
Copies of reference material for Section B of the written examination will be provided for each examinee.
The reference material will be reviewed by the chief examiner and will consht of Technical Specifications, operating / abnormal procedures, administrative procedures, Emergency Plans as available to the plant operators.
7.
Video taping capabilities can be utilized. The facility should contact j
the chief examiner for restrictions related to its usage, j
8.
Since comen tasks and detailed systems knowledge will be probed during the walkthrough portion of the operating test, operators will be iequested not to discuss the walkthrough with other examinees until after the complete examination has been administcred.
9.
An attempt will be made to distinguish between R0 and SR0 knuwledge and abilities, to the extent that such a distinction is supported by the tacility training materials, l
l i
ENCLOSURE 3 NRC RULES AND GUIDANCE FOR EXAMINEES 1.
Use black ink or dark pencil ONLY to facilitate legible reproduction.
2.
Print your name in the blank provided on the cover sheet of the examination.
3.
Fill in the date on the cover sheet of the examination, if necessary.
4.
Answer each question on the examination.
If additional paper is required, use only the lined paper provided by the examiner.
5.
Use abbreviations only if they are cmnonly used in facility literature.
6.
The point value for each question is indicated in parentheses af ter the question and can be used as a guide for the depth of answer required.
7.
Show all calculations, methods or assumptions used to obtain an answer to a mathematical problem, whether asked for in the question or not.
8.
Unless solicited, the location of references need not be stated.
9.
Partial credit may be given. Therefore, ANSWER ALL PARTS OF THE QUESTION i
AND 00 NOT LEAVE ANY ANSWERS BLANK.
1 10.
If parts of the examination are not clear with respect to their intent, I
ask questions of the examiner only.
- 11. You must sign the statament on the cover sheet that indicates the work on the examination is your own and that you have not received or been given any assistance in completing the examiration.
This must be signed AFTER the examinat;on has been completed.
- 12. Rest room trips are to be limited and only one examinee at a time may leave.
You must avoid all contact with anyone outside the examination room to avoid even the appearance or possibility of cheating, i
13.
Cheating on the examination would result in a revocation of your license and could result in more severe penalties.
- 14. Each section of the examination is designed to take approximately 90 minutes to con'plete.
You will be given two hours to complete each section for a total of four hours.
- 15. Due to the existence of questions that will require all examinees to refer to the same indications or controls, particular care must be taken to maintain individual examination security and avoid any possibility of compromise or t.ppearance of cheating.
- 16. When you are finished and have turned in your completed examination, leave the examination area.15-401 Rev 5 10/1/88 DRAFT ADMINISTRATION OF NRC REQUALIFICATION PROGRAM EVALUATIONS A.
Purpose This standard provides general guidance and requirements to NRC examiners for the administration of NRC requalification examinations. This program evaluates the effectiveness of a facility's licensed operator requalification training program to maintain the competency and currency of licensed operators. This is done by evaluating the ability of the facility to adequately prepare written examination questions and simulator scenarios and their ability to properly evaluate their operators' performance.
In addition, satisfactory completion of the examinations by individual operators would satisfy the regulatory requirement to pass an NRC administered requalification exanination prior to i
This document is not a substitute for the operator licensing reguiationsandissubjecttorevisionorotherinternaloperatorlicensing policy changes.
B.
Program Description The NRC requalification examinations administered under this standard per the provisions of 10 CFR 55.5g(a)(2)(iii) attempt to minimize the potential for an adverse impact on the safe operations of facilities, and to provide the staff with an assessment of the effectiveness of facility requalification training programs.
The requirements and procedures are derived based on a systems approach to training (SAT) program in accordance with INPO Guideline 86-025, and rely upon existing requalification program standards for guiding the NRC examination development and implementation. This approach will allow the NRC to atiminister l
requalification examinations that are consistent with existing facility Examiner Standards 0001.0.0 of 28
ES-401 hev i 13/1/08 0
developed programs, thereby reducing the impact on the facilities and improving the reliability of the NRC assessment of requalification training programs.
Each requalification examination will be developed by en examination team consisting of IIRC examiners and facility representatives and will be reviewed by facility representatives and by the Resident Inspector where practical.
The examination will be based upon the facility requalification program and its learning objectives, to the extent practical.
This approach will result in more technically sound and cperationally oriented examinations.
In addition, coevaluation of operator performance by the NRC and the facility will enhance the ability of the NRC to assess both individual nd program performance.
The NRC administered requalification examination is cor. posed of an operating test and a written examination.
The operating test consists of a simulator evaluation which emphasizes tisw-critical and team-dependent behavior with a flagging systen to enable follow-up of individual weaknesses.
The walk-through evaluation will cover plant systems identified by the NRC and the facility that,are important to the safe operation of the facility as contained within Groul1 I ard 11 of the "Examiners' Handbook for Developing Licensing Eneminations' (See page 2-18 of NUREG/BR-0122).
The written examination consists of a two section open book examination:
Section A will be a ' Plant Cperations" section and is administered on a static sirulator; Section B will be a "Linits and Controls" section and is administered in a classroor. setting, C.
Administrative Controls The NRC will plan to examine an average of 201 of the licensed operators and senior operators at each facility on an annual basis. The following criteria outline the requirements for the administration of the NRC Requalification Evaluation Program.
1.
Fscility Notification i
l 9
Examiner Standards 0002.0.0 of 28
ES 401 hev 5 10/1/88 The attached eerporate notification letter is used to inform facility program managers of the impending requalification program evaluation. This letter should be uniled at least 90 days in advance.
Site visits should be scheduled to coincide with the requalification training cycle of the facility. The facility should know 30 days in advance which operators will be evaluated.
2.
Selectiam of Operators The facility will propose operators for NRC examination 60 days prior to the administration date.
The NRC will select crews and individuals to be examined based on the following criteria:
a.
Number of individuals on crew not examined during current license terr.,
i b.
Nurber of examinees required to simple 20% of licensed personnel.
c.
Number of staff (not crew) licensed personnel not examined during current licer.se term, and ld.
Length of time until license expiration.
.e.
Priority given to crew (s) in training during the examination week (s),
i
.I An operator that has previously passed an NRC requalification examination during the tervi of the license may be included in the simulator crew i
evaluation, if he/she is a member of a crew selected. Such an individual will not be required to take the written or walk-through examination.
Poor j
performance in the simulator examination would be evaluated as described in Section D.7.a.
j The sample will include additional shifts composed of licensed personnel who are not routinely performing shif t duties, in order to sample approximately 20 percent of the operators each year.
If a facility evaluates mixed crews (shift and non-shift license holders), mixed crewt, may be evaluated, i
i l
Examiner Standards 0003.0.0 of 28
18 401 Nov 5 10/1/08 3.
Facility Involvement a.
The facility will be requested to provide two employees, one from the operations staff and one from the training staffs, to assist the NRC to form the NRC e w ination team. The esployee from operations
$11 be an active licensed $% thegloyee from training should g. preferably be a dicensed SNO, but may be a aprtified instrseteg.
'The function of these individuals is to provide facility specific technical assistance to the NRC in the development and review of the written examination ittes, plant walk-through topics, and simlator scenarios.
If necessary due to insufficient facility staff resources, facility representatives may prticipate in the actual conduct of the operating test or written examination.
b.
To ensure examination security, the individuals shall not in any way be involved in requalification program activities between the start of their involvement in examination development and the administra-tier. of the exerirations. Also, the individuals shall be instructed and agree not to comunicate in any f ashion the content or scope of the examination to unauthorized persons.
Each individual will be required to sign a statement prior to reviewing any examination material that he/she will not knowingly divulge information concerning the NRC examination, and then to sign a second statement after examination administration that he/she has not knowingly given inforwtion about the specific content of the examination to any l
)
unauthorized person.
2 l
c.
For the simulatiun facility examinations, a facility management j
)
representative with responsibilities for the conduct of plant operations (as a mirimum, first level above shift supervisor) be 4
J present during administration of these examinations.
I j
d.
In addition, the facility will be required to grade, in parallel with j
the NRC.
4 Timetable for Administration of Requalification Examinations Examiner Stardards 0004.0.0 of 28
t$ 4C1 Rev 5 10/1/08 The facility is nottfied.
90 days in advance The facility provides proposed materials for cr.amina-60 days in advance tion construction (including written examination question and sissalator scenario banks and jcb performance measures with follow-on questions for importantsafetysystems).
The facility proposes crew composition.
NRC notifies facility of crew selections.
30 days in advance NRC team visits the facility to prepare for the 14 days in advance examinations.
NRC prepares for examination in the Regional office.
7 days in advance Examination Week NRC administers requalification examinations to selected crews.
l 5.
lExaminationAcministration q
The simulator examination is normally administered first, followed by the plant walk-thrJugh evaluations for each crew.
The two part written examination will j
be asninistered after all simulator and plant walk-through evaluations have j
been coepleted.
This order of administration is intended to provide for an operational atacsphere prior to the written examination.
Each portion of the i
requalification examination shall include the appropriate oral briefing or rules handout.
i l
6.
Regelification Program Evaluation 1
j a.
Program Evaluation I
l Examiner Standards 0005.0.0 of 28
l i
IS 401 Rev i 10/1/88 A peogram evaluation should normally be based on a minimum sample g;ph ( K and senior operators.
If the p1 #5 MWMintM 5% M66%%%'"-
( program ovaluattom will te deferred wat114pc1vsion of the next annual cycle, Snless the lateria results warrant Maediate action.
A satisfactory requalification program must meet each of the following:
(1) 901 pass / fail decisions agreement between the NRC and facility 1
grading of the written and operating examinations.
(2) At least 801 of all operators pass the examination.
(3) The program is judged satisfactory in accordance with the guidance given for the simulator evaluation.
-(4) The program meets the requirements of 10CFR55.59 or, in lieu of paragraphs (c)(2), (c)(3), and (4), is based on systerns approach to training.
When using the above percer' ages, fractions of individaals should be rounded up to the next highest number.
For example, if twelve (12) licensed individuals are evaluated, 801 passing would be 9.6, thus ten of twelve passing should be considered as meeting the 80% requirement. This rule should be applied j
+hroughout this standard.
l 1
b.
Reference Material Evaluation NRC reserves the right to der.lare a facility training program unsatisfactory and to postpone NRC akinistered requalification examinations if the facility generated materials are inadequate for l
examination preparation.
Enforcement action may be considered if j
necessary to bririg facility generated material to the level of quality for examination preparation.
7.
Requalification Performance for an Individual l
Examiner Standards 0006.0.0 of 28
ES 401-Rev 8 10/1/08 For an indiviesel to successfully pass the requalification examination as graded by the ERC, the following apply:
a.
Achieve at least 801 overall on the written examination, b.
Sattsfactorily cosplete the walk-through examination, 6nd i
c.
Be judged satisfactory on the simulator examination.
8.
Exanination Process Critique The operators will be provided with an examination process critique sheet and encouraged to complete the form at the conclusion of the examination process.
The facility will be responsible for collating the critiquet and providing them to the HQ Branch Chief, with a copy to the Regional Branch Chief, af ter the administration of the examination.
The Regional Chief Examiner will compute the mean and standard deviation for each item and include the results in the exam report.
I D.
Simulator Evaluations I
The purpose of this portion of the examination is to evaluate the crew's time-critical and team dependent behavior and individual performance during I
simulated events, a
Untilfullcompliancewith10CFR55.45(b),forplantswithoutasimulation facility, the facility shall propose an alternate method such as a control room occkup for the conduct of this portion of the examination. NRC 'till determine i
the usefulness of such alternatives.
)
1.
Scenario Scope and Cont?nt i
j The facility shall provide at least 15 simulater scenarios to the examination j
team. The scenarios must be approximately 50 minutes in length and both I
i Examiner Standards 0007.0.0 of 28
ES-401 Rev 5 10/1/08 comprehensive and realistic, i.e., not a series of unrelated events. The examination team should verify the scenarios against the facility requalification program learning obejettves and compare the tritical tasks with the K/A catalog to ensure they possess japortance factors of 3.5 and above. The scenarios may be modified by the team based upon the technical advisement of the facility representatives.
The scenarios should be based upon lessons covered in the requalification cycle, recent industry events, LCRs, emergency and abnormal procedures, and design and procedural changes that exercise the crew's obt11ty to use facility procedures in accident prevention and mitigation.
Each scenario should exercise each s:rew member as appropriate to h4/her position during the scenario. That is, each individual should be required to perform at least one critict1 task during this portion of the operc. ting test.
Each simulator evaluation should place individual crew mnters in the most senior watch standing position in which the individual normally operates on shift.
Each evaluation shall exercise the crews' abilities in the use of Emergency Operating Procedures, Technical Specifications, and the Emergency Plan.
2.
. Scenario Complexity, Standards and Criteria The NRC will review scenarios to ensure tasks are compatible with NUR[0-1122/ll23
- Knowledge and Activities Catalogs for Nuclear Iower Plant
)
Operators' for abilities with importance factors of 3.0 and above.
The following should also be considered:
1 (a) too complex beyondthescopeoflessonscovered(unrealistic) beyond the ability of procedures beyond the simulation facility capability (b) too simple should be integrated events no team dependent behavior no time critical behavior absence of equipment malfunctions The examination team will review scenarios to ensure expected responses are i
provided.
Examiner Standards 0008.0.0 of 28
ES.401 Rev 5 10/1/08 The NRC and the facility representatives as the examination team will jointly identify 'tritical tasks' for each scenario that are crucial to the maintenance of plant safety. To identify critical tasks, review the JTA or other facility training material for identification of critical tasks for each scenario.
In addition, the knowledge or ability should be rated 3.5 or above in the K/A catalog. The facility representatives' review should substantiate the task as critical to plant safety. Should the crew or individual incorrectly perfors these critical tests, this would establish a basis for an unsatisfactory evaluation.
3.
Simulator Examination Length and Position Rotation A scenario's contact time (not including time spent on briefings, simulator setup or sisulation facility problems) should average 50 minutes, based upon real tire performance, j
Position rotation will be dependent upon facility rotation practices.
Crews will,be evaluated in toe simulator as they are configured by the tecility for j
operating the plant.
Control manipulative skills will be evaluated in the i
welk-through.
For example, senior operators may be required to perforr control board operations on the simuistor.
The number of scenarios shall be sufficient, with a min % m of two per crew, to ensure that each examinee is tested to the extent that he/she may operate during both normal and emergency operating conditions.
4.
Recording and Flagging Systen Any individual weakness observed on competencies detailed on NRC Fors1157, f
page 4, will be potential items for follow-up.
Follow-up may not be 1
i I
l Examiner Standards 0009.0.0 of 28 l
l-ES 401 Rev 5 10/1/08 required if the examiner's concerns are resolved during the facility led scenario crittgue.
Passive observations will be made. Questioning of operators during the scenario should only be done after the completion of the facility led scenario critique. Follow up questions for flagged items should be asked at this time.
The Chief Examiner may question the plant management representative for clarification, if necessary, a.
Individual observations - record any deficiency noted.
Failure for an individual, on the simuletor portion only, shall be based upon significant deficiencies that could challenge the safety status of the facility such as:
1 (1) inability to effectively manipulate controls (2) failure to actuate a reactor trip when required (3) failure to ensure ESFAS systems inject / actuate when required i
(4) failure to e:nergency borate or initiate ADS when required (5) violation of Technical Specifications (6) f ailure to take any other action or cor.bination of actions tnat would prevent a challenge to plant safety.
(7)
Initiate inappropriate actions or corbination of actions that create a challenge to plant safety.
Significant deficiencies will nonna11y be associated witn identified critical tasks as indicated in D.2 above.
b.
Team observations - evaluation of crew performance will be mode as directed in the "Simulator Crew Evaluation Form."
i c.
Flagged itees of weakness noted on the simulator examination should be covered through follow-up questions af ter completion of the l
scenario critique to determine if there is a need for remedial training, unless resolved during the facility led critique. These questions should also reference a K/A to indicate importance, and be reviewed by the facility representatives.
Examiner Standards 00010.0.0 of 28 l
ES401 Rev 5 10/1/04 5.
Rus6er of Eqaminers Required The minimum number of examiners required is one for every two operators plus i
one examiner as coordinator and ;he Chief Examiner, who may act as the coordi-nator or as an examiner.
6.
Assistance of Facility Operations Management j
The facility will be requested to provide a member of the plant operations i
4 staff to observe the process. The Chief Examiner is the principal point of contact between the facility management and the NRC.
i 7.
Individual / Crew / Program Simulator Evaluation Guidance Following each scenario, the crew performance will be critiqued by the facility graders at the simulation facility.
The evaluation is to be observed by the NRC examiners.
Use the ' Crew Evaluation rom
- to document the results of the examination.
The facility evaluators should also complete a copy of this fom I
or an equhalent facility form for each crew.
'a, Individual Evaluation l
(1) An individual may fail the NRC examination based on significant l
deficiencies (see D.4) observed during the simulator examination, i
(2)
An individual may fail the NRC excmination based on follow-up questioning on deficiencies observed in the simulator.
i (3)
An individual who has passed an NRC requalification examination and is being used es a crew member may fail the examination or be found to require remedial training as a result of poor performance meeting the criteria of D.4.a.
1 I
Examiner Standards 00011.0.0 of 28 l
ES 401 Rev 5 CD/1/08 (4)
NRC or facility evaluator judgement that a crew is W5AY!$ FACTORY in the simulater will not necessarily result in en UNSATISFACTORY individual performarce evaluation. That is, an individual operator mey have made only minor errors in execution such that they were overcome during the questioning aft.r the scenario critiques and may not have contributed significantly to the poor crew performance.
i b.
Crew Evaluation (1)
Crew performance shall be evaluated in accordance with the "Simulator Crew Performance Evaluation Form."
1 (2)
If a crew is judged UNSATISFACTORY in the simuistor by either the NRC or the facility evaluators, the crew must be taken off l
shift, given remedial training, and reexanined prior to resuming i
licensed duties.
NRC should administer the test if the
)
facility's requalification program is currently judged UNSATISFACTORY and NRC has not verified that adequate corrective measures have been instituted.
Otherwise, the facility will be permitted to administer the reeraninatior..
i c.
Program Evaluation I
(1)
A program may be judged UNSATISFACTORY if the NRC judges at least one crew UNSATISFACTORY and the facility evaluators judge j
thatcrewSATISFACTORY;(e.g.,facilityevaluatorsfeelno l
remedial training is necessary).
l (2)
A program may be judged UNSATlsFACTORY if there is less than 901 i
l agreement between the NRC and the facility on the individual l
pass / fail determinations with the facility evaluating fewer i
individuals UNSATISFACTORY.
i Examiner Standards 00012.0.0 of 28
l U-401 Rev 5 10/1/08 (3)
If the facility evaluators judge see crew performance IMSAfil.
FACTORY end the NRC does not, remedial training is indicated but the program will not be penalized for holding a higher standard of operator performance.
(4)
If two or more crews are determined to be UNSATISFACTORY, regardless of individual failures, the overall program will be judged UM5AT15 FACTORY.
8.
Video Taping The following basic guidelines apply:
l a.
If equipeent is available, the simulator examination will be video taped.
I b.
Image clarity and degree of coverage must be adequate to make taping useful.
]
c.
After initial set up of the camera by the licensee's persernel under i
observation of the Chief Examiner, the recording will be made with l
l unattendedcamera(s). The only intervention will be to change the tape.
]
d.
The critiques will be video taped if the scenarios are video taped.
I e.
Two copies of the video tape will be made, one for NRC and one for i
the facility, i
The video tape will be used primer 11y to resolve areas of contention between l
the facility and NRC examiners' parallel evaluations of the operators.
Additionally, the facility, the examiners, and operators will be provided an opportunity to review those portions of the video tape that directly affect the pass / fail decisions, e.g., performance of previously identified critical items, if they so request.
NRC will supply the video tapes and maintain custody of I
one copy and leave one copy with tt.e facility. After examination tesults are i
j Examiner Standards 00013.0.0 of 28
__.____________i
ES-601 Sev 8 10/1/00 finalind, had all cor.flicts resolved, the lutC and th facility will erau the video tapes. The facility tape custodian will sign a statement upon receipt of the video tape indicating that the tape will be used only for its intended prupose and will be promptly erased at the conclusion of the examination, l
E.
Walk-Through Evaluations The purpose of this portion of the operating test is to assess the individual's understanding of and the ability to perform actions associated with plant I
systems and manipulations that operators may either perform, or direct the perforrance of, and to assess the requalification program's effectiveness in i
keeping the operator's knowledge current with respect to these important safety-relcted tasks and the associated systems.
1.
Applicable Plant Systems The facility will identify those plant systers applicable to maintenance of public health and safety in the mitigation of the consequences of an event, and those systems that can directly initiate an event. Criteria for system selection 1
include:
a.
Systems covered during the facility requalification cycle.
b.
New or recently modified systems.
c.
Systees the subject of recent facility LERs or vendor notices (e.g.,
GESILs).
i
+
d.
PRA identified risk dominant systems /corponents for plant or vendor generic plants.
1 i
j e.
NRC Information Notices.
1 1
I Examiner Standards 00014.0.0 of 28
ES-601 Rev i 10/1/88 The NRC team will evaluate the facility identified systems Absent other information, systees should be selected from those identified in troups 1 end 11 of the 'Exambers Handbook for Developing Licensing Examinations,'
NURES/BR 0122 with at least 501 of the selected systems from Group 1 of the Handbook.
Selected systems will be discussed with the facility representatives to ensure optimum site specific relevance.
NRC additions and/or substitution will be discussed with the facility representatives.
2.
Identify Tasks The facility representatives will review the facility JTA including learning objectives and NUREG-1122/1123 highlighting tasks /ebilities for the identified i
syster.s that meet the following criteria:
a.
Are applicable to the facility.
b.
Are at the A0/R0/SR0 level - RO is retponsible for A0/R0 tasks.
The SFC is respersible for A0/R0/SR0 tasks.
)
'c.
Have a K/A ability rating of 3.5 or higher.
Itees may be used that have ratings below 3.5 after NRC examiners consult with the facility j
representatives, j
i A list of plant specific tasks should also be developed. These are I
tasks / abilities that may not be specifically addressed by the JTA or NUREG-1122/1123 but which have been covered, due to special needs, in the I
requalification program, e.g., special procedures, EDG operations.
The facility representatives will review and concur on job applicability and importance for the complete list of tasks.
l I
l Eneminer Standards 00015.0.0 of 28
i I'
t$ 401 Rev 5 10/1/08 3.
Job Perfoemance Measures (JPMs)
For each of the tasks identified, review the JTA and/or learning objectives and appropriate cross references to determine the training mode.
Seventy five JPMs shall be subsitted by the facility. Refer to the Lesson Plan, Qualification i
Card or Job Performance Measure which addresses the task.
The JPMs should include:
1 s.
Initial conditions b.
Initiating cues c.
References i
d.
Perforr.ance elements and standards e.
Cues l
f.
Any appropriate output staterer.ts.
t j
g.
Appropriate knowledge areas for in-depth questions and answers.
I l
Criteria for satisfactory completion of the task will be identified prior to l
l l
administration of the examination. Critical steps which may not be included in f
the facility documentation of the JPH Mould be identified.
l l
The team should review JPM criteria and critical steps hased on the technical I
support of the factitty representatives. The examination team should review
)
proposed JPNs per the criteria in the "JPM Quality Checklist," All questions
{
and answers will be reviewed by the facility representatives for job relevance j
and safety significance prior to examination administration.
If the NRC asked l
follow up or probe questions during the walk-through, these questions should j
j be reviewed as soon as possible af ter the walk-through is completed.
2 i
[naminer Standards 00016.0.0 of it
E8 401 Rev 5 10/1/88 4.
Time Al kcation The walk-threegh will be plannad for approximately 2-1/2 hoer in length.
This includes both the control room and in plant time and is reflective of actual examination contact time. Time required for monexamination evolutions / items will not be considered during examination planning. These evolutions /itees include, but are not limited to, the followingt a.
Transit tire to and froe the plant site.
l b.
Time spent coeplying with facility security and radiological admin-i istrative requirements.
c.
Transient tine from the control room to in plant locations.
d.
Transit time from one in-plant location to another.
.e.
Time required for the operator to locate examination aides, e.g.,
d procedures, prints, etc.
l S.
lExanination Construction and Administration l
.The walk-through will be constructed and administered as follows:
\\
l a.
Time should be allotted during the operattag test for evaluating the perforr.anceoften(10)JPMs.
A minimum of four (4) JPMs should be evaluated outside the control room.
Five of the ten tasks should be
'camon' tasks. These common tasks will be administered to each i
operator and used in detervining the effectiveness of the j
requr.lification prograr in preparing the opeistors to perfor1n the j
tas ks, i
b.
A JPM worksheet will be completed for each task.
l t
i
)
Examiner Standards 00017.0.0 of 26
hev 5 1
/
c.
The detailed geestions as recorded on the 'JPM Question Fors' will be based upon the testing objectives established in part E.3 above and sheeld have a K/A rating of 3.0 er above. Question construction should meet the guidelines established for the development of the written examination. The K/A for each question and correct answer will be indtrated.
Each JPM will have a minlaum of two questions which directly relate to the #N, and which i
address the basic understanding and application of knowledge of the JPM.
Correct responses for these questions should not normally be i
directly found in control room references, i
d.
Each examination will be reviewed by the Chief Examiner also, using the guidance in NUREG-1021.
)
i e.
The facility examiners will cceduct the walk-through while an NRC l
examiner grades in parallel.
The NPC examiner may ask questions of 1
the operator necessary and appropriate to ensure adequate coverage of the content of the walk-throu...h at the completion of each JFM, and j
to ensure that the operator h!t demonstrated satisfactory understar. ding and applicatien f '<nowledge regarding the JFM, the NRC examiner sust ensure that the facility evaluator is conducting an appropriate examination, f.
Af ter administration, the NRC shall resolve with the f acility representatives all unforeseen technical questions or issues that could i
result in an individual failing the examination.
The facility training
)
manager will be infonnally briefed on the preliminary results of the examinations.
i l
6.
Indivissal and Program Evaluation j
j j
a.
Individual Performance Criteria in order to be judged satisfactory on the walk-through portion of the examination, each operator shall:
i
)
Examiner Standards 00018.0.0 of 28 i
ES-401 Rev 5 10/1/08 (1) satisfactorily cseplete DOS of the JPMs seministered and 1
(t) correctly answer 70% of the questions related to the 10 tasks.
l b.
Program Performance Criteria (1)
If greater than 251 of the operators incorrectly answer more than 201 of the cosmon task questions (for example, 3 out of 10 questions),
a prograssnatic weakness is indicated in the area of plant systems f
knowledge and systems interactions.
(2)
If one or more facility evaluator (s) evaluates two or more j
operators as satisfactory when the NRC evaluator evaluates the j
sane operators as unsatisf actcry, the program will ce found to be unsatisfactory.
F.
Written Enanthation I
The purpose of the two section open reference written emarination is to assess i
the individual's kr.owledge of plant systan;s, procedures, and operating limits including the Technical Specifications.
I Section A is a ' Plant Operations' sectlen and is normally administered on a
)
static simulator.
Until full cos:pliance with 10 CFR 55.45(b), for facilities
]
without a simlator, section A should be administered in a control room mockup 1
if it exists.
If neither is available, then a classroom setting should be I
used.
MRC and the facility being evaluated will agree on the coepensatory
)
measures (computer printouts, photographs, etc.), required for a classroom or l
mockup setting. Section A is designed to evaluate the operator's knowledge of plant systems, integrated plant operations and instruments and controls.
In j
addition, recognition of Technical Specification (TS) LCOs and the operator's i
ability to diagnose postulated events shall be evaluated.
I l
i Examiner Standards 00019.0.0 of 28
_ _ _ _ ~
El 401 hev 5 10/1/08 Section B is a ' Limits and Controls" section and is shinistered in a classroom setting. Section B of th written enemination is designed to evaluate th ability of the operator to analyze a given set of conditions and determine the proper procedural and abinistrative guidance.
1.
Scope and Content i
a.
Section A - Plant Operations I
This section of the written examination is designed to utilize the i
sisolator as a reference tool in answering questions.
The questions I
should be rel6ted to plant systens, controls, and recognition of TS LCOs in an open reference format.
This section has a minimum of two "frozen
- conditions on the simulator, one condition being at power with some equipment in an abrorral status; one condition for which the plant has experienced a rajor transient resulting in ESFAS initiation.
i
,b.
Section 6 - Limits and Controls This section of the written esamination is designed to utilize plant procedures (including emergency, normal operation, and abnorral) and j
ahinistrative controls (including TS, E-plan, Administrative I
Procedures) in an open reference fonnat.
t 1
2.
Item Developrent and Review l
a.
Proposed examination itees shall be provided as an iteel bank by the i
facility.
Proposed itees should be reviewed for appropriateness, clarity, and importance to safety, as described in the "Guidelines l
l
)
J i
i Enartur 5tindards 00020.0.0 of 28
85 401 hey 5 10/1/34 for the Development and Review of Open Reference tassinattees.' Preposed items may be modified, deleted er replaced if Mcessary.
b.
A technical review of the references provided for each test item will be conducted to verify ites accuracy.
Each test item will be reviewed for question construction with emphasis on its applicability to an open reference examination using 'NRC Checklist for Open Reference Test items.'
I tech item should be reviewed for an associated learning objective.
Learning objectives should be verified as job related and relevant.
Review of lesson materiel for test items should be conducted to determine if the learning objectives are related to corresponding task (s) in the facility JTA and tbs l
K/A catalog.
The X/A rating should be 3.0 or above.
If a clear tie to the JTA does not exist, the applicability of the item shall be discussed with the f acility representatives.
Return any test iters which do not meet these criterid to the facility training departrent for modification.
l
}
Lipon coepletion of question revision, the facility representatives should review all test iters evaluating them for (1) apprcpriateness, (2) tire I.
required to answer each iten, (3) technical accuracy (4) clarity, and (5) K/A l
and objective references.
Af ter the facility representatives have completed the review, they will provide the results to the NRC for use in final I
examination development.
l 4
Serpling Plan J
The facility will provide a sampling plan (i.e., test specifications which identify the percentage to be saspled of each topic area) with their proposed j
i iters as described in F.2 above. The facility sampling plan should document I
the validity of test items by linking each ites topic with:
(1) a K/A with an importance value of 3.0 or greater (or equivalent safety rating from a facility l
JTA); (2) a facility learning objectives; and (3) safety-related tasks as I
1 Examiner Standards 00021.0.0 of 26
fl 401 Rev 5 10/1/88 identified by the facility JTA. The lutC's sampling plan will encrpass the facility's sampling plan with an addition of no more than 20s of items beyond the facility's sampling plan. The facility's saspling plan should indicate the percentage of items en Parts A and 8 of topics covered during the prior full reevalificatteet program. The plan should indicate the systematic approach to training basis and other materials used to develop the plan and document the comprehensiveness per 10 CFR 55.5g(a)(2).
5.
Construct Additional items should it be necessary to develop additional 1'.em a satisfy the sampling plan, request the f acility to do so.
Also, tha :.4 team r.ay develop itens as specified in paragraph 1.4.A above.
These iten should also be reviewed in accordance with this standard.
6.
Construct the Final Examination l
Using the tirei required to answer the test items provided by the f acility, construct an examination using the sampling plan.
A cerpetent operator should be able to corplete the emanir,ation in 1 bour and 15 minutes. Once the exarination is constructed, use Exar.iner Standard ES-107 to perform a QA review.
The E t team will review the final examination with the facility representatives for clarity and technical accuracy. Use the attached instructions and cover sheets for examination administration.
The examination shall be entered
)
into the EQB main bank af ter administ"* ion.
The enartijattor ray be constructed in alternate forms, i.e., each operator may have a differen't sequence of questions on his/her examination. This assists in e11einsting the need for having multiple sets of reference material.
Handouts (e.g., plant curves, blank forms, etc.) may be provided with the test to help relieve the burden on the utti'ty to provide additionci sets of reference
- material, i
Examiner Standards 00022.0.0 of 28
j ES-401 Rev 5 10/1/88 7.
Procedure for Parallel Grading Using the examination and toy, the facility and NRC will independently grade each section of the written examination. The grading of all written examina-tions shall be completed witt.in 15 working days of the examination administration date.
8.
Individual and Program Evaluation a.
'ndividual Performance Criteria In order to be judged satisfactory on the written portion of the examination, each operator must achieve at least 80% overall score as graded by the NRC.
b.
Program Performance Criteria In order for a f acility's raqualification program to be judged satisfactory, the following criteria must be met for the individuals evaluated:
(1) Ninety percent (905) pass / fail decisions agreement between NRC and facility grading, and (2) at least 80% of all operators pass the examination.
9.
Examination Proctoring Each section of the examination shall be proctored in accordance with ES-201.i and form ES-201-3 sb il be completed, f
10.
Reference Material The facility shall be responsible for providing the following reference materials:
Examiner itandards 00023.0.0 of 28
4 ES-401 Rev 5 10/1/88 During the ' Plant Operations" (Section A) portion of the examination, one copy of all controlled material available in the control room should be available to examinees. Examination reference material will NOT ine19de meterial that is intended for training use only. All reference material must be authorized for actual operation of the power plant.
During the ' Limits and Controls" (Section B) portion of the examination, each examinee shall have available for use the following matertal (complete, controlledcurrentissue):
a.
Technical Specifications l
b.
Plant procedures (EOP/A0P/0P, etc.)
c.
Energency Plan (as available in the control roor) i d.
Adrinistrative procedures applicable to operations l
.e.
Other plant reference material normally available in the control roon:
(e.g., curves and data book, forms, plant drawings, flow charts, I
etc.).
NOTE:
"Non-controlled" reference material, such as the Emergency Procedure Owner's Group Basis Documents will not be provided unless the facility certifies that these documents are authorized to be used during plant operations.
G.
Actions Required for Unsatisfactory Individual or Program Evaluation 1.
Unsatisfactory Individual Evaluation l
If an operator fails an NRC administered requalification examination, the operator shall be removed from licensed duties until remediation and Examiner Standards 00024.0.0 of 28
l l
ES-401 Rev 5 10/1/88 reexamination has been completed satisfactorily. INtC should shinister the test if the facility's requalification program is currently judged Ult 5AT15FAC-TORY and the stC has not verified that adequate corrective measures have been instituted. Otherwise, the facility will be permitted to administer the reexamination for returning the individual to licensed duties; however, license renewal would require another NRC administered exam.
2.
Unsatisfactory Requalification Program Evaluation For any program evaluated as unsatisfactory, the following actions are required unless findings indicate otherwise.
Additional actions may be taken at the discretion of the Regional administrator or his designee.
The sequence of actiuns below is not required.
The determination whether plant shutdown is required should be ongoing until the Regional Administrator or his designee has reviewed all in (d) below.
a.
Require the facility to identify program deficiencies and corrective actions required to improve operator performance, b.
Meet with senior facility management to review audit findir.gs, identified deficiencies, root causes, corrective actions proposed, schedule for corrective action implementation, and follow-up inspections and examinations, c.
Detennine the required corrective actions by the facility, the required follow-up by the NRC, and the schedule for each.
d.
The Regional Administrator or his designee shall determine whether plant shutdown pending completion of corrective action is required.
This decision should be based on the:
(1)
Significance of generic performance deficiencies identified during the program evaluation, Examiner Standards 00025.0.0 of 28
ES-401 Rev 5 10/1/08 (t) Recent SALF perforCr.ce, especially as related to Criterion 7 Training Effectiveness and Qualification, (3) Recent facility events which relate to licensed operator performance, and (4) Reconnendations by resident inspectors.
H.
Operator License Renewal Policy 1.
Licenses for operators and senior operators will be renewed upon timely application, as described in ES-109/110 if the individual in question has su:cessfully passed an NRC requalification examination within the term of his/her license.
2.
If an individual licensee has been administered an NRC requalification examination two (2) times during the terin of his/her license without pass-j ing any examination, then his/her license will be terminated or will expire without renewal and he/she may apply for license under 10 CFR 55.31 1
and sucessfully cceplete an NRC license exarrination to maintain a license.
1 3.
If an individual has not successfully passed an NRC requalification
. examination within the term of his/her license but has failed an MRC requalification examination fewer than two (2) times, then the individual license is extended under the timely application provisions until the next NRC requalification examination. Af ter the results of that examination, the individual will confort with either H.1 or H.2 above.
4.
If an individual has passed one (1) MRC requalification examination curing the ters of his/her license then the provisions of H.1 apply without regard to the facility requalification program status or any subsequent or prior unsuccessful NRC requalification examination (s).
Examiner Standards 00026.0.0 of 28 lu-
ES-401 Pev 5 10/1/88 I.
Final Regeelification Program Evaluation Report A final requalification program evaluation report similar to the final examina-tion report for a licensing examinktion shall be prepared when the grading of requalification examinations has been completed.
A complete copy of the report shall be filed in the facility requalification file. A copy of the NRC Admin-istered Requalification Examination Results Sumary, shall be forwarded to the Management Assistant, Regional Support and Cversight Section, OLB. The results sumary is required to verify OLTS data and for statistical data.
J.
Record Retention 1.
A facility requalification file shall be maintained for each facility.
All evaluation forms, records, assignment sheets, and correspondence j
I relating to the requalification program audit for the latest two evaluations shall be retained.
, hen the requalification evaluation has been completed by the Regional 2.
W Office, a copy of all NRC administered written, oral and simulator examination results shall be supplied to the facility.
The facilities are
- required to maintain these records until the operator's or senior operator's license is renewed in accordance with 10 CFR 55.59, i
3.
'A copy of the results sumary shall be sent to the Management Assistant, Regional Support and Oversight Section, OLB.
These sumaries shall be used for statistical data gathering.
4.
Material relating to an individual failure shall be retained by the Regional Office as necessary to support denial of license renewal per 10CFR55.67(b)(2)(iv).
Examiner Standards 00027.0.0 of 28
E5-401 Rev 5 10/1/08 Attachments:
1.
Corporate Notification Letter 2.
Examination Process Critique Sheet 3.
Simulator Crew Evaluation Fors 4.
JPM Work Sheet 5.
JPM Question Form 6.
Briefing Sheet 7.
Written Examination Cover Sheet 8.
JPM Quality checklist 9.
Simulator Scenario Review Checklist 10.
Guidelines for the Development and Review of Open Reference Exaninations 11.
NRC Checklist for Open Reference Test items I
l Examiner Standards 00028.0.0 rif 28
ATTACMMENT 2 EXAMINEE CRITIQUE OF NRC REQUALIFICATION EXAMINATION As part of its efforts to improve all aspects of the NRC requalification examination process, NRC is asking all test-takers to provide feedback on these examinations.
Please use this form to comment on the NRC requalification examination that you have just completed.
After you have answered the questions below, and provided any comments that you wish to make, please give this form to your training manager who will send it to the NRC Regional Branch Chief.
Please take a few minutes and respond to each of the following questions, by circling the number that best represents your experience on the examination just completed.
Each question is written in the form of a statement, and the number which you circle indicates how strongly you agree or disagree with that statement.
For example, if you circle number 1 for a particular question, that indicates that you are in strorig agreement with the statement.
If you circle number 4, that indicates mild disagreement.
If you circle number 3, that indicates that you neither agree nor disagree with the statement made in that question.
THERE ARE NO RIGHT OR WRONG RESPONSES.
Your participation is strictly voluntary, but please remember that your responses, together with any comments that you might have, will help to improve the requalification examination process.
All responses are anonymous.
WRITTEN EXAMINATION:
1.
The content of the questions reflected the content of the requalification
,progran at my plant.
Strongly agree Strongly disagree 1
2 3
4 5
Comments:
l 1
)
2.
The overall time allotted for the examination was sufficient to answer all the questions.
Strongly agree Strongly disagree 1
2 3
4 5
Comments:
3.
The information to be used to answer each question was available.
Strongly agree Strongly disagree 1
2 3
4 5
Comments:
4.
There was an opportunity for me to have any questions regarding the
. examination clarified prior to the end of the examination.
Strongly agree Strongly disagree 1
2 3
4 5
Comments:
I l
2 1
1
OPERATING TEST:
1.
The content of the JPMs and the qwstions on the walk-through portion was consistent with the conient of the requalification program at my plant.
Strongly agree Strongly disagree 1
2 3
4 5
Comments:
2.
The overall time allotted for the walk-through portion of the operating test was appropriate.
Strongly agrae Strongly disagree I
2 3
4 5
Comments:
3.
The scenarios used for crew evaluation on the simulation facility portion of the operating test were consistent with team (crew) training at my plant.
Strongly agree Strongly disagree 1
2 3
4 5
Coments:
J 3
)
4.
The instructions for the conduct of both portions of the operating test were clearly explained by the exami mr(s).
Strongly agree Strongly disagree 1
2 3
4 5
Comments:
ADDITIONAL COMMENTS ON THE CONTENT AND/0R CONDUCT OF THE NRC REQUALIFICATION WRITTEN EXAMINATION AND OPERATING TEST JUST COMPLETED:
PLEASE CIRCLE THE LEVEL OF THE EXAMINATION JUST COMPLETED:
R0 SRO 1
PLEASE DO NOT INDICATE YOUR NAME ON THIS QUESTIONNAIRE.
THANK-YOU.
This request is covered by Office of Manage.nent and Budget Clearance Number 3150-0011 which expires December 31, 1989.
Comments on burden and duplication may be directed to the Office of Management and Budget, Room 3208, New Executive Office Building, Washington, DC 20503.
4 4
1 l
ATTACMENT 3 INSTRUCTIONS ON USE OF $1MULATOR CREW EVALUATION FORM Enclosed is an evaluation fom for use during the trial r.inulator examination component of the requalification examination.
In keeping with the purpose of i
the requalification exam, these scales are geared toward evaluating the crew as a whole, rather the.n individual operators.
Please follow the instructions below when rating team performance on the simulator examination:
1.
heview the rating scales prior to the onset of the simulator examination to familiarize yourself with each performance issue to be evaluated.
2.
Use the "Operator Actions" Form (ES-302, attachment 6), and the expected operator actions included on that form, to make notes (Jurino the examir.ation, as described in ES-302.
3.
Isenediately after the simulator examination is over, evaluate the crew by completing the Simulator Crew Evaluation Form.
Be sure to address all the 1
rating factors for all 6 competencies.
4.
Provide an overall rating of "satisfactory" or "unsatisf actory" for each competence.
If you gave the crew more than one rating of "1" on the rating factors for that competence, your score for them on that competence overall should be "unsatisfactory." Although part of the purpose of the walk-through is to follow up on areas of weakness noted during the simulator portion of the examination, if the crew receives a rating of "unsatisf actory" on one or more competences, their overall evaluation should also be l
unsatisfactory, 5.
Tnere is space for comments beneath each competence rating and below the overall rating.
This space is provided if you feel the need to annotate or explain rating (s).
In particular, use this space to document the failure of an individual candidate due to his/her exceptionally poor performance.
We need your feedback to evaluate the usefulness of these ratings scales.
Af ter ycu have had an opportunity to use the form, please provide us with the following information:
Were the scales clear, concise, and easily understood?
j Were the scales written so that you cnuld evaluate crew (versus individual) performance?
Could you comfortably decide which ratings to give the crew, based on the decriptions of the performance provided?
If not, how could these descript. ions be improved?
Were there rating issues which you believe are not appropriate for simulator requalification examinations?
Were there any performance issues missing from these rating scales?
Please note your comments and suggestions on the page provided at the back of the evaluation fom and pt ovide them to John Hannon at the exit meeting.
Thanks very much for your assistance.
5
4 51MULATOR EXAMINATION SLM448Y $NEET CREW ME M ERS:
I Name Position i
4 OVERALL TEAM RATING ON THE SIMULATOR EXAMINATION:
Satisfactory Unsatisfacto.y Comments:
i Please use the space below to note your comments on the evaluation form (attach additional pares, if n' cessary):
)
l i
j 1
6
UNDERSTANDING / INTERPRETATION OF ANNUNCIATOR / ALARM $1GNALS DID THE CREW.
(a) NOTICI and ACKNOWLEDGE alares, anti ATTEND T0 alares in order of their importance/ severity?
3 2
1 All alares that directly Minor awareness or Failed to notice related to significant response difficulties and/or extremely changes in plant or lapses slov at responding conditions vere noted to significant alarms at critical times; easily distracted by nuisance alarms (b) Correctly INTERPRET the meaning and significance of alarms and annunciators (including the use of the Alarm Response Procedures, as appliccole)?
3 2
1 Crei readily determined Minor inaccuracies in Significant misin-what failures / events alarm interpretation terpretations, alarbs were indicating but without safety resulting in plant related consequences degradation (c) l VERIFY that annunciators / alarm signals were consistent with plant / system conditions?
3 2
1 All necessary verifi-Minor lapes in alarm Verification of
]
cations performed, verification, but no failed systems in:1uding the identi-inappropriate actions was poor or ficetion of erroneous taken as a result of altogether absent alarms inadequate verification SCORE ON UNDERSTANDING / INTERPRETATION OF ANNUNCIATORS / ALARM SIGNALS:
Satisfactory Unsatisfactory Comments:
h-I 7
4 e
DIAGNOSIS OF EVENTS / CONDITIONS BASED ON $1GNALS/ READINGS 010 THE CREW:
(a)
RECOGNIZE off-normal trends / status?
3 2
1 Timely and accurate Recognition of trends Failed to recognize recognition of trends at time of, but not trends, even after even prior to alares prior to, sounding of sounding of alarms and alarms annunciators (b)
USE INFORMATION and use REFERENCE MATERIAL (prints, books, charts) to aid in the diagnosis / classification of events and conditions?
3 2
1 Correct, timely use Minor errors by crew Failure to use of information and in use or interpretation reference material, reference material of information and misuse / misinterpretation led to accurate reference material of information resulted diagnoses in improper diagnoses (c). Correctly DIAGNOSE plant conditions based on those control room indications?
3 2
1 Diagnoses by crew Minor errors /diffi-Faulty diagnoses were accurate and culties in diagnoses resulted in timely incorrect control manipulations SCORE ON DIAGNOSIS OF EVENTS / CONDITIONS BASED ON SIGNALS / READINGS:
Satisfactory Unsatisfactory Comments:
1 8
UpeERSTANDING OF PLANT / SYSTEMS RESPONSE DID THE CREW:
(a) LOCATE and INTERPRET control room indicators correctly and afficiently to ascertain and verify the status / operation of plant systems?
3 1
1 Accurate and efficient Minor errors in locating Serious omissions instrument location &
or interpreting instruments delays or inaccuracies interpretation by all and displays; some crew made in instrument crew members members required assistance interpretation (b) Demonstrate an UNDERSTANDING of how the plant, systems, and components operate, including setpoints, interlocks, and automatic actions?
3 2
1 All crew members demon-Minor instances of errors Inadequate knowledge strated thorough dee to gaps in crew of system / component understanding of how knowledge of system /
operation resulted in systems / components
- omponent operation; some serious mistakes or operate crew members required plant degradations assistance (c). Demonstrate an understanding of how their ACTIONS (or inaction) affected system / plant conditions?
3 2
1 All members understood Actions or directives Crew appeared to act the effect that indicated minor without knowledge of, actions or directives inaccuracies in under-or disregard to, effect had on plant / system standing by individuals, on plant conditions but actions were corrected by team SCORES ON UNDERSTANDING OF PLANT / SYSTEM RESPONSE:
Satisfactory Unsatisfactory Coeveent:
l 9
COMPLIANCE /USE OF PROCEDURES AND TECHNICAL $PECIFICATIONS DID THE CREW:
(a) REFER To the appropriato procedures in a timely manner?
3 2
1 Crew used procedures Minor failures by Failed to correctly as required; knew crew to refer to refer to procedures when what conditions were procedures without required, resulting in covered by procedures prompting, but did faulty system operation and where to find them affect plant status (b) CORRECTLY IMPLEMENT procedures, including following procedural steps in correct sequence, abiding by cautions and limitations, selecting correct paths on decision blocks, and correctly transitioning between procedures?
3 2
1 Timely, accurate Minor instances of Importance procedural enactment of procedural misapplication, but steps were not enacted steps by crew, corrections made in correctly, which led demonstrating thorough sufficient time to to impeded and/or slow understanding of avoid adverse impact recovery or unnecessary procedural purposes / bases degradation (c) '. RECOGNIZE E0P ENTRY CONDITIONS and carry out appropriate immediate actions without the aid of references or other forms of assistance?
3 2
1 Consistently accurate Minor lapses or Failed to accurately and timely errors; individual recognize conditions recognition and crew members needed or execute actions, implementation assistance from others even with use of aids i
to implement procedures (d) CORRECTLY RECOGNIZE and COMPLY with Technical Specifications and Action Statements of LCOs?
3 2
1 Recognized and Minor difficulties in Failure to recognize /
fully complied with referring to and/or comply with Tech Spec LCOs/ Action Statements applying Tech. Specs.;
LCOs crew had to prompt SRO on TS requirements SCORE ON COMPLIANCE /USE OF PROCEDURES AND TECHNICAL SPECIFICATIONS:
Satisf actory Unsatisfactory Comments:
10 I
r
_.. - _ _ _ ~,_ _ _
CONTROL 90ARD OPERATI00t$
DID THE CREW:
(a)
LOCATE CONTROLS efficiently and accurately?
3 2
1 Controls and Instances of Instances of failure indicators were hesitancy /
to locate controls located without difficulty in jeopardized system hesitation by locating controls status individual operators by one or more operators (b) MANIPULATE CONTROLS in an accurate and timely manner?
3 2
1 Smooth manipulation Minor shortcomings Mistakes made in of the plant in manipulations, manipulating controls within controlled but recovery from caused system parameters errors without transients and causing problems related problems (c) ~ Take KANUAL CONTROL of automatic functions, when appropriate?
3 2
1 All operators took Minor delays and/or failed to control cont,rol, and smcothly prompting necessary automatic systems operatored automatic before overriding /
manually, even when systems manually, without operating automatic aniple time and assistance, thereby functions, but plant indications existed averting adverse events transients were avoided when possible SCORE ON CONTROL BOARD 09ERATIONS:
i Satisfactory __
Unsatisfactory Comments:
i 11
COMUNICATIONS/ CREW INTERACTIONS DID THE CREW:
(a) EXCHANGE complete and relevant infonnation in a clear, accurate, and attentive manner?
3 2
1 Members informed each Communications generally Members did not other of relevant info.
complete and accurate, inform each other and actively sought and but some instances of of abnormal indica-listened to info. from needing to be prompted, tions or when others as/when necessary or failing to acknowledge performing evolutions; or respond to info, from inattentive when others important info. was requested or provided (b)
INTERACT with other regarding issues / circumstances outside of their individual area of responsibility to facilitate safe plant conditions?
3 2
1 Members assumed Members listened to Members were responsibility for each others conversations inattentive to what issues outside their in general; major technical was happening own boards, as errors corrected around them; poor appropriate coordination of activities (c),MAKE TEAM DECISIONS in a timely, ef fective manner?
3 2
1 All' individuals provided Major team decisions Leader or other crew 1
input to decisions, generally included members did not ace.ept Decisions resulted in early, input from most crew input froe others, recuperative action members, but some resulting in incorrect delays or other or untirely decisions /
problems in reaching directives effective decisions SCORE ON C0mVNICATIONS/ CREW INTERACTIONS:
Satisf actory Unsatisfactory Comments:
12
ATTACHMENT 4 JOB PERFORMANCE MEASUME WORKSHEET Facility:
Task
Title:
Task No.:
K/A
Reference:
Operator:
Evaluator:
Date:
Applicable methods of testing:
Simulate performance Actual performance Cla,sroom Simulator Plant READ TO THE OPERATOR I wiki explain the initial conditions, which step (s) to simulate or discuss, and provide initiating cues.
When you complete the task successfully, the objective for this job performance measure will be satisfied.
Initial Conditions:
Task Standards:
Required Materials:
Generil
References:
Initiating Cues:
13
PERFORMANCE INFORMATION (Denote critical steps with a check mark)
Performance step:
1 Standard:
Comment:
Performance step:
Standard:
Comment:
Performance step:
i Standard:
Com: tent:
l
. Performance step:
Standard:
l Comment:
i Performance step:
Standard:
j l
Comment:
f 1
Terminating cue:
14 l
VERIFICATION OF COMPLETION Job Performance Measure No.
Operator's Name:
Date performed:
Evaluator:
Number of attempts:
Result:
(Denote by an 5 for satisfactory or a U for unsatisf actory and requires remedisi training).
1 Evaluator's signature and date:
l l
(
i 15 i
rr------
.-,p.
w
,w_--
g-
ATTACMENT 5 SYSTEM ndhtK-THROUGH TEST PLAN Page_ of_
Facility:
System Topic:
Performance Measurement:
Question
_ Rating 1.nswe r:
Re spor.se/ Comment s:
Question
_ Rating Answer:
Response / Comments:
Question
- KA Rating Answer:
l Response /Connents:
Question
- KA Rating Answer:
i Response /Coopents:
~
16
$YSTEM tmLK-THit00GH FOLLEUP 00CLMENTAT10N Question Berivation (can be completed after examination):
Simulator Evaluation Deficiency (identify)
Topic Area Evaluation: 1. Performance Measurement
- 2. Question Number Question (to be completed during exam):
Response (to be completed during exam):
Question Derivation (can be completed after examination):
Simulator Evaluation Deficiency (identify)
~
Topic Area Evaluation: 1. Performance Measurement
- 2. Question Number _
Question (to be completed during exam):
Response (to be completed during exam):
Question Derivation (can be completed after examination):
Simulator Evaluation Deficiency (identify)
Topic Area Evaluation: 1. Performance Measurement
- 2. Question Number Question (to be completed during exam):
Respon:,e (to be completed during exam):
Question Derivation (can be completed after examination):
Simulator Evaluation Deficiency (identify)
Topic Area Evaluation: 1. Performance Measurement 2, Question Number Question (to be completed during exas):
i Response (to be completed during exam):
f 17
1 ATTAC MENT 6 BRIEFING CHECKLIST - SYSTEM WALK-THROUGH 1.
If the NRC examiner is a visitor, escort responsibility for ensuring compliance with safety, security and radiation protection procedures is the resper.sibility of the operator escorting the examiner.
2.
Plant equipment thould not be, operated.
Nothing the facility or NRC exteiner says or a:h vill M intended to violate that principle.
3.
If clarification of questions is needed during the walk-through, there should be no hesitation to request the examiner reword or clarify the question.
4.
The exariner will be taking notes throughout the test to document operator performance.
Frequently an examiner will stop questioning for this purpose.
The amount of note-taking is not dependent upon the operator's level of performance. The examiner must document satisfactory as well as less than satisf actory performance.
5.
The walk-through is considered "open book " The reference material in the f.scility/ control room which is normally available to operators is available, including calibration curves, previous log entries, piping and instrumentation
' diagrams, calculation sheets, and procedures.
However, operators are respons-ible for knowing from memory the immediate actions of emergency and other
', procedures as appropriate to the facility.
6.
'The system walk-through has been planned for approximately two hours in length.
However, there is no specific time limit for the walk-through.
lThe examiner will take whatever time is necessary to cover the areas selected, in the depth and scope required.
There will be a minimum of 4
, Job Ferformance Measures (tasks) evaluated from the e.ontrol room and 4
, Job Performance Measures evaluated outside of the control room.
- However, the total number of JPM's will be no less than ten (10).
7.
The examiner will explain what tasks are to be completed, which steps to simulate or discuss and provide initial conditions.
The operator is to i
proceed with completing the task as if directed by plant procedures and/or shif t supervision.
During the task the examiner will supply the necessary t
plant conditions and/or parameters needed to simulate the task.
The operator should explain each step of the task to the examiner before doing it.
4 8.
When all of the steps for each task are performed correctly, the criteria for the examination will have been completed, s
9.
If the operator feels the need for a break during the walk-through, the operator should request this from the examiner.
The examiner is not l
allowed to reveal the results of the walk-through at its conclusion.
10.
The lac examiner may ask clarifying questions of the operator at the end of each JPM.
1 j
18 l
Part B - For tests with simulation facility available 1.
The primary responsibility is to operate the simulation facility as if it were the actual plant.
1 2.
Team work and communication between operators is evaluated.
It benefits the exam process to verbalize observations, analysis, and reasons for actions more than normally would be done during actual plant operations.
3.
If an operator recognizes an incorrect decision, response, answer, analysis, action taken, or interpretation of the team of which the operator is a part but fa.Ils to correct, then the examiner may assume that that operator agrees with the incorrect item.
4.
A rough log may be kept during each exercise that would be sufficient to complete necessary formal log entries which may be evaluated under admin-istrative topics.
5.
A designated facility instructor will act as the auxiliary operators, radiation health and chemistry technicians, maintenance supervisors, plant management, and anyone else needed outside the control room area.
6.
The facility examiner will provide a shif t turnover before the exercise begins.
The shift turnover will include present plant conditions, power history, equipment out of service, abnormal conditions, surveillance due,
'and instructions for the shift.
7.
'The control board switches may be purposely misaligned to enhance a simulated scenario or transient where appropriate and is not part to the
' evaluation.
If misaligned, they should be tagged or otherwise highlighted as appropriate to the f acility.
The examiner will not misalign switches
'during the scenario as an awareness drill.
Note: The chief examiner has the option to tell the operators that no control board switches will be misaligned on a given scenario or set of scenarios.
If no switches are misaligned, the chief examiner may wish to reduce the time it takes of the operators to complete the board walkdown and accept the shift.
8.
Operators will be allowed three to five minutes to familiarize themselves with the status / conditions of the control boards prior to the start of the exercise.
9.
The simulation facility part of the examination will consist of a minimum of tae exercises lasting approximately 50 minutes each.
There will be a short break between exercises to set up the initial conditions for the next exercise.
10.
If operators have any questions concerning the administration of the operating I test, those questions should be answered prior to the start of the test.
19
ATTAC E NT 7 U. 5. NUCLEAR REGULATORY COMIS$!0N REACTOR OPERATOR REQUALIFICATION EXAMINATION l
FACILITY:
REACTOR, TYPE:
f DATE ADMINISTFRED:
I OPERATOR:
7 1 0T SECTION CATEGORY OPERATOR'S CATEGORY f
VALUE SCORE VALUE l
A Plant Proficiency l
B Limits and Controls l
1 Final Grade O
I i
i 20 s
ATTACwENT S JOB PERFORMANCE BEASURE QUALITY CHECKLIST 1
1.
Supported by facility's job / task analysis.
Operationally important (meets threshold criteria of K/A 3.5 or as 2.
determined by the facility and agreed to by MRC).
3.
Designated es either SRO only or both R0/SRO.
Time validated (time allowed for task completion indicated on JPH).
4.
Questions related to the task need not be time validated.
5.
Either the task itself or the questions that follow require operator pretier, recognition and diagnosis.
Questions require knowledge correct
. actions for abnorral system responses whenever applicable.
Performance standards are specific in that exact control and indication 6.
nomenclatureandcriteria(switchposition,meterreading)arespecified, even if such criteria are not specified in the procedural step.
Critical elements and associated performance standards are identified and 7.
agreed to by the facility and the NRC.
t 8.
Performance standards should provide complete and proper systee response cues where appropriate such that the examiner can properly cut the operator when asked.
s
1
.t.
Answers to questions at the end of the task are 100T found directly in the 9.
procedure just used (i.e., do not ask why a certain caution exists if the caution itself identifies the answer).
i d
9 9
9 I
i I
I i
i l
l I
}
ATTAD9 0 7 9 Rev. 1 4/15/98
$1PULA,'OR LCENAR10 REVIEW CHECKLIST (Attach separate copy to each scenario reviewed)
SCENARIO IDENTIFIER:
REVIEWER:
Scenariocontainsatleastone(1) time-criticelcrewresponse.
1.
Time-critical is defined to require affirr.ative action by one or more crew member (s) in order to prevent or mitigate an event within a limited time.
Scenario contains et least one (1) team dependent :rew response.
2.
Team-dependent is defined to require transfer of information betweer crew members in order to prevent or mitigate an event.
3.
The scenario events involve each crew member.
4.
Scenario is composed of related or linked events.
__ 5.
Scenario requires the use of: (check those that apply)
Abnonnal operating procedures Emergency operating procedures Technical Specifications Energenr.y plan implementing procedures
i t-6.
Critical tasks aret i
Identified by facility 1
i Required to prevent or mitigate event Importance factor 3.5 in K/A Catalog er facility JTA 3.0 Scer.ario events have K/A importance factor 7.
i 7
e P
1 P
i i
j i
'l 1
I i
I I
e-
__y__
=
l ATTAC MENT 10 GUIDELINES FOR THE DEVELOPMENT AND REVIEW 0F OPEN REFERENCE EXAMINATIONS
!. Introduction The following guidelines are intended for thnse who are involved in the development and/or review of test items for the written portion of the NRC Requalification Examination.
As described in ES-601, "Administration of NRC Requalification Program Evaluations,"
the written examination consists of two sections, one that utilizes the simulator to provide a context for questions on plant systems and controls, and a second that focuses on plant procedures and administrative controls.
Both sectiont are administered in an "open reference" femat.
Candidates are allowed to use reference material, ine.luding simulator displays, during examination administration.
The intent of the change from closed to open reference written exarinations, is twofold:
(1) Examination Validity.
By permitting the use of references that are available to the rprator, the conditions and requirements of the written examination more closely approximate those of the actual job.
The information provided to the operators in the test items can and should closely parallel the information typically available to them on the job, while the responses elicited by the questions should be similar or identical to the
)
decisions, solutions, and actions required for effective job performance.
In other words, the open reference format enhances the match between M demands and test demands -- a cornerstone of examination validity.
i
(2)Leve_1gfKnowledee. N open reference format also enhances examination validity by elevating the level of knowledge of the test itest. As described later in these guidelines, candidate access to references preelddes the use of questions that test for the mere recall of facts and specifics.
Instead, open reference test items require test takers to demonstrata that they con apply, analyze, evaluate, or otherwise USE knowledge to handle the problems and issues encountered on the job.
11.
Open Reference Guidelines Most principles for effective test item construction apply equally to all types of written questions, regardless of fomat. Therefore, open reference test item developers and reviewers should consult references such as NUREG BR-0122 "Exaniners' Her.dbook for Developing Operator Licensing Examinations,"
INP0's "Principles of Training System Developrent Addendum I, Test Item Developmert,* and NUMARC's "Guidelines for Developing Written Test items for the NRC Requalification Exam." The guidelines below are those that have been found to be especially pertiner.t to the creation of open reference test iters.
These guidelines are divided into five categories:
l (1)SelectionofTestTopics (2) General Guidelines for Sections A & B (3)SpecificGuidelinesforSectionA,"PlartOperations" (4) Ideas for Open Reference Formats (5) Open Reference Test Iten Review Checklist (1)SelectionofTestTopics f
Test item topics for the NRC requalification examination should be selected based on the following criteria.
A.
Requalification Training Program Curriculum. Test topics should be based l
on the curriculum of the most recent operator requalification program training cycle. However, NRC may substitute up to 201 of the examination topics selected by the facility with subjects not emphasized durirg the requalification cycle under 10 CFR 55.59.
l
5.
Perfomsace Basis Like the requalification program itself, test topics should be dream from a job-task analysis (JTA) of the operator and senior operator positions. The facility should validate their test itses by demonstrating a link between each item and the following JTA products:
/
important operator tasks as identified by the JTA important K/As (rated 3.0 or higher) as identified in the NRC K/A Catalog (NUREG 1122/1123) or a facility-specific K/A catalog facility learning objectives identified as important to safety i
Adequacy o_f, Test Coverage. Thefacility'sproposedsamplingplan(or f
C.
curriculurr evaluation plan) should by checked to ensure that it provides balanced, cacprehensive coverage of the topics covered during the requalificatien training cycle.
Facility test item topics may be revised if subject areas are found to be under or over-represented in the sampling plan relative to their coverage in the requalification program.
In addition, 201 may be substituted by the NRC.
Recent safety-related issues and events (e.g., relevant LERs) 4 should be addressed in the sampling plan.
(0) heneralGuid*%es The following guidelines should be followed in the construction and review of test items for both parts of the written examination. These guidelines are l
intended to supplement, not replace, the good practice criteria found in NUREG BR-0122 and related docunents.
t l
-l A.
Operational Orientation. As discu:; sed earlier, examination validity is j
enhanced to the extent that the demands of the test match the demands of the
(
l job. Therefore, in addition to being derived from important K/As and testing objectives, the context and stipulations of test items should mirror the situations encountered in the work setting.
The following example illustrates l
effective and ir. effective ways to design test items f om K/As and learning
)
objectives, i
l i
K/A:
Knowledge of the design attributes of the Turbine Driven Auxiliary Feedweter Pump Differential Pressure Controller.
Task: Operate the TDAFWP controls during all modes of plant operation.
Terminal Learning Objective:
The student will be able to operate the TDAFWP Differential Pressure Controller without error during a loss of feedwater event.
EnablingObjective:
Upon completion of this lesson, the student will be able to explein the operation of the TDAFWP Differential Pressure Controller.
Poor Test Item:
State the parameters used by the TDAFWP Differential Pressure Controller.
l Better Test Item:
Prior to isolating the 'C' steam generator (per EPP11), it was n,oted that the transducer. fed auxiliary feed flow indicators for the "C" i
steam generator were reading greater than the flow indicators to the 'A" and l
- B" steam generators.
What is the reeson for this flow deviation?
Notice that the second test item requires the candidate to demonstrate mastery of the knowledge by applying it to an actual job situation.
In developing items, it may be useful to ask oneself "why is the K/A important to 1
satisfactory job performance?" and 'in what situation will be operator need this K/A?' The answers to these questions can provide a basis / context for test items.
s j
B.
Level of Knowledge.
The operational orientation required of test items on the open reference examinations, as well as the candidates' access to controlled documents, precludes the use of qcestions that test for mere recall or memorization.
Rather than requiring candidates to simply recognize or recall facts and specifics, open reference test items should have the i
candidates demonstrate understanding by requiring them to use their knowledge to address real. life situations and problems. A test itse at the
higher level of knowledge requires candidates to determine or identify the l
appropriate fact, rule, or principle to a novel siteetion and then correctly apply it. A description of each level of knowledge, elong with cosmon verbs and example westions is found in Table 1.
C.
Realistic Context.
To provide additional assurance of examination validity, the situation or problem posed in the open reference test itse should be as similar as possible to the actual situations that candidates j
encounter on the job.
Situations described ia the questions should not only be realistic, but should also be free of connon "context" problems, including "bachards logic" and "window dressing."
Backwards logie questions provide candidates with information they nonrtally For have to produce, while asking thern for information they normally receive.
example:
K/A:, Ability to calculate shutdown margins.
Bachard Logic iter.:
Given a shutdown margin is 5.51, how long has the unit i
been shut down?
Better Iter:
The unit has been shut down for x hours.
Calculate the shutdown margin.
1 1
Questions with window dressing have additional, unnecessary infonnation, typically in an attempt to make a memory level ites more operationally oriented.
For example:
1 i
l i
i l
4
Item with Window Dressing: The plant has tripped due to the effect of a tornado crossing the site boundary. You, as Shift Supervisor, direct the phone talker to cosplete the 15 minute notifications. He informs you that the nomal notification network is inoperable. What method do you direct him to use for completing the 15 minute notification?
Revised Iter:
If the normal notification network is inoperable, what method do you direct the phone talker to use to complete the 15 rinute notification af ter the plant has tripped?
Another comon problem when constructing questions with realistic contexts is that quite of ten "real world" situations have more than one correct solution or response.
Check the question and references over carefully to ensure that each ' test item has only one correct answer.
D.
Questior Novelty.
One of the most effective ways to ensure that candidates have higher levels of knowledge is to present them with nevel situitions and recuire them to both realize what information is relevant ano how to apply it.
If test questions do not contain unique or varied i
circumstances compared to that which was presented in training, the item will be reduced to simple recall.
Review the training material to ensure that questions do not include overly familiar conditions.
Keep in mind, however, that all conditions and situations should be reasonable, realistic, and safety.related.
Use o_f References.
References should be considered tools that candidates f
E.
use to solve problems.
It should be the proper use of these tools that is 1
tested during the open reference examination, not the recall of facts and specifics. Purely "look up' questions should not be included in the t
examination; rather, questions should be restricted to those that test to see if candidates can identify, locate, or select appropriate reference information to produce organized responses and satisfactory solutions to job j
related problems and issues.
For example.
l
Pure "Look up' Itse:
In the event that a safety limit is violated, the reactor shall bei a.
pit.ced in a hot shutdown condition within 1 hou-b.
placed in a tot shutdown condition within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> c.
placed in a cold shutdown condition within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> d.
placed in a cold shutdown r.ondition within 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />, i
Better Use of
References:
While operating at 1001 power, VCT and pressurizer alarms and indications show decreasing pressurizer level. Also, the blowdowr and main stean radiation monitors have alarmed.
While following the i
appropriate Abnormal or Erergency Procedures, yeu as the Shift Supervisor must evaluate the existing conditior.
Based on this information, the following emergency classification should be declared:
,a.
kctification of Unusual Event
'b.
Alert c.
Site Area Emergency d.
General Energency F.
Difficulty Level.
It is not unusual for test constructors to believe, erroneously, that open reference test items should be sione difficult, to corpensate for the candidates' access to reference material.
Frequently, this increased difficulty is in the form of requiring knowledge of more obscure or otherwise unnecessary inforration. Both open and closed reference examination items should have the same standard of difficulty; that it, difficulty should be based on the job demands and responsibilities of operators.
G.
Time Limits.
Relative to closed reference examinations, candidates take considerably longer to answer open reference test items.
(Weaker candidates especially have been found to spend an appreciable amount of exam time l
\\
consulting references versus writing responses).
It is tapertent to provide candidates en ample amount of time to complete the examination, although not so much tiet as to allow less than competent operators the opportunity to locate answers without prior familiarization. The following guidelines should be used to determine the appropriate length of the er. amination:
1.
Each two hour examination section should be constructed to take a competent operator an expected I hour 15 minutes to cosplete.
2.
Wher. possible, the response tine of each ques'. ion sheuld be estimated by havir.g a subject matter expert actually answer the question, including searching through references.
H.
Correct r. ode of Measurement.
No matter how high their importance i
ratings or operatier.a1 relevance, certain operator knowledges, skills and abilities are not amenable to written testing.
For example:
1 Arrange the major steps in the proper secuence to start, parallel, and 1
IoadDG.2:
l l
Use Governer Control to increase DG-2 KW Raise DG eed to 900 RPM Press start button on A130B Match Voltage with Bus 1A2 Voltage j
Close Breaker IAD2.
1 i
l Despite its operational orientation, the underlying skill addressed in the
)
j above test item would be better assessed by having the candidate sirolate or step through the steps during either the sisolator or walkthrough portions of the operating examination. Table 2 provides en overview of the best uses for each test mode. Make sure that the K/As and learnin0 objectives selected for the written examination can be effectively measured during the written testing.
i
$pecific Guidelines f_or Section A. "Plant Operatices' (3) o 1
The following guidelinet are specific to the Plant Operations section of the written examination, performed on a static simulator. These guidelines are divided into two sections, Question Development and Simulator Setup.
Question Development e.
To ensure that the operators' knowledge of plant operations is adequately evaluated, Section A of the written examinetion should address a mix of normal, abnonnal and emergency rodes of operation, b.
Questions should require the operators, to the extent possible, to refer to control room indications in forr.ulating their responses, as in the following example:
yhich one of the following describes the location of the steam break?
(a)
Inside containment, upstretin of the steam line flow transmitters I
('b )
Inside containment, downstrean of the steam line flow transmitters (c) Outside containment, between 'C' M51V and 'C' main steam line check valve.
(d) Outside containment, between 'C' MSly and 'C' main steam line containtnent penetration.
I c.
The neer of scenarios used should be minimized due to the extensive amount of time necessary to set up, run and check the transients, i.
i d.
The number of malfunctions / failures for each scenario should be l
limited.
In general, the scenario should contain one major failure (e.g.,
f LOCA, $bTR, steam line breaks, ejected control rods, loss of all AC power).
In addition to the major failures, no more than four minor failures should be used (e.g., failure of a safety related pump to start, failed pressurizer i
1 l
I pressura meter indication, nucletr instrumentation failure). In many cases, one mejor failure and 2-3 minor failures will provide sufficient effects to test a wide range of objectives.
e.
Questions may be used that do not relate to the transient but use the simulator as a frare of reference only, provided the candidates are aware of this lack of relationship to the transient.
f.
Special attention should be given to ensure that multiple questions sterming fror one event do not suffer from double jeopardy. The candidate should be able to be understand and correctly answer each question based only en the inforv.ation given in the question, rather than on the answer to a previous question.
Sinuletor Setup g.
Prior to the test, the simulator recorders should be rotated to provide clean readings, and the recorders should be checked for proper operation, b.
All indications should be checked (e.g., bulbs, meters, manual loader indications, etc) to ensure they are in proper working order.
1.
When the simulator has been frozen, the chart recorder drive power should be secured, if necessary.
j.
Prior to administering the test, the simulator indications should be verified proper based on expected question responses, k.
Any "first-out' annunciators that would normally blink to announce first-out conditions should be frozen and provided to candidates.
1.
If a transient is stabilized by use of plant procedures, the step at which the simulator is fronn should be noted and this information recorded on the simulater sperations summary theet. Progress of the procedure step in effect should also be given to the examinees as necessary.
l (4)
Ideas g g Book Formats.
Table 3 provides a list of sample formats to assist question developers in i
generating performance 4ased, open reference test items.
1 l
l I'
i i
I O
t l
I
j l
i TABLE 1 LEVEL OF KNOWLEDGE DESCRIPTIONS AND EXAMPLES 1.
MEMORT The memory level involves the recall of f acts and specifics, the recall of methods and processes, or the recall of a pattern, structure, or setting.
For measurement purposes, the recall situatio: involves little rcre than bringing to mind the appropriate material.
Appropriate verbs include:
To define To develop To di?.tinguish To outline i
To recall To identify To recognize To list For the operator licensing examination sore exar.ples include:
1 State the basis for a procedural step or caution.
State the basis for a procedural change.
]
State the purpose of a specific procedure.
l Items that require only memorization or recall are not permitted on open-reference examinations.
2.
COMPREHENSION Comprehension represents the lowest level of true understanding.
It refers to a type of understanding such that the individual knows what is l
r
i' being ensuunicated and can make use of the material without necessarily relatlog it to other material or realizing its fullest 1sp11 cations. AppropriaQ ver6s include:
To translate To estimate To prepare To differentiate To comprehend To explain To interpret To susmarize To grasp To demonstrate by example To distinguish To see implication effects, and consequence Questions based at this level and above are permissible to ask on an open-reference examination.
3.
APPLitATION These types of questions require candidates to apply the knowledge to various concrete situations.
The knowledge may be in the form of general ideas, rules of procedures, or generalized methods. Appropriate verbs include:
To apply To predict l
To employ To use To relate To develop For the operator licensing examination examples of this cognitive level would include:
Calculation of plant parameters.
Use of reference material such as graphs, charts, curves, etc.
4.
ANALYSIS The analysis level involves the breakdown of a coemunication into its constituent elements or parts such that the relative hierarchy of ideas is made clear
and/or the relations between the ideas expressed are made explicit.
Appropriate verbs include:
To discriminate To categorize To analyze To choose To detect To discover To infer To select 5.
SYNTHESIS Synthesis involves the putting together of elements and parts so as to form a whole. Appropriate verbs include:
To create To perceive To propose To organize l
To integrate To prepare To plan To compile To design To incorporate To synthesize To visualize 6.
E. VALUATION The evaluation level involves judgment about the value and methods for given j
l purposes. Appropriate verbs include:
To judge To evaluate l
To assess To decide To compare To determine i
To appraise Evaluation is the optimum level of open-reference procedures questions.
Typically questions at this level would propose a situation and require the j
j operator to use analysis and synthesis to formulate judgments / decisions about appropriate actions, j
l 4
l
E1AMPLt$
A knowledge level question, requiring the operator to respond from memory on the basis of a oeution:
A cautism at the beginning of FRP.H.1 requires RCS bleed and feed to be started at STEP 9 if ANY $/G wide range level decreases to less than 60%
[631). Which item below is the basis for this caution?
A) Steps I through 8 deal only with diagnostic evaluation of the event ar.d ray be time consuming.
B) Steps 1 through 8 consist of a ' loop' that is difficult to exit from.
C) Wide range level is not calibrated (therefore not accurate) for l
601[63%).
i D) This level assur.es S/0 dryout is imminent and PCS bleed and feed must l
be started immediately for core cooling, i
1 A comprehension level question, requiring the operator to differentiate between types of turbine runbacks.
i l
I Indicate what type of turbine runback occurred: LOAD LIMIT, LOAD REFERENCE or BOTH.
i j
l l
)
1
The following question falls into the polication level (i.e., app' lying a procedure).
H8R Unit #2 is at 1001 power, steady state.
- ,e HP 97 ct1culator in the control room is 005.
Based on the values gi4en b110w, determine the "QUA!: RANT POWER TILT" as defined in Technico Specifications.
NORMAL UPPER DETECTOR CURRENTS N.41 = 230 h.42 = 235 N-43 = 232 N.44 = 236 NORMAL LOWER DETECTOR CURRENTS R.41 = 233 N.42 = 238 N 43 = 231 N-44 = 240 PRESENT INDICATED DETECTOR CURRENTS UPPER LOWER N-41 232 236
'H.42 238 243
-N-43 235 234 N.44 239 243 l
The following question is at the analysis level, it requires the operator analyze conditions and discriminate an idea:
Determine the posting required for a room using the results of the following radiological survey:
- 1. AIRBORNE ACT!YITY:
6.34 E.9 uct/cc (Co-60)
- 2. FLOOR SMEAR:
Beta-610 dpe/un squared; Alpha. 4dpe/cn squared
- 3. EQUIPMENT SMEAR:
Beta.1800 dpe/cm squared: Alpha. 16 dpe/cm squared
- 4. GENERAL RADIATION LEVEL:
110 ar/hr
~
i l
The following question represents the synthesis / evaluation level putting together of elements and parts to form a whole:
Deterstne if plant conditions satisfy the requirements of foldout *0* $1 TERMINATI0h CRITERIA. Justify your answer with specific values of any l
)
required plant parameters, i
j l
l t
i l
t 1
I
}
I J
l l
i, t
i i
i I
I i
i i
I TABLE 2 OVERVIEW 0F THE BEST U5ES FOR EACH TEST MODE Written:
- Knowledge and abilities that are difficult to infer fres behavior alone.
- Itnowledge of factual information.
- Paper 4 pencil abilities and skills (e.g., calculations)
- Responses requiring information that can be supplied on paper
- Interpretation of reference, if open reference i
J Walkthrough:
- - Areas needing interpretation l
- Areas needing props l
- Knowledge of locations
- Interpretation of references
- Administrative requirements i
4 Simulator:
Overall ability to operate i
Integrated use of knowledge and ab Hties l
I Cossnunications l.
Team Interactions l
Time-critical l
I r
I i
i i
l 1
l
]
i i
1 l
l l
l
i TA8LE 3 ERAMPLE FORMATS FOR OPEN REFERENCE QUESTIONS PPMIDE THE OPERATOR WITH:
REQUIRE THE OPERATOR 70:
1.
Plant / System /Com)onent Diagnosecauseoftheprobles(s) l Condition (s)/ Pro)1ee(s) 2.
Plant / System / Component Identify location of probles(s)
Conditiem(s)/Probleas(s)
J
\\
action (s)ppropriate(recuperative)
Indicate a
)
3.
Plant /Systen/Cor ponent J
Condition (s)/Probles(s) i i
4 Plant / System / Component Indicate actions to achieve conditions specified effect i
- 5. Plant /Ky; tem / Component Identify precipatory events /
Co M tions actions l
I
- 6. Pl[ ant / System / Component Classify / Categorize or I
Conditions /
otherwise indicate if i
conditions meet specified criteria i
I i
7.
Proposed / Hypothetical Cossient on Appropriateness /
I course of action /recosewndation Acceptability of these
.i actions /recosmiendations 1
L 8.
Plant conditions and operator Indicate purpose of/ reasoning actions / procedural steps behind these actions / steps l
9.
Requisite data Cosgutation of parameters I
(
l
- 10. Plant conditions and/or Predict expected plant /
l operator actions system /corponent response (s) 1
- 11. Systes/ Component status Indicate effect on same or othersystem(s)/ component (s) i 12.
Plant,*5ys tem /Cosponent Indicateprcperprocedure(s)/
Conditions references to turn to.
I l
86 ATTAC H NT H NRC CHECKLIST FOR OPEN REFERENCF TEST ITEMS Item Level 1.
Does each test ites have a docueented link to important operator tasks, K/As, and/or facility learning objectives?
2.
Is each test item operationally criented, i.e., is there a satch between job demands and test decands?
4 3.
Is the question at least at the 'corprehension' level of knowledge?
4 Is the cor. text of the questions realistic and free of window dressir.g and backwards logic?
t l
5.
Does the item require an appropriate use of reference material, i.e., is I
it free of "look up' questions?
6.
Is the item at the correct level of difficulty for the job position?
7.
I.s the item appropriate for the written examination and the selected written exam forrat (e.g., shcrt answer; multiple choice)?
8.
is an appropriate mix of operating modes presented in the scenarios in j
5ection A "Plant Operations?'
4 9.
D,o questions in Section A take advantage of the simulator control room l
Setting?
]
- 10. Is the item fre2 of double jeopardy?
- 11. Is the item clear, precise and easy to read and understand?
- 12. Is there only one correct answer to the question?
i
- 13. Does the item pose situations and problems other than those presented during training?
Test Level i
1.
Does the facility sampling plan adequately cover the requalification topics?
2.
Does the facility sampling plan ensure comprehensive, balanced coverage of the requalification program topics?
i 3.
Can the test be coepleted in the time allotted?
t l
n
_ _,, _