ML20154Q473

Confirms Visit to Evaluate Requalification Program During Wk of 890227.Ref Matl Requirements & Related Documentation Encl

ML20154Q473
Person / Time
Site:	Point Beach
Issue date:	09/28/1988
From:	Wright G NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:	Fay C WISCONSIN ELECTRIC POWER CO.
References
NUDOCS 8810030410
Download: ML20154Q473 (84)

Text

, - -

MP 2 81988 i

Docket No. 50-266 Docket No. 50-301 Wisconsin Electric Power Company ATTN: Mr. C. W. Fay Vice President Nuclear Power Department 231 West Michigan, Room 308 Milwaukee, WI 53201

SUBJECT:

REQUALIFICATION PROGRAM EVALUATION In a telephone conversation between Dr. R. Bruno, Training Coordinator, and D. Damon, Chief Examiner, arrangements were made for an evaluation of the requalification program at Point Beach Nuclear Plant. The evaluation visit is scheduled for the week of February 27, 1989.

For this visit, the NRC examiner will administer NRC prepared operating and written examinations. The NRC examiners will discuss with the appropriate facility personnel and operators the schedule and the process for these exami-nations. For the examiners to adequately prepare for this visit, it will be necessary for the facility to furnish the approved reference material listed in Enclosure 1, "Reference Material Requirements". Dr. Bruno has been advised of our reference material requirements and where they are to be sent.

NRC reserves the right to declare a facility training program unsatisfactory and to postpone NRC administered requalification examinations if the facility generated materials are inadequate for examination preparation. Enforcement action may be consiaered if necessary to bring facility generated material to the level of quality for examination preparation. The facility's good faith effort will be taken into consideration during the interim development period.

Additionally, it is requested that a licensed SR0 from both the Point Beach Operations Department and Training Department be designated as the facility representatives for these examinations. These individuals must not be scheduled for an NRC-administered examination during this visit, or participate as an instructor once selected. Also, the facility representatives will be required to certify that, as a result of their involvement, no portion of the examination has been knowingly compromised.

The facility representatives shall be restricted (1) from knowingly comuni-cating)by and (2 any means the content or scope of the exam to unauthorized personsfrom p nation, or tutoring in which an identified regualification examinee (s) will be present. These restrictions shall apply from 60 days prior to the exar: date, h0030410000920 y ADock 05000266 PNU UF

'1

• SEP 2 81988 Wisconsin Power Company 2 The facility management is responsible for providing adequate space and accomodations to properly develop and conduct the examinations. Enclosure 2 "Administration of Requalification Examinations," describes our requirements for developing and conducting the examinations. Dr. Bruno has also been informed of these requirements. Also, a facility operations management repre-sentative should observe the simulation facility examination process at the site.

Enclosure 3 contains the "NRC Rules and Guidance for Examinees" that will be in effect during the administration of the written examination. The facility management is responsible for ensuring that all operators are aware of these rules.

This request for information was approved by the Office of Management and Budget under Clearance Number 3150-0101, which expires May 31, 1989, Coments on burden and duplication may be directed to the Office of Management and Budget, Reports Management Room 3208, New Executive Cifice Building, Washington, D. C. 20503.

Thank you for your consideration in this matter. If you have any questions on the evaluation process, please contact T. M. Burdick at 312/790-5566.

Sincerely.

0RICli4L S! CUED 01. GE0ffREY 0. WRIGllt Geoffrey C. Wright, Chief Operations Branch

Enclosures:

1. Reference Material Requirements for Requalification pogram Evaluations

2. Administration of Requalification Examinations

3. NRC Rules and Guidance for Examinees

,See Attached Distribution Rlll Ril RIII C

r1 Damon/mc Burdick Wright

p.

SEP 2 8 tage Wisconsin Power Company' 3

, Distribution cc w/ enclosures:

J. J. Zach, Plant Manager R. J. Bruno, Training Manager DCD/DCB(RIDS)

Licensing Fee Management Branch Resident Inspector, RIII Virgil Kanable, Chief-Boiler Section Charles Thompson, Chairman Wisconsin Public Service Connission R.I.Braund(SLO),

WI Div. of Emergency Government Lawrence J. McDonnell, Chief Radiation Protection Section WI Department of Health and Social Services, Division of Health cc w/o enclosures:

K. E. Perkins, LOLB N. H. Wagner, Project Manager, NRR R. DeFayette, Section Chief. DRP

4 .

ENCLOSURE 1 REFERENCE MATERIAL REQUIREMENTS

1. For the written examination, the following items must be provided to the NRC 60 days prior to the examination date:

Proposed R0 and SRO requalification written examination. test items. (A minimum of 350 per section of the examination.)

Since the written examination is open reference, examination items must meet the following:

a. Items that require only memorization or recall are not permitted;

b. Items should require that the examinee comprehend, interpret, inte-grate, or apply available information;

c. Items should contain situations, aspects, or conditions that do not duplicate lesson plans or references; and

d. Items should require examinees to locate and use references.

The written examination will be composed of two sections, each designed to be completed in 1-1/2 hours. Each section will be separate. Section A will be administered on a static simulator; Section B will be administered in a classroom setting. Section A is designed to evaluate the operator's knowledge of plant systems, integrated plant operations, and instrumenta- ,

tion and controls. In addition, recognition of Technical Specification LCOs and the operator's ability to diagnose postulated events should be evaluated. Section 8 of the written examination is designed to evaluate the ability of the operator to analyze a given set of conditions and deter-mine the proper procedural and/or administrative guidance.

! 2. All reference material and objectives for the proposed operating test items.

For the simulation facility, the following items must be provided to the NRC 60 days prior to the examination date:

1

1. A minimum of 15 scenarios The scenarios should sample areas such as LERs, emergency and abnormal procedures, and design and procedural changes that exer-cise the crew's ability to use facility procedures in accident prevention and mitigation. The scenarios should evaluate each ,

crew member as appropriate to his/her license, and shall exercise their abilities in the use of Emergency Operating Procedures, i

Technical Specifications, and the Emergency PN n. The scenario's l net time (not including time spent on briefings, setup or simula-l tion facility problems) should average 50 minutes, Lased upon l

real time performance.

i I

o Enclosure 1 (Continued) 9

. For the plant walk through examination, the following items must be prc/ided to the NRC 60 days prior to the examination date:

(1) A list of systems and topics appropriate to the plant walk through examination that were covered during the requalification cycle and are important to safety. All reference material required to support an examination on these topics should be provided.

, (2) Seventy five (75) job performance measures.

These performance measures should be both in plant and control room operator functions, that are required for the safe operation of the facility. They shall include acceptable performance criteria.

(3) Any additional reference material required for examination pre-paration will be requested by the examination team.

'3. A sampling plan shall be provided by the facility which indicates the relative emphasis of topics which were included in the most recent requal-ification training cycle.

I

ENCLOSURE 2 ADMINISTRATION OF REQUALIFICATION EXAMINATIONS

1. 20 percent of the facility licensed operators shall be selected for evalua-tion. Nor ally thc crew currently in the requalification cycle will be selected. A random sample without replacement will be used to preclude a satisfactory operator fron being subject to reexamination by the NRC during the term of the license. The sample will include other shift (s) made up of licensed personnel who are not routinely performing shift duties.

2. The simulator and a simulator operator (s) will be provided for examination development. The date(s) and duration of time needed to develop the examin-ations will be agreed upon by the chief examiner and the facility.

3. The reference material used in the simulator will be reviewed by the chief examiner. No material will be made available that is solely used for training.

4. A single room shall be provided for completing Section B of the written examination. The location of this room and supporting rest room facili-ties shall be such as to prevent contact with all other facility and/or contractor personnel during the duration of the examination.

5. Minimum spacing is required to ensure examination integrity as determined by the chief examiner. Minimum spacing should be one examinee per table, with a 3 foot space between tables. No wall charts, models, and/or other training materials shall be present in the examination room.

6. Copies of reference material for Section B of the written examination will be provided for each examinee. The reference material will be reviewed by the chief examiner and will consist of Technical Specifications, operating /

abnormal procedures, administrative procedures, Emergency Plans as avail-able to the plant operators.

7. Video taping capabilities can be utilized. The facility should contact the chief examiner for restrictions related to its usage. ,

8. Since cosmon tasks and detailed systems knowledge will be probed during the walk through portion of the operating test, operators will be requested not to discuss the walk through with other examinees until after the com- 4 plete examination has been administered.

9. An attempt will be made to distinguish between R0 and SRO knowledge and abilities, to the extent that such a distinction is supported by the facil-ity training materials.

e

ENCLOSURE 3

{

NRC RULES AND GUIDANCE FOR EXAMINEES

1. Use black ink or dark pencil ONLY to facilitate legible reproductions.

2. Print your name in the blank provided on the cover sheet of the examination.

3. Fill in the date on the cover sheet of the examination, if necessary.

4. Answer each question on the examination. If additional paper is required, use only the lined paper provided by the examiner.

5. Use abbreviations only if they are commonly used in facility literature.

6. The point value for each question is indicated in parentheses after the question and can be used as a guide for the depth of answer required.

7. Show all calculations, methods or assumptions used to obtain an answer to a mathematical problem, whether asked for in the question or not.

8. Unless solicited, the location of references need not be stated.

9. Partial credit may be given. Therefore, ANSWER ALL PARTS OF THE QUESTION AND DO NOT LEAVE ANY ANSWERS BLANK.

10. If parts of the examination are not clear with respect to their intent, ask questions of the examiner only.

11. You must sign the statement on the cover sheet that indicates the work on the examination is your own and that you have not received or been given any assistance in completing the examination. This must be signed AFTER the examination has been completed.

12. Rest room trips are to be limited and only one examinee at a time may leave.

You must avoid all contact with anyone outside the examination room to avoid even the appearance or possibility of examination compromise.

13. Cheating on the examination would result in a revocation of your license

, and could result in more severe penalties. '

i

14. Each section of the examination is designed to take approximately 90 minutes to complete. You will be given two hours to complete each section for a total of four hours.

15. Due to the existence of questions that will require all examinees to refer to the same indications or controls, particular care must be taken to main-tain individual examination security and avoid any possibility of compromise or appearance of cheating.

16. When you are finished and have turned in your completed examination, leave the examination area.

1. ES-601 THIS DOCUMENT CONTAINS THE Rev 5 10/01/88 REQUIREMENTS AND PROCEDURES

' N' . ' > FOR REQUALIFICATION EXAMINATIONS AND IS PROVIDED FOR YOUR GUIDANCE AND INFORMATION IN PREPARING THESE

( EXAMINATIONS.

ADMINISTRATION OF NRC REQUALIFICATION PROGRAM EVALUATIONS A. Purpose The NRC requalification examinations are administered under this standard per the provisions of 10 CFR 55.59(a)(2)(iii). The program described in this standard will minimize the potential for an adverse impact on the safe operations of facilities, provide the staff with an assessment of the effectiveness of the facilities requalification training programs, and meet the requirements of 10 CFR 55.57(b)(2)(iv).

B. Scope

. This standard provides general guidance and requirements to NRC examiners for the administration of NRC requalification examinations. This program evaluates the effectiveness of a facility's licensed operator requalifica-tion training program to maintain the competency area currency of licensed operators. This is done by evaluating the ability of the facility to ade-quately prepare written examination questions and simulator scenarios and their ability to properly evaluate their operators' performance. In addi-tion, satisfactory completion of the examinations by individual operators would satisfy the regulatory requirement to pass an NRC administered requal-ification examination prior to license renewal. This document is not a

( substitute for the operator licensing regulations and is subject to revi-sion or other internal operator licensing policy changes.

The requirements and procedures are derived based on a Systems Approach to Training (SAT) program in accordance with INPO Guideline 86-025, and rely upon existing renualification program standards for guiding the NRC exami-

nation developmeat and implementation. This approach will allow the NRC to administer requalification examinations that are consistent with exist-ing facility developed programs, thereby reducing the impact on the facil-ities and improving the reliability of the NRC assessment of requalifica-tion training programs.

Each requalification examination will be developed by an examination team consisting of NRC examiners and facility representatives and will be f reviewed by facility representatives and by the Resident Inspector where practical. The examination will be based upon the facility requalification program and its learning objectives, to the extent practical. The facility should utilize the facility specific Job and Task Analyses as the basis for the examination development. Importance Factor requirements shall be substantiated by the facility. NUREG 1122/1123 should be used to provide additional guidance for identifying job specific importance factors. This approach will result in more technically sound and opera-tionally oriented l

examinations. In addition, coevaluation of operator performance by the NRC and the facility will enhance the ability of the NRC to assess both individual and program performance.

1 i

l 1 of 76

1. ES-601

' Rev 5 10/01/88 s

The NRC administered requalification examination is composed of an opera-ting test and a written examination. The operating test consists of a

( simulator evaluation which emphasizes time-critical and team-dependent behavior with follow-up of individual weaknesses. The walk-through evalu-ation will cover plant systems identified by the NRC and the facility that are important to the safe operation of the facility. The written examina-tion consists of a two section open book examination: Section A will be a "Plant Operations" section and is administered on a static simulator; Section B will be a "Limits and Controls" section and is administered in a classroom setting or in the simulator.

C. Administrative Controls

1. Introduction The following criteria outlines the requirements for the administra-tion of the NRC Requalification Program.

2. Responsibilities

a. Notifications (1) NRC notifies the facility to be evaluated at least 90 days in advance of exam date using the attached "Corporate Noti-fica-tion Letter" (Attachment 1). Site visits should be scheduled to coincide with the requalification training

( cycle of the facility, if possible.

(2) The facility responds to the "Corporate Notification Letter" at least 60 days in advance of the evaluation by providing to the NRC the following:

• Reference Materials (enclosure 1 to Attachment 1)

• Proposed list of operators

• Two staff employees assigned to assist NRC Team (3) The NRC will confirm with the facility at least 30 days in advance which operators have been selected to participate in the evaluation.

(4) The NRC will notify the facility immediately of any indivi-dual whose performance on the examination is sufficiently poor to require immediate removal from licensed activity.

b. Selection of Operators The facility will propose operators for NRC examination 60 days prior to the administration date. The NRC will select crews and individuals to be examined based on the following criteria:

(1) Number of individuals on crew not examined during current license term, 2 of 76

' ' ' ES-601 o Rev 5 10/01/88 (2) Number of examinees required to obtain an adequate sample of at least 12 for program evaluation purposes.

( ,

(3) If circumstances exist which preclude the selection of 12 operators, program evaluation will be deferred until 12 operators from consecutive evaluations have been examined.

(4) Number of staff (not crew) licensed personnel not examined during current license term.

(5) Length of time until license expiration.

(6) Priority given to crew (s) in training during the examination week (s).

(7) Minimize perturbation on facility schedules and operations.

An operator who has passed on NRC requalification examination during the term of his/her license may be included in the simu-lator crew evaluation. Such an individual will not be required to take a written or walk-through examination. This individual will not be included as one of the miminum 12 operators required for program evaluations. Unsatisfactory performance, however, in the simulator examination will be' evaluated as described in section D.1.

( The per'sonnel sampled may include crews of personnel who are not routinely performing shift duties in order to supplement the sample to an adequate size. Mixed crews of shift and non-shift license operators shall not be allowed unless the facility rou-tinely evaluates such mixed crews in their requalification train-ing program.

c. Facility Involvement (1) The facility will be requested to provide two employees, one from the operations staff (required) and one from the training staff (optional), to complete the NRC examination team (Attachment 1). The employee from operations shall be an active licensed SR0; the employee from training should preferably be a licensed SRO, but may be a certified instructor. The function of these individuals is to pro-vide facility specific technical assistance to the NRC in the development and review of the written examination items, plant walk-through topics, and simulator scenarios. If necessary the facility representatives may participate in the actual conduct of the operating test or written exa:nina-tion as a facility evaluator.

(2) To ensure examination security, the facility representa-tives who are on the examination team shall have exam secur-ity restrictions placed upon them by the facility. These restrictions begin upon facility submittal of the response 3 of 76

! ES-601

• Rev 5 10/01/88 L

to the component notification letter (60 days prior to exam date) and continue until the exam administration is cone.luded.

(a) Facility representatives shall not knowingly communicate by any means the content or scope of the exam to unauthorized persons.

(b) Facility representatives shall not participate in any facility programs such as instruction, examination or tutoring in which an identified requalification examinee (s) will be present.

Each facility representative shall be required to sign an Exam Security Agreement (Attv.hment 2) prior to exam involve-ment as stated above and a M st Exam Security Agreement (Attachment 3) at the conclusion of the exam process. These agreements reinforce adherence to the above restrictions.

(3) The facility evaluators must provide by the end of each day, preliminary pass / fail results for the simulator and walkthrough portions of the examination and final results prior to the exit brief.

(4) The facility shall grade the written and the operating exams in parallel with the NRC Team.

d. FacilithReferenceMaterial

(

(1) The facility shall supply the reference materials as requested in the Corporate Notification Letter (Attachment 1). The NRC shall evaluate the facility reference materials for adequacy for exam preparation using the ES 601 Examination Check List (Attachment 4)

(2) NRC reserves the right to declare a facility training program

unsatisfactory and to postpone NRC administered requalifica-tion examinations if the facility generated materials are inadequate for examination preparation. Enforcement action may be considered if necessary to bring facility generated material to the level of quality for examination preparation.

(3) A General Sampling Plan shall be provided by the facility which summarizes the specific examination subject require-ments as outlined in the appropriate sections of this examin-ation standard. This sampling plan should indicate the emphasis which each requalification topic received during the most recent requalification cycle.

3. Administrative Procedure

a. Examination Administration The simulator examination is normally administered first, followed by the plant walk-through evaluations for each crew. The two 4 of 76 f

ES-601

• Rev 5 10/01/88

, part written examination will normally be administered after all simulator and plant walk-through evaluations have been completed.

( This order of administration is intended to provide for an oper-ational attnosphere prior to the written examination. However, a different order of administration may be used if deered appro-priate by the Chief Examiner and approved by the regional section chief. Each portion of the requalification examination shall include the appropriata oral briefing or rules handout,

b. Requalification Program Evaluation A program evaluation should normally be based on a minimum sample size of at least 12 licensed operators and senior operators. If the actual sample size does not meet this minimum then the pro-gram evaluation will be deferred until inclusion of the next annual cycle, unless the interim results warrant immediate action (refer to E.2).

A satisfactory requalification program must meet each of the following:

(1) 90% pass / fait decisions agreement between the NRC and facility grading of the written and operating examinations.

(2) At least 75% of all operators pass the examination; not including individuals selected who had previously passed the examination.

(

(3) The program is judged satisfactory in accordance with the guidance given for the simulator evaluation.

(4) The program meets the requirements of 10CFR55.59 [ para-graphs (c)(2), (c)(3), and (4)] or, is based on systems approach to training.

Criteria for evaluation of Programmatic Weaknesses include the following examples:

(a) 50% of examinees miss greater than one same common task.

(b) 50% of examinees miss greater than one same common task question.

(c) Failure to train and evaluate operators in all of the positions permitted by their individual license. Espec-ially, SR0s Acting as Ras and SR0s not directing operators.

This is not intended to cover utilization of SR0s on CR Panels during only the simulator portion of the examination.

(d) Failure to train operators for "in-plant" JPMs.

5 of 76

o ,

. ES-601 Rev 5 10/01/88 (e) 75% of examinees successfully complete 80% of the common L task questions. .

(f) greater than 1 facility evaluator determined to be unsatisfactory per the ' Evaluating Facility Evaluators Checklist" (Attachment 5).

Note: Failure to meet 3 of these items will mandate a facility program determination of unsatisfactory.

When using the above percentages, fractions of individuals should be rounded up to the next highest number. For example, if fifteen (15) licensed individuals are evaluated, 75% passing would be 11.25, thus eleven of fifteen passing would not be considered as meeting the 75%

requirement. This rule should be applied throughout this standard.

Overall program adequacy shall be determined based on the NRC examiners evaluations. If the facility evaluators judge greater than 25% of their operators as unsatisfactory, while the NRC does not, then cor-rective actions may be required but the facility, program would not be found unsatisfactory due to the facilities more stringent grading standards.

c. Requalification Performance for an Individual For an individual to successfully pass the requalification examin-( ation as graded by the NRC, he/she shall satisfactorily complete:

(1) The written examination as described in 0.3.c(2)

(2) The walkthrough examination as described in D.2.c(2)(a)

(3) Satisfactorily complete the simulator examination as described in D.1.c(2)(b).

d. Crew Performance Evaluations at Non$imulator Facilities Until full compliance with 10 CFR 55.45(b), plants without a i simulation facility shall propose an alternate method such as a t control room mockup for the conduct of this portion of the examina-tion. NRC will determine the usefulness of such alternatives.

e. Timetable for Administration of Requalification Examinations 90 days in advance The facility is notified.

60 days in advance The facility provides proposed material for examination construction (including written examination question and simula-tor scenario banks and job performance measures with follow-on questions for important safety systems).

6 of 76

. ES-601 Rev 5 10/01/88 The facility proposes crew composition.

and team members. ,

Facility team member enter security restrictions.

30 days in advance -

NRC notifies facility of crew selections.

14 days in advance -

NRC team visits the facility to prepare for the examinations.

Facility supplies simulator operator.

7 days in advance -

NRC prepares for examination in the Regional office.

Examination Week -

NRC administers requalification exam-inations to selected crews.

D. Implementation

1. Simulator Evaluations

a. Introduction During the simulator portion of the requalification examination

( each crew will be evaluated on at least two simulator scenarios.

Scenarios shall be used to evaluate:

• crew time critical and team dependent behavior l

• facility requalification training program

• individual competencies and weakness The scenarios should be developed by the facility and reviewed per Attachment 4 and revised by the NRC. The NRC may augment the facility derived scenarios with no more than 20% NRC devel-

oped scenarios. These scenarios will N *' 4 wed with the facil-ity evaluators prior to administratio,. Cr ws sna11 be evalu-ted by both NRC and facility evaluators with W eu erving facility led critiques. Normally NRC evaluators a y w questions of operators after the critique to follow up on deficiencies observed during the simulator evaluation. In those cases where immediate followup question (s) prior to the critique may be necessary, said questions will be approved by the chief examiner.

The simulator examination provides a comprehensive evaluation of the integrated knowledge and skill requirements in a real time environment and determines if there are areas in which retraining l

i is needed to upgrade licensed operators and senior operator knowledge.

i 7 of 76

e

• ? ES-901 Rev i 10/01/88

b. Responsibilities l.

(1) Facility (a) A facility management representative with responsi-bilities for the conduct of plant operations (as a minimum, first level above shift supervisor) should be present during administration of the simulator examin-ations. The NRC Chief Examiner will be the principal point of contact between the facility management and the NRC.

(b) The facility shall provide at least 15 simulator scenarios to the examination team. The scenarios should be approximately 50 minutes in length and both comprehensive and realistic, i.e., not a series of unrelated events.

(c) The scenarios should be based upon lessons covered in the requalification cycle, recent industry events, LERs, emergency and abnormal procedures, and design and procedural changes that exercise the crew's ability to use facility procedures in ecm dent prevention and mitigation.

(d) The facility representatives should substantiate and

( recommend changes as appropriate to NRC modifications to tasks that are critical to plant safety.

(e) The facility shall provide a qualified simulator operator 14 days prior to the exara date, who will be subject to all security restrictions.

(2) NRC Responsibilities (a) The minimum number of HRC examiners required for admin-istration of the simulator scenarios is one for every two operators. Another examiner shall be present as a coordinator. The Chief Examiner or his designee must be present during the conduct of all simulator scenarios, and may act as 11ther the coordinator or an examiner. *

(b) The NRC will review scenarios to ensure tasks contain

- abilities from NUREG-1122/1123 "Knowledge and Abilities

.latalo'l.4 for Nuclear Power Plant Operators" with import" -

ance f .: 4s of 3.0 and above. Facility specific K/A catal.' 4 - 'id be used when available, however, import-

. . Aust be based on public health and safety. '

i <.4 1 should also M considered:

- P wplex beyond the scope of lessons covered salistic) beyond the ability of procedures beyond the simulation facility capability 8 of 7F

~ _ -- _ - _ _ _ _

• ? ES-601 Rev 5 10/01/88

• too simple should be integrated events

( -

no team dependent behavior no time critical behavior absence of equipment malfunctions (3) Examination Team (a) The NRC and the facility representatives as the examina-tion team will jointl Critical Tasks ISCT"*yfor identify Individual each scenario thatSimulator are cru-cial to the maintenance of Any disagree-ments of selection of ISCT' splant safety.

will be raviewed and approved by the regioral Section Chief. To identify ISCTs, review the JTA or other facility training material.

(b) The examination team should verify the scenarios against the facility requalification program learning objectives and compare the ISCT's to the'K/A catalog to ensure they possess abilities with importance factors of 3.5 and above. The scenarios may be modified by the team based upon the technical advisement of the facility representatives.

(c), The examination team shall review those scenarios

( selected for use against the Simulator Scenario Review Checklist (Attachment 6). Due to a simulator scenario not going as planned unanticipated actions by operators may be identified post-scenario by the facility and/or NRC as "critical." Those items will be reviewed in accordance with the guidance provided in paragraph 3-b above.

"Individual Simulator Critical Tasks are defined as step (s) to be taken by an individual during the simulator examination that could challenge the safety status of the facility if not performed properly. Examples of essential safety actions are given in Section c(2)(b).

c. Requalification Process (1) Conduct of Requalification (a) A scenaria's contact time (not including time spect on briefings, sin >ulator setup or simulation facility pro-

blems) should average 50 minutes.

(b) Position rotation shall be dependent upon facility rotation practices. However failure by the facility to rotate each SRO such that the individual would be 9 of 76 f

. ES-601

• Rev 5 10/01/88 required to direct other operators may be identified Crews will be evaluated in the

( as a program weakness.

simulator as they are configured by the' facility for operating the plant. When a simulation facility is available control manipulative skills shall be evaluated in the walk-through examination.

(c) The number of scenarios shall be sufficient, with a minimum of two per crew, to ensure that each examinee is tested to the extent that he/she may operate during both normal and emergency operating conditions within the rotation practices of the facility.

(d) Each scenario should exercise each crew member as appropriate to his/her position during the scenario.

That is, scenarios should be designed so that each individual performs at least one ISCT during this por-tion of the operating test. Each simulator evaluation should place individual crew members in the most senbr watch standing position in which the individual normally operates on shift. Each evaluation should exercise the crews' abilities in the use of Emergency Operating Procedures, Technical Specifications, and the Emergency Plan (See Attachment 6).

(e) Passive observations shall be made. Questioning of operators should normally be done only after the com-(. pletion of the facility led scenario critique. Ques-tions asked immediately after the scenario (prior to the critique) should be approved by the Chief Examiner.

Follow-up questions for flagged items should be asked at this time. The Chief Examiner may question the plant management representative for clartfication, if necessary.

(f) Flagged items of individual weaknesses noted on the simulator examination should be covered through follow-up questions after completion of the scenario critique to determine if them is a need for remedial training, unless resolved during the facility led critique. These questions should also reference a K/A to indicate importance, and be reviewed by the facility rep estn-tatives af ter the fact.

(g) The candidates will be briefed prior to the start of any exercises using the "Briefing Checklist" (Attach-ment 7). Crews may be briefed individually or as a group.

(2) Evaluations

. (a) Crew Evaluations

1. Following each scenario, the crew performance will be critiqued by the facility graders at 10 of 76

• I ES-601 Rev 5 10/01/88 the simulation facility in accordance with

( their procedures. The critique is to be observed by the NRC examiners.

2. Crew performance should be evaluated and ,

documented in accordance with the "Simulator Crew Performance Evaluation Form (Attachment 8)." The facility evaluators should also complete a copy of this form or an equivalent facility form for each crew.

3. An incorrect performance by a crew of one ISCT may establish a basis for an UNSATIS-

""1RY trew evaluation. Incorrect perform-anc6 . two or more ISCT's shall result in an unsatisfactory crew evaluation for the simulator operating test.

4. A crew judged UNSATISFACTORY in the simulator by the NRC evaluators shall be taken off shift, given remedial training, and reexamined prior to resuming licensed duties. NRC should administer the test if the facility's requal-ification program is currently judged UNSATIS-FACTORY and NRC has not verified the adequate

. corrective measures have been instituted.

( Otherwise, the facility will be permitted to administer the reexamination.

(b) Individual Evaluations

1. Individual observations record any defi-ciency noted using ES-301 Attachment 4 or the appropriate facility scenario form.

Individual Simulator Critical Tasks are defined as step (s) to be taken by an indivi-dual during the simulator examination that could challenge the safety status of the facility if not preformed properly. Examples of essential safety actions are:

• ability to effectively manipulate conrols e ability to actuate a reactor trip

• ability to ensure ESFAS systems inject /

actuate e ability to emergency borate or initiate ADS

• prevent violation of Technical Specifications a ability to take any other action or combination of actions that would prevent .

a challenge to plant safety '

11 of 76

- ES-601 Rev 5 10/01/C3

• ability to preclude inappropriate actions

( or combination of actions that create a challenge to plant safety N

2. Significant deficiencies will normally be associated with identified ISCT's. Should an individual incorrectly perform one ISCT, this could establish a basis for an UNSATIS-FACTORY evaluation of his/her simulator examination. Should 2 or more ISCT's be incorrectly performed, an individual examin- ,

ation failure is mandated.

3. An individual who has passed a previously administered NRC requalification examination and is being used as a crew member may fail the examination or be found to require remedial training as a result of poor performance meeting the crittria of D.I.c(2)(b) above.

4. NRC or facility evaluator judgment that a crew is UNSATISFACTORY in the simulator will not necessarily result in an UNSATISFACTORY individual performance evaluation.

5. Any individual weakness observed on tasks

• detailed on ES-301, Attachment 4, will be

( potential items for follow-up. Follow-up '

may not be required if the examiner's concerns are resolved during the facility led scenario '

critique.

(c) Program Evaluation

1. A program may be judged UNSATISFACTORY if the NRC judges at least one crew UNSATIS- '

FACTORY and the facility evaluators judge l the same crew SATISFACTORY; (e.g., facility evaluators feel no remedial training is  !

.ecessary),

' i

2. A program may be judged UNSATISFACTORY if there is less than 90% agreement between the NRC and the facility on the individual pass /

fail determinations with the facility evalu-ating fewer individuals UNSATISFACTORY. i

3. If the facility evaluators judge crew perfor- i mance UNSATISFACTORY and the NRC does not, remedial training is indicated but tha pro-  !

gram will not be penalized for holding a j

' higher standard of operator performance.

12 of 76 f

ES-601 Rev 510/01K2

4. If two or more crews are determined to be

( UNSATISFACTORY by the NRC regardless of individual ~ failures, the overall program will be judged UNSATISFACTORY.

d. Enhancements (1) The following guidelines apply to video taping.

(a) If equipment is available at the site, the simulator examination will be video taped.

(b) After initial set up of the camera by the licensee's personnel under observation of the Chief Examiner, the recording will be made with unattended camera (s).

The only intervention will be to change the tape.

(c) The critiques will be video taped if the scenarios are video taped.

(d) The NRC will be provided a copy in VHS format of the tape before leaving the site.

(2) The video tape will be used primarily to resolve areas of contention between the facility and NRC examiners'

. parallel evaluations of the operators. Additionally,

( the facility, the examiners, and the operators will be provided an opportunity to review those portions of the video tape that directly affect the pass / fail decisions, e.g., performance of previously identified critical items, if they so request. The facility will supply the video tapes. After examination results are finalized, and all conflicts resolved, the NRC and the facility will erase the video tapes. The facility tape custodian will sign a statement (Attachment 9) upon receipt of the video tape indicating that the tape will be used only for its intened purpose and will be promptly erased as specified above. The NRC

! will return its copy of the video tape within 30 days <

of exam results finalization, unless individual exam results are appealed.

2. Walk Through Evaluations

a. Introduction

< The purpose of this portion of the operating test is to assess the individual understanding of and the abili* to perform actions associated with plant systems and manipulatior ..at operators may either perform, or direct the performance e , and to assess the requalification program's effectiveness in keeping the oper-ator's knowledge current with respect to these important safety-

, related tasks and the associated systems. When a simiulation 13 of 76

': ES-601 Rev 5 10/01/88 facility is available control manipulative skills shall be evalu-ated in the walk-through. For example, senior operators may be.

(- required to perform control board operations on the simulator,

b. _ Responsibilities (1) Facility (a) The facility will identify those plant systems applicable to maintenance of public health and safety in the mitigation of the consequences of an event, and those systems that can directly initiate an event. Criteria for system selection include:

• Systems covered during the facility requalification cycle.

• New or recently modified systems.

• Systems the subject of recent facility LERs or vendor notices (e.g., GE SILs).

• PRA identified risk dominant systems / components for plant or vendor generic plants.

.* NRC Information Notices.

(b) Th'e facility representatives will review the facility JTA including learning objectives and NUREG-1122/1123 high-lighting tasks / abilities for the identified systems that meet the following criteria:

• Are applicable to the facility.

• Are at the A0/R0/SRO level RO is responsible for A0/R0 tasks. The SRO is responsible for l A0/R0/SRO tasks.

• Have a K/A ability rating of 3.0 or higher.

Items may be used that have ratings below 3.0 if j

proper facility justification exists.

• If a facility specific JTA is used in lieu of NUREG-1122/123, the importance ratings must be cased on public health and safety.

(c) A list of plant specific tasks should also be developed.

These are tasks / abilities that may not be specifically addressed by the JTA or NUREG-1122/1123 but which have been covered, due to special needs, in the requalifica-tion program, e.g., special procedures EDG operations.

The facility representatives will review and concur on job applicability and importance for the complete list of tasks. The tasks should indicate which steps are 14 of 76

. . ES-601 Rev 5 10/01/88

\

Critical step (s) is/are the step (s) which critical.

( when not performed or when performed incorrectly will, prevent the system from functioning correctly or suc-cessful task completion.

(d) For each of the tasks identified, review the JTA and/

or learning objectives and appropriate cross references to determine the training mode. Seventy-five JPMs shall be submitted by the facility. Refer to the Lesson Plan, Qualification Card or Job Performance Measure which addresses the task. The JPMs should include:

• Initial conditions

• Initiating cues .

• References

• Performance elements and standards

• Cues

• Any appropriate output statements.

• In-depth questions and answers related to the JPM. (Attachment 10)

• Criteria for satisfactory completion of the task.

• Performance elements that are critical.

• Validated task time limits.

• Elements steps sequence restrictions.

(e)* The questions and answers shall:

(

• be based on the task objectives in the JTA

• supported by knowledge rating >73.0

• meet the written guidance in Attachment 11

• include the correct answer, & K/A

• emphasize knowledges required for task performance not demonstrated during the specific JPM

• two or more questions per task, based on the number of knowledge areas in the task analysis

• the answer shall not be directly given in the procedures used to complete the task, i.e. , lookup (2) NRC (a) The NRC team will evaluate the facility identified systems. Absent other information, systems should be selected from those identified in Groups I and II of the "Examiners Hand-book for Developing Licensing I Examinations," NUREG/BR-0122 with at least 50% of the selected systems from Group I of the Handbook.

15 of 76

ES-601 Rev 5 10/01/88 (b) Selected systems wil' % discussed with the facility representatives to e optimum site specific rele .

vance. NRC addit- t 1/or substitution will be dis-cussed with the L tlit) representatives to verify applicability.

(3) Examination Team (a) The team can propose up to 20% of the selected JPM criteria and critical steps based on the technical support of the facility representatives. The examina-tion team should review proposed JPMs per the criteria in the "JPM Quality Checklist" (Attachment 12). All questions and answers will be reviewed by the facility reprasentatives for job relevance and safety signifi-cance prior to examination administration. If the NRC or the facility evaluator asked follow-up or probe questions during the walk-through, these questions should be reviewed as soon as possible with the facil-ity representative af ter the walk-through is completed.

(b) Each examination set shall be reviewed by the Chief ,

Examiner to ensure ES-601 guidance is adhered to.

c. Walk-Through-Process

( (1) Conduct (a) The walk-through will be planned for approximately 2-1/2 hours in length. This includes both the control room and in plant time and is reflective of actual examination contact time. Time required for nonexamin- ,

ation evolutions / items will not be considered during '

examination planning. These evolutions / items include, but are not limited to, the following: [

• Transit time to and from the plant site.

• Time spent complying with facility security and  ;

radiological administrative requirements.

• Transient time from the control room to in plant .

locations, j

• Transit time from one in plant location to another.

(b) The walk-through will be constructed and administered as follows:

i

• Time should be allotted during the operating test for evaluating the performance of ten (10) JPMs. A minimum of four (4) JPMs should be evaluated outside the con-trol room. Five of the ten tasks shall be "common" tasks. These common tasks will be administered to i

16 of 76

': ES-601 Rev 5 10/01/88 each operator and used in determining the effectiveness

( of the requalification program in preparing the operators to perform the tasks.

• A JPM worksheet (Attachment 13) will be completed for each task.

(c) The facility examiner will brief the candidate using the "Briefing Checklist" (Attachment 7). If desired, the candidates may be briefed as a group prior to the start of any walk-thrus. The facility examiners will conduct the walk-through while the NRC examiner grades in parallel. The NRC examiner may ask questions of the operator necessary and appropriate to ensure adequate coverage of the content of the walk-through at the completion of each JPM, and to ensure that the operator has demonstrated satisfactory understanding and application of knowledge regarding the JPM. The NRC examiner must ensure that the facility evaluator is conducting an appropriate examination. If the NRC Examiner has determined that the examination being conducted is inadequate to allow a pass / fail determin-ation then, first, the NRC examiner shall privately counsel the facility evaluator. If the examination conduct remains inadequate, the NRC examiner shall

. . stop the examination and request another facility

( evaluator to conduct the examination.

(d) After administration of each JPM, the NRC shall resolve with the facility representatives all unforeseen tech-nical questions or issues that could result in an indivi-dual failing the examination.

(2) Evaluations (a) Individual Performance Criteria In order to be judged satisfactory on the walk-through portion of the examination, each operator shall:

(1) NOTE: Satisfactory. completion of each JPM within double the allotted contact time is required (unless both the NRC examinee and facility evalu-ator agree that the progress to completion is acceptable). Satisfactorily complete 80% of the JPMs administered.

(2) Correctly answer 70% of the prewritten questions related to the 10 tasks. Followup questions will be used only to confirm step or element performance and/or confirm the initial response to a prewritten question.

(b) Facility Program Performance Criteria 17 of 76

ES-601 Rev 5 10/01/88 (1) 90% pass / fail decision agreement between the NRC

( and facility grading of the walk-through examina*

tions.

(2) At least 75% of all operators pass the walk-through examination.

3. Written Examination Evaluations

a. Introduction .

The purpose of the two section open reference written examination is to assess the individual's knowledge of plant systems, pro-cedures, and operating limits including the Technical Specifica-

'tions,

b. Responsibility (1) Facility Responsibility (a) The facility shall provide a bank of at least 700 requalification examination question items.

These items should be reviewed for appropriateness, clarity, and importance to safety, as described in the

. "Guidelines for the Development and Review of Open Reference Examinations" (Attachment 14),

(

Additionally the following must be provided for each item:

Applicable K/A ref. and values

• Reference JTA

• Estimated time to answer

• Appropriate learning objective

• Training material reference

• The licensee should indicate which questions in ,

the bank are applicable to the current requal  :

cycle (b) Section A of the written examination is designed to utilizir the simulator as a reference tool in answering  !

questions. The questions should be related to plant systems, controls, and recognition of TS LCOs in an open reference format.

(c) This section has a minimum of two "frozen" conditions i, on the simulator, one condition being at power with some equipment in an abnormal status; one condition '

for which the plant could have experienced a major transient resulting in ESFAS initiation.

(d) Section B of the written examination is designed to I utilize plant procedures (including emergency, normal 18 of 76

. ES-601 Rev 5 10/01/88 operation, and abnormal) and administrative controls

[ (including TS, E plan, Administrative Procedures) in ,

( an open reference format.

(e) After the facility representatives have completed the review, they will provide the results to the NRC for use in final examination development. The results will include the time required to answer test items.  :

(f) The facility will provide a sampling plan (i.e., test specifications which identify the percentage to be sampled of each topic area) with their submitted bank items. The facility sampling plati should document the validity of test items by linking each item topic withi

• a K/A with an importance value of 3.0 or greater (or equivalent safety rating from a facility JTA);

e a facility learning objectives; and

• safetyrelated tasks as identified by the facility JTA.

The facility's sampling plan should indicate the per-centage of items on Parts A and B of topics covered

. during the prior two years. The plan should indicate the systematic approach to training basis and other

( materials used to develop the plan and document the comprehensiveness per 10 CFR 55.50(a)(2).

(g) The facility shall be responsible for provioing all required reference material.

(2) NRC Examination Team Responsibility (a) A technical review of the references provided for each test item will be conducted to verify item accuracy.

Each test item will be reviewed for question construc- '

tion with emphasis on its applicability to an open reference examination using, (Attachment 11) "NRC Check-list for Open Reference Test Items."

(b) Each test item should be reviewed for an associated learning objective. Learning objectives should be verifiedasjobrelatedandrelevant. Review of lesson material for test items should be conducted to deter-mine if the learning objectives are related to cor-responding task (s) in the facility JTA and the K/A cr.talog. The K/A rating should be 3.0 or above.  ;

(c) If a clear tie to the JTA does not exist, the applic-ability of the item shall be discussed with the facil-ity representatives.

19 of 76

ES-601 Rev 5 10/01/88

~

(d) Should it be necessary to develop additional items to satisfy the sampling plan, request the facility to do.

.( so.

(e) The NRC may augment the facility's sampling plan. The NRC sampling plan will encompass the facility's sampling  ;

plan with an addition of no more than 20% of items beyond the facility's sampling plan.

(f) Proposed items may be modified, deleted, or replaced ,

if it is determined necessary during the NRC review.  :

(g) The facility representative will review all revised '

test items, evaluating them for:

e appropriateness,

• time required to answer each item,

• technical accuracy, clarity, and

• K/A and objective references.

Examiner Standard ES-401 Attachment 3 should be used to perform a QA review of the examination. The NRC team will review the final examination with the facil-ity representatives for clarity and technical accuracy. ,

, Attachment 1 provides guidance for examination admin- -

istration. The examination shall be entered into the

( EQB main bank after administration, j

+

(h) The examination may be constructed in alternate forms, [

1.e., each operator may have a different sequence of [

questions on his/her examination. This assists in eliminating the need for having multiple sets of refer-ence material. Handouts (e.g., plant curves, blank forms, etc.) may be provided with the test to help relieve the burden on the utility to provide additional sets of reference material,

c. Written Examination Process (1) Written Examination Conduct (a) Each section of the examination shall be proctored in accordance with ES-201.1 and form ES-201-3 shall be completed.

(b) Section A is a "Plant Operations" section and is normally administered on a static simulator. Until full compliance with 10 CFR 55.45(b), for facilities without a simulator, section A should be administered in a control room mockup if it exists. If neither is available, then a classroom setting should be used.

NRC and the facility being evaluated will agree on the compensatory measures (computer printouts, photograohs, 20 of 76

i

~

ES-601 l Rev 5 10/01/88 l

)

etc.), required for a classroom or mockup setting.

Section A is designed to evaluate the operator's know-(- ledge of plant systems, integrated plant operations l and instruments and controls. In addition, recognition of Technical Specification (TS) LCOs and the operator's ability to diagnose postulated events shall be evaluated.

(c) Section B is a "Limits and Controls" section and is normally administered in a classroom setting, although it may be administered in the simulator if deemed appropriate by the Chief Examiner. Section B of the written examination is designed to evaluate the ability of the operator to analyze a given set of conditions and determine the proper procedural and administrative guidance.

(d) The facility shall be responsible for providing the following reference materials:

During the "Plant Operations" (Section A) portion of the examination, one copy of all controlled material available in the control room should be available to examinees. Examination reference material will NOT include material that is intended for training use only. All reference material must be authorized for

. actual operation of the power plant.

(e) During the."Limits and Controls" (Section B) portion of the examination, each examinee shall have available for use the following material (complete, controlled, current isssue):

• Technical Specifications >

• Plant procedures (EOP/AOP/0P, etc.)

• Emergency Plan (as available in the control room)

• Administrative procedures applicable to operations

• Other plant reference material normally available in the control room (e.g., curves and data book, forms, plant drawings, flow charts, etc.).

NOTE: "Noncontrolled" reference material, such as the Emergency Frocedure Owner's Group Basis Documents will not be provided unless the facility certifies that these docu-ments are authorized to be used during plant operations.

(2) Wr3tenExaminationEvaluations Using the examination and key, the facility and NRC will independently grade each section of the written examination.

The grading of all written examinations shall be completed within 20 working days of the examination administration date. Grades will be recorded on the written examination cover sheet (Attachment 15).

21 of 76

'! ES-601 Rev 5 10/01/88 (a) Individual Performance Criteria In order to be judged satisfactory on the written portion of the examination, each operator must achieve at least 80% overall score as graded by the NRC.

(b) Program Performance Criteria In order for a facility's requalification program to be judged satisfactory, the following criteria must be met for the individuals evaluated:

• Ninety percent (90%) pass / fail decisions agreement between NRC and facility grading, and

• at least 75% of all operators pass the examination.

E. Actions Required for Unsatisfactory Individual or Program Evaluation

1. Unsatisfactory Individual Evaluation If an operator fails an NRC administered requalification examination, the operator shall be removed from licensed duties until remediation and reexamination has been completed satisfactorily. NRC should administer the test if the facility's requalification program is currently judged UNSATISFACTORY and the NRC has not verified that adequate corrective measures have been instituted. Otherwise, the

( facility will be permitted to administer the reexamination for return-ing the individual to licensed duties; however, license renewal would require another NRC administered exam.

2. Unsatisfactory Requalification Program Evaluation For any program evaluated as unsatisfactory, the following actions are required unless findings indicate otherwise. Additional actions may be taken at the discretion of the Regional Administrator or his designee. The sequence of actions below is not required. The deter-mination whether plant shutdown is required should be ongoing until the Regional Administrator or hi designee has reviewed all in (d) below,

a. Require the facility to identify program deficiencies and correc-tive actions required to improve operator performance,

b. Meet with senior facility management to review audit findings, identified deficiencies, root causes, corrective actions proposed, schedule for corrective action implementation, and follow-up inspections and examinations.

c. Determine the required :orrective actions by the facility, the required follow-up by the N'lC, and the schedule for each.

22 of 76

ES-601

. Rev 5 10/01/88

d. The Regional Administrator or his designee shall determine whether plant shutdown pending completing of corrective action is required.

This decision should be based on the:

(1) Significance of generic performance deficiencies identified during the program evaluation, (2) Recent SALP performance, especially as related to Criterion 7, Training Effectiveness and Qualification, (3) Recent facility events which relate to licensed operator performance, and (4) Recommendations by resident inspectors.

F. Operator License Renewal Policy

1. Licenses for operators and senior operators will be renewed upon timely application, as described in ES-109/110 if the individual in question has successfully passed an ND.C requalification examination within the term of his/her license.

2. If an individual licenses has been administered an NRC requalification eomination two (2) times during the term of his/her . license without passing any examination, than th individual will be removed from licensed duties and the individuals license will be terminated or will expire without renewal. To resume licensed duties the individual

( mutt apply for license under 10 CFR 55.31 and successfully complete an NRC license examination.

3. If an individual has not successfully passed an NRC requalification exanination within the term of his/her license but has failed an NRC reqJalification examination only once, then the individual license is extended under the timely application provisions until the individual cont'orms with either F.1 or F.2 above.

G. Final Requalification Program Evaluation Report A final requalification program evaluation report similar to the final examination report for a licensing examination shall be prepared when the grading of requalification examinations has been completed. A complete copy of the report shall be filed in the facility requalification file. A copy of the NRC Administered Requalification Examination Results Summary, shall be forwarded ta the Management Assistant OLB. The results summary it, required to verify OLTS d.ata and for statistical data.

H. Record R Qention

1. A facility requalification file shall be maintained for each facility.

All facility evaluation forms, records, assignment sheets, and cor-respondence relating to the requalification program audit for the

(

latest two evaluations s5all be retained.

23 of 76

• I ES-601 Rev 5 10/01/88

2. When the recualification evaluation has been completed by the Regional

( Office, a copy of all NRC administered written, oral and simulator examination results shall be supplied to the facility.. The facilities are required to maintain these records until the operator's or senior operator's license is renewed in accordance with 10 CFR 55.59 or for two years after license expiration. ,-

a

3. A copy of the results summary shall be sent to the Management Assistant, OLB. These summaries shall be used for statistical data gathering.

4. Material relating to an individual failure shall be retained by the Regional Office as necessary to support denial of license renawal per .

10 CFR 55.57(b)(2)(iv). This shall include: [

a. Examination cover page for all exams.

b. The portions of the examination that resulted in the failure.

1 Attachments:

1. Corporate Notification Letter

2. Exam Security Agreement

3. Post E::am Security Agreement -

4. Evaulation Checklist for ES 601

.- 5. Evaluation of Facility Evaluators

6. Simulator Scenario' Review Checklist j

' ( 7. Briefing Sheet i

8. Simulator Crew Evaluation Furm l 9. Video Tape Agreement i 10. System Walk Through Test Plan i 11. NRC Checklist for Open Reference Test Items

! 12. JPM Quality Checklist j 13. JPM Work Sheet 1 14. Guidelines for the Development and Review of Open Reference Examinations

15. Written Examination Cover Sheet l

{

l

\

l l

24 of 76

r . .

ES-601  :

• Rev 5 10/01/88 p 2

ATTACHMENT 1  :

k CORPORATE NOTIFICATION LETTER l (INSERT CORPORATE ADORESSEE)

SUBJECT:

REQUALIFICATION PROGRAM EVALUATION In a telephone conversation between M (title) and M (Section Chief), arrangements were made for an evaluation of the requalifica-tion program and licensed personnel at the (facility name). The evaluation i visit is scheduled for the week of (date).  :

For this visit, the NRC examiners will cdminister NRC prepared operati g snd l written examinations. The NRC examiners will discuss with the appropriate '

facility personnel and operators the schedule and the process for these examina-

. tions. For the examiners to adequately prepare frar this visit, it will be necessary for the facility to furnish the approved items listed in Enclosure 1 "Reference Material Requirements." M has been advised of our refer-l ence material requirements and where they are to be sent.

NRC reserves the right to declare a facility training program unsatisfactory and to postpone NRC administered requalification examinations if the facility s generated materials are inadequate for examination preparation. Enforcement action may be considered if necessary to bring facility generated material to

the level of quality necessary for examination preparation. l

( Additionally, it is requested that a licensed SRO from both the l (plant name) Operations Department and Training Department be deslinatedi as the facility representatives for these examinations. These individuals sust not be

- scheduled for an NRC administereu examination during this visit, or perticipate

as an instructor once selected. Also, the facility representatives will be (

required to certify that, as a result of their involvement, no portion of the examination has been knowingly compromised.

1 The facility representatives shall be restricted (1) from knowingly communicating  ;

by any means the content or scope of the exam to unauthorized persons and (,1) from participating in any facility programs such as instruction, examinatior,  ;

or tutoring in which an identified requalification examinee (s) will be pressnt, j These restrictions shall apply from 60 days prior to the exam date, j The facility management is responsible for providing adequate space and accom-modationt to properly develop and conduct the examinations. Enclosure 2, '

"Administration of Requalification Examinations," describes our requirements

] for developing and conduct.ing the examinations. M has also been i 4

informed of these requirements. Also, a facility operations management repre- l sentative should observe the simulation facility examination process at the .

I site.

j Enclosure 3, contains the "NRC Rules and Guidance for Examinees" that will be c in effect during the administration of the written examination. The facility l management is responsible for ensuring that all operators are aware of these [

l f

rules. Enc 404vre-4r "Regtrirementranti-Procedure 5 for-RequalificartWExamina- l

-ti=," is-includad for-your guidance and1nformation in prepagg_for-these- L examinations._. 7 25 of 76 ,

ES-601

- Rev 5 10/01/88 This request for information was approved by the Office of Management and Budget under Clearance Number 3150-0101, which expires May 31, 1989. Comments on ,

burden and duplication may be directed to the Office of Management and Budget, Reports Management Room 3208, New Executive Office . Washington, DC.

Thank you for your consideration in this matter. If you have any questions on the evaluation process, please contact (Regional Section Chief and telephone number).

Sincerely, (Appropriate Regional Title)

Enclosures:

1. Reference Material Requirements

2. Administration of Requalification Examinations

3. NRC Rules and Guidance for Examinees DISTRIBUTION:

Project Manager ,

Branch Chief, OLB

( Resident Inspector Regional Section Leader Examiner (s)

Facility Training Coordinator Facility Operations Manager 26 of 76

- ES-601 Rev 5 10/01/88 ENCLOSURE 1 REFERENCE MATERIAL REQUIREMENTS

1. For the written examination, the following items must be provided to the NRC 60 days prior to the examination date:

Proposed R0 and SRO requalification written examination. test items. (A minimum of 350 per section of the examination.)

Since the written examination is open reference, examination items must meet the following:

a. Items that require only memorization or recall are not permitted;

b. Items should require that the examinee comprehend, interpret, inte-grate, or apply available information;

c. Ittms should contain situations, aspects, or conditions that do not duplicate lesson plans or references; and

d. Items should require examinees to locate and use references.

The written examination will be composed of two sections, each designed to be completed in 1-1/2 hours. Each section will be separate. Section A I will be administered on a static simulator; Section B will be administered in a classroom setting. Section A is designed to evaluate the operator's knowledge of plant systems, integrated plant operations, and instrumenta-tion and controls. In addition, recognition of Technical Specification LCOs and the operator's ability to diagnose postulated events should be evaluated. Section D of the written examination is designed to evaluate the ability of the operator to analyze a given set of conditions and deter-mine the proper procedural and/or administrative guidance.

2. All reference material and objectives for the proposed operating test items.

For the simulation facility, the following items must be provided to the NRC 60 days prior to the examination date:

1. A minimum of 15 scenarios The scenarios should sample areas such as LERs, emergency and abnormal procedures, and design and procedural changes that exer-cise the crew's ability to use facility procedures in accident prevention and mitigation. The scenarios should evaluate each crew member as appropriate to his/her license, and shall exercise their abilities in the use of Emergency Operating Procedures.

Technical Specifications, and the Emergency Plan. The scenario's net time (not including time spent on briefings, setup or simula-tion facility problems) should average 50 minutes, based upon real time performance.

27 of 76

D ES-601

. Rev 5 10/01/88 For the plant walk through examination, the following items must be provided to the NRC 60 days prior to the examination date:

( (1) A list of systems and topics appropriate to the plant walk through examination that were covered during the requalification cycle and are important to safety. All reference material required to support an examination on these topics should be provided.

(2) Seventy five (75) job performance measures.

These performance measures should be both in plant and control room operator functions, that are required for the safe oneration of the facility. They shall include acceptable performance criteria.

(3) Any additional reference material required for examination pre-paration will be requested by the examination team.

' 3. A sampling plan shall be provided by the facility which indicates the relative emphasis of topics which were included in the most recent requal-ification training cycle.

(

I 28 of 76

- ES-601 Rev 5 10/01/88 ENCLOSURE 2 ADMINISTRATION OF REQUALIFICATION EXAMINATIONS

1. 20 percent of the facility licensed operators shall be selected for evalua-tion. Normally the crew currently in the requalification cycle will be selected. A random sample without replacement will be used to preclude a satisfactory operator from being subject to reexamination by the NRC during the term of the license. The sample will include other shift (s) made up of licensed personnel who are not routinely performing shift duties.

2. The simulator and a simulator operator (s) will be provided for examination development. The date(s) and duration of time needed to develop the examin-ations will be agreed upon by the chief examiner and the facility.

3. The reference material used in the simulator will be reviewed by the chief examiner. No material will be made available that is solely used for training.

4. A single room shall be provided for completing Section B of the written examination. The location of this room and supporting rest room facili-ties shall be such as to prevent contact with all other facility and/or contractor personnel during the duration of the examination.

5. Minimum spacing is required to ensure examination integrity as determined by the chief examiner. Minimum spacing should be one examinee per table, with a 3 foot space between tables. No wall charts, models, and/or other

( training materials shall be present in the examination room.

6. Copies of reference material for Section B of the written examination will be provided for each examinee. The reference material will be reviewed by the chief examiner and will consist of Technical Specifications, operating /

abnormal procedures, administrative procedures, Emergency Plans as avail-able to the plant operators.

7. Video taping capabilities can be utilized. The facility should contact the chief examiner for restrictions related to its usage.

8. Since common tasks and detailed systems knowledge will be probed during the walk through portion of the operating test, operators will be requested not to discuss the walk through with other examinees until after the r.om-plete examination has been administered.

9. An attempt will be made to distingu;sh between RO and SRO knowledge and abilities, to the extent that such a distinction is supported by the facil-ity training materials.

29 of 76

ES-601 Rev 5 10/01/88 ENCLOSURE 3 NRC RULES AND GUIDANCE FOR EXAMINEES

1. Use black ink or dark pencil ONLY to facilitete legible reproductions,

2. Print your name in the blank provided on the cover sheet of the examination.

3. Fill in the date on the cover sheet of the examination, if necessary.

4. Answer each question on the examination. If additional paper is required, use only the lined paper provided by the examiner.

5. Use abbreviations only if they are commonly used in facility literature.

6. The point value for each question is indicated in parentheses after the question and can be used as a guide for the depth of answer required.

7. Show all calculations, methods or assumptions used to obtain an answer to a mathematical problem, whether asked for in the question or not.

8. Unless solicited, the location of referent,es need not be stated.

9. Partial credit may be given. Therefore, ANSWER ALL PARTS OF THE QUESTION AND 00 NOT LEAVE ANY ANSWERS BLANK, Ifpartsoftheekaminationarenotriearwithrespecttotheirintent,

( 10.

ask questions of the examiner only.

11. You must sign the statement on the cover sheet that indicates the work on the examination is your own and that you have not received or been given any assistance in completing the examination. This must be signed AFTER the examination has been completed.

12. Rest room trips are to be limited and only one examinee at a time may leave.

You must avoid all contact with anyone outside the examination room to avoid even the appearance or possibility of examination compromise.

13. Cheating on the examination would result in a revocation of your license and could result in more severe penalties.

14. Each section of the examination is designed to take approximately 90 minutes to complete. You will be given two hours to. complete each section for a total of four hours.

N 15. Due to the existence of questions that will require all exaninees to refer to the same indications or controls, particular care must be taken to main-tain individual examination security and avoid any possibility of compromise or appearance of cheating.

16. When you are finished and have turned in your completed examination, leave the examination area.

30 of 76

ES-601 Rev 5 10/01/88 4

ATTACHMENT 2 EXAM SECURITY AGREEMENT I agree that I will not knowingly divulge any information Print Name concerning the requalification examination scheduled for to any unauthorized persons. I understand that I am not to participate in any instruction involving those operators or senior operators scheduled to be admin-istered the above requalification examination commencing 60 days prior to the examination.

/

Signature /Date

(

31 of 76

u; ,

l 1

ES-601 Rev 5 10/01/88 ATTACHMENT 3

(' POST EXAM SECURITY AGREEMENT I did not, to the best of my knowledge, divulge any Print Name information concerning the examination administered on

.to any unauthorized p*.rsons. I did not participate in providing any instruction to those operators and senior operators who were administered the above requal-ification examination commencing 60 days prior to the examination.

/

Signature /Date i

i k

(

i i

t i

! t t

I i

i l

j  ! I 4 . ,.

, n i

32 of 76 e li  !

- ES-601 Rev 5 10/01/88 ATTACHMENT 4 EVALUATION CHECKLIST FOR ES 601 The attached sheets represent the requirements for Facility Generated Reference material. It is intended to give a go-no go analysis of submitted materials comparability to ES 601 requirements. The important checks presently in ES 601 have been incorporated into this checklist. The Evaluator may use NUREG 1220 for a more in depth analysis as appropriate. The Evaluator would make a deci-sion of satisfactory or unsatisfactory based on how well the facility's material met the minimum requirements.

Specific items, as noted in ES 601, may be resolved via negotiations with the facility staff.

l

\

i 33 of 76

v c .

- ES-601 Rev 5 10/01/88

( FACILITY GENERATED REFERENCE MATERIAL EVALUATION I. Quantity:

ES 601 establishes that the facility shall supply the following minimum

- Reference Material to the NRC 60 days prior to the examination.

Reference Required Actual material minimum submitted Comment A. Open reference written examination 350 per section items B. Simulator scenarios 15 C. Job performance 75 measure D. Technical specifications 1 copy E. Plant procedures 1 set

( F. Emergency Plan 1 copy G. Administrative 1 copy .

procedures H. Sampling plan 1 plan I. Reference material necessary to identify critical tasks (eg)

JTA, K/A catalog 1 set J. Systems Training Reference Material 1 set 34 of 76

o . ES-601

. Rev 5 10/01/88 II. Functionality

(- Circle one A. The Reference Material is legible yes no .

B. Reference Material is arranged and properly labeled for its function yes no C. The Reference Material shows indication of a systematic .

approach yes no ,

D. Reference Material is available to verify that test items are appropriate, relevant job related, and technically accurate yes no E. Reference Material is available to adequately support the examination topics yes no Comments t

t F

h

[

i I

I 35 of 76

.- ES-601 Rev 5 10/01/88

' III. Quality Required Actual Exam sections minimums (% of Minimum) Comments A. Written e items are appropriate examination e important to safety

• clear .

• applicable to open ref examination

• associated with K/A > 3.0 J

• Job relevint Sampling plan Identifies: ,

• K/A > 3.0

• Leariiing objectives

• Safety related tasks

• % part A and B e Indication of SAT basis

(

36 of 76

. ES-601 Rev 5 10/01/88 III. Quality (Continued)

Required Actual Exam sections minimums (% of Minimum) Comments B. Walkthrough

• Applicable plant systems identified by selection criteria:

- Systems covered in requal cycle

- New or recently modified systems 4

- Recent facility LERs or vendor notices

- PRA identified risk dominant

- NRC notices

(' *[ asks /abilitiesfor identified systems are:

- Applicable to the facility

- At the A0/R0/SRO level

- Have a K/A value

> 3.5

• JPMs include:

- Initial conditions

- Initiating cues

- References

- Performance elements and standards

- Cues

- Appropriate output statements 37 of 76

. ES-601 Rev 510hl/88 III. Quality (Continued)

Required Actual Exam sections minimums (% of Minimum) Comments Walkthrough - Appropriate knowledge  !

'(Continued) areas for in-depth questions and answers

- Designated for SRO only or both R0/SRO

- Either the task itself or the following questions require operator problem recognition and ,

diagnosis i

- Critical elements identified

- Answers to questions ,'

,are not found directly in the procedures

(

j

• Associated procedures 1 are included as I reference for JPM

• Each question associated ,

with the JPM is keyel

. to the appropriate K/A

• Each JPM has a minim mi ,

of 2 questions directly i

' related to the JPM knowledge requirement l l

i f

1 v

1 38 of 76  ;

. ES-601 Rev 5 10/01/88 III. Quality (Continued)

Required Actual Exam sections minimums (% of Minimum) Comments Simulator

• Approximately 50 min.

Examination in length

• Comprehensive and realistic

• Verified against learning objectives

)

• Critical tasks are identified and compared i with facility JTA or K/A catalog to verify

importance factor > 3.5

• Scenarios are based on:

i - Lessons covered in l

.requal cycle

] (

! - Recent industry events j - LERs

)

l - Emergency and abnormal procedures i

- Design and procedural changes 4

• Scenarios exercise i

crew's ability to use facility procedures in accident prevention and i mitigation i-

• Scenario events have a K/A importance factor of*

l 3.0 I

• Scenario contains at i least one time-critical response

• Scenario contains at l 1 east one team-dependent crew response j

39 of 76

ES-601

' Rev 5 10/01/88

. 1 III. Quality (Continued) 1 Required Actual Exam sections minimums (% of Minimum) Comments 2

• The scenario events a involve each crew member  ;

• Scenario is composed of related or linked events ,

^

• Scenario requires the ,

use of one of the following:

- Abnormal procedures

- Emergency operating I

procedure

- Technical specifications j

- Emerg. plan .

! implementing procedures  ;

i ,

I i

! i a

6 3

I i i

i

i 8

t i

i I

b

t l

i; I

) ,

l J

l 40 of 76  :

1

~'

. ES-601 Rev 5 10/01/88 ATTACHMENT 5

( .

EVALUATION OF FACILITY EVALUATORS Cues o Verbal cues not excessive o Non-verbal cues (e.g., body language) not excessive o Doesn't give away which steps are "critical."

Evaluation Skills o Detection skills (ability to pick-up on errors) o Probe as required o Properly grades down JPMs that exceed time guidelines o Separates out training knowledge (i.e., doesn't give credit due to evaluator's recollection of operator's training program performance) o Judgement errors k

41 of iS

• I

- ES-601 Rev 5 10/01/88 l ATTACHMENT 6 Rev 1

( 4/15/88 i

SIMULATOR SCENARIO REVIEW CHECKLIST (Attach separate copy to each scenario reviewed)

This form completes examination team review l

SCENARIO IDEdTIFIER: REVIEWER:

1. Scenario contains at least one (1) time-critical crew response.

Time-critical is defined to require affirmative action by one or more crew member (s) in order to prevent or mitigate an event within a limited time.  ;

2. Scenario contains at least one (1) team-dependent crew response.

Team-dependent is defined to require transfer of information between crew members in order to prevent or mitigate an event.

3. The scenario events are designed to involve each crew member.

4. Scenario is composed of related or linked events.

( 5. Scenario requires the use of: (check those that apply)

(Note: The scenario set must include each of the following) ,

Abnormal operating procedures Emergency operating procedures Technical Specifications Emergency plan implementing procedures

6. Critical tasks are:

Identified by facility Required to prevent or mitigate event Importance factor 3.5 in K/A Catalog or facility JTA Reviewed and approved by Exam Team

7. Scenario events have K/A importance factor 3.0 42 of 76

, --- u ,

ES-601 ,

Rev 5 10/01/88 f

ATTACHMEkT 7 BRIEFING CHECKLIST - SYSTEM WALK-THROUGH

1. If the NRC examiner is a visitor, escort responsibility for ensuring compliance with safety, security and radiation protection procedures is the responsibility of the operator escorting the examiner.

2. Plant equipment should not be operated. Nothing the facility or NRC examiner says or asks vill be intended to violate that principle.

3. If clarification of questions is needed during the walk-through, there should be no hesitation to request the examiner reword or clarify the question. ,

4. The examiner will be taking notes throughout the test to document operator 4 .

performance. Frequently an examiner will stop questioning for this purpose.

The amount of note-taking is not dependent upon the operator s level of performance. The examiner must document satisfactory as well as less than satisfactory performance.

5. The walk-through is considered "open book." ine reference material in the I

facility / control rc.: which is normally ava.cble to operators is available, including calibration curves, previous log entries, piping and instrument-ation diagrams, calculation sheets, and procedures. However, operators are responsible for knowing from memory the immediate actions of emergency and other procedures as appropriate to the facility.

~

6. The system walk-through has been planned for approximately two hours in length. However, there is no specific time limit for the walk-through.

The examiner will take whatever time is necessary to cover the areas selected, in the depth and scope required. There will be a minimum of 4 t

Job Performance Measures (tasks) evaluated from the control room and 4 Job 4 Performance Measures evaluated outside of the control room. However, the total number of JPM's will be no less than ten (10).

7. The examiner will explain what tasks are to be completed, which steps to simulate or discuss and provide initial conditions. The operator is to proceed with completing the task as if directed by plant procedures and/or i shift supervision. During the task the examiner will supply the necessary i plant conditions and/or parameters needed to simulate the task. The,oper-l ator should explain each step of the task to the examiner before doing it.

8. When all of the steps for each task are performed correctly, the criteria for the examination will have been completed.

+

9. If the operator feels the need for a break during the walk-through, the operator should request this from the examiner. The examiner is not allowed to reveal the results of the walk-through at its conclusion.

The NRC examiner may ask clarifying questions of the operator at the end I

j 10.

t of each JPM.

l l

43 of 76

r ES-601 Rev 5 10/01/88

11. The NRC examiner will indicate to the operator that no aspects of his examination should be discussed with any other examinees until the con-( clusion of the examination.

Part 8 - For tests with simulation facility available i ,

1. The primary responsibility is to operate the simulation facility as if it were the actual plant. I

2. Team work and communication between operators is evaluated. It benefits  !

the exam process to verbalize observations, analysis, and reasons for l actions more than normally would be done during actual plant operations. l

3. If an operator recognizes an incorrect decision, response, answer, analysis, action taken, or interpretation of the team of which the operator is a part but fails to correct, then the examiner may assume that that operator agrees with the incorrect item. l I

4. A rough log may be kept during each exercise that would be sufficient to complete necessary formal log entries which may be evaluated under admin-

! istrative topics.

4

5. A designated facility instructor will act as the auxiliary operators, radiation health and chemistry technicians, maintenance supervisors, plant management, and apyone else needed outside the control room area.

l I

( 6. The facility examiner will provide a shift turnover before the exercise begins. The shift turnover will include present plant conditions, power i history, equipment out of service, abnormal conditions, surveillance due, and instructions for the shift.

1

7. The control board switches may be purposely misaligned to enhance a simula-  ;

ted scenario or transient where appropriate and is not part to the evalua-tion. If misaligned they should be tagged or otherwise highlighted as  ;

appropriate to the facility. The examiner will not misalign switches dur-l

ing the scenario as an awareness drill.

~

Note: The chief examiner will tell the operators that'no control board switches l will be misaligned on a given scenario or set of scenarios.

8. Operators will be allowed three to five minutes *to familiarize themselves i with the status / conditions of the control boards prior to the start of the ,

experience. [

9. The simulation facility part of the examination will consist of a minimum j

! of two exercises lasting approximately 50 minutes each. There will be a l short break between exercises to set up the initial conditions for the

• l next exercise.  ;

)

10. If the operators have any questions concerning the administration of the operating test, those questions should be answered prior to the start of j l the test.

i 1

44 of 76

ES-601 Rev 5 10/01/88

. t i

11. The NRC examiner will indicate to the operator that no aspects of his examination should be discussed with any other examinees until the con ,

^

( clusion of the examination.

a r.

i

! l i .

g.

{

[

t e

i .

l j i i

, 1 i

I l I

L 1

I

, t i

l I

i

! =

i

+

1 o

I f i  :

I

I i

i i

1

)

l >

I I

i I

45 of 76 e

ES-601

- Rev 5 10/01/88 ATTACHMENT 8 l

INSTRUCTIONS ON USE OF SIMULATOR CREW EVALUATION FORM Enclosed is an evaluation form for use during the trial simulator examin-ation component of the requalification examination. In keeping with the l 1 purpose of the requalification exam, these scales are geared toward evalu- l ating the crew as a whole, rather than individual operators. Please follow  ;

I the instructions below when rating team performance on the simulator examin-ation: l

1. Review the rating scales prior to the onset of the simulator examin- I ation to familiarize yourself with each performance issue to be evaluated, j

2. Use the "Operator Actions" Form (ES-302 attachment 6), and the i' expected operator actions included on that form, to make notes during the examination, as described in ES-302.

1 3. Immediately after the simulator examination is over, evaluate the  ;

crew by completing the Simulator Crew Evaluation Form. Be sure to address all the rating factors for all 6 competencies.

I 4. Provide an overall rating of "satisfactory" or "unsatisfactory" i for each competence. If you gave the crew more than one rating of "1" on I the rating factors for that competence, your score for them on that com-  ;

petence overall should be "unsatisfactory." Although part of the purpose j (~ of the walk-through is to follow up on areas of weakness noted during the i j simulator portion of the examination, if the crew receives a rating of  !

"unsatisfactory" on one'or more competences, their overall evaluation should ,

j also be unsatisfactory. l

) 5. There is space for comments beneath each competence rating and i l

below the overall rating. This space is provided if you feel the need to l s annotate or explain rating (s). In particular, use this space to document  :

the failure of an individual candidate due to his/her exceptionally poor I performance. l i i i t i

j i  !

J i

1 1 L i '

i ('

)

i 46 of 76 l

. ES-601 Rev 5 10/01/88 SIMULATOR EXAMINATION SLM4ARY SHEET CREW MEMBERS:

Name Position ,

I OVERALL TEAM RATING ON THE SIMULATOR EXAMINATION:

Satisfactory Unsatisfactory .

f Coments:  !

L I

( -

t h

Please use the space below to note your coments on the evaluation form (attach additional pages, if necessary): ,

I l

i l

I l

t L

i I

}.

j i

i l

47 of 76 ,

i

ES-601

- Rev 5 10/01/88 UNDERSTANDING / INTERPRETATION OF ANNUNCIATOR / ALARM SIGNALS DID THE CREW:

(a) NOTICE and ACKNOWLEDGE alarms, and ATTEND T0 alarms in order of their l importance/ severity?

l 3 2 1 All alarms that directly Minor awareness or Failed to notice related to significant response difficulties and/or extremely changes in plant or lapses slow at responding conditions were noted to significant alarms at critica) times; easily distracted by nuisance alarms (b) Correctly INTERPRET the meaning and significance of alarms and annunciators (including the use of the Alarm Response Procedures, as applicable)?

3 2 1 Crew readily determined Minor inaccuracies in Significant misin-what failures / events ,

alarm interpretation terpretations, alarms were indicating but without safety resultin in plant

( related consequences degradat on (c) VERIFY that annunciators / alarm signals were consistent with plant / system conditions?

3 2 1 All necessary verifi- Minor lapes in alarm Verification of cations performed, verification, but no failed systems including the identi- inappropriate actions was poor or fication of erroneous taken as a result of altogether absent alarms inadequate ve-ification SCORE ON UNDERSTANDING / INTERPRETATION OF ANNUNCIATORS / ALARM SIGNALS:

Satisfactory Unsatisfactory Comments: _

i 48 of 76

_y y-ES-601 Rev 5 10/01/88 DIAGNOSIS OF EVENTS / CONDITIONS BASED ON SIGNALS / READINGS DID THE CREW:

(a) RECOGNIZE off-normal trends / status?

3 2 1 Timely and accurate Recognition of trends Failed to recognize recognition of trends at time of, but not trends, even after even prior to alarms prior to, sounding of sounding of alarms and alarms annunciators (b) USE INFORMATION and use REFERENCE MATERIAL (prints, books, charts) to aid in the diagnosis / classification of events and conditions?

3 2 1 Correct, timely use Minor errors by crew Failure to use of information and in use or interpretation reference material, reference material of information and misuse / misinterpretation led to accurate reference material of information resulted diagnoses in improper diagnoses (c) Correctly DIAGNOSE plant conditions based on those control room indications?

3 ,

2 1 Diagnoses by crew Minor errors /diffi- Faulty diagnoses were accurate and culties in diagnoses resulted in timely incorrect control manipulations SCORE ON DIAGNOSIS OF EVENTS / CONDITIONS BASED ON SIGNALS / READINGS.

Satisfactory Unsatisfactory Comments:

49 of 76

---v-- - - - - - - - -- - - - - -

- ". ES-601 Rev 5 10/01/88 UNDERSTANDING OF PLANT / SYSTEMS RESPONSE i DID THE CREW:

(a) LOCATE and INTERPRET control room indicators correctly and efficiently to ascertain and verify the status / operation of plant systetas?

3 2 1 Accurate and efficient Minor erre.rs in locating Serious omissions instrument location & or interpreting instruments delays or inaccuracies i

interpretation by all and displays; some crew made in instrument crew members members required assistance interpretation

)

i (b) Demonstrate an UNDERSTANDING of how the plant, systems, and components

operate, including setpoints, interlocks, and automatic actions?

l t 3 2 1 All crew members demon- Minor instances of errors Inadequate knowledge

. strated thorough due to gaps in crew of system / component i understanding of how knowledge of system / operation resulted in systems / components component operation; some serious mistakes or operate crew members required plant degradations assistance

( (c) Demonstrate an understanding of how their ACTIONS (or inaction) affected

,I system / plant conditions?

3 2 1 All members understood Actions or directives Crew appeared to act the effect that indicated minor without knowledge of actions or directives inaccuracies in under- or disregard to, effect had on plant / system standing by individuals, on plant conditions but actions were corrected by team SCORES ON UNDERSTANDING OF PLANT / SYSTEM RESPONSE:

Satisfactory Unsatisf actory Comment:

50 of 76

v ,

ES-601 Rev 5 10/01/88 ,

l  ;

COMPLIANCE /USE OF PROCEDURES AND TECHNICAL SPECIFICATIONS

(- DID THE CREW:

(a) REFER T0 the appropriate procedures in a timely manner?

3 2 1 Crew used procedures Minor failures by Failed to correctly as required; knew crew to refer to refer to procedures when what conditions were procedures without required, resulting in t covered by procedures prompting, but did faulty system operation and where to find them affect plant status (b) CORRECTLY IMPLEMENT procedures, including following procedural steps in correct sequence, abiding by cautions and limitations, selecting correct paths j on decision blocks, and correctly transitioning between procedures?  ;

3' 2 1 Timely, accurate Minor instances of Importance procedural  ;

enactment of procedural misapplication, but steps were not enacted i steps by crew, corrections made in correctly, which led i 1

demonstrating thorough sufficient time to to impeded and/or slow [

understanding of avoid adverse impact recovery or unnecessary  ;

procedural purposes / bases degradation l (c) RECOGNIZE E0P ENTRY CONDITIONS and carry out appropriate immediate actions I j without the aid of references or other forms of assistance? [

t 3 2 1 i

Consistently accurate Minor lapses or Failed to accurately and timely errors; individual recognize conditions  ;

, recognition and crew members needed or execute actions.

J implementation assistance from others even with use of aids i to implement procedures [

i (d) CORRECTLY RECOGNIZE and COMPLY with Technical Specifications and Action I 1 Statementt. of LCOs?  !

3 2 1 I

Recognized and Minor difficulties in Failure to recognize / i j

fully complied with referring to and/or comply with Tech Spec ,,

i LCOs/ Action Statements applying Tech. Specs.; LCOs L crew had to prompt SRO [

! on TS requirements k I

SCORE ON COMPLIANCE /USE OF PROCEDURES AND TECHNICAL SPECIFICATIONS:

Satis, to,, Unsatis,- to,,

)) Coments:

l r

i 51 of 76  !

t

__,-_7

- ES 601 Rev 5 10/01/88 CONTROL BOARD OPERATIONS DID THE CREW:

(a) LOCATE CONTROLS efficiently and accurately?

3 2 1 g t

Controls and Instances of Instances of failure indicators were hesitancy / to locate controls located without difficulty in jeopardizedsystem hesitation by locating controls status f individual operators by one or more l operators j (b) MANIPULATE CONTROLS in an accurate and timely manner? l i

3 2 1 l '

l Smooth manipulation Minor shortcomings Mistakes made in l of the plant in manipulations, manipulating controls  !

within controlled but recovery from caused system j parameters errors without transients and causing problems related problems l t

(c) Take MANUAL CONTROL of automatic functions, when appropriate? (

( 3 2 1  !

l All operators took Minor delays and/or Failed to control I

, control, and smoothly prompting necessary automatic systems i

! operatored automatic before overriding / manually, even when  !

systems manually, without operating automatic ample time and [

assistance, thereby functions, but plant indications existed l l

averting adverse events transients were i

l avoided when possible l l

SCORE ON CONTROL BOARD OPERATIONS

Satisfactory _

Unsatisfactory ,

i Comments:  ;

1 k

i 3

i

l [

I J

i i

52 of 76 {

. ES-601 Rev 5 10/01/88 COM4UNICATIONS/ CREW INTERACTIONS DID THE CREW:

. (a) EXCHANGE complete and relevant informatiori in a clear, accurate, and attentive manner?

3 2 1 Members informed each Communications generally Members did not other of relevant info, complete and accurate, inform each other and actively sought and but some instances of of abnormal indica-listened to info, from needing to be prompted, tions or when others as/when necessary or falling to acknowledge performing evolutions; or respond to info. from inattentive when others important info was requested or provided (b) INTERACT with -ther regarding issues / circumstances outside of their individual area of responsibilsty to facilitate safe plant conditions?

3 2 1 Members assumed Members listened to Members were responsibility for each others conversations inattentive to what issues outside their in general; major technical was happening own boards, as

• errors corrected around them; poor

( appropriate coordination of activities (c) MAKE TEAM DECISIONS in a timely, effective manner?

1 2 1 All individuals provided Majorteamdecisions Leader or other crew input to decisions, generally included members did not accept Decisions resulted in early, input from most crew input from others, recuperative action members, but some resulting in incorrect delays or other or untimely decisions /

problems in reaching directives effective decisions SCORE ON COMMUNICATIONS / CREW INTERACTIONS:

Satisfactory ,,, Unsatisfactory Comments:

53 of 76

, ES-601  !

Rev 5 10/01/88 i

! . }

i 4

ATTACHMENT 9 l

t . ,

1 VIDEO TAPE AGREEMENT  ;

i I hereby agree that I assume responsibility for this video tape of the simulator l I examination that took place on __ _ a t  ;

. I will use this video tape only to make an evaluation -

4 of the examination results and af tet* 411 conflicts have been resolved I will j erase the video tape. I also agree to make no copies of this video tape. l (NAME) l

(TITLE) l t

i  !

(FACILITY) 1  !

1 1

i i

I

)

i i

I I

4 j

f r

! i i >

n

! l 1 ,

l i

l r

54 of 76 l b

q ES-601 Rev 5 10/01/84 ATTACHMENT 10 SYSTEM WALK-THROUGH TEST PLAN Page ,,,,,,, o f ,,__,

Facility: System l Topic: Performance Measurement:

Question  : KA ,

Rating Answer:

Response / Comments:

Question  : KA Rating Answer:

( .-

Response / Comments' Question  : KA , Rating Answer:

Response / Comments:

Question  : KA Rating Answer:

Response / Comments:

55 of 76

o

• ES-601 1 Rev 5 10/01/88 ;

SYSTEM WALK-THROUGH FOLLOW UP DOCUMENTATION f k -

Question Derivation (can be completed after examination):  ;

Simulator Evaluation Deficiency (identify)  !

Topic Area Evaluation: 1. Performance Measurement

2. Question Number Question (to be completed during exam):

Response (to be completed during exam):

Question Derivation (can be completed after examination):

Simulator Evaluation Deficiency (identify)

Topic Area Evaluation. 1. Performance Measurement

2. Question Number Question (to be completed during exam):

Response (to be completed during exam):

(

Question Derivation (can be completed after examination):

Simulator Evsluation Deficiency (identify)

Topic Area Evaluation: 1. Performance Measurement

2. Question Number Question (to be completed during exam):

Response (to be completed during exam):

Question Derivation (can be completed after examination):

Simulator Evaluation Deficiency (identify)

Topic Area Evaluation: 1. Performance Measurement 2 Question Number Question (to be completed during exam):

Response (to be completed during exam):

56 of 76

q .

ES-601 .

- l Rev 5 10/01/C3 ATTACHMENT 11 l 1 ( l NRC CHECKLIST FOR OPEN REFERENCE TEST ITEMS Item level f

1. Does each test item have a documented link to important operator tasks, I K/As, and/or facility learning objectives? f

2. Is each test item operationally oriented, i.e., is there a match between f I

job demands and test demands?

l

3. Is the question at least at the "comprehension" level of knowledge? }

i

4. Is the context of the questions realistic and free of window dressing and backwards logic?  !

[

5. Does the item require an appropriate use of reference material, i.e., is <

l it free of "look up" questions? j i

6. Is the item at the correct level of difficulty for the job position? j a 7. Is the ites appropriate for the written examination and the selected  !

i written exam format (e.g., short answer; multiple choice)? l i

8. Is an oppropriate. mix of operating modes presented in the scenarios in l Section A "Plant Operations?"

( j l 9. Do questions in Section A take advantage of the simulator control room I setting? [

l

10. Is the item free of double jeopardy?

! 11. Is the item clear, precise and easy to read and understand?  ;

12. Is there only one correct answer to the question? [

] t

13. Does the item pose situations and problems other than those presented dur-  !

ing training? (

l 1 Test Level f t

l 1. Does the facility sampling plan adequately cover the requalification topics? I J.

2. Does the facility sampling plan ensure comprehensive, balanced coverage of ,

) the requalification program topics? L b

3. Can the test be completed in the time allotted? l I

I l

t

. I l

I

) L i 57 of 76 1

'. ES-601

- Rev 5 10/01/88 JOB PERFORMANCE MEASURE QUI,LITY CHECKLIST

1. Supported by facility's job / task analysis.

2. Operationally important (meets threshold criteria of K/A 3.i or as deter-mined by the facility and agreed to by NRC).

3. Designated as either SRO only or both R0/SRO.

4. Time validated (time allowed for task completion iMicated on JPM).

Questions related to the task need not be time validated,

5. Either the task itself or the questions that follow require operator problem recognition and diagnosis. Que'stions require knowledge correct actions for abnormal system responses whenever applicable.

6. Performance standards are specific in that exact control and indication nomenclature and criteria (switch position, meter reading) are specified, even if such criteria are not specified in the procedural step.

7. Critical elements and associated performance standards are identified and agreed to by the facility and the NRC.

8. Performance standards should provide complete and proper system response cues where appropriate such that the examiner can properly cue the

{

operator when asked.

9. Answers to questions at the end of the task are NOT found directly in the procedure just used (i.e., do not ask why a certain caution exists if the caution itself identifies the answer).

58 of 76

O .

'. E$-601 i Rev 5 10/01/C3 l ,

l 3 ATTACHMENT 13 J08 PERFORMANCE MEASURE WORKSHEET Facility:

Task

Title:

Task No.: [

. lob Performance Measure No.:  !

K/A Reference-l Operator:  !

Evaluator

Date:

1 Appilcable methods of testing:  !

l Simulate performance Actual performance  !

Classroom Simulator Plant l READ TO THE OPERATOR [

I will explain the initial conditions, which step (s) to simulate or discuss,  !

and provide initiating cues. When you complete the tJsk successfully, the [

l objective for this job performance measure will be satisfied. l l Initial Conditions:

Task Standards: [

( Required Materials:

General

References:

[

Initiating Cues:

i 1 .

i 59 of 76

ES-601

• Rev 5 10/01/88 PERFORMANCE INFORMATION (Denote critical steps with a check mark)

Performance step:

Standard:

Coment:

Performance step:

Standard:

Coment:

Performance step:

Standard:

Comment:

Performance step:

Standard:

Coment:

{'

Performance step:

Standard:

Comment:

Terminating cue:

4 60 of 76

• ES-601

+

Rev 5 10/01/88 VERIFICATION OF COMPLETION Iq '

Job Performance Measure No.

Operator's Name:

Date performed:

Evaluator:

Number of attempts:

Result: (Denote by an S for satisfactory or a U for unsatisfactory and requires remedial training).

Evaluator's signature and date: ,

k P

f

)

i 61 of 76

- ' ES-601 Rev 5 10/01/88 ATTACHMENT 14 GUIDELINES FOR THE DEVELOPMENT AND REVIEW 0F OPEN REFERENCE EXAMINATIONS I. INTRODUCTION The following guidelines are intended for those who are involved in the develop-ment and/or review of test items for the written portion of the NRC Requalifi-cation Examination. As described in ES-601, "Administration of NRC Requalifi-cation Program Evaluations," the written examination consists of two sections, one that utilizes the simulator to provide a context for questions on plant systems and controls, and a second that focuses on plant procedures and admin-istrative controls. Both sections are administered in an "open reference' format. Candidates are allowed to use reference material, including simulator displays, during examination administration. ,

The intent of the change frca closed to open reference written examinations, is twofold:

1. Examination Validity By permitting the use of references that are available to the operator, the conditions and requirements of the written examination more closely approximate those,of the actual Job. The information provided to the

{. operators in the test items can and should' closely parallel the information typically available to them on the job, while the responses elicited by the questions should be similar or identical to the decisions, solutions, and actions required for effective job perforaance. In other words, the openreference'formatenhancesthematchbetweenJobdemandsandtestdemands

--a cornerstone of examination validity.

2. Level of Knowledge The open reference format also enhances examination validity by elevating the level of knowledge of the test items. As described later in these guidelines, candidate access to references precludes the use of questions that test for the mere recall of facts and specifics. Instead, open ref-erence test items require test takers to demonstrate that they can apply, analyze, evaluate, or otherwise USE knowledge to handle the problems and issues encountered on the job.

II. Open Reference Guidelines Most principles for effective test item construction apply equally to all types of written questions, regardless of format. Therefore, open reference test item developers and reviewers should consult references such as NUREG BR-0122, "Examiners' Handbook for Developing Operator Licensing Examinations," INP0's "Principles of Training System Development Ad.iendum I, Test Item Development,"

and NUMARC's "Guidelines for Developing Written Test Items for the NRC Requali-i fication Exam." The guidelines below are those that have been found to be especially pertinent to the creation of open reference test items. These guide-

- lines are divided into five categories:

62 of 76

. -v .

ES-601

- Rev 5 10/01/88 (1) Selection of Test Topics I (2) General Guidelines for Sections 1 & 2

\

(3) Specific Guidelines for Section 1, "Plant Operations" (4) Ideas for Open Reference Formats (5) Open Reference Test Item Review Checklist

1. Selection of Test Topics Test item topics for the NRC requalification examination should be selected based on the following criteria.

A. Requalification Training Program Curriculum .

Test topics should be based on the currict. lum of the most recent operator requalification program training cycle. However, NRC may substitute up to 20% of the examinatien topics selected by the facil-ity with subjects not emphasized during the requalification cycle under 10 CFR 55.59. ,

B. Performance Basis Like the requalification program itself, test topics should be drawn from a job-task analysis (JTA) of the operator and senior operator positions. The facility should validate their test items by demon-strating a link between each item and the following JTA products:

importa.nt operator tasks as identified by the JTA l - important K/As (rated 3.0 or higher) as identified in the NRC K/A Catalog (NUREG 1122/1123) or a facility-specific K/A catalog facility learning objectives identified as important to safety C. Adequacy of Test Coverage The facility's proposed sampling plan (or curriculum evaluation plan) should be checked to ensure that it provides balanced, comprehensive coverage of the topics covered during the requalificatien training cycle. Facility test item topics may be revised if subject areas are found to be under or over-represented in the sampling plan relative to their coverage in the requalification program. In addition, 20%

may be substituted by the NRC. Recent safety-related issues and events (e.g., relevant LERs) should be addressed in the sampling plan.

2. General Guidelines The following guidelines should be followed in the construction and review of test items for both carts of the written examination. These guidelines are intended to supplement, not replace, the good practice criteria found in NUREG BR-0122 and related documents.

63 of 76

l

~

ES-601 Rev 5 10/01/88 A. Operational Orientation As discussed earlier, examination validity is enhanced to the extent that the demands of the test match the demands of the job. Therefore, in addition to being derived from important K/As and testing objectives, the context and stipulations of test items should mirror the situations encountered in the work setting. The following example illustrates effective and ineffective ways to design test items fror4 K/As and learning objectives.

K/A: Knowledge of the design attributes of the Turbine Driven Auxiliary Feedwater Pump Differential Pressure Controller.

Task: Operate the TDAFWP controls during all modes of plant operation.

Terminal Learning Objective: The student will be able to operate the TOAFWP Differential Pressure Controller,without error during a loss of feedwater event.

EnablingObjective: Upon completion of this lesson, the student will be able to explain the operation of the TOAFWP Differential Pressure Controller.

Poor Test Item: State the parameters used by the T0AFWP Differential Pressure Controller.

Better Test Item: Prior to isolating the "C" steam generator (per EPP11), it was noted that the transducer-fed auxiliary feed flow indicators for the "C"

( steam' generator were reading greater than the flow indicators to the "A" and "B" steam generators. What is the reason for this flow deviation?

Notice that the second test item requires the candidate to demonstrate mastery of the knowledge by applying it to an actual job situation. In developing items, it may be useful to ask oneself "why is the K/A important to satisfactory job performance?" and "in what situation will be operator need this K/A?" The answers to these questions can provide a basis / context for test items.

B. Level of Knowledge The operational orientation required of test items on the open reference examinations, as well as the candidates' access to controlled documents, precludes the use of questions that test for mere recall or memor-ization. Rather than requiring candidates to simply recognize or recall facts and specifics, open reference test items should have the candidates demonstrate understanding by requiring them to use their knowledge to address real-life situations and Problems. A test item at the higher level of knowledge requires candidates to determine or identify the appropriate fact, rule, or principle to a novel situation and then correctly apply it. A description of each level of knowledge, along with common verbs and example questions, is found in Table 1.

64 of 76

.___. . -- . = _ _ - . __. - . - . . - _ . _ - _ _ _ _ _ - -

l

, ES-601 1 Rev 5 10/01/88 C. Realistic Context To provide additional assurance of examination validity, the situation or problem posed in the open reference test item should be as similar as possible to the actual situations that candidates encounter on the job. Situations described in the questions should not only be realistic, but should also be free of common "context" problems, including "back-wards logic" and "window dressing."

Backwards logic questions provide candidates with information they normally have to produce, while asking them for information they normally receive. For example:

K/A: Ability to calculate shutdown margins.

Backward Logic Itert Given a shutdown margin is 5.5%, how long has the unit been shut down?

Better Item: The unit has been shut down for x hours. Calculate the shutdown margin.

Questions with window dressing have additional, unnecessary information, typically in an attempt to make a memory level item more operationally oriented. For

( $$$ $[......................................................................

2 Item with Window Dressing: The plant has tripped due to the effect of a tornado crossing the site boundary. You, as Shift Supervisor, direct the phone talker to complete the 15 minute notifications. He informs you that the normal notifi-cation network is inoperable. What method oo you direct him to use for complet-ing the 15 minute notification?

Revised Item: If the normal notification network is inoperable, what method do you direct the phone talker to use to complete the 15 minute notification after the plant has tripped?

Another common problem when constructing questions with realistic contexts is that quite often "real world" situations have more than one correct solution or response. Check the question and references over carefully to ensure that each test item has only one correct answer.

D. Question Novelty One of the most effective ways to ensure that candidates have higher levels of knowledge is to present them with novel situations and require them to both realize what information is relevant and how to apply it. If test questions do not contain unique or varied circum-stances compared to that which was presented in training, the item 65 of 76

. ES-601 Rev 5 10/01/88 will be reduced to simple recall. Review the training material to ensure that questions do not include overly familiar conditions.

( Keep in mind, however, that all conditions and situations should be reasonable, realistic, and safety-related.

E. Use of References References should be considered tools that candidates use to solve problems. It should be the proper use of these tools that is tested during the open ieference examination, not the recall of facts and specifics. Purely "look up" questions should not be included in the examination; rather, questions should be restricted to those that test to see if candidates can identify, locate, or select appropriate reference information to produce organized responses and satisfactory solutions to job related problems and issues. For example:

Pure "Look up" Item: In the event that a safety limit is violated, the reactor shall be:

a. placed in a hot shutdown condition within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />

b. placed in a hot shutdown condition within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />

c. placed in a cold shutdown condition within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />

d. placed in a cold shutdown condition within 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.

Better Use of

References:

While operating at 100% power, VCT and pressurizer alarms and indications,show decreasing pressurizer level. Also, the blowdown

( and main steam radiation monitors have alarmed. While following the appropriate Abnormal or Emergency Procedures, you as the Shift Supervisor must evaluate the existing condition Based on this information, the folicwing emergency classification should be declared:

a. Notification of Unusual Event

b. Alert

c. Site Area Emergency

d. General Emergency F. Difficulty Level l

It is not unusual for test constructors to believe, erroneously, that open reference test items should be more difficult, to compensate for i the candidates' access to reference material. Frequently, this increased difficulty is in the form of requiring knowledge of more obscure or l

otherwise unnecessary information. Both open and closed reference examination items should have the same standard of difficulty; that is, difficulty should be based on the job demands and responsibilities of operators.

l l

66 of 76 l

l

.- \

. 1 ES-601 Rev 5 10/01/88 l G. Time Limits Relative to closed reference examinations, candidates'take considerably longer to answer open reference test items. (Weaker candidates espec-ially have been found to spend an appreciable amount of exam time consulting references versus writing responses). It is important to provide candidates an ample amount of time to complete the examination, although not so much time as to allow less than competent operators the opportunity to locate answers without prior familiarization. The following guidelines should be used to determine the appropriate length of the examination:

1. Each two hour examination section should be constructed to take a competent operator an expected 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> 30 minutes to complete.

2. When possible, the response time of each question should be estimated by having a subject matter expert actually answer the j question, including searching through references.

H. Correct Mode of Measurement No matter how high their importance ratings or operational relevance, certain operator kne ledges, skills and abilities are not amenable to written testing. For example:

( Arrange the major steps in the proper sequence to start parallel, and load DG-2:

Use Governer Control to increase OG-2 KW Raise OG Speed to 900 RPM Press start button on A130B Match Voltage with Bus 1A2 Voltage Close Breaker 1A02.

Decnite its operational orientation, the underlying skill addressed in the above test item would be better assessed by having the candidate simulate or step through the steps during either the simulator or walkthrough portions of the operating examination. Table 2 provides an overview of the best uses for each test mode. Make sure that the K/As and learning objectives selected for the

' written examination can be effectively measured during the written testing.

3. Specific Guidelines for Section A, "Plant Operations" The following guidelines are specific to the Plant Operations section of the written examination, performed on a static simulator. These guide-lines are divided into two sections, Question Development and Simulator Setup.

67 of 76

ES-601

.~o Rev 5 10/01/88 Question Development

(- a. To ensure that the operators' knowledge of plant operations is adequately evaluated, Section A of the written examination should address a mix of normal, abnormal and emergency modes of operation.

b. Questions should require the operators, to the extent possible, to refer to control room indications in formulating their responses, as in the following example:

Which one of the following describes the location of the steam break?

(a) Inside containment, upstream of the steam line flow transmitters (b) Inside containment, downstream of the steam line flow transmitters (c) Outside containment, between "C" MSIV and "C" main steam line check valve.

(d) Outside containment, between "C" MSIV and "C" main steam line contain-ment penetration.

c. The number of scenarios used should be minimized due to the extensive amount of time necessary to set up, run and check the transients,

d. The number of malfunctions / failures for each scenario should be limited.

In general, the scenario should contain one major failure (e.g., LOCA,

( SGTR, steam line breaks, ejected control rods, loss of all AC power).

In addition to the major failures, no more than four minor failures should be used (e.g., failure of a safety related pump to start, failed pressurizer pressure meter indication, nuclear instrumentation failure). In many cases, one major failure and 2-3 minor failures will provide sufficient effects to test a wide range of objectives.

e. Questions may be used that do net relate to the transient but use the simulator as a frame of reference only, provided the candidates are aware of this lack of relationship to the transient,

f. Special attention should be given to ensure that r.iultiple questions stemming from one event do not suffer from double jeopardy. The can-didate should be able to be understand and correctly answer each ques-tion based only on the information given in the question, rather than on the answer to a previous question.

Simulator Setup

g. Prior to the test, the simulator recorders should be rotated to pro-vide clean readings, and the recorders should be checked for proper operation,

b. All indications should be checked (e.g., bulbs, meters, manual loader indications, etc) to ensure they are in proper working order.

68 of 76

ES-601 o

• Rev 5 10/01/88

(

i. When the simulator has been frozen, the chart recorder drive power should be secured, if necessary.

J. Prior to administering the test, the simulator indications should be verified proper based on expected question responses.

k. Any "first-out" annunciators that would normally blink to announce first-out conditions should be frozen and provided to candidates.

1. If a transient is stabilized by use of plant procedures, the step at which the simulator is frozen should be noted and this information recorded on the simulator operations summary sheet. Progress of the procedure step in effect should also be given to the examinees as necessary.

4. Ideas for Open Book Formats Table 3 provides a list of sample formats to assist question developers in generating performance-based, open reference test items.

k 69 of 76

6

' ES-601 Rev 5 10/01/88 TABLE 1

'( '

LEVEL OF KNOWLEDGE DESCRIPTIONS AND EXAMPLES

1. MEMORY The memory level involves the recall of facts and specifics, the recall of methods and processes, or the recall of a pattern, structure, or setting.

For measurement purposes, the recall situation involves little more than bringing to mind the appropriate material. Appropriate verbs include:

To define To develop To distinguish To outli.ie To recall To identify To recognize To list For the operator licensing examination some examples include:

State the basis for a procedural step or caution.

State the basis for a procedural change.

State the purpose of a specific procedure.

Items that require only memorization or recall are not permitted on  ;

open-reference examinatior.s.

2. COMPREHENSION ,

Comprehension represents the lowest level of true understanding. It refers to a type of understanding such that the individual knows what is being communicated and can make use of the material without necessarily relating it to other material or realizing its fullest implications.

Appropriate verbs include:

To translate To estimate To prepare To differentiate To comprehend To explain To interpret To summarize To grasp To demonstrate oy example To distinguish To see implication effects, and consequence i Questions based at this level and above are permissible to ask on an j open-reference examination.

3. APPLICATION These types of questions require candidates to apply the knowledge to various concrete situations. The knowledge may be in the form of general l ideas, rules of procedures, or generalized methods. Appropriate verbs include:

I l

To apply To predict l ' To employ To use To relate To develop 70 of 76

, ES-601 Rev 5 10/01/88 For the operator licensing examination examples of this cognitive level would include:

( ,

Calculation of plant parameters.

Use of reference material such as graphs, charts, curves, etc.

4. ANALYSIS The analysis level involves the breakdown of a communication into its con-stituent elements or parts such that the relative hierarchy of ideas is made clear and/or the relations between the ideas expressed are made explicit.

Appropriate verbs include:

To discriminate To categorize To analyze To choose To detect To discover To infer To select

5. SYNTHESIS Synthesis involves the putting together of elements and parts so as to form a whole. Appropriate verbs include:

To create To perceive To propose -

To organize To integrate To prepare

(- To plan To compile To design To incorporate To synthesize To visualize

6. EVALUATION The evaluation level involves judgment about the value and methods for given purposes. Appropriate verbs include:

To judge To evaluate To assess To decide To compare To determine To appraise Evaluation is the optimum level of open-reference procedures questions.

.lypically questions at this level would propose a situation and require the operator to use analysis and synthesis to formulate judgments / decisions about appropriate actions.

EXAMPLES A knowledge level question, requiring the operator to respond from memory on the basis of a caution:

A caution at the beginning of FRP-H.1 requires RCS bleed and feed to be started at STEP 9 if ANY S/G wide range level decreases to less than 60%

[63%). Which item below is the basis for this caution?

71 of 76

. ES-601 Rev 5 10/01/88 A) Steps 1 through 8 deal only with diagnostic evaluation of the event and may be time censuming.

( ,

B) Steps 1 through 8 consist of a "loop" that is difficult to exit from. ,

C) Wide range level is not calibrated (therefore not accurate) for 60%

[63%].

D) This level assumes S/G dryout is imminent and RCS bleed and feed must be started immediately for core cooling.

A comprehension level question, requiring the operator to differentiate between types of turbine runbacks.

Indicate what type of turbine runback occurred: LOAD LIMIT, LOAD REFERENCE or BOTH.

The following question falls into the application level (i.e., applying a pro-cedure).

HBR Unit #2 is at 100% power, steady state. The HP-97 calculator in the control room is 005. Based on the values given below, determine the "QUADRANT POWER TlLT" as defined in Technical Specifications.

( NORMAL UPPER DETECTOR CURRENTS N-41 = 230 N-42 = 235 N-43 = 232 N-44 = 236 NORMAL LOWER DETECTOR CURRENTS N-41 = 233 N-42 = 238 N-43 = 231 N-44 = 240 PRESENT INDICATED DETECTOR CURRENTS UPPER LOWER N-41 232 236 N-42 238 243 N-43 235 234 N-44 239 243 The following question is at the analysis level, it requires the operator analyze conditions and discriminate an idea:

Determine the posting required for a room using the results of the follow-ing radiological survey:

72 of 76

,- , ES-601 Rev 5 10/01/88

1. AIRBORNE ACTIVITY: 6.34 E-9 uci/cc (Oc-60) t 2. FLOOR SMEAR: Beta- 610 dpm/cm squared; Alpha- 4dpm/cm squared

3. EQUIPMENT SMEAR: Beta- 1800 dpm/cm squared: Alpha- 16 dpm/cm squared

4. GENERAL RADIATION LEVEL: 110 mr/hr The following question rep itents the synthesis / evaluation level - putting together of elements and patts to form a whole:

Determine if plant conditions satisfy the requirements of foldout "B" SI TERMINATION CRITERIA. Justify your answer with specific values of any required plant parameters. i e

I.

( ,

?

I 73 of 76

~

ES-601 Rev 5 10/01/88 TABLE 2

{s OVERVIEW OF THE BEST USES FOR EACH TEST H0DE Written:

Knowledge and abilities that are difficult to infer from behavior alone.

Knowledge of fectual information.

- Paper & pencil abilities and skills (e.g. , calculations)

Responses requiring information that can be supplied on paper Interpretation of reference, if open reference Walkthrough:

- ^

,.reas needing interpretation Areas needing props Knowledge of locations Interpretation of references

- Administrative requirements

( Simulator:

Overall ability to operate Integrated use of knowledge and abilities Communications Team Interactions Time-critical I

l l

l l

t 74 of 76

ES-601 Rev 5 10/01/88 TABLE 3

{s.

• EXAMPLE FORMATS FOR OPEN REFERENCE QUESTIONS PROVIDE THE OPERATOR WITH: REQUIRE THE OPERATOR TO:

1. Plant / System / Component Diagnose cause of the problem (s)

Condition (s)/ Problem (s)

~2. Plant / System / Component Identify location of problem (s)

Condition (s)/ Problems (s)

3. Plant / System / Component Indicate appropriate (recuperative)

Condition (s)/ Problem (s) action (s)

4. Plant / System / Component Indicate actions to achieve conditions specified effect

5. Plant / System / Component Identify precipatory events /

Corditions actions

6. Plant / System / Component Classify / Categorize or otherwise Conditions indicate if conditions meet sp2cified criteria

7. Proposad/ Hypothetical Comment on Appropriateness /

( course of action / recommendation Acceptability of these actions / recommendations

8. Plant conditions and operator Indicate purpose of/ reasoning '

actions / procedural steps behind these actions / steps

9. Requisite data Computation of parameters

10. Plant conditions and/or Predict expected plant /

operator actions system / component response (s)

11. System / Component status Indicate effect on same or other system (s)/ component (s)

12. Plani./ System / Component Indicate proper procedure (s)/

Conditions references to turn to.

i 75 of 76

+ b

( .

s -,

Es401 U.3. M Af#t IEGLANORf ConMS90N REQUAUFICATION EXAMINATION REPORT D M NEr5 M UCDGE N0.: 000ZT NO.:

DMRAn0N M SS.

FACRJTY NAlE:

SUWWARY DMNADON M R0 / 940 FVtET / SECOPC DM LAST D M M WRITTEN .

M Mtc Dwaet (MtM): Fm Danet (MtM):

MADES Mtc FM A

e om f, SWULATOR M Mtc DMNDt (MtM): FM DMNDt (MtM):

PASS / TE PASS / F E WALX THROUGH E Mtc DMNDt (MM): FM DMNDR (MtM):

i No. m N 1 QUEEONS CORRECT i

RECOWMENDAT10N5 i NRC EXAMMERS wrocer Cstat sa m t aRTTIN PASS /FE m u ren PA$s / ra nunacuei Pass /ra SECDON CHET l PASS /FE sowutt

r T ES-601

• 0 Rev 5 10/01/88 a

ATTACHMENT 15

/,

U. S. NUCLEAR REGULATORY COMMISSION REACTOR OPERATOR REQUALIFICATION EXAMINATION FACILITY:

REACTOR TYPE:

DATE ADMINISTERED:

OPERATOR:

% OF SECTION CATEGORY OPERATOR'S CATEGORY VALUE SCORE VALUE A Plant Proficiency .

B Limits and Controls Final Grade l

f 1

(

76 of 76 L