ML20151K259

Provides Ref Matl Requirements for Operator Licensing Exams to Be Held During Wk of 881031,per 880602 Telcon

ML20151K259
Person / Time
Site:	Fermi
Issue date:	07/28/1988
From:	Wright G NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:	Sylvia B DETROIT EDISON CO.
References
NUDOCS 8808030151
Download: ML20151K259 (78)

Text

- - - - - - - - - - -

.r -

..  ; p r JUL 2 8 W ,

T Docket No. 50-341 The Detroit Edison Company ATTN:' B. Ralph Sylvia Senior Vice President Nuclear Operations 6400 North Dixie Highway Newport, MI 48166

SUBJECT:

REQUALIFICATION PROGRAM EVALUATION In a telephone conv(rsation on June 2,1988, between Mr. G. Overbeck, Training Manager, and Mr. G. M. Nejfelt, NRC Chief Examiner, arrangements were made for an evaluation of the requalification program at the Fermi 2 Nuclear Power Station. The evaluation visit is scheduled for the week of October 31, 1988.

For this visit, the NRC examiner will administer NRC prepared written examinations and operating tests. When the NRC examiner arrives at the site, he will meet with the appropriate facility personnel to review the schedule for these examinations. For the examiner to adequately prepare for this visit, it will be necessary for the facility to furnish the approved reference material listed in Enclosure 1, "Reference Material Requirements for Requalification Program Evaluations," at least 60 days prior to the examination date. Mr. Overbeck has been advised of our reference material requirements and where they are to be sent.

NRC reserves the right to declare a facility training program unsatisfactory and to postpone NRC administered requalification examinations if the facility generated materials are inadequate for examination preparation. Enforcement action may be considered if necessary to bring facility generated material to the level of quality for examination preparation.

Additionally, it is requested that licensed SR0 from both the facility's Operations Department and Training Department be designated as the facility representatives for these examinations. These individuals must not be scheduled for an NRC-administered examination during this visit, or participate as an instructor once selected. Also, the reviewers will be required to certify that, as a result of their review, no portion of the examination has been knowingly compromised.

\\

esososotsi esc 72e PDR ADOCK 05o00341 q A/

,/

V PDC

_ _ _ _ _ . _ _ ._________u

~

3 i y Nl281888 i t

1 The Detroit Edison Company 2 '

The facility management is responsible for providing adequate space and accommodations to properly develop and conduct the examinations. Enclosure.2, "Administration of Requalification' Examinations," describes our. requirements for developing and conducting the examinations. Mr. Overbeck has also been informed of these requirements. Also, a facility operations management representative should observe the simulation facility examination nrocess at the site.

Enclosure 3, contains the "NRC Rules and Guidance for Examinees" that will be in effect during the administration of the written examination. The facility management is responsible for ensuring that all operators are aware of these rules. Enclosure 4, "Requirements and Procedures for Requalification Examinations," is included for your guidance and information in preparing for these examinations.

This request for information was approved by the Office of Management and Budget under Clearance Number 3150-0101, which expires May 31, 1989.

Comments on burden and duplication may be directed to the Office of Management and Budget, Reports Management Room 3208, New Executive Office Building, Washington, D. C. 20503.

Thank you for your consideration in this matter. If you have any questions on the evaluation process, please contact G. M. Nejfelt, (312) 370-5528.

Sincerely, Original Signed By M. J. Jordan Fer: Geoffrey C. Wright, Chief Operations Branch

Enclosures:

1. Reference Material Requirements

2. Administration of Requalification Examination

3. NRC Rules and Guidance for Examinees

4. Requirements and Pro: cdures for Requalification Examination See Attached Distribution RIII I RIII

%' e 1 p Ti 3{0 Y F!kf Ndffel/av Jo an Wright

r e ,

',(( ,. ,. k dUL 1 8 1988 p ,

The Detroit Edison Company 3 Distribution <

cc w/ enclosures:

Patricia Anthony, Licensing P. A. Marquardt, Corporate ,

Legal Department DCD/DCB (RIDS)

Licensing Fee Management Branch Resident Inspector, RIII Ronald Callen, Michigan lPublic Service Commission Harry H. Voight, Esq.

Michigan Department of Public Health Honroe County Office of

, Civil Preparedness G. Overbeck, Training Manager cc it/o enclosures:

T. R. Quay, Project Manager, NRR J. N. Hannon, Branch Chief, OLB

<R. W. Cooper, Section Chief, DRP

, , W. Tucker , Operations Superintendent E. Preston, Operations Engineer i

n W

i $

f

\

U00178D STAT 56 NUCLEAR REGULATORY COMMl8860N enaaneessions.o.s.seems

\..... ENCLOSURE 1 REFERENCE MATERIAL REQUIREMENTS For the written examination, the following items post be provided to the MRC 60 days prior to the examination date:

1. Propcsed R0 and SRO requalification written examination test items. (A minimum of 350 per section of the examination. /6. var,/,v&#ren J,4 d m eye,ni.v.4m w:ahe4.,.nelo.'che y cne wij). 0 Since the written examination is open reference, examination items must meet the following:

a. Ites:s that require only memorization or recall are not peruitted;

b. Items should require that the exaninee comprehend, interpret, integrate, or apply available information;

c. Iter.s should contain situations, aspects, or conditions that do not duplicate lesson plans or references; and .-

d. Iter.s should require examinees to locate and use references.

.The written examination will be composed of two sections, each designed to be conspleted in 1-1/2 hours. Each action will be separate. Section A will be administered on a static sirolator; Section B will be administered

'in a classroom setting. Section A is designed to evaluate the operator's knowledge of plant systems, integrated plant operations, and instrumenta-

' tion and controls. In addition, recognition of Technical Specification LCOs and the operator's ability to diagnose postulated events should be evaluated. Section B of the written examination is designed to evaluate the ability of the operator to analyze a given set of conditions and determine the proper procedural and/or administrative guidance.

2. All reference material and objectives for the proposed test items.

' For the simulation facility, the following items must be provided to the NRC 60 days prior to the examination date:

1. A minimum of 15 scenarios The scenarios should sample areas such as LERs, emergency and abnormal procedures, and design and procedural changes that exercise the crew's ability to use facility procedures in accident prevention and mitigation. The scenarios should evaluate each crew member as appropriate to his/her license, and shall exercise their abilities in the use of Emergency Operating l

i

- procedures T

• The scenario'echnical s not time fSpecifications, and the Emergency P1er,.e setuporsimulationfacilityproblems)shouldeverage50 minutes, based upon real time performance.

For the plant walk through examination, the following items must be provided to the NRC 60 days prior to the examination date:

(1) A list of systems and topics appropriate to the plant walk (

through examination that were covered during the requalification cycle and are important to safety. All reference material required to support an examination on these topics should be provided.

(2) Seventy five (75) job perforr.ance measures.

These performance reasures should be both in plant and control room operator functions, that are required for the safe opera-tion of the facility. They shall include acceptable performance criteria.

(3) Any additional reference material required for examination preparation will be requested by the examination team.

e 4

9 0

t weise stAtas

< P NUCLEAR REGULATORY 00MM18810N mammeston,s.a.seems

\..... ENCLOSURE t ADMINISTRATION OF REQUALIFICATION EXAMINATIONS

1. 20 percent of the facility licensed operators shall be selected for evaluation. Normally tht: crew currently in the requalification cycle will be selected. A random sample without replacement will be used to preclude a satisfactory operator from being subject to reexamination by the NRC dvring the term of the license. Thesamplewillincludeothershift(s) made up of licensed personnel who are not routinely performing shift duties.

2. The simulator and a simulator operator (s) will be provided for examination development. Thedate(s)anddurationoftimeneededtodevelopthe examinations will be agreed upon by the chief examiner and the facility.

3. The reference material used in the simulator will be reviewed by the chief examiner. No material will be made available that is solely used for

' training.

4. 'A single room shall be provided for completing Section 8 of the written

.'ex amir,a t ion. The location of this room and supporting rest room facili-f ' ties shall be such as to prevent contact with all other facility and/or

. contractor personnel during the duration of the examination.

5. ' Minimum spacing is nquired to ensure examination integrity as determined

'by the chief examiner. Minimum spacing should be one examinee per table, with a 3 foot space between tables. No wall charts, models, and/or other

' training materials shall be present in the examination room.

6. Copies of reference material for Section 8 of the written examination will be provided for each examinee. The reference material will be reviewed by the chief examiner and will consist of Technical Specificaticns, operating /abnorral procedures, administrative procedures, Emergency Plans as available to the plant operators.

7. Video taping capabilities can be utilized. The facility should contact t

the chief examiner for restrictions related to its usage.

8. Since common tasks and detailed systemt, knowledge will be probed during the usik through portion of the operating test, operators will be re-quested not to discuss the walk tirough with other examinees until after the complete examination has been administered.

9. An attempt will be made to distinguish between R0 and SRO knowledge and abilities, to the extent that suci a distinction is supported by the facility training materials, t ._ --- - .

s

• y a.

yppttDSTATas

/ NUCLEAR REGULATORY 00MM188H3N i I, f maanemerrow.e.t. asses

\,...../

ENCLOSURE 3 NRC RULES AMD GUIDANCE FOR EXAMINEES

1. Use black ink or dark pencil DNLY to faci)itate legible reproductions.

2. Print your name in the blank provided on the cover sheet of the examination.

3. Fill in the date on the cover sheet of the examination, if necessary.

4. Ansver each question on the examination. If additional paper is required, use only the lined paper provided by the examiner.

5. Use abbreviations only if they are cosinonly used in facility literature.

6. The point value for each question is indicated in parentheses after the question and can be used as a guide for the depth of answer requir M.

7. Show all calculations, methods or assustptions used to obtain an answer to '

• a mathematical problem, whether asked for in the question or not.

8. 1Jnless solicited, the location of references need not be stated.

9. yartial credit may be given. Therefore, ANSWER ALL PARTS OF THE QUESTION AND DO NOT LEAVE ANY ANSWERS BLANK.

10. If parts of the examination are not clear with respect to their intent,

'ask questiuns of the examiner only.

11. You must sign the statement on the cover sheet that indicates the work on the examination is your own and that you have not received or been given any assistance in corpleting the examination. This must be signed AFTER the examination has been completed.

12. Rest room trips are to be limited and only one examinee at a time may leave. You must avoid all contact with anyone outside the examination room to avoid even the appearance or possibility of examint. tion compromise.

13. Cheating on the examination would result in a revocation of your license and could result in more severe penalties.

14. Each section of the examination is designed to take approximately 90 minutes to complete. You will be given two hours to cosplete each section I for a total of four hours.

3 l

l

m

• \

edu.ow e(M t

', *M' ES-601

. Rev 5 10/1/88

/ft/ M f, 4 A ffa ~  % W

• G meU75 a J & x.vh u es AetM% 19cMXn).

• DRAFT ADMINISTRATION OF NRC REQUALIFICATION PROGRAM EVALUATIONS l l

A. Purpose This standard provides general guidance and requirements to NRC examiners for the administration of HRC requalification examinations. This program evaluates the effectiveness of a facility's licensed operator requalification training program to maintain the competency and currency of licensed operators. This is done by evaluating the ability of the facility to adequately prepare written examination questions and simulator scenarios and their ability to properly evaluate their operators' performance. In addition, satisfactory completion of the examinations by individual operators would satisfy the regulatory requirement to pass an NRC administered requalification examination prior to ,

license renewal. This document is not a substitute for the operator licensing reguiationsandissubjecttorevisionorotherinternaloperatorlicensing policy changes.

B. Program Description The NRC requalification examinations administered under this standard per the provisionsof10CFR55.59(a)(2)(iii)attempttominimizethepotentialforan

~

adverse impact on the safe operations of facilities, and to provide the staff with an assessment of the effectiveness of facility requalification training programs.

The requirements and procedures are derived based on a systems approach to training (SAT) program in accordance with INPO Guideline 86-025, and rely upon existing requalification program standards for guiding the NRC examination development and implementation, This approach will allow the NRC to administer requalification examinations that are consistent with existing facility

. _ . . _ xamjnerStandards 0001.0.0 of 28

l l

l .

, ES-401 Rev 5 10/1/88 developed programs, therm dr9 the impact on the facilities and improving the reliability of the Ni t m erat of requalification training programs.

Each requalification examination will be developed by an examination team consisting of NRC examiners and facility representatives and will be reviewed by facility representatives and by the Resident Inspector where practical. The examination will be based upon the facility requalification program and its learning objectives, to the extent practical. This approach will result in more technically sound and operationally oriented examinations. In addition, coevaluation M operator performance by the NRC and the facility will enhance the ability of the NRC to assess both individual and program performance.

The NRC administered requalification examination is composed of an operating test and a written examination. The operating test consists of a simulator evaluation which emphasizes time-critical and team-dependent behavior with a flagging system to enable follow-up of individual weaknesses. The walk-through evaluation will cover plant systems identified by the NRC and the facility that,are important to the safe operation of the facility as contained within ,

Croups I and 11 of the "Examiners' Handbook for Developing Licensing Examinations" (See page 2-18 of NUREG/BR-0122). The written examination

onsists of a two section open book examination: Section A will be a "Plant Operations" section and is administered on a static simulator; Section B will be a "Limits and Controls" section and is administered in a classroom setting.

C. Administrative Controls The NRC will plan to examine an average of 20% of the licensed operators and senior operators at each facility on an annual basis. The following criteria outline the requirements for the administration of the NRC Requalification Evaluation Program.

1. Facility Notification Examiner Standards 0002.0.0 of 28

d ES-601 Rev 5 10/1/88 The attached corporate notification letter is used to inform facility program managers of the impending requalification program evaluation. This letter should be mailed at least 90 days in advance. Sita visits should be scheduled to coincide with the requalification training cycle of the facility. The j facility should know 30 days in advance which operators will be evaluated. l 1

2. Selection of Operators The facility will propose operators for NRC examination 60 days prior to the administration date. The NRC will select crews and individuals to be examined based on the following criteria:

a. Number of individuals on crew not examined during current license term,

b. Nunter of examinees required to sample 20% of licensed personnel,

c. Muterofstaff(notcrew)licensedpersonnelnotexaminedduring ,

[ current licer.se term, and Id. Length of time until license expiration.

.e. Priority given to crew (s) in training during the examination week (s).

An operator that has previously passed an NRC requtlification examination during the term of the license may be included in the simulator crew evaluation, if he/she is a merber of a crew selected. Such an individual will not be required to take the written or walk-through examination. Poor performance in the simulator examination would be evaluated as described in Section D.7.a.

l The sample will include additional shifts composed of licensed personnel who are not routinely performing shif t duties, in c,rder to sample approximately i 20 percent of the operators each year. If a facility evaluates mixed crews (shift and non-shift license holders), mixed crews may be evaluated.

{

Examiner Standards 0003.0.0 of 28

3 *

't .

ES-601 Rev 5 10/1/88 1 3. Facility involvement

a. The facility will be requested to provide two employees, one from the operations staff and one from the training staffs, to assist the NRC to fona the NRC examination team. The employee from operations shall be an active licensed SR0; the employee from training should preferably be a licensed SRO, but may be a certified instructor.

The function of these individuals is to provide facility specific technical assistance to the NRC in the developeent and review of the written examination items, plant walk-through topics, and simulator scenarios. If necessary due to insufficient faellity staff resources, facility representatives may participate in the actual conduct of the operating test or written examination.

b. To ensure examination security, the individuals shall not in any way be involved in requalification program activities between the start

. of their involvement in examination development and the administra-tion of the examinations. Also, the individuals shall be instructed.

and agree not to comunicate in any fashion the content or scope of '

the examination to unauthorized persons. Each individual will be

. required to sign a statement prior to reviewing any examination meterial that he/she will not knowingly divulge information concerning the NRC examination, and then to sig a second statement after extmination administration thtt he/she has not knowingly given information about the specific content of the examination to any unauthorized person.

c. For the sis.alation facility examinations, a facility management representative with responsibilities for the conduct of plant operations (as a minimum, first level above shift supervisor) be present during administration of these examinations.

d. In addition, the facility will be required to grade, in parallel with the NRC.

4. Time *able for Administration of Reoualification Examinations Examiner Standards 0004,0.0 of 28

a ES-601 Rev 5 10/1/88 90 days in advance - The facility is notified.

60 days in advance - The facility provides proposed materials for examina-tion construction (including written examination question and simulator scenario banks and job performance measures with follow-on questions for important safety systems).

- The facility proposes crew conposition.

30 days in advance - NRC notifies facility of crew selections.

14 days in advance - NkC team visits the facility to prepare for th' examinations.

7 days in advance - NRC prepares for examination in the Regional office.

Examination Week - NRC administers requalification examinations to -

selected crews.

5. . Examination Administration The simulator examination is normally administered first, followed by the plant walk-through evaluations for each crew. The two part written examination will be administered after all simulator and plant walk-through evaluations have been completed. This order of administration is intended to provide for an operational atmosphere prior to the written examination. Each portion of the requalification examination shall include the appropriate oral briefing or rules handout.

6. Requalification Program Evaluation

a. Program Evaluation i

l l

Examiner Standards 0005.0.0 of 28

ES-601 Rev 5 10/1/88 A program evaluation should normally be based on a minimum simple sire of at least 12 licensed operators and senior operators. If the annual 20% sample does not meet this minimum samplo size then the program evaluation will be deferred untti inclusion of the next annual cycle, unless the interim results w6rrant issnediate action.

A satisfactory requalification program must meet each of the following:

(1) 901 pass / fail decisions agreement between the NRC and facility grading of the written and operating examinations.

(2) At least 80% of all operators pass the examination.

(3) The progra:r. is judged satisfactory in accordance with the guidance giver, for the simulator evaluation.

-(4) The program meets the requirements of 10CFR55.59 or, in lieu of paragraphs (c)(2),(c)(3),and(4),isbasedonsystemsapproachto ,

trafring.

When using the above percentages, fractions of individuals should be rov 7ded up to the r. ext highest number. Forexample,iftwelve(12)licersedindividuals are evaluated, 80% passing would be 9.6, thus ten of twelve passing should be considered as meeting the 801 rtiquirement. This rule should be applied throughout this standard,

b. Reference Material Evaluation NRC reserves the right to declare a facility training program unsatisfactory and to postpone NRC adsinistered requalification examinations if the facility generated materials are inadequate for examination preparation. Enforcement action may be considered if necessary to bring facility generated material to the level of quality for examination preparation.

7. Requalification Performance for an Individual Examiner Standards 0006.0.0 of 28

. - ES-601 Rev 5 10/1/88 For an individual to successfully pass the requalification examination as l

graded by the NRC, the following apply:

a. .i 'ieve at least 80% overall on the written examination.

b. Satisfactorily complete the walk through examination, and

c. Be judged satisfactory on the simulator examination.

8. Examination Process ritique The operators will be provided with an examination process critique sheet and encouraged to complete the form at the conclusion of the examination process.

The fa.111ty will be >sponsible for collating the critiques and providing them to the HQ Branch Chief, with a copy to the Regicnal Branch Chief, after the administration of the examination. The Regional Chief Examiner will compute the kan and stanciard deviation for each item and include the results in the exam report. .

D.

.'$imulator Evaluations The purpose cf this p'ortien of the examination is to evaluate the crew's time-l critical and team-dependent behavior and individual performance during

! simulated events.

Until full compliance with 10 CFR 55.45(b), for plants without a simulation facility, the facility shall propose an alternate method such as a co.4 trol room

mockup for the conduct of this portion of the examination. NRC will dete - %

the usefulness of such alternatives.

1. Scenario Scope and Content l The facility shall provide at least 15 simulator scenarios to the examination team. The scenarios must be approximately 50 minutes in ler.gth and both i

Examiner Standards 0007.0.0 of 28 l- - _

'S-601 Rev 5 10/1/88 comprehensive and realistic, i.e., not a series of unrelated events. The examination team should verify the scenarios against the facility requalification program learning obejetives and compare the critical tasks with the K/A catalog to ensure they possess importance factors of 3.5 and above. The scenarios may be modified by the team based apor, the technical advirement of the facility representatives. The scenarios should be based upon lessons covered in the requalificatior, cycle, recent industry events, LERs, emergency and abnormal procedures, and design and procedural changes that exercise the crew's ability to use facility procedures in accident prevention and mitigation. Each scenario should exercise each crew member as appropriate to his/her position during the scenario. That is, each individual should be required to perform at least one critical task during this portion of the operating test. Each simulator evaluation should piece individual crew members in the most senior watch standing position in which the individual normally operates on shift. Each evaluation shall l

exercise the crews' abilities in the use of Emergency Operating Procedures, Technical Specifications, and the Emergency Plan.

2. . Scenario Complexity, Standards and Criteria

• The NRC will review scenarios to ensure tasks are compatible with

NUREG-1122/1123 "Knowledge and Activities Catalogs for Nuclear Power Plant Operators" for abilities with importance factors of 3.0 and above. The following should also be considered

(a) too complex - beyondthescopeoflessonscovered(unrealistic)

- beyond the ability of procedures

- beyond the simulation facility capability 1

(b) too simple - should be integrated events

- no team dependent behavior

- no time critical behavior

- absence of equipment malfunctions The examination team will review scenarios to ensure expected responses are l

provided.

Examiner Standards 0008.0.0 of 28

, l ES-601

.. Rev 5 10/1/88 The NRC and the facility representatives as the exse.ination team will jointly identify "critical tasks" for each scenario that are crucial to the maintenance of plant safety. To identify critical tasks, review the JTA or other facility training material for identification of critical tasks for each scenario. In addition, the knowledge or t11ty should be rated 3.5 or above in the K/A catalog. The facility repts.,entatives' review should substantiate the task as critical to plant safety. Should the crew or individual incorrectly perform I these critical tasks, this would establish a basis for an unsatisfactory evaluation.

3. Simulator Examination Length and Position Rotation A scenario's contact time (not including time spent on briefings, simulator setup or simulation facility problems) should average 50 minutes, based upon real time perfomance.

Position rotation will be dependent upon facility rotation practices. Crews will be evaluated in the simulator as they are configured by the facility for operating the plant. Control manipulative skills will be evaluated in the walkl through. For exarple, senior operators may be required to perferm control board operations on the sinulator.

The number of scenarios shall bat sufficient, with a minimum of two per crew, to ensure that each examinee is tested to the extent that he/she may operate during both normal and emergcncy operating conditions.

4. Recording and Flagging System Any individual weakness observed on competencies detailed on NRC Form 157, page 4, will be potential items for follow-up. Follow-up may not be Examiner Standards 0009.0.0 of 28

, . ES-601

. Rev 5 10/1/88 required if the examiner's concerns are resolved during the facility led scenario critique.

Passive observations will be made. Questioning of operators during the scenario should only be done after the completion of the facility led scenario I r.ritique. Follow-up questions for flagged items should be asked at 'this time.

The Chief Examiner may question the plant management representative for clarification, if necessary,

s. Individual observations - record any deficiency noted. Failure for an individual, on the simulator portion only, shall be based upon significant deficiencie.1 that could challenge the safety status of the facility such as:

(1) inability to effectively manipulate controls (2) failure to actuate a reactor trip when required (3) failure to ensure ESFAS systems inject / actuate when required (4) failure to emergency borate or initiate ADS when required ,

(5) violation of Technical Specifications (6) failure to take any other action or combination of actions that l -

would prevent a challenge to plant safety.

. (7) Initiate inappropriate actions or combination of actions that create a challenge to plant safety.

Significant deficiencies will nonna11y be associated with identified l critical tasks as indicated in D.2 above.

b. Team observations - evaluation of crew perfonnance will be made as directed in the "Simulator Crew Evaluation Form."

c. Flagged items of weakness noted on the simulator examination should be covered through follow-up questions after completion of the scenario critique to determine if there is a need for remedial training, unless resolved during the facility led critique. These questions I should also reference a K/A to indicate importance, and be reviewed by the facility representatives.

l Examiner Standards 00010.0.0 of 28

' ES-601 Rev 5 10/1/88

5. Number of Examiners Required The minimum number of examiners required is one for every two operators plus one examiner as coordinator and the Chief Examiner, who ray act as the coordi-nator or as an examiner.

6. Assistance of Facility Operations Management The facility will be requested to provide a member of the plant operations staff to observe the process. The Chief Examiner is the principal point of contacc i)etween the facility management and the NRC.

7. Individual / Crew / Program Simulator Evaluation Guidance Following each scenario, the crew performance will be critiqued by the facility graders at the simulation facility. The evaluation is to be observed by the NRC examiners. Use the "Crew Evaluation Form" to document the results of the examir,ation. The facility evaluators should also complete a copy of this form,'

or an eouivalent facility form for each crew.

.' a . Individual Evaluation ,

(1) An individual may fail the NRC examination based on significant deficiencies (see D.4) observed during the simulator excmination.

(2) An individual may fail the NRC examination based on iollow-up questioning on deficiencies observed in the simulator.

(3) An individual who has passed an NRC requalification examination and is being used as a crew member may fail the examination or be found to require remedial training as a result of poor performance meeting the criteria of D.4.a.

Examiner Standards 00011.0.0 of 28

ES-601 Rev 5 10/1/88 (4) MRC or facility evaluator judgement that a crew is UNSATISFACTORY in the simulator will not necessarily result in an UNSATISFACTORY individul performance evaluation. That is, an individual operator may have made only minor errors in execution such that they were overcome during the questioning after the scenario critiques and may not have contributed significant'iy to the poor crew performance,

b. Crew Evaluation (1) Crew perfonnance shall be evaluated in accordance with the "Simulator Crew Performance Evaluation Form."

(2) If a crew is judged UNSATISFACTORY in the simulator by either the NRC or the facility evaluators, the crew must be taken off shift, given remedial training, and reexamined prior to resuming licensed duties. NRC should administer the test if the f acility's requalification program is currently judged .

UNSATISFACTORY and NRC has not verified that adequate corrective measures have been instituted. Otherwise, the facility will be permitted to administer the reexamination.

I

c. Program Evaluation l

(1) A program may be judged UNSATISFACTORY if the NRC judges at least one crew UNSATISFACTORY and the facility evaluators judge that crew SATISFACTORY; (e.g., facility evaluators feel no remedial training is necessary).

(2) A program may be judged UNSATISFACTORY if there is less than 90%

agreement between the NRC and the facility on the individual i

pass / fail determinations with the facility evaluating fewer individuals UNSATISFACTORY.

Examiner Standards 00012.0.0 of 28

_s .

'. - ES-601

- Rev 5 10/1/88 (3) If the facility evaluators judge one crew performance UNSATIS-FACTORY and the NRC does not, remedial training is indicated but the program will not be penalized for holding a higher standard of operator performance.

(4) If two or more crews are determined to be UNSATISFACTORY, regardless of individual failures, the overall program will be judged UNSATISFACTORY.

8. Video Taping The following basic guidelines apply:

a. If equipment is available, the simulhtor examination will be video taped.

b. Image clarity and degree of coverage p;ust be adequate to make taping useful. .

c. After initial set up of the camera by the licensee's personnel under

- observation of the Chief Examiner, the recording will be made with unattendedcamera(s). The only intervention will be to change the tape.

d. The critiques will be video taped if the scenarios are video taped,

e. Two copies of the video tape will be made, one for NRC and one for the facility.

The video tape will be used primarily to resolve areas of contention between the facility and NRC examiners' parallel evaluations of the operators.

Additionally, the facility, the examiners, and operators will be provided an opportunity to review those portions of the videv tape that directly affect the pass / fail decisions, e.g., performance of previously identified critical items, if they so request. NRC will supply the video tapes and maintain custody of one copy and leave one copy with the facility. After examination results are Examiner Standards 00013.0.0 of 28 l ---- - - - . - __

l .

ES-601

~

Rev 5 10/1/88 finalized, and all cor.flicts resolved, the NRC and the facility will erase the videc tapes. The facility tape custodian will sign a statement upon receipt l of the video tape indicating that the tape will be used only for its intended prupose and will be prceptly erased at the conclusion of the examination.

E. Walk-Through Evaluations The purpose of this portion of the operating test is to assess the individual's understanding of and the ability to perfom actions associated with plant systems and manipulations that operators may af ther perform, or direct the performance of, and to assess the requalification program's effectiveness in keeping the operator's knowledge current with respect to these important safety-related tasks and the associated systems.

1. Applicable Plant Systems The facility will identify those plant systems applicable to maintenance of public health and safety in the mitigation of the consequences of an event, and those, systems that can directly initiate an event. Criterit. for system selection include:

a. Systems covered during the facility requalification cycle.

b. New or recently modified systems.

c. Systems the subject of recent facility LERs or vendor notices (e.g.,

GE SILs).

d. PRA identified risk dominant systems / components for plant or vendor generic plants.

e. NRC Information Notices.

Examiner Standards 00014.0.0 of 28

~' I

, Rev 5 10/1/88 The NRC team wird evaluate the facility identified systems. Absent other information, systems should be selected from those identified in Groups I and II of the "Examiners Handbook for Developing Licensing Examinations,"

NUREG/BR-0122 with at least 50% of the selected systems from Group I of the j

, Handbook.

Selected systems will be discussed with the facility representatives to ensure optimum site specific relevance. NRC additions and/or substitution will be discussed with the facility representatives.

2. Identify Tasks The facility representatives will review the facility JTA including learning objectives and NUREG-1122/1123 highlighting tasks / abilities for the identified systems that meet the following criteria:

a. Are applicable to the facility.

b. Are at the A0/R0/SR0 level - RO is responsible for A0/R0 tasks. The SR0 is responsible for A0/R0/SR0 tasks.

c. Have a K/A ability rating of 3.5 or higher. Items may be used that have ratings below 3.5 after NRC examiners consult with the facility representatives.

A list of plant. specific tasks should also be developed. These are tasks / abilities that may not be specifically addressed by the JTA or NUREG-1122/1123 but which have been covered, due to special needs, in the requalification program, e.g., special procedures, EDG operations. The facility representatives will review and concur on job applicability and importance for the complete list of tasks.

Examiner Standards 00015.0.0 of 28 ___ ___

's .

x. .

ES-601 Rev 5 10/1/88

3. Job Performance Measures (JPMs)

For each of the tasks identified, review the JTA and/or learning objectives and appropriate cross references to determine the training mode. Seventy-five JPMs shall be submitted by the facility. Refer to the Lesson Plan, Qualification Card or Job Performance Measure which addresses the task. The JPMs should include:

a. Initial conditions

b. Initiating cues

c. References

d. Performance elements and standards

e. Cues ,

.f. Any appropriate output statements.

g. Appropriate knowledge areas for in-depth questions and answers.

Criteria for satisfactory completion of the task will be identified prior to administration of the examination. Critical steps which may not be included in the facility documentation of the JPM should be identified.

The team should review JPM crite.-ia and critical steps based on the technical support of the facility representatives. The examination team should review proposed JPMs per the criteria in the "JPM Quality Checklist." All questions and answers will be reviewed by the facility representatives for job relevance and safety significance prior to examination administration. If the NRC asked follow-up or probe questions during the walk-through, these questions should be reviewed as soon as possible after the walk-through is completed.

MrAa GK 6L& & RfLFD _ _

. i ES-601 Rev 5 10/1/88

4. Time Allocation The walk-through will be planned for approximately 2-1/2 hours in length.- This includes both the control room and in-plant time and is reflective of actual examination contact time. Time required for nonexamination evolutions / items will not be considered du-ing examination plonning. These evolutions / items include, but are not limited to, the following:

a. Transit time to and from the plant site.

b. Time spent complying with facility security and radiological admin-istrative requirements.

.c. Transient time from the control room to in plant locations,

d. Transit time from one in-plant location to another.

.e. Time required for the operator to locate examination aides, e.g., -

[ procedures, prints, etc.

5. lExaminationConstructionandAdministration

.The walk-through will be constructed and administered as follows:

a. Time should be allotted during the operating test for evaluating the performanceoften(10)JPMs. A minimum of fou': (4) JPMs should be evaluated outside the control room. Five of the ten tasks should be "cosenon" tasks. These comon tasks will be administered to each operator and used in determining the effectiveness of the requalification program in preparing the operators to perform the l tasks.

l

b. A JPM worksheet will be completed for each task.

Examiner Standards 00017.0.0 of 28

ES-601 Rev 5 10/1/88

c. The detailed questions as recorded on the "JPM Question Form" will be based upon the testing objectives established in part E.3 )

above and should have a K/A rating of 3.0 or above. Question  ;

construction should meet the guidelines established for the development of the written examination. The K/A for each question and correct answer will be indicated. Each JPM will have a minimum of two questions which directly, relate to the JPM, and which address the basic understanding and application of knowledge of the JPM. Correct responses for these questions should not normally be directly found in control room references.

d. Each examination will be reviewed by the Chief Examiner also, using the guidance in NUREG-1021.

e. The facility examiners will conduct the walk-through while an NRC examiner grades in parallel. The NRC examiner may ask questions of

- the operator necessary and appropriate to ensure adequate coverage of the content of the walk-through at the completion of each JPM, and ,

to ensure that the operator has demonstrated satisfactory understanding and application of knowledge regarding the JPH, the

- NRC examiner must ensure that the facility evaluator is conducting

', an appropriate examination.

'f. After administration, the NRC shall resolve with the facility representatives all unforeseen technical questions or issues that could result in an individual failing the examination. The facility training manager will be informally briefed on the preliminary results of the examinations.

6. Individual and Program Evaluation

a. Individual Performance Criteria In order to be judged satisfactory on the walk-through portion of the examination, each operator shall:

Examiner Standards 00018.0.0 of 28 L__ _ .-_ . __ - - _ - - -. - - -. - _ . -- . --

s .

ES-601 ,

Rev 5 10/1/88 l (1) satisfactorily complete 805 of the JPMs administered and (2) correctly answer 70% of the questions related to the 10 tasks.

b. Program Performance Criteria (1) If greater than 25% of the operators incorrectly answer more than 20*

of the common task questions (for example, 3 out of 10 questions),

a programatic weakness is indicated in the area of plant systems knowledge and systems interactions. y (2) If one or more facility evaluator (s) evaluates two or more operators as satisfactory when the NRC evaluator evaluates the s6me operators as unsatisfactory, the program will be found to be unsatisfactory.

F. Written Examinatfor.

The purpose of the two section open reference written examination is to assess the individual's knowledge of plant systems, procedures, and operating limits inc1'uding the Technical Specifications.

Sectic.n A is a "Plant Operations" section and is normally administered on a static simulator. Untilfullcompliancewith10CFR55.45(b),forfacilities without a sinJ1ator, section A should be administered in a control room mockup if it exists. If neither is available, then a classroom setting should be used. NRC and the facility being evaluated will agree on the compensatory l

measures (computerprintouts, photographs,etc.),requiredforaclassroomor mockup setting. Section A is designed to evaluate the operator's knowledge of plant systems, integrated plant operations and instruments and controls. In addition, recognition of Technical Specification (TS) LCOs and the operator's J

ability to diagnose postulated events shall be evaluated.

l l

Examiner Standards 00019.0.0 of 28 o

O l

ES-601 Rev 5 10/1/88 Section B is a "Limits and Controls" section and is administered in a classroom setting. Section 8 of the written examination is designed to evaluate the ability of the operator to analyze a given set of conditions and determine the proper procedural and administrative guidance.

1. Scope and Content

a. Section A - Plant Operations This section of the written examination is designed to utilize the simulator as a reference tool in answering questions. The questions should be related to plant systems, controls, and recognition of TS LCOs in an open reference format.

This section has a minimum of two "frozen" conditions on the simulator, one condition being at power with some equipment in an abnormal status; one condition for which the plant has experienced a

mejor transient resulting in ESFAS initiation. .

,b. Section B - Limits and Controls

(

This section of the written examination is d'esigned to utilize plant procedures (including emergency, normal operation, and abnormal) and administrative controls (including TS, E-plan, Administrative Procedures)inanopenreferenceformat.

2. Item Development and Review t

a. Proposed examination items shall be provided as an item bank by the l facility. Proposed items should be reviewed for appropriateness, clarity, and importance to safety, as described in the "Guidelines Examiner Standards 00020.0.0 of 28

n .- -

~

, E5401

. Rev 5 10/1/88 for the Development and Review of Open Reference Examinations." Proposed items may be modified, deleted or replaced if necessary.

b. A technical review of the references provided for each test item will  !

be conducted to verify item accuracy. Each test item will be reviewed for question construction with emphasis on its applicability to an open reference examination using "NRC Checklist for Open Reference Test Items."

Each item should be reviewed for an associated learning objective. Learning objectives should be verified as job related and relevant. Review of lesson material for test items should be conducted to determine if the learning objectives are related to corresponding task (s) in the facility JTA and the K/A catalog. The K/A rating should be 3.0 or above.

If a clear tie to the JTA does not exist, the applicability of the item shall be discussed with the facility representatives. Return any test iters which do not meet these criteria to the facility training department for modification. .

Upon'. completion of question revision, the facility representatives should review all test items evaluating them for (1) appropriateness, (2) time required to answer each item, (3) technical accuracy, (4) clarity, and (5) K/A and objective references. After the facility representatives have completed the review, they will provide the results to the NRC for use in final examination development.

4. Sampling Plan The facility will provide a sampling plan (i.e., test specifications which identify the percentage to be sampled of each topic area) with their proposed items as described in F.2 above. The facility sampling plan should document the validity of test items by linking each item topic with: (1) a K/A with an importance value of 3.0 or greater (or equivalent safety rating from a facility JTA); (2) a facility learning objectives; and (3) safety-related tasks as Examiner Standards 00021.0.0 of 28

. . 1

. 1

~

' ES-601

. Rev 5 10/1/88 identified by the facility JTA. The NRC's sampling plan will encompass the facility's sampling plan with an addition of no more than 20% of items beyond the facility's sampling plan. The facility's sampling plan should indicate the percentage of items on Parts A and 8 of topics covered during the prior full reovelification program. The plan should indicate the systematic approach to training basis and other materials used to develop the plan and document the comprehensivenessper10CFR55.59(a)(2). 1 l

5. Construct Additional Items i Should it be necessary to develop additional items to satisfy the sampling plan, request the facility to do so. Also, the NRC team may develop items as specified in paragraph 1.4.A above. These items should also be reviewed in accordance with this standard.

6. Construct the Final Examination Using the time required to answer the test items provided by the facility, ,

construct an examination using the sampling plan. A competent operator should be able to complete the exanination in I hour and 15 minutes. Once the examination is constructed, use Examiner Standard ES-107 to perform a QA review. The NRC team will review the final examination with the facility representatives for clarity and technical accuracy. Use the attached instructions I

and cover sheets for examination administration. The examination shall be entered into the EQB main bank after administration.

The examination may be constructed in alternate forms, i.e., each operator may have a different sequence of questions on his/her examination. This assists in eliminating the need for having multiple sets of reference material. Handouts (e.g., plant curves, blank forts, etc.) may be provided with the test to help relieve the burden on the utility to provide additional sets of reference material.

Examiner Standards 00022.0.0 of 28

ES-601 Rev 5 10/1/88

7. Procedure for Parallel Grading Using the exarination and key, the facility and NRC will independently grade each section of the written examination. The grading of all written examina-tions shall be completed within 15 working days of the examination administration date.

8. Individual and Program Evaluation i

a. Individual Performance Criteria In order to be judged satisfactory.on the written portion of the examination, each operator must achieve at least 80% overall score as graded by the NRC.

.b. Program Performance Criteria In order for a facility's requalification program to be judged satisfactory, the following criteria must be met for the individuals

. evaluated:

(1) Ninety percent (90%) pass / fail decisions agreenent between NRC and facility grading, and (2) at least 80% of all operators pass the examination.

9. Examination Proctoring

, Each section of the examination shall be proctored in accordance with ES-201.1 l and form ES-201-3 shall be completed.

l

10. Reference Material 1

i 1

The facility shall be responsible for providing the following reference materials:

l Examinor Stondordo. _ _ , 00023a00 0 ef_28

ES-601 Rev 5 10/1/88

_ Daring the "Plant Operations" (Section A) portion of the examination, one copy

of all controlled material available in the control roem should be ara 11able to examinees. Examination reference material will NOT include material that is intended for training use only. All reference material must be authorized for actual operation of the power plant.

During the "Lir.its and Controls" (Section B) portion of the examination, each examinee shall have available for use the following material (complete, controlledcurrentissue):

n. Technical Specifications l b. Plantprocedures(E0P/A0P/0P,etc.)

c. Emergency Plan (as available in the control room) d Administrative procedures applicable to operations

e. Other plant reference material normally available in the control room (e.g., curves and data book, forms, plant drawings, flow charts, etc.).

NOTE: *Non-controlled" reference material, such as the Emergency Procedure Owner's Group Basis Documents will not be provided unless the facility certifies that these documents are authorized to be used during plant operations.

G. Actions Requi. red for Unsatisfactory Individual or Program Evaluation

1. Unsatisfactory Individual Evaluation If an operator fails an NRC administered requalification examination, the operator shall be removed from licensed duties until remediation and i l

1 l

l l

Examiner Standards 00024,0.0 of 28 j L _

s 3 ,

1

~

~ ~' '

ES-601 Rev 5 10/1/88 yJ reexamination has been completed satisfactorily. MRC should administer the i test if the facility's requalification program is currently judged UNSATISFAC-TORY and the NRC has not verified that adequate corrective measures have been l t

instituted. Otherwise, the facility will be permitted to administer the reexaminttion for returning the individual to licensed duties; however, license renewal would require another NRC administered exam.

2. Unsatisfactory Requalification Program Evaluation For any program evaluated as unsatisfactory, the following actions are required unless findings indicate otherwise. Additional actions may be taken at the discretion of the Regional administrator or his designee. The sequence of actions below is not required. The deterinination whether plant shutdown is required should be ongoing until the Regional Administrator or his designee has reviewed all in (d) below.

a. Require the facility to identify program deficiencies and corrective actions required to improve operator performance. ,

Meet with senior facility management to review audit findings.

[b.

identified deficiencies, root causes, corrective actions proposed, schedule for corrective action implementation,'and follow-up inspections and examinations,

c. Deterinine the required corrective actions by the facility, the required follow-up by the NRC, and the schedule for each.

d. The Regional Administrator or his designee shall determine whether l

plant shutdown pending completion of corrective action is required.

This decision should be based on the:

l 1

(1) Significance of generic perforinance deficiencies identified during the program evaluation, i i

. Examiner Standards 00025.0.0 of 28

~

E5-601 Rev 6 10/1/88 (2) Recent SALP parformance, especially as related to Criterion 7, i Training Effectiveness and Qua'fification, (3) Recent facility events which relate to licensed operator performance, and (4) Recossnendations by resident inspectors.

H. Operator License Renewal Policy

1. Licenses for operators and senior operators will be renewed upon timely application, ar, described in ES-109/110 if the individual in question has successfully passed an MC requalification examination within the term of his/her license.

2. If an individual licensee has been administered an NRC requalification

' examination two (2) times during the term of his/her license without pass.

.ing any examination, then his/her license will be terminated or will ,

[expirewithoutrenewalandhe/shemayapplyforlicenseunder10CFR55.31 and sucessfully ccerplete an NRC license examination to maintain a license.

3. If an individual has not successfully passed an NRC requalification l , examination within the term of his/her license but has failed an NRC requalificationexaminationfwerthantuo(2) times,thentheindividual license is extended under the timely application provisions until the next NRC requalification examination. After the results of that examination, the individual will conform with either H.1 or H.2 above.

4. If an individual has passed one (1) NRC requalification examination curing the tenn of his/her license then the provisions of H.1 apply without regard to the facility requalification program status or any subsequent or prior unsuccessful HRC requalification examination (s).

Examiner Standards 00026.0.0 of 28

u ,

- ES-601 Rev 5 10/1/88

1. Final Requalification Program Evaluation Report

, A final requalification program evaluation report similar to the final examina-tion report for a licensing examination shall be prepared when the grading of reque.)ification examinations has been completed. A complete copy of the report shall be filed in the facility requalification file. A copy of the NRC Admin-1stered Requalification Examination Results Susuary, shall be forwarded to the Management Assistant, Regional Support and Oversight Section, OLB. The results sumary is required to verify OLTS data and for statistical data.

J. Record Retention

1. A facility requalification file shall be maintained for each facility.

All evaluation forms, records, assignment sheets, and correspondence relating to the requalification program audit for the latest two evaluations shall be retained.

2. ,

When the requalification evaluation has been completed by the Regional Office, a copy of all NRC administered written, oral and simulator

' examination results shall be supplied to the facility. The facilities are required to maintain these records until the operator's or senior

[ operator's license is renewed in, accordance with 10 CFR 55.59.

3. 'A copy of the results sumary shall be sent to the Management Assistant, Regional Support and Oversight Section, OLB. These sumaries shall be used for statistical data gathering, i

j 4. Material relating to an individual failure shall be retained by the

( Regional Office as necessary to support denial of license renewal per l 10CFR55.57(b)(2)(iv).

l l

Examiner Standards 00027.0.0 of 28

1. O, L t

ATTACHMENT 2 EXAMINEE CRITIQUE OF NRC REQUALIFICATION EXAMINATION As part of its efforts to improve all aspects of the NRC requalification examination process, NRC is asking all test-takers to provide feedback on these examinations. Please use this form to comment on the NRC requalification examination that you have just completed. After you have answered the questions below, and provided any comments that you wish to make, please give this form to your training manager who will send it to the NRC Regional Branch Chief, Please take a few minutes and respond to each of the following questions, by circling the number that best represents your experience on the examination just completed. Each question is written in the form of a statement, and the number which you circle indicates how strongly you agree or disagree with that statement. For example, if you circle number I for a particular question, that l

indicates that you are in strong agreement with the statement. If you circle number 4, that indicates mild disagreement. If you circle number 3, that indicates that you neither agree nor riisagree with the statement made in that question.

THERE ARE NO RIGHT OR WRONG RESPONSES. Your participation is strictly voluntary, but pleate remember that your responses, together with any comments that you might have, will help to improve the requalification examination process. All responses are anonymous.

WRITTEN EXAMINATION: ,

1. 'The content of the questions reflected the content of the requalification

,prograr at my plant.

Strohgly agree Strongly disagree 1 2 3 4 5 l Comments:

l l

l l

l l

l l

I 1

2. The overall time allotted for the examination was sufficient to answer c.

all the questions.

i Strongly agree Strongly disagree 1 2 3 4 5 Comme..ts:

3. The information to be used to answer each question was available.

Strongly agree Strongly disagree 1 2 3 4 5 Comments:

E

4. There was an opportunity for me to have any questions regarding the examination clarified prior to the end of the examination.

Strongly agree Strongly disagree 1 2 3 4 5 Comments:

2

OPERATING TEST: l i

1. The content of the JPMs and the questions on the walk-through portion was consistent with the content of the requalification program at my plant.

Strongly agree Strongly disagree 1 2 3 4 5 Comnents:

2. The overall time allotted for the walk-through portion of the operating test was appropriate.

Strongly agree Strongly disagree

. I 2 3 4 5 Comments:

3. The scenarios used for crew evaluation on the simulation facility portion of the operating test were consistent with team (crew) training at my plant.

Strongly agree Strongly disagree 1 2 3 4 5 Comments:

3

. 4 .

4. The instructions for the conduct of both portions of the operating test were clearly explained by the examiner (s).

Strongly agree Strongly disagree 1 2 3 4 5 Comments:

ADDITIONAL COMMENTS ON THE CONTENT AND/0R CONDUCT OF THE NRC REQUALIFICATION WRITTEN EXAMINATION AND OPERATING TEST JUST COMPLETED:

PLEASE CIRCLE THE LEVEL OF THE EXAMINATION JUST COMPLETED: RO SRO PLEASE 00 NOT INDICATE YOUR NAME ON THIS QUESTIONNAIRE.

THANK-YOU.

l This request is covered by Office of Management and Budget Clearance

' Number 3150-0011 which expires December 31, 1989. Comments on burden and duplication may be directed to the Office of Management and Budget, Room 3208, l

New Executive Office Building, Washington, DC 20503.

l l

4

ATTACHMENT 3

! INSTRUCTIONS ON USE OF SIMULATOR CREW EVALUATION FORM Enclosed is an evaluation form for use during the trial simulator examination component of the requalification examination. In keeping with the purpose of the requalification exam, these scales are geared toward evaluating the crew as a whole, rather than individual operators. Please follow the instructions below when rating team performance on the simulator examination:

1. Review the rating scales prior to the onset of the simulator examination to familiarize yourself with each performance issue to be evaluated.

2. Use the "Operator Actions" Form (ES-302, attachment 6), and the expected operator actions included on that form, to make notes during the examination, as described in ES-302.

3. Immediately after the simulator examination is over, evaluate the crew by completing the Simulator Crew Evaluation Form. Be sure to address all the rating factors for all 6 competencies.

4. Provide an overall rating of "satisfactory" or "unsatisfactory" for each competence. If you gave the crew more than one rating of "1" on the rating factors for that competence, your score for them on that competence overall should be "unsatisfactory." Although part of the purpose of the walk-through is to follow up on areas of weakness noted during the simulator portjon of the examination, if the crew receives a rating of "unsatisfactory"

• on one or more competences, their overall evaluation should also be unsatisfactory.

l

, 5. There is space for comments beneath each compets!nce rating and below the overall rating. This space is provided if you feel the need to annotate or explain rating (s). In particular, use this space to document the failure of an individual candidate due to his/her exceptionally poor performance.

We need your feedback to evaluate the usefulness of these ratings scales.

Atter you have had an opportunity to use the form, please provide us with the following information:

- tlere the scales clear, concise, and easily understood?

- Were the scales written so that you could evaluate crew (versus individual) performance?

- Could you comfortably decide which ratings to give the crew, based I on the decriptions of the performance provided? If not, how could these descriptions be improved?

- Were there rating issues which you believe are not appropritte for simulator requalification examinations?

- Were there any performance issues missing from these rating scales?

Please note your comments ard suggestions on the page provided at the back of the evaluation form and provide them to John Hannon at the exit meeting.

Thanks very much for your auistance.

l l

5

\

f SIMULATOR EXAMINATION SUMARY SHEET CREW MEMBERS:

Name Position OVERALL TEAM RATING ON THE SIMULATOR EXAMINATION:

Satisfactory Unsatisfactory Coments:

Please use the space below to note your coments on the evaluation form (attach additional pages, if necessary): '

1 l

l C

. ,r 1 UNDERSTANDING /INTERPRETATIDH 0F ANNUNCIATOP/ALARN SIGNALS

.I ],

DID THE CREW:

(a) NOTICE and ACKNOWLEDGE ala ms, and ATTEND TO alarms in order of their importance/ severity?

3 2 i All r, tares that directly Ninor awareness or Failed to notice related to significant response difficulties and/or extremely changes in plant or lapses slow at responding conditions were noted to significant alarms at critical times; easily distracted by nuitance alarms (b) Correctly INTERPRET the meaning and significance of alarms and annunciators (including the use of the Alarm Response Procedures, as applicable)?

3 2 1 Crew'readily determined Minor inaccuracies in Significant misin-what failures / events alarm interpretation terpretations, alarks were indicating but without safety resultino in plabt related consequences degradation (c) ' VERIFY that annunciators / alarm signals were consistent with p' ant / system condjtions?

3 2 1 All necessary verifi- Minor lapes in alarm Verification of cations performed, verification, but no failed systems including the identi- inappropriate actions was poor or fication of error,eous taken as a result of altogether absent alarms inadequate verification SCORE ON UNDERSTANDING / INTERPRETATION OF ANNUNCIATORS / ALARM SIGNALS:

Satisf actory Unsatisfactory Cornments: _

~

7

e . .

DIAGNOSIS OF EVENTS / CONDITIONS BASED ON. SIGNALS / READINGS DID THE CREW:

(a) RECOGNIZE off-normal trends / status?

3 2 1 Timely and accurate Recognition of trends Failed to recognize recognition of trends at time of, but not trends, even after even prior to alarms prior to, sounding of sounding of clarms and alarms annunciators (b) USE INFORHATION and use REFEPENCE MATERIAL (prir.ts, books, charts) to aid in the diagnosis / classification of ever.ts and conditions?

3 2 1 Correct, timely use Minor errors by crew Failure to use of information and in use or interpretation reference material, reference material of information and misuse / misinterpretation led to accurate reference material of information resulted diagnoses in improper diagnoses (c) . Correctly DIAGNOSE plant conditions based on those control room indications? ,

j 3 2 1 Diagnoses b;, crew Minor errors /diffi- Faulty diagnoses

were accurate and culties in diagnoses resulted in l timely incorrect control manipulations SCORE ON DIAGNOSIS OF EVENTS / CONDITIONS BASED ON SIGNALS / READINGS

l Satisfactory Unsatisfactory l

Comments:

l 8

. _ _ _ _ _ - _ _ ~ _ _ . _ _ _ _ _ _ .

UNDERSTANDING 0F PLANT / SYSTEMS RESPONSE

. 010 THE CREW:

(a) LOCATE and INTERPRET control room indicator: correctly and efficiently to ascertain and verify the status / operation of plant systems?

3 2 1 Accurate and efficient Minor errors in locating Serious omissions instrument location & or interpreting instruments delays or ir, accuracies interpretation by all and displays; some crew made in instrument crew members members required assistance interpretation (b) Demonstrate an UEDERSVANDING of how the plant, systems, and components operate, including setpoints, interlocks, and automatic actions?

3 2 1 All crew me:sbers demon- Minor instances of errors Inadequate knowledge strated thorough due to gaps in crew of system / component understanding of how knowledge of system / operation resulted in systems / components component operation; some serious mistakes or operate crew members required plant degradations assistance l

(c) Demonstrate an understanding of how their ACTIONS (or inaction) affected '

system / plant conditions?

I l 3 2 1 All members understood Actions or directives Crew appeared to act

( ,the effect ti.at indicated minor without knowledge of, l actions or directives inaccuracies in under- or disregard to, effect l

had on plant / system standing by individuals, on plant

conditions but actions were corrected by team SCORES ON UNDERSTANDING OF PLANT / SYSTEM RESPONSE

Satisfactory Unsatisfactory Comment:

l 1

1

-9 e 9

~

COMPLIANCE /USE OF PROCEDURES AND TECHNICAL SPECIFICATIONS DID THE CREW:

(a) REFER T0 the appropriate procedures in a timely manner?

3 2 1 Crew used procedures Minor failures by Failtd to correctly as required; knew crew to refer to refer to procedures when what conditions were procedures without required, resulting in covered by procedures prompting, but did faulty system operation

, and where to find them affect plant status (b) CORRECTLY IMPLEMENT procedures, including following procedural steps in correct sequence, abiding by cautions and limitations, selecting correct paths on decision blocks, and correctly transitioning between procedures?

3 2 1 Timely, accurate Minor instances of Importance procedural enactment of procedural misapplication, but steps were not enacted steps by crew, corrections made in correctly, which led demonstrating thorough sufficient time to to impeded and/or slow understanding of avoid adverse impact recovery or unnecessary procedural purposes / bases degradation (c) ' RECOGNIZE E0P ENTRY CONDITIONS and carry out appropriate immediate actions withc"t the aid of references or other forms of assistance?

3 2 1 Conshstently accurate Minor lapses or Failed to accurately and timely errors; individual recognize conditions recognition and crew members needed or execute actions, implementation assistance from others even with use of aids to implement procedures (d) CORRECTLY RECOGNIZE and COMPLY with Technical Specifications and Action Statements of LCOs?

3 2 1 Recognized and Minor d'fficulties i in Failure to recognize /

fully complied with referring to and/or comply with Tech Spec LCOs/ Action Statements applying Tech. Specs.; LCOs crew had to prompt SRO on TS requirements SCORE ON COMPLIANCE /USE OF PROCEDURES AND TECHNICAL SPECIFICATIONS:

Satisfactory Unsatisf actory Comments:

10

n; i ,

CONTROL 80ARD OPERATIONS DID THE CREW:

(a) LOCATE CONTROLS efficiently and accurately?

3 2 1

~

Controls and Instances of Instances of failure indicators were hesitancy / to locate controls located without difficulty in jeopardized system hesitation by locating controls status individual operators by one or more operators (b) MANIPULATE CONTROLS in an accurate and timely manner?

3 2 1 Smooth manipulation Minor shortcomings Mistakes made in of the plant in manipulations, manipulating controls within controlled but recovery from caused system parameters errors without transients and causing problems related problems (c) 'Take MANUAL CONTROL of automatic functions, when appropriate?

1  :

3 2 All operators took Minor delays and/or Failed to control cont,rol, and smuothly prompting necessary automatic systems operatored automatic before overriding / manually, even when ample time and systems manually, without operating automatic assistance, thereby functions, but plant indications existed averting adverse events transients were avoided when possible SCORE ON CONTROL BOARD OPERATIONS:

Satisfactory Unsatisf actory Comments:

11

- 1 a .

l l

COMMUNICATIONS / CREW INTERACT!DNS  !

DID THE CREW:

(a) EXCHANGE complete and relevant information in a clear, accurate, and I attentive mannse? )

l 3 2 1 Members informed each Communications generally Members did not other of relevant info. complete and accurate, inform each other and actively sought and but some instances of of abnormal indica-listened to info. from needing to be prompted, tions or when others as/when necessar, or failing to acknowledge performing evolutions; or respond to info. from inattentive when others important info. was requested or provided (b) INTERACT with other regarding issues / circumstances outside of their individual area of responsibility to facilitate safe plant conditions?

3 2 1 Members assumed Members listened to Members were responsibility for each others conversations inattentive to what issues outside their in general; major technical was happening own boards, as errors corrected around them; poor appropriate coordination of -

activities (c) ,MAKE TEAM DECISIONS in a timely, effective manner?

3 2 1 All jndividuals provioed Major team decisions Leader or other crew input to decisions. generally included members did not accept Decisions resulted in early, input from most crew input from others, recuperative action members, but some resulting in incorrect delays or other or untimely decisions /

problems in reaching directives effective decisions SCORE ON COMMUNICATIONS / CREW INTERACTIONS:

Satisf actory Unsatisfactory l Comments:

l l

l l

l l 12

1

- j ATTACHMENT 4 5

JOB PERFORMANCE MEASURE WORKSHEET Facility:

Task

Title:

Task No.:

Job Performance Measure No. :

K/A

Reference:

Operator:

Evaluator: Date:

Applicable methods of testing:

Simulate performance Actual performance Classroom Simulator Plant READ TO THE OPERATOR I wiki explain the initial conditions, which step (s) to simulate or discuss,-

and provide initiating cues. When you complete the task successfully, the obje.ctive for this job performance measure will be satisfied.

Initial Conditions:

Task Standards:

Required Materials:

General

References:

Initiating Cues:

13

e PERFORMANCE INFORMATION (Denote critical steps with a check mark)

"-  ! Performance step:

Standard: -

Comnent:

Performance step:

Standard:

Comment:

Performance step:

Standard: '

Comment:

Performance step:

< Standard:

Comment:

Performance step:

Standard:

Comment:

Terminating cue:

14 i _ _, ___ _ _ _ _ _ _ __ _ __ . _ _ _ _ . . . _ , _ _ _ _ _ - _

n;.

. \

• VERIFICATION OF COMPLETION

{

( '; Job Performance Measure No.

Operator's Name:

Date performed:

Evaivator:

Number of attempts:

Result: (Denote by an 5 for satisfactory or a U for unsatisfactory and requires remedial training).

Evaluator's signature and date: .

4 8

4 e

G e

e e

5 d

15

• i .

w ATTACHMENT 5 x (

SYSTEM WALK-THRODGH TEST PLAN Page__of_ ,

Facility: System Topic: Performance Measurement:

Question  : KA Rating Answer:

Response / Comments: _

Question  : KA Rating Answer:

Resp'onse/ Comments :

l l

Question  : KA Rating Answer:

Response / Comments:

Question  : KA Rating Answer:

l Response / Comments:

i 16 L

SYSTEM WALK-THROUGH FOLLOW-UP DOCUMENTATION

Question Derivation (can be completed af ter examination)

Simulator Evaluation Deficiency (identify)

Topic Area Evaluation: 1. Performance Measurement

2. Question Number _

Question (to be completed during exam):

Response (to be completed during exam):

Question Derivation (can be completed after examination):

Simulator Evaluation Deficiency (identify)

Topic Area Evaluation: 1. Performance lleasurenent

2. Question Number ,

Question (to be completed during exam): _ _

Response (to be completed during exam):

Question Derivation (can be completed af ter examination):

Simulator Evaluation Deficiency (identify) ,,

Topic Area Evaluation: 1. Performance Measurement

. 2. Question Number Question (t'o be completed during exam):

Response (to be completed during exam): ,

Question Derivation (can be completed after examination):

Simulator Evaluation Deficiency (identify)

Topic Area Evaluation: 1. Performance Measurement

2. Question Number Question (to be completed during exam):

Response (to be completed during exam):

17

~

~ ~ '

g. . ,

ATTACHMENT 6 BRIEFING CHECKLIST - SYSTEM WALK-THROUGH

1. If the NRC examiner is a visitor, escort responsibility for ensuring compliance with safety, security and radiation protection procedures l' is the responsibility of the operator escorting the examiner.

l

2. Plant equipment should not be operated. Nothing the' facility or NRC examiner says or asks will be intended to violate that principle.

3. If clarification of questions is needed during the wolk-through, there should be no hesitation to request the examiner reword or clarify the question.

4. The examiner will be taking notes throughout the test to document operator performance. Frequently an examiner will stop questioning for this purpose.

The amount of note-taking is not dependent upon the operator's level of performance. The examiner must document satisfactory as well as less than satisfactory performance.

5. The walk-through is considered "open book." The reference material in the facility / control room which is normally available to operators is available, including calibration curves, previous log entries, piping and instrumentation

' diagrams, calculation sheets, and procedures. However, operators are respons-ible for knowing from memory the immediate actions of emergency and other

' procedures as appropriate to the facility. .

6. 'The system walk-through has been planted for approximately two hours in length. However, there is no specific time limit for the walk-through.

,The examiner will take whatever time is necessary to cover the areas selected, in the depth and scope required. There will be a minimum of 4

, Job Performance Measures (tasks) evaluated from the control room and 4

, Job Performance Measures evaluated outside of the control room. However, the total number of JPM's will be no less than ten (10).

7. The examiner will explain whst tasks are to be completed, which steps to simulate or discuss and provide initial conditions. The operator is to proceed with completing the task as if directed by plant procedures and/or shift supervision. During the task the examiner will supply the necessary plant conditions and/or parameters needed to simulate the task. The operator should explain each step of the task to the examiner before doing it.

8. When all of the steps for each task are performed correctly, the criteria for the examination will have been completed.

9. If the operator feels the need for a break during the walk-through, the operator should request this from the examiner. The examiner is not allowed to reveal the results of the walk-through at its conclusion.

10. The NRC examiner may ask clarifying questions of the operator at the end of each JPM.

18

i +

4 Part 8 - For tests with simulation facility available i

1. The primary responsibility is to operate the simulation facility as if it were the actual plant.  ;

1

2. Team work and communication between operators is evaluated. It benefits I the exam process to verbalize observations, analysis, and reasons for i actions more than normally would be done during actual plant operations.

3. If an operator recognizes an incorrect decision, response, answer, analysis, action taken, or interpretation of the team of which the operator is a part i but fails to correct, then the examiner may assume that that operator l agrees with the incorrect item. '

4. A rough log may be kept during each exercise that would be sufficient to complete necessary formal log entries which may be evaluated under admin-istrative topics.

5. A designated facility instructor will act as the auxiliary operators, radiation health and chemistry technicians, maintenance. supervisors, plant management, and anyone else needed outside the control room area.

6. The facility examiner will provide a shift turnover before the exercise begins. The shift turnover will include present plant conditions, power history, equipment out of service, abnormal conditions, surveillance due,

'and instructions for the shif t.

7. ',The control board switches may be purposely misaligned to enhance a ,

simulated scenario or transient where appropriate and is not part to th'e i ' evaluation. If misaligned, they should be tagged or otherwise highlighted as appropriate to the facility. The examiner will not misalign switches lduring the scenario as an awareness drill.

1 l Note: The chief examiner has the option to tell the operators that no control l board switches will be misaligned on a given scenario or set of scenarios.

If no switches are misaligned, the chief examiner may wish to reduce the time it takes of the operators to comp *sete the board walkdown and accept the shift.

8. Operators will be allowed three to five minutes to familiarize themselves with the status / conditions of the control boards prior to the start of the exercise.

9. The simulation facility part of the examination will consist of a minimum of two exercises lasting approximately 50 minutes each. There will be a short break between exercises to set up the initial conditions for the next exercise.

10. If operators have any questions concerning the administration of the operating test, those questions should be answered prior to the start of the test.

19

- . j r

ATTACHMENT 7 U. S. NUCLEAR REGULATORY COMMISSION

, REACTOR OPERATOR REQUALIFICATION EXAMINATION FACILITY:

REACTOR TYPE:

DATE ADMINISTERED:

OPERATOR: )

l

% OF SECTION CATEGORY OPERATOR'S CATEGORY VALUE SCORE VALUE A Plant Proficiency B Limits and Controls 1

Final Grade 6

20 i

. '. ATTACmENT 8 JOB PERFORMANCE MEASURE QUALITY CHECKLi$T i

1. Supported by facility's job / task analysis.

2. Operatierellyimportant(meetsthresholdcriteriaofK/A3.5oras determined by the facility and agreed to by NRC).

3. Designated as either SRO only or both R0/SRO.

4. Time vandated (time allowed for task completion indicated on JPM).

Questions related to the task need not be time validated.

5. Either the task itself or the questions that follow require operator problec recognition and diagnosis. Questions require knowledge correct factions for abnormal system responses whenever applicable. ,

6. ' Performance standards are specific in that exact control and indication nomenclature and criteria (switch position, meter reading) are specified, even if such criteria are not specified in the procedural step.

7. Critical elements and associated performance standards are identified and agreed to by the facility and the NRC.

l

8. Performance standards should provide complete and proper system response cues where appropriate such that the examiner can properly cue the ,

operator when asked.

1

9. Answers to questions at the end of the task are. MOT found directly in the procedurejustused(i.e.,donotaskwhyacertaincautionexistsifthe caution itself identifies the answer).

8 O .

O f

l l

\

' - ATTAC W NT 9 Rev. 1 ,

4/15/88 l l

SIFULATOR SCENARIO REVIEW CHECKLIST (Attach separate copy to each scenario reviewed)

REVIEWER:

SCENARIO IDENTIFIER:

1. Sceneriocontainsatleastone(1) time-criticalcrewresponse.

Time-critical is defined to require affirtrative action by one or more crew member (s) ir, order to prevent or mitigate an event within a limited time.

2. Scenario contains at least one (1) team-dependent crew respcase.

Tean-dependent is defined to require transfer of information between crew members in order to prevent or mitigate an event.

'3. The scenario events involve each crew member.

4. Scenario is composed of related or linked events.

5. Scenario requires the use of: (check those that apply)

Abnormal operating procedures Emergency operating procedures Technical Specifications i

Emergency plan implementing procedures l

s

.t.

6. Critical tasks are:

Identified by facility Required to prevent or mitigate event importance factor 3.5 in K/A Catalog or facility JTA Scerario events have K/A importance factor 3.0 7.

t b o

e

i ,

3, ,

ATTACHMENT 10

(

GUIDELINES FOR THE DEVELOPMENT AND REVIEW 0F OPEN REFERENCE EXAMINATIONS ,

1. Introduction The following. guidelines are intended for thnse who are involved in the development and/or review of test items for the written portion of the NRC Requalification Examination. As described in ES-601, "Administration of NRC

/ Requalification Program Evaluations," the written examination consists of two sections, one that utilizes the simulator to provide a context for questions on plant systems and controls, and a second that focuses on plant procedures and administrative controls. Both sections are administered in an "open reference" format. Candidates are allowed to use reference material, including simulator displays, during examination administration.

The intent of the change from closed to open reference written examinations, is twofold:

l i

l (1) Examination Validity. By permitting the use of references that are avtilable to the operator, the ccnditions and requirements of the written examination rore closely approximate those of the actual job. The i

information provided to the operators in the test items can and should closely l parallel the information typically available to them on the job, while the responses elicited by the questions should be similar or identical to the' decisions, solutions, and actions required for effective job perforvence. In otherwords,theopenreferenceforretenhancesthematchbetweenjobdemands 1 and test demands -- a cornerstone of examination validity.

i l

l

j 'I

.'t -

l l

( (2) Level gf Knowledge. The open reference format also enhances examination validity by elevating the level of knowledge of the test items. As described later in these guidelines, candidate access to references precludes the use of questions that test for the mere recall of facts and specifics. Instead, open reference test items require test takers to demonstrate that they can apply, j analyze, evaluate, or otherwise USE knowledge to handle the problems and )

issues encountered on the job.

11. Open Reference Guidelines j Most principles for effective test item construction apply equally to all types of written questions, regardless of format. Therefore, open reference test item developers and reviewers should consult references such as NUREG BR-0122, "Examiners' Hendbook for Developing Operator Licensing Examinations,"

INP0's "Principles of Training System Development Addendum I, Test Item Developmert," and NUKARC's "Guidelines for Developing Written Test items for the NRC Requalification Exam." The guidelines below are those that have been found to be especially pertiner.t to the creation of open reference test items.. These guidelinas are divided into five categories:

(1) Selection of Test Topics (2) General Guidelines for Sections A & B (3) Specific Guidelines for Section A, "Plant Operations" (4) Ideas for Open Reference Formats (5) Open Reference Test Item Review Check 1ht (1) Selection of Test Topics Test item topics for the NRC requalification examination should be selected based on the following criteria.

A. Requalification Training Program Curriculum. Test topics should be based on the curriculum of the most recent operator requalification program training cycle. However, NRC nay substitute up to 20'4 of the examination topics selected by the facility with subjects not emphasized during the requalification cycle under 10 CFR 55.59.

o

[ *

- s S. Performance, Basis. Like the requalification program itself, test topics shouls be drawn from a job-task analysis (JTA) of the operator and senior operator positions. The facility should validate their test items by demonstrating a link between each item and the following JTA products:

- important operator tasks as identified by the JTA

- irportant K/As (rated 3.0 or higher) as identified in the NRC K/A Catalog (NUREG 1122/1123) or a facility-specific K/A catalog I - facility learning objectives identified as important to safety 1

C. Adequacy of Test Coverage. The facility's proposed sampling plan (or curriculum evaluation plan) should by checked to ensure that it provides balanced, cocprehensive coverage of the topics covered during the requalificaticn training cycle. Facility test item topics may be revised if subject areas are found to be under or over-represented in the sampling plan relative to their coverage in the requalification program. In addition, 201 may be substituted by the NRC. Recent safety-related issues and events (e.g., relevant LERs) should be addressed in the sampling plan. *

(2) henoral Guidelines The ollowing guidelines should be followed in the construction and review of test items for both parts of the written examination. These guidelines are intended to supplesunt, not replace., the good practice criteria found in NUREG BR-0122 and related documents.

A. Operational Orientation. As discussed earlier, examination validity is enhanced to the extent that the demands of the test satch the demands of the job. Therefore, in addition to being derived from important K/As and testing objectives, the context and stipulations of test items should mirror the situations encountered in the work setting. The following exarple illustrates effective and ineffective ways to design test items free K/As and learning objectives.

, f K/A: Knowledge of the design attributes of the Turbine Driven Auxiliary Feedwater Pump Differential Pressure Controller.

Task: Operate the TDAFWP controls during all modes of plant operation.

Terminal Learning Objective: The student will be able to operate the TDAFWP Differential Pressure Controller without error during a loss of feedwater event.

Enabling Objective: Upon completion of this lesson, the student will be able to explain the operation of the TDAFWP Differential Pressure Controller.

Poor Test Item: State the parameters used by the TDAFWP Differential Pressure Controller.

Better Test Item: Prior to isolating the "C" steam generator (per EPP11), it was n,oted that the transducer. fed auxiliary feed flow indicators for the "C" steam generator were reading greater than the flow indicators to the "A" and .

"B" steam generators. What is the reason for this flow deviation?

Notice that the second test item requires the candidate to demonstrate mastery of the knowledge by applying it to an actual job situation. In developing items, it say be useful to ask oneself "why is the K/A important to satisfactory job performance?" and "in what situation will be operator need this K/A?" The answers to these questions can provide a basis / context for test items.

B. Level of Knowledge. The operational orientation required of test items on the open reference examinations, as weil as the candidates' access to controlled documents, precludes the use of questions that test for mere recall or memorization. Rather than requiring candidates to simply recognize or recall facts and specifics, open reference test items should have the candidates demonstrate understanding by requiring them to use their knowledge to address real-life situations and problems. A test item at the

^ ^

,1 % ,

higher level of knowledge requires candidates to determine or identify the appropriate f act, rule, or principle to a novel situation and then correctly apply it. A description of each level of knowledge, along with common verbn and example questions, is found in Table 1.

C. Realistic Context. To provide additional assurance of examination validity, the situation or problem posed in the open reference test item should be as similar as possible to the actual situations that candidates encounter or. the job. Situations described in the questions should not only be realistic, but should also be free of common "context" problems, including "backwards logic" and "window dressing."

Backwards logic questions provide candidates with information they normally For have to produce, while asking them for information they norum11y receive.

example:

X/A: , Ability to calculate shutdown margins.  :

Backward Logic Iten: Given a shutdown margin is 5.51, how long has the unit been shut down?

Calculate the Better Iter: The unit has been shut down for x hours.

shutdown margin.

.................................e............. ...........................

Questions with window dressing have additional, unnecessary inforsation, typically in an attempt to make a memory level item more operationally oriented. For example:

i Item with Window Dressing: The plant has tripped due to the effect of a l tornado crossing the site boundary. You, as Shift Supervisor, direct the l phone talker to complete the 15 minute notifications. He informs you that the normal notification network is inoperable. What method do you direct him to j use for completing the 15 minute notification? )

\

Revised Iter: If the nomal notification network is inoperable, what methed j do you direct the phone talker to use to complete the 15 rinute notification after the plant has tripped? l Another ccoon problem when constructing questions with realistic contexts is that quite often "real world" situations have more than one correct solution or response. Check the question and references over carefully to ensure that each' test item has only one correct answer.

D. puestionNovelty. One of the most effective ways to ensure that -

candidates have higher levels of knowledge is to present them with novel situa'tions and reouire them to both realize what information is relevant and how to apply it. If test questions do not contain unique or varied circumstances compared to that which was presented in training, the item will be reduced to simple recall. Review the training material to ensure that questions do not include overly familiar conditions. Keep in mind, however, that all conditions and situations should be reasonable, realistic, and safety-related.

E. Use g References. References should be considered tools that candidates use to solve problems. It should be the proper use of these tools that is tested during the open reference examination, not the recall of facts and specifics. Purely "look up" questions should not be included in the examination; rather, questions should be restricted to those that test to see if candidates can identify, locate, or select appropriate reference information to produce organized responses and satisfactory solutions to job related problems and issues. For example:

[

Pure "Look up* Itein: In the event that a safety limit is violated, the reactor shall be:

a. placed in a hot shutdown condition within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />

b. placed in a hot shutdown cor.dition within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />

c. placed in a cold shutdown condition within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />

d. pieced in a cold shutdown condition within 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.

Better Use of

References:

While operating at 100% power, VCT and pressurizer alarms and indications show decreasing pressurizer level. Also, the blowdowr and rain stean radiation monitors have alarmed. While following the appropriate Abnormal or Emergency Procedures, you as the Shift Supervisor must evaluate the existing condition.

Based on this information, the following emergency classification should be declared:

,a. Notification of Unusual Event I

'b. Alert

c. Site Area Emergency

d. Gene al Emergency F. Difficulty Level. It is not unusual for test constructors to believe, erroneously, that open reference test items should be more difficult, to cocpensate for the candidates' access to reference reterial. Frequently, this increased difficulty is in the form of requiring knowledge of more obscure or otherwise unnecessary information. Both open and closed reference examination items should have the same standard of difficulty; that is, difficulty should be based on the job demands and responsibilities of operators, l G. Time Limits. Relative to closed reference examinations, candidates take '

considerably longer to answer open reference test items. (Weaker candidates especially have been found to spend an appreciable amount of exam time

~. - - -

-r- - . - - - - -

._--__--_____m-- _.,____.,__m_ _ - -___ - _. ,-.m-_ ,,7_ . , - - _._ , ,_-- -- -, . . - - - - , - -

t

• i consulting references versus writing responses). It is impertant to provide f candidates an ample amount of time to complete the examination, although not so much tint as to allow less than coe.petent operators the opportunity to locate answers without prior familiarization. The following guidelines should be used to determine the appropriate length of the examination:

1. Each two hour examination section should be constructed to take a competent operator an expected 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> 15 minutes to complete.

2. Wher, possible, the response tine of each question should be estimated by having a subject matter expert actually answer the question, including searching through references.

H. Correct Mode of Measurement. No matter how high their importance ratings or operational relevance, certain operator knowledges, skills and abilities are not amenable to written testing. For example:

Arrange the major steps in the proper sequence to start, parallel, and ,

Ioad00-2:

Use Governer Control to increase DG.2 KW

- Raise DG Speed to 900 RPM Press start button on A130B Match Voltage with Bus IA2 Voltage Close Breaker IAD2.

Despite its operational orientation, the underlying skill addressed in the above test item would be better assessed by having the candidate simulate or step through the steps during either the sirolator or walkthrough portions of the operating examination. Table 2 provides an overview of the best uses for each test mode. Make sure that the K/As and learning objectives selected for the t

written examination can be effectively measured during the written testing.

(3) Specific Guidelines for Section A, "Plant Operations

• The followins guidelines are specific to the Plant Operations section of the written examination, performed on a static simulater. These guidelines are divided into two sections. Question Development and Simulater Setup.

Question Development

a. To ensure that the operators' knowledge of plant operations is adequately evaluated, Section A of the written examination should address a mix of normal, abnonnal and emergency redes of operation,

b. Questions should require the operators, to the extent possible, to refer to control room indications in forr.ulating their responses, as in tha following example:

yhich one of the following describes the location of the steam break? ,

(a) Insice containment, upstrean of the steam line flow transmitters

('b) Inside containment, downstream of the steam line flow transmitters (c) Outside containment, between "C" MSIV and "C" main steam lin'e check valve.

(d) Outside containment, between "C" MSIV and "C" main steam line containment penetration.

c. The nes6er of scenarios used should be minimized due to the extensive amount of time necessary to set up, run and check the transients.

d. The number of malfunctions / failures for each scenario should be limited. In general, the scenario should contain one major failure (e.g.,

LOCA, SGTR, steam line breaks, ejected control rods, loss of all AC power).

In adiition to the major failures, no more than four minor failures should be used (e.g., failure of a safety related pump to start, failed pressurizer

. .. i

. i pressure meter indication, nuclear instrumentatica failure). In many cases, one major failure and 2-3 minor failures will trovide sufficient effects to test a wide range of objectives.

e. Questions may be used that do not relate to the transient but use the simulator as a frare of reference only, provided the candidates are aware of this lack of relationship to the transient.

f. Special attention should be given to ensure that multiple questions steming from one event do not suffer from double jeopardy. The candidate should be able to be understand and correctly answer each question based only on the infomation given in the question, rather than on the answer to a previous question.

Sirolator Setup

g. Prior to the test, the simulator recorders should be rotated to provide clean readings, and the recorders should be checked for proper operation. .

l l h. All indications should be checked (e.g., bulbs, meters, r,anual loader

indications, etc) te ensure they are in proper working order.

l .

1. When the simulator has been frozen, the chart recorder drive power should be secured, if necessary.

J. Prior to administering the test, the simulater indications should be verified proper based on expected question responses.

L Any "first-out" annunciators that would normally blink to announce first-out conditions should be frozen and provided to candidates.

1. If a transient is stabilized by use of plant procedures, the step at which the simulator is frozen should be noted and this information recorded on the simulator operations sumary sheet. Progress of the procedure step in effect should also be given to the examinees as necessary.

i

• i i

l t

. (4) Tdeas for Open Book Formats.

Tat,le 3 provides a list of sample formats to assist question developers in enerating performance 4ased, cpen reference test items.

e  %

e O

4 e

i 1

1 i

l 4

.g ,

TABLE 1 LEVEL OF KNOWLEDGE DESCRIPTIONS AND EXAMPLES

1. MEM0RY The memory level involves the recall of facts and specifics, the recall of methods and processes, or the recall of a pattcen, structure, or setting.

For measurement purposes, the recall sitt.ation involves little more than bringing to mind the appropriate material. Appropriate verbs include:

To define To develop To distinguish To outline To recall To identify ,

To recognize To list For the operator licensing examination some examples include:

State the basis for a procedurai step or caution.

State the basis for a procedural change.

State the purpose of a specific procedure.

Items that require only memorization or recall are not permitted on open-reference examinations.

2. COMPRElENSION Comprehension represents the icuest level of true understanding. It refers to a type of understanding such that the individual knows what is i

being cosaunicated and can make use of the material without necessarily

- relating it to other material or realizing its fullest implications. Appropriate Y verbs include:

To translate To estimate To prepare To differentiate To comprehend To explain To interpret To sumarize To grasp To deinonstrate by example To distinguish To see implication effects, and consequence Questions based at this level and above are permissible to ask on an open-reference examination.

3. APPLICATION These types of questions require candidates to apply the knowledge to various i;oncrete situations. The knowledge may be in the form of general ideas, rules of procedures, or generalized methods. Appropriate verbs include: .

To apply To predict To employ To use To relate To develop For the operd or licensing examination examples of this cognitive level would include:

Calculation of plant parameters.

Use of reference material such as graphs, charts, curves, etc.

4. ANALYSIS The analysis level involves the breakdown of a cosinunication into its constituent elements or parts such that the relative hierarchy of ideas is made clear

.,m_ _ - . _ . . . - . - - .

and/or the relations between the ideas expressed are made explicit.

Appropriate verbs include:

To discriminate To categorize ,

To analyze To choose To detect To disco)er To infer To select

5. SYNTHESIS Synthesis involves the putting together of elements and parts so as to fonn a whole. Appropriate verbs include:

To create To perceive To propose To organize To integrate To prepare To plan To compile To design To incoro m te

', To synthesize To visut. 1 ,

6. . EVALUATION The evaluation level involves judgment about the value and methods for given l

I purposes. Appropriate verbs include:

! To judge Ta evaluate l To assess To decide i To compare 70 determine To appraise Evaluation is the optimum level of open-reference procedures questions.

Typically questions at this level would propose a situation and require the operator to use analysis and synthesis to formulate judgments /decisiens about appropriate actions, l

EXAMPLES A knowledge level question, requiring the operator to respond from memory on the basis of a caution:

A caution at the beginning of FRP-H.1 requires RCS bleed and feed to be started at STEP 9 if ANY S/G wide range level decreases to less than 60%

[631]. Which item below is the basis for this caution?

A) Steps 1 through 8 deal only with diagnostic evaluation of the event ar.d may be time censuming.

B) Steps 1 through 8 consist of a "loop" that is difficult to exit from.

C) Wide range level is not calibrated (therefore not accurate) for 60%[63%).

D) This 2evel assumes S/G dryout is intinent and RCS bleed and feed must be started immediately for core cooling.

A comprehension level question, requiring the operator to differentiate between types of turbine runbacks.

Indicate what type of turbine runback occurred: LOAD LIMIT, LOAD REFERENCE or BOTil.

The follwing question falls into the application level (i.e., applying a procedure).

HBR Unit #2 is at 100% pwer, steady state. The HP-97 calculator in the control room is 005. Based on the values given belw, determine the "QUADRANT POWER TILT" as defined in Technical Specifications.

NORMAL UPPER DETECTOR CURRENTS R.41 = 230 N-42 = 235 N-43 = 232 N-44 = 236 NORPAL LOWER DETECTOR CURRENTS N-41 = 233 N-42 = 238 N-43 = 231 H-44 = 240 PRESENT INDICATED DETECTOR CURRENTS I .

UPPER LOWER l N-41 232 236 .

'N 42 238 243 l .N.43 235 234 b.44 239 243 t

t l The follwing question is at the analysis level, it requires the operator I

analyze conditions and discriminate an idea:

Determine the posting reqaired for .) room using the results of the follwing radiological survey:

1. AIRBORNE ACTIVITY: 6.34E-9uci/cc(Co-60)

2. FLOOR SMEAR: Beta. 610 dpm/cm squared; Alpha- 4dpm/cm squared

3. EQUIPMENT SMEAR: Beta.1800 dpm/cm squared: Alpha- 16 dpe/cm squared

4. GENERAL RADIATION LEVEL: 110 mr/hr i

The following question represents the synthesis / evaluation level -- putting together of elements and parts to form a whole:

Determine if plant conditions satisfy the requirements of foldout "B" SI TERMINATION CRITERIA. Justify your answer with specific values of any required plant parameters.

t

-- , , , - > > - , , ----- n- - . - ... ----,- - --- -~-----e --

- o TABLE 2 OVERVIEW 0F THE BEST USES FOR EACH TEST MODE Written: .

- Knowledge and abilities that are difficult to infer from behavior alone.

- Knowledge of factual information.

- Paper & pencil abilities and skills (e.g., calculations)

- Responses requiring fpformation that can be supplied on paper

- Interpretation of referente, if open reference Walkthrough:

- Areas needing interpretation

- Areas needing props

- Knowledge of locations

- Interpretation of references

. - - Administrative requirements Simulator: .

- Overall ability to operate

- Integrated use of knowledge and abilities

- Conmunications

- Team Interactions

- Tire-critical

\.  % .

. l l

e '

TABLE 3 EXAMPLE FORMATS FOR OPEN REFERENCE QUESTIONS 1

- PROVIDE THE OPERATOR WITH: REQU)RE THE OPERATOR 70:

1. Plant / System / Component Diagnose cause of the problem (s)

Condition (s)/ Problem (s)

2. Plant / System / Component Identify location of problem (s)

Condition (s)/ Problems (s)

3. Plant / System / Component Indicate a Condition (s)/ Problem (s) action (s)ppropriate(recuperative)

4. Plant / System / Component Indicate actions to achieve conditions specified effect

5. Plant / System / Component Identify precipatory events /

Conditions actions

6. Pl[ ant / System /Compenent Classify / Categorize or Conditions / otherwise indicate if conditions meet specified criteria

7. P.roposed/ Hypothetical Coment nn Appropriateness /

course of action /recomendation Acceptability of these actions /recomendations

8. Plant conditions and operator Indicate purpose of/ reasoning actions / procedural steps behind these actions / steps

9. Requisite data Computation of parameters

10. Plant conditions and/or Predict expected plant /

operator actions system / component response (s)

11. System / Component status Indicate effect on same or othersystem(s)/ component (s)

12. Plant / System /Cwponent Indicate proper procedure (s)/

Conditions references to turn to.

s c.

ATTACHMENT 11 NRC CHECKLIST FOR OPEN REFERENCE TEST ITDIS Item Level

1. Does each test iter, have a documented link to important operator tasks, K/As, and/or facility learning objectives?

2. Is each test item operationally oriented, i.e., is there e match between job demands and test demands?

3. Is the question at least at the "comprehension" level of knowledge?

4 Is the centext of the questions realistic and free of window dressir.g end backwards logic?

5. Does the item require an appropriate use of reference material, i.e., is it free of "look up" questions?

6. Is the item at the correct level of difficulty for the job position?

7. Is the item appropriate for the written examination and the selected written exam format (e.g., shcrt answer; multiple choice)?

8. I's an appropriate mix of operating modes presented in the scenarios in ,

S.ection A "Plant Operations?"

9. Do questions in Section A take advantage of the simulator control room s,etting?

10. I's the item fr of double jeopardy?

11. I's the item clear, precise and easy to read and understand?

12. Is there only one correct answer to the question?

13. Does the item pose situations and problems other than those pr&sented during training?

Test Level

1. Does the facility sampling plan adequately cover the requalification topics?

2. Does the facility sampling plan ensure comprehensive, balanced coverage of the requalification program topics?

3. Can the test be completed in the time allotted?

.__ _ _ ___ -__. _ - . ____ . _ . _ . _____ __ _ - _ . . . _ _ .__