Text

Operator Licensing Examination Standards for Power Reactors
	ML20206N219
Person / Time
Issue date:	04/30/1999
From:	; Office of Nuclear Reactor Regulation
To:	;
References
	NUREG-1021, NUREG-1021-R08, NUREG-1021-R8, NUDOCS 9905170173
	Download: ML20206N219 (500)
	v • d • e

{{#Wiki_filter:,- NUR EG-1021, Rev. 8 I Operator Licensing Examination Standards for Power ~ Reactors j Tusi-- l A f i M

l

M m U.S. Nuclear Regulatory Commission g Office of Nuclear Reactor Regulation Washington, DC 20555-0001 ,fa a. y 9 73 990430

   $                PDR L

NUREG-1021, Rev. 8 [ Operator Licensing Examination Standards for Power Reactors t i! U.S. Nuclear Regulatory Commission Office of Nuclear Reactor Regulation Washington, DC 20555-0001 g% , % y /

  ; =:

i

NUREG-1021, Rev. 8 Operator Licensing Examination 1 Standards for Power Reactors Manuscript Completed: March 1999 Date Published: April 1999 Division ofInspection Program Management OITice of Nu: lear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, DC 20555-0001 saano uy

    -ga
     % . . . . . .o O

z

AVAILABILITY NOTICE Availability of Reference Materials Cited in NRC Publications NRC publications in the NUREG series, NRC regu- NRC Public Document Room lations, and Title 10, Energy, of the Code of Federal 2120 L Street, N.W., Lower Level Regulations, may be purchased from one of the fol- Washington, DC 20555-0001 lowing sources: < http://www. nrc.g ov/N R C/PD R/pd r1.htm >

1. The Superintendent of Documents 3W2% m bcah 20244m U.S. Government Printing Office Microfiche of most NRC docurrents made publicly RO. box 37082 available since January 1981 may be found in the Washington, DC 20402-9328 Local Public Document Rooms (LPDRs) located in

     < http://www. access.gpo. gov /su_ docs >       the vicinity of nuclear povter plants. The locations 202-512-1800                                    of the LPDRc may be obtained from the PDR (see Previ US Paragraph) or through:

2. The National Technical Information Service Springfield, VA 22161-0002 <http://www.nrc. gov /NRC/NUREGS/

     <http://www.ntis. gov /ordernow>                     SR1350/V9/lpdr/htmi>

703 -487-4650 Publicly released documents include, to name a The NUREG series comprises (1) brochures few, NUREG-series reports; Federal Register no-(NUREG/BR-XXXX), (2) proceedings of confer- tices; applicant, licensee, and vendor documents ences (NUREG/CP-XXXX), (3) reports resulting and correspondence; NRC correspondence and from international agreements (NUREG/lA-XXXX), internal memoranda; bulletins and information no-(4) technical and administrative reports and books tices; inspection and investigation reports; licens- [(NUREG-XXXX) or (NUREG/CR-XXXX)), and (5) ee event reports; and Commission papers and compilations of legal decisions and orders of the their attachments. Commission and Atomic and Safety Licensing Boards and of Office Directors' decisions under Documents available from public and special tech-Section 2.206 of NRC's regulations (NUREG. nical libraries include all open literature items, such XXXX) as books, journal articles, and transactions, Feder-al Register notices, Federal and State legislation, A single copy of each NRC draft report is available and congressional reports. Such documents as free, to the extent of supply, upon written request theses, dissertations, foreign reports and transla-as follows: tions, and non-NRC conference proceedings may be purchased from their sponsoring organization. Address: Office of the Chief information Officer Reproduction and Distribution Copies of industry codes and standards used in a Services Section substantive manner in the NRC regulatory process U.S. Nuclear Regulatory Commission are maintained at the NRC Library, Two White Flint Washington, DC 20555-0001 North, 11545 Rockville Pike, Rockville, MD E-mail: < DISTRIBUTION @nrc. gov > 20852-2738. These standards are available in the Facsimile: 301 - 415- 2289 library for reference use by the public. Codes and A portion of NRC regulatory and technicalinforma-standards are usually copyrighted and may be Purchased from the originating organization or, if tion is available at NRC's World Wide Web site: they are American National Standards, from-

      <http://www.nrc. gov >

American National Standards Institute All NRC documents released to the public are avail- 11 West 42nd Street ; able for inspection or copying for a fee, in paper, New York, NY 10036-8002 microfiche, or, in some cases, diskette, from the <http://www. ansi.org > Public Document Room (PDR): 212 -642 - 4900 J el I l

ABSTRACT V NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," establishes the policies, procedures, and practices for examining licensees and applicants for reactor operator and senior reactor operator licenses at power reactor facilities pursuant to Title 10, Part 55, of the Code of Federa/ Regulations (10 CFR Part 55). The related guidance that was previously published in the " Examiners' Handbook for Developing Operator Licensing Written Examinations" (NUREG/BR-0122, Rev. 5, dated March 1990) has been incorporated herein. NUREG/BR-0122 is no longer in effect. These examination standards are intended to assist NRC examiners and facility licensees to better understand the processes associated with initial and requalification examinations. The standards also ensure the equitable and consistent administration of examinations for all applicants. These standards are for guidance purposes and are not a substitute for the operator licensing regulations (i.e.,10 CFR Part 55), and they are subject to revision or other changes in internal operator licensing policy. This revision implements an amendment to 10 CFR Part 55 that allows facility licensees to prepare the entire operator licensing examination and to proctor and grade the written portion of the examination. The NRC will prepare the examinations at least four times per year to maintain the proficiency of its examiners, as necessary to ensure quality, and upon written request by facility licensees consistent with NRC staff availability. This revision will be used for all examinations administered after the effective date of the associated amendment to 10 CFR Part 55 or at an earlier date agreed upon by the facility V licensee and its NRC Regional Office. j

 \                                                    iii                     NUREG-1021, Revision 8

w-TABLE OF CONTENTS TABLE OF CONTENTS

    . EXECUTIVE

SUMMARY

ABBREVIATIONS ES-101 . Purpose and Format of Operator Licensing Examination Standards ES-102 - Regulations and Publications Applicable to Operator Licensing ES-201 - Initial Operator Licensing Examination Process ES-202 Preparing and Reviewing Operator License Applications ES-203 : [ Deleted] ES-204 Processing WahTrs Requested by Reactor Operator and Senior Reactor Operator Applicants ES-205 Procedure for Administering the Generic Fundamentals Examination Program v ES-301 Preparing Initial Operating Tests ES-302 Administering Operating Tests to initial License Applicants ES-303 Documenting and Grading initial Operating Tests ES-401 Preparing Initial Site-Specific Written Examinations ES-402 Administering Initial Written Examinations ES-403 Grading initial Site-Specific Written Examinations ES-501 initial Post-Examination Activities ES-502 Processing Requests for Administrative Reviews and Hearings After Initial License Denial ES-601 Conducting NRC Requalification Examinations ES-602 Requalification Written Examinations v NUREG-1021, Revision 8

~ 1 TABLE OF CONTENTS ES-603 Requalification Walk-Through Examinations ES-604 Dynamic Simulator Requalification Examinations

 ' ES-605             License Maintenance, License Renewal Applications, and Requests for Administrative Reviews and Hearings                                          ,

l ES-701 Administration of Initial Examinations for Senior Operators Limited to Fuel l Handling ES-702 Administration of Requalification Examinations for Senior Reactor Operators Limited to Fuel Handling APPENDIX A Overview of Generic Examination Concepts APPENDIX B Written Examination Guidelines APPENDIX C Job Performance Measure Guidelines APPENDIX D Simulator Testing Guidelines APPENDIX E Policies and Guidelines for Taking NRC Examinations APPENDIX F Glossary NUREG-1021, Revision 8 vi

i J (9

 /
    /

EXECUTIVE

SUMMARY

I Title 10, Part 55, of the Code of Federa/ Regu'etions (10 CFR Part 55) requires app!icants for reactor operator (RO) and senior reactor operator (SRO) licenses to pass a written examination and an operating test that are developed and administered in accordance with 10 CFR 55.41 and 55.45 or 55.43 and 55.45, respectively. Although license examiners from the U.S. Nuclear j Regulatory Commission (NRC) have historically prepared all of the licensing examinations ' using facility-provided reference materials, the NRC has now amended Part 55 by adding a new section ( 55.40) that allows facihty licensees to develop and submit, upon approval by an authorized representative of the facility licensee, proposed examinations for NRC review and approval. The NRC will prepare the examinations if requested in writing by a facility licensee and may elect to prepare the examinations, in lieu of allowing a specific facility licensee to do so, as necessary to maintain the proficiency of its examiners or the quality of the examinations. Facility licensees that elect to prepare their own examinations shall develop and submit their proposed examinations based on the guidelines and instructions contained herein. Section 107 i of the Atomic Energy Act of 1954, as amended, requires the Commission to prescribe uniform l licensing conditions for operators. Therefore, the NRC discourages facility licensees from using testing methodologies that do not conform to the policies, procedures, and practices defined in this NUREG. Nevertheless, facility licensees may propose alternatives to specific guidance in NUREG-1021, and the NRC will review and rule on the acceptability of the alternatives. The NRC will make a reasonable attempt to administer all license examinations on the dates requested by the facility licensees. At times, however, resource limitations may compel the staff to prioritize its examination review and development activities based on need and safety O) ( considerations. Facility licensees are strongly encouraged to schedule their initial license examinations and to resolve any applicant eligibility questions with their NRC regional office before commencing a license training class. For Revision 8, NUREG-1021 was reorganized to more clearly identify the various organizational responsibilities. It incorporates the methodology and lessons learned from the 1 pilot examination program described in GL 95-06 and changes made in response to the public comments on interim Revision 8 solicited in connection with the regulatory amendment noted above. This revision also formally implements Revision 2 of NUREGs-1122 and 1123, the

      " Knowledge and Abilities Catalogs" for pressurized and boiling water reactors, respectively.

In addition, Revision 8 of NUREG-1021 supersedes Revision 5 of NUREG/BR-0122,

      " Examiners' Handbook for Developing Operator Licensing Written Examinations," dated March 1990.

The following list summarizes the significant changes from Revision 7. 1 m NUREG-1021, Revision 8

   )                                                    vii t

1

EXECUTIVE

SUMMARY

Abbreviations This list has been added to provide a central location for (lefining the acronyms and abbreviations used throughout this NUREG. ES-101 No significant changes. ES-102 NUREG-1560, " Individual Plant Examination Program: Perspecth es on Reactor Safety and Plant Performance," and NUREG-1600," General Statement of Policy and Procedure for NRC Enforcement Actions, have been added to the list of documents applicable to the operator licensing program. ES-201 Each NRC regional office will prepare at least one examination p er year. Facility licensees that elect to have the NRC prepare their examinations shall submit a written request per 10 CFR 55.40(c), and the NRC will schedule the examination consistent with staff availabihty. Facility licensees that write their own examinations will conduct the following activities based on the guidance in NUREG-1021: Observe various examination security and integrity ciiteria, including limits on the activities of personnel having knowledge of the examination contents, physical security expectations and considerations, and limits on the use of examination banks. Approximately 75 days before the scheduled examir ation date, prepare and submit for NRC review and comment an integrated examination ; outline, in accordance with ES-301, ES-401, and the associated quality ' checklist. 3 Approximately 45 days before the scheduled examination date, prepare ) and submit for NRC review and comment the complete examination, in accordance witn ES-301 and ES-401, along with a s.tatoment indicating the source of each test item proposed for use on the examination. Make examination changes as agreed upon with tho NRC. Facility licensees shall designate a point of contact to work with the NRC chief examiner. Pursuant to 10 CFR 55.40(b)(3), an authorized representative of the facility licensee shall approve the submittals before sending them to the NRC for review and comment. The amount of reference material requested from the facility 1:censee will be adjusted based on the NRC's level of involvement in the exaniination development ; process. : The NRC regional offices may separate the written examinations and operating tests by up to 30 days without NRR program office approval. ; ES-202 The eligibility criteria for senior reactor operators limited to fuel handling (LSROs) l have t,een moved from ES-701 to ES-202. 1 NUREG-1021, Revision 8 viii 1 _____._d

(] (/ EXECUTIVE

SUMMARY

j ES-202 To make the standard conform with 10 CFR 55 and current practice, ES-202 now l includes a provision requiring that facility licensees submit a written request to have j a license examination administered to an applicant. l If more than six months pass since an applicant's medical examination, the facility shall certify that the applicant has not developed any condition reportable under 10 CFR 55.25. The regions will verify that an applicant's name does not appear on the " Restricted Individuals List" before accepting the application. The requirement for five significant reactivity manipulations has been clarified. ES-204 The provision for LSROs to be licensed at more than one site has been moved from ES-701 to ES-204. The regions may, under certain circumstances, waive the requirement for an examination for applicants that were previously licensed at the same facility. The regions may, under the long-term shutdown conditions specified in 10 CFR 55.31(a)(5), accept an application and administer an examination before the applicant completes the required control manipulations. t i

 ;   /        The regions may waive the requirement for a new medical examination for up to two years from the dato of the last examination if the facility licensee certifies that the applicant has not developed any medical condition reportable under 10 CFR 55.25.

ES-205 Facility licensees f hould notify the NRR operator licensing program office if they must modify their previously submitted registration letter for the generic fundamentals examination by adding or deleting a person. ES-301 Dominant accident sequences, as determined by the facility licensee's probabilistic risk assessment or individual plant examination, should be considered for sampling during the operating test. A site-specific task list may be used to supplement or override, on a case-by-case basis, selected individual items in the NRC's knowledge and abilities catalog. The instructions for developing the operating test outline and the final test items have been separated to facilitate their sequential preparation, review, and approval. Generic guidelines (i.e., those that apply to both initial and requalification examinations) for developing the walk-through and dynamic simulator tests have been relocated to Appendix C," Job Performance Measure Guidelines," and Appendix D, " Simulator Testing Guidelines," respectively. (h NUREG-1021, Revision 8 (v) ix

EXECUTIVE

SUMMARY

ES-301 The job performance measures (JPMs) used in Category B, " Control Room Systems and Facility Walk-through," of the operating test will no longer include prescripted follow-up questions. Examiners may continue to ask follow-up questions for cause based on the applicant's performance of each JPM. The number of altemate path JPMs has been increased to 40 percent of the category. No more than 80 percent of any applicant's walk-through test may be taken directly from the facility licensee's item bank without significant modification, and no more than 30 percent of the walk-through may be repeated from the last NRC license examination at the facility. A quality checklist has been included as an attachment to this standard to highlight various criteria and promote consistency. Each applicant's dynamic simulator test shall include at least one new or significantly modified scenario. The required reactivity manipulation may be conducted under normal or controlled abnormal conditions. A quality checklist has been included as an attachment to this standard to promote consistency by highlighting and suggesting target ranges for various criteria, including simulator critical tasks. The target ranges are based on a study of simulator scenarios used during past initial operator licensing examinations. No dynamic simulator scenarios or JPMs will be repeated on successive days. ES-302 NRC examiners may use additional surrogate operators to augment the simulator crews if the technical specifications require the facility licensee to operate with more than two ROs in the control room. A shift technical advisor (STA) may also be used consistent with facility operating practice. SRO-upgrade applicants, while in an RO position during the simulator test, do not have to be monitored individually by an NRC examiner. , The facility licensee and NRC chief examiner should confirm that the simulator ) instructor's station, programmers' tools, and extemal interconnections do not compromise the integrity of the operating test. Appendix D briefly describes a number of vulnerabilities. The practice of conducting an exit briefing with the facility licensee after the operating tests are complete has been adopted as policy. The operating test briefing for the applicants has been moved to Appendix E,

           " Policies and Guidelines for Taking NRC Examinations."

ES-303 The simulator operating test grading guidelines for errors having serious safety consequences (including critical tasks) have been clarified. Missing a critical task does not necessarily mean that an applicant will fail the simulator test, nor does success on every critical task prevent the examiner from recommending a failure if the applicant had other deficiencies that, in the aggregate, justify the failure based on the competency evaluations. 1 NUREG-1021, Revision 8 x

 ~.
   /  ~~

EXECUTIVE

SUMMARY

i i x ES-303 The operating test documentation requirements have been clarified: examiners must now briefly describe the error that the applicant made to justify a grade of "2" on any rating factor, but unsatisfactory grades on specific systems or rating factors that do not result in a test failure do not necessarily require detailed documentation. , The applicants' responses to JPM follow-up questions will be evaluated based l primarily on safety-significance. ES-401 This standard now includes instructions, an example method, and forms for use in { systematically developing the written examination outline. References to l NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations," have been deleted. References for guidance in deve!oping multiple choice test items have been changed from NUREG/BR-0122 to Appendix B, " Written Examination Guidelines." Facility licensees shall submit an outline approximately 75 days before the j examination date, followed by the " ready-to-use" examination approximately 45 ) days before the examination date. A site-specific task list may be used to supplement or override, on a case-by-case basis, selected individual items in the NRC's knowledge and abilities catalog; a site-specific task list may not be used in place of the entire catalog. l This standard now includes several criteria to ensure the integrity of examinations developed by facility licensees. These criteria include limits on the number of questions that can be taken directly from the facility licensee's item bank or can be i repeated from earlier quizzes and examinations. ; In an effort to maintain examination quality and consistency,50 to 60 percent of the ) questions on the examination (including 10 new questions) shall be written at the ; comprehension / analysis level. ; The NRC regional office will conduct a 30-question acceptance review of the written examination upon receipt. The region is expected to communicate the results of the review to the facility licensee if six or more questions are found to be unacceptable. As a final check for technical accuracy, facility licensees should consider administering the examination to one or more previously uninvolved licensed personnel (under security aare(ments). ES-402 The time permitted to tc 5e written exam'ination has been increased from four to five hours. If necessary on a case-by-case basis, the NRC regional office may authorize additional extensions in 30-minute intervals. ( ( xi NUREG 1021, Revision 8

r EXECUTIVE

SUMMARY

ES-402 Facility licensees that prepare their own examinations will generally administer the written examinations after they are approved by the NRC. The NRC may delay an examination if necessary to achieve a quality product. The facility licensees will document for subsequent review by the NRC any auestions posed by rid answers provided to the license applicants during the examination. If NRC examiners are on site, they may periodically monitor the administration process. The guidelines for briefing the applicants who will take the examination have been moved to Appendix E. 4 ES-403 Facility licensees should collect and consider any questions and comments made by the applicants after the examinations are administered. Facility licensees that prepare and administer the written examinations will grade the examinations, review the grading, evaluate the applicants' performance, and submit the results to the NRC for review and approval. The facility licensee shall justify all recommended question deletions and changes to the answer key. The discussion of examination grading quality reviews has been moved to ES-501. ES-501 This standard summarizes the documentation that facility licensees are expected to provide to the NRC if they develop and administer (in the case of the written) the license examinations. In addition, the standard now summarizes the post-examination quality review process that was previously contained in ES-403. If a facility licensee recommends deleting or changing five percent or more of the questions on a written examination that it developed, it may be asked to explain why the changes were necessary. The NRC regional offices may delay issuing the licenses for applicants who pass the written examination with insufficient margin to ensure that the licensing decision will be sustained if additional questions are deleted or changed upon appeal. , Applicants will be notified in writing if their licensing action is delayed. The examination report shall address any significant issues that the region or facility licensee encountered in developing the examination. The record keeping requirements, including the submittal of proposed examinations to the public document room (PDR), have been revised to reflect the new examination process and document management system. ES-502 The NRR operator licensing program office will determine whether to refer an appeal to the affected regional office, evaluate the appeal internally, or convene a panel. Appeal panels, when required, will normally consist of a branch chief and two examiners or subject matter experts. Facility licensees may be requested to provide reference materials and technical information as necessary for the NRC to evaluate informal appeals. I NUREG-1021, Revision 8 xn l l l l t . _ _ - - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _

m l l

   \                                    EXECUTIVE

SUMMARY

ES-502 The detailed administrative review procedures and sample letters have been removed and incorporated in a separate intemal document. ES-601 The NRC will continue to monitor licensees for indications of undue stress during requalification examinations, however the stress feedback forms have been eliminated. The amount of test item duplication from recent examinations, which could affect examination validity and integrity, will be considered when evaluating the effectiveness of the requalification program. ES-602 References to NUREG/BR-0122 have been changed because the guidance for developing multiple choice test questions is now in Appendix B. Because the guidelines previously documented in Attachment 1 to ES-602,

             " Policies and Guidelines for Taking NRC Written Examinations," are generally the same for initial and requalification examinations, they have been consolidated in Appendix E.

ES-603 ~ Attachment 1 to ES-603, " Guidelines for the Development and Use of Alternate Path'JPMs," Attachment 3, " Walk-Through Evaluation Guidelines," Form ES-603-1, "JPM Quality Checklist," and Form ES-603-2, "JPM Worksheet," have been moved to Appendix C because they apply to both initial and requalification examinations. g Attachment 2 to ES-603, " Briefing Checklist - System Walk-Through," has been moved to Appendix E. ES-604 Attachment 1 to ES-604, " Critical Task Methodology," and Attachment 3,

             " Quantitative and Qualitative Scenario Attributes," have been moved to Appendix D because they apply to both initial and requalification examinations. Attachment 2,
             " Dynamic Simulator Briefing Checklist," has been moved to Appendix E.

ES-605 The policy on standir.g proficiency watches and renewing inactive licenses has been clarified. The NRC regional offices may, under certain circumstances, authorize an operator to temporarily suspend participation in the facility licensee's requalification training program. ES-701 The eligibility criteria for LSROs have been moved to ES-202, and the provision for LSROs to be licensed at more than one site is now discussed in ES-204. The standard has also been edited to clarify the differences between the full-scope SRO and the LSRO examinations. The number of systems tested in Category B of the operating test has been decreased from six to five, and the requirement to test a normal evolution during each of the two discussion scenarios in Category C has been eliminated. xiii NUREG-1021, Revision 8

e EXECUTIVE

SUMMARY

ES-701 The number of subject areas to be evaluated with questions when it is not practical to conduct or simulate a job performance measure has been decreased. ES-702 This standard has been edited to clarify the differences between the full-scope and the LSRO requalification examinations. Whenever possible, the facility licensee should include an LSRO on the requalification examination team. Appendix A This new appendix discusses the generic examination concepts that play a role in the operator licensing process. It includes much of the information that was previously contained in NUREG/BR-0122 as well as discussions of new topics that have a bearing on the level of difficulty of an examination. Appendix B This new appendix incorporates the guidance for developing written test questions that was previously contained in NUREG/BR-0122. It focuses primarily on multiple-choice questions, the only type currently permitted on the initial operator licensing examination, and includes examples to illustrate various psychometric concepts. Appendix C This new appendix summarizes the guidelines concerning job performance measures that apply to both initial and requalification examinations. Much of this information was previously contained in Attachments to ES-603. There are no significant policy changes. Appendix D This new appendix summarizes the dynamic simulator scenario guidelines that apply to both the initial and requalification examination programs. Much of the information was previously contained in ES-301 and Attachments to ES-604. Appendix D also describec a number of simulator security vulnerabilities (related to features of the instructor's station, programmers' tools, and external interconnections) that NRC examiners and facility personnel should consder when preparing and administering operating tests. Appendix E This new appendix summarizes all of the policies and guidelines applicable to examinees who will be taking an initial or requalification examination. The information was previously contained in ES-302, ES-402, ES-602, ES-603, and ES-604. The policy on examining senior reactor operator upgrade applicants on the control boards has been added to the simulator test briefing list. Appendix F This new appendix provides a central location for defining terms used throughout ; this NUREG. I Any reference to the plant's technical specifications includes the plant's other technical requirements documents. NUREG-1021, Revision 8 xiv l l

p ABBREVIATIONS

 .Q       AC         attemating current
         ' ADS       automatic depressurization system AFW-       auxiliary feedwater ANSI /ANS  American National Standards Institute /American Nuclear Society AO         auxiliary operator AOP        abnormal operating procedure APRM      . average power range monitor
         'ARP        alarm (or annunciator) response procedure ATWS(T]    anticipated transient without scram [ trip]

B&W Babcock and Wilcox BWR boiling water reactor CAL confirmatory action letter CCW component cooling water CFR Code of FederalRegulations CRD control rod drive CRT criterion-referenced test CT critical task CTMT containment CVCS chemical and volume control system DAS dominant accident sequence

         .DC          direct current DHR        decay heat removal DIPM       Division of Inspection Program Management
     ] , EAL          emergency action level ECA-       emergency contingency action (procedure)

ECCS emergency core cooling system ECP estimated critical pocition EDG emergency diesel generator i EHC electrohydraulic control EOP emergency operating procedure EPIP emergency plan implementing procedure EQB examination question bank i i ES examination standard ESF engineered safety feature FHE fuel handling equipment FRP functional recovery procedure FSAR final safety analysis report GFE generic fundamentals examination GL generic letter GUI graphic user interface HP health physics HPCI high pressure coolant injection HPCS' high pressure core spray HVAC heating, ventilation, and air conditioning IC instrumentation and control INPO Institute of Nuclear Power Operations O) l

  %J xv                 NUREG-1021, Revision 8

ABBREVIATIONS 1 IP inspection procedure l IPE individual plant examination l lRM intermediate range monitor ' JPM Job performance measure

JTA job task analysis K/A knowledge and ability KSA knowledge, skill, and ability LCO limiting condition for operation LER licensee event report LOCA loss of coolant accident LPCI low pressure coolant injection LPCS low pressure core spray LPRM local power range monitor LSRO limited senior reactor operator MiP master inspection plan MSIV main steam isolation valve NEl Nuclear Energy Instituto NRC Nuclear Regulator Commission NOP normal operating procedure NRR Office of Nuclear Reactor Regulation NRT norm-referenced test )

NWPA Nuclear Waste Policy Act(of 1982) OJT on-the-job training OLA operator licensing assistant l OLTS operator licensing tracking system l OMB Office of Management and Budget i PCIS primary containment isolation system PDR public document room PORV power-operated relief valve i PPR plant performance review PRA probabilistic risk assessment PWR pressurized water reactor QA quality assurance RBM rod block monitor RCA radiologically control!ed area RCIC reactor core isolation cooling RG Regulatory Guide RHR residual heat removal RMCS reactor mar.ual control system RO reactor operator ROI report on interaction RM radiation monitor RPIS rod position indication system RPS reactor protection system RPV reactor pressure vessel RWST refueling water storage tank

         , OVMon 8                       Wi

'[] ABBREVIATIONS ' _ LJ S satisfactory SALP. systematic assessment of licenseo performance SAT systems approach to training. SGTS standby gas treatment system SD standard deviation SGTR steam generator tube rupture Si safety injection

   -SLC        standby liquid control SME        subject matter expert SRO        senior reactor operator SRP        Standard Review Plan
   .SRV        safety relief valve STA        shift technical advisor TDAFW(P) - turbine-driven AFW (pump)

TS technical specification (or other technical requirements document) U unsatisfactory UPS uninterruptible power supply W/T walk-through ; I O l l l xvii NUREG-1021, Revision 8

b) (. ES-101 PURPOSE AND FORMAT OF OPERATOR LICENSING EXAMINATION STANDARDS A. PURPOSE i Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55) requires that applicants 1 for reactor operator (RO) and senior reactor operator (SRO) licenses pass written examinations j and operating tests (both initially and for requalification). Moreover, the regulations mandate j that the license examinations must be developed and administered in accordance with 10 CFR ! 55.41 and 55.45 for ROs, or 10 CFR 55.43 and 55.45 for SROs. ! l The " Operator Licensing Examination Standards for Power Reactors" (NUREG 1021) establish the procedures and practices for administering the required initial and requalification written examinations and operating tests. These standards describe the provisions of the act and j regulations on which the program is based. They also ensure the equitable and consistent ! administration of examinations to all applicants and licensed operators at all facilities subject to l the regulations. 1 B. FORMAT l Each standard explains the rules, procedures, and practices for a particular aspect of the l program. For ease of reference, each examination standard (ES) is assigned a three-digit number, and related standards are grouped together in the sense that standards beginning with the same digit apply to related aspects of the program, as follows: V ES 1xx - General ES-2xx - Initial pre-examination activities

ES-3xx - Initial operating tests ES-4xx - Initial written examinations ES-5xx - Initial post-examination activities l ES-6xx - Requalification examinations j ES-7xx - Fuel handling examinations l l l i

   "/                                                   1 of 1                     NUREG-1021, Revision 8 4

I

/                                                   ES-102 V                          REGULATIONS AND PUBLICATIONS APPLICABLE TO OPERATOR LICENSING A.      PURPOSE This standard lists the U.S. statutes and the regulations of the U.S. Nuclear Regulatory Commission (NRC) that establish the requirements for conducting operator licensing                   ,

examinations. It also identifies the regulatory guides and NUREG reports that establish the I procedures for implementing the regulations and administering the examinations, as well as standards of the American National Standards Institute /American Nuclear Society (ANSI /ANS) that may provide additional guidance. Regulatory guides (RGs), NUREG reports, and industry stendards are not requirements, except as specified in Commission orders or as committed to by the facility licensee. The appropriate revisions should be consulted as referenced in the facility's final safety analysis report (FSAR) , or approved training program. The following paragraphs summarize the latest revisions of I these documents. B. STATUTES

1. Atomic Energy Act of 1954 Section 107 of the Atomic Energy Act of 1954 (42 U.S.C. 2137), as amended, requires l

/} ( / that the NRC prescribe uniform conditions for licensing individuals as operators of production and utilization facilities, determining the qualifications of these individuals, l i and issuing licenses to such individuals. I

2. Nuclear Waste Policy Act of 1982 Section 306 of the Nuclear Waste Policy Act of 1982(42 U.S.C.10226,96 Stat. 2201 at j 2262 - 2263) directs the NRC to establish requirements governing (1) simulator training !

for applicants for operator licenses and for operator requalification training programs, (2) NRC administration of requalification examinations, and (3) operating tests at civilian i nuclear power plant simulators. C. REGULATIONS i l

1. 10 CFR Part 2. Rules of Practice The regulations in 10 CFR Part 2 govern the conduct of all proceedings under the Atomic Energy Act of 1954, as amended, and the Energy Reorganization Act of 1974 with regard to (a) granting, suspending, revoking, amending, or taking other action with respect to any license; (b) imposing civil penalties; and (c) public rulemaking.

10 CFR 2.103(b)(2) establishes the applicant's right to demand a review of a proposed license denial, and defines the applicant's appeal and hearing rights. (D

V 1 of 6 NUREG-1021, Revision 8

i i ES-102 l Subpart G, " Rules of General Applicability," govems all adjudications initiated by the issuance of an order to show cause, an order designating the time and place of a hearing requested by a person charged with a violation, and a notice of hearing. Subpart L, " Informal Hearing Procedures for Adjudications in Materials and Operator Licensing P:omedings," governs proceedings for the issuance, renewal, or licensee-initiated amendment of an operator or senior operator license.

2. 10 CFR PM 9. Public Records The regu!ations in ~10 CFR Part 9 prescribe the rules governing the NRC's public records that relate to any proceeding subject to 10 CFR Part 2.

I Subparts A and B describe and implement the regttirements for balancing the public's l rights to inforrration under the Freedom ofInformation Act and the NRC's responsibility to protect personal information under the Privacy Act. Subparts C and D implement the provisions of the Sunshine Act, conceming the opening of Commission meetings to public observation. They also describe the procedures governing the production of agency records, information, or testimony in response to subpoenas or demands of courts or other judicial authorities in State and Federal proceedings.

3. 10 CFR Part 20. Standards for Protection Aaainst Radiation l

The regulations in 10 CFR Part 20 establish standards for protection against radiation ] hazards arising from licensed activities. Some of the materialis appropriate for I l inclusion in the examinations administered to candidates for RO or SRO licenses.

4. 10 CFR Part 50 Licensina of Production and Utilization Facilitie.g 10 CFR 50.34(b)(8) requires that the FSAR include a description of the operator requalification program. That description forms the basis for the inspection, audit, and approval of requalification programs.

10 CFR 50.54(I-1) requires facility licensees to implement an operator requalification program that meets the requirements of 10 CFR 55.59(c) within 3 months after receiving a facility operating license. Notwithstanding the provisions of 10 CFR 50.59, the licensee may not decrease the scope of its approved requalification program without authorization from the Commission. 10 CFR 50.54(k) - (m) contain regulations restricting control manipulations to licensed operators. These regulations are conditions of all facility licenses issued under 10 CFR Part 50. 10 CFR 50.74 requires facility licensees to notify the Commission within 30 dns if there is a change in the status of a licensed RO or SRO. NUREG-1021, Revision 8 2 of 6 l

7T ES-102

5. 10 CFR Part 55. Ooerators' Licenses 10 CFR Part 55 is the implementing regulation that establishes the requirements and the regulatory basis for licensing and requalifying ROs and SROs.

D. REGULATORY GUIDES ',

1. Reculatorv Guide 1.8. " Qualification and Trainina of Personnel for Nuclear Poweg fjants." Revision 2. Aoril 1987 Section C.1 of this RG currently endorses, with exception, ANSI /ANS 3.1-1981, "American National Standard for Selection, Qualification, and Training of Personnel for Nuclear Power Plants" (effective March 31,1988). The NRC is currently reviewing, and is expected to endorse, with exception, the 1993 version of ANSI /ANS 3.1.

2. Reaulatorv Guide 1.33. "Quali+v Assurance Proaram Reauirements -Operations" Appendix A to this RG contains a list of typical procedures for pressurized water reactors anc? boiling water reactors.

3. Reaulatorv Guide 1.114. " Guidance on Beina an Operator._gt the Controls of a Nuclear Power Plant"

   /    )
   'y/       This RG describes a method acceptable to the NRC staff for complying with the Commission's regulations in 10 CFR 50.54(k) - (m), which require the presence of an RO at the controls of a nuclear power unit and an SRO in the control room from which the nuclear power unit is being operated.

4. BeaulatonLGuide 1.134. " Medical Egluatip._n of Licensed Personnel for Nuclesr Power Plants." Revision 3. March 1928 This RG currently endorses ANSI /ANS 3.4-1996, " Medical Certification and Monitoring of Personne! Requiring Operator Licenses for Nuclear Power Plants," with exceptions.

However, facility licensees may continue to use the 1983 version of ANSI /ANS 3.4, which was previously endorsed in its entirety by Revision 2 of RG 1.134, dated April 1987,

5. Reaulatory Guide _L.149 "Nuclearfower Plant Simulation Facilities for Use in Operator License Examinations."fevisionj!. April 1996 This RG currently endorses, with exception, ANSI /ANS 3.5-1993, " Nuclear Power Plant Simulators for Use in Operator Training and Examination." It is expected that Revision 3 will endorse ANS!/ANS 3.5-1998. However, facility licensees may continue to use the e 1985 version of ANSI /ANS 3.5, which was previously endorsed, with exceptions, by Revision 1 of the RG dated April 1987.

i 3 of 6 NUREG-1021, Revision 8

ES-102 E. NUREG REPORTS

1. NUREG-0660. Vol.1. "NRC Action Plan Devetooed as a Result of the TMI-2 Accident."

May1980 item I.A.4.2 of this document describes the guidelines for long-term simulator upgrades.

2. NUREG-0737. " Clarification of TMI Action Plan Reauirements." November 1980 !

l This document clarifies the following action plan items which are intended to upgrade the training, licensing, education, and experience of operators on the basis of experience gained from the accident at Three Mile Island, Unit 2: Item I.A.2.1, "Immediate Upgrading of RO and SRO Training and Qualifications" Item 1.A.2.3, " Administration of Training Programs" Item 1.A.3.1, " Revised Scope and Criteria for Licensing Exams" Item 11.B.4, " Training for Mitigating Core Damage"

3. NUREG-0800. " Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants. LWR Edition." July 1981 Section 13.2, " Reactor Operator Training," describes the training and licensing of operators and identifies information to be submitted by applicants for construction permits and operating licenses.

4. NUREG-1122. "Knowledae and Abilitics Cataloa for Nuclear Power Plant Operators:

Eressurized Water Reactors." Revision 2 This document provides the basis for developing content-valid licensing examinations for operators at pressurized water reactors (PWRs). It contains knowledge and ability (K/A) statements that have been rated for their importance to ensuring that the plant is operated in a manner consistent with the health and safety of plant personnel and the public.

5. NUREG-1123. "Knowledae and Abilities Cataloa for Nuclear Power Plant Operators:

e Reactors." Revision 2 Boilina Wate_r This document provides the basis for developing content-valid licensing examinations for operators at boiling water reactors (BWRs). It contains K/A statements that have been rated for their importance to ensuring that the plant is operated in a manner consistent with the health and safety of plant personnel and the public. NUREG-1021, Revision 8 4 of 6 4 w

l p) ES-102 V

6. NUREG-1291. "BWR and PWR Off-Normal Event Descriptions." November 1987 The reactor event descriptions in this document provide a reliable, performance-based source of information that examiners may use to design simulator scenarios that will be a valid test of an applicant's ability to safely and competently perform alllicensed duties and responsibilities.

7. NUREG-1560. " Individual Plant Examination Proaram: Persoectives on Reactor Safety and Plant Performance" l

This report provides perspectives gained by reviewing 75 individual plant examination (IPE) submittals pertaining to 108 nuclear power plant units. Chapter 13, " Operational Perspectives," is of particular interest because it identifies a number of important human actions that should be considered for evaluation on BWR and PWR licensing and requalification examinations.

8. NUREG-1600. " General Statement of Policy and Procedure for NRC Enforcement Actions" This report addresses the NRC's expectations regarding compliance with 10 CFR 55.49,

               " Integrity of Examinations and Tests," and possible enforcement actions against parties subject to that regulation (i.e., Part 55 license holders and applicants and Part 50 O          licensees).

L./I 9. NUREG/BR-0122. " Examiners' Handbook for Devetooina Ooerator Licensina Written Examinations." Revision 5. March 1990 This docuinent, which presented a procedure for systematically constructing content-valid licensing examinations for nuclear power plant operators, has been incorporated into the examination standards in NUREG-1021, Revision 8. It may be used for historical perspective, but is no longer used for developing examinations. F. INDUSTRY STANDARDS

1. ANSI /ANS 3.1. "American National Standard for Selection. Qualification and Trainina of Personnel for Nuclear Power Plants" This standard provides criteria for selecting and training nuclear power plant employees performing a variety of functions at various levels of responsibility (e.g., managers, supervisors, operators, and technicians). RG 1.8, Revision 2, endorsed, with exceptions, the 1981 version of the standard; the 1987 version was never endorsed by the NRC; the 1993 version is currently under review by the NRC.

l f'% ( "' / 5 of 6 NUREG-1021, Revision 8 l

ES-102

2. ANS 3.2 (ANSI N18.7-1976). " Administrative Controls and QA for the Operational Phase of Nuclear Power Plants" This standard provides guidance ano recommendations for administrative rules of practice and related subjects and for preparing procedures and audit programs. See RG 1.33.

3. ANSI /ANS 3.4-1996. " Medical Certification and Monitorina of Personnel Reauirina Ooerator Licenses for Nuclear Power Plants" This standard is the basic document covering the general health and disqualifying conditions applicable to license applicants and licensed personnel. Revision 3 of RG 1.134 currently endorses this standard with exceptions, but facility licensees may continue to use the 1983 version, which was previously endorsed in its entirety by Revision 2 of the RG.

4. ANSI /ANS 3.5-1993. " Nuclear Power Plant Simulators for Use in Operator Trainina' This standard establishes the minimum functional requirements and capabilities for nuclear power plant simulators for use in operator training. Revision 2 of RG 1.149 endorses this standard, with exceptions, and it is expected that Revision 3 of RG 1.149 will endorse the 1998 version of this standard. However, facility licensees may continue to use the 1985 version, which was previously endorsed, with exceptions, by Revision 1 of the RG.

NUREG-1021, Revision 8 6 of 6

ES-201 V INITIAL OPERATOR LICENSING EXAMINATION PROCESS

 . A.       . PURPOSE l

This' standard describes the activities that must be completed to prepare for initial operator licensing examinations (including written examinations and operating tests) at power reactor

 ' facilities. It includes instructions for scheduling and coordinating examination development, assigning NRC examiners and facility personnel, maintaining examination security, and obtaining reference and examination materials from the facility licensee.-

B. BACKGROUND Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55) requires that applicants for reactor operator (RO) and senior reactor operator (SRO) licenses pass a written examination and an operating test. The regulation allows power reactor facility licensees to prepare the site-speci'ic written examinations and operating tests subject to the following j conditions: (1) the facility licensee shall prepare the examinations and tests in accordance with the criteria contained herein; (2) the facility licensee shall establish, implement, and maintain 1 procedures to control examination security and integrity; (3) an authorized representative of the facility licensee shall approve the examinations and tests before they are submitted to the NRC for review and approval; and (4) the facility licensee shall obtain NRC approval of its proposed written examinations and operating tests. Moreover, the regulation requires that the license ] examinations be developed and administered in accordance with 10 CFR 55.41 and 55.45 for ROs or 10 CFR 55.43 and 55.45 for SROs.

  ' Facility licensees may propose attematives from the examination criteria contained herein and evaluate how the proposed alternatives provide an acceptable method of complying with the Commission's regulations. The NRC staff.will review any proposed attematives and make a decision regarding their acceptability. The NRC will not approve any alternative that would compromise its statutory responsibility of prescribing uniform conditions for the operator licensing examinations.

The NRC will continue to prepare the examinations upon written request by facility licensees (consistent with NRC staff availability) and retains the authority to develop the examinations on a case-by-case basis if it loses confidence that a facility licensee will develop examinations I upon which the NRC can base its licensing decisions. If the NRC determines that a facility is I 1 unable to develop acceptable examinations, the examinations could be delayed until sufficient NRC resources can be scheduled to develop and conduct the examinations, or until the facility licensee can develop an acceptable examination. Each NRC regional office will also prepare at least one examination per calendar year to certify new examiners, as required, and to maintain examiner proficiency. The NRC will make a reasonable attempt to administer all license examinations on the dates requested by facility licensees. At times, however, resource limitations may compel the staff to ! prioritize its examination review and development activities based on need and safety ! considerations. Examinations for fewer than three applicants should be scheduled only under extenuating circumstances such as a shortage of licensed ROs or SROs at the facility. If a 1 of 24 NUREG-1021, Revision 8

ES-201 facility licensee has fewer than three license applicants, the examinations may be delayed until more applicants are trained. Moreover, facility licensees that elect to have the NRC prepare their licensing examinations should keep in mind that it takes the NRC more time to prepare than to review an examination and that the NRC will require greater flexibility to schedule those services. Other pre-examination activities, such as submitting and reviewing license applications and eligibility waivers and administering the generic fundamentals examination program, are addressed in ES-202, ES-204, and ES-205. Specific instructions for developing, adrninistering, and grading the written examinations and operating tests are found in ES-401 through ES-403 and ES-301 through ES-303, respectively. Post-examination administrative activities, including management review of the examination results and preparation of examination reports, are discussed in ES-501. Cross-references to each of these standards have been made where appropriate. C. RESPONSIBILITIES Facility licensees and NRC staff should use Form ES-201-1, " Examination Preparation Checklist," to track the examination preparations. As noted on the form, the target due dates I can be adjusted as necessary to accommodate a given situation. The NRC chief examiner will initial the items as they are completed and ensure that the original form is retained for the master examination file (refer to ES-501).

1. Facility Licensee if a facility licensee requests the NRC to prepare the iicensing examinations, then only those items identified with an asterisk (*) are applicable.

a*. The facility licensee is expected to apprise its NRC regional office of changes in its examination requirements. The facility licensee should respond in writing to the NRC's annual administrative letter soliciting estimated operator licensing needs and notify its NRC regional office if its examination requirements change significantly from those stated in its response. Facility licensees are strongly encouraged to schedule their examinations and to resolve any applicant eligibility questions with their NRC regional office be' ore commencing an initial license training class. In accordance with 10 CFR 55.40(c), facility licensees that elect to have the NRC prepare, proctor, and grade their operator licensing examinations shall submit a written request (to the responsible NRC regional office) for those examinations pursuant to 10 CFR 55.31(a)(3). b*. In accordance with 10 CFR 55.49, facility licensees and applicants shall not engage in any activity that compromises the integrity of any application, test, ur examination required by 10 CFR Part 55. Attachment 1 of this ES summarizes several examination security and integrity considerations. NUREG-1600, NUREG-1021, Revision 8 2 of 24

ES-201

       " General Statement of Policy and Procedures for NRC Enforcement Actions,"

addresses possible enforcement actions against parties subject to the requirements in the regulation (i.e., Part 55 license applicants and licensees and Part 50 licensees).

c. Pursuant to 10 CFR 55.40(b)(2), facility licensees that elect to prepare their own examinations shall establish, implement, and maintain procedures to control examination security and integrity. Attachment 1 discusses a number of examination security and integrity guidelines that may be appropriate for incorporation in those procedures.

d*. All facility and contractor personnel involved with an examination are subject to the restrictions stated in Section D of this ES. Any questions regarding those restrictions should be resolved with the NRC chief examiner before granting an individual access to the licensing examination. The facility licensee shall designate a point of contact to work with the NRC chief examiner and assign additional personnel as required to ensure that the examinations are developed, reviewed, administered, and oraded in accordance with the applicable examination standards. The facility licensee may use contractors or other outside assistance to develop the examinations, but the licensee bears full responsibility for the product, including conformance with the examination criteria and maintenance of examination security and integrity.

e*. The facility contact shall submit the required reference materials, examination outlines, and examinations, as applicable, based on the level of facility participation. Form ES-201-1 specifies target due dates for the various materials; the actual dates may be adju.sted with prior agreement from the NRC regional office,

f. The examination outlines and the examinations shall be prepared in accordance with the guidelines in ES-301, ES-401, and ES-701, as applicable. The proposed outlines and examinations shall cover all portions of the license examination (written, dynamic simulator, and walk through) at all license levels relevant to the applicants (RO, SRO, and limited SRO) to be tested.

A facility supervisor or manager shall independently review the examination outline (s) and the proposed examination (s) before they are submitted to the NRC regional office per item (g) below. In conducting this review, the facility shall use Forms ES-201-2, " Examination Outline Quality Checklist," ES-301-3, " Operating Test Quality Checklist," ES-301-4, " Simulator Scenario Quality Checklist," and ES-401-7, " Written Examination Quality Checklist."

g. Pursuant to 10 CFR 55.40(b)(3), an authorized representative of the facility licensee shall approve the proposed examination (s) before they are submitted to

'k% 3 of 24 NUREG-1021, Revision 8

i the NRC regionrJ office for review and approval. The outline (s) and l examination (s) should be forwarded to the NRC regional office with a cover letter signed by the facility representative. The materials must be complete and ready-to-use. l

h. In its examination submittal to the NRC, the facility licensee (or its contractor) )

l shall indicate the source of each test item proposed for use on the written i examination and the operating tests. The following information " hall be included for eLch item: State the source of each item (e.g., is the item taken directly, without changes, from the facility licensee's bank, another facility's bank, the NRC's bank, or an old NRC exam; is the item a modified version of a bank item; or is the item new?). Items that the facility licensee (or its contractor) has obtained from another bank and deposited in its own bank may be treated as ' bank" items provided they have a equal chance of being selected for use on the examination. Items from another bank may be treated as new items if they have not been made available for review and study by the license applicants and there is no basis (e.g., historical precedent or reciprocal arrangements with the other facility licensee) for the applicants to predict their use on the examination. For those items that are taken directly from the facility licensee's bank, state if and when the item was used on the last two NRC license examinations at the facility or to evaluate the applicants' performance during their current license training class. For those items that are derived by modifying existing bank items, note the changes that were made or submit a copy of the item from which it l originated. i*. The facility licensee shall make its simulation facility available, as necessary, for NRC examiners to prepare for and administer the operating tests. The NRC will take reasonable efforts to minimize the impact on other training activities. Before developing or administering an initial licensing examination, facility licensees are encouraged to review the simulator examination security considerations in Appendix D to NUREG-1021 for applicability to their facility. Because facility licensees are more familiar than the NRC examiners with the unique capabilities, limitations, and vulnerabilities of their simulators, it is expected that the licensees will take responsibility for determining and implementing whatever measures might be necessary to ensure the integrity of the operating tests. j*. The facility licensee shall meet with the NRC in the regional office or at the l faciPy, as necessary and appropriate, to review the examinations and discuss poutial changes. NUREG-1021, Revision 8 4 of 24 L_ __ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ES-201 If the examination was p repared by the NRC, the facility reviewers should make l their comments and recc mmendations on a copy of the written examination (s) l' and operating test (s) provided to them by the NRC examiner. Simple editorial changes that do not change the intent of the question require no justification; however, every substantive change (e.g., deleting a question, replacing a distractor, or revising an answer) must be supported by approved facility reference material. If the facility licensee has significant concems with the content or difficulty of the NRC prepared examinakn or the changes that the NRC has directed the facility licensee to make in its prc posed examination, the facility licensee is er ccuraged l to communicate thoso cor cems to the NRC and, if app;opriate, to request a ! meeting with the NRC to address the concerns. The NAC chief examiner is l normally the first poirat of cantact for resolving any concerns regarding the examination, if the concerls are not resolved at that level, the facility licensee should contact NRC regional management, and,if necessary, the chief of the NRR operator licensing pro gram office for resolution.

k. If the facility licensee develcped the examinations,'it will generally make any necessary changes as agreed upon with the NRC; however, the NRC retains final authority to approve the examinations.

l'. In accordance with ES-202, tie facility licensee chall submit the license applications along with a letter requesting that licensing examinations be administered.

2. NRC Reaional Manaaement. Suoervicion. and Desianitgg

a. The regional office shall schedale the NRC's initial operator licensing examinations and shall arrange for the develcpment, administration, and grading of those examinations as discunsed below. The regional office shall periodically review each facility licensee's e:: amination re,quirements and shall negotiate with the facility licensee's training representatives as necessary to schedule specific examination dates consistent with operational requirements and NRC resource availability. Each regional office shall plan to prepare at least one complete examination per calendar year.

b. Approximately six months before aach ant cipated examination date, the regional office should contact the facility licensee and confirm the examination date(s) and the expected number of applicants to be examined. The regional office should use that information to estimate the required number of NRC examiners and to make preliminary work assignments.

c. The regional office should contact the fa:ility licensee by telephone at least four months before the scheduled exam; nations to reconfirm the expected number of applicants and the examination dates, and to make other preliminary arrangements for developing the examinations. The person who contacts the O- 5 of 24 NUREG-1021, Revision 8

facility licensee shall discuss the following examination arrangements, as applicable, depending on the facility licensee's level of participation in the examination development process: the examination integrity and security requirements and considerations (refer to Attachment 1) tne requirement for an authorized representative of the facility licensee to approve the examination outlines and examinations before they are submitted to the NRC for review the need to have the examination outlines delivered to the NRC approximately 75 days before the scheduled examination date the need to have the reference materials necessary for the NRC to develop the examination (if applicable; refer to Attachment 2) delivered to the regional office at least 75, but preferably 90, days before the scheduled examination date the guidelines for developing, administering, and grading the written examinations, as applicable (i.e., the effective version of ES-401, ES-402, and ES-403, respectively) the need to have the simu!ator available and the guidelines fcr developing and administering the operating tests (i.e., the effective version of ES-301 and ES-302, respectively) the nead to have the examinations and the supporting reference materials (refer to Attachment 2) delivered to the NRC regional office approximately 45 days before the scheduled examination date the option to submit some sample test items (e.g.,5 to 10 written questions,1 scenario, and 1 to 2 job performance measures) for preliminary NRC review and comment (This could increase the efficiency of the examination review process by promoting early identification and correction of generic examination development concerns.) the requirements (refer to 10 CFR 55.31) and guidelines (refer to ES-202) for submitting the license applications The NRC regional office may negotiate earlier due dates with the facility contact but should refrain from advancing the dates if it is unlikely that the review will begin promptly Mer the material arrives in the regional office. The regional office should also keep the facility contact informed of the dates by which the region expects to provide its comments regarding the licensee's submittals.

d. The NRC regional office shall normally issue a letter confirming the arrangements no later than 120 days before the examination begins. The letter should be addressed to the person at the highest level of corporate management l who is responsible for plant operations (e.g., Vice President of Nuclear l Operations). Attachment 3 is an example of such a letter; the exact wording may !

be modified as necessary to reflect the situation. 3 l

e. Approximately four months before the scheduled examination, the NRC regional office will assign the required number of examiners to develop, prepare for, and ]

NUHRG-1021, Revision 8 6 of 24 ,

i

I ES-201 administer the examination as arranged with the facility licensee. The regional office will also designate a chief examiner to coordinate the examination project with the facility licensee and other examiners assigned to the examination. When making assignments, the region should consider each examiner's certification status, other examination commitments, possible conflicts of interest (as discussed in Section D of this E3), and general availability. Regional management should try to. assign a sufficient number of examiners so that no examiner will have to administer more than four operating tests per week.

f. The regional office will evaluate each examination assignment to determine if some or all of the assigned examiners should make a separate preparatory site visit. The purposes of such a visit may include providing examiner orientation, retrieving additional reference material, or reviewing and validating the examinations. When making a decisicn, the region should carefully weigh the costs and benefits associated with each additional trip to the facility. The region should also consider such factors as the experience of the assigned examiners, the quality of the facility licensee's examinations (if applicable), the number of written examinations and operating tests to be validated, and the status of the simulation facility (e.g., is it new or recently upgraded?). In addition, the region should consider the attemative of reviewing the written examination (s) and operating test (s) with the facility licensee via telephone (if the examination quality is high) or in the regional office, as well as the attemative of validating the operating test (s) emsite at the beginning of the examination week.

g. Upon receiving the preliminary license applications, approximately 30 days before the examination date, the regional office shall review the applications in accordance with ES-202. In addition, the regional office shall evaluate any waiver requests in accordance with ES-204 to dcitermine if the applicants meet the eligibility criteria specified in 10 CFR 55.31.

After reviewing and approving the preliminary license applications and resolving all waiver requests, the region will prepare an examination assignment sheet (in the format of Attachment 4) as far in advance as possible, but at least two weeks before the scheduled examination date. The region will review and revise the assignment sheet as necessary after receiving and ovaluating the final license applications. The assignment sheet will identify the chief and other examiners by name and list the applicants by name, docket number, and type of examination (e.g., SRO upgrade, RO written only) to be administered. All applicants listed on the assignment sheet should be administered complete examinations (written and operating) as indicated under " Examination Type" unless waivers have been ; granted in accordance with ES-204. A copy of the assignment sheet will be j distributed to all assigned examiners, the NRR operator licensing program office, and regional distribution. i O 7 of 24 NUREG-1021, Revision 8 l

ES-201

h. The responsible regional supervisor will review the examination outlines and the I draft examinations and evaluate any recommended changes and corrections noted during the chief (and other) examiner's review (refer to ES-301 and ES-401 for additional guidance regarding examination reviews). The supervisory review is not intended to be another detailed review, but rather a check to ensure that all applicable administrative requirements have been implemented. If the outlines, examinations, and recommended changes are acceptable, the supervisor will authorize the chief examiner to resolve any noted deficiencies with the author or facility contact.

41 any of the facility-developed examination materials (written, walk-through, or simulator) require substantive changes and cannot be made to conform with the examination standards by the end of the designated examination review week, regional management shall consult the NRR operator licensing program office and make a decision whether to proceed with the facility-developed examinations or develop the examinations in-house. If the region does not have the resources to ensure that acceptable examinations are prepared by the scheduled edministration date, regional management shall negotiate with the facility licensee to reschedule the examinations as necessary. Although it is generally easier to postpone the written examination and focus on the operating tests so that they can be administered on schedule and without affecting examinations at other facilities, regional management may delay either part of an examination for up to 30 days. The regional office shall consult the NRR program office regarding any examination delay and notify the facility licensee in writing of the reasons for the delay. The responsible supervisor will also ensure that any significant deficiencies and prcblems are addressed in the examination report in accordance with ES-501. l

i. After the chief examiner has verified that the necessary changes and corrections !

have been made, the responsible supervisor will review and approve the examinations for administration. Before signing the applicable quality checklist (i.e., Form ES-301-3 and/or Form ES-401-7), the supervisor must be satisfied that the examination is acceptable for administration.

j. If there is an indication that an examination may have been compromised, the responsible supervisor will take action as necessary to ensure and restore the integrity and security of the examination process. Actions may include not giving the examination, making additional changes to the examination, voiding the results if the examination has already been given, reevaluating the licensing decisions pursuant to 10 CFR 55.61(b), and possibly imposing enforcement action in accordance with NUREG-1600. The supervisor shall keep regional ,

management and the NRR operator licensing program office informed of any ! concerns regarding examination integrity or security. l l NUREG 1021, Revision 8 8 of 24 l l

f \ ES-201

3. Assioned NRC Examiners

a. . When assigned to administer operating tests for the first time at a particular facility, the examiner should inform the chief examiner and the responsible supervisor so that arrangements can be made to conduct an orientation trip to the facility as described in Item C.2.f, if deemed appropriate.

l

b. NRC examiners monitor and ensure the integrity of the examination process, if they perceive that a compromise has occurred, they must immediately report it to the responsible regional supervisor so that the necessary actions can be taken to restore the integrity of the examination. Attachment 1 summarizes several examination security and integrity considerations that examiners should note when reviewing the facility licensee's procedures established pursuant to 10 CFR 55.40(b)(2), as applicable.

c. The assigned examiners shall review and inventory the reference materials received from the facility licensee in response to the 120-day corporate notification letter. The purpose of this review is to determine if the materials are complete and adequate to enable the regional office to review or develop the examinations, as applicable. If it is not, the reviewer (s) shall inform the chief examiner and the responsible supervisor and request that the facility licensee send any additional materials that might be required. If necessary, an examiner may review and select additional reference materials during a site orientation trip O d.

(refer to item C.2.f). The chief examiner will work with the assigned examiners and the designated facility contact, as applicable, to ensure that the examination outlines and examinations are developed in accordance with the applicable examination standards. The chief examiner should adapt the level of oversight and coordination based upon the experience of the individuals who are preparing the examinations. Facility employees are generally less familiar with the examination standards and will require more oversight to ensure that a quality examination is ready on time,

e. The chief examiner will review the examination outlines using Form ES-201-2,

            " Examination Outline Quality Checklist," as a guide. A thorough and timely review (i.e., within 5 working days) will minimize the potential for significant problems with the examinations.

The ch.'ef examiner will note any necessary changes and forward the outlines to the responsible supervisor for review and comment before resolving any deficiencies with the author or facility contt.ct. If the outlines are significantly deficient, refer to item C.2.h for additional guidance.

f. The chief examiner will review the written examinations and operating tests for <

quality in accordance with the applicable checklists (refer to ES-301 and ES-401) forwarded with the examination. If the chief examiner wrote the operating tests, 9 of 24 NUREG-1021, Revision 8

ES-201 another NRC examiner shall perform the independent review. The regional office may conduct additional reviews at its ciscretion if resources permit. It is especially important that facility-developed examinations and tests be reviewed promptly because of the extra time that may be required if extensive changes are necessary. The written examination sampling review (as described in Section E of ES-401) should be completed within one week after receiving the j examination, and the balance of quality reviews should be completed within two weeks after the examinations and tests are received from the author or facility contact. The chief examiner will note any necessary changes and forward the examinations and tests to the responsible supervisor for review and comment before reviewing the examinations with the author or facility contact. There are no minimum or maximum limits on the number or scope of changes the NRC may direct the facility licensee to make to its proposed examinations, provided they are necessary to make the examinations conform with established acceptance criteria or to attain an appropriate level of examination difficulty. Chief examiners shall exercise their experience and judgement to ensure that ! the level of difficulty remains consistent with that expected on NRC-prepared ; examinations. If the examinations are significantly deficient, refer to item C.2.h { for additional guidance. The chief examiner shall document the responsible ! supervisor's authorization to proceed with for the facility review by initialing item 11 on Form ES-201-1.

g. Upon supervisory approval, generally about two weeks before the examinations are scheduled to be given, the chief examiner will review the written examinations and operating tests with the facility licensee.

The chief examiner may conduct the examination review via telephone, in the regional office, or at the facility, as appropriate to the circumstances, depending on the extent of the changes, and as approved by the responsible regional , supervisor (refer to Item C.2.f). 1 If the examination was prepared by the NRC, the regional office will provide a , copy of the written examination (s) and operating test (s) to the facility reviewers j after they sign the security agreement (Form ES-201-3). The facility reviewers shou ld make their comments directly on the examination (s), return the marked- { up copy (ies) to the NRC chief examiner, and ensure that he or she understar.ds l their comments and recommendations. The facility reviewers may retain a copy 1 of the marked-up examination (s), subject to the physical security considerations in Attachment 1. , if the facility reviewers have significant disagreements with the chict examiner, the chief examiner willinform the responsible regional supervisor so that the . disagreements can be resolved before the examinations are administered. 1 NUREG-1021, Revisie 18 10 of 24 9If l

I[} ES-201 v h. After the examination corrections have been made, the chief examiner shall verify that the changes are appropriate and route the examinations and the mark-up drafts to the responsible supervisor for final approval,

i. As soon as possible after the responsible supervisor has approved the operating

      . tests for administration, the chief examiner shall distribute copies of the scenarios, job performance measures (JPMs), and questions to the other assigned examiners so that they can familiarize themselves with those materials and be better prepared to probe the applicants' deficiencies if required.             ;
   };   The chief examiner should work with the designated facility contact to schedule the operating tests to optimize efficiency and the mix of RO and SRO applicants in the crews assembled for the simulator examinations. The numbei of applicants on a crew shall not exceed the number of assigned examiners (i.e.,

one-on-one evaluations are mandatory), except as noted below. However, if the facility licensee's technical specifications routinely require more than two ROs to be stationed in the control room, the chief examiner may authorize the use of additional surrogates. Only one individual (applicant or surrogate) is allowed to fill a shift supervisor or manager position during the simulator operating test. If a three-person operating crew consists entirely of SRO-upgrade applicants , (who do not have to be evaluated on the control boards), the region may assign ! only two examiners to observe the crew. Although the applicants in the RO and b balance of plant positions may not be individually evaluated, they will be graded V and held accountable for any errors that occur as a result of their action (s) or inaction (s). SRO-instant applicants will always be individually evaluated regardless what operating position they are filling during a given scenario. Normally, for purposes of test integration and continuity, the same examiner should administer all three operating test categories to an applicant. However, under certain circumstances, the walk-through portion of the operating test may l be divided among different examiners. Such division is appropriate if a facility licensee's simulator is not located near the plant, because of limitations in l l examiner resources or scheduling, c. a y licensee requests examinations , for an unusually large group of applicants. Rewr to ES-302 for specific { instructions regarding administration of the operating tests. I i l l Operating tests will normally be administered on regular work days. If weekend l or shift work is required to administer the operating tests, the chief examiner will coordinate the arrangements with the assigned examiners and the facility i licensee. l l I The written examinations may be administered as soon as they and the license applications (inc;uding any applicable waivers) have been approved. The region shall not allow the written examination and operating test dates to diverge by more than 30 days without obtaining concurrence from the NRR operator ! licensing program office. l 11 of 24 NUREG-1021, Revision 8 l

1 1 1 if, as an efficiency measure, the facility licensee prepared the written examination or operating tests in conjunction with another facilbj, then the two examinations / tests rnust be administered at the same time. If the examination schedule has to be changed on short notice, the chief examiner will work with his or her supervisor and the designated facility contact to reschedule the examinations to a time when examiners are available and other examinations are not affected.

k. If the facility licensee will administer the written examinations, the chief examiner shall review the ES-402 requirements (e.g., proctoring and responding to applicant questions) and confirm the app!icant's status on the assignment sheet (i.e., examination type and waivers) with the facility contact before the l

examinations are given. D. PERSONNEL RESTRICTIONS It is impossible to define criteria that anticipate every possible conflict-of-interest issue. Supervisors must apply sound judgment to the facts of each case. If any doubt exists regarding 1 a particular case, the supervisor should consult with regional management and/or the NRR operator licens:ng program office to resolve the issue.

1. NRC Examiners i

a. The regional office shall not assign an examiner who failed an applicant on an s

operating test to administer any part of that applicant's retake operating test. '

b. If an examiner was previously employed by a facility licensee (or one of its contractors) and was significantly involved in training the current license applicants, the regional office will ^not assign that examiner any direct responsibilities for developing or administering written examinations or operating tests at that facility. Regional management will control other in-office l examination activities concerning the facility, such as technical consultation and quality reviews of examinations.

1

c. If an examiner is assigned to an examination that might appear to present a conflict of interest, the examiner shall inform his or her immediate supervisor of the potential conflict. Such notifications should include the following information:

1 the nature and extent of previous personal and professional relationships I with the applicants anything that could affect the administration, performance, evaluation, or results of the examination anything that could create the appearance of a conflict of interest NUREG-1021, Revision 8 12 of 24

O ES-201 O 2. ' Facility Personnel

a. The facility licensee shall minimize the number of personne! who have detailed knowledge of the NRC licensing examination. Moreover, the facility licensee should limit each person's access to only thu ertions of the examination for which the individual bears responsibility (e.g. udividuals who prepare the simulator scenarios may not require access ti written examination).

b. All personnel who will receive detailed knowledge of any portion of the NRC licensing examination, including the examination outline, must acknowledge their responsibilities by reading and signing Form ES-201-3, " Examination Security Agreement," before they obtain detailed knowledge and again after the examinations are complete. Examples of prohibited activities for personnel who have signed Form ES-201-3 include the following:

the design and administration of any classroom and simulator instruction (including scheduled sessions, individual coaching, and remedial training) specifically for the license applicants (Simulator booth operation is acceptable if the individual does not select the training content or provide direct or indirect feedback. Continued participation in requalification training for groups including SRO upgrade applicants is also acceptable, as long as it is documented on Form ES-201-3 and is limited to areas in which the instructor has no examination knowledge.) h V all on-the-job training, practice, coaching, and sign-offs the preparation, review, grading, and evaluation of periodic quizzes, examinations, and simulator exercises (Individuals on the security agreement may prepare and grade the audit examination subject to an NRC review for test item duplication.) Supervisors and managers having knowledge of the examination content may continue their general oversight of the training program fo; the license applicants, including the review of examinations, quizzes, and remedial training programs. However, those supervisors and managers may not provide individual applicant feedback regarding the content of those examinations, quizzes, or programs. The original security agreement forms must be submitted to the NRC regional office for retention after the examinations are complete. E. ~ ATTACHMENTS / FORMS Attachment 1, " Examination Security and Integrity Considerations" Attachment 2, " Reference Material Guidelines for Initial Licensing Examinations" Attachment 3, " Sample Corporate Notification Letter" Attachment 4, " Sample Examination Assignment Sheet" Form ES-201-1, " Examination Preparation Checklist" Form ES-2012, " Examination Outline Quality Checklist" Form ES-201-3, " Examination Security Agreement" 13 of 24 NUREG-1021, Revision 8

f ES-201 Examination Security and Attachment 1 Integrity Guidelines NRC and facility licensee personnel must be attentive to examination security measures to ensure compliance with 10 CFR 55.49; moreover, pursu int to 10 CFR 55.40(b)(2), facility licensees that elect to prepare their own examinations must establish, implement, and maintain procedures to control examination security and integrity. At the time the examination arrangements are confirmed, an NRC examiner shall review the facility licensee's security procedures and brief the facility contact on the following examination security guidelines. Although these guidelines are not regulatory requirements, facility licensees are encouraged to consider them when establishing their own procedures. Physical Security Guidelines

1. The NRC expects that personnel will be aware of the facility licensee's physical security measures and requirements (as documented in the facility licensee's approved procedures), sign the NRC's examination security agreement, and understand their security responsibilities, including the limits on their interaction with the license applicants (as discussed in Section D.2 of ES-201), before they are given knowledge or custody of any examination materials.

2. All examination-specific materials (i.e., the examination outlines and final examinations) shall be positively and continuously controlled and protected as sensitive information (i.e., under lock-and-key or in the custody of someone who has sig.ied the security agreement). Drafts, copies, and waste materials must also be contmiled and disposed of properly.

The NRC expects that the examinations will NOT be developed and stored on a computer network to which the license applicants or other persons not on the security agreement could gain access. t

3. The examination outlines, written examinations, and operating tests that are sent to the I NRC regional office shall be p! aced in a double envelope. The inner envelope shall be l conspicuously marked "FOR OFFICIAL USE ONLY" and "TO BE OPENED BY l ADDRESSEE ONLY." Furthermore, the cover letter forwarding the examination materials shall state that the materials be withheld from public disclosure until after the examinations are complete. ,

J The facility licensee should follow up on its examination mailing by communicating with the NRC chief examiner to ensure that the package was received. The examination outlines and examinations shall not be transmitted via non-secure electronic means (e.g., the Internet); they may be transmitted via the NRC's " AUTOS" local area network in the resident inspector's office. l

4. The facility licensee is expected to immediately report to the NRC chief examiner any l indications or suggestions that examination security may have been compromised, even l l if the situation is identified and corrected before the examination is submitted to the I i 1 NUREG-1021, Revision 8 14 of 24 j

?' O ES-201 2 Attachment 1 NRC for review and approval. The NRC will evaluate such situations on a case-by case basis and determine the appropriate course of action.

5. The facility licensee and the NRC should determine if examination security problems were noted in the past and ensure that corrective actions have been taken to preclude recurrence.

l

6. The facility licensee and the chief examiner will review the simulator security l considerations in Appendix D to ensure that the instructor station features, programmers' tools, and extemal interconnections do not compromise examination ;

integrity. The primary objective is to ensure that the exam material cannot be re.id or recorded at other unsecured consoles, and that examination materials are either physically secured or electronically protected when not in use by individuals listed on the security agreement. Examination Bank Limitations

1. The facility licensee and chief examiner shall ensure that written examinations and operating tests conform with the guidelines in ES-301 and ES ^01 regarding the use of items taken directly from the bank, modified items, and new items.

2. If the facility licensee has an open bank, it will not place any new or modified test items '

(written questions, job performance measures, or simulator scenarios) that will be used O'() on the examination in its examination bank until after the last examination has been administered. Other Considerations

1. The NRC will consider an examination to be potentially compromised if any activity occurs that could affect the equitable and consistent administration of the examination, regardless of whether the activity takes place before, during, or after the examination is administered.

2. The license applicants should not be able to predict or narrow the possible scope or content of the licensing examination based on the facility licensee's examination practices (other than those authorized by this NUREG or in writing by the NRC).

3. Facility licensees are responsible for the integrity, security, and quality of examinations prepared for them by contractor personnel.

l p 15 of 24 NUREG 1021, Revision 8

ES-201 Reference Material Guidelines Attachment 2 For Initial Licensing Examinations This attachment discusses the reference materials that facility licensees are expected to provide for each NRC initial licensing examination. The regional office will customize the list of reference materials as required to support the specific examination assignment; additional materials may be requested at a later time if necessary to ensure the accuracy and validity of the examinations. In determining the need for reference materials, the regional office will consider the facility licensee's level of participation in the examination development process. If the facility licensee will be preparing the examinations, it may be sufficient to obtain only those references necessary to review and validate the items that appear on the examination, plus a set of key procedures and other documents required to prepare for the operating tests. The regional office will duly consider the administrative burden it places on facility licensees and request only those materiais that are actually necessary for the NRC examiners to prepare for the examinations. All reference materials provided for the license examinations should be approved, final issues and should be so marked. If any of the materialis expected to change before the scheduled examination date, the facility licensee should reach agreement with the NRC chief examiner regarding changes before the examinations are administered. The reference materials may be submitted on computer diskettes (in a format compatible with the NRC's wor

nrocessing software), as hard copy, or a combination as arranged with the NRC chief exan. ar. If the facility licensee prepares the examinations, the hard-copy references should normally be limited to those materials required to validate the selected test items. All procedures ar d reference materials should be bound with appropriate indices or tables of contents so that they can be used efficiently; a master table of contents should be provided for all materials sent. Failure to provide complete, properly bound, and indexed reference material may prompt the NRC to return the material to the person at the highest level of corporate management responsible for plant operations. The returned reference materials will be accompanied by a cover letter explaining the deficiencies in the material and the basis for postponing or canceling the examinations. >

Unless otherwise instructed by the NRC regional office, the facility licensee is expected to provide the following reference materials for each NRC initial licensing examination:

1. Materials used by the facility licensee to ensure operator competency

a. The following types of materials used to train applicants for initial RO and SRO licensing, as necessary to support examination development:

learning objectives, student handouts, and lesson plans system descriptions of all operationally relevant flow paths, components, controls, and instrumentation NUREG-1021, Revision 8 16 of 24 ,

1 1 ES-201 2 Attachment 2 material used to clarify and strengthen understanding of normal, abnormal, and emergency operating procedures complete, operationally useful descriptions of all safety system interrdions and, where available, balance-of-plant system interactions und6; emergency and abnormal conditions, including consequences of anticipated operator errors, maintenance errors, and equipment failures, as well as plant-specific risk insights based on a probabilistic risk analysis (PRA) and individual plant examination (IPE) These materials should be complete, comprehensive, and of sufficient detail to support the development of accurate and valid examinations without being redundant.

b. Questions and answers specific to the facility training program that may be used in the written examinations or operating tests

c. Copies of facility-generated simulator scenarios that expose the applicants to abnormal and emergency conditions, including degraded pressure control, degraded heat removal capability, and containment challenges, during all modes of operation, including low-power conditions (A description of the scenarios used for the training class may also be provided.)

d. All JPMs used to ascertain the competence of the operators in performing tasks within the control room complex and outside the control room (i.e., local operations) as identified in the facility's job task analysis (JTA) (JPMs should evaluate operator responsibilities during normal, abnormal, and emergency conditions and events, and during all modes of operation including cold shutdown, low power, and full power.)

2. Complete index of procedures (including all categories sent)

3. All administrative procedures applicable to reactor operation or safety

4. All integrated plant procedures (normal or general operating procedures)

5. All emergency procedures (emergency instructions, abnormal or special procedures)

6. Standing orders (important orders that are safety-related and may modify the regular procedures)

7. Surveillance procedures that are run frequently (i.e., weekly) or that can be run on the simulator

8. Fuel handling and core loading procedures (if SRO applicants will be examined)

\ 17 of 24 NUREG-1021, Revision 8

ES-201 3 Attachment 2

9. All annunciator and alarm procedures

10. Radiation protection manual (radiation control manual or procedures)

11. Emergency plan implementing procedures

12. Technical Specifications or similar technical requirements documents (and interpretations, if available) for all units for which licenses are sought

13. System operating procedures

14. Technical data book and plant curve information used by operators as well as the facility .

precautions, limitations, and set points document l l l

15. The following information pertaining to the simulation facility:

a. list of allinitial conditions

b. list of all malfunctions with identification numbers and cause and effect !

l information, including a concise description of the expected result or range of results that will occur upon initiation and an indication of which annunciators will be actuated as a result of the malfunction

c. a description of the simulator's failure capabilities for valves, breakers, indicators, and alarms

d. the range of severity of each variable malfunction (e.g., the size of a reactor coolant or steam leak, or the rate of a component failure such as a feed pump, turbine generator, or major valve)

d. a list of modeling conditions (e.g., simplifications, assumptions, and limits) and problems that may affect the examination

f. a list of any known performance test discrepancies not yet corrected

g. a list of differences between the simulator and the reference plant's control room

h. simulator instructor's manual

16. Any additional plant-specific material that has been requested by the NRC examiners to develop examinations that meet the guidelines of these standards and the regulations NUREG-1021, Revision 8 18 of 24 l

l i ES-201 Sample Corporate Attachment 3 Notification Letter

                                                                                                   - (Datel (Name. Title)

(Name of facilitv) (Address) l

     . (City. State. Zio code)                                                                                 l

Dear (Name):

l 1 in a telephone conversation on (date) between Mr/Ms. (Name. Title) and MrJMs. (Name. Title), arrangements were made for the administration of licensing examinations at Ifacility name) j during the week (s) of (date). i [As agreed during the telephone conversation, your staff]((The NRC)) will prepare the examinations based on the guidelines in Revision 8 of NUREG-1021, " Operator Licensing i Examination Standards for Power Reactors."[ The NRC regional office will discuss with your staff any changes that might be necessary before the examinations are administereri.j(( Your staff will be given the opportunity to review the examinations during the week of IdgLe).)) To meet the above schedule, it will be necessary for your staff to furnish the Jcxamination outlines by Ldata). The written examinations, operating tests, and the suppe. ting] reference materials identified in Attachment 2 of ES-201 [will be due] by (date). [ Pursuant to i 10 CFR 55.40(b)(3), an authorized representative of the facility licensee shall approve the , outlines, examinations, and tests before they are submitted to the NRC for review and approval. ) All materials shall be complete and ready-to-use.) Any delay in recel/ing the sequired (examination and] reference materials, or the submittal of inadequate or incomplete materials, may cause the examinations to be rescheduled. i in order to conduct the requested written examir.aiioi.: snd ooera%g tests, it will be necessary I for your staff to provide adequate space and accommodations la accordance with ES-402, and to make the simulation facility available on the dates noted above. In accordance with ES-302, your staff should retain the original simulator performance data (e.g., system pressures, temperatures, and levels) generated during the dynamic operating tests until the examination [ i results are final. f I Appendix E of NUREG-1021 contains a number of NRC policies and guidelines that will be in effect while the written examinations and operating tests are being administered. To permit timely NRC review and evaluation, your staff should submit preliminary reactor operator and senior reactor operator license applications (Office of Management and Budget , (OMB) approval number 3150-0090), medical certifications (OMB approval number 3150-0024), and waiver requests (if any)(OMB approval number 3150-0090) at least 30 days before the first examination date, if the applications are not received at least 30 days before the examination date, a postponament may be necessary. Signed applications certifying that all training has been completed should be submitted at least 14 days before the first examination date. 19 of 24 NUREG-1021, Revision 8

I l l ES-201 2 Attachment 3 l This letter contains information collections that are subject to the Paperwork Reduction Act of

    -1995 (44 U.S.C. 3501 et seq.). These information collections were approved by the Office of Management and Budget, approval number 3150-0101, which expires on September 30,2000.

The public reporting burden for this collection of information is estimated to average [500] ((50)) hours per response, including the time for reviewing instructions, gathering and maintaining the data needed, { writing the examinations, ]and completing and reviewing the collection of information. Send comments on any aspect of this collection of information, including suggestions for reducing the burden, to the information and Records Management Branch (T-6 F33), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by Internet electronic mail at BJS1 @ NRC. GOV; and to the Desk Officer, Office of Information and l Regulatory Affairs, NEOB-10202 (3150-0101), Office of Management and Budget, Washington, DC 20503. The NRC may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Thank you for your cooperation in this matter. {Name) has been advised of the policies and guidelines referenced in this letter. If you have any questions regarding the NRC's examination procedures and guidelines, please contact (name of reg,ional contact) at (telephone number), or 10ame of resoonsible regio. nal supervisorl at [tglephone number). Sincerely, (Aporopriate reaional representative. Titlej Docket No.: 50-(Number) Distribution: PubUc NRC Document Control System } Regional Distribution [] Include only for examinations to be prepared by the facility licensee. (()) Include only for examinations to be prepared by the NRC. NUREG-1021, Revision 8 20 of 24

t I /G 4, ES-201 Sample Examination Assignment Sheet Attachment 4 l l MEMORANDUM TO: (List NRC examiners by name) FROM: (Reaional Suoervisor's Name. Title)

SUBJECT:

EXAMINATION ASSIGNMENTS i APPLICANT DOCKET NO. EXAMINATION TYPE O 1 1 Facility and location Facility contact NRC chief examiner I Written examinations to be prepared by (RO) (SRO) Dates of Examinations 1 NRC Supervisor ! cc: Resident inspector Project manager (Standard regional distribution) NRR operator licensing program office l m 21 of 24 NUREG-1021, Revision 8

ES-201 Examination Preparation Checklist Form ES-201-1 Facility: Date of Examination: Examinations Developed by- Facility / NRC (circle one) Target Chief Date* Task Description / Reference Examiner's initials

         -180        1. Examination administration date confirmed (C.1.a; C.2.a & b)
         -120        2. NRC examiners and facility contact assigned (C.1.d; C.2.e)
         -120        3. Facility contact briefed on security & other requirements (C.2.c)
         -120        4. Corporate notification letter sent (C.2.d)

(-90] [5. Reference material due (C.1.e; C.3.c)]

          -75        6. Integrated examination outline (s) due (C.1.e & f; C.3.d)
          -70        7. Examination outline (s) reviewed by NRC and feedback provided to facility licensee (C.2.h; C.3.e)
          -45        8. Proposed examinations, supporting documentation, and reference materials due (C.1.e, f, g & h; C.3.d)
          -30        9. Preliminary license applications due (C.1.l; C.2.g; ES-202)
          -14        10. Final license applications due and assignment sheet prepared (C.1.l; C.2.g; ES-202)
          -14         11. Examination approved by NRC supervisor for facility licensee review (C.2.h; C.3.f)
          -14        12. Examinations reviewed with facility licensee (C.1.j; C.2.f & h; C.3.g)
           -7         13. Written examinations and operating tests approved by NRC supervisor (C.2.i; C.3.h)
           -7        14. Final applications reviewed; assignment sheet updated; waiver letters sent (C.2.g, ES-204)

15. Proctoring / written exam administration guidelines reviewed with

           -7             facility licensee and authorization granted to give written exams (if applicable)(C.3 k)
           -7         16. Approved scenarios, job performance rneasures, and questions distributed to NRC examiners (C.3.i)

Target dates are keyed to the examination date identified in the corporate notification letter. i They are for planning purposes and may be adjusted on a case-by-case basis in coordination ! with the f acility licensee. [] Applies only to examinations prepared by the NRC. . O1I i NUREG-1021, Revision 8 22 of 24 l l

i l l l l 1 ES-201 Examination Outline Form ES-201-2 j Quality Checklist

 !p  i
                                 =

I

 \v}    Facihty-                                                                                   Date of Examination:                      I l

Initials item Task Description a b* c

1. a. Venfy that the outrsie(s) fit (s) the appropriate model per ES 401. J W b. Assess whether the outline was systematically prepared and wnether all knowledge and ability R

categones are appropriately sampled. g l

c. Assess whether the outline over emphasizes any systems, ervolutions, or generic topics.

E N d. Assess whether the repetition from previous examinabon outlines is excessive. l

2. a. Using Form ES-301-5, venfy that the proposed scenario sets cover the required number of !

normal evolutions, instrument and component failures, and major transients. I S ! I b. Assess whether there are enough scenano sets (and spares) to test the projected number and i M mix of applicants in accordance with the expected crew composition and rotation schedule without compromising exam integnty; ensure each applicant can be tested using at least one new or significantly modified scenario, that no scenarios are duplicated from the applicants' audit test (s)*, and scenar'os will not be repeated over successive days.

c. To the extent possible, assess whether the outline (s) conform (s) with the qualitative and quantitative entena specified on Form ES-3014 and described in Appendix D.

3. a. Venfy tnat (1) the outline (s) contain(s) the required number of control roem and in-plant tasks, W (2) no more than 30% of the test rnaterialis repeated from the last NRC examination.

/n\         /      (3)* no tasks are duplicated from the applicants' audit test (s), and k   )        T      (4) no more than 80% of any operaiing test is taken directty from the licensee's exam banks.

v

b. Venfy that:

(1) the tasks are distributed among the safety function groupings as specified in ES-301, (2) one task is conducted in a low-power or shutdown condition, (3) 40% of the tasks require the applicant to implement an afternate path procedure, I (4) one in-plant task tests the applicant's rtsonse to an emergency or abnormal condition, and l (5) the in-plant walk-through requires the applicant to enter the RCA.

c. Verify that the required administrative topics are covered, with emphasis on performance-based activities. _

d. Determine il there are enough different outlines to test the projected number and mix of applicants and ensure that no items are duplicated on successive days.

4. a. Assess whether plant-specific priorities (including PRA and IPE insights) are covered in the aopropriate exam section.

G E b. Assess whether the 10 CFR 55.41/43 and 55.45 sampling is appropriate. N E c. Ensure that K/A importance ratings (except for plant-specific pnonties) s.re at least 2.5. R A d. Check for duplication and overiap among exam sections. _

e. Check the entire exam for balance of coverage.

f. Assess whether the exam fits the appropriate job level (RO or SRO).

Printed Name / Signature Date

a. Author

b. Facihty Reviewer (*)

c. Chief Examiner

( J

d. NRC Supervisor

\. / (*) Not applicable for NRC-developed examinations. 23 of 24 NUREG-1021, Revision 8

3- t a t E d h d o T 1 0 ee t en O h iz t s r ly 2- nn edl N O t r oho be d S f eo t isdc ia E E st o n s mis t nI i f T a aut e e ,i c m d a erc g imn di o p e A r n el e w 'e og u at as D - o e dulos f F e n ne s st rs nsa _ b hebecno r t oi o it int n cdlias eyr sn p e soio t t

                                             / io                   id ame n c
                                                                                           )

2 ( l x foha n vt nidt alyaf naic c a ma e an , E R

             )            cl                                           xos                 U (hs pch    ol niaes i

dn eint gao T k w piacniy i t f o ni t A e nia N es wne eia t n a i smn G o spd n exm nai I - es o seir me S h r h sn t et ae ai n t lceax i f r pot oy t pmexma Cue Rng f nkcc e c u ex Nion E da axoh e e lutobd edf t f eet is n T

                                      ,s    oi e                     h lpe t                    A d s enan hc                                              gml ic              D e n eo( o hci of it sic                                            noc e im ees t

t e at r nlaR s sn ansel ct eN ehh ct t omanmc ni i amie ne nl ot d i t n i t ax r odu mi ah r c t r c nu ee r z nef aqno n t t ) imee ons a espo nr r 1 m ei t it en i ( r lt n a E e x h eidue t m mim R r g et d gt uivi n a e a nsm r e o r e ed U A r a y in o paromesy r a ag f ng e i ar T A n 4 t i r sb na r xuma y ye N G 2 u c e no es nt aruow at m oO i f e o en ey I c i o ,f s S e l it n ch S Cat Rmuioyec a nm mi t il o s s s ew 4 2 n it ea r st io Nr o laer vl ug t r f ein ph a t t ef hinepcat o t c n t yt

                              ,me st                                  det oil t

un uclat co sih o r izip np Y im oa p r T a best rl oda agn nsicfore t i I x h er e L E elu i uysy t u eso I B h nl t di gv oopotet t anh net I S ia ledt a et i i uio N wlynodsthdd ne . yt n ak t O ognmifoc h omd e ah c a P S kniat w mreeii ms to eb t E d o It o ahl m ead t e R ent tar wf liwop t gd e / r s iuef E izk hf no aoo I.m l aodnmn icn it d vh ec . L T e! aiaieo t t mnC aR I T sws naI aonsec

                      !                                                t p!       t oo r

eieie

                                           ,l n            nrFr mN             B dI et d              mro v iec be                      d id o

r h e O J ut au naxmt

                                                                               .f ir h                             al                                     t r     hye                    I              e p y q eI ghdi         e    et it               v          e,              b c                            a l

e are r, ntr n ch a g id de 8 egeiuafs y n d e vz E n n o v an n F. tse a o oi r o haIer.imeCet r hm it a lw r o M is it c of p A aiRd i a r y n h v n I x l nor t N e k) ou,a i t im t au f eeNnoi s u eur m a y (s ed D R a h tae t e e h t n dmc e x mkt E , x e niht E- e an T 1 E- gighcd yat b) nn ss t f o waw elua N 2 0 e dey s Ce s I r rdereai sio o t s e e vo el R 1 P l wmRe gt a P bt h t ,e P G@ t : 1 0 of Nszu i id a nin r egc b S E E knoenioe om 2- ud R ct eh mh ot t cia h inr t t r e . . . . T O U S a a yd ur cx t ou st n o . . . . . . . . . 01 . . 2345 E 1 I db aapae 2 Tdi n 1 234567891 1 1 1 1 1 N N

n

       ~
    /*                                                   ES-20P.

( PREPARING AND REVIEWING OPERATOR LICENSING APPLICATIONS l A. PURPOSE I l This standard provides instructions for facility licensees and applicants to prepare and the NRC to review initiallicensing applications, it also discusses the experience, training, education, and certification requirements and guidelines that an applicant should satisfy before being allowed to take an NRC reactor operator (RO), senior reactor operator (SRO), or limited senior reactor operator (LSRO) licensing examination. B. BACKGROUND Tha Cornmission-approved licensed operator training programs at most power reactor facilities are based on a systems approach to training (SAT) and use simulation facilities that have 'oeen either certified by the facility Scensee or determined to be acceptable by the Commission under i 10 CFR 55.45(b). In accordance with 10 CFR 55.31(a)(4), these facilities are not required to ; include details of the applicant's qualifications, experience, and training on the NRC license ) application form. In lieu of these details, the Commission will generally accept certification by ; an authorized representative of the facility licensee that the applicant has successfuliy completed the facility's Commission-apprcved training program. j If the facility licensee does not have a SAT-based licensed operator training program that uses a simulation facility acceptable to the Commission, the NRC will not accept the license C application unless it includes the details of the applicant's qualifications and training. Detailed ( license eligibility requirements and guidelines are derived from 10 CFR Part 55, Subpart D,

        " Applications," and Regulatory Guide (RG) 1.8, Revision 2, " Qualification and Training of Personnel for Nuclear Power Plants," respectively. With respect to license applicants, PD 1.8       i endorses, with exceptions, the guidance in American National Standards Institute / American Nuclear Society (ANSI /ANS)-3.1-1981, " Selection, Qualification and Training of Personnel for      j Nuclear Power Plants." NRC examiners should refer to those documents as necessary when               I evaluating the eligibility of applicants at facilities that do not use an NRC-approved or facility-certified simulator as part of a SAT-based licensed operator training program.

C. RESPONS181LITIES l The regulatory requirements associated with the license application process are detailed in Subpart D, " Applications," of 10 CFR Part 55. The medical requirements for license applicants and licensed operators appear in Subpart C, " Medical Requirements," of 10 CFR Part 55. These requirements should be referred to as necessary when preparing and reviewing license applications. , 1

1. Apolicant/ Facility Licensee

a. To apply for an RO or SRO license, an applicant must submit an NRC Form 398,

                           " Personal Qualifications Statement - Licensee," and an NRC Form 396,
                           " Certification of Medical Examination by Facility Licensee." (Computer-generated duplicates are acceptable.) The application is not complete until both
   \~

1 of 11 NUREG-1021, Revision 8

c ES-202 forms are filled out, signed by the appropriate personnel, and received by the NRC. Detailed instructions for completing NRC Form 398 are provided with the form. Additionalinstructions regarding waivers of training, experience, and examination requirements are provided in ES-204. If the applicant is reapplying after a license denial,10 CFR 55.35 applies, and the applicant must complete and submit a new Form 398. The applicant may file the second application two months after the date of the first final denial, a third application six months after the date of the second final denial, and successive applications two years after the date of each subsequent denial. Each new Form 398 shall describe the extent of the applicant's additional training since the denial and shall include a certification by the facility licensee that the applicant is ready for reexamination. if the applicant previously passed either the written examination or the operating test, he or she may request a waiver of that portion of the licensing examination. Such waivers are limited to the first reapplication and must be requested within one year of the date of the failed examination. Refer to ES-204 for a more detailed discussion of this and other waiver criteria The medical data in support of NRC Form 396 are normally good for six months from the date of the medical examination. If more than 6 months have passed since the date of an RO or SRO instant applicant's medical examination, the ! facility licensee shall certify in writing that the applicant has not devek> ped any physical or mental condition that would be reportable under 10 CFR 55.25. If the time since any applicant's last medical examination is expected to exceed 24 l months before the licensing action is completed, the applicant shall be I reexamined by' a phy&k:n and the facility licensee shall recertify the applicant's I medical fitness on NRC Form 396 before the NRC regional office issues the license. ! If an applicant is reapplying after withdrawing a previous application or accepting a finallicense denial, he or she may request a waiver of a medical reexamination j by checking item 4.f.4 on NRC Fom1398. The time since the last medical l examination can not exceed 24 months and the applicant must certify in item 17, l "Cornments," of the form that he or she has not developed any physical or mental condition tnat would be reportable under 10 CFR 55.25,

b. Each applicant (except those applying for an LSRO license) must satisfactorily complete the NRC's generic fundamentals examination (GFE) section of the written operator licensing examination for the applicable vendor. Refer to ES-205 for more information on the GFE program.

Applicants do not need to take the GFE if they were previously issued an RO or SRO license or an instructor certificate based on a site-specific written examination (on the same type of facility) that was administered between February 1982 and November 1989 and included the material covered by the NUREG 1021, Revision 8 2 of 11

e

     ]                                                                                         ES-202 GFE. Enter the date of the examination in item 4.g on NRC Form 398 and an explanation in item 17; a waiver is not required.

c. As noted in ES-201, the facility licensee should submit preliminary, uncertified license applications and medical cedfications for review by the NRC regional office at least 30 days before the examination date. This will permit the NRC to make preliminary eligibility determinations, process the medical certifications, t

evaluate any waivers that might be appropriate, and obtain additional information, if necessary, while allowing the facility licensee to finish training the applicants before the certified applications are due.

d. The facility licensee's senior management representative on site must certify -

when an applicant has completed all of the facility licensee's requirernents for the desired license level (i.e., experience, control manipulations, training, and medical).- Such certification involves placing a check in item 19.b of NRC Form 398, signing the form, and submitting it to the NRC regional office at least 14 days before the examination date. The facility must also submit a written request that the written examination and operating test be administered to the applicant.

e. When the NRC regional office denies a license application, the applicant may not accept the proposed denial. In such instances, the applicant may request that the Director, Division of Inspection Program Management (DIPM), Office of Nuclear Reactor Regulation (NRR), review the application denial or request a hearing in accordance with 10 CFR 2.103(b)(2). Further action will be taken in accordance with ES-502.

f. The facility licensee is expected to inform the NRC regional office in writing if it desires to withdraw an application before the licensing process is complete.

2. NRC Reaional Office j

a. The NRC regional office shall review the preliminary applications as soon as possible after they are received. In that way, the regional office can process the medical certifications, evaluate and resolve any waiver requests in accordance with ES-204, and obtain from the facility licensee any additional information that might be necessary in order to support the final eligibility determinations.

With regard to the medical certifications, the regional office shall forward the applicant's NRC Form 396 and the supporting medical evidence to the NRC physician at the Headquarters Health Unit or the regional contract physician for evaluation any time the examining physician has recommended that the - applicant be issued a restricted license or that an existing restriction be changed (by checking block A.4 or A.5 on Form 396). 3 of 11 NUREG-1021, Revision 8

ES-202 The NRC will not process a retake application if the applicant's request for reconsideration or a hearing on the previous license denial is still outstanding (refer to ES 502). Before entering the applicants' data in the operator licensing tracking system (OLTS), the region shall verify that none of the applicants' names appear on the

         " Restricted Individuals List" found on the NRC's web site at http1/www. internal.

nrc. gov /OE/ restrict.htm. The region Lnali check with the appropriate contact in the Office of Enforcement by telephone or electronic mail to verify that the information on the subject individual is current before using the information on the list to deny a licensing action. s

b. The regional office will verify that the applicant has successfully passed the GFE, if required, and review the data on NRC Form 398 to ensure that it is complete.

Affirmative responses to items 12.a and 12.b of NRC Form 398, indicate that the applicant has successfully completed a Commission-approved, SAT-based training program that uses a simulation facility acceptable to the Commission under 10 CFR 55.45(b). If the facility licensee checks "yes" in response to these items, the licensee need not complete items 13, " Training," 14, " Experience," and 15, " Experience Details," of NRC Form 398. The region ,may accept the application without further review unless there is reason to request further information conceming the applicant's qualifications. Occasionally, a facility licensee completes items 13,14, and 15 even though they are not required as explained above. In such instances, the region may review the information provided against the eligibility guidelines in Section D for the requested license level and resolve any deviations with the facility licensee. New applications must include the number of significant control manipulations in item 13.3; at least five are required on the facility for which the license is sought. Every effort should be made to diversify the reactivity and power changes for each applicarit. Startups, shutdowns, large load changes, and changes in rod programming are some examples; these changes could be accomplished manually using such systems as rod control, chemical shim control, and recirculation flow control. This requirement can only be waived or deferred under the conditions specified in 10 CFR 55.31(a)(5); situations other than those specified would require an exemption in accordance with 10 CFR 55.11. For ROs applying for an SRO license, certification that the operator has successfully operated the controls of the facility as a licensed operator shall be accepted as evidence of having completed the required manipulations. If an applicant checks "no" in response to items 12.a and 12.b on Form 398, the region shall review the application against the specific RO, SRO, or LSRO eligibility guidelines described in Section D. NUREG-1021, Revision 8 4 of 11

l h' G ES-202 If the applicant is reapplying after a previous examination failure and license denial, the region shall evaluate the applicant's additional training to determine if the facility licensee made a reasonable effort to remediate the deficiencies that caused the applicant to fail the previous examination.

c. The region may determine that the preliminary application is incomplete, that more information _is necessary to make a waiver determination, or that the applicant does not meet the requirements in 10 CFR 55.31. In such instances, the region will note the deficiencies and request that the facility licensee supply additional information when it submits the final, certified license application (or soonerif possible).

Conversely, the region may determine that the preliminary application is

complete, and the applicant meets the eligibility requirements or is expected to meet the requirements pending the receipt of additionalinformation. In such instances, the region shall enter the applicant's name, docket number, and )

examination requirements on the examination assignment sheet in accordance l with ES-201.

d. Upon receiving the final, certified license application, the reviewer shall evaluate any new informat!on to ensure that the eligibility criteria are satisfied. If so, the reviewer shall check the " meets requirements" block at the bottom of Form 398, p sign and date the form. If necessary, the reviewer shall add the applicant's name and other data to the examination assignment sheet in accordance with ES-201. The reviewer shall also ensure that the assignment sheet accurately reflects any examination waivers that may have been granted in accordance with ES-204.

If the region determines that the applicant still does not meet the eligibility requirements, the regional licensing authority will discuss its decision with the NRR operator licensing program office and notify the applicant in writing that the application is being denied and identify the deficiencies on which the denial is based (Attachment 1). The responsible regional supervisor, or designee, shall check the "does not meet requirements" block at the bottom of Form 398, and shall sign and date the form. The applicant's name shall be stricken from the examination assignment sheet; the applicant shall not be permitted to take the licensing examination until the region determines that he or she meets the eligibility criteria. With prior approval from NRR, as necessary, in accordance with ES-204, the region may administer a license examination to an applicant who has not satisfied the applicable training or experience requirements at the time of the examination, but is expected to complete them shortly thereafter. Assuming that the applicant passes the examination, the region shall not issue the applicant's license until the facility licensee certifies that all of the requirements have been completed. (Refer to ES-501 for additional guidance.) ! 5 of 11 NUREG 1021, Revision 8

ES-202 D. LICENSE ELIGIBILITY GUIDELINES Revision 2 of Regulatory Guide (RG) 1.8, " Qualification and Training of Personnel for Nuclear Power Plants," describes a method acceptable to the NRC staff for complying with the Commission's regulations with regard to the training and qualifications of nuclear power plant personnel. For the positions of shift supervisor, senior operator, and licensed operator, this RG generally endorses the guidelines contained in ANSl/ANS-3.1-1981; specific clarifications, additions, and exceptions are noted in Section C, " Regulatory Position," of RG 1.8. The license eligibility guidelines in RG 1.8, Revision 2, and ANSI /ANS-3.1-1981 are summarized below. Except as specifically noted below, experience and training are separate aspects of license eligibility. As stated in NUREG-1262 (in response to Question No.113), a person should meet the experience guidelines before entering the license training program. Time spent in training before entering the license training program may qualify as experience, but time spent in an NRC-approved training program leading up to license eligibility should normally not be double-counted as experience. The NRC regional office should refer all questions regarding license eligibility to the NRR operator licensing program office for resolution.

1. Reactor Operator

a. Experience (1) The applicant should have a minimum of three years of power plant experience, at least one of which is spent at the nuclear power plant for which the license is sought (preferably in the performance of nonlicensed operator duties).

(2) The applicant should spend at least six months performing plant operational duties as a nonlicensed operator at the nuclear power plant for which the license is sought.

b. Training (1) The applicant should complete at least 13 weeks as an extra person on shift in training for the RO position. This training should include all phases of day-to-day operations and be conducted under the supervision of licensed personnel.

(2) The applicant should be trained in nuclear power plant fundamentals and plant systems, use of those systems to control or mitigate an accident during which the core is severely damaged, and operating practice. (3) The applicant should complete at least 500 hours of lectures on the principles of reactor operation, design features and general operating characteristics of the nuclear power plant involved, instrumentation and NUREG-1021, Revision 8 6 of 11

[^\ ES-202 d control (IC) systems, safaty and emergency systems, standard and j emergency operating procedures, and radiation control and safety 1 procedures. (4) The applicant should satisfactorily complete an NRC-approved training program involving at least one week at a nuclear power plant simulator. The simulator training center should certify the applicant's ability during a reactor startup to manipulate the controls, keep the reactor under control, predict instrument responses, use instrumentation, follow procedures, and explain annunciator alarms that occur during operation.

c. Education The applicant should have a high school diploma or equivalent.

2. Senior Reactor Operator

a. Experience (1) The applicant should have a minimum of four years of responsible power plant experience, as defined in RG 1.8. At least two of those four years .

should be nuclear power plant experience, and at least six months of the l nuclear power plant experience should be at the plant for which the l

 \g                   applicant seeks a license.

(2) The applicant should have actively performed licensed RO duties for at least one year at the facility for which the SRO license is sought. The NRC may accept any one or more of the following education or experience qualifications to satisfy this requirement provided that the applicant supplies sufficient details in the license application for the staff to make a judgement regarding equivalence: A four-year degree in engineering or the equivalent (e.g., a degree in engineering technology or the physical sciences that includes course work in physics, mathematics, or engineering; a professional engineer's (PE) license obtained by passing the PE examination). At least one year as an active licensed RO at a comparable facility (same vendor, similar vintage) or 18 months as an RO at a noncomparable commercial power reactor. At least two years in a position equivalent to a licensed RO at a military reactor. Experience obtained in licensed positions (or their equivalent) on p other large-scale reactors will be evaluated on a case-by-case 7 of 11 NUREG-1021, Revision 8

i basis. Applicants must also submit a waiver request in I accordance with ES-204 if they want this experience to apply toward the requirement. (3) During the two years of nuclear power plant experience, the applicant should participate in reactor operator activities at power levels greater than 20 percent for at least six weeks.

b. Training (1) The applicant should complete at least 13 weeks as an extra person on shift in training for the SRO position. 'his training should include all phases of day-to-day operations and be conducted under the supervision of licensed perconnel. Any portion of the 13 weeks that is spent at or above 20 percent power may also be used to satisfy the experience guideline in Section D.2.a(4).

l (2) If the applicant has not held an RO license at the facility and one of the ! qualifications specified in Section D.2.a(2) is substituted for that experience, the training guidelines of Sections D.1.bs4) and D.1.b(5) should be met. The applicant should satisfactorily complete a training program that is comprehensive in its coverage of both RO and SRO knowledge, skills, and abilities and must take an SRO-instant license examination. (3) The applicant should be trained in nuclear power plant fundamentals and plant systems, use of those systems to control or mitigate an accident during which the core is severely damaged, and operating practice. (4) The applicant should also complete the additional instruction specified in Se tion 5.2.1.6 of ANSI /ANS-3.1-1981 in subjects related to the duties of I at

                          'n '

c. Education 1

The applicant should have a high school diploma or equivalent.

3. Limited Senior Reactor Operator

a. Experience The applicant should have three years of nuclear power experience that includes l

active participation in at least one refueling outage at the site for which the I license is sought or at a similar facihty. Six months of the nuclear power plant experience must be at the site for which the LSRO license is sought or at a similar facility owned by the same facility licensee. NUREG-1021, Revision 8 8 of 11

o ES-202

b. - Training The applicant is expected to have satisfactorily completed a training program that ensures that he or she is qualified to supe.rvise fuel handling operations.

The program should be based on a systems approach to training and is expected to include instruction in at least the following areas: (1) nuclear power plant and health physics fundamentals and the principles of reactor theory and thermodynamics (2) design features of the nuclear power plant pertaining to fuel handling activities, including plant systems and equipment associated with fuel handling operations, pertinent IC systems, and features of the emergency core cooling systems (ECCSs) associated with the refueling mode of - operation (3) the use of installed plant systems to control or mitigate an accident in which the core is damaged during refueling operations (4) operating practices and procedures that pertain to refueling, including administrative, operational, surveillance, emergency, radiation control, and safety procedures; the technical specifications applicable to refueling; and the requirements concerning communications and t interfaces with the main control room The applicant should also complete a minimum of 80 hours of on-the-job training (OJT) in refueling activities, including manipulation of the refueling bridge or similar refueling equipment.

c. Education The applicant should have a high school diploma or equivalent.

4. Cold License Eliaibility Cold examinations are those administered before the unit completes preoperational testing and the initial startup test program as described in the FSAR.

a. Each applicant must satisfactorily complete the training programs described in Section 13.2 of the FSAR and approved by the NRC. The NRC's review and approval are based on information contained in Section 13.2.1 of the Standard Review Plan (SRP) (NUREG-0800).

Note: These NRC-approved training programs typically require ten startups on a research reactor. This requirement may be waived if the applicant has completed a plant-referenced simulator training program accredited by the p Institute of Nuclear Power Operations (INPO). 9 of 11 NUREG-1021, Revision 8 )

ES-202

b. In lieu of the control manipulations on the facility for which the license is sought (per 10 CFR 55.31(a)(5)), the Commission may accept evidence of satisfactory performance of simulated control manipulations as part of a Commission-approved training program on a simulation facility acceptable to the Commission under 10 CFR 55.45(b).

E. ATTACHMENTS / FORMS , " Sample initial Application Denial from Region" O NUREG-1021, Revision 8 10 of 11

E O ES-202 Sample initial Application Denial from Reaion NRC Letterhead Attachment 1 (date) (Anolicant's name) (Street address) (City. State. Zio code)

Dear (Name):

This is to inform you that your application of (date) for a (reactor ooerator. senior reactor ooerator) license submitted in connection with the (facility name) is hereby denied. (Reaion to discuss deficiencies and which part of 10 CFR 55.31. ES-202. NRC- aooroved facility trainina oroaram. or Reaulatorv Guide 1.8 was involved.) When you have met the requirements of 10 CFR 55.31, you may submit another application. If you do not accept this denial, you may, within 20 days of the date of this letter, take one of the following actions: You may request that the NRC reconsider the denial of your application by writing to the Director, Division of Inspection Program Management, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, DC 20555. Your request must include specific reasons for your belief that your application was improperly denied. If the NRC determines that the denial of your application remains appropriate, you still have the right to request a hearing pursuant to 10 CFR 2.103(b)(2), as described below. You may request a hearing in accordance with 10 CFR 2.103(b)(2). Submit your request, in writing, to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, DC 20555, with a copy to the Associate General Counsel for Hearings, Enforcement, and Administration, Office of the General Counsel, at the same address. If you have any questions, please contact (name) at (teleohone number). Sincerely, (Regional branch chief or above) Docket No. 55-(number) cc: (Facility reoresentative who sianed the aoolicant's NRC Form 398) CERTIFIED Mall- RETURN RECEIPT REQUESTED 11 of 11 NUREG-1021, Revision 8

ES-204 PROCESSING WAIVERS REQUESTED BY REACTOR OPERATOR AND SENIOR REACTOR OPERATOR APPLICANTS A. PURPOSE This standard provides guidance concerning the processing of waivers requested by reactor operator (RO) and senior reactor operator (SRO) license applicants at power reactor facilities. B. BACKGROUND in accordance with 10 CFR 55.35, "Reapplications," and 10 CFR 55.47, " Waiver of Examination and Test Requirements," an applicant may request to be excused from a written examination or an operating test. The NRC may waive any or all of the examination requirements if it determines that the applicant has presented sufficient justification. In an effort to expedite the resolution of applicant requests, the Office of Nuclear Reactor Regulation (NRR) has delegated the authority to grant routine waivers of certain operator licensing requirements to the NRC regional offices. C. RESPONSIBILITIES

1. Aoolicant/ Facility Licensee

a. An applicant may request a waiver of a license requirement by checking the appropriate block in item 4.f on NRC Form 398, " Personal Qualifications j Statement - Licensee." The applicant should also explain the basis for requesting tha waiver in item 17, " Comments."

b. The facility licensee's senior management representative on site must certify the final license application, thereby substantiating the basis for the applicant's waiver request,

c. Facility licensees having units designed by the same nuclear steam supply system vendor and operated at approximately the same power level may request I dual licensing for their operators. Similarly, if the units of a multi-unit facility are nearly identical, the facility licensee may request a waiver of the examination requirements for the second and subsequent units.

In either case, the facility licensee must justify to the NRC that the differences between the units are not so significant that they could affect the operator's ability to operate each unit safely and competently. Further, the facility licensee must submit for NRC review the details of the training and certification program. The analysis and summary of the differences on which the applicants must be trained will include the following, as applicable: facility design and systems relevant to control room personnel technical specifications procedures (primarily abnormal and emergency operating) b D 1 of 5 NUREG-1021, Revision 8

i l l l [ control room design and instrument location operational characteristics administrative procedures related to conduct of operations at a multi-unit site (e.g., shift manning and response to accidents and fires) the expected method of rotating personnel between units and the I refamiliarization training to be conducted before an operator assumes j responsibility on a new unit {

2. NRC Reaional Office

a. The regional office will evaluate waiver requests on a case-by-case basis against the waiver criteria discussed in Section D of this ES.

b. The regional office may grant routine waivers identified in Section D.1 without first obtaining concurrence from the NRR operator licensing program office.

However, waivers of experience requirements, completion of training, or completion of examinations (e.g., the generic fundamentals examination) not specifically identified in Section D.1 must be approved by NRR. The regional office should evaluate the waiver request and forward its approvdl , recommendation to the NRR operator licensing program offico for concurrence. ! l The region does not require written concurrence from NRR to deny an l applicant's waiver request, but it should discuss its decision with the operator licensing program office before informing the applicant; formal concurrence may be desirable in some cases.

c. If additional information is required to reach a decision on the waiver request, the region shall generally request the necessary information from the facility licensee in accordance with ES-202.

d. Upon deciding to grant or deny a waiver, the regional office shall promptly notify the applicant in writing concerning the disposition of the request, including an explanation for the denial. If time is too short to notify the applicant in writing before the examination date, the regional office shall notify the facility training representative by telephone concerning the disposition of the waiver request and provide a follow-up written response to the applicant. The regional office shall include the NRR operator licensing program office on distribution for all waiver disposition letters.

e. The region shall document the disposition of every waiver request, whether granted or denied, by completing the block designated "For NRC Use" on the applicant's NRC Form 398 and by entering the data in the operator licer. sing tracking system (OLTS).

NUREG-1021, Revision 8 2 of 5 O1

I A ES-204

f. NRC examiners assigned to a particular examination will be notified of approved waivers by the appropriate regional supervisor and by an entry on the examination assignment sheet (ES-202, Attachment 4).

g. If the applicant is determined to be ineligible to take the licensing examination, the regional office shall issue a denial letter in accordance with ES-202.

D. WAIVER CRITERIA

1. Routine Waivers

a. If a1 applicant fails onlythe written examination or one category of the operating test, the region may waive those examination areas (categories) that were passed. This is only applicable for the first retake examination and only if it takes place within one year of the date on which the denial of the original ,

application became final.

b. The region may waive training requirements specified in the final safety analysis report (FSAR) when the FSAR authorizes waiver of those specific requirements and the applicant otherwise meets NRC requirements (e.g., waiver of some training requirements for applicants previously licensed at a comparable facility),

c. The medical data in support of NRC Form 396 are normally good for six months G from the date of the medical examination for a person applying for an RO or an SRO instant license. For reapplications following a license denial or withdrawal of an application, waivers extending the six month period may be granted if the date of the original medical examination is within 24 months of the anticipated licensing date and item 17," Comments," of NRC Form 398 certifies that the applicant has not developed any physical or mental condition that would be reportable under 10 CFR 55.25. For renewal and SRO upgrade applicants, the medical examination documented on NRC Form 396 is good for two years from the date of the medical examination.

d. Substitutions allowed by Regulatory Guide 1.8, Revision 2, are not considered to be waivers and, therefore, do not require approval. For example, substitution of related technical training for up to two years of experience for an SRO or up to one year for an RO is not a waiver. However, training for the examination applied for may not be counted as related technical training.

e. If the facility licensee certifies that the applicant has successfully completed a ;

training program accredited by the Institute of Nuclear Power Operations using ! an acceptable simulation facility, the region may waive the requirement for ten startups on a research reactor typically required by NRC-approved cold license training programs.

f. For those facilities unable to meet the requirement for six weeks on shift at g greater than 20 percent power (because of extended plant shutdowns or other i'

3 of 5 NUREG-1021, Revision 8

l l ES-204 l l extraordinary circumstances), this requirement may be waived upon application if i the following criteria are satisfied: i l (1) Facility training objectives for the desired licensed position have l been developed using a properly validated job and task analysis (JTA). (2) The facility licensee's training program is based on a systems approach to training (SAT) using the five elements defined in 10 CFR 55.4. 1 l (3) The facility licensee can accomplish the training objectives l required for plant operation at greater than 20 percent power l using a plant-referenced or NRC-approved simulation facility. i

g. If an operator was previously licensed at a facility and reapplies for a license at the same facility and license level, the region may, pursuant to 10 CFR 55.47, waive the requirement for the applicant to pass a written examination and an operating test if it finds that the applicant (1) previously discharged his or her responsibilities competently and safely and is capable of continuing to do so (2) terminated participation in the facility licensee's requalification program less than two years before the date of the license application (3) successfully completed " Additional Training," pursuant to 10 CFR l 55.59(b), and a facility-prepared written examination and l operating test which ensure that the applicant is up-to-date in the licensed operator requalification training program i (4) will successfully complete at least 40 hours of shift functions under the direction of an operator or senior operator, as ;

appropriate, and in the position to which the applicant will be ; assigned (see 10 CFR 55.53(f)) before being assigned to licensed I duties l l (5) complies with the requirements of 10 CFR 55.31 I

h. If an applicant at a facility that has completed preoperational testing is unable to perform the five significant control manipulations required by 10 CFR 55.31(a)(5) because of an extended shutdown, the region may process the application, administer the examination, and issue a conditional license that is only valid with the reactor in cold shutdown and refueling. The region will not remove the license condition until the facility licensee supplies the required evidence that the NUREG-1021, Revision 8 4 of 5

ES-204 applicant has successfully completed the control manipulations (refer to ES-501). . Situations other than those specified in the regulation may require an exemption I and must be processed through the NRR operator licensing program office. ; {

2. Examination Waivers for Previousiv Licensed Ooerators at Comparable Facilities (

Depending on the justification provided by the applicant and the facility licensee, NRR , will consider examination waivers for operators who were previously licensed at a comparable facility. Pursuant to 10 CFR 55.47, the Commission may waive any or all requirements for a written examination and operating test.

3. Multi-Unit Examination Waivers

a. Generally, personnel will not be examined on or allowed to hold licenses for "different units" simultaneously. "Different units" owned or managed by a single facility licensee are defined for purposes of this standard as follows:

units having the same vendor but significantly different age and/or power level (e.g., Dresden Units 1 and 2) ; 1 units having the same vendor and similar design but different locations (e.g., Sequoyah and Watts Bar, Byron and Braidwood) , units having different vendors (PWR only) but located on the same site a (e.g., Arkansas Units 1 and 2, Millstone Units 2 and 3) NRR may authorize a limited senior reactor operator (LSRO) to be licensed at I multiple sites, provided that the units are manufactured by the same vendor and are of similar design. The applicant must pass an examination that addresses the differences in the designs, procedures, technical data, and administrative controls of the separate facilities for which the license is being sought.

b. With regard to the examination requirements for ' identical" second or subsequent units at the same site, NRR may waive any of all requirements for a written examination and operating test if it finds that the applicant meets the

! criteria specified in 10 CFR 55.47, as noted in item D.2 above. If the situation warrants, the Commission may impose other examination requirements, such as NRC-administered operating tests and written examinations concerning the plant differences. l I l l I 5 of 5 NUREG-1021, Revision 8

ES-205 PROCEDURE FOR ADMINISTERING THE GENERIC FUNDAMENTALS EXAMINATION PROGRAM A. ' PURPOSE This standard describes the procedures and policies pertaining to administration of the generic fundamentals examination (GFE) section of the written operator licensing examination at power reactor facilities. It describes how the examinations are scheduled and constructed, how to solicit facility licensees for applicants to take the examinations, and how to promulgate the examination results. B. BACKGROUND Sections 55.41 and 55.43 of 10 CFR Part 55 require that the written operator licensing examinations for reactor operators (ROs) and senior reactor operators (SROs) include questions on various mechanical components, principles of heat transfer, thermodynamics, and fluid mechanics. These regulations also require that the written examination address fundamentals of reactor theory, including the fission process, neutron multiplication, source effects, control rod effects, criticality indications, reactivity coefficients, and poison effects. The fundamental knowledge and abilities (K/As) required of an onerator do not vary significantly between license levels or among facilities of the same vendor type. As a result, the NRC implemented the GFE program to standardize the fundamental examinabon coverage for all - applicants at pressurized and boiling water reactors (PWRs and BWRs). Having passed a GFE j as an RO or an SRO applicant, an operator will not have to take 2nother GFE unless he or she transfers to a facility of the other vendor type. The GFE program does not pertain to limited senior reactor operator (LSRO) license applicants. Applicants do nd neec to take the GFE (nor obtain a waiver) if they were previously issued an RO or SRO licenee e, an instructor certificate based on a site-specific written examination (on the same type c facility) that was administered between February 1982 and November 1989 and included the material covered by the GFE. Applicants who were issued a license before 1982 will have to take the examination or apply for a waiver in accordance with ES-204. j l The GFE examinations for BWRs and PWRs are typically administered twice a year, on the j I Wednesday following the first Sunday in April and October. C. RESPONSIBILITIES

1. Facility Licensee

a. The facility licensee must certify that all individuals who plan to take the GFE are enrolled in a facility sponsored training program that will satisfy the eligibility requirements for an RO or SRO license. The operator trainees need not complete all of the training required for the license before they take the GFE.

l l 1 of 13 NUREG-1021, Revision 8 l l

ES 205 The facility licensee may use tbc sample registration letter enclosed with the NRC notification letter (Attachment 1) or any similar format that contains the required inforrnation and certification. If the facility licensee must add or delete 1 an individual after submitting its registration letter, it should inform the Office of ! Nuclear Reactor Regulation (NRR) operator licensing program office of the i change, as specified in the examination cover letter, before the examinations are I administered.

b. When the examinations are received from the GFE contractor, the facility licensee shall reproduce and safeguard the examinations as described in the examination cover letter.

c. On the designated examination day, the facility licensee shall adrainister and proctor the GFE in accordance with the instructions contained in the examination package.

The facility licensee will start and stop the GFE in accordance with the time zone map contained within the examination package. Late arrivals will be allowed to take the examination; however, all examinees must hand in their examinations at the completion time designated in the proctor instructions enclosed with the examination cover letter (refer to Section C.2.d).

d. No later than the day after the GFE is administered, the facility licensee shall i

send the following items via ovemight mail to the name and address designated in the examination package: the original answer sheets the signed exam cover sheets the signed security statements

2. NRR Ooerator Licensina Prooram Office and GFE Contractor

a. The NRR operator licensing program office will designate a coordinator to oversee the GFE activities with the regional offices, the GFE contractor, and the facility licensees.

b. The NRC will send a notification letter (Attachment 1) to each facility licensee 60 days before the GFE administration date. The letter will notify the facility licensee of the date of the examination and request a registration letter listing the licensed operator trainees o whom the facility licensee plans to administer the examination. A sample registration letter is enclosed with the notification letter.

c. The GFE contracior will prepare the examinations as described in Section D of this ES. The examiner assigned responsibility for developing the GFE shall submit the examinations to the NRR GFE coordinator and any other designated

! reviewers at least 20 calendar days before the scheduled administration date. The NRR operator licensing program office will provide comments and NUREG-1021, Revision 8 2 of 13

ES-205 recommended changes to the examination author as soon as possible. The final examinations should be ready at least 14 days before the GFE administration date.

d. The GFE contractor will assemble the approved examination packages as described below, and mail the packages to the names and addresses designated by the participating facility licensees. The examinations should normally be mailed one week before the examinations are scheduled to be administered.

The examination packet will contain the following information, enclosures, and ) attachments: cover letter (Attachment 2 is a sample letter) proctor instructions security agreement single copies of appropriate exam, forms A and B exam time zone map sample answer sheet facility docket number sheet applicant docket number sheet appropriate number of unswer sheets applicant answer sheet instructions

 , e. On the day that the GFE is administered, the NRR GFE coordinator and GFE

\ contractor shall be available to answer questions from facility proctors if the need arises.

f. When the examination answer sheets are received from the facilit'. licensees, the GFE contractor shall score, grade, and tabulate the overall item statistics, and generate facility and regional grade reports for each GFE examination. The contractor shall forward the regional and facility grade reports, including individual scores and copies of individual answer sheets, and corrected answer keys to the applicable regional office for distnbution.

The GFE contractor shall develop individual item statistics on all questions used l on the GFE examinations. Questions with acceptable statistical characteristics shall be moved into the " validated" GFE question bank. The contractor will provide copies of all grade reports to the NRR GFE coordinator, along with the following additionalitems: exam-wide item statistics (PWR and BWR) analysis reports of soecific items deleted or answers changed corrected answer keys original answer sheets original signed exam cover sheets signed security statements O 3 of 13 NUREG-1021, Revision 8

l ES-205

g. The NRR operator licensing program licensing assistant will ensure that copies of the final master BWR and PWR examinations are placed in the NRC's Public I Document Room.

3. NRC Reoional Office

a. Regional management should assign an individual to coordinate GFE administration in the region.

b. The regional operator licensing assistant (OLA) shall assign a docket number to each individual identified in the facility licensee's registration letter. The OLA shall forward the list of names and docket numbers for each facility to the GFE contractor, with a copy to the NRR GFE coordinator, no later than 20 days before the examination administration date.

c. The regional GFE coordinator should keep the NRR GFE coordinator informed of any changes in the number of applicants scheduled to take the GFE at any facility,

d. The regional office shall distribute the GFE examinations to their respective facility licensees. Sample cover letters for facility licensees that did and did not participate in the examination are provided in Attachment 3.

e. The regional OLA shall update the applicants' status (pass or fail) in the operator licensing tracking system (OLTS) and ensure that a hard copy of the GFE results is placed in each applicant's docket file.

D. EXAMINATION SCOPE AND STRUCTIIRE Each GFE shall contain 100 questions covering the " Components" and " Theory" (including reactor theory and thermodynamics) sections of NUREG-1122, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors," or NUREG-1123, "Knowiedge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors." The passing grade for the GFE is 80 percent. The knowledge and ability (K/A) topics applicable to the GFE for PWRs and BWRs have been categorized into various component, reactor theory, and thermodynamics groups as shown in . The attachment also identifies the number of test questions required to evaluate each topic. The NRC will use Institute of Nuclear Power Operations (!NPO) Academy Documents 89-003, " Generic Fundamentals Test Item Catalog - PWR Operator," and 89-004, " Generic Fundamentals Test item Catalog - BWR Operator," as the primary sources of test questions when developing the GFE. The ratio of previously used (i.e., " validated") test questions to new or unvalidated test questions will be adjusted as the size of the validated question bank increases. NUREG-1021, Revision 8 4 of 13

l- \ ES-205

The questions used on the GFE examination shall conform with the applicable construction and style guidelines in Appendix B. E. ATTACHMENTS / FORMS i l Attachment 1, " Sample Notification Letter' s Attachment 2, " Sample Examination Cover Letter" Attachment 3, " Sample Results Letter" Attachment 4, 'GFE Test item Distribution" l l l l 1 V l 1 l l l I 5 of 13 NUREG-1021, Revision 8 4 t.

ES-205 Sample Notification Letter Attachment 1 NRC Letterhead (Date) (Name. Title) (Facility name) (Street address) (City. State Zio code)

Dear (Name):

The NRC plans to administer the generic fundamentals examination (GFE) section of the written operator licensing examination on (date_1 To register personnel to take the GFE, an authorized representative of your facility must submit a letter to the appropriate regional administrator with a copy addressed as follows: Chief, Operator Licensing, Human Performance, and Plant Support Branch Mail Stop OWFN 9 D24 U.S. Nuclear Regulatory Commission Washington, DC. 20o35 Your letter should identify the individuals who will take the examination, and it should certify that they are enrolled in a facility licensee-sponsored program leading to NRC operator or senior operator licensing and that they will have completed their fundamentals training by the date of the examination. The letter should also identify the personnel who will have access to the examinations before they are administered (e.g., proctors) and the address to which the examinations are to be sent. To allow the NRC to assign docket numbers, the letter should be received by both the NRC regional administrator and the Chief, Operator Licensing, Human Performance, and Plant Support Branch,30 days before the examination date. A sample registration letter is enclosed. Sincerely, (Acoropriate reaional representative) Docket No. 50-{t[ umber)

Enclosure:

As stated t NUREG-1021, Revision 8 6 of 13

1 i O ES-205 2 Attachment 1 Enclosure (Name) Regional Administrator U.S. Nuclear Regulatory Commission Region (Number) l (Street address) 1 (City. State Zio code) Dear (Name). (Facility name) requests to have the following (number) individuals take the (BWR or PWR) I generic fundamentals examination (GFE) section of the written operator licensing examination to be administered on (date): Name Date of Birth Previous Docket No. (Insert the name. date of birth. and orevious 10 CFR Part 55 Docket Number (if aoolicab_lq) for each oerson.) { i All of the listed personnel are enrolled in the (facility name) operator licensing training program ; and will have completed the generic fundamentals portion of the program by the examination date. The following personnel will have access to the examinations before they are administered: Name Title (Insert the name and title of each cerson who will have access to the examinations before thev i are administered (e.o.. oroctors). Please address the examinations as follows: Name. Title , Street address City. State Zio code l if you have any questions, please contact (facility contact name) at (telechone number). Sincerely, l Name. title l cc: 1 Chief, Operator Licensing, Human Performance, and Plant Support Branch l l 1 l l l N i- 7 of 13 NUREG-1021, Revision 8 l 1

                    ~         ~~
  ~j205 E                                 Sainple Examination Cover Let3t                      Attachment 2

{Qate) (Name. Title of desianated addresitee.J LEacilitv name) (Strget address) { Gill.StBle._Ep code)

Dear { blame):

Your facility is scheduled to administer the generic fundamentals examination (GFE) section of the NRC's written operator licensing examination on {dat.e_J. {Name of contractor) is authorized under contract to support the NRC in the administration of GFE activities. Note: For security reasons, please open the sealed envelope now and page-check the examination using the enclosed checklist. Tnen contact immediately and no later than (date) one of the persons listed below informing (him or ber) that you have received this package and noting any discrepancies: (Name),(Telephone Number) (Name). { Telephone Number) This letter and its enclosures provide the instructior.s and guidelines for adrninistering the GFE and retuming the comp (eted exams and related materiais to (Name of contracigr). Please read this letter and follow the directions in the accompanying enclosures now, Enclogre 1. Security Aareement. Please referto the enclosed NRC Secunty Agreement. A copy of this agreement must be completed by each and every exam administrator and/or proctor seeing or having knowiedge of the GFE contents. For security reasons, the number of

 . persons seeing or having knowledge of this exam's contents before the exam must be limited to three persons who have a need to know.
                                                                                                              }

The top portion of the security agreement is expected to be completed now, and the bottom l portion immediately after the exam has been completed. Fillin the spaces for each individual's name and the name of the facility for both portions, and have the individual (s) sign the ( form (s). l Please note: The signed security agreements must be retumed to {Name of contracto_rj along } whh the completed exam answer sheets before any scoring will be performed. I i l l l \ L ( l NUREG-1021, Revision 8 8 of 13 9I ' l t I _-_-_---____-________-______-__________________________a

O

ES-205 2 Attachment 2 d ' Enclosure 2. Exam Copies. Two' single copies of Forms A and B of the exam are provided. These altemative forms are identical in content; however, for security purposes, the test item ' sequence on each form is different to reduce the possibility of an applicant copying any answers from a nearby test answer sheet. (See the separate Proctor instructions in Enclosure 3 for further exam administration instructions.) You are responsible'for reproducing the number of exam copies required for'the number of indiwduals taking the exam.' Prior to the exam, store the original copies in a locked cabinet or safe and reproduce the necessary number of copies only on the day immediately preceding the exam; in this case, copies should be made on (date). Please note: your total number of copies should consist of one half Form A and one half Form B.L After making the necessary number of j copies, secure the original and all copies from view of unauthorized persons, storing them in a j locked cabinet or safe until the exam date. Each individual taking the exam must sign the security statement on the exam cover page. This page must be removed from the exam copy and mailed to (Name of c_q_Ang1gr) n r along with the answer sheets and administrator / proctor security agreements. After the exam has been given, the exam copies become public knowledge and no longer need 4 security. Exam copies, therefore, may be kept or disposed of as desired. Enclosure 3. Proctor instructions. The proctor instructions detail the guidelines for administering the exam. Please note that the specific instructions presented are designed to be

   \.      adhered to and followed identically by each proctor at all facilities. This process will ensure     I uniform administration and equity of results nationwide. As noted in the Proctor Instructions, all GFE exams will be administered at the same time in accordance with the local time zone in which the facility is located.                                                                     I En_ closure 4. Exam Answer Sheets. The appropriate number of answer sheets (extra copies ir:cluded) is enclosed for the number of applicants you identified to take the exam. All           1 applicants must use the original enclosed answer sheets for recording answers during the            l exam.

Summary of items to be Returned to (Name of contractor) The following items are to be mailed via Overnight Dellvery Service to {N._a_m._e of contractor) and postmarked no later than (date). completed answer sheets

                  . applicant-signed exam cover sheets                                                        ;

administrator / proctor-signed security statement (s) ). b

   \'                                                       9 of 13                  NUREG-1021, Revision 8 m

1 l E5-205 3 Attachment 2 i Mail all of the above exam-related materials addressed as follows: ) I Elil_[D_tl (Nag 19 of contractor) { Street address) . (City. State Zin codej l 1 For further questions regarding the specifics of this exam, please contact (Name) at (telephe_ne . l number). For questions regarding the GFE in general, please contact [N_3mej, NRC, at {lgjgohone numb _qt). For matters regarding candidate y/ithdrewals or cancellations, contact either[Name) or Name) i at (telechone number) for specific guidance. I (Name), Chief Operator Licensing, Human Performance, and Plant Support Branch Division of Inspection Program Management Office of Nuclear Reactor Regulation

Enclosures:

l As stated Distributio.n: w/o enclosures Director, DIPM Chief, Operator Licensing, Human Performance, and Plant Support Branch NRR GFE Coordinator Project Manager j Public NUREG-1021, Revision 8 10 of 13

ES-205 Sample Results Letter Attachment 3 ['] NRC Letterhead (Date) (Name. Title) (Facility name) (Street address) ~ (City. State Zio code] j l

Dear (Name):

l l (*) On (date), the NRC administered the generic fundamentals examination (GFE) section of the written operator licensing examination to employees of your facility. Enciosed with this , letter are copies of both forms of the examination, including answer keys, the grading results for ! your facility, and copies of the individual answer sheets for each of your employees. Please forward the results to the individuals along with the copies of their respective answer sheets. A )

         "P" in the column labeled RESULTS indicates that the individual achieved a passing grade of             j 80 percent or better on the GFE. Those individuals having an "F" in the RESULTS column failed the examination.

1 (") On (datej, the NRC administered the generic fundamentals examination (GFE) section l of the written operator licensing examination. j ( (") Your facility did not participate in this examination. However, a copy of the master i (BWR or PWR) examination, with the answer key, is enclosed for your information.

   -     if you have any questions concerning this examination, please contact {Name of the NRR GFE              )

(m) ' coordinator) at {p_ hone number). ! Sincerely, LAparoJLriate recional reorosentative) Docket No. 50-(Number) (*)

Enclosures:

1. Examination Form "A" and "B" with enswers

2. Examination Results Summary for (Facility Name)

3. Individual Answer Sheets

(**)

Enclosure:

As stated [ Paragraphs marked (*) apply only to those facility licensees that participated in the examination, while paragraphs marked (") apply only to those facility licensees that did not l participate in the examination.] l n ( )

   ~/                                                      11 of 13                  NUREG-1021, Revision 8 i
                                                                      ~                          _

n l j i i ES-205 GFE Test item Distributidii Attachment 4 m ! Pressurized Water Reactors No. of Wl j K/A Topic items i Group I Com_ conents I 191001 Valves 4 ! 191002 Sensors and Detectors 10 l 191003 Controllers and Positioners 5 l 191004 Pumps 7 { 191006 Heat Exchangers and Condensers 3 191008 Breakers, Relays, and Disconnects 7 l1

                                            ._                                      j Grouo il Components                                               l 191005        Motors and Generators                                   5 191007        Demineralizers and lon Exchangers                       3 Group I Reactor Theory 192004        Reactivity Coefficients                                 4 192005        Control Rods                                            4 192000        Reactor Operational Physics                             8 Grouo ll Rqacipr Theory 192003        Reactor Kinetics and Neutron Sources                    2 192006        Fission Product Poisons                                 6

( l Group Ill Reactor Theoty 192001 Neutrons 1 192002 Neutron Life Cycle 2 192007 Fuel Depletion and Bumable Poisons 1 Grouo l Thermody,namics l 193009 Core Thermaiumits 2 1 193010 Brittle Fracture and Vessel Therrnal Stress 5 l l Greuo il Thermodynamics 193003 Steam 2 193007 Heat Transfer 2 193008 Thermal Hydraulics 8 Groupu Ill Thermodynamics 193001 Thermodynamic Units and Properties 1 193004 Thermodynamic Processes 2

193005 Thermodynamic Cycles 1 193006 F.!uid Statics and Dynamics 5 t l Total items 100 NUREG-1021, Revision 8 12 of 13 i

t I l

_ES-205 2 Attachment 4 v Boiling Water Reactors No. of K/A Topic items Componenta 291001 Valves 5 l 291002 Sensors and Detectors 9 1 291003 Controllers and Positioners 3 j 291004 Pumps 8 l 291005 Motors and Generators 5 l 291006 Heat Exchangers and Condensers 6 ! 291007 Demineralizers and lon Exchangers 3 ; 291008 Breakers, Relays, and Disconnects 5 Grouo i Reactor Theorv , 292004 Reactivity Coefficients 2 l 292005 Control Rods 4 292008 Reactor Operational Physics 8 Group 11 Reactor Theorv l 292001 Neutrons 2 l 292002 Neutron Life Cycle 2 292003 Reactor Kinetics and Neutron Sources 3 292006 Fission Product Poisons 6 h %Y 292007 Fuel Depletion and Burnable Poisons 1 Group i Thermodynamics 293007 Hsat Transfer and Heat Exchangers 3 293009 Core Thermal Limits 7 Group ll Thermodynamics 293003 Steam 2 293004 Thermodynamic Processes 2 293008 Thermal Hydraulics 7 293010 Brittle Fracture and Vessel Thermal Stress .2 Group lli Thermodynamics 293001 Thermodynamic Units and Properties 1 293005 Thermodynamic Cycles 1 293006 Fluid Statics 3 Total items 100

   !:.                                                                        ---=

[ \- 13 of 13 NUREG-1021, Revision 8

    -                                                                                           ~~

ES-301 V , _ _, PREPARING INITIAL OPERATING TESTS A. PURPOSE . i All applicants for reactor operator (RO) and senior reactor operator (SRO) licenses at power j reactor facilities are required to take an operating test, unless it has been waived in accordance l with 10 CFR 55.47 (refer to ES-204). The spe:ife content of the operating test depends on the j type of license for which the applicant has apNieo. This standard describes the procedure %r developing operating tests that meet the ! requirements of 10 CFR 55.45, inclNing the use of reactor plant simulation facilities and the I conduct of multi--unit evaluatiom,. l l B. BACKGROUND To the extent applicable, the operating test will require the applicant to demonstrate an understanding of, and the ability to perform, the actions necessary to accomplish a i representative sampling frorn the 13 items identified in 10 CFR 55.45(a) (all 13 items do not I need to be sampled on every operating test). In addition, the content of the operating test will be identified, in part, from learning objectives contained in the facility licensee's training prograrn and from information in the final safety analysis report, system description manuals and operating procedures, the facility license and license amendments, licensee event reports, l g and other materials requested from the facility licensee by the Commission. j t i V The structure of the operating test is dictated, in part, by 10 CFR 55.45(b)(1). It states that the test will be administered in a plant walk-through and in either a simulation facility (as defined in 10 CFR 55.4) that the Commission has approved or a simulation facility consisting solely of a plant-referenced simulator (also as defined in 10 CFR 55.4) that has been certified to the Commission by the facility licensee. The walk-through portion of the operating test consists of two categories, each focusing on specific knowledge and abilities (K/As) required for licensed operators to safely discharge their assigned duties and responsibilities. A third category of the operating test is administered on an NRC-approved or facility-certified simulation facility. Unless specifically waived in accordance with ES-204 and documented on the Examination Assignment Sheet (Form ES-201, Attachment 4), all three categories must be completed for every license applicant. Each category of the operating test is briefly described below. Section D of this standard provides detailed instructions for developing each category. Procedures for administering and grading the operating test are contained in ES-302 and ES-303, respectively.

1. feaftegory A. " Administrative Tooics" This category of the operating test covers K/As that are generally associated with the administrative control of the plant. It implements items 9 through 12 of 10 CFR 55.45(a) and is divided into four administrative topics, as described below. The depth of coverage required in each topic is based on the applicant's license level. The O

1 of 26 NUREG-1021, Revision 8

ES-301 applicant's competence in each topic is evaluated by administering job performance measures (JFMs) or by asking specific questions. Topic A.1, " Conduct of Operations," evaluates the applicant's knowledge of the daily operation of the facility. The following subjects are examples of the types of information that should be evaluated under this topic: shift turnover shift staffing requirements temporary modifications ci procedures reactor plant startup requirements modo changes plant parameter verification (estimated critical position (ECP), heat balance, etc.) short-term information (e.g., night and standing orders) key control security (awareness and familiarity) fuel handling Topic A.2, " Equipment Control," addresses the administrative requirements associated with managing and controlling plant systems and equipment. The following subjects exemplify the types of information that should be evaluated under this topic: surveillance testing maintenance tagging and clearances temporary modification of systems familiarity with and use of piping and instrument drawings Topic A.3, " Radiation Control," evaluates the applicant's knowledge and abilities with respect to radiation hazards and protection (of plant personnel and the public). The following subjects exemplify the types of information that should be evaluated under this topic: ute and function of portable radiation and contamination survey instruments and personnel monitoring equipment knowledge of significant radiation hazards the ability to perform procedures to reduce excessive levels of radiation and to guard against personnel exposure radiation exposure limits and contamination control, including permissible levels in excess of those authorized radiation work permits control of radiation releases Topic A.4 " Emergency Plan," evaluates the applicant's knowledge of the emergency plan for the facility, including, as appropriate, the responsibility of the RO or SRO to l NUREG 1021, Revision 8 2 of 26 l l . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ES-301 { decide whether the plan should be executed and the duties assigned under the plan. The following subjects are examp5s of the types of information that should be evaluated under this topic: lines of authority during an emergency emergency action levels and classifications emergency facilities emergency communications emergency protective action recommendations Category A is administered in a one-on-one, walk-through format and the four topic areas are graded collectively (refer to ES-302 and ES-303).

2. Cateoorv B. " Control Room Systems and Facility Walk-throuah" This category of the operating test is used to determine if the applicant's knowledge in the area of plant system design is adequate and to determine if the applicant is able to safely operate those systems. As such, this category implements the requirements of items 3,4,7,8, and 9 identified in 10 CFR 55.45(a). It also encompasses several types of systems, including primary coolant, emergency cooiant, decay heat removal, auxiliary, radiation monitoring, and instrumentation and control.

f Category B is divided into two subcategories. The first and larger subcategory (B.1,

       " Control Room Systems") focuses on those systems with which licensed operators are most involved (i.e., those having controls and indications in the main control room). The second subcategory (B.2, " Facility Walk-Through") ensures that the applicant is familiar with the design and operation of systems located outside the main control room. The applicant's knowledge and abilities relative to each system are evaluated by administering JPMs and, when necessary, specific follow-up questions based on the applicant's performance of each JPM.

Subcategories B.1 and B.2 are administered in a one-on-one, walk-through format and are graded collectively (refer to ES-302 and ES-303).

3. Cateaorv C. "Intearated Plant Operations" This category of the operating test implements items 1 through 8 and 11 through 13 identified in 10 CFR 55.45(a). This is the most performance-based category of the operating test and is used to evaluate the applicant's ability to safely operate the plant's systems under dynamic, integrated conditions.

The simulator test is administered in a team format with up to three applicants (or surrogates) filling the RO and SRO license positions (as appropriate) on an operating l crew. (Refer to ES-201 for additional guidance on crew composition and ES-302 for test administration instructions.) This format enables the examiner to evaluate each applicant's ability to function within the control room team as appropriate to the assigned 3 of 26 NUREG-1021, Revision 8 l

ES-301 position, in such a way that the facility licensee's procedures are adhered to and that the limitations in its license and amendments are not violated (refer to 10 CFR55.45(a)(13)). Each team or crew of applicants is administered a set of scenarios designed so that the examiners can individually evaluate each applicant on a range of competencies applicable to the applicant's license level. Appendix D describes those comnetencies, and Forms ES-303-3 and ES-303-4, the " Integrated Plant Operations Competency Grading Worksheets' for ROs and SROs, break down each competency into a number of specific rating factors to be considered during the grading process (refer to ES-303). Each applicant must demonstrate proficiency on every competency applicable to his or her license level. The only exception is that SRO Competency Number 5, " Control Board Operations," is optional for SRO-upgrade applicants (i.e., SRO-upgrade applicants do not have to fill a position that requires control board operations; however, if they do rotate into such a position, they will be graded on this competency even though they may not be individually observed by an NRC examiner, as discussed in ES-302). C. RESPONSIBILITIES

1. Facility Licensee The facility licensee is responsible for the following activities, as applicable, depending upon the examination arrangements confirmed with the NRC regional office in accordance with ES-201 approximately four months before the scheduled examination date:

a. Prepare proposed examination outlines in accordance with Section D and submit them to the NRC regional office for review and approval in accordance with ES-201.

b. Submit the reference materials necessary for the NRC regional office to prepare and/or review the requested examination (s) (refer to ES-201, Attachment 2).

c. Prepare and review the final operating tests in accordance with the previously approved examination outi;ne(s) and the instructions in Sections D and E, and submit the tests to the NRC regional office in accordance with ES-201.

d. Make the simulation facility available, as necessary, for NRC examiners to prepare for the operating tests.

e. Meet with the NRC in the regional office or at the facility, when and as necessary, to review the proposed operating tests and discuss potential changes (refer to ES-201).

1 NUREG-1021, Revision 8 4 of 26 Ol. i

h l 1 [ ES-301

f. Revise the operating test outlines and the final tests as applicable and as agreed upon by the NRC regional office (refer to ES-201). The NRC retains final authority to approve the operating tests.

I

2. NRC Reaional Office i

! The NRC regional office is responsible for the following activities: i a. Ensure that the operating tests are developed in accordance with Section D.

b. Ensure that the operating tests are reviewed for quality in accordance with Section E.

c. Meet with the facility licensee, when and as appropriate, to prereview the operating tests in accordance with ES-201.

D. INSTRUCTIONS Prepare each category of the operating test in accordance with the following general guidelines and specific instructions:

1. General Guidelines

,r l[

a. In an effort to reduce examination preparation effort, the same operating test

'( may be used to examine multiple applicants and simulator crews. Depending on the number and license level of the applicants being examined, it might be possible to use the same set of JPMs and scenarios to examine all of the applicants if the operating test is administered in multiple segments (e.g., single scenarios or two-four JPMs) each of which can be given to all of the applicants in a single day. The facility licensee and the NRC chief examiner shall discuss the options and reach agreement on the process before developing the operating tests. To minimize predictability and maintain test integrity, varied subjects, systems, and operations shall be evaluated with applicants that are not being examined at the same time, unless measures are taken to preclude interaction among the applicants. The same JPMs and simulator scenarios shall not be repeated on successive days. Operati.ng tests written by the facility licensee may not duplicate test items (simulator scenarios or JPMs) from the applicants' audit test (or tests if the applicant is retaking the examination) given at or near the end of the license training class. Simulator events and JPMs that are similar to those that were

tested on the audit examination are permitted provided the actions required to mitigate the transient or complete the task (e.g., using an alternate path as discussed in Appendix C) are significantly different from those required during LA the audit examination. The facility licensee shall identify for the NRC chief

 \  ]

5 of 26 NUREG-1021, Revision 8 l\d

ES-301 examiner those simulator events and JPMs that are similar to those that were tested on the audit examination. Sufficient operating test materials shall be developed to ensure that all applicants can be tested with the available personnel according to the schedule agreed upon by the NRC regional office and the facility licensee (refer to ES-201).

b. To the extent permitted for each category of the operating test, select and modify testing materials (i.e., JPMs, questions, and simulator scenarios) from the facility's examination banks. Every selected test item must satisfy the qualitative and quantitative criteria specified for the applicable section of the operating test or be modified accordingly.

c. Consider the K/As associated with normal, abnormal, and emergency tasks and evolutions as a source of topics for use in evaluating applicant competency in each category of the operating test.

The knowledge and abilities associated with the tasks and questions planned for the operating test should have importance factors of at least 2.5. Tasks with importance factors of less than 2.5 may be used if there is a substantive reason for including them (e.g., a recent licensee event or a significant system modification). The K/As should be appropriate to the plant-specific requirements for the applicant's license level. Refer to the facility's job and task analysis (if available), learning objectives, and other reference material to confirm that the operating test is correctly oriented to the facility and the applicant's license level. The facility licensee's site-specific task list may be used to supplement or override, on a case-by-case basis, selected individual items in the NRC's K/A catalogs. In order to maintain examination consistency, the site-specific task list shall not be used in place of the entire K/A catalog.

d. When selecting and developing materials (JPMs, scenarios, and questions) for the operating test, ensure that the materials contribute to the test's overall capacity to differentiate between those applicants who are competent to safely operate the plant and those who are not. Additionally, all of the test items should ,

include the three facets of test validity (i.e., content, operational, and discrimination) discussed in Appendix A. Any test items that, when missed, would raise questions regarding adequate justification for denying the applicant's license should not be included on the operating test. l

e. SRO applicants, whether upgrade or instant, will be examined for the highest on-shift position for which the SRO's license is applicable (e.g., shift supervisor),

I regardless of the position to be assigned when licensed. SRO applicants should demonstrate their supervisory abilities and an attitude of responsibility for safe operation, and are expected to assume a management role during plant NUREG-1021, Revision 8 6 of 26 (

ES 301 transients and upset conditions while taking Category C of the operating test. The operating test briefing, discussed in Appendix E, ensures that the applicants are advised of this policy. Differences in administrative controls and facility design will affect the SRO's responsibilities, but, in general, the following guidelines should be used to differentiate the SRO operating test from that of an RO.

               ' in directing licensed activities, the SRO must evaluate plant performance and make operational judgments accordingly. SRO applicants should, therefore, be more knowledgeable in areas such as operating characteristics, reactor behavior, and instrument interpretation.

In directing licensed activities, the SRO must have a broader and more thorough knowledge of facility administrative controls and methods, including limitations imposed by the regulations and the facility's technical specifications and their bases. The SRO may be assigned responsibilities for auxiliary systems that are outside the control room (e.g., waste disposal and handling systems) and are not normally operated by licensed operators. Becauce the SRO may have these additional responsibilities, the SRO license applicant should demonstrate knowledge of the designs of such systems as they relate to O maximum permissible concentrations, effluent release rates, and other radiological considerations.

f. Incorporate facility-specific and industry-gensric operating experience into the operating test whenever possible. Documentation such as licensee event reports, significant event reports, and service information letters are readily available sources of operationally oriented plant anomalies.

Evaluate the dominant accident sequences (DASs) for the facility to determine if they are suitable for testing, on a sampling basis, during the dynamic simulator 1 or walk-through tests. DASs are those sequences that contribute significantly to the frequency of core damage as determined by the facility licensee's probabilistic risk assessment (PRA) or individual plant examination (IPE). The PRMPE should also be used to identify risk-important operator actions. Chapter 13, " Operational Perspectives," of NUREG-1560, " Individual Plant Examination Program: Perspectives on Reactor Safety and Plant Performance," identifies a number of important human actions that may be appropriate for evaluation on the operating test. In determining what actions to evaluate, do not overlook actions that are relied upon or result in specific events being driven to low risk contribution. This will help identify those human actions, assumed to be very reliable, that might otherwise not show up in a list of risk-dominant actions. I 7 of 26 NUREG-1021, Revision 8 l

ES-301

g. If the applicants at a facility qualify for dual or multi-unit licenses, the operating tests should evaluate their knowledge of the design, procedural, and operational differences between the units.

Divide the operating test coverage among the units and do not become predictable by conducting the walk-through tests on only one unit. Different applicants may be examined on different units, or each applicant may be asked to explain or demonstrate his or her understanding of variations in control board layouts, systems, instrumentation, and procedural actions between the units at the facility. Most dual- or multi-unit stations have a simulator that is modeled after only one of the units. Therefore, ensure that the applicants are properly tested on the different systems, control board layouts, and any other differences between the units during the walk-through portion of the operat:ng test. For example, after administering Category C of the operating test on Browns Ferry Unit 1, the control room systems portion of Category B of the operating test could be administered on Unit 2 or Unit 3 or both.

h. The operating test should examine a broad range of knowledge and abilities, systems and components, and operations and events. The three categories of the test should not be redundant, nor should they duplicate material that is covered on the written examination. It is particularly important that Categories B and C be developed and reviewed as a package to preclude the same tasks and events from appearing on both parts of the test.

i. Every'acet of the operating test, including the prescripted Category A questions and answers, the JPMs for Categories A and B, and the Category C simulator scenarios, should be planned, researched, validated, and documented to the ,

maximum extent possible before the test is administered.

j. Examiners who will be administering the operating tests but were not involved in their development are expected to research and study the topics and systems to be examined on the operating test so that they are prepared to ask whatever performance-based follow-up questions might be necessary to determine if the applicant is competent in those areas. As stated in 10 CFR 55.45(a), the operating test requires the applicant to demonstrate an understanding of and the I ability to perform the actions necessary to accomplish a representative sample i from among 13 items listed in the rule. If the applicant correctly performs a JPM j (including both critical and noncritical steps) and demonstrates familiarity with the l

equipment and procedures, it is not necessary to ask any follow-up questions. However, if the applicant fails to accomplish the task standard for the JPM or , demonstrates a lack of understanding regarding the equipment and procedures such as having difficulty locating information, control board indications, or controls, the examiner must be prepared to ask performance-based follow-up questions, as necessary, to c'arify or confirm the applicant's understanding of the system as it relates to the task that was performed. j NUREG-1021, Revision 8 8 of 26 1

ES-301 Examination team members are strongly encouraged to meet as a group with the chief examiner to review the examination materials after they have been approved for administration by the responsible supervisor. The discussions ! should focus on those test items that might require extensive cuing by the examiner and those that are unique to the facility and require a response different from what the examiner might expect based on past experience.

k. JPMs should include the elements identified in Appendix C (e.g., initiating and terminating cues, critical steps, and performance criteria). The guidelines and forms (or equivalents) in that appendix should be used when developing new JPMs. Facility procedures may be adapted for use as JPMs by identifying critical steps and entering comments on how to execute particular steps.

l. The prescripted questions for Category A and the performance-based follow-up questions for Category B may include a combination of open- and closed-reference items. Open-reference items that require applicants to apply their knowledge of the plant to postulated normal, abnormal, and emergency situations are preferred. Closed-reference items may be used to evaluate the immediate actions of emergency and other procedures, certain automatic actions, operating characteristics, interlocks, set points, and routine administrative activities, as appropriate to the facility.

Refer to Attachment 1 for more guidance regarding the development and use of O prescripted open reference questions for Category A of the walk-through test. To the extent possible, the concepts in the attachment should also be applied to performance-based follow-up questions.

m. If it becomes necessary to deviate from a test outline that has been approved by the NRC chief examiner in accordance with ES-201, discuss the proposed deviation with the chief examiner and obtain concurrence before proceeding with the changes. Be prepared to explain why the original proposal could not be implemented and why the proposed replacement is considered an acceptable substitute.

2. Specific Instructions for Cateaory A. " Administrative Tooics' Although the administrative topics may be examined separately, it is preferable, whenever possible, to link, associate, or integrate them with tasks and events condacted during Categories B and C. However, it is important to keep in mind that the applicant's proficiency in the administrative topics should be deliberately evaluated and not inferred solely from observations made during the simulator portion of the operating test.

a. For each of the administrative topics listed below, select the required number of subjects to be evaluated during the operating test. Section B.1 provides examples of the types of subjects that should be evaluated under each topic; the lists are not all-inclusive.

l O 9 of 2fi NUREG-1021, Revision 8 I

ES-301 Toole Number of Subiects A.1, " Conduct of Operations" 2 A.2, " Equipment Control' 1 A.3, " Radiation Control" 1 A.4, " Emergency Plan" 1 K/As associated with each administrative topic can be found in Section 2 of the applicable NRC K/A catalog (i.e., NUREG-1122 for PWRs and NUREG-1123 for BWRs). For Topic A.4, only those K/As related to the emergency plan and implementing procedures (not those associated with the emergency operating procedures (EOPs)) are applicable to this category of the operating test.

b. For each administrative subject, determine the best method for evaluating the applicant's knowledge or ability in that area. Although a performance-based evaluation, using a single administrative JPM is generally preferred, two prescripted questions may be used to conduct the evaluation in each specific subject area selected for evaluation. The questions may be associated with Category B JPMs or they may be administered separately.

c. In general, SROs have more administrative responsibilities than ROs, so SRO applicants should be evaluated in greater depth on the administrative topics. RO applicants need only understand the mechanics and intent of the related subjects, as they pertain to tasks at the facility.

d. The following specific guidelines should be applied when selecting or developing prescripted questions or JPMs to confirm the applicant's competence with regard to each topic:

Tooic A.1. " Conduct of Operations' Many of these subjects can be : overed within the framework of a shift turnover l or by integrating them ireto other discussions, as they apply, throughout the l examination. ! l The subject of fuel handling can be covered in the control room, but attempt to cover this subject in the fuel handling areas of the plant whenever possible. The RO applicant should be aware of his or her duties in the control room during fuel handling. These duties include monitoring instrumentation and responding to alarms from the fuel handling area, communicating with the fuel handling and storage facility, and operating systems from the control room in support of l (re) fueling operations. For the SRO applicant, evaluate topics such as core alterations, new and spent fuel storage and movement, the design of the fuel i handling area, use of the fuel handling tools, and fuel handling casualties. i The applicant's security awareness should be evaluated by observing his or her l behavior during the operating test. However, passive observations, in and of I l themselves, are insufficient to justify an evaluation in that subject area. It is NUREG-1021, Revision 8 10 of 26

h\J ES-301 appropriate to question an SRO applicant on applicable aspects of the facility's security plan and the operating crew's interactions with the security shift supervisor. Tooic A.2. "Eauipment Control" These subjects can be evaluated within the framework of a normal maintenance evolution. For example, ask the applicant to demonstrate how he or she would take a failed system or component out of service, initiate maintenance on the system, and test the system before placing it back in service. Tonic A.3. " Radiation Control" This topic is best covered in conjunction with the JPMs prepared for Category B.2 of the walk-through (i.e., local systems and operations), it is most appropri-ate to evaluate these subjects during the required entry into the radiologically controlled area (RCA). The levels of knowledge expected of RO and SRO applicants in some radiation control subjects are significantly different, The RO's duties generally require knowledge of radiation worker responsibilities and operation of plant systems associated with liquid and gaseous waste releases. Therefore, the depth to which RO applicants are evaluated should be limited to their responsibilities and c\ (b the monitoring requirements before, during, and after the release. The SRO, however, may be involved in reviewing and approving release permits and should be cognizant of the requirements associated with those releases, as well as their potential effect on the health and safety of the public. The SRO applicants may be asked to discuss or simulate (i.e., with a JPM) a planned release (e.g., liquid, gaseous, or containment purge) when examining these topics. Tooic A.4. "Emeraency Plan" There are significant differences between the knowledge required of RO and SRO applicants in this area. RO applicants should be familiar with the emergency plan and with their plant-specific responsibilities under the emergency plan implementing procedures (EPiPs). SRO applicants, however, must demonstrate additional knowledge based upon their responsibility to direct and manage the implementation of the EPIPs during the initial phases of an emergency. Because of this, SRO applicants should have a more detailed understanding of the EPIPs, in general, and be familiar with event classification procedures, protective action recommendations, and communication i requirements and methods. l This topic is best evaluated by integrating it into a discussion of a Category C transient that requires implementation of the emergency plan, or by conducting a JPM requiring use of the emergency plan. Such a JPM can be conducted ; i r \ E 11 of 26 NUREG-1021, Revision 8 I i

I I l I l {

l ES-301 immediately following a simulator scenario or during the walk-through (Category A or B) examination.

e. The planned administrative subjects should normally take no more than 1 hour and 1.5 hours to administer to RO and SRO applicants, respectively.

f. On Form ES-301-1, " Administrative Topics Outline," briefly describe the administrative subjects selected for evaluation and the method (s) by which each subject will be evaluated. The method of evaluation should include the title of any planned JPMs and a brief summary of the proposed questions.

g. Forward the completed outline to the NRC chief examiner so that it is received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed; the outline is normally due approximately 75 days before the scheduled examination date. Refer to ES-201 for additional instructions regarding the review and submittal of the examination outline.

The NRC chief examiner and responsible supervisor shall review the test outline coverage as soon as possible in accordance with ES-201 and forward any comments to the originator for resolution.

h. After the NRC chief examiner approves the operating test outline, prepare the final Category A test materials (i.e., the JPMs, prescripted questions, and answers) in accordance with the general operating test guidelines in Section D.1, the open-reference question guidelines in Appendix B, and the JPM guidelines in Appendix C.

i. When the materials are complete, review the quality of the final Category A walk-through test using Form ES-301-3, " Operating Test Quality Checklist." This review shall be performed in conjunction with the associated Category B walk-through and the dynamic simulator operating test as noted in Sections 0.3 and D.4.

Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner, as appropriate, for review and approval in accordance with Section E. The test must be received by the NRC chief examiner approximately 45 days before the scheduled administration date, unless other arrangements have been made.

3. Soecific Instructions for Cateaorv B. " Control Room Systems and Facility Walk-Throuah" This category of the operating test evaluates the applicant on systems-related K/As by having the applicant perform selected tasks and, when necessary, based on the applicant's performance, probing his or her knowledge of the task and its associated system with specific follow-up questions. The Category B tasks are in addition to and should be different from the events and evolutions conducted during Category C,

       " Integrated Plant Operations "

NUREG-1021, Revision 8 12 of 26 l

                                                                                             )
                                                                                             \

i ES-301

a. Refer to Section 1.9 of the K/A catalog applicable to the type of reactor for which the applicant is seeking a license (i.e., NUREG-1122 for PWRs and NUREG-1123 for BWRs). From the nine safety function groupings identified in the catalog, select the appropriate number of systems (see the table below) to be evaluated for each subcategory of the test based on the applicant's license level.

The emergency and abnormal plant evolutions (E/ APES) listed in Section 1.10 of the appropriate NUREG may also be used to waluate the applicable safety function (as specified for each E/ APE in the first tier of the written examination outlines attached to ES-401). License Level: Subcateoorv B.1 Subcateoorv B.2 Tg.tgl RO 7 3 10 SRO-instant (1) 7 3 10 SRO-upgrade (U) 2 or 3 3 or 2 5 The 10 systems and evolutions selected for RO and SRO-I applicants should evaluate at least 7 different safety functions. All of the systems and evolutions in each subcategory of the test should be selected from different safety function lists, and the same system or evolution should not de used to evaluate more than one safety function in each subcategory. For PWR operating tests, the primary and secondary systems listed under Safety Function 4, " Heat Removal J From Reactor Core,"in Section 1.9 of NUREG-1122 may be treated as separate ; safety functions; i.e., two systems, one primary and one secondary, may be selected from Safety Function 4. The S systems and evolutions selected for an SRO-U applicant should evaluate at least 5 different safety functions. One of the control room systems or evolutions (Subcategory B.1) must be an engineered safety feature, and the same system or evolution should not be used to evaluate more than one safety function. Keep in mind that the systems and evolutions selected for evaluation in Subcategories B.1 and B.2 must be oriented toward control room operations and local operations, respectively.

b. For each system selected for evaluation, select from the applicable K/A catalog or the facility licensee's site-specific task list one task for which a JPM exists or can be developed. Review the associated simulator outline if it has already been prepared (refer to Section D.4), and avoid those tasks that have already been selected for evaluation on the dynamic simulator test.

The JPMs should, individually and as a group, have meaningful performance requirements that will provide a legitimate basis for evaluating the applicant's understanding of and ability to safely operate the associated systems and the plant (as required by 10 CFR 55.45). O () 13 of 26 NUREG-1021, Revision 8

ES-301 in order to protect the integrity and security of the examination process, no more than 80 percent of any applicant's walk-through test (i.e,8 out of 10 or 4 out of 5 JPMs, as applicable) may be taken directly from the facility's testing materials without significant modification. A significant modification means that at least one condition has been substantively changed in a manner that alters the course of action of the JPM. Additionally, no more than 30 percent of the walk-through test may be repeated from the last NRC licensing examination at the facility. At least one of the tasks shall be related to a shutdown or low-power condition, and 40 percent of the tasks (i.e.,4/10 for ROs and instant SROs and 2/5 for upgrade SROs) shall require the applicant to execute alternate paths within the facility's operating procedures. In addition, at least one of the tasks conducted in the plant (i.e., Subcategory B.2) shall evaluate the applicant's ability to implement actions required during an emergency or abnormal condition, and another shall require the applicant to enter the RCA. This provides an excellent opportunity for the applicant to d?scuss or demonstrate the radiation control subjects described in Administrative Topic A.3. If it is not possible to develop or locate a suitable task /JPM for each of the selected systems, return to Step (a), above, and select a different system or evolution. After identifying a JPM for each system, list it and its associated safety function number on Form ES-301-2, " Control Room Systems and Facility Walk-Through Test Outline." Also indicate the type of JPM by entering the applicable code (s) identified at the bottom of the form.

c. Forward the completed walk-through test outline to the NRC chief examiner so that it is received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed; the outlines are normally due approxinsately 75 days before the scheduled examination date. Refer to ES-201 for additional instructions regarding the review and submittal of examination outlines.

The NRC chief examiner and responsible supervisor shall review the test outline in accordance with ES-201 and forward any comments to the originator for resolution.

d. After the NRC chief examiner approves the operating test outline, prepare the final Category B JPMs in accordance with the general guidance in Section D.1 and the JPM guidelines in Appendix C.

e. When the materials are complete, review the completed wa!k through test for quality using Form ES-301-3, " Operating Test Quality Checklist," and make any changes that might be necessary. To minirnize duplication, this review shall be !

performed in conjunction with the associated administrative topics and the i simulator operating test (refer to Sections D.2 and D.4). l Submit the entire operating test package to the designated facility reviewer or i the NRC chief examiner, as appropriate, for review and approval in accordance i NUREG-1021, Revision 8 14 of 26 l

r ES-301 with Section E. The test must be received by the NRC chief examiner approximately 45 days before the scheduled review date, unless other arrangements have been made.

4. Specific Instructions for Cateaory C. "Intearated Plant Ooerations"

a. Based on the anticipated crew compositions, determine the number of scenarios and scenario sets necessary to rotate each RO and SRO-l applicant into the lead reactor operator position so that he or she can perform a direct reactivity l manipulation. For e . ample, a crew consisting of two ROs and one SRO l will l normally require three scenarios to evaluate each applicant's performance on the reactor controls; however, a surrogate SRO will have to fill the supervisory role while the SRO-l applicant is in the lead operator position. Similarly, the crews 1 and scenarios will have to be planned so that every SRO applicant (U and l} fills the supervisory role for at least one scenario.

SRO-U applicants are given credit for their previous RO license evaluation and experience and are normally not required to manipulate the controls. It may be possible to significantly reduce the number of simulator scenario sets required to examine a large group of applicants by administering the same set of scenarios on the same day to two (or more) different crews of applicants. However, provisions must be made to ensure that the crews remain out of contact until all crews have completed the set of scenarios (refer to ES-302). Additional or replacement scenarios should also be prepared and available while administering the operating tests in accordance with ES-302 in case one of the planned scenarios does not work as intended.

b. The simulator operating tests (i.e., scenario sets) will be constructed by selecting i and modifying scenarios from existing facility licensee or NRC scenario banks and by developing new scenarios.

In order to maintain test integrity, every applicant shall be tested on at least one i new or significanty inodified scenario that he or she has not had the opportunity to rehearse or practice. A significant modification means that at least one condition or event has been substantively changed to alter the course of action in the scenario. Furthermore, any other scenarios that are extracted from the facility licensee's bank must be altered to the degree necessary to prevent the applicants from immediately recognizing the scenarios based on the initial conditions or other cues.

c. The initial conditions, normal operations, malfunctions, and major transients should be varied among the scenarios and should include startup, low-power, and full-power situations. Review the associated walk-through outline if it has already been prepared (refer to Section D.3), and take care not to duplicate operations that will be tested during the walk-through portion of the operating q test.

15 of 26 NUREG-1021, Revision 8

1 l l l ES-301 j

d. In order to maximize the quality and consistency of the operating tests, develop new scenarios in accordance with the instructions in Appendix D. Modify existing i scenarios, as necessary, to make them conform with the qualitative and i quantitative attributes described in that appendix and enumerated on Form ES- l 301-4, " Simulator Scenario Quality Checklist." The quantitative attribute target l ranges that are specified on the form are not absolute limitations; some j scenarios may be an excellent evaluation tool but may not fit within the ranges. j A scenario that does not fit into these ranges shall be evaluated to enecre that j the level of difficulty is appropriate. Whenever possible, the critical tasks should be distributed so that each applicant is required to respond.

Each scenario set must, at a minimum, require each applicant to respond to the , types of evolutions, failures, and transients in the quantities identified for the I applicant's license level on Form ES-301-5, " Transient and Event Checklist." An i applicant should only be given credit for those events that require the applicant to perform verifiable actions that provide insight to the applicant's competence. The required instrument and component failures should normally be completed before starting the major transient; those that are initiated after the major transient should be carefully reviewed because they may require little applicant ; action and provide little insight regarding their performance. Each event should only be counted once per applicant; for example, a power change can be counted as a normal evolution OR as a reactivity manipulation, and, similarly, a component failure that immediately results in a major transient counts as one or the other, but not both. Any normal evolution, component failure, or abnormal event (other than a reactor trip or other automatic power reduction) that requires the operator to perform a controlled power or reactivity change will satisfy the requirement for a reactivity manipulation. This includes events such as an emergency boration, a dropped rod recovery, a significant rod bank realignment, or a manual reactor power reduction in response to a secondary system upset. Such events may produce a more timely operator and plant response than a normal power change. Furthermore, each scenario set must also allow the examiner to evaluate the applicant's performance on each competency and rating factor germane to the applicant's license level. Use Form ES-301-6, " Competencies Checklist," to verify that the competencies are adequately evaluated by entering the scenario and event numbers that are intended to assess each competency. If the facility licensee normally operates with and is required by its technical specifications to have more than two ROs in the control room, the chief examiner may authorize the use of additional surrogates to fill out the crews. In such cases, take care in planning the scenarios to ensure that the additional operators do not reduce the examiners' ability to evaluate each applicant on the required number of events and on every competency and rating factor. NUREG-1021, Revision 8 16 of 26

ES-301 J Appendix D provides detailed instructions for completing Form ES-D-1, the

                 " Scenario Outline," and Form ES-D-2, the expected " Operator Actions," that examiners will use to administer the simulator operating tests. In order to minim!ze the amount of rework that might be required as a result of changes in the planned scenario events, Form ES-D-2 should be completed after the NRC chief examiner has had the opportunity to review and comment on the proposed simulator operating test outlines (i.e., Form ES-D-1) in accordance with ES-201.

I

e. When the proposed simulator operating test outlines are complete, forward them I to the NRC chief examiner so they are received by the date agreed upon with the {

NRC regional office at the time the examination arrangements were confirmed; j i the outlines are normally due approximately 75 days before the scheduled examination date. Refer to ES-201 for additionalinstructions regarding the review and submittal of the examination outlines. l The NRC chief examiner shall review the operating test outlines in accordance with ES-201, and forward any comments to the originator for resolution.

f. After the NRC chief examiner approve's the operating test outlines, prepare the final simulator test materials by revising Form (s) ES-D-1 as requested by the NRC chief examiner and completing a detailed operator action form (ES-D-2) for each event. All substantive operator actions (e.g., opening, closing, and throttling valves; starting and stopping equipment; raising and lowering level, flow, and pressure; making decisions and giving directions; not acknowledging alarms or verifying automatic actions) shall be documented, and critical tasks shall be identified. Events that do not require an operator to take one or more substantive actions will not count toward the minimum number of events required for each operator per Form ES-301-5.

g. Review the completed simulator operating test for quality using Form ES-301-4,

                  " Simulator Scenario Quality Checklist," and make any changes that might be necessary. This review shall be performed in conjunction with the associated walk-through test (refer to Sections D.2 and D.3) to minimize duplication.

Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner, as appropriate, for review and approval in accordance with Section E. The test must be received by the NRC chief examiner - approximately 45 days before the scheduled administration date, unless other arrangements have been made. E. - QUALITY REVIEWS

1. Facility Manaaement Review if the operating test was prepared by the facility licensee, the preliminary outline and the proposed test shall be independently reviewed by a supervisor or manager before they A are submitted to the NRC regional office for review and approvalin accordance with ES-17 of 26 NUREG-1021, Revision 8

ES-301 201. The reviewer should evaluate the outline and test using the crite ia on Forms ES-201-2, ES-301-3, and ES-301-4 and include the signed forms (for each different operating test) in the examination package submitted to the NRC in accordance with ES-201.

2. NRC Examiner Review

a. The NRC chief examiner shall independently review each operating test for content, wording, operational validity, and level of difficulty. As a minimum, the chief examiner .chall check the items listed on Forms ES-301-3 and ES-301-4, as applicable. The examiner should keep in mind that counting the number of scenario quantitative attributes is not always indicative of the scenario's level of difficulty Although there are no definitive minimum or maximum attribute values that can be used to identify scenarios that will not discriminate because they are too easy or difficult, scenarios that fall outside the target ranges specified on Form ES-301-4 should be carefully evaluated to ensure they are appropriate, if the chief examiner wrote the operating test, another NRC examiner shall perform the independent review.

b. The chief examiner should review the operating tests as soon as possible after receipt so that supervisory approval can be obtained before the final review with the facility licensee, which is normally scheduled about two weeks before the administration date. It is especially important that the chief examiner promptly review tests prepared by a facility licensee because of the extra time that may be required if extensive changes are necessary. The chief examiner shall consolidate the comments from other regional reviewers and submit one set of comments to the author,

c. If the facility licensee developed the operating test, then the facility licensee is primarily responsible for technical accuracy and compliance with the restrictions concerning the use of examination banks. However, the chief examiner is expected to use his or her best judgment and take reasonable measures, including selective review of reference materials and past tests, to verify these items. j

d. The chief examiner will note any changes that need to be made and forward the tests to the responsible supervisor for review and comment in accordance with Section E.3 before reviewing the examinations with the author or facility contact.

There are no minimum or maximum limits on the number or scope of changes the chief examiner may direct the author or facility contact to make to the proposed tests, provided that they are necessary to make the tests conform with established acceptance criteria. Refer to ES-201 for additional guidance l regarding NRC response to facility-developed examinations that are significantly l deficient. . l l NUREG-1021, Revision 8 18 of 26

, 1 l

 /O                                                                                                 ES-301 b           e. Upon supervisory approval, and generally at least 14 days before the operating tests are scheduled to be given, the chief examiner will review the tests with the facility licensee in accordance with ES-201.

Tests that were developed by the NRC should be clean, properly formatted and l

                    " ready-to-give" before they are reviewed with the facility licensee. The region
                   . should not rely on the facility licensee to ensure that the tests are of acceptable
                   ' quality to administer,

f. After reviewing the' tests with the facility 1:censee, the chief examiner will ensure that any cornments and recommendations are resolved and the tests are revised as necessary. - if the facility licensee developed the tests, it will generally be expected to make whatever changes are recommended by the NRC. -j i

g. After the necessary changes have been made and the chief examiner is satisfied i with the test, he or she will sign Form (s) ES-301-3 and forward the test package !

to the responsible supervisor for final approval.

3. ' NRC Supervisorv Review

a. Per ES-201, the responsible supervisor shall review the operating tests before authorizing the chief examiner to proceed with the facility prereview. The g supervisory review is not intended to be another detailed review, but rather a general assessment of test quality, including a review of the changes

( recommended by the chief examiner, and a check to ensure that all of the x 1 applicable administrative requirements have been implemented. l l

b. The responsible supervisor should ensure that any significant deficiencies in the ;

original operating tests submitted by a facility licensee are evaluated in I accordance with ES-201 to determine the appropriate course of action. At a l minimum, the supervisor should ensure that they are addressed in the final examination report in accordance with ES-501.

c. Following the facility review, the responsible supervisor should again review the tests to ensure that the concerns expressed by the facility licensee and the chief examiner have been appropriately addressed. The supervisor shall not sign Form (s) ES-301-3 until he or she is satisfied that the examination is acceptable to be administered.

F. ATTACHMENTS / FORMS

    ' Attachment 1,            "Open-Reference Question Guidelines" Form ES-301-1,            " Administrative Topics Outline" Form ES-301-2,            " Control Room Systems and Facility Walk-Through Test Outline" Form ES-301-3,            " Operating Test Quality Checklist" Form ES-301-4,            " Simulator Scenario Quality Checklist" Form ES-301-5,            " Transient and Event Checklist" Form ES-301-6,            " Competencies Checklist" 19 of 26                   NUREG-1021, Revision 8

ES-301 _ Open Reference Question Guidelines Attachment 1

1. The most appropriate format is the short-answer question, which requires the applicant to compose a response rather than select from among a set of alternative responses, as is the case with multiple-choice, matching, and true/ false questions.

2. Provide clear, explicit directions / guidelines for answering the question so that the applicant understands what constitutes a fully correct response. Choose words carefully to ensure that the stipulations and requirements of the question are appropriately conveyed. Words such as " evaluate," " outline," and " explain," can invite a lot of detail that is not necessarily relevant.

3. Make sure that the expected response matches (cnd is limited to) the requirements posed in the question. Consider the amount of partial credit to be granted for an incomplete answer. For questions requiring computation, specify the degree of precision expected. Try to make the answer turn out to be whole numbers.

4. Avoid giving away part or all of the answer by the way the question is worded. For example: "If the letdown line became obstructed, could boration of the plant be accomplished shortly after a reactor trip to put the plant in cold shutdown? If so, how?"

A test-wise applicant can realize that the answer has to be yes, or else the second part of the question would have read something like "If not, why not?"

5. Avoid what could be considered ' trick" questions in which the expected answer does not precisely match the question. For example, asking "How [do] the Si termination criteria change following an Si reinitiation?" implies that the termination criteria will change, when in actuality they do not.

6. Do not use direct look-up questions that only require the applicant to recall where to find the answer to the question. The operational orientation required of questions on the walk-through test and the applicant's access to reference documents, argue against the use of questions that test for recall and memorization. Any questions that do not require any analysis, synthesis, or application of information by the applicant should be answerable without the aid of reference materials. Refer to ES-602, Attachment i, for a more detailed discussion of direct look-up questions.

7. Questions should also adhere to the generic item construction principles and guidelines in Appendix B. Moreover, Form ES-602-1, "NRC Checklist for Open-Reference Test items," contains a list of questions that can be used to evaluate the suitability of the questions for the walk-through portion of the operating test. Although the checklist was developed for use in evaluating requalification written examinations, all of the criteria except 9,10,11, and the K/A rating on item 7 are generically applicable.

NUREG-1021, Revision 8 20 of 26

l , O ES-301 Administrative Topics Outline Form ES-301-1

V i

Facility: Date of Examination: ! Examination Level (circle one): RO/SRO Operating Test Number: l l l Administrative Describe method of evaluation: Topic / Subject 1. ONE Administrative JPM, OR Description 2. TWO Administrative Questions ! A.1 l l , ~A.2 i t A.3 i i A.4 O 21 of 26 NUREG-1021, Revision 8 i'

r- 1 ES-301 Control Room Systems and Facility Walk-Through Test Outline Form ES-301-2 i i l Facility: Date of Examination: l Exam Level (circle one): RO / SRO(l) / SRO(U) Operating Test No.: j B.1 Control Room Systems { l System / JPM Title Type Safety l Code' Function a. b. l l c. d. e.

f. l 9-B.2 Facility Walk-Through l

a. l l

b. c.

Type Codes: (D)irect from bank, (M)odified from bank, (N)ew, (A)lternate path, (C)ontrol i room, (S)imulator, (L)ow-Power, (R)CA NUREG-1021, Revision 8 22 of 26 l

l

[in ES-301 _Ogggi_np Test Quality Checklist , Form ES-301-3

    \                                                                      =

su.=::w---~w - Operating Test Nurnber- I _ Facility

_

Datg Examinatiort , inifais

1. GENERAL CRITERIA J a b c

a. The coerating test conforms with the previoidy approved outline; changes are corwistent with , I sa nphng requirements (e.g.,10 CFR 55.45, opetatk>nM importarvte, saf eN function distnbutiony

b. There is no day-to-day repetition between this and otner operating tests to be administered during this exarnination.

c. The operating test shisli not duplicate items from the apolicants' audit test (sgs,ee Section D.1 e).

d. Overtap with the written examination and between operating test categorias is within acceptable i limrts. __ )

e. It appears Wat the operating test will differentiate be' ween competent and less-than-competent l applicants at the designated license level. ,

2. WALK THROUGH (CATEGOR( A & B) CRITERtA - - -

a. Each JPM includes the fonowing, an applicatWe:

inidal conditions initiating cues references ancf tools, including associatd procedures validated tirne limits (average time allowed for completion) and specific designation if downed l to be time entical by the facility licertsee specific performance criteria that inc ude

                                 - detailed expucted actions with exact critena tuid nomenclature

[p\ - system resporise and other examiner cues t

   \
       ]                         - statements describing important observations to be ma.fe by the applicant
                                 - cr+teria for successful completion of the task d

identification of cntical steps and their associater1 performance standards 1 restrictions on the sequence of steps,if aophcable

b. The prescripted quastions in Category A are predominantly open reference and meet the critena in Attachment 1 of ES-301.

c. Repetition from operating tests used during the previous licensing examirnNon is within acceptable timits (30% for the walk through) and do not comprorree test intognty.

d. At least 20 percent of the JPMs on each test are new or sponNicantly modpi

3. SIMULATOR (CATEGORY C) CRITERIA -

a. The associated simulator operating tests (scenarb sets) have been reviewed in accordance with Form ES 3014 and a copy is attactes. -

y Pnnted Name / Signature Date

a. Author

b. Facility Reviewer (*)

c. NRC Chief Examiner (*) __

d. NRC Supervisor (*)

(*) The facility signature is not appucable for NRC developed tests: two independent NRC reviews are required. I U 23 of 26 NUREG-1021, Revision 8

ES-301 Simulator Scenario Quality Checklist Form ES-301-4 m _, Facility- Date of Exam: Scenario Numbers- / / Operating Test No.: QUALITATIVE ATTRIBUTES Initials a b c __

1. Tte initial conditions are realistic, in that some equiprnent and/or instrumentation may be out of servios, but it does not cue the operators into expected events.

2. The scenarios consist mostfy of related events.

3. Each event description consists of a the point in the scenano when it is to be irutiated the malfunction (s) that are entered to initiate the event the symptoms / cues that will be visible to the crew the expected operator actions (by shift position) the event termination point (if applicable)

4. No more than one rm mechanistic failure (e.g., pipe break) is incorporated into the scenario without a credible preceding incident such as a seismic event.

6. The events are valid vMh rgard to physics and thermodynamics.

tt Sequencing and timing of c<ents is reasonable, and allows the examination team to obtain complete evaluation resutts commensurate with the scenano nbiectives.

7. If time compression techniques are used, the scenario summary clearty so indicates. Operators have suffident time to carry out expected activities without undue time constraints. Cues are given.

8 The simulator rnodeling is not altered.

9. The scenarios have been validated. Any open simulator performance deficiencies have been evaluated to ensure that functional fidelity is mentained v!Nie running the planned scenanos.

10. Every operator will be evaluated using at least one new or sigrwficantly modified scenano. AM other scenarios have been aftered in accordance with Section 0.4 of ES-301.

11. Allindividual operator competencies can be evaluated, as venfied using Form ES 3016 (subrnit the form along with the simulator scenarios).

( 11 Each applicant will be significardy invotveo in the minimum nurnber of transients and events l Specified on Form ES 3015 (submit tne form with the simulator scenaMos) g The level of difficulty is aporopnate to support licensing decisions for each crew position. TARGET QU ANTITATIVE ATTRIBUTES (PEFl SCENARIO; SEE SECTION D.4.D) Actual Attributes - - -

1. Total malfunctions (5-0) / /

2. Ma? functions after EOP entry (1-2) / /

3. Abnormal events (2-4) / /

4. Major transients (12) / /

5 EOPs entered /reqeiring substantive actions (12) / / j l

6. EOP contingercies requiring substantive actions (0-2) / /

7. Cntical tasks (2-3) / /

es g

l 1

}

ES-301 _, Transient and Event Checklist , Form ES-301-5 l OPERATING TEST NO.: l A Scenario Number ignt Evgtjon Fgum 1 2 3 4 i Reactivity l 1 Normal 1 RO instrument 2 i Component 2 Major 1 Reactivity 1 Normal 0 As RO Instrument 1 Component 1 Major 1 SROI ( I Reactivity 0 LJ Normal 1 As SRO i instrument 1 j Component 1 Major 1 Reactivity 0 Normal 1 l SRO-U Instrument 1 l Component 1 Major 1 Enter the operating test number and Form ES-D-1 event numbers for ! Instructions: (1) each evolution, type. l (2) Reactivi manipulations may be conducted under normal or controlled I abnorm conditions (refer to Section D.4.d) but must be significant per j Section C.2.a of Appendix D. l Author: Chief Examiner: (-m# 3 25 of 26 NUREG-1021, Revision 8

ES-301 Competencies Checklist Form ES-30'l-6 Applicant #1 f Applicant #2 Applicant #3 RO/SRO-l/SRO-U RO/SRO_-l/SRO-U RO/SRO-1/SRO-U Competencies _ SCENARIO SCENARIO SCENARIO 1 2 3 4 1 2, 3 4 1 2 3 4 Understand and interpret Annunciators and Alarms Diagnose Events and Conditions Understand Plant l and System Response Comply With and Use Procedures (1) Operate Control Boards (2) Communicate and Interact With the Crew Demonstrate Supervisory Ability (3) Comply With and Use Tech. Specs. (3) Notes- 1 l (1) includes Technical Specification compliance for an RO. (2) Optional for an SRO-U. (3) Only applicable to SROs. ; 1 instructions: Circle the applicant's license type and enter one or more event numbers that will allow the examiners to evaluate every applicable competency for every applicant. Author ) Chief Examiner: i NUREG-1021, Revision 8 26 of 26 9i' l l 1 l

r j l eS-302 (j ADMINISTERING OPERATING TESTS TO INITIAL LICENSE APPLICANTS A. PURPOSE 1 This standard describes how to administer operating tests to initial license applicants in , accordance with the requirements of 10 CFR 55.45. It inc!udes policies and guidelines for i administering both the walk through and the integrated plant operations categories of the operating test. It is assumed that the operating test was prepared in accordance with ES-301. I B. BACKGROUND As noted in ES-201, faci!ity licensees w01 generally prepare proposed operating tests in accordance with ES-301 and submit them to the responsible NRC regional office for review and approval. Regardless of whether it was prepared by the facility licensee or the NRC, every operating test will be independently administered and graded by an NRC licensing examiner in accordance with the instructions contained herein and in ES-303. C. RESPONSIBILITIES

1. Facility Licensee .

The facility licensee is responsible for the following activities:

a. Make the plant and simulation facility available, as necessary, for validating and y/ administering, Category A (administrative topics), Category B (control room and in-plant systems), and Category C (integrated plant operations) of the operating tests.

b. Safeguard the integrity and security of the operating tests in accordance with facility procedures established pursuant to 10 CFR 55.40(b)(2) and the guidelines discussed in Attachment 1 of ES-201.

c. Provide administrative and logistics support (e.g., personnel to operate the simulation facility, surrogate operators, etc.) to facilitate the administration of the operating tests in accordance with Section D.

d. Inform the NRC regional office in writing if an applicant withdraws from the examination process before it is complete.

2. NRC Reaional Office The NRC regional office is responsible for the following activities:

a. Work with the facility contact to coordinate the operating test administration schedule in a manner that maximizes efficiency and maintains security.

Normally, the operating tests should be administered within 30 days before or after the written examinations. The region shall obtain concurrence from the n

  !    )
     -                                                    1 of 10                       NUREG-1021, Revision 8

ES-302 NRR operator licensing program office if the examination dates diverge by more than 30 days. (Refer to ES-201 for additional guidance regarding examinations y that have to be rescheduled to achieve an acceptable product.) f b. Administer the operating tests in accordance with Section D. D. TEST ADMINISTRATION INSTRUCTIONS AND POLICIES

1. Q,eneral s

a. Before beginning the operating test, an examiner shall brief the applicant (s) using Parts A, C, D, and E of Appendix E. To save time, it is recommended that the examiner (s) brief the applicants as a group. ,

I

b. If an applicant requests to withdraw during any part of the examination process, {

the examiner shall inform the applicant that this will result in automatic license denial and that he or she may reapply in accordance with 10 CFR 55.35. The chief examiner will request the facility licensee to document th; '.4pplicant's withdrawal in a letter to the NRC regional administrator.

c. Each applicant listed on the examination assignment sheet (see ES-201, Attachment 4) shall be administered an operating test as indicated under

            " Examination Type."

d. For purposes of test integration and continuity, the chief examiner should generally schedule the same examiner to administer all three operating test categories to an applicant. However, under certain circumstances, such as when a licensee's simulation facility is not located near the plant or if a licensee requests examinations for an unusually large group of applicants, the {

responsible regional supentisor may authorize the chief examiner to divide the ! operating test categories and subcategories among different examiners (simulator operating tests consisting of multiple scenarios shall not be divided i among examiners). The chief examiner will be responsible for ensuring that f each applicant gets a complete operating test and that the tests are thoroughly j and accurately documented. I i Normally, an NRC examiner will be assigned to individually evaluate each ) applicant during the simulator operating test. However, if a three-person i operating crew consists entirely of senior reactor operator (SRO) upgrade l applicants (who do not have to be evaluated on the control boards), the chief examiner may assign only two examiners to observe the crew. Although the applicants in the reactor operator and balance of plant positions may not be i individually evaluatod, they will be held accountable for any errors that occur as a i result of their action (s) or inaction (s) and graded on their ability to " Operate the j Control Boards" (i.e., SRO Competency 5). SRO instant applicants will always ! be individually evaluated by an NRC examiner regardless what operating position they are filling during a given scenario. I NUREG-1021, Revision 8 2 of 10 I ______-______--__-_j

ES-302

e. The examiner is expected to administer the planned operating test in accordance with the prepared and approved walk-through test outlines (Forrns ES-301-1,

           " Administrative Topics Outline," and ES-301-2, " Control Room Systems and Facility Walk-Through Test Outline") and simulator scenarios (Forms ES-D-1,

Scenario Outline," and ES D 2, " Operator Actions"). Examiners shall document '

every significant aspect of each applicant's performance for later evaluation, but thay shall not use the applicant's unplanned actions and statements to displace any part of the planned operating test.' Normally, examiners should substitute or replace planned operating test materials only if it is determined that an item is invalid or impc'.sible to perform or simulate because of unanticipated access restrictions or equipment failures.

f. Examiners may administer the same operating test (walk through and simulator) to consecutive applicants and crews on the same day, but they must ensure that J the security of the operating test is maintained. The same simulator scenarios shall not be repeated during successive days.

If previously agreed upon by the facility licensee, examiners may also administer the same operating test (walk through and simulator) by dividing the test into segments that can be administered to all of the applicants on the same day. This will minimize the amount of effort required to develop different operating tests but will complicate the scheduling process.

g. . The examiner should normally administer Categories B and C of the operating test first and attempt to concurrently evaluate as many of the planned administrative subjects in Category A as possible. The remaining administrative I subjects should then be evaluated in accordance with the approved outhne,

h. The examiner must take sufficient notes to facilitate the thorough docurnentation of any and all applicant deficiencies in accordance with ES-303. The examiner must be able to cross-reference each comment to a specific JPM, simulator event, or question.

1. The making of videotapes during the administration of operating tests is not authorized.

j. The number of persons present during an operating test should be limited to ensure the integrity of the test and to minimize distractions to the applicants.

Except for the simulation facility operators, no other member of the facility's staff shall be allowed to observe an operating test without the chief examiner's permission. Facility management and other personnel deerned necessary by the facility licensee should generally be allowed access to the examination (under security agreements, as appropriate), provided the simulation facility can accornmodate them and there is no impact on the applicants. (' 1 3 of 10 NUREG 1021, Revision 8 1

ES-302 Although the simulation facility operator will normally assume the role of the other personnel that the applicants direct or notify regarding plant operations, the chief examiner may permit other members of the facility training or operations staff (e.g., a shift technical advisor (STA)) to augment the operating shift team if necessary. The chief examiner shall fully brief those individuals regarding their responsibilities, reporting requirements, duties, and level of participation before the operating test begins. The examiners must not restrict the surrogate operators' activities to such an extent that the applicants being evaMated are required to assume responsibilitie.s beyond the scope of their position. The surrogate operators will be expected to assurne the it.!! responsibilities of the roles they take in the operating test. Consultations with an STA shall be conducted in accordance with the fac;lity licensee's j normal control room practice; e.g., an STA shall not be stationed in the simulator if they are on-call at the site. If the facility licensee normally operates with and is required by its technical specifications to have more than two reactor operators (ROs) in the control room, the chief examiner may authorize the use of additional surrogates to fill out the crews. In such cases, examiners must take care that the presence of additional operators does not dilute the examiners' ability to evaluate each applicant during the required number of events 1 and on every applicable competency and rating factor. Examiners shall not hesitate to run additional scenarios, as necessary, to ensure that every applicant is given the opportunity to demonstrate his or her competence. Only one individual (applicant or surrogate) is allowed to fill a shift supervisor or manager position during the simulator operating test. Under no circumstances will another applicant be allowed to witness an operating test. Operating tests are not to be used as training vehicles for future applicants. Other examiners may observe an operating test as part of their training or to audit the performance of the examiner administering the operating test. The chief examiner may permit other NRC employees, such as resident inspectors, regional personnel, researchers, or NRC supervisors, to observe an operating test. Personnel who are not NRC employees (e.g., representatives from the Institute of Nuclear Power Operations (INPO)) may observe the operating tests with prior approval from the NRR operator licensing program office. The chief examiner will control the observers activities in accordance with guidance provided by NRR. The examiner should also give the applicant the opportunity to object to the presence of observers.

k. The chief examiner should confirm with the facility licensee that the simulator instructor's station, programmers' tools, and external interconnections do not NUREG-1021, Revision 8 4 of 10 ,

I

FL l ES-302 compromise operating test security while conducting examinations (refer to Section F of Appendix D). The primary objective is to ensure that the exam l material cannot be read or recorded at other unsecured consoles and that l examination material is either physically secured or electronically protected when l not in use by individuals listed on the security agreement.

                          ~
                                                                                                 )

i j i

l. The chief examiner should arrange for any NRC examiners who are not familiar with the facility to obtain a tour before they administer any operating tests. .The tours shall not be conducted or observed by any of the applicants. In addition, i l the tours should concentrate on areas of the plant that will be used during the l examination process, such as the control room, the simulation facility, and planned waik-through locations.

m. The chief examiner will conduct an exit briefing with the facility licensee after the operating tests are complete. The briefing should address any generic weaknesses noted during the operating tests and tiny other significant issues (e.g., problems with the reference material, the simulation facility, or the plant) that might be addressed in the examination report. The individual operating test i

results are predecisional until approved by NRC management in accordance with ES-501 and shall not be shared with the facility licensee during the exit briefing.

2. Walk-Throuah (Cateoories A and B) ;

l I lg a. The examiner should validate any JPMs that were not previously validated by the facility licensee or by the NRC during a preparatory site visit. This is particularly important for complex JPMs and those that require the applicant to implement an i alternative method directed by plant procedures.

b. To the extent possible, the examiner should have the applicant perform the
control room JPMs on the simulator, rather than asking the applicant to describe l how he or she would accomplish the task.

t l- If the examiner observes a discrepancy between the simulator setup and the conditions specified in a JPM, then the examiner shall stop the JPM and correct the situation, as necessary. If the task can be completed with different values (e.g., wind direction when determining a protective action recommendation during an emergency), then the examiner shall document the differences and coordinate with the facility contact and the NRC chief examiner to validate the applicant's response under the actual conditions. L The chief examiner is expected to coordinate the administration of the JPMs to j maximize the use of the simulator. To increase efficiency, different JPMs may be administered simultaneously to multiple applicants, but the examiners must ensure that mutual interference is minimized and test integrity is riot compromised. 5 of 10 NUREG-1021, Revision 8 L

r I ES-302 When JPMs or discussions are conducted in the control room, the examiners shall make every effort to accommodate and not interfere with normal shift operations. The chief examiner should request that the facility training manager notify the shift supervisor when the NRC will be conducting examination activities in the control room. If the number of persons or the noise levelin the control room is excessive, the examiner should, if possible, move to a quieter location, modify the sequence of the JPMs and return when the level of activity in the control room has abated, or ask the facility training manager to address the issue.

c. The examiner should encourage the applicant to sketch diagrams, flow paths, or other illustrations to aid in answering the examiner's questions. In all cases, the examiner shall collect the supporting material because it provides additional documentation to support a pass or fail decision (refer to ES-303). To facilitate i' copying, the applicant's drawings should be restricted to one side of separate sheets of 8.5-inch by 11-inch paper; the back of Form ES-303-1 or its attachments shall not be used for this purpose.

d. The examiner shou:d encourage the applicant to use such material as facility forms, schedules, and procedures if they are relevant to the questions asked. !

e. The examiner should be careful not to infer excessive system and administrative knowledge from observations made during Category C. The examiner should keep in mind that the applicant's proficiency in every administrative topic and i each control room and in-plant system should be deliberately evaluated in accordance with the operating test that was prepared in accordance with ES- 1 301.

f. As stated in 10 CFR 55.45(a), the operating test requires the applicant to )

demonstrate an understanding of and the ability to perform the actions necessary to accomplish a representative sample from among 13 items listed in the rule. If ihe applicant correctly performs a JPM (including both critical and noncritical steps) and demonstrates familiarity with the equipment and procedures, the examiner should infer that the applicant's understanding of the i system / task is adequate and refrain from asking follow-up questions. However, ! if the applicant fails to accomplish the task standard for the JPM, exhibits behavior that demonstrates a lack of familiarity with the equipment and procedures, or is unable to locate information, control board indications, or controls, the examiner should ask performance-based follow-up questions as 4 necessary to clarify or confirm the applicant's understanding of the system as it l relates to the task that was performed. Similarly, if the applicant gives an ambiguous answer to a prescripted administrative question in Category A, the examiner is expected to ask probing questions to ensure that the applicant understood the original question and the applicable knowledge or ability. The examiner shall document all perforrnance-based questions and answers for later evaluation. NUREG-1021, Revision 8 6 of 10 O1 l l I

1 1 ES 302 If an applicant volunteers additional or corrected information after having } completed a task or question, the examiner shall offer the applicant the opportunity to take whatever actions would be required in a similar situation in

the plant. The examiner will record any revisions to previously performed tasks or answers for consideration when grading the operating test per ES-303.

g. The examiner should practice other good walk-through evaluation techniques as discussed in Section D of Appendix C.

l

3. Simulator Test (Cateoorv C)

a. Before administering the test (s), the examiners will validate each scenario on the simulator to ensure that it will run as intended. Scenarios that were adapted from previous NRC examinations at the facility or from the facility licensee's bank -

may not require real-time validation. At a minimum, the examiners will " dry run" ) those events having variable inputs and questionable outcomes and discuss the remainder of the scenario with the facility's simulator instructor to ensure that it ! will run as planned. In some cases, the scenarios can be validated while the applicants are taking the written examination. However, it muy be beneficial to validate the scenarios during a preparatory site visit as determined by NRC regional management (refer to ES-201). l b. The examiners will take precautions to prevent the scenarios from being i revealed to the applicants before the tests begin. If significant portions of the scenarios are dry run or otherwise reviewed with the simulator instructor (s), the chief examiner shall ask the instructor (s) to sign a security agreement (Form ES 201-3) to protect the integrity of the simulator test.

c. The examiners should revise all copies of Forms ES-D-1 and ES-D-2 to reflect any changes made to the scenario events or the expected operator actions as a result of the scenario validation runs and reviews. These revisions should be neatly written in ink so that the forms can be used in the final write-up of the simulator test, as discussed in ES-303.

d. The examiners should review the scenarios together and discuss the required procedures, technical specifications, special circumstances, and so forth, related to the scenarios.

e. Immediately before beginning the simulator tests, the examiners should review the scenario events with the simulator operator and provide him or her with a copy of Form ES-D-1. This review should familiarize the operator with the l sequence of events to ensure that they will proceed as planned. This is

! ' particularly important if the simulator operator during the test is not the same l individual who assisted in validating the scenarios.

 'V                                             7 of 10 -                NUREG-1021, Revision 8

r l ES-302

f. The examiners should identify important plant parameters to be monitored during each simulator scenario. The chief examiner should ask the simulator operator to record selected parameters, if possible, on the facility's safety parameter display system (s). Parameter readings should be collected at meaningful intervals, depending on the parameter, the nature of the event, and the capability of the simulation facility. The chief examiner should retain the recordings as backup documentation to augment the notes taken by the examiners during the simulator test.

g. The examiner in charge of each scenario should arrange a suitable communication system with the simulator operator so that he or she can be prompted to insert the malfunctions without cuing the applicants. Malfunctions may be planned for a predetermined time or power level so that the examiners and the facility operator are aware of the event that is occurring or pending.

If necessary, the examiners may use time compression to speed up the response of key parameters so that the scenario can proceed to the next event ! within a reasonable time. Time compression is acceptable as long as it is used judiciously and the operators are given sufficient time to perform the tasks that they would typically perform in real time. If the examiners intend to use time compression, they should inform the applicants of that fact during the operating test briefing (refer to Section D.1.a). The examiners should also mitigate the j potential for negative training by debriefing the applicants after any scenario in which time compression was used.

h. Before beginning each scenario set, the examiners should have the simulator l

operator advance any control room strip chart recorders that may prove useful in recreating the sequence of events. The charts should be clearly marked with the date, time, and examiner's initials so that they can be accurately matched with the correct operating crew. I. The chief examiner should ensure that the simulator operator (or examiner) playing the role of other plant personnel is aware of the time scale for responding to the applicants' requests for information. For example, fast-time could be I specified for auxiliary operator checks or lineups to prevent long delays in simulated operations, while maintenance and chemistry sample information can be provided with normal time delays to present the applicants with the same analysis problems that they will face as operators. J. Before the simulator test begins, the examiners shall caution the simulator operator to provide only information that is specifically requested by the applicants and does not compromise the integrity of the examination. When the simulator operator is briefing the applicants or communicating with them on the telephone, the examiners should monitor the conversation to ensure that the information provided is appropriate and does not cue the applicants. 1 NUREG-1021, Revision 8 8 of 10

ES-302

k. Each examiner should use the expected actions and behaviors listed on Form ES-D-2 as a guide while administering the simulator tests. If an applicant pedorms as expected, the examiner may simply note in the left-hand column of the form the time when the expected actions occurred. However, if an applicant does not perform as expected, the examiner should note the applicant's actions (or lack of actions) next to or below the expected action and follow up with appropriate questions after the simulator scenario is completed (refer to Section D.3.1).

Each examiner must determine the best way to docurnent the applicant's actions. Some examiners record a minute-by-minute account of all key plant events and the applicant's actions as they occur; other examiners only record the applicant's .significant actions. Each individual examiner should develop his or her own examination documentation technique; however, the documentation must provide an adequate basis for a licensing decision. In addition, the examiner's notes must provide sufficient information to allow the examiner to confidently judge the applicant's performance on the competencies described in Appendix D.

l. Examiners shall limit discussions with the applicants during the scenarios both to maintain realism and to avoid distracting the applicants from operating the plant.

The examiners' questions during the scenarios should be limited to those that are necessary to assess the applicants' understanding of plant conditions and (O") the required operator actions. Whenever possible, the examiner shal! defer questioning the applicant until a time when the applicant is not operating or closely monitoring the plant (preferably after the simulator has been placed in

       " freeze"). The examiner's follow-up questions or concerns can generally be addressed during a brief question and answer period after each scenario or during the control room systerns and facility walk-through portion of the operating test (i.e., Category B) if it is performed after the simulator test.

m. The examiners who administer the simulator test shall confer immediately after completing the scenario set to compare notes and to verify that each exarniner observed his or her applicant perform the required number of transients and events in a manner sufficient to justify an evaluation of all the required competencies. If necessary, the examiners shall run an additional scenario to ensure that the required evolutions and competencies are covered. All scenarios will be planned and documented in accordance with Section D of ES-301.

The chief examiner shall ensure that the examiners' observations are consistent and that their findings are mutually supportive. If a performance deficiency is

       " shared" by more than one applicant, it should be noted by both evaluating examiners, ideally, this cross-check should be accomplished as soon as possible after running the scenarios while still at the facility. The cross-check must be accomplished before finalizing the examination results in accordance with ES-303.

/~N (d) 9 of 10 NUREG-1021, Revision 8

n. If the applicants did not perform as expected, the examiner shall ask the simulator operator to provide copies of the logs, charts, and other materials that may be required after leaving the facility to evaluate and document the applicants' performance. The examiner of record shall retain all documentation related to any operating test failure until the proposed denial becomes final or a license is issued.

The chief examiner should also ask the simulator operator to retain copies of the same materials until all applicants are licensed or all appeals are settled, as suggested in the sample corporate notification letter shown in ES-201, Attachment 3.

o. If the simulation facility should become inoperable and cause excessive delay of the operating tests, the chief examiner should discuss the situation with the facility licensee and the responsible regional supervisor so that management can make a decision regarding the conduct of the operating tests. It may be l

necessary to reschedule the simulator examinations for a later date. The simulator should be considered inoperable under any of the following conditions: The simulator exhibits a mass / energy imbalance, erratic logic, or inexplicable panel indications during model execution. The simulator exhibits unplanned and unexplained events or malfunctions that cause the applicants to divert from the expected responses and success path of the planned scenario. The simulator automatically goes to the " freeze" state during a scenario or a "beyond simulated limits" alarm is received on the instructor station. 1 The simulator instructor informs the examination team that a software I module has halted or " kicked out." l Occurrence of any of these abnormal simulator operating conditions during an I examination constitutes sufficient cause to stop the scenario. Evaluations of the applicants' performance during any of these simulator malfunction conditions may be unreliable. 1 J When the simulator has been restored to full operability, the chief examiner will j determine if the scenaric, requires replacement, may be resumed in progress, or ) may be restarted from the beginning. l l NUREG-1021, Revision 8 10 of 10 9i l

l l 0 , ES-303 DOCUMENTING AND GRADING INITIAL OPERATING TESTS

   ~ A. PURPOSE This standard describes the procedures for documenting all categories of the operating test, collating the data to arrive at a pass or fail recommendation, and reviewing the documentation to ensure quality.

B. BACKGROUND This standard assumes that the operating test was prepared and administered in accordance with ES-301 and ES-302, respectively. The procedures contained herein require the examiner to evaluate each applicant's performance on the operating test and make a judgement as to whether the applicant's level of knowledge and understanding meet the minimum requirements to safely operate the facility for which the license is sought. The examiner evaluates each noted deficiency in light of the total breadth of knowledge and ability demonstrated by the applicant in that subject area. When used to evaluate an applicant's performance on all or part of the operating test as discussed herein, the terms " satisfactory" and " unsatisfactory" are defined as follows: S - Satisfactory Working Knowledge and Understanding The applicant may have some slight or minor difficulty describing system interactions. Q Competence in the operation of equipment associated with the system is very good, although there may be some hesitation while discussing or performing some tasks. The applicant appears to be familiar with the equipment and procedures. U - Unsatisfactory or Poor Working Knowledge and Understanding ; Tne applicant has difficulty answering questions in depth and describing the interactions j of systems. Discussions or behavior while operating equipment show lack of familiarity l with the equipment and procecures. The applicant is unable to answer questions or l provides incorrect or incomplete answers. The applicant is unfamiliar with the subject or system, as evidenced by hesitant answers, inability to locate information, inability to locate control board indications or controls, and lack of knowledge of system operating procedures. 1 C. RESPONSIBILITIES :

1. Facility Licensee The facility licensee's responsibilities are limited to providing the NRC examiners with whatever additional reference materials and information the examiner might require to evaluate the applicants' performance on the operating tests. Such materials might include simulator strip chart recordings that document plant status during the simulator scenarios, and procedures that document the expected operator actions, l 1 of 27 NUREG-1021, Revision 8

ES 303

2. NRC Examiner of Record As soon as possible after administering the test, the examiner of record shall review, evaluate, and finalize each applicant's operating test documentation in accordance with the instructions in Section D.

If an applicant made an error with serious safety consequences, the examiner may recommend an operating test failure even if the grading instructions in Section D would normally result in a passing grade. Under such circumstances, the examiner shall thoroughly justify and document the basis for the failure in accordance with Section D.3.b. Moreover, the NRC regional office shall obtain written concurrence from the NRR operator licensing program office before I completing the licensing action.

3. NRC Chief Examiner

a. The chief examinet shall arrange a meeting of the NRC examination team members after the simulator scenarios are complete. Such meetings enable the examiners to compare notes to ensure that the documentation for applicants on the same operating crew is consistent and mutually supportive. ,

i i

b. The chief examiner shall work with the other examiners on the team to resolve j any technical questions that might arise during the grading process, and j communicate any additional reference material requirements to the facility j contact.

c. The chief examiner or a management-approved designee will review the grading of each operating test to verify that the examiner's comments appropriately support his or her recommendation and to ensure that the operating test meets l the requirements of ES-301. If the chief examiner or designee does not agree with any of the examiner's recommendations, he or she shall confer with the examiner before overturning the recommendation. Such disagreements are not common and usually arise because an unsatisfactory grade is not adequately justified. It is, therefore, very important for examiners to be complete and accurate in their grading and documentation.

d. The chief examiner or designee shall make an independent pass-or- fail recommendation, sign the " Final Recommendation" block on Form ES-303-1, i

                " Individual Examination Report," and forward the package to the responsible supervisor for review in accordance with ES-501. The supervisor must concur in any recommendation to overturn the examiner's results, and the specific reasons I

for this action must be explained on Form ES-303-2, " Operating Test Comments." NUREG-1021, Revision 8 2 of 27 Ol l

O e ES-303 C D. GRADING AND DOCUMENTATION INSTRUCTIONS (

1. Review and Cateaorize Rouch Notes and Documentation

a. Review the job performance measures (JPMs) and simulator scenarios that were performed and the prescripted Category A and performance-based followup questions that were asked. Evaluate all rough notes and documentation generated while administering the operating test to determine the areas in which the applicant was deficient. If the applicant generated or used any material (such as figures, drawings, flowcharts, or forms) during the operating test, the material may be used to aid in documenting the applicant's performance. If it contributes to an unsatisfactory performance evaluation, the material shall be appropriately marked and cross-referenced to the applicable deficiency and attached to the examination package for retention.

b. Verify the validity and technical accuracy of any performance-based questions that were asked during the operating test but had not been prescripted, as well as any unexpected events or actions that occurred during the simulator operating test. If necessaiy, work through the chief examiner to obtain any additional reference material that might be required to resolve any technical questions.

c. On the rough notes and documentation, label or highlight every action, response, O note, or comment that may constitute a performance deficiency.

d. Label each deficiency related to the applicant's administrative and plant system knowledge and abilities with the alphanumeric code of the administrative topic (e.g., A.1) or the control room or plant system (e.g., B.1.c or B.2.a) to which it applies.

e. Review each simulator operating test performance deficiency. Using as a guide the competency and rating factor descriptions in Appendix D and on Form ES-303-3 (RO) or Form ES-303-4 (SRO), code each deficiency with the number and letter of the rating factor (s) it most accurately reflects (e.g., C.4.a). Whenever possible, attempt to identify the root cause of the applicant's deficiencies and code each deficiency with no more than two different rating factors. However, one significant deficiency may be coded with additional rating factors if the error can be shown, consistent with the criteria in Section D.3.b, to be relevant to each of the cited rating factors.

As stated in ES-302, it is essential that the simulator operating test documentation is consistent and mutually supportive for all applicants in an operating crew. Operating errors that involved more than one applicant should be noted by each applicant's evaluating examiner. If the examination team j members do not have the opportunity to discuss and compare their observations before leaving the site, the chief examiner shall schedule a conference call after the examiners retum to their respective offices.

3 of 27 NUREG-1021, Revision 8

ES-303

2. Evaluate the Apolicant's Performance After categorizing and coding the rough notes, review, evaluate, and grade the applicant's performance in operating test Category A, " Administrative Topics," Category B, " Control Room Systems / Facility Walk-Through," and Category C, " Integrated Plant Operations," as follows:

a. Form ES-303-1, Category A Review the identified deficiencies and decide whether the applicant's knowledge and understanding of each administrative topic was satisfactory or unsatisfactory (refer to the discussion in Section B). Grade any JPMs that were used to evaluate the administrative topics as described in Section D.2.b below.

Document the grade for each administrative topic by placing an "S" or "U" in the appropriate block on page 2 of Form ES-303-1. Document and justify every deficiency in accordance with Section D.3. After grading all four of the topics in Category A, assess the applicant's topic grades and deficiencies and assign a single "S" or "U" grade for the category, if the applicant has a "U" in only one administrative topic, the examiner may fail the applicant in Category A depending on the safety significance of the identified deficiency. However,if the applicant has a "U"in two or more of the administrative topics, the examiner must assign a grade of "U" for Category A. Place the assigned grade in the " Administrative Topics" block of the " Operating Test Summary" on page 1 of Form ES-303-1. Enter "N/E" (not examined)if this category was waived in accordance with ES-204.

b. Form ES-303-1, Category B 1

On page 2 of the applicant's Form ES-303-1, enter the names of the systems ! and JPMs examined during operating test Subcategories B.1, " Control Room j Systems," and B.2, " Facility Walk-Through." To determine a grade for the systems /JPMs listed on Form ES-303-1, evaluate each deficiency coded in the rough notes for Category B. If the following criteria i are met, assign a satisfactory grade by placing an "S" in the " Evaluation" column for that system /JPM; otherwise enter a "U": Time-critical JPMs must be completed within the allotted time. The task standard for the JPM must be accomplished by correctly completing all of the critical steps. If the applicant initially missed a critical step, but later performed it correctly and accomplished the task standard without degrading the condition of the system or the plant, the applicant's performance on that JPM should be graded as satisfactory. However, the applicant's error shall be documented in accordance with Section D.3. NUREG-1021, Revision 8 4 of 27 ___._______J

i l ES-303

                ' The responses to any performance-based follow-up questions asked pursuant to Section D.2.f of ES-302 must confirm that the applicant's understanding of the system /JPM is satisfactory.

If the follow-up questions reveal that the applicant's understanding of the system /JPM is seriously deficient, the examiner may recommend an l unsatisfactory grade for the system even though the applicant successfully completed the task standard for the JPM. The basis for the recommendation shall be thoroughly justified and documented in accordance with Section D.3. Conversely, if the applicant did not accomplish the task standard and follow-up questioning revealed that the failure was caused by a deficiency in the procedure or some other factor beyond the applicant's contre the I

                ' examiner may still recommend a satisfactory grade for the system /JPM.

Once again, the basis for the recommendation shall be thoroughly l justified and documented in accordance with Section D.3.

After grading the applicant's performance with respect to all the Category B [ systems, determine an overall grade for Category B by calculating the l percentage of satisfactory system grades. If the applicant has an "S" on at least 80 percent of the systems examined, the applicant passes Category B and receives an "S" overall. If the applicant has an "S" on fewer than 80 percent of the systems, the applicant fails Category B and receives a "U" overall. Document the applicant's grade by placing an "S" or a "U" in block B, " Control Room Systems and Facility Walk-Through," in the " Operating Test Summary" on ! page 1 of Form ES-303-1. Enter "N/E" if this category was waived in accordance l with ES-204. Document and justify every deficiency in accordance with Section j D.3. i

c. Form ES-303-1, Category C Using Form ES-303-3 or ES-303-4, depending on the applicant's license level, evaluate any deficiencies coded for Category C. Circle the integral rating value (1 through 3) corresponding to the behavioral anchor that most accurately reflects the applicant's performance. ' A rating of "1" would be justified if the 1 applicant missed a critical task (i.e., by omission or incorrect performance) or committed multiple errors of lesser significance that have a bearing on the rating factor. Missina one or more critical tasks does not necessarily mean that the aoolicant will fail the simulator test. nor does success on every critical task orevent the examiner from recommendina a failure if the aoolicant had other deficiencies that. In the aaoreaate. iustify the failure based on the comoetency evaluations. As discussed in ES-301, Competency 5 is optional for SRO I upgrade applicants. However, the examiner shall evaluate Competency 5 if the applicant rotated into an operating crew position that required the applicant to manipulate the controls.

V 5 of 27 NUREG-1021, Revision 8 f

ES-303 Multiply each integral rating value by its associated weighting factor to obtain a e numerical measure of the eoplicant's performance on each rating factor. Then, I circle the corresponding numbers on page 3 of the RO or SRO applicant's Form l ES-303-1. I For each competency on page 3 of Form ES-303-1, sum the circled rating factor grades and enter the resulting competency grade in the " Total" column. (The grades should range between 1 and 3.) Using the following evaluation criteria, determine if the applicant's overall performance in Category C is satisfactory or unsatisfactory and document the grade by placing an "S" or a "U" in block C, " Integrated Plant Operations." in the

             " Operating Test Summary" on page 1 of Form ES-303-1. Enter "N/E" if this category was waived in accordance with ES-204.

If the " total" grade for al/ competencies is greater than 1.8, the applicant's performance is generally satisfactory. If the " total" grade for Competency 6, " Communications and Crew Interactions," is less than or equal to 1.8 but greater than 1.0, andthe individual " total" grades for allother competencies are 2.0 or greater, the applicant's performance is satisfactory. If the " total" grade for Competency 6 is 1.0, orthe " total" grade for any other competency is 1.8 or less, the applicant's performance is unsatisfactory. Note: Competency 5, " Control Board Operations," is optional for SRO upgrade applicants. However, if it is evaluated, it shall be factored into the applicant's final grade. Document and justify every deficiency in accordance with Section D.3.

3. Finalize the Documentation

a. Review and finalize the simulator scenarios that were run for Category C of the operating test.

Complete Form ES-D-1, " Scenario Outline," by entering the applicants' names, the positions they occupied during the scenario, and the facility's name on the top of the form. Also enter on Form ES-D-1 any scenario revisions made during the test, so that each form accurately shows all of the events that actually occurred during each scenario. Change the event numbers, malfunction numbers, malfunction types, and descriptions as necessary to reflect the "as run" conditions. These changes may be made using pen-and-ink or by retyping the scenario, provided that the final form is clear and legible. Update each Form ES-D-2, " Operator Actions," to reflect the "as run" conditions. Discard or mark as "not used" any events that were not run, and fill out new NUREG-1021, Revision 8 6 of 27

l l I O ES-303 V forms for any events that were run but not originally planned. Neatly enter notes, , comments, and additional actions in the spaces between the expected operator i actions.

The final Forms ES-D-1 and ES-D-2 must be a clear, legible, and sequential record of the actual events and actions that occurred during the simulator operating test. The forms sent to the applicant shall not contain any rough notes or irrelevant comments. Any events or malfunctions that did not function as expected c. are not useful in evaluating the applicants (e.g., a surveillance test thit rcqtio . a long time to perform) should be noted on the master copy of the secarico to aid in future scenario preparation.

b. Review the applicant's Form ES-303-1 and the rough documentation. Justify in detait on Form ES-303-2, " Operating Test Comments," every knowledge or ability deficiency that contributed to a failure in any operating test Category.

Provide the following specific information, as applicable: the question asked or task administered (i.e., describe the JPM or the simulator scenario and event, as well as the applicant's position on the operating crew) O - the applicant's incorrect answer or action and an indication of whether the . ij action was a JPM critical step the lack of knowledge or ability that the applicant demonstrated

the consequences of the applicant's incorrect answer or action f the correct answer or action, with an se copriate facility reference (e.g., f lesson plan, system description, proceisre name and number) { the K/A number and its importance rating (as given in NUREG-1122 or NUREG-1123) and the facility's leaming objective the item from 10 CFR 55.45(a) that the applicant did not understand or was unable to perform General statements (such as "did not know decay heat removal system") are not adequate. Whenever possible, substantiate comments with printouts or strip chart J recordings generated during the simulator operating test and drawings and ; illustrations generated by the applicant. j Deficiencies that do not contribute to an operating test Category failure shall also j be documented; however, a brief statement describing the error and the , expected action or response is generally sufficient. Examiners should keep in j 7 of 27 NUREG-1021, Revision 8 i i

ES-303 mind that their licensing recommendation and the associated documentation are subject to review by the chief examiner and NRC regional office management. Therefore, the documentation should contain sufficient detail so that the independent reviewer, responsible supervisor, and licensing official can make a logical decision in support of the examiner's recommendation to deny or issue the license. Retain rough documentation until the chief examiner and NRC regional office management have reviewed the examiner's recommendations and concurred in j the results (refer to ES 501). Examiners shall retain all applicable notes and l documentation associated with proposed denials until the denials become final. Examiners are advised that such notes would be subject to disclosure if requested under the Freedom of information Act.

c. Cross-reference each comment on Form ES-303-2 with the specific task, subject, or competency rating factor to which it applies on the applicant's Form ES-303-1. Do this by entering the applicable alphanumeric subject reference from Form ES-303-1 (e.g., A.2, B.1.c, C.4.b) in the left-hand column of Form ES-303-2, and entering the page number on which the comment is found in the appropriate block on Form ES-303-1.

4. Make a Final Recommendation

a. After grading and documenting the operating test, make an overall recommendation by checking the " Pass" or " Fail" (or " Waive" if the entire operating test was waived in accordance with ES-204) block, and signing and dating the " Examiner Recommendations" section on the applicant's Form ES-303-1. Make a " pass" recommendation only if allsummary blocks of the operating test contain satisfactory (S) grades or the letters "N/E," indicating that the applicant was not examined in that area. l l

b. If the written examination was not waived and the written examination data have i not yet been entered on Form ES-303-1, route the examination package to the l written examination grader (or NRC reviewer if the examination was graded by l the facility licensee) for processing in accordance with ES-403. If the written I examination results have already been entered, forward the examination package to the chief examiner for review.

E. ATTACHMENTS / FORMS , l Form ES-303-1, " Individual Examination Report" Form ES-303-2, " Operating Test Comments" Form ES-303-3, "RO Competency Grading Worksheet for Integrated Plant Operations" Form ES $03-4, "SRO Competency Grading Worksheet for Integrated Plant Operations" NUREG-1GL i, Revision 8 8 of 27

; ES-303                           individual Examination Report                   Form ES-303-1

( U.S. Nuclear Regulatory Commission l Individual Examination Report Applicant's Name Docket Number 55-I R Examination Type (Initial or Retake) Facility Name Reactor Operator Hot Ser:ior Reactor Operator (SRO) Instant Cold Facility SRO Upgrade BWR Desenpton SRO Limited to Fuel Handling PWR Written Examination Summary NRC Author / Reviewer Total Examination Points NRC Grader / Reviewer Total Applicant Points Date Administered Applicant Grade % Operating Test Summary Administered by Date Administered 1 A. Administrative Topics B. Control Room Systems and Facility Walk-Through C. Integrated Plant Operations (Simulator Operating Test) Examiner Recommendations Check Blocks Pass Fail Waive Signature Date Written Examination Operating Test Final Recommendation License Recommendation issue License Signature - Supervisor Date Deny License O) i V 9 of 27 NUREG-1021, Revision 8

ES-303 2 Form ES-303-1 l i Applicant Docket Number: 55- Page of l l Category Evaluation Comment Page ) (S or U) Number ; 7 A. Administrative Topics l l

1. Conduct of Operations j 1

2. Equipment Control

3. Radiation Control 1

4. Emergency Plan B.1. Control Room Systems

a. i b.

c. d. e. f. 9 B.2 Facility Walk-Through a. b. c. NUREG-1021, Revision 8 10 of 27

y I

   - ES-303                                        3.a                            Form FC-303-1 Applicant Docket Number: 55-                                                ' Page of i

C. Reactor Operator integrated Plant Operations ) (Simulator Operating Test) Grading Summary Competencies / _ Comment J Rating Factors . Weight 3.0 2.0 1.0 Total Page No.

1. ' Alarms / Annunciators ;

a. Notice /Acknewledge 0.30 0.90 0.60 0.30

b. - Interpret / Verify 0.40 1.20 0.80 0.40

c. Frioritize 0.30 0.90 0.60 0.30

2. Diagnosis. ,

a. Recognize 0.40 1.20 0.80-- 0.40 l

b. Use Reference Material 0.20 0.60 0.40. 0.20

c. Diagnose 0.40. 1.20 0.80 0.40 I l

3. System Response

a. Locate / interpret 0.33. 1.00 0.67 0.33 .

b. System Operation Knowledge 0.33 ~ 1.00 0.67 0.33

c. Effect of Actions .0.33 1.00 0.67 0.33

4. Procedures / Tech Specs

a. Reference 0.20' O.60 0.40 0.20

b. Eop Entry /immediate Actions 0.40 1.20 0.80 0.40

c. Procedure Compliance 0.20 0.60 0.40 0.20

d. Tech Spec Entry 0.20 0.60 0.40 0.20

5. Control Board Operations

a. Locate 0.25 0.75 0.50 0.25

b. Manipulate 0.25 0.75 0.50 0.25

c. Response 0.25 0.75 -0.50 0.25

d. Manual Control 0.25 0.75 0.50 0.25

6. Communications

a. Provide Information 0.33 1.00 0.67 0.33

b. Receive Information 0.33 1.00 0.67 0.33

c. Carry Out Instructions 0.33 1.00 0.67 0.33 O

11 of 27 NUREG-1021, Revision 8

I ES-303 3.b Form ES-303-1 { 1 Applicant Docket Number: 55- Page of C. Senior Reactor Operator Integrated Plant Operations (Simulator Operating Test) Grading Summary l Competencies / Comment l Rating Factors Weight 3.0 2.0 1.0 Total Page No.

1. Alarm.s/ Annunciators

a. Prioritize 0.30 0.90 0.60 0.30

b. Interpret 0.35 1.05 0.70 0.35

c. Verify 0.35 1.05 0.70 0.35

2. Diagnosis

a. Recognize 0.25 .0.75 0.50 0.25

b. Accuracy 0.25 0.75 0.50 0.25

c. Diagnose 0.25 0.75 0.50 0.25

d. Crew Response 0.25 0.75 0.50 0.25

3. System Response

a. Interpret 0.35 1.05 0.70 0.35

b. Attentive 0.20 0.60 0.40 0.20

c. Plant Effects 0.45 1.35 0.90 0.45

4. Procedures

a. Reference 0.25 0.75 0.50 0.25

b. Correct Use 0.50 1.50 1.00 0.50

c. CrewImplementation 0.25 0.75 0.50 0.25

5. Control Board Operations

a. Locate 0.25 0.75 0.50 0.25

b. Manipulate 0.25 0.75 0.50 0.25

c. Response 0.25 0.75 0.50 0.25

d. Manual Control 0.25 0.75 0.50 0.25

6. Communications

a. Clarity 0.45 1.35 0.90 0.45

b. Crew Informed 0.35 1.05 0.70 0.35

c. Receive Information 0.20 0.60 0.40 0.20

7. Directing Operations

a. Timely Action 0.20 0.60 0.40 0.20

b. Safe Directions 0.40 1.20 0.80 0.40

c. Oversight 0.20 0.60 0.40 0.20

d. Crew Feedback 0.20 0.60 0.40 0.20

8. Technical Specifications

a. Recognize 0.40 1.20 0.80 0.40

b. Locate 0.20 0.60 0.40 0.20

c. Compliance 0.40 1.20 0.80 0.40 NUREG-1021, Revision 8 12 of 27

I l l ES-303 Operating Test Comments Form ES-303-2 l Applicant Docket Number: 55- Page of Form ES-303-1 Comments Cross Reference O V i 13 of 27 NUREG-1021, Revision 8

ES-303 RO Competency Grading Worksheet Form ES-303-3 , for Integrated Plant Operations _

1. UNDERSTAND AND INTERPRET ANNUNCIATORS AND ALARM SIGNALS DID THE APPLICANT:

(a) NOTICE and ACKNOWLEDGE alarms? 3 2 1 Consistent and Minor difficulties Failed to notice and timely or lapses in awareness acknowledge important acknowledgment or response alarms; distracted by nuisance alarms; etc. [.30 X = ] 1 (b) Correctly INTERPRET and VERIFY that annunciators and alarm signals were l consistent with plant and system conditions (including the use of alarm response procedures (ARPs), when necessary)? 3 2 1 Consistent and Minor inaccuracies Significant inaccur-efficient in interpreting acies resulted in j interpretation and verifying signals plant degradation; j and verification poor use of ARPs j (.40 X = } (c) ATTEND to ANNUNCIATORS and ALARM SIGNALS in order of importance and ) severity? l i 3 2 1 ! I Consistent Minor inaccuracies Did not prioritize l attention in and oversights attention to signals; all cases inattentive to important alarms j (.30 X = ] l NUREG-1021, Revision 8 14 of 27 1 1 l

-[ ES-303 2 Forrn ES-303-3 I f

2. DIAGNOSE EVENTS AND CONDITIONS BASED ON SIGNALS AND READINGS I DID THE APPLICANT:

I l (a) RECOGNIZE off-normal trends and status? 3 2 1 Quick and accurate Some delays in Serious omissions, recognition recognizing off- delays, or I normal conditions inaccuracies in recognizing events [.40 X = ] l (b) Correctly USE REFERENCE MATERIAL (prints, books, charts) to aid in diagnosing and I classifying events and conditions? I 3 2 1 (j ( Correctly used Minor errors in Did not use or references, using or relying incorrectly used when necessary on references references to l diagnose events 1 I [.20 X .-

                                                                                          ]      !

(c) Correctly DIAGNOSE plant conditions based on control room indications? 3 2 1 Diagnoses were Minor errors or Faulty diagnoses accurate difficulties adversely affected in diagnoses plant status [.40 X = ] b 15 of 27 NUREG-1021, Revision 8

g ES-303 3 Form ES-303-3

          . 3. UNDERSTAND PLANT AND SYSTEM RESPONSE DID THE APPLICANT:

(a) LOCATE and correctly INTERPRET relevant instruments and other indicators of plant and system response (s)? I 3 2 1 Accurate and Minor errors in Serious omissions efficient location locating and inter- or inaccuracies and interpretation preting instruments in interpreting of instruments and displays instruments [ 33 X = ] l (b) Demonstrate KNOWLEDGE of SYSTEM OPERATION, including set points, I iniedocks, and automatic actions? 3 2 1 Demonstrated Minor instances of inadequate knowledge thorough under- errors caused by in- resulted in plant standing of adequate knowledge degradation system operations ,- [.33 X = ] (c) Demonstrate an understanding of how his or her ACTIONS (or inaction) AFFECT PLANT and SYSTEM CONDITIONS 7 3 2 1 Understood the Minor misunder. Appeared to act effect of actions standing of effect without knowledge on plant and of actions on plant of or regard for systems and systems effect on pisnt and systems 3 [.33 X = ] NUAEG-1021, Revision 8 16 of 27 i

'!\

f gO ' ES-303 4 Form ES-303-3 U.

4. COMPLY WITH AND USE PROCEDURES AND TECHNICAL SPECIFICATIONS DID THE APPLICANT:

(a) REFER TO the appropriate procedure in a timely manner? 3 2 1

                      ' Quickly located                                                       Minor difficulties and     Problems and failures appropriate ,                                                         oversights in referring    in referring to
                      . procedures                                                            to appropriate             procedures in procedures                 important instances

[.20 X = ] (b) RECOGNIZE EOP ENTRY CONDITIONS and carry out appropriate immediate actions without the aid of references or other forms of assistance? , H 3 2 1 Consistent, accurate Minorlapses or errors, Did not accurately and timely . but actions generally execute actions recognition appropriate O- [.40 X = ] (c) . COMPLY WITH procedures (including precautions and limitations) in an accurate and timely manner? 3 2 1 Accurate and Few errors; corrections Many significant timely compliance made in sufficient time errors; excessive

to avoid adverse effect ' assistance required

{.20 X = ] (d) RECOGNIZE plant conditions that are addressed in technical specifications? 3 2 1 Recognized and Minor assistance Did not recognize complied with required to recognize' conditions and/or LCOs and action conditions and/or comply with LCOs and 4 statements comply with LCOs and action statements action statements [.20 X = ]

 't 17 of 27                NUREG-1021, Revision 8 J
                                                . . . . . . . . . . . . . . . . _ . . , _ _ _               _.__.__.m.,_

ES-303 Form ES-303-3

5. OPERATE THE CONTROL BOARD DID THE APPLICANT:

(a) LOCATE CONTROLS efficiently and accurately? j 3 2 1 Promptly located Some minor hesitancy Unable to locate appropriate controls and difficulty in controls without in allinstances locating controls assistance [.25 X = ] (b) MANIPULATE CONTROLS in an accurate and timely manner? 3 2 1 ! Control manipulations Minor shodcomings, Improper manipul-were consistently but efficiently ations resulted in accurate and mitigated any major system timely resulting consequences perturbations [.25 X = ] (c) ACT appropriately in response to INSTRUMENT READINGS? 3 2 1 Responses were Generally adequate Failed to react appropriate and response; some appropriately to timely errors and lapses instrument readings without assistance [.25 X = ] (d) Take MANUAL CONTROL of automatic functions when appropriate? j i 3 2 1 i L Took manual control Minor delays and some Depended on automatic l when appropriate prompting necessary actions; had to be ' before overriding prompted to take l automatic functions manual control I = r [.25 X ] l NUREG-1021, Revision 8 18 of 27 I l l

y

    ,O   ES-303                                          6                            Form ES-303-3 O    6. COMMUNICATE AND INTERACT WITH OTHER CREW MEMBERS DID THE APPLICANT:

(a) PROVIDE clear and accurate INFORMATION on system status to others for the performance of theirjobs? 3 2 1 Provided others Minor instances of Failure to accurately with accurate and needing to be _ provide important pertinent information prompted for input; information to others some incomplete and jeopardized plant inaccurate information status [.33 X = } ) (b) Effectively RECEIVE INFORMATION from others (including requesting,

       .        acknowledging, and attending to information)?

3 2 1 (Q Responded and reacted appropriately Minor instances of failure to inattentive to information to information ' acknowledge or provided by others from others respond to information [.33 X = ] (c) CARRY OUT the INSTRUCTIONS of the supervisor successfully? 3 2 1 Ably carried out Minor hesitancy and Failed to promptly all supervisory difficulty in following and accurately follow objectives; dis- orders, but ultimately directions; blindly cussed instructions complied successfully complied with when questionable erroneous orders [.33 X = ]

   ,O kd                                                                      NUREG-1021, Revision 8 19 of 27

l ES-303 SRO Competency Grading Worksheets Form ES-303-4 for Integrated Plant Operations

1. UNDERSTAND AND INTERPRET ANNUNCIATORS AND ALARM SIGNALS DID THE APPLICANT:

(a) NOTICE and ATTEND to annunciator and alarm signals in order of their importance and severity? 3 2 1 Responded accurately Minor ditficulties Failed to attend to ar.d efficiently in in attending to signals or prioritize important allinstances or prioritizing attention alarms; responded slowly; distracted by nuisance alarms [.30 X = ] (b) Correctly INTERPRET the meaning and significance of alarms and annunciators (including the use of alarm response procedures (ARPs), when necessary)? 3 2 1 Understood and Minor inaccuracies Misinterpretations, quickly determined or delays in delays, or misuse of what failures alarms interpreting alarms ARPs resulted in were indicating plant degradation i [.35 X = ] ! (c) VERIFY that annunciator and alarm signals were consistent with plant and system conditions? 3 2 1 Ensured proper Minor lapses in Failed to correctly verification when alarm verification, verify signals on necessary but no inappropriate important occasions; actions ' wore taken as did not notice incon-a result of inadequate sistencies between verification alarms and plant conditions [.35 X = ] l NUREG-1021, Revision 8 20 of 27 9)! l

k O ES-303 <

2- Form ES-303-4 2.' DIAGNOSE EVENTS AND CONDITIONS BASED ON SIGNALS AND READINGS

      ~ DID THE APPLICANT:
      . '(a)   RECOGNIZE off-normal trends and status?

3 2 1 Quick and accurate Some delays in . Serious omissions, recognition recognizing off . delays, or inaccuracies normal conditions in recognizing trends [.25 X = ] (b) Ensure the collection of CORRECT, ACCURATE, and COMPLETE information and reference material on which to base diagnoses? 3 2 1

        <      Ensured that all            Minor instances of over-    Serious instances of relevant indications         looking, overrelying on,    misusing or failing to use and references were          or misinterpreting indic-   important information or data checked                      ations and/or references                                    !

[.25 X = ] (c) - Correctly DIAGNOSE plant conditions based on control room indications? 3 2 1 Diagnoses of plant Minor errors or Faulty diagnoses conditions were difficulties in diagnosing adversely affected accurate conditions plant status [.25 X = ] (d) Ensure that CORRECT and TIMELY DIAGNOSTIC ACTIVITIES were carried out by the CREW 7 3 2 1 Ensured effective Minor errors or Faulty diagnostic activities diagnostic activities difficulties in by crew adversely and diagnoses by crew diagnosing by crew affected plant status [.25 X = ] 21 of 27 NUREG-1021, Revision 8

1 ES 303 _ 3 Form ES-303-4

3. UNDERSTAND PLANT AND SYSTEM RESPONSE DID THE APPLICANT:

(a) INTERPRET control room indicators correctly and efficiently to ascertain and verify the status and operation of plant systems? 3 2 1 Accurate and efficient Minor errors in Serious omissions, interpretation of interpreting delays, or inaccuracies instruments and instruments and in interpreting displays displays instruments and displays [.35 X = ] (b) Remain ATTENTIVE to control room indications? 3 2 1 Regularly scanned Sporadically scanned Rarely scanned indications; indications; anticipated indications; minor failed to anticipate changes in plant lapses in anticipating predictable changes in ! conditions due to predictable changes plant status events in progress [.20 X = ] ! l (c) Demonstrate, through directives and actions, a thorough UNDERSTANDING of how the PLANT, SYSTEMS, and COMPONENTS OPERATE AND INTERACT (including set points, interlocks, and automatic actions)? 3 2 1 Demonstrated thorough Minor errors because of inadequate knowledge understanding of how gaps in knowledge of of system and component systems and components how systems and operation resulted in operate and interact components operate serious mistakes or plant degradation [.45 X = ] NUREG-1021, Revision 8 22 of 27 Ol l

) l

 '/     -

ES-303 4 Form ES-303-4

4. COMPLIANCE WITH AND USE OF PROCEDURES DID THE APPLICANT:

(a) REFER to correct procedures and procedural steps when appropriate? 3 2 1 Requested or readily Minor lapses in referring Failed to correctly located all appropriate to or locating refer to procedures ! procedures as necessary appropriate procedures in important instances l (.25 X = ] l (b) USE PROCEDURES CORRECTLY, including following procedural steps in correct sequence, abiding by procedural cautions and limitations, selecting correct paths on decisions blocks, and correctly transitioning between procedures? 3 2 1 Accurately and Minor errors, but Significant errors ! i promptly executed made necessary impeded or slowed

  'y/            procedural steps               corrections in              recovery or degraded          ;

a timely fashion plant unnecessarily [.50 X = ] (c) Ensure the safe, efficient IMPLEMENTATION of procedures BY THE CREW 7 3 2 1 Kept crew informed of Crew occasionaCy had Read procedures to procedural status; got to question SRO him/herself; failed acknowledgment from regarding status; to coordinate or verify crew when reading allowed lapses in crew's use of procedures procedures implementation by crew [.25 X = ] Pi 23 of 27 NUREG-1021, Revision 8

~~

ES-303 5 Form ES-303-4

5. OPERATE THE CONTROL BOARDS l l

DID THE APPLICANT: 1 I (a) LOCATE CONTROLS efficiently and accurately? 3 2 1 I Promptly located Some minor hesitancy or Unable to locate l appropriate controls difficulty in locating controls without j in allinstances controls assistance l 1 [.25 X = ] (b) MANIPULATE CONTROLS in an accurate and timely manner?

)!

3 2 1 I Manipulations were Minor shortcomings, improper manipulations l consistently accurate but any resulting caused major system j and timely consequences were perturbations readily mitigated (c) ACT appropriately in response to INSTRUMENT READINGS? 3 2 1 Responses were Generally responsive, Failed to react appropriate and but some minor appropriately to timely errors and lapses instrument readings without assistance [ 25 X = ] (d) Take MANUAL CONTROL of automatic functions when appropriate? 3 2 1 Took manual control Minor delays; some Depended on automatic as appropriate prompting necessary actions; required before overriding prompting to take automatic functions manual control [.25 X = ] NUREG 1021, Revision 8 24 of 27

ES-303 6 Form ES-303-4 (

6. COMMUNICATE AND INTERACT WITH THE CREW AND OTHER PERSONNEL DID THE APPLICANT:

(a) Communicate in a clear, easily-understood manner? 3 2 1 Communications were At times, communi- Communications were timely, clear, and cations were ill-timed, vague or easy to hear and confusing, hard to difficult to hear or understand hear or understand understand [.45 X = ) (b) Keep crew members and those outside the control room informed of plant status? 3 2 1 Provided others with Had to be prompted for Failed to provide accurate, pertinent information in some minor important information information throughout instances; gave some C scenarios incomplete or inaccurate information

[.35 X ] (c) ENSURE RECEIPT of clear, easily-understood communications from crew and others? 3 2 1 Requested information Failed to require Failed to request needed or clarification when or acknowledge infor- information; inattentive necessary; understood mation from others when information was communications from provided; failed to others correct serious misunderstandings among crew members [.20 X = ] , 25 of 27 NUREG-1021, Revision 8

7 ES-303 7 Form ES-303-4

7. DIRECT SHIFT OPERATIONS DID THE APPLICANT:

(a) Take TIMELY and DECISIVE ACTION when problems arose? 3 2 1 Took early remedial Minor instances of Failed to take timely action; action when necessary failure to take action resulted in deterioration of within a reasonable plant conditions period of time [.20 X = ] (b) Provide TIMELY, WELL THOUGHT OUT DIRECTIONS that facilitated CREW PERFORMANCE and demonstrated appropriate CONCERN for the SAFETY of the plant, staff, and public? 3 2 1 Directives enabled Minor instances of Directives inhibited safe safe, integrated irmorrect, trivial, performance; crew had crew performance or difficult-to-carry- to explain why orders could out orders not or should not be followed [.40 X = ] (c) Stay in a position of OVERSIGHT and provide an APPROPRIATE AMOUNT of DIRECTION and GUIDANCE 7 3 2 1 Stayed involved but Crew occasionally had to Lost the big picture; crew not intrusive; request assistance, had to repeatedly request anticipated crew's which interfered or provide guidance; needs and provided with their ability to failed to verify that directives guidance when carry out actions were correctly implemented necessary [.20 X = ] (d) SOLICIT and INCORPORATE FEEDBACK from the crew to foster an effective, team-oriented approach to problem solving and decision making? 3 2 1 Involved crew in At times, failed to involve Made decisions without problem-solving process crew in decision making crew participation or con-as appropriate, leading when it would have been sultation; crew divisiveness to effective team appropriate, detracting from was counter productive decision making team-oriented approach [.20 X = ] NUREG-1021, Revision 8 26 of 27

ES-303 8 Form ES-303-4

8. COMPLY WITH AND USE TECHNICAL SPECIFICATIONS !

DIO THE APPLICANT: I (a) RECOGNIZE when conditions were covered by technical specifications (TS)? 3 2 1 Recognized TS Minor errors and Failed to correctly limiting conditions for misunderstandings recognize situations l operation and action with respect to covered by TS and statements without use TS applications action statements of references [.40 X = ] (b) LOCATE the appropriate TS quickly and efficiently? 3 2 1 Located applicable Had difficulty locating Could not locate TS quickly and TS; had to search appropriate TS accurately through index and body p of document [.20 X = ] (c) Ensure correct COMPLIANCE with TS and LCO action statements? 3 2 1 Directives were based Needed some assistance Applied incorrect TS on correct under- from crew to ensure to situation; allowed standing of TS compliance crew to violate TS action statements [.40 X = ] p. 27 of 27 NUREG-1021, Revision 8

 %.w..

O ES-401 h PREPARING INITIAL SITE-SPECIFIC WRITTEN EXAMINATIONS A. PURPOSE This standard specifies the requirements, procedures, and guidelines for preparing site-specific written exam! nations for the initial licensing of reactor operator (RO) and senior reactor operator

   '(SRO) applicants at power reactor facilities.

B. BACKGROUND The content of the written licensing examinations for ROs and SROs is dictated by 10 CFR 55.41 and 55.43, respectively. Each examination shall contain a representative selection of questions conceming the knowledge, skills, and abilities (K/As) needed to perform duties at the desired license level. The written operator licensing examination is administered in two sections, including a generic fundamentals examination (GFE) and a site-specific examination. The GFE covers those K/As that do not vary significantly among reactors of the same type (i.e., pressurized or boiling water) and is generally administered early in the license training process (refer to ES-205 for a description of the program). The instructions in this standard apply only to the site-specific examination. Except as noted in Section D.1.b, the " Knowledge and Abilities Catalog (s) for Nuclear Power Plant Operators: Pressurized [and Boiling) Water Reactors," NUREG-1122 and -1123, O . respectively, provide the basis for developing content-valid licensing examinations. Each K/A stem statement has been linked to the applicable item number in 10 CFR 55.41 and/or 55.43. Preparing the license examination using the appropriate K/A catalog, in conjunction with the instructions in this NUREG, will ensure that the examination includes a representative sample of the items specified in the regulations. C. RESPONSIBILITIES

1. Facility Licensee The facility licensee will perform the following activit;es, as applicable, depending upon the examination arrangements confirmed with the NRC regional office (in accordance with ES-201) approximately four months before the scheduled examination date:

a. Prepare the proposed examination outline (s) in accordance with Section D.1,

                     - and submit the outline (s) to the NRC regional office for review and approval in accordance with ES-201.

b. Submit the reference materials necessary for the NRC regional office to prepare and/or validate the requested examination (s) (refer to ES-201, Attachment 2).

c. Prepare the proposed examination (s) in accordance with Sections D.2 through D.4, review the examination (s) in accordance with Section E, and submit the examination (s) to the NRC regional office in accordance with ES-201.

1 of 45 NUREG-1021, Revision 8 i I

d. Meet with the NRC in the regional office or at the facility, when and as necessary, to review the proposed examination (s) and discuss potential changes (refer to ES-201).

e. Revise the proposed examination outline (s) and examination (s) as agreed upon with the NRC regional office; however, the NRC retains final authority to approve the examination.

2. NRC Realonal Office The NRC regional office wili perform the following activities:

a. Ensure that the examinati'ans are prepared in accordance with Section D.

b. Ensure that the examinations are reviewed for quality as described in Section E.

c. Meet with the facility iicensee, when and as appropriate, to prereview the examination (s) in accordance with ES-201.

D. EXAMINATION PREPARATION

1. Develoo the Outline Develop each written examination outline in accordance with the following general instruction ,. ;

1

a. Select the appropriate examination outline model for the licensing examination being developed (i.e., RO or SRO, BWR or PWR) from Forms ES-401-1 through ES-401-4; Form ES-401-5, " Generic Knowledge and Abilities Outline," applies to all examinations,

b. Systematically select K/As from NUREG-1122 (for PWRs) or -1123 (for BWRs) to complete each of the three tiers (i.e., Tier 1, Emergency and Abnormal Plant Evolutions; Tier 2, Plant Systems; and Tier 3, Generic Knowledges and Abilities) of the examination outline, in order to maintain examination consistency, the l facility licensee's site-specific K/A list shall not be used in place of the K/A 1 catalog. Attachment 1 provides an example of an acceptable methodology for selecting 100 K/As within the defined structure of the examination outline. Other methodologies may be used provided they are reproducible and scrutable and yield an examination outline that is free of bias and adheres to the applicable examination model. The examination author may be requested to explain to the NRC chief examiner the systematic process thst was used to develop the i examination outline.

Distribute the K/As among the three tiers as specified for the applicable outline, select topics from as many different systems and evolutions as possible, and distribute the topics among the K/A categories (e.g., K1 through K6, A1 through i NUREG-1021, Revision 8 2 of 45

OG ES-401 A4, and G, or generic, for Tier 1), with at least two topics from each category applicable to each tier of the outline. This will help ensure that a valid sample is maintained even if selected questions have to be deleted during the grading process. Avoid selecting more than two or three topics from a given system or evolution unless they are related to a plant specific priority (refer to item (c) below. The topics for the generic K/A category in Tiers 1 and 2 (i.e., Column "G" on Forms ES-401-1 through ES-401-4) and the four K/A categories in Tier 3 (i.e., l

                        " Conduct of Operations,"" Equipment Control," Radiation Control," and                                           l
                        " Emergency Procedures / Plan") shall be selected from Section 2, " Generic Knowledges and Abilities," of the applicable K/A cataleg. However, only those topics that are relevant to the selected evolution or system shall be included in the sample for Tiers 1 and 2.

l If the systematic selection process identifies a K/A topic having an importance rating that is below 2.5 or a K/A category that contains no K/A topics, systematically select another K/A category and topic. K/A topics with importance ratings below 2.5 may be justified on the basis of plant-specific priorities. Enter the K/A numbers, a brief description of each topic, the topics' importance ratings for the license level of the exam, and the point totals (system, category, group, and tier) on the examination outline. ( Ensure that the outline used during successive audit and licensing examinations does not become repetitive and predictable. If a facility licensee proposes to use an outline that was previously used at another facility, it shall identify the source of the outline and explain what effect its reuse is expected to have on examination integrity.

c. After systematically selecting 100 K/As for the examination, the examination author may identify up to ten additional K/As based on the facility licensee's site-specific task list or other plant-specific, high-priority topics (e.g., operating events or problems, PRA-identified risk-important systems and operator actions', and recent technological developments) that are appropriate for testing on the written examination. Enter the applicable information in the space provided at the bottom of Tier 2, Group 3 of the examination outline (i.e., the system / topic, an indication of which systematically selected K/A the plant-specific priority topic will replace, a brief explanation for making the substitution, and the proposed number of examination points applicable to the topic).

d. After completing the outline, check the celected K/As for balance of coverage within and across the three tiers. Ensure that every applicable K/A category is

        ' Chapter 13 of NUREG-1560. ' Individual Plant Examination Program: Perspectives on Reactor safety and Plant Performance.' identifies a number of risk-important human actions that may be appropriate for examination. In determining important operator actions. do not overtook actions that are relied upon or result in specific events being driven to low nsk contribution. This will help identify those human actions, assumed to be very reliable, that might otherwise not show up in a list of riska$ominant actions.

3 of 45 NUREG-1021, Revision 8

l l j ES-401 3 sampled at least twice within each of the three tiers and that the outline for an SRO examination adequately samples the additior/J topics required by 10 CFR 55.43(b). Make any adjustments that might be necessary by systematically selecting replacement K/As. Also check the overall balance of the entire i licensing examination, including the walk-through and the dynamic simulator test, and make any necessary adjustments. ; i

e. Review and submit the completed outline to the NRC chief examiner for review and approval in accordance with ES-201. Facility-developed outlines shall be l independently reviewed by a facility supervisor or manager before being submitted to the NRC regional office in accordance with ES-201. Facility j licensees are responsible for ensuring that contractor-prepared outlines meet the j guidelines herein. The NRC must receive the outlines by the date agreed upon when the examination arrangements were confirmed (normally approximately 75 days before the scheduled examination date).

f. The NRC chief examiner will review the outline within five working days and )

provide comments and recommended changes, as appropriate. The chief examiner shall approve the site-specific item or topic substitutions.

2. Select and Develoo Questions i f

I

a. Prepare the site-specific written operator licensing examination using a combination of existing, modified, and new questions in accordance with the previously approved examination outline (refer to item D.1 and ES-201) and the '

criteria summarized below. l If it becomes necessary to deviate from the previously approved examination l outline, discuss the proposed deviations with the NRC chief examiner and obtain { concurrence. Be prepared to explain why the original proposal could not be I implemented and why the proposed replacement is considered an acceptable I substitute.

b. Ensure that each question is technically accurate and free of the following psychometric flaws that could diminish the validity of the examination. Appendix B provides a detailed discussion and examples of questions containing each of these and other errors; the parenthetic references identify the applicable sections of Appendix B and its Attachment 2. Appendices A and B contain more detailed instructions and guidelines for preparing and formatting content-valid examinations and should be referred to as necessary while preparing the examination.

implausible distractors (C.2.g, h, k; D) confusing or ambiguous language (C.1.c; E) l confusing or inappropriate negatives (C.2.e; E.3) l - collection of true/ false statements (C.2.c; F) backward logic (C.1.h; G) specific determiners (C.2.m) NUREG-1021, Revision 8 4 of 45

p ES-401 l

-Q

?

c. Ensure that the questions will differentiate between competent and less-than-competent applicants, that they are appropriate for the job level being examined, l and that they are operationally oriented when possible. Refer to Appendix A )

(Section C.2) and Appendix B (Section C.1.a and Section B of Attachment 2) for l additional discussion of and examples to illustrate the concept of operational validity. Establish a level of difficulty that discriminates between applicants who have and have not mastered the required knowledge, skills, and abilities. See Appendices A (Section C.3) and B (Section C.1.e and Section C of Attachment 2) for further ) guidance on sotting individual test question level of difficulty. The applicant I should be able to complete and review the examination within five hours. J l In order to maintain examination quality and consistency, between 50 and 60 percent of the questions on the examination shall be written at the { comprehension / analysis level. The cognitive level of any question drawn directly j from a bank will be counted at its face value. Refer to Appendix B (Section C.1.d ) and Section A of Attachment 2) for further guidance regarding the levels of j knowledge and sample questions written at each level.

d. When both RO and SRO examinations are to be given at the same time, duplicate no more than 75 percent of the RO examination questions on the SRO i examination, even though the RO and SRO examination outlines differ by only l

11 points (i.e., SROs have 11 fewer points in Tier 2,7 additional points in Tier 1, ' and 4 additional points in Tier 3). The 25 SRO-level questions shall evaluate the additional knowledge and abilities required for the higher license level per 10 CFR 55.43(b) or the facility licensee's teaming objectives and should be distributed among the three tiers of the examination. Questions related to 10 CFR 55.41(b) topics may also be appropriate SRO-level questions if they evaluate knowledge and abilities at a level that is unique to the SRO job position. Similarly, SRO examinations that are not developed in conjunction with an RO examination shall also include at least 25 questions that evaluate knowledge and abilities required for the SRO license level per 10 CFR 55.43(b) or the facility licensee's learning objectives. The remaining questions may be appropriate for either license level.

e. All test questions shall be in the multiple choice format described in Appendix B.

Each question shall have four possible answer choices and be worth one point. l f. To avoid compromising the integrity and security of the examination and to enhance consistency, observe the following limits on question repetition and l bank use when preparing the examination: Repeat no more than 25 questions on the examination (or examination , set if an RO and SRO examination are prepared concurrently) from i examinations, quizzes, or tests administered to the license applicants O 5 of 45 NUREG-1021, Revision 8 1 __]

during their license training class or from the past two licensing examinations at the facility (an RO and SRO examination given at the same time count as ons examination). The facility test / quiz limit does not apply to NRC-developed examinations because those materials are generally not available to NRC examiners. Facility-written examinations shall repeat no questions directly from the applicants' audit examination (or examinations in the case of retake applicants) or similar testing vehicle given at or near the end of the license training class, unless the two examinations are written independently (i.e., no interface between the examination authors). In such cases, up to five questions may be coincidentally duplicated, and the facility licensee shall identify the duplicates to the NRC chief examiner. Take no more than 50 percent of the questions for the examination directly from the facility licensee's written examination question bank without significant modification. Questions that the facility licensee has obtained from another bank and deposited in its own bank may be treated as " bank" questions provided they have an equal chance cf being selected for use on the examination. Write at least 10 new questions at the comprehension and analysis level, as described in Appendix B. Questions from another bank may be treated as new items if they have not been made available for review and study by the license applicants and there is no basis (e.g., historical precedent or reciprocal arrangements with the other facility licensee) for the applicants to predict their use on the examination. ! Select the remaining questions for the examination from the facility l licensee's bank, but significantly modify each question by changing the j pertinent conditions in the stem and at least one distractor. The intent or ! objective of the question does not necessarily have to be changed, but adding or deleting irrelevant information and making minor changes (e.g., the unit number, component train, or power level when it makes no difference) would not be considered a significant modification to the i question. j

g. A technical reference and a cross-reference to the facility licensee's examination i question bank, if applicable, shall be noted for every question. If the facility licensee has a learning objective applicable to the question, it should be referenced as well. However, the absence of a learning objective does not I invalidate the question provided it has an appropriate K/A and technical l reference. Refer to ES-201 for additionalinstructions regarding the l documentation of the source of questions on facility-written examinations.

l l NUREG-1021, Revision 8 6 of 45 9 i

ES-401

To facilitate the review process, examination authors should consider providing a brief explanation of why the answer is correct, and each of the distractors is plausible but incorrect. This optionalpractice increases the efficiency of the examination review process and promotes the detection and correction of problem questions before the examinations are administered. ) l Reference materials such as diagrams, sketches, and portions of facility procedures may be used on a selective basis as attachments to the written examination. Ensure that any reference material used in the examination is easy to read and clearly marked, provides an effective and objective way for the applicant to demonstrate knowledge of the topic or concept, and does not give away the answers to other questions on the examination. Form ES-401-6 is a sample worksheet for use in preparing the written examination questions. Facility licensees may use that or a similar form to document the information related to each proposed question that is submitted to the NRC for review and approval.

3. Review and Submit the Examination

a. Review the entire examination to ensure that the criteria on Form ES-401-7,

              " Written Examination Quality Checklist," are satisfied.

O i / b. Forward the examination package, including all proposed attachments, and the completed quality checklist to the first reviewer. Section E provides instructions for conducting the quality reviews. Facility-developed examinations must be reviewed by a supervisor or manager before they are sent to the NRC regional office in accordance with ES-201. Facility authors shall submit their examinations for management review in time to support their delivery to the NRC regional office approximately 45 days before the scheduled examination date. NRC examiners shall submit their examinations to the chief examiner for review at least one week before the scheduled prereview by the facility licensee (refer to l ES-201).

4. Assemble the Examinations

a. Format the examinations using the one-question-per-page layout specified in Appendix B or by placing as many complete questions as possible on each page,

b. Use a cover sheet in the format shown in Form ES-401-8, " Site-Specific Written Examination Cover Sheet," for all RO and SRO written examinations. Fill out all items in the upper section of the cover sheet, except the name of the applicant, p when preparing the examinations.

7 of 45 NUREG-1021, Revision 8

ES-401 E. QUALITY REVIEWS When reviewing questions, reviewers should try to put themselves in the position of the applicants by attempting to answer the questions without using reference material or referring to the answer key. Reviewers should ensure that the conditions and requirements posed in the question are complete and unambiguous, all necessary information is provided, all unnecessary information is deleted, the intended answer clearly follows from what is asked in the question, and the distractors are plausible.

1. Facility Manaoement Review if the examination was prepared by the facility licensee, it shall be independently reviewed by a supervisor or manager before it is submitted to the NRC regional office for review and approval in accordance with ES-201. The reviewer should evaluate the examination using the criteria on Form ES-401-7 and include the signed fonn in the examination package submitted to the NRC. Facility licensees are responsible for ensuring that contractor-prepared examinations meet the guidelines herein and are encouraged to verify the origin of the questions used to construct the examination.

2. NRC Examiner Review

a. The NRC regional office staff shall review the examination as soon as possible after receipt so that supervisory approval can be obtained before the final review ,

with the facility licensee, which is normally scheduled about two weeks before the examination date. It is especially important that the region promptly review examinations prepared by a facility licensee because of the extra time that may be required if extensive changes are necessary. The chief examiner shall consolidate the comments from all NRC reviewers and submit one set of comments to the author or facility contact.

b. If the NRC prepared the examination, the NRC chief examiner shall independently review all examination questions for content, wording, operational validity, and level of difficulty. As a minimum, the chief examiner shall check the ,

items listed on Form ES-401-7. Item 4 on the quality checklist applies only to the 4 last two NRC licensing examinations at the facility. Moreover, item 5 and the facility reviewer blocks in Column "b" are not applicable for NRC-prepared examinations. If the chief examiner wrote the examination, another NRC examiner must perform the independent review.

c. If the facility licensee developed the examination, the licencee is primarily responsibie for ensuringcompliance with the items listed or Form ES-401-7.

However, the regional office staff is expected to take reasonable measures, including the selective review of reference materials, individual questions, and past examinations, to verify these items when reviewing the examination; i exclusive reliance on the facility author's and reviewer's initials is not adequate. Depending upon the expected technical quality of the examination and the time available before the scheduled review with the facility licensee, the regional NUREG-1021, Revision 8 8 of 45

F ES-401

 'u)     office staff shallindependently review and verify the technical accuracy of a
        - sample of the written examination questions.

With regard to assessing the psychometric quality of the proposed examination questions, the regional office shall begin by systematically selecting a sample of 1 questions for detailed review. The sample is based on the nominal 50/40/10 i (bank / modified /new) question distribution discussed in Section D.2.f above and the question background information provided by the facility licensee (using I Form ES-401-6 or similar method). The sample shallinclude 10 of the new questions on the examination and 20 additional questions that are randomly selected from among the remaining questions that have not been prevalidated through successful use on an NRC licensing examination administered at that facility since October 1,1995. The regional offics shall conduct and document the review of the 30 selected questions using Form ES-401-9," Written Exam; nation Review Worksheet." When the sample review is complete, the chief examiner shall consult with the responsible supervisor and proceed as directed to evaluate the remainder of the examination.

d. There are no minimum or maximum limits on the number or scope of changes the regional office may direct the author or facility contact to make to the proposed examinations, provided that they are necessary to make the examinations conform with established acceptance criteria. All uriacceptable I flaws identified by using Form ES-401-9 shall be corrected by rewriting or replacing the questions before the examination is administered. Other flaws of a minor nature (e.g., editorial clarifications or enhancements) should, as time -

permits, be corrected before the examination is administered, but the NRC expects such corrections to be made before the question is reused on another NRC examination.

e. Upon supervisory approval, generally at least 14 days before the examinations are scheduled to be given, the chief examiner will review the written examinations with the fa::ility licensee in accordance with ES-201.

When providing feedback to the facility licensee regarding unacceptable questions, the chief examiner shall, at a minimum, explain how the Appendix B psychometric quantitative and qualitative attributes are not being mot. For example,if the question is determined to have more than one irnplausible distractor, the attendant explanation shall articulate the reasons the examiner believes each of the faulty distractors is not credible. Examinations that aro written by the NRC shall be clean, properly formatted, and

           " ready-to-give" before they are reviewed with the facility licensee. The region shall not rely on the facility licensee to ensure that the quality of the examination is acceptable for administration.

f. Af ter reviewing the examination with the facility licensee, the chief examiner will O

g- 9 of 45 NUREG-1021, Revision 8

I I ES-401 i ensure that any comments and recommendations are resolved and the l examination is revised as necessary. If the facility licensee developed the examination, it will generally be expected to rnake whatever changes are recommended by the NRC. 1

g. After the necessary changes have been made and the chief examiner is satisfied with the examination, he or she will sign the quality checklist and forward the examination package to the responsible supervisor for final approval. If the examination was written by the facility licensee, the chief examiner should include a copy of the original submittal with the examination package.

I

3. NRC Supervisorv Review 1l I

a. The responsible supervisor shall review all questions determined to have unacceptable flaws in accordance with Form ES-401-9 before any comments are provided to the facility licensee. The responsible supervisor shall review the entire examination before authorizing the chief examiner to proceed with the facility prereview per ES-201. The supervisory review is not intended to be another technical review, but rather a general assessment of examination quality, including a review of the changes being recommended by the chief examiner, and a check to ensure that all the applicable administrative )

requirements have been implemented.

b. Based on the results of the sampling review conducted in accordance with ;

Section E.2.c above, the responsible supervisor (in coordination with regional j management and the NRR operator licensing program office, as appropriate) will continue the examination review as follows: If fewer than six of the 30 sampled questions contain unacceptable flaws as determined by using Form ES-401-9, then the regional office shall review in detail the remainder of the examination (excluding those questions that were prevalidated by the NRC) using Form ES-401-9 and provide comrnents to the facility licensee for rework and correction. The NRC-validated questions need not be reviewed in detail but will be evaluated as necessary to complete Form ES-401-7 (including the identification and correction of technical and psychometric flaws that cause the question to have no or multiple correct answers) before reviewing the examination with the facility licensee. The responsible supervisor will review and approve each comment that would require the facility licensee to rework an NRC-validated question. If six or more of the 30 sampled questions contain unacceptable flaws as deterrr.ined by using Form ES-401-9, then the regional office may return the written examination (with explanatory comments) to the facility ! licensee for rework and correction without reviewing the remainder of the examination (refer to Section C.2.h of ES-201 for additional guidance regarding examination delays). The facility licensee will be expected to correct the unacceptable flaws in the sampled questions and like-kind l flaws that exist in the remainder of the examination. When the facility j NUREG 1021, Revision 8 10 of 45 l l ____________.______.___.____________________________.______.______.__________._.________________________________________o

p D ES-401 o- , licensee resubmits the examination, every question (excluding the NRC- l validated questions) will be subject to NRC review using Form ES-401-9. i The NRC-validated questions will be reviewed as discussed above. l [ Altematively, if the responsible supervisor concludes that the remainder l of the examination (excluding the NRC-validated questions) can be l l reviewed and corrected in time for the scheduled examination date, the I l regional office should continue the review using Form ES-401-9 and provide comments to the facility licensee for correction.

                                  .                                                                          I l           c.     ' Tha responsible supervisor should ensure that any significant deficiencies in the        !

original examinations submitted by a facMy licensee are evaluated in j (' accordance with ES-201 to determine the appropriate course of action. At a minimum, the supervisor should ensure that they are addressed in the final examination report in accordance with ES-501.

d. Following the facility review, the responsible supervisor should again review the examination to ensure that the concems expressed by the facility licensee and i the NRC have been appropriately addressed. The supervisor shall not sign l Form ES-401-7 until he or she is satisfied that the examination is acceptable to j be adminictered.

4. Facility Peer Review 5

As a final check of the examination's technical accuracy, facility management

should consider administering the NRC-approved examination (under security agreements) to one or more licensed perso..nel who were previously uninvolved in developing the examination. As discussed in Section D.2.d of ES-201, the NRC regional office may deny the facility licensee's proposal to use certain individuals (e.g., the applicants' supervisors or coworkers) to validate the i examination. Any comments made and problems identified during the trial l

administration shall be discussed with the NRC chief examiner and resolved before the examination is administered to the license applicants. The intent of the review is to identify and correct deficiencies that may affect the validity of the examination. F. ATTACHMENTS / FORMS Attachment 1, " Example Systematic Sampling Methodology" Form ES-401-1, "BWR SRO Examination Outline" Form ES-401-2, "BWR RO Examination Outline" Form ES-401-3, "PWR SRO Examination Outline" Foim ES-401-4, "PWR RO Examination Outline" Form ES-401-5, " Generic Knowledge and Abilities Outline" Form ES-401-6, " Sample Written Examination Question Worksheet" Form ES-401-7, " Written Examination Quality Checklist" Form ES-401-8, " Site-Specific Written Examination Cover Sheet" Form ES-401-9, " Written Examination Review Worksheet" 11 of 45 NUREG-1021, Revision 8

f I l ! ES-401 Example Systematic Sampling Methodology Attachment 1 The following process, which uses the BWR SRO outline (Form ES-401-1) for illustration, may be usedfor each group in Tiers 1 and 2 of the examination outline.

1. Review each group and delete those items (Emergency / Abnormal Plant Evolutions (E/ APES) for Tier 1 and systems for Tier 2) that clearly do not apply to the facility for which the examination is being written; be prepared to explain the basis for the deletions to the NRC chief examiner.

2. Sequentially number the remaining items in the group and sequentially annotate the same number of tokens. If we assume that none of the 20 E/ APES in Tier 1, Group 1 was deleted in Step 1, there should be 20 tokens, numbered from 1 to 20.

1 I

a. If the number of items remaining in the group (in this case f.o; smaller than the required number of points for the group specified in the rigt ' .id column of the excmination outline (in this case 26), then each item in the ; oup would be sampled at least one time. The rest of the sample would be determined by randomly selecting and removing tokens (in this case 6 of the 20) until the required total number of points is reached. Update Form ES-401-1 to note the selected items.

b. If the number of items remaining in the group is larger than the required number of points for the group (e.g., Tier 1, Group 2 has 20 items but only requires 17 points), then randomly select and remove the required number of tokens and note them on Form ES-401-1.

3. After selecting the topics to be sampled in each group as described in Step 2, count the number of K/A categories in the group (e.g.,6 for each group in Tier 1; i.e., K1, K2, K3, A1, A2, and G) and sequentially annotate the same number of tokens (in this case 6).

For each E/ APE (and system) selected in Step 2, randomly select and remove a token end note the K/A category on Form ES-401-1. If the E/ APE (or system) was sampled more than once per Step 2.a randomly select a second K/A category. If the selected K/A category contains no K/A statements having an importance rating that is above 2.5, systematically select another K/A category, unless the lower importance is justified bssed on plant-specific priorities. Then replace all the tokens in the container and repeat the process for every selected item in each group.

4. Use a similar method to randomly select from among the K/A statements under each selected K/A category. Describe each K/A topic in the space provided on Form ES-401-1 and enter the importance rating. K/As having importance ratings less than 2.5 can be used if justified based on plant priorities; the facility contact should be prepared to explain the basis to the NRC chief examiner.

For Tier 3 (Plant-Wide Generics) of the examination outline, randomly select K/As from Section 2 of the NRC K/A catalog so that each of the four K/A categories (i.e.," Conduct of Operations,"

 " Equipment Control," Radiation Control," and " Emergency Procedures / Plan") has at least two items.

NUREG 1021 Revision 8 12 of 45

ES-401 BWR SRO Examination Outline Form ES-401-1 Facility: Date of Exam: Exam Level: K/A Category Points Tier Group Point K K K K K K A A A A G Total 1 2 3 4~ 5 6 1 2 3 4 rp m- w - 1, 1 w& :1 C, 2 ;m 26 Emergency & ,3 -j y; g. g Abnormal 2 m a cg se q 17 Plant .we e m xm n+ Evolutions Tier Totals yt re i" L pM L" . y! M. 43 1 23 2. 2 13 Plant Systems 3 4 Tier 40 Totals e 3. Generic Knowledge and Abilities Cat 1 Cat 2 Cat 3 Cat 4 17 Note: 1. Ensure that at least two topics from every K/A category are sampled within each tier (i.e., the " Tier Totals" in each K/A category shall not be less than l two).

2. Actual point totals must match those specified in the table.

3. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specic priorities.

4. Systems / evolutions within each group are identified on the associated outline.

5. The shaded areas are not applicable to the category / tier.

6.* The generic K/As in Tiers 1 and 2 shall be selected from Section 2 of the DA Catalog, but the topics must be relevant to the applicable evolution or system.

7. On the following pages, enter the K/A numbers, a brief description of each topic, the topics'importance ratings for the RO license level, and the point totals for each system and category. K/As below 2.5 should be justified on the basis of plant-specific priorities. Enter the tier totals for each category in the table above.

13 of 45 NUREG-1021, Revision 8 i!

1 1 t s 0 n 6 4- io 2 S E F m r o P I p m O

( s c ip o T A/ K 1 p u o r G

1 r ie : la eT n-t o is lt T un t n Oio t io nlu P oo iv p taE u 5 n o 4 O t r imn G f ala o xP El 4 a G 1 Om Rr Son Rb 2 WA A Ddn a 1 y c A n e g r 3 e K m E 2 K 1 K 5 6

              /                                   8          /.p    5       r e       5 n

r w

e 5

t e w o

                                                                                    /.c o                                                   m      v P               5   1               d            e /

e P1 n it / 7 o T e o c n C /.p n / 9 M r r u L d/ nn 9 C u A 2 o t g e t r e 2 aw n F f m it n / e ta a t

e y o 3

e i d e t in 3 r e a / to nn te g t e s s / le v 5 T d A m n R a lo 5 / W p W i e v ek R a o r 8 o e e

e lo 1 o o / e e h m l e sn d fa L r u L r o y / d e C r r u ig e o o L eU r e y n S e s r u s P it iv M n s a ts u s s H T P r Pr s a H o

          /   t e    s  e   s  n      A     a  e   n   s   s     t te  no   e  t n      is e                                         o  n   n l

e t a e o tc b e e e a oe lp r r a R A le d r P r o e o til le e i v m P Wr P is e C R P P W ia R a m r s R S m i c c r n m is s dc ns m n e N o o o l le e o A

                                                      !e   o     in   e   w on   e        :

s R C 1 t c r p t n e o ite t c o a r o it ia la

r / a t c w p e t R s g y w a is t n p L Cwo s- t n , o e a y u t le f in e s o p r f o to 1 E M R e D r S r e p l o f O D r R e C u o MD f O C T 2 A R S A l P a v m r le r p t c y A it r R h w h h d o t n h f u h h p h w a RM h h o r 0

          /    a  C ig  o ig  ig    a n

c n o ig e ig ig u ig o e CR ig ig g 1 O E P S H L H H I I C H R H H S H L R SPA H H te 1 3 6 7 9 0 3 4 5 6 7 3 4 5 6 7 0 1 7 8 0 a G 0 4-0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 0 1 0 2 0 2 0 2 0 2 0 2 0 3 0 3 0 3e 0v 3 0 0 0 C E S 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5o 9b 5 9 0 0 A

R E 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2A 2 5 K U N

1 8 - s n o 1 t n .x 0 4-S io P 7 1 is _e E iv e m r o p

                   .                                                                                                  R F         m                                                                                                      1 I

2 0 1 G _ E R _ U N _ ) ( s ic p o T A

K 2 p u o r G

1 w l: eT ta o _ n- T hs t uno t n _ Oitu io nl P . oo iv p _~ t eE u 5 _ n t o r 4 inn G f . / ra al o xP _ El 5 _ a G 1 Om Rr _ Son Rb 2 WA A Bdn a 1 y c A - n e g r 3 e K n r E 2 K 1 K - l a . e r h ig a it e r o 5 n A C 3

                                                                           /                H       e  /

p n d / 5 l e a a n fe r e 6 8 7 v e e o m io c m /

                                         /
                                                   /                         e   r     r    it    f i     u a    D t

c r o r r 3 e r 8 r & L A A S n i i

                                                /                            r              lt u  F           w /

2 u 5 A 5 4 5 e t n t n n h h e P ip t a / W . / / / t e e e ig ig F f o V r / C t n e a V H H C T le r e er s n o g r W m m ty e s s r e D r v p u C i it in 1 u n n t n t n t n . f o s. f o o e m t a f o f o a lo t a lo ia ia e e e a L t a L e r lo o / s r o t n t n m m m 8 . S s r r T e s s s C p e P o o n n n / e s e e p s s p : o C C e s - I

e t e o n t a t n m o o . n m m n ia ia ia lp l e e e L L t n w u e io y y t n t n t n it S la m a m 4 C la G Wr m T la la o d o P T s s r a r a o o o n t o - N o& n t o e o n l t o t o C D l e d C C C T C1 ia n ia le T t tu R l e r n 4-.9 y y y O t n

                 /             T   ib t

c t w T n h w p o o. / r r r e

                 #    r/   M     r o  r  a  n   y  r o

r o te S C y p c c els a a a io E ono f o T u R e C o D r r e f o f o D r S u e5 S/ Sev d n d n d5n ir F P P lai la la la v o9 o o/ t n y A it t r a s it n h h h it it d s s s s h h he gr he c c5 cle a r o

E alu s o r a ia ig ig ig r a r a a n o o ig H ig H iu Hta igL Hno e/ e/ Ser Se ev l P g Pc L P M H H H P P L L Sn e 4i o I r 1 1i 2 4 5 8 1 2 8 9 0 1 2 8 9 2e r 3it 5u 6L 0 t a 0 0C 0 0 0 0 1 1 1 1 2 2 2 2 2 0 3p 3a 0i 3a 0i t 3s 0s 3r 0e 0 0 C 4- 0wo 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 5 0m 5 5d 5d 5e 5a t 0 A S 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9e 9a 9a 9r 9 2W 0 / K E 2R 2 2 2 2 2 2 2 2 2 2 2 2 2 2T 2R 2R 2P 6

1 1 t s 0 n 3 O 4 io 2 S P E n ac . p F I m

( s ic o o T A i : K la t o T t n io P p u o r G G 4 e1 A in> u ltuo 3 Or A nG/ o2 itr 5 ae ni A 2 iT m-4O f as x t o Eac A 6 1 Oi Ry s SS 6 Rtn K W!a BP 5 K 4 K 3 K 2 K 1 K f r fo t o r t e u la s h e u n n s S g e io ie y d o e lo e d t r lp R r lo n ta ia p M e t d o o n y r n n r u o 8 l e t n C e i x a u o m o M ) y r o m u S p r s s C it u n a C n c it u A m S e le b o N o n n r d a r v i r :

      /      w it c    e         o    t s

n n a te T P e t is T s iS

      #     lo   e    g         M                S   M      e L          M la  i v

F j n r e e i T T n r D t o m n I m g M R le r M r a C ib t e la T C T e e ts o : E n a P io T e : r u a ic y t n R lc l I y it a C ( R L B C u C T

                                                           /  Wr     r   r a io S     lu P      n            /   r       y     P      r      tc    d P 1

c io e a r N L o o le n 2 S r i I t S S c M e C a ! S / s t c t c S E s o y ic R C a C C C S r u R lc S m R V a a T G c r 0 I C e H P lo P P L P o P u I C D ir I C H R e e G C D e g o 1 R R R H Is H S R S A N R A P P R S R R A S - S E 1 0 4-5 0 0 1 0 2 2 0 0 2 0 2 0 0 0 3 0 2 0 0 0 6 0 2 0 0 0 7 0 2 L 1 0 0 9 0 2 2 0 0 9 0 2 0 0 0 1 1 2 0 0 0 2 1 2 4 0 0 5 1 2 5 0 0 5 1 2 0 0 0 6 1 2 0 0 0 7 1 2 0 0 0 8 1 2 1 0 0 3 2 2 2 0 0 3 2 2 1 0 0 6 2 2 2 0 0 9 3 2 0 0 0 1 4 2 2 0 0 9 5 2 S 0 0 0 1 6 2 1 0 0 2 6 2 E 0 0 0 4 6 2 1 0 0 0 9 2 C A

K e ta GO E R U N

                                                                                                    ;f 1      _

8 _ n _ 1 J. o w/ 0 3 _ 4- 1 is S N i a, E v e _ m r R o p , F n 1 r I 2 I 0 1 G E R U N

( s icp o T A

                  /                                                                                              :

K la t o T t i n o P p u o r G l G 4 e2 A n lp iu tuo 3 Or A nG / o2 itr ae ni 2 A 5 4 iT m- N f

 -       as x       1 o

Em e A 7 Ot Ry s 1 SS 6 t K Rn Wla BP 5 K 4 K 3 K 2 K 1 K e d o e d s M o ie g M r i n y ia l lo a r i x r o p u te e C S t n A n a e d o m lo lo m lo n it u W a o o ip t r b g C g N P P u n a. r e i n A

                 /                      g       /

s

s q o n t r i r V u ls s n u u C e t a is o ;u a h o r r E e G w D t n H e t o n i m lc i o o g g ) r C o e u n T T n a e d e C l a o io m i A T t s a o C I:

i k in e D c M t c o o

t n t y r it n C I C ld a br F / i r n e t n e n S dy la w P P n e u C t c o R e n io H u U o a L r o A s tor l o m o P S S c L L H T ( le it P u p - D C C M r C d t S M

R

R V n t c S E a ia t r r y ic u I M le I a fg d e n t m r o R M S W e W h P B R H H u S ia e P C f a i r o s n o g C R R R R R S R R I R R F M M R U D O R F C I C e Oa 1 0 4-S 1 0 0 1 0 2 0 0 1 0 4 0 0 1 0 6 0 0 1 0 1 0 0 2 0 0 0 0 4 0 0 0 0 5 0 0 0 0 4 1 2 0 0 5 1 3 0 0 5 1 0 0 0 9 1 0 0 0 0 3 0 0 0 4 3 3 0 0 9 3 0 0 0 5 4 1 0 0 9 5 2 0 0 2 6 0 0 0 3 6 0 0 0 1 7 0 0 0 2 7 0 0 0 6 8 3 0 0 0 9 0 0 0 0 0 0 0 0 0 0 t C A

a E 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 4 K l(

h 1 1 t s t s 0 n 4 i n o 9 4- io S P P E m r o p F i m n

                  )                                                                 o s

( s a c e ip R o T : A/ la t K o T t n io . P . p u or G G . r. 4 f o A t n e m e3 3 e n c lip tu A a u o s lp Or e e it R nG 2 i o/2 A io d itr r e 5 ae P d 9_ ni iT n 4 m- fc i e _ f as x 1 A i c e m _ o m - Em 0t e

S-p o c

e 8 1 s t t ry 6 n R SS K l a Rn t P Wla _ BP 5

                    'k                                                          ,

4 K 8 3 K - 2 K l 1 K 4 !li ic p o m T

s in m m - ai n i e ) de a ts y o t e !e S ls it 8 e M b C m m e o d a a im n a v r n te e m l ( N ir D P a g S t a t e : ls l a ic

n t a s n n in a t o is

                     #     d-i    o     in                        t T          v c     io   e  e      o    le     o                                         e m     a            o  h  d     it      s    T                                 y n

i r a ta s it R te d t C R o e t n r s o g C e i t n V io y R sr lo dr t e io r S is o r o s a V r o P P 1 lo r e P a t t y c 2 n c w t n c r 6 0 t w l e ia a d e o ic o u e a la e g 1 e 0 r - C T F M R R P R e p 1 3 1 0 1 0 0 0 2 t a S G 0 0 0 0 0 0 0 0 0 C t E 4- 0 0 0 0 0 0 0 0 A n S 1 0 S 3 3 5 3 6 5 8 G 8 8 0 9 la R E 2 2 i 2 2 2 2 2 2 K P U l _ ; N lllf f! lll >t

[' ES-401 _ BWR RO Examination Outline Form ES-401-2 Y Facility: Date of Exam: Exam Level: K/A Category Points Tier Group Paint I K K K K K K A A A A G Total l 1 2 3 4 5 6 1 2 3 4

1. 1 Y -

o M 13 Emergency & ,~ M4 . ! Abnormal 2 e n: 1,_ 19 ' l Plant Evolutions 3 N NY . 4

                                                        ,E gg        a        y   ,

Tier 36 Totals T T. 1 28 2. Plant 2 19 Systems 3 4 Tier 51 Totals I f%

3. Generic Knowledge and Abilities Cat 1 Cat 2 Cat 3 Cat 4 I

Note: 1. Ensure that at least two topics from every K/A category are sampled within each tier (i.e., the " Tier Totais" in each K/A category shall not be less than two).

2. Actual point totals must match those specified in the table.

3 Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities.

4. Systems / evolutions within each group are identified on the associated outline.

5. The shaded areas are not applicable to the category / tier.

6.* The generic K/As in Tiers 1 and 2 shall be selected from Section 2 of the K/A Catalog, but the topics must be relevant to the applicable evolution or system.

7. On the following pages, enter the K/A numbers, a brief description of each topic, the topics' importance ratings for the RO license level, and the point totals for each system and category. K/As below 2.5 should be justified on the basis of plant-specific priorities. Enter the tier totals for each category in i the table above.

i

Q 19 of 45 NUREG-1021, Revision 8

2-1 t s 0 n 3 4 io 1 P O S E mr . o p F m I

( s icp o T A

K 1 p u or G

1 r i e : T la e-n t o is T n t ltuo n Oiu t io P el taE o v p u 5 o 4 O in nt r ma G f aP l o x 0 Ela G 2 Omr Ro Rn b 2 WA Bd n a A 1 y c A n e g r 3 e K m E 2 K 1 K r e w 5

/.

n o c o it 1 P1 ec 3 ! c n dI O n u

p o nn n F ir 2

it i 2

                                                         /  t awo   e d                  n     g ekn T     3         5           5   3 ty r    /  le         d           /  le        o          8

_ e o e v / A / e v sn r m fa t r e e y 1 e r e e rU dy n S s u L r u

                                              /     r u  u L s  r it iv  M        s  r Pr   H              o
                  /    m r

s e e s s t A s s s e e no t n iS m e r ta e c a R e r t a oe e i r G f Wr P r e C P r P W il t ia m v N a e r o l R S l r o dc ns n : e n 1 t c o le t n te l e t c w o on ia ls a R

ibr / a tc a v, y e e w y a L Cwo t n , u M e d e r o to 1 E T R e D r e lp D r R o MD C T 2 P A R v m t c A y A n R h w h d o h h a RM h r 0

                  /   ia  C ig  o     ig    a   c ig  ig  e CR    g       o   1 E   M   S H  L      H     n   n H   H  R  SP   iM g

e - A i i

   ._ 0  1            5   6 7  9      0   4   5   4   5  1  7 3e 0       t a   G 0   0 0  0      1   1   1   2   2  3       0       C    E O_

4- 0 0 0 0 0 0 0 0 0 0 0v 0 S 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5o 9b 0 0 A

R E 2 2 2 2 2 2 2 2 2 2 2A 5 _ K U [ .F. N

2-I 8 _ 1 s n j., 0 t n 9 o 4 S io P 1 is E iv e R m r o p , F m 1 I 2 0 1 G _ E R U N _

( s c ip o T . A

K 2 p u - o G 1 r ie : la - T t _ s-n o - is T ltn t uo in Oiu t o _ nlo P _ o iv p tE u 5 x , a or 4 mtn G f ma l o aP x 1 Ela 2 _ G Om r _ Ro _ Rn b 2 _ WA A Bd n a 1 y A c A. 3K E

w. _

1 8 2 K . 1

            -               K l

e _ r 5 o 6 6

                                                                                                    /.p           5   5 C     3  /

r

r 8

                                                                                                                  /   /  9 4

n d / w w / 8 m 5 le v le / io e c m P P 5 W / 7 e / e v s ls / u 7 & T e L e d

                                                                   /.p t

r r v c n o u C C / 9 C r r u r t e a 9 F c A D 2 5 C iA 5 e 5 e r R u a 5 t n / / t t a / e L.

e F f o f o f o / / m e e f o t a ta t h y V le

                                                          /    e    e     t         s n

n r e e a d ig s r s s v e r u T m a s io W p r u W W a ta te s e s s e r u n R s i 1 R H R o o o t lo t f o s o t a lo o f o ta / m a lo n a L n l L L t a r o d e L lo s o e r o o a o e 8 S e r r e P n s s s p P T e P e s e d e e e e p a a e s p P r it a / :

                            /     t e     n t

e t t p n le t e o I m n t n n n a e ls e le a m b m o o t u o A. le ite a e i lp o lp p W m e A lp L. n e e o lt n S m 4 C e T is R p o P is m T iss is t n e R t o a m& n m m r T s m m C s s o e n T N o o o o l ia e o a o m 3 e in l le e e V O C1 ia C C t c T r p o it C o t n R r p a r p r p C. . it s t n

                            /
                             #     r/   M    r   r    a   M    w y    p  R s-  r  C        e   C      p t

n w y p p c t n - e io ono f o o e T r u f f o r tr f u o r u u e o f f ir F P E o R C D S lo O la o e o S C D S S S C O lait l l P s a ia t r v s h h h . h t n y it h h h h n h it d h h c r A ita t t o r s r r r r a s ig w ig ig a

alu o a a ig ig ig ig o ig a a o ig ig ig o e g E Pc L P P H H H H C H P P In L H H H H L H S H T'l e

     @.se         1               1ir   2  3   4     8    1  2     3   6  7   8    9      0    2    6   7   8      9  0  3  4    8   0     ta 0

4-0C 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 1 0 1 0 1 0 1 0 2 0 2 0 2 0 2 0 2 0 2 0 3 0 3 0 3 0 3 0 0 0 C S 5w 9l o 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 5 9 0 0 A

2F 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 6 K _E 2

b I nl: 2-1 t s 0 n 4 O 4- io S P E m r o p F m I 1

( s c ip o T A I K 3 p u o r G/ 1 r ie l: a T t w- o i rs T

      !tn                                      t n

uo io Oiu t P nl i t oov p u 5 aE o 4 O nt r in ma G f o l aP x 2 Ela G 2 Om r Ro Rn b 2 WA A Bd n a 1 y A c n e g r 3 e K m E 2 K K 1 l a a e r it n e A

r p n a e m o e r f f i u it c A D S n 4 t n h h u / e g ig F i ty g n m H H e i n t n t n 8 fa lo 8 ia e e n o / t n S C t s o m n m n : o

e n n e C ia ia s iS w y t n t n la i v m o id r t o a d c c d a C o C o T e N t u A n y y t n R

                /   h S    g  o c     r    r              ,

in e5 a a io 1 E f o S/ i a d5 P 2 P le u n/ o y A s u he c5 cte r 0 s gr o fe iu ev 1 O-

o e/ g E L R Hta Ser Se e - 1 1 3 2e r 5u 6L t a G 0 4-2 0 2 0 3p 0 3s 0s 3r 0e C E S 5 9 5 9 5m 9e 5e 9r 5a 9 t A

R E 2 2 2T 2P 2W U _ K_ N

2- 8 1 t s n Q) 0 4-n io 8 o 2 is m . S P i v E . m e _ r o p

                  .                                                                           R F

I m 1 2 0 1 G E R U _ N . ) _ ( s c ip o _ T A

K la t _ o _ T _ t _ n io P p u o r G G 4 e1 A n) - hu t uo 3 OrG A

n2 o cr 2 5 aie A 4 nT m- f as r t o xm A 3 Ee t 2 _ Os Ry S 6 Rtn K Wa BlP 5 K 4 _ K 3 K 2 K _ 1 K f r f o t o t u la h g u r n s S e e o ie y lo s it a r lp R r lo e n t ia p e t n e r d e n l u r t n o d e ix S u o m o M n m u s C a C n o u A m s e le _ N o C r d a r r v

               /               w it    )                  t s       n   e   P    e  e s

n

                #             lo   c e

g r a t S e ta L la lc F j

                                        <5              M i

r T n w r e t o m i u n n e M r a i br d T e o I m R l e e t a t s a : E P io T u e t n y dy r it I C ( L B C lc T F Wr S la P n y u / r r io H u o / r a r N o o o P S c L S S M e a / s t t t S s y D C S r ic

R I C it a C C S M R lc C S S V c a c c a T G R M I C e H P lo P C P M R P u I C D m I C R e a e e G D r o P L R ir g C R R R R H Is L H S R t S A N R A P P S R R R S E t e 1 1 2 5 2 0 0 0 1 0 2 0 0 0 3 0 4 0 5 0 0 0 0 1 0 2 0 2 0 0 0 1 0 2 0 0 0 0 0 a 0 4-0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C S 1 1 1 2 3 6 7 9 9 1 2 5 5 5 6 7 8 3 3 9 1 9 5 9 5 1 6 4 6 A 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 2 3 4 / _ E 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 K

l 3 - 1 ts 0 n 9 4- io 1 S E F m r o P p O im

( s c ip o T A

          /                                                                                        :

K la t o T t n io P p u o r G G 4 e2 A in) u tt uo 3 OrG A n/2 o ir 5 t aie 2 nT A 4O f im- o as xm 1 A 4 Eet 2 Os Ry S 6 Rtn K Wa BlP 5 K 4 K 3 K 2 K 1 K e d e m o d s is M o ie n g e M r a n d y ia h c i lo o a r l ix r e _ o M p u e e M _ C y S A n n t a 8 a m d o o m e lo p r lo a n it u it u W n a v o o e e g C N i r P S P t S a. t b b n A g io D g / T / n a i r i r i r r n : s

in s s t e s t t s o V i iS

           #   d                         u M    u  a       n is    i M  H       lo la   v n              to       r o T r

o e G e D D it n n T o t o m e a n o T C T h e e d n l a

C la o o C m ir A C T e ts d o o C : : : R i n o ic D ic M it c y o o t t n t n R y R it n I C I C I C br C r / r n e r R n e , S la w P P P d n u r t c C t c o t o a e n io lo u U o a T o A le s r d l o m o P 1 r S c d L L L t le ( a it a P n r u p 2 t n C M r ic C t u S M

A

R

R n n c a E S E i o t n r m y r 0 t i a fg d e c t o o S W e W h P B b H H ia e C P C f a i r e o s n o g 1 C R R R R S R R R R R M M R A U D O R F S C C - e GO I t 1 3 4 6 1 0 0 0 2 0 1 0 1 0 0 1 2 0 0 0 0 1 3 0 0 a 0 4-0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C E S 1 0 1 0 1 0 2 0 4 9 5 0 4 1 5 1 9 1 6 2 0 3 9 3 5 4 6 5 2 6 2 6 3 6 1 7 2 7 6 8 0 9 0 9 0 0 0 0 A

R

 -   E         2   2 2 2  2     2  2 2 2   2   2  2  2    2  2   2 2   2   2   2   2  2   3   4  K   U N

l i 2-8 1 t s t s n 0 n n io O 4- io 4 io S P P is E v e m r o R p F n 1 i 2 0 1 G E R U N n

                )                                            o s                                          s

( a c e ip R o T A

l: K t a o T t n io P p u o r G G r f o t 4 n A e e3 m ln>u i s e c tuo 3 ie la OrG A it p i r e n2 o ir io r R taie 2 P d 5 A e 4 m' nT im-as xm Eet A 1 ic f ic e p d n e n. f 5 o S- n 2 Os RyS t n v 6 la & - Rtn K P R Wa BlP _ 5 _ K _ 4 _ K 3 K 2 K 1 K c ip o T

p m u e ) n t s 0 a y 1 e le t n s S ti e b C e m o d lo la im r m O a P n t r m N a ip u n te : la

                  /     e   g     q    o         n  ls              t r

o n E C n I a o c- li e o le t o T m o g g it s T y e n o in a la s it t s i g C l k it e t n r y n d n a e n V io io S lo e t e P r is o a L s a V r o P r e P H V w t t c y r ic v le l e I d n a o f a u u S a a e g ic T r F F M R l P R t e e p

 -       1            1    0    0     3  0  0  2      a             S-0            0    0    0     0  0  0  0    C                t 4            0    0    0     0  0  0  0    A                 n 5    3    4     9  8  8  0                     la S             1    3    3     3  6  8  9    /

K E 2 2 2 2 2 2 2 P l I W

ES-401 PWR SRO Examination Outline Form ES-401-3 l Facility: , Date of Exam: Exam Level: K/A Category Points Tier Group Point K K K K K K A A A A G Total 1 2 3 4 5 6 1 2 3 4

1. 1 ? # 9 24 Emergency & ; w y W .

2 - e a 16 Abnormal . - Plant 3 D O' b D ' 3 Evolutions g, "m T lE' Tier l? E, 43 i A Totals M 1 19 2. 2 17 Plant Systems 3 4 Tier 40 Totals

3. Generic Knowledge and Abilities Cat 1 Cat 2 Cat 3 Cat 4 Note: 1. Ensure that at least two topics from every K/A category are sampled within each tier (i.e., the " Tier Totals" in each K/A category shall not be less than two).

2. Actual point totals must match those specified in the table.

3. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities.

4. Systems / evolutions within each group are identified on the associated outline.

5. The shaded areas are not applicable to the category / tier.

6.* The generic K/As in Tiers 1 and 2 shall be selected from Section 2 of the K/A Catalog, but the topics must be relevant to the applicable evolution or system.

7. On the following pages, enter the K/A numbers, a brief description of each topic, the topics' importance ratings for the RO license level, and the point totals for each system and category. K/As below 2.5 should be justified on the basis of plant-specific priorities. Enter the tier tctals for each category in the table above.

NUREG-1021, Revision 8 26 of 45 l

3- 8 1 t s n 0 4- i n o 4 i o S P 2 i s E v m e r o p

                 .                                                                                                                 R F                                                                                                                                ,

I m 1 2 0 1 G E R . U N

( s c ip o T A

K 1

        - p u

o r G

1 r ie : eT n-la t hs o T tun Oho t n i nlu o oo P tiv p aE u 5 mt o r 4 mn ala G f

 's   xP El o

a G 7 Om Rr 2 Son Rb 2 WA A Pdn a 1 y A c n e g r 3 e K m E 2 K 1 K 3 4 e

n /.e 8 i n o / 1 L 9 4 n it r r e / 4 6 / 5 / 4 io a i C m / 4 8 / g t c 1

                          /            n                  ta  m  a     S           /

s t e / / y n / n 9 n 1 3 i la W a te T 4 u R r e c

                                                                                                       . t i   il o   ig
                                                                                                                       /

u

                   /       d o    /     m r

r u g c S4 P / B e t a g r o r y F la R t n e t n S )/ - m a v e C a t i v w e T a 2r e g u t s

                                                                                      . t s W       E     t n       M   i ty       a    l o            4       N        h o  o 1   f n    u      n    a                    e       t c  7 e       r 1 m

t s c e m I r g A / f a d h

                       /   t r

n 3 n S / s 0 1 1

C o /w En i lo a I W d ic o T C o n Y S i t d o o / ia & n E n o t t n e Wa r o c V le c a v r 9 R o M T i lo t n a

X-C t s n 6 e o

W R A n o & k r r e E R / C d e k C o is i t 9 it a e is a e s

S e l o a c lo I N d lo c O C o c 0 r n n v n t u C id r t r f o In b u o N m o r u n nu o o a Ee O e u a is- t n C a R t n t L e g E B p r EH S o A iq e n o s ) S f N o S k id ia f la W m T Cev d n k c l L lc s 7 e r o o s: s u C

a s d y o d C o a u O C o 0 t s la

le e t e M ; c e ;i R la it ! a e ) L E t a c s

            #        o d  b    r   u R

3 n C 5s C B V t N r 6 & u a e o t E u e a B O P 1 e f ta 0s 8 f n f n f i 0

4 6 q e P n p r e e 2 C A g o ip Ee 0 o o o e o F A 1 0 e R L T i t p p g A 0 R

E e r s c /c E s s id s t n

E E d 3 y A n o o o r C E C s o it n Nx

                                                                'E
                                                                       /

s it a s c s a W / / n a h 0 A r o

E C D r n a O 7 m A B- W o t o c o l B W( W ig g I L L & 1

                                         /       9;    E  L     (       :   L   S  L    A   L   P   (         (

f 3 H & e 0er 2

 &  1 0

4-1 0 0 3 0 0 5 0 0 1 1 0 4 0 1 O 5 1 0 0 E

4 2 0 6 2 0 9 2 0 4u 0t 1 1 A 1 5 0 5 5 0 7 5 0 9 5 0 2 6 0 7 6 0 8 6 0 9 6 0 4 7 0 0 E 6 7 0 0 A t C a _ S 0 0 0 0 0 0 0 0 E E 0 0 W 0 0 0 0 0 0 0p 0u

E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

W 0 0

W /A E 0 0 0 0 W W 0 B 0 0 0 0R C 0 0 0 0 0 0 0 0 0 B 0 B K

3-1 t s 0 n 6 O 4- io 1 S P E m r o p F m f k

( s ic p o T A

K 2 p u o r G/ 1 r ie : la eT to n- T is lt un t n Oio t io nlu P oo itv p aE u 5 n o 4 O it r mn G f al a o xP El 8 a G 2 Om Rr Son Rb 2 A WA Pd n a 1 y A c n e g r 3 e K m E 2 K 1 K s 4 s

                    -                                                                o        9 ip r

3 /.s 4

                                              /             m                   4 L
                                                                                      -       /
                                 /

s 2 e / n T t n e c

ts 7 3 r r e l. e o r e r r p y

                                                                            /    e  f R

it o p ic u / 3 e t a s c t c id e e e S I r n e 9 n a c D R k N / u w a ts / F u e c a lo 7 e k a t p d T r a n R A - t n M t r

                                                                /     g  e   u   e                   o ty                e       n     a           n   I n L   R   F e  t a        d w    it 8 e
                   )

2 c w lo t n 4 o N a e a 7 ia 8 0 a o o a / C e R e e n H R

r fa S E p 3 d C lo e g e b u b u ia e 6

s d a i A y n

E1 S / A lo o y o m e r u n a t T T M ta4

                                                                                           /   s u  m R  t r

e : o

               /        1         r o  C          c   C   t s       ia   r r

e c r n v s iS e C/ / 4 p C n r s y s R d o r o f o u/ e la t n e o la i m ;y 0r k c / a O A e o S e r e e t a ta s qk en w o s A e m c e to v a 1 e a p V L C g r t c P cr m r e s di P a m u R T e N Evo b ir r k a O e a R H r u r te pt n o aS C G m e in t r R

                   &c    n u   T    e   e  L      m     e     e     o   n  9   e  L   Int  a    la t ta s

n l a t n

                #  2e         e  iz    r      E     R   R  iz   S   I C   G
                                                                                )

6 5e D t s y n I n io E 0R R n r u B 3 0 f r f f 0 0H f n o f o P 1 E-f o o fo u3 o o o e S o 2 P / t n ibr s l E s s m m E Ey d C itc y A s ia / s s s s/ s a a / /r s ic M s n r 0

               /   Wn Bio la P

u e r m W s s o s o en ro s o s o e e E C Wa s o c R h ig s o u o g 1 T o O E P E L L Pi L L t S t S ;nd L A A F e - (t a L ( H L G t 1 4 8 4 1 7z 0 0 8 9 0 2 5 7c 2 3 7 8 4 0oc 8 0 1 6 5 9 ta 0 4 0h A A 0 0 0 0 E 1 1 2 0 2 0 2n 0f u 3 0 3 0 3 0 3 0 5 0 Ee 5 0 6 0 6 0 1 6 0 0 E C E 0ib / / / E / E S 0a 0t W W 0 0 0 0 W / 0 0 0 0 0l a 0 0 0 0 0 0 0 0 0 0 0 WS 0 0 0 0 0 0

                                                                                                    /  0 0
                                                                                                           /

E A

R E 0S B B 0 0 B W 0 0 0M 0 0 0 0 0 Bo f 0 0 0 W 0 C K U N

l 1 14 8 3-1 s n 0 t n 3 o . io i 4 S P is _

 .C  E m

r o p

                 .                                            R v

e F m 1 _ I 2 0 1 G _ E R U N _

             )                                                        _

( s _ c . ip o T A

K 3 p u or G _

1 - r e : i la eT t in- o s T ltun t n Oio t o i nlu P _

-)

oo itv p aE u 5 -

 . nt                                                  o r    4     _

imn a G f al xP o . El 9 a G 2 Om Rr Son Rb 2 WA A Pd n , a 1 y c A n e g r 3 e K m E 2 K 1 K 8 n / t s 4 o 2 n e 6

it c / e r e n d u / r u u n ic s n s F io c o o s t c A lc it a e t y n g 6 n u 2 r p 5 e u n / E t / - / f r c e r fa la i l d e d n A g e g S M n w a a v in

               /         a  o   s le        k  O    d s

e le P s a o H e e e r o la m v lu i L lo t o a e le te D ta F N L u is R y 8 S e r t T e r F f P c / C n n t n

               /       )   f n   g   R    e   e
                #  iz  8   O   O    e   n            m     io E

r u 0 f E g s G n P o i A r d s F s s

s 4 e o e m ia y r A e W s 1 m o c a t n o b

               /     r      o  E  E l

F x e o g E P B & E t e ( L 3 5 7 6 S C t 1 8 6 6 1 0 0 3 5 a 0 2 0 3 0 5 0 E A A 1 A 1 1 C 4 0 0 0

                               /  /   /
                                            /  E    E      A S           0   0   )   W  W   W     E  /    /
                                                           /

K E 0 0 N B B B C W W

3-1 t s 0 n 9 4- io O 1 S P E m r o p F I m

( s ic p o T A/ K la t T t o n io P p u o r G G 4 e1 n A i) ltu uo Or 3 A nG / o2 itr 5 ae ni A 2 4 O iT m- f o as x 1 Em e A 0 Ot 3 Ry s SS 6 Rtn K Wla PP 5 K 4 K 3 K 2 K 1 K n io ta t u r c e lo A r t a r e t s o w 8 m n e it d g a C o r u n n o e n in n N p t a n o e o r io e e o M F it o : s

m F it it a e g y u l it iS

           #       u  m      a        r    n     y        c  b        a  n  la  v P   u  y    c t

n u a n i r s o t o m e v lo t e i e t a lio o r p e ts e o p M T e e t n V f w m r r e g i R D i t s D r la a I r u e C r e S t e r ts is n t n y o d n S n r p t s t a la a D io i o S d o a d o t s m n n n e w m w s t 1 o e n e e e e ta d E ic d a ia P 2 R C la r it is I T m d m s e / t r a d y r ic e o r n n n n e y c R G a r o 0 lo o e a e i o e F ia r le e R 1 n P a ia g O r t c m e r d - t n a lc o t n C t n d n n l E iu ts a e 1 o e e h ig n d o u c n

                                       -   o  e  o  o ia ix u C   iq   a  e r

t a G 0 4- C R C E R N I C Ic C C M A D L W A C E S 1 0 3 0 4 0 3 1 4 1 5 1 7 1 2 2 5 2 6 2 6 5 9 5 1 6 3 6 8 6 1 7 2 7 A

R E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 K U N

  ]                                                                           '

3-8 1 ts n 0 4-n io 7 1 ios S P i E v m e r o p R F 1 Im 2 0 1 G - E R U N

( s c ip o T A

               /                                                                                        :

K la t o T t n io P p u o r G G 4 e2 A n N> t u Oruo 3 G A n/ o2 itr ae ni 2 A 5 iT 4 m- f as x o Em e 1 A 1 Ot s 3 Ry SS 6 Rtn K Wla PP 5 K 4 K 3 K 2 K 1 K lo r t n o C e g r u r g e lo n la P n r v d to a i m g t n to i o n t r a n o lo a m a 4, n n r e to N E C t r t n e r e m la o n in o n e R e m a v it e o s

o e r o in e o u G M la 4 C u C n m u e b e M ip u S t m b t m s o r in g C q r e ir le n r o e t e s le t s d m r u E o t a R t is s o e T t n o r e v e it c n to o P lo t a e D ie it t a n s y a C P r L e I c e o g r h i r D a o t n S lo t or r t n t n P in e e A la id W t n io o y r e r e a e R e n R r ic y a it c e P le g l C c P e n d e e r c R r e n iz iz lc m e m u n G d s t n n i A t m y r o e r u r u o r u n g n F a n n c e s s it a n o n r o t c g r s s t n- ia o ia t H m a e le g e o P r ia g

  '                    e   s   s   c       t   r  t   n       a  n  d   E   r e    c      lu             e a     e   e   a    n   n dy   n  e  le   e ia    n          o       c  it a  e t

n 1 0 e m r r e o o o p u t o C m r ir t ir o ta 4- R E P P R N C H C S F S M C A E P C S F C C 2 6 0 1 2 6 7 8 9 3 4 5 9 5 2 4 3 5 9 6 3 A S 0 0 1 1 1 1 2 2 2 3 3 3 3 5 6 6 7 7 7 8 0 / - E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 K

,ll

3-1 t s s n 4 t n O io O o i S P P E m r o p F I m n

          )                                          o s

( s a ic e p R o T A

          /                                 :

K la t o T t in o P p u o r G G r f o t 4 n A e e3 m in) l e tu c Or uo 3 s ie a lp A e nG / it r io2 io r R tr d ae ni 2 A P e O iT ic d 5 m- f n 4 as ic e x e f Em e 1 A p S-m m o Ot s o 2 Ry t n c 3 SS e 6 la R Rtn K P Wla PP 5 K 4 K 3 K 2 K 1 K icp o T lo r

t n m o e ) k C ts 0 n y 1 a s S it e T s 8 h r a p m m a c te y i l( n N ta n a B r o io v e W : s la

o u e t a t o iS

            #      O    g  n    r e           la           T    v m

e

f n i br n t o e m e li o u e T y e R i o G it R t s le T r r t n r y t a R C / p e e i io , S a n ta A io r H r e t n m t P P 1 u n 2 i iz e br W e y ic la r n D u e r f 0 u u o p T m o g ic 1 O d s m ic u e - s a n v r e p 1 is e e r m o te ia r e t s n ta S- G 0 4- R P C S M S I C t n E S 5 0 7 0 0 0 1 4 5 4 6 7 8 7 A

a R U l E 0 0 0 0 0 0 0 K P L N

l IO ES-401 PWR RO Examination Outline Form ES-401-4 h i Facility: Date of Exam: Exam Level: K/A Category Points Tier Group Point K K K K K K A A A A G Total 1 2 3 4 5 6 1 2 3 4 l

1. 1 ei $ ~

16 Emergency & , e - - Abnormal 2 m - - 17 Plant 3 0- d

3 Evolutions

                                                         *         ~

Tier 36 Totals '

4 1 23 2.

Plant 2 20 Systems 3 8 l Tier 51 Totals

3. Generic Knowledge and Abilities Cat 1 Cat 2 Cat 3 Cat 4 13 Note: 1. Ensure that at least two topics from every K/A category are sampled within each tier (i.e., the " Tier Totals" in each K/A category shall not be less than two).

2. Actual point totals must match those specified in the table.

3. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities.

4. Systems / evolutions within each group are identified on the associated outline.

5. The shaded areas are not applicable to the category / tier.

6.* The generic K/As in Tiers 1 and 2 shall be selected from Section 2 of the K/A Catalog, but the topics must be relevant to the applicable evolution or system.

7. On the following pages, enter the K/A numben, a brief description of each topic, the topics' importance ratings for the RO license level, and the point totals for each system and category. K/As below 2.5 should be justified on the basis of plant-specific priorities. Enter the tier totals for each category in the table above.

33 of 45 NUREG-1021, Revision 8

4-1 t s 0 n 6 4- io 1 S P e E m r o p F m I

( s c ip o T A

K 1 p u o r G

1 r ie la T t e- o in s T ln t t n uo O6u t io P nlo o iv p tE a u nt o r 5 in G 4 e ma l aP x f o Ela G 4 Omr 3 Ro Rn b 2 WA A Pd n a 1 y A c n e g r 3 e K m E 2 K

1 K 4 e

                          /.e     8           n
                                  /    m    i L                                        4 n             r       r  e            4            6               5   /     4 io i

e t s m / 4 8 / g C t y a S

n

9 t c 1

la a S e T 4 s u

r /.c it y i n / n d r W t4 S P / B e a r g lo ig ty u o u g lo / t a v e o r a F R ta n t r ) 2r

                                                      -  m          .

E t C iv y N i n o fe g u u t s W n e M it c 7 t e lo 4 lo 1 Es in c In e m I r g A / 8 r / 0 1 o C /n lo a ic o T o n f a t n o s 1 / n C e Wra o V c v 9 o M C i lo t n N n S C n o E

                          &    o  t n

r u ii c r r 6 le r e / R T d o a X- o

it 9 e s 5 t e e / E S e lo C a c lo I N i e k c ita n s 0a v s t f n b o is m c n 0 r o o e Ee O n u C r a it s- t r o I u C N v a u E p r

                                            /H           e     o A     e      n    s )     S                        e N

t S fu B P E S d k n o s 7 r o f o :

             /   /e la W    y c

m o r e Cev C n o c la la t lc u O C o 0 E te a t c s s la R M ;i R i e ) L s

              #  lb       3;   n  C   iz    5s          C     B  V    N   r  6   )   &       u q

a e o t o E a P 1 e r 0s 8 f n f i F 0 4 6 L T 1 r e C A g fo u3 s Ee 0 o fo o A 1 0 e R 3 2 P p R

E r e s s/ /c x E s io s s t n

E E d h 0 y r 0 A o s en WE / s ta s s W / / a ig A o

E n 7 C m o ro W o t o o la B W W n g 1 1 E L Pi B-( L S L L P ( ( ( I H & e - G I 9; t ; 3 2 e

1 5 5 0 4 6 7c 0er 1 1 5 7 2 7 8 9 4 0 6 0 t a 0 0 0 1 0 E 2 0 2 2nu 4u 0t 1 A 5 0 5 0 5 0 6 0 6 0 6 0 6 0 7 0 E 7 0 A C E 4- / 0 0f / / S 0 0 0 0 W 0 0 0 0 0l 0a 0p 0u

E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 W 0 0 W A

R E 0 0 B 0 0 0M 0R C 0 0 0 0 0 0 0 0 B 0 B K U N lll1ll<

4-8 - s n 1 0 4 t n io 7 i o S P 1 i s E v m e R

  . fL r

o p , F m 1 I 2 0 1 G E R U N

( s ic p o T A

K 2 p u or G/ 1 r i e : T la t e-n o hs T tn t uo n Otu i io P lo p v u 5 E o r 4 G f o

  .\U (    Ela Omr Ro Rn   b G

2 5 3 WA A Pd n a 1 y A c n e g r 3 e K m E 2 K 1 K . s _ s _ 4 3

                                                                         /                                  L o               9 ip            3                   4 r            /             /.s   /.

n 2 1 4

                                                                                                         /
                                                                                                              -        9     /.

T t c io / 3 r r e / le i, k- r n s r t a p

                                                                                     /        7       /   e               .

R o f e e ic n u m / 3 e s le i s t d r p e a ta n e 9 1 c ic 3 e i m e r I N k / r u w a R t w

                           /       a             c        /         R      r k a       c   7              d    r          e    s    /

la e A t e S e a tp e T a n F y w R n e @ t n a T M / g e n L u e t t s a w o t a ) e t 4 /o I N a R F a d 7 it e 1 2 c m lo S i n e W a ia R be r f d / 0 a 3 3 n o a / w e e n H R / d S

a e m h it W d o d R lo o r E E1 C/

                                   ;y 0r 1
                                         /

4 r 4

S p r o C p

A a OL

A C O ia t C n o W C A C & y s c is n o e n lo C o r o t S m e sy t n e is n a R g n a e e i d ta e T t u r o a b T t u r o a ia M f o s e ta4 u/ 6

r e w o d R id a u q a s u o e s s m r la A R t a n e y r e v o c e t ls a o a R t n 1e v V k L e C g g t c r c m r r - r P m r R T r e e i iL G n R T IWd S m ip r c N s o Evo u r a k d O e ia a u e n n e a C dr r i

u C &c T e e a e ts L m de e H d e So t e e L t la la t ia la t n

                        #   c     2e      u   e iz   r  r   u       E        R   R   t        in   G  G
                                                                                                         )

5e D t t s t n n io u d e 0P R n r u B B O 3 R a f f 6 0H f n S y o io P 6 f f E 0 o o p o o 0 o e n p E- t i s ll e f o 2 m m Ey C t y P A it n p n br s a g A E s 0 s s ic s s a a E r s k c d ic M h c n r

                        /   o   o r Wm     la P

u er m r a C W s E o s s o n s it o s o e te E C Wa s o c c R ig u o g E C D Bu (t T P S L O o & L L A L L t S S (

                                                                                                              ;d n L  A     A  A  H  F       e a 1  4             L   8:  L                                         4                         9 1             1  3   7z     0  0   8   9  1 4    0         1   2   5   9    2   3    7  8  4   0oc    8  9     0  1 6  0      ta
        ]

1 0 0 0 0 A A 0 0 1 O 2 2 2 3 3 3 3 5 5 5 6 6 C 4- 0 0 0M 0 0 0 0 E 1 0 0 0 0 0 0 0 0 Ee 0 0 0 0 1 E E E W E E/

                                         /  /

0 0 0a t 0 0 0 0 0 0 0 0 0 0 0 S 0 0 0 0 A S 0 0 0t W W 0 0 0 0 0 0 0 0 0 0 0 W f 0 0 0 0

                                                                                                                                   /  E      /

E 0 0 0S B B 0 0 0 W B W W 0 0 0 0 0 0 0 0 Bo 0 0 0 0 W C _ K

l _ 4 _ 1 t s 0 n 3 4- io S P - E F m r o I p m O

( s c ip o T A

K 3 p u o r G

1 r : ie l a T t o e- T n is t ltn uo n Oiu t io P nlo p ov u 5 itE a o 4 int n G r f o . ma l aP O x 6 Ela G 3 Omr _ Ro Rn b 2 WA A Pd n a 1 y A c n e g r 3 e K m E 2 K 1 K 8

n t s 4 o 2 n e 6

it e r e c n

d u / r u u n ic s n s F i o c o o s t c A lc n ita 2 e ty n g 6 8 u r p 5 8 e u n / / E / - f r d tc e r / n fa la i l e ir A g e g S M d n w A n a a v n o

                 /          a   o  t n    s le        k  O i

d ls i S e le p s a o H e le ie e r o a i v m v u lo t o a e le u te m u R D L t a F T e N L is r y 8 S r e t R r e F f

                                -  t s P     c  /     C   n     n    t n
                 /            f n                  e
                          )

O in O e , g R io 8 iz 8 e n G m 1 E r 0 f E g i s n P 2 u A/ fo o s r d y a i P s 4 e o e m 0 s s r A

                 /     e WB s

r s o oL s 1 E E m lo F c x a e t n o o g 1 E P L & E S t C e 1 8 ( 6 6 5 3 1 5 0 7 0 6 3 5 ta G 0 2 3 5 6 E A A 1 C E A 9 1 1 0 0 0 0 S 4-0 0 0 0

W

E E

E

                                                             /      A
                                                                    /

R E 0 0 0 0 0 0 0 0 B B B C W W K U N

4-8 1 s n io t 0 n 3 _ 4- io 2 _ S P is v

 .j  E m                                                                        R e

N r o p F 1 m I 2 0 1 G _ E _ R U _ N

( s c ip o T A

            /                                                             :

K la to T t n io P p u o r G G _ 4 e1 A n> eu uo 3 OrG A

n2 o hr aie 2 5 nT A 4 f im- o as

xm Ee t 1 A 7 3 Os RyS 6 Rtn K Wa PlP 5 K 4 K l 3 K 2 K 1 K n o it a t u r c e l o A r t a e r w s e t o w m r o r i d g a u m e n N C t n e ir

p e e e o it K F o s m m F a e 9 y la it n la

               #        u   u  y  t    r    6             c     s   o      t n

e P n u ; A o o m e iv t lo t e e t a e r e g e p M T t r n V f a n r e C r e r t s is n t s y D a d S n p e t a e e D o in S lo n i t n s- e w m w it o d o o a d e s m e e ta d E d s a ia P _ R C la r In T m A s e / a d y r c e r n n e y R G a r o lo o e a e ia e e F ia r e R g r t c im n e r o C d l d t s a e t n a e ig lc c t n n in i x iu a e t 1 o e h n u - o e o a u iq r a . 0 C R C E N i n C tc C M A L W A C _ 4-S 1 3 4 3 5 7 2 5 6 9 5 1 6 8 6 1 7 2 7 A

0 0 0 1 1 1 2 2 5 K E f 0 0 0 0 0 0 0 C 0_ 0 0 0 0 0

4- _ 1 t s _ 0 n 0 2 4- io S P E O_ m r o p F I m

( s ic p o T A

          /                                                                                  l K                                                                                  t a

o T t _ _ n io P p u o r G G 4 e2 A in2 lt u uo 3 OrG A

      /                                                                                                       _

n2 o _ ir t 2 5 aie A 4 _ nT f _ im-as o - xm 1 8 O_ Ee A 3 t Os RyS 6 Rtn K Wa PP 5 K 4 K 3 K 2 K 1 K - r g e l o n o n 8 r o ta i r m a g t n o lo it g n n r e o n N n C t r n ta n n m la a o n it n o i lo n o e i a v it i t e o : s i

o e o it lo e o b u b u G M la iS C r u C n a m u y e o t S m i r i r n t o v m e s te o ic r a g C r t e t t le o r T e e t n r s v d t s r p r u lo o a R is is s it e R t o e itc In n S P ta e D D ie a ta n t n s a C P r e te o r h i r D o id I , y lo L n r t n t n P a e A la l a W i o - S o y r r o r o a e e n R r ic c y a it c F 1 C c e e P i e le e e r i r c R g i n r e 2 n iz iz it s lc m m u G d s t c t c n s A y 0 r o e r u r u r o o u n n F n n e e g s it a n tor r o t g s s t P n i ia t m a e le t e o P g 1 c r e s s c - ta t n a in d E E r e c lu e a e e a d n n n e e a n o c it a e t G 1 e m r r e o o o o p o C C m r r t r a E i t i 0 R E P P R R N C C S S M C A D E P" C S F C O 4-S 2 0 6 0 0 1 2 4 6 6 2 9 2 3 3 5 3 9 3 5 5 2 6 3 6 4 6 3 7 5 7 9 7 6 8 A

R E 0 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 K U N

4 8 s s n 1 0 t n 8 t n o 4 - io P io P iS S iv E e n m r .R o p F- m 1 I 2 0 1 G E R U N n

                )                                                                o s

( s a c e ip R o T A

                 /                                                     :

K la t o T t n io _ P _ p u o r G I G r f o _ t n - 4 e _ e3 A n) m hu l e s c _ tuo 3 e a _ OrG A it lp

i r e n2 o ior R _ ir aie t 2 P i 5 A m 4 nT ic f 4 f im- ic . o as 1 e -. xm A p -. 9 ' Ee S- -. 3 Os t t sc Ry n e S 6 la Rtn K P R Wa PP l 5 K 4 K 3 K 2 K 1 K ic p lor o t n T o lo / C t r n m eg o e ) k r C t s 0 n u y 1 a la P s s S it e T r v i a m m h e o w p i t a c m f a t n y ( N la n ta e e B o r : v e W r n : s la

o u R e. : e t a t o

                   #      O         e      . ip   n    r               la m

e

n U, u ib e n t o T y m le S e ts y R t a e li R e r W C t i d l t o n h i e E q g n T

ew e p e G n t r e a i A r t n T t n io it i P r o r H e n e i l m t e P u b i n d n l a iz e n m e n a D ru W e m y r ic u r u o n g T e m n o f ic d s p i a o H m c u ia g e is s m t n r dy le a n iv r t n s p 1 e er o o u e ia r e t s o t a S-0 R P C C H F S t M S n C C t n I A 4-e S 5 0 7 0 8 0 7 2 8 2 4 3 1 4 5 4 6 7 7 8 3 0 A l a E 0 0 0 0 0 0 0 0 0 0 1

                                                                        /-              P K_

ES-401 Generic Knowledge and Abilities Outline (Tier 3) Form ES-401-5 Facility: Date of Exam: Exam Level: 01 r Category K/A # Topic Im'p. Points 2.1. 2.1. Conduct of 2.1. Operations 2.1. 2.1. 2.1. Total 2.2. 2.2. 2.2. Equipment 2.2. I Control 2.2. 2.2. l Total 2.3. 2.3. 2.3. , Radiation 2.3. Control 2.3. 2.3. Total 2.4. 2.4. , Emer0ency 2.4. Procedures / 2.4. Plan 1 2.4. _ 2.4. l Total Tier 3 Point Total (RO/SRO) _ 13/17 _ NUREG-1021, Revision 8 40 of 45 9lI

ES-401 Sample Written Examination Form ES-401-6 . <A Question Worksheet ! U Examination Outline Cross-reference: Level RO SRO Tier # Group # K/A # Importance Rating Proposed Question: l i Proposed Answer: Explanation (Optional): 1 Technical Reference (s): (Attach if not previously provided)

! Proposed references to be provided to applicants during examination: Learning Objective: (As available) i Question Source: Bank # l Modified Bank # (Note changes or attach parent) ! New Question History: Previous NRC Exam Previous Quiz / Test ! Question Cognitive Level: Memory or Fundamental Knowledge l Comprehension or Analysis 10 CFR Part 55 Content: 55.41 55.43 Comments: r

 /                                              41 of 45                   NUREG-1021, Revision 8 t

ES-401 Written Examination Form ES-401-7 Oaty Checklist O Facility- Date of Exam: Exam Level: RO/SRO Initial item Description a b* c'

1. Questions and answers technically accurate and applicable to facility

2. a. NRC K/As referenced for all questions

b. Facility leaming obiectives referenced as available

3. RO/SRO overlap is no more than 75 percent, and SRO questions are appropriate per Section D.2.d of ES-401

4. No more than 25 questions are duplicated from (practice NRC Other exams, quizzes, and] the last two NRC licensing exams; enter the actual number of duplicated questions at right

5. [No (Less than 5 percent) question duplication from the license screening / audit exam (if independently written)]

6. Bank use meets limits (no more than 50 Bank Modified New percent from the bank, at least 10 percent new, and the rest modified); enter the actual question distribution at right

7. Between 50 and 60 percent of the questions on Memory C/A the exam (including 10 new questions) are wntten at the comprehension / analysis level; enter the actual question distribution at right

8. References / handouts provided do not give away answers

9. Question distribution meets previously approved examination outline; deviations are justified

10. Question psychometric quality and format meet ES, Appendix B. guidelines

11. The exam contains 100, one-point, multiple choice items; the total is correct and agrees with value on cover sheet Printed Name / Signature Date

a. Author

b. Facility Reviewer (*)

c. NRC Chief Examiner (*)

d. NRC Regional Supervisor (*)

Note:

The facility reviewer's signature is not applicable for NRC-developed examinations; two independent NRC reviews are required.

            # See special instructions (Section E.2.c) for items 1,4,5, and 6.

[] The items in brackets do not apply to NRC-prepared examinations. NUREG-1021, Revision 8 42 of 45 1

I 1 ES-401 Site-Specific Written Examination Form ES-401-8 h Cover Sheet 1 U.S. Nuclear Regulatory Commission Site-Specific Written Examination 1 Applicant Information ] Name: Region: I / ll / Ill / IV Date: Facility / Unit: 1 License Level:RO / SRO Reactor Type: W / CE / BW / GE Start Time: Finish Time: 1 Instructions Use the answer sheets provided to document your answers. Staple this cover sheet on top of the answer sheets. The passing grade requires a final grade of at least 80.00 percent. Examination papers will be collected five hours after the examination starts. ( l I Applicant Certification l All work done on this examination is my own. I have neither given nor received aid. Applicant's Signature Results Examination Value Points Applicant's Score Points l Applicant's Grade Percent 43 of 45 NUREG-1021, Revision 8

G

9-1 0 4-S

o y r O E ) y t c r a m r o f s m it o ) e ) a F n o m S ( it a m. o )s r

o, m fr n t r o m nlo n e e f in t s wla og m s y n e s b )k .ni c n e

le d d e t n ebo en h a n lb e t t i n o a e ic ntot s e n d o e it a t p h a r cdu e la n e c t n inrq ir 6 a c c u o iuh o lp c la qi t it x ] s a e m o t new or d

E E tp a rno ce t t e e c n g n o, r a c a d g d w 2 o l e is lo f h t n e . n t n mt e u f o a g hc lee io lp e n ie ts f nh une r a iv i t t a nt ( oe n a r er la c t nin g k e d idt e er ns O o r r o o it n t n a ( r e f o e piasr d o t ot p e r e a e oi n t h ,.i n 4 e t n ie. re in - e s s(t n n n i, e e i, o ) it ndm t n p eol e . o m ime t r h le i it s wtep s a e m v ns c t a m )c. s a s , rt o e, d a,eg t . e tc . e u r b e it eg be( rss . iro on t g g n t q rn s Aet it oe n o e /f n ne a e it e m,l, u Keru p ah i c d ne me as r r s g h s es a lav osiust g e 5 n e n u mr ik ig g n t a m t 4 inr n i q ir

o .

S E

it c on H(

it a r h ap,s a alcs s e nre iuq ibr f o ExW 5 / u it r r r e st c ht ah e, b e t t U t a o ) t lcrn tn nrij no ( r a 4 n w - sn ml r a lu nee u,i n m a i efo re e icr 4 ei e i on ne ic lc c oh t kdc ra f it siy t lb te t v s a f f i gr . mte p p efic to a m t ier w Bw i ia mda - (eet . etas a uc a e hp o qerd t la n d y rde pue c c WR F s o nu a s e s he he scr c y ca a s t n # itnu

it i

                                           )         e              wfcla i

t t,oo ap )n p e d d F( ( sif ! e.ts fom o (U B n c/ , t n o a a r 5

                                                       -   dt: a pr     e eu        g itec (il e                                                                         n i x it              r                                            e dt:savn C             u             o h          1         ecs,t  e d           (                                    e d n          f it        a i

irs n t ret e e c ienheo f t b o i B e t e tlit it e s gn noua in meca f r M gtic w pp J. ix a is ecll r t 4 k d n n u i ish d er.J ce or e edi slip c dir elap a s A bnoi e e. u . ysq iu#e a n e JL p io n t ieo nr o it ht t ti r p s io wi(f lel l oba rrn uo s w la it A e u s lais o q e t flc c ct eeuinit oe dr i a ejoag orbk rof nic u h q e o r a t h u io rp e ca h q esnetcc) r P t rt e telhlo l in s w fe e c muilal or nhaie et cws t he lap la dt es

                          . .       R

[ f o ea oonc s ct o n (a noe t o ra e s x t r t hf e F ct co ais cde a t v i, t, i

                                           )       f ic        CrD                  K o yn sesrt      eos               ri bkt   ehde r n m
                                                    )                        r t

r O D pico r aco o nss j s e u e athene mg m L at m F ( O afit cs et rc a ri r iai o

T e L( ffua i sr i nqnqu d i u in f out h g ix strcit sods x osircr eoe j m c s d e lu ty osihed i s a 8 y s e lw ic bkdcn ea c r ror b nnnn e eoooo r n P u o tl o e o t 'e a t C n i f f a irmmw n atttiiiit w s, o pee shr am reeee pissss 3 k d e g is s uuuu e ivn mu ec f o o f ott rssa nte o orqqqq r it i v t o le le peeere peeee e a e SF v v phhhon phhhh h r R le e l aTTTMO e a e TTTT t

                                                                                                                            "U
                 .D)5                        e e h                                                                     n               ,

h h t ht o "y 1 2O- L (1 t t k k d n 2 r e r e ec e c e s ar 0 t t n n h h a o 1 K)H E E C- C- B F 1 1 . O/ F G L( E O 0 4- R S s O 1 . 2 3

4. 5 6 U

E N

9- 8 r 1 0 4-n io is S v E e _ R m r 1 o 2 F 0 1 G E R U . N n io t s 6

           .@    E                   ..

5 4 S f 2 5

           .      /

E

o U 5

                           . 4 kd r c

s aa w a Bw R s t n

                 /
                  # itnu t

e 1 n a o it C u b in o M J. 4 k bo n JU la . it r a l F s w a P dt es i ic t r CrD - e F - m T

o h . c y s s e P u C 3 mus _ tec o S F -

             . D )5                  .

2O- L (1 K)H - 1 1 .O/F - 0 L( - 4-( S E s O

m 4 .. [ - ES-402 ADMINISTERING INITIAL WRITTEN EXAMINATIONS 7 -A. PURPOSE i 7 ; i This standard specifies the requirements and procedures for administenng wntten examinations - l for the initial licensing of reactor operator (RO) and senior reactor operator (SRO) applicants at power reactor facilities. The standard includes instructions for proctoring the examinations and conducting post-examination reviews of NRC-developed examinations. B.' BACKGROUND ! CAs noted in ES 201, facility licensees will generally prepare the written operator licensing examinations, subject to review and approval by the NRC. Generally, examinations that are prepared by the facility licensee will also be administered by the facility licensee in accordance with the instructions contair.ed herein. C. BESPONS!SILITIES

1. Facility Ucensee i

a. The facility licensee shall safeguard me integrity and security of the examinations in accordance with ES-201.

O b .' The facility licensee shall provide a single roctn suitable for administering the wntten examination. To ensure examination integrity, the room shall be large , enough so that there is only one applicant per tabic, with a 3-foot space between tables. The examination room and supporting restroom facilities (i.e., the examindon area) shall be located to prevent the applicants from having contact with all other , facility and contractor personnel during the written examination. j

c. If desired and compatible with examination secunty requirements, the facility licensee may arrange for the applicants to have lunch, coffee, or other refreshments during the examination. j l

d. Before the scheduled examination date, the facility licensee should famihanze '

the applicants with the examination policies and guidelines contained in Appendix E. ! e.- The facility licensee shall provide the necessary copies of the approved examinations, answer sheets, and handouts (e.g., equation sheets, selected p technical specifications, and steam tables) for each applicant, as directed and I- approved by the NRC chief examiner. ;

The facility licensee may use machine-gradable answer sheets if desired, but this is not required. {

,D .                                                               1 of 6                     NUREG-1021. Revision 8 fw                                                                                                                     {

T

,l.

ES-402 ;

f. If the f acMity licensee developed the examination, it shall also administer the

, examination to the applicants identified on the examination assignment sheet j j (Attachment 4 of ES-201) as arranged with the NRC chief examiner and in j accordance with the specific instructions in Section D. l l

g. The facility licensee will send a letter to the NRC regional administrator to formally withdraw the applications of any indhriduals whose names appear on the examination assignment sheet (Attachment 4 of ES-201) but will not be taking the examination. l

h. As discussed in Section E, the facihty licensee should provide the NRC regional {

office with formal comments for consideration during the grading process (refer to ES-403). The facility licensee may also request an informal meeting with the NRC chief examiner to discuss the examination questions and resolve facility concerns.

2. NRC Reaional Office

a. The NRC regional office may administer the examination, at its discretion, in j accordance with the specific instructions in Section D, even if the examination was developed by the facility licensee. ~However, the regional office will generally arrange for the facility licensee to administer the examination. (Refer to ES-201 for further instructions on examination scheduling.)

If the NRC developed the examination, the regional office may arrange for an NRC examiner or the facility licensee to administer the examination.

b. If the facility licensee will conduct the examinations while the NRC examiners are ,

on-site, the chief examiner shou!d inspect the examination facilities to ensure ! their adequacy. In addition, the NRC examiners should periodically monitor the exam to ensure that the proctor is appropriately addressing the applicants' questions. If this is not feasible, the regional office should consider having an examiner check the facilities during the preparatory site visit (if one is deemed necessary) or upon arriving at the site for tha operating tests. 1 ! If tha facility hcensee will conouct the examinations when no NRC examiners are I on-site, the chief examiner will ensure that an NRC point of contact is available in the regional office to respond to facility questions while the examinations are being given. If the NRC prepared the examination, an examiner familiar with the e:camination content must be available to respond to the applicants' questions by telephone. The written examinations may be administered as soon as they and the license applications (including any applicable waivers) have been approved. The region shall not allow the written examination and operating test dates to diverge by s more than 30 days without obtaining concurrence from the NRR operator licensing program office. (Refer to ES-201 for additional guidance regarding examinations that have to be rescheduled to achieve an acceptable product.) NUREG-1021, Revision 8 2 of 6 G, ) _

m ;

  ,                                                                                                        1 I
                                                                                                           )

1 o ES-402 'k.Oj' l

            . c. When the applicants have completed the written examination, the chief examiner         l may conduct an examination review with the faci!Ry staff as described in Section       i E below.                                                                               l D. EXAMINATION ADMINISTRATION INSTRUCTIONS                                                       ,

1. Make Preparations

a. Arrange for the appl; cants to be proctored at all times while taking the written examination. Ensure that the proctor clearly understands his or her responsibilities (refer to Section D.2) before the examinations are distributed.

If the NRC will administer the examinations, the chief examiner should consider using the following resources to ensure adequate proctoring: NRC secretarial help another examiner other NRC employees The examiner may arrange for facility employees to proctor the examination for brief periods if it is necessary for the examirter to go to the restroom.

b. At least one individual who is familiar with the intent of the questions (i.e., an f NRC examiner or facility employee who took part in the examination l

[D) - development; shall be available to clarify examination questions for the applicants during the examination,

c. Remove from the examination area, or otherwise remove from the applicants' view, any wall charts, models, or other training materials that might compromise examination integrity.

l

d. Only NRC-approved applicants are allowed to take the examination. If applicable, the NRC examiner shall verify each applicant's identity and examination level against the examination assignment sheet (Attachment 4 of ES-201) before beginning the examination. Any errors or absences shall be resolved with the facility staff, and the assignrnent sheet shall be updated as required.

e. If possible, the RO and SRO applicants shall be seated at alternate tables. The proctor shall construct a chart illustrating the seatng arrangement of the applicants during the examination.

2. Star 1.the Examination i i

a. Remind the applicants that they may use calculators to complete the examination, and that only reference materials provided with the examination are allowed in the examination area (i.e., the examination room and supporting g restroom facilities).

k NUREG-1021, Revision 8 3 of 6 l

E i ES-402

b. Pass out the examinations, blank answer sheets, and all required handouts as i approved by the NRC chief examiner (e.g., steam tables, equation sheets, and ;

selected technical specifications). Instruct the applicants not to review the examination until told to do so.

c. Provide each applicant with a copy of Appendix E, " Policies and Guidelines for Taking NRC Examinations," and brief the applicants on the rules and guidelines that will be in effect during the written examination (i.e., review Parts A and B of the appendix). If time permits and the operating tests have not yet been administered, review those policies and guidelines (i.e., Parts C, D, and E of Appendix E) as well; this will save time later and give the applicants greater {

opportunity to resolve any questions they may have. i

d. Instruct the applicants to verify the completeness of their copies by checking l each page of the examination. j i

e. Answer any questions that the applicants may have regarding the examination l

policies. Start the examination, and record the time.

3. Monitor the Examination

a. The proctor shall give full attention to the applicants taking the examination. The proctor shall not read procedures or other material, grade examinations, or engage in any other activities in a manner that may divert his or her attention from the applicants and possibly cause the examination to be compromised.

b. Personnel responding to questions raised by tne applicants during the examination must be extremely careful not to lead the applicants or give away answers when clarifying questions. If the proctor has any doubt about how to respond to an applicant's question, it is best to withhold additional guidance and instruct the applicant to do his or her best with the information that is provided.

Any question changes or clarifications shall be made on a chalk board or white board,if available, and called to the attention of all the applicants. Changes made to questions during the examination should be made in ink on the NRC master copy and on a copy that is retained by the facility staff after the examination is administered. Changes shall be reviewed and approved by the NRC chief examiner as part of the grading process (refer to ES-403). All applicant questions regarding specific written examination test items and all statements of clarification shall also be documented (verbatim if possible) for future review by the NRC chief examiner and for reference in resolving grading conflicts.

c. The proctor shall periodically advise the applicants of the time that remains to complete the examination. Normally, a chalk board or white board is available and can be used for this purpose.

NUREG-1021, Revision 8 4 of 6

    '4.  'Comolete the Examination
         . a.        As the applicants completo the examination, ensure that they sign the examination cover sheet and staple it on top of their answer sheets. Collect the
                    . examination packages, including the questions and answer sheets, and any .

reference material provided with the examination. . Verify that all applicants have entered their names on both the answer and cover sheets, and record the official start time and the time at which each applicant completed the examination in the space provided on the examination cover sheet. b .- Retain the cover and answer sheets for grading in accordance with ES-403. The ,

                    - question books may be distributed to the applicants after the last examination J

has been collected. !

c. Remind the applicants to leave the examination area, as previously defined,

d. When five hours have elapsed, instruct the remaining applicants to stop work,

                    ' sign their examination cover sheets, and tum in their examinations. The time        !

allowed to complete the examination shall not be extended without prior approval j by the NRC regional office, if it becomes necessary, the regional office may authorize additional time in 30-minute increments. 1

e. Deliver the completed examination packages, the marked-up master I examinations, the list of applicant questions and answers, and the seating chart to the NRC chief examiner or the appropriate facility representative, as l

applicable, for review and grading in accordance with ES-403. E. POST-EXAMINATION REVIEWS

1. If the NRC administered the examination, the chief examiner shall ensure that the master copy of the examination reflects all changes made to questions during the administration of the examination. The chief examiner will then provide a copy of the master examination and answer key to the facility staff and answer any questions they may have regarding the NRC's examination review and comment process.

2. If the NRC developed the examination, the chief examiner will also provide the facility licensee with a copy of the examination as edited during the facility prereview. If the facility reviewers believe that the NRC did not adequately resolve the prereview comments, they should address those concerns in a formal comment letter. i

3. The NRC chief examiner will request that the facility prereviewers confirm that they did ,

not divulge any information about the examination (s) by having them sign the post-examination security statement (Form ES-201-3) after the examinations are completed.

4. The facility licensee should submit formal comments within five working days after the examination is administered. However, the facility licensee may expedite the grading process by giving draft comments to the NRC chief examiner before he or she leaves

 .O                                                    5 of 6                   NUREG-1021, Revision 8

ES-402 the site. The NRC will consider comments not submitted within the requested time on a case-by-case basis; however, late comments may delay the examination grading process. The facility licensee is also encouraged to collect and consider comments from the license applicants and include them in its submittal to the NRC.

5. The facility licensee should submit all comments in the following format:

List the question, answer, and reference. State the comment and make a recommendation whether the answer should be changed or the question should be deleted. If the facility licensee does not support an applicant's comment, it should briefly explain the reason for its rejection. Support the comment with a reference, and provide a copy if it was not included in the original reference material submittal. (Note: The NRC will not change the examination without a reference to support the facility's comment.)

6. Formal comments should be signed by an authorized facility representative and addressed to the responsible NRC regional office, with a copy to the NRC chief examiner.

1 I I l l l l NUREG-1021, Revision 8 6 of 6 Ojl l l l 1 ______________________________-_-___w

I l l I l LYT ES-403 l GRADING INITIAL SITE-SPECIFIC WRITTEN EXAMINATIONS A. PURPOSE This standard explains the requirements and procedures for grading the site-specific written ! examinations for the initial licensing of reactor operator (RO) and senior reactor operator (SRO) applicants at power reactor facilities. The standard includes instructions for evaluating and revising the examinations after they are administered, grading the examinations, and conducting the first review of the graded examinations. B. BACKGROUND. l As discussed in ES 201, facility licensees will generally develop and administer the initial operator licensing written examinations, subject to review and approval by the NRC. Facility licensees will also be expected to grade the written examinations, evaluate the outcome, and - submit the examination results to their NRC regional office for review, approval, and licensing action in accordance with ES-501. C. RESPONSIBILITIES

1. Facility Licensee

a. If the facility licensee developed and administered the written examinations, the licensee is also expected to perform the following grading activities, as described in Section D:

Review and resolve any questions and comments that arose during and after the examination (refer to ES-402). Grade the examinations and review the grading using Form ES-403-1,

                             " Written Examination Grading Quality Checklist."

I Evaluate the applicants' performance on the examination. i Facility management will review the examination grading based on the guidance in ES-501 and forward the graded examinations and all associated documentation to the NRC ch%f examiner so that it is received, when practical, within five working days after the examination was administered.

b. If the NRC developed the examinations, the facility licensee's responsibility is limited to providing the NRC chief examiner with comments and recommendations regarding question deletions and answer key changes. Such comments and recommendations should normally be received within five working days after the exit meeting; any delay in submitting the comments will likely result in a comparable delay in the finallicensing actions. (Refer to ES-402 for l additional instructions regarding the post-examination review and commerit ,

l process.) r lV 1 of 5 NUREG-1021, Revision 8

i

2. NRC Reaional Office

a. If the facility licensee grades the examinations, the regional office shall provide guidance and assistance, as necessary, to ensure that the facility licensee complies with the instructions in Section D.

b. If the NRC developed the examinations, the regional office should grade the examinations in accordance with Section D after receiving any comments and recommendations from the facility licensee (refer to ES-402). The regional office may take advantage of the facility licensee's machine grading capability if it is available.

c. After the examinations have been graded, the regional office shall review the grading, process the documentation, and complete the licensing actions in accordance with ES 501.

D. GRADING INSTRUCTIONS The author of the examination should normally grade the examination; however, the examination may be graded by another equally qualified individual if the author is not available, the number of applicants is unusually large, or the NRC regional office or facility licensee wishes to expedite the grading process. The examinations shall be graded as expeditiously as possible, in accordance with the following instructions:

1. Evaluate Questions and Comments

a. Evaluate all questions posed by the applicants during the examination, any pen-and-ink changes made on the master examination during its administration, and I any post-examination comments or recommendations received from the facility licensee and applicants after the examinations were administered. Determine if any questions should be deleted from the examination, or if any answers need to be changed. Do not delete any question or change any answer unless there is a valid reference to support the change. .An unreasonable assumption on the part of an applicant does not justify the acceptance of an alternate answer.

If there is some doubt whether the NRC chief examiner will accept a proposed change, the grader is encouraged to discuss the matter with the chief examiner before proceeding with the grading process. This may help to minimize the need for grading corrections during the quality reviews. For each comment and recommendation, the NRC chief examiner shall docurnent the reason that the question was changed or the comment was not l accepted; this information will be included in the examination report as discussed in ES-501. l l

b. If it is determined that there are two correct answers, both answers will be l

l accepted as correct. However,if three or more answers could be considered l NUREG-1021, Revision 8 2 of 5 l

ES-403 correct or there is no correct answer, the question shall be deleted. Annotate the recommended changes on the master examination and answer key and , document the reason for every change or deletion. )

c. Those applicant questions, facility comments, and recommendations that do not result in answer key changes or question deletions, should be evaluated to I determine if the associated test questions might benefit from editorial changes before they are used on another examination.

2. Grade the Examinations

a. Copy each applicant's answer sheet, and set the copies aside for later use

              - during the grading review process.
      ' b. On each applicant's original answer sheet, indicate in redpen orpencilwhich questions were answered incorrectly, note their correct answers, and indicate which questions (if any) were deleted, if the answer sheet is more than one page long, it is helpful to note the total number of incorrect answers on each page to aid in tabulating the final grade.

If the examinations are graded by machine, attach a copy of each applicant's ] profile report to his or her answer sheet, or manually annotate the answer sheet i as noted above. I e I

c. If it is necessary to change a grade during the grading process, do so by lining out the original grade in such a way that it remains legible. Briefly explain the reason for the change on the applicant's answer sheet, and initial the change.

Under no circumstances will a grader use " white-out" or other methods that obscure the change,

d. After grading all the questions, enter the " Examination Value" (i.e., the original test point total minus the point value of any deleted questions), the " Applicant's Score," and the " Applicant's Grade" (i.e., the Applicant's Score divided by the Examination Value) in the 'Results" section of the applicant's written examination cover sheet.

If a facility chooses to share its preliminary grades with the applicants, it should caution them that the outcome may change if the NRC does not accept all of the

              . facility licensee's recommended changes to the examination answer key.

3. Evaluate and Review the Gradina

a. -. Evaluate the applicants' performance on each examination question to identify any indications of a problem with the question or a deficiency in the applicants' training program. A table that summarizes the applicants' answers on each question, or a computerized item analysis (if the examinations were graded by machine) may be used to identify items with which the applicants had problems.

3 of 5 NUREG-1021, Revision 8

1 l l l I ES-403 If it appears that a test question was faulty, determine whether the question should be deleted, the answer key should be changed, and/or the question { should be revised before reuse. Then regrade the examinations as necessary. If it appears that the training program was deficient, determine the need for remedial training and/or a program upgrade.

b. After evaluating the examinations, review the grading in detalland complete Form ES-403-1, " Examination Grading Quality Checklist."

c. Forward the examination package (i.e., the master examination and answer key, justification for any examination changes, any item analysis that was performed, the applicant's examination cover and answer sheets (the graded original and one clean copy), and Form ES-403-1) to the designated facility representative (if applicable) or to the NRC chief examiner for review in accordance with ES-501.

E. ATTACHMENTS / FORMS Form ES-403-1, " Written Examination Grading Quality Checklist" O NUREG-1021, Revision 8 4 of 5

J

.O       ES-403                               Written Examination Grading                    Form ES-403-1 Q                                                  Quality Checklist Facility-                                    Date of Exam:                  Exam Level: RO/SRO initials item Description                                 a        b       c

1. Answer key changes and question deletions justified and documented

2. Applicants' scores checked for addition errors (reviewers spot check > 25% of examinations)

3. Grading for all borderline cases (80% +/- 2%) reviewed in detail

4. AH other failing examinations checked to ensure that grades are justified

5. Performance on missed questions checked for training deficiencies and wording problems; svaluate validity of p questions missed by half or more of the applicants Printed Name / Signature Date

a. Grader 1

b. Facility Reviewer (*) _

i

c. NRC Chief Examiner (*) l l

d. NRC Supervisor (*) )

(*) The facility reviewer's signature is not applicable for examinations graded by the NRC: two independent NRC reviews are required. l .i

 \                                                        5 of 5                    NUREG-1021, Revision 8

O O O

i O ES-501 INITIAL POST-EXAMINATION ACTIVITIES A. PURPOSE l l This standard describes and coordinates the activities that must be completed after the written examinations and operating tests have been administered and graded in accordance with the l ES-300 and ES-400 series. Specifically, the standard includes instructions for assembling and reviewing the examination package, notifying the facility licensee and applicants of the examination results, preparing the examination report, and retaining examination records. B. BACKGROUND The goal of the NRR operator licensing program office is to complete licensing or denial actions within 30 days after the facility licensee submits the graded examinations or its formal written examination comments to the NRC. The NRC and facility licensee staffs should establish their priorities and schedules to achieve this goal. Applicants must achieve a grade of 80 percent or greater on the written examination and a grade of " satisfactory" on all three categories of the operating test to qualify for a license. C. RESPONSIBILITIES

1. Facility Licensee 1O' lt a. If the facility licensee participated in developing, administering, and grading the examination, the licensee shall forward the following examination documentation to the NRC chief examiner (' addressee only") so that it is received, when practical, within five working days after the examinations were administered:

the graded written examinations (i.e., each applicant's original answer and examination cover sheets) plus a clean copy of each applicant's answer sheet (ES-403) the master examination (s) and answer key (s), annotated to indicate any changes made while administering (ES-402) and grading the examination (s) (ES-403) any questions asked by and answers given to the applicants during the written examination (ES-402) any comments made by the applicants after the written examination with an explanation why the comment was accepted or rejected (ES-402) the written examination seating chart (ES-402) a completed Form ES-403-1," Written Examination Grading Quality Checklist" (ES-403 and Section D.1) the results of any written examination performance analysis that was performed (ES-403) original Form (s) ES-201-3, " Examination Security Agreement," with a pre-and post-examination signature by every individual who had detailed knowledge of any part of the written examination or operating tests before they were administered. [ \ 1 of 22 NUREG-1021, Revision 8

ES-501 Refer to the referenced Examination Standards for a more detailed discussion of each documentation requirement.

b. If the facility licensee did not participate in developing, administering, and grading the examination, the licensee should submit comments and recommendations regarding the NRC-developed written examination to the NRC regional office within five working days (when practical) after the exit meeting.

The facility licensee should also include and consider comments made by the license applicants that took the examination. (Refer to ES-402 for more detailed instructions.)

2. NRC Reaional Office

a. The NRC regional office shall ensure that the operating tests and written examinations are graded in accordance with ES-303 and ES-403, respectively.

b. The NRC regional office shall ensure that the examination results and licensing recommendations receive the required reviews and approvals in accordance with Section D, that the associated administrative requirements are completed in accordance with Section E, and that the required records are retained in accordance with Section F.

The regional office may use Form ES-501-1, " Post-Examination Check Sheet," to track completion of the administrative items after the examinations are administered.

c. NRC regional management should also review the overall examination results and any generic findings, deficiencies, or issues to determine if any follow-up action is required.

If the facility licensee recommends deleting or changing the answers to five percent or more of the questions on a written examination that it developed, the regional office should request that the facility licensee explain why so many post-examination changes were necessary and what actions w;ll be taken to improve future license examinations. If ten percent or more of the examination questions are deleted during the grading process, the region shall evaluate the remaining examination to ensure that the test outline sampling requirements in ES-401 are still satisfied. The training and assessment specialist on the program office staff should be consulted if the validity of the examination is in question. If the content validity of the examination is affected (e.g., several knowledge and ability (K/A) topics are not covered, or the majonty of the remaining K/As are associated with a small number of systems) as a result of deleting questions, NRR operator licensing program office will make a decision whether the examination should be voided. NUREG-1021, Revision 8 2 of 22

D. EXAMINATION REVIEWS AND LICENSING ACTION Except as noted below, the quality reviews generally constitute spot checks, or sampling, to follow up on the work pedormed by the written examination and operating test graders in accordance with ES-403 and ES-303, respectively. If the quality reviews indicate significant problems, additional detailed review will be necessary. Reviewers should discuss all grading discrepancies with the grader or previous reviewer before making any changes. In addition, the reviewers shall document any changes by carefully lining out the original entry so that it remains legible, entering the revision with a brief explanation, and initialing the change. Reviewers shall not use " white-out" or other methods that obscure the original entry.

1. Facility Manaaement if the facility licensee graded the written examinations, a supervisor or manager shall confirm the quality of the grading and sign the bottom of Form ES-403-1 before sending the examinations to the NRC regional office.

The NRC will consider the signed form to represent facility management concurrence with the individual and collective examination results, including the justification for any examination changes.

2. NRC Chief Examiner (or desianated alternate)

The written examination grading shall be independently reviewed by at least two NRC personnel using Form ES-403-1 as a guide. If the examination was graded by the chief examiner, then another examiner shall conduct the independent review. If the chief examiner conducts the independent review, he or she may not pedorm the supervisory review required by Section D.3.

a. If the facility licensee graded the written examinations, the chief examiner shall immediately inventory the examination package to ensure that all required materials have been submitted. The chief examiner shallinform the responsible ,

supervisor of any obvious deficiencies, and shall contact the facility licensee to i determine the status of any missing documentation.

b. The chief examiner shall independently analyze each examination and answer key change made or recommended by the facility licensee or a license applicant to determine whether it is justified. The chief examiner shall ensure that the reason for accepting or rejecting each change or recommendation is documented in the examination report. The report shall briefly state the region's basis for accepting or rejecting each facility comment; simply stating ;

concurrence with no explanation is not sufficient. The chief examiner will not ; accept a change to the examination unless the facility licensee submits a valid reference to support its recommendation. N 3 of 22 NUREG-1021, Revision 8

ES-501

c. The chief examiner shall review the remaining items on Form ES-403-1. The chief examiner should apply his or her judgment when reviewing the examination results and adjust the level of the review based on the performarice of the applicants and the facility licensee (e.g., the number of questions changed or deleted, the average grade, the number of borderline or failing grades, etc.). If the examination was graded by machine or using a template, the chief examiner shall ensure that the template accurately parallels the approved answer key.

The chief examiner shall independently grade every borderline examination (i.e., those between 78% and 82%) using the final, approved answer key and the clean applicant answer sheets provided by the facility licensee.

d. The chief examiner should review the written examination results and the facility i licensee's performance analysis (if applicable) for indications of the following: !

1 deficiencies in the applicants' training program, so that they may be ! addressed in the examination report I poor question construction, so the applicants are not graded unfairly, any significant problems can be addressed in the examination report, and the questions are corrected before reuse l any indications that the examination was compromised

e. When satisfied with the examination grading, the chief examiner shall sign and date Form ES-403-1 and pass it on to the responsible supervisor for management review (see item D.2.h).

The chief examiner shall also ensure that the written examination results and the names of the NRC examiners who wrote, graded, or reviewed the examinations are recorded in the " Written Examination Summary" section of each applicant's

          " Individual Examination Report," Form ES-303-1.

f. The chief examiner shall also review, in detail, the other examiners' operating test documentation to ensure that the test (as given) and its grading meet the requirements in ES-301 and ES-303. In so doing, the chief examiner shall i ensure that the other examiners' operating test comments support the pass or I fail recommendations and check for consistent documentation and grading among the operators tested on the same simulator crew.

If the documentation is accurate and complete, and the licensing recommendation is appropriate, the chief exarniner shall check " Pass" or " Fail" and sign and date the " Final Recommendation" block on Form ES-303-1. If the licensing recommendation is not appropriate based on the documentation presented, the chief examiner shall discuss the examination findings with the NRC examiner of record and resolve any disagreement. NUREG-1021, Revision 8 4 of 22

ES-501 If the chief examiner administered the operating test, the responsible regional supervisor shall designate another examiner to independently review the documentation and sign the " Final Recommendation" block on page 1 of Form ES-303-1.

g. The chief examiner shall record the results of the written examinations and operating tests on Form ES-501-2, " Power Plant Examination Results l Summary." -

l

h. The chief examiner shall ensure that the examination documentation is complete

! and contains a!! of the items identified in Section F before forwarding the entire package to the responsible supervisor for review and approval in accordance with Section D.3. l If the written examinations were administered much before the operating tests, the chief examiner should enter that data on the form and forward it with the l completed written examination package to the responsible supervisor for review and approvalin advance of the operating test results.

3. NRC Manaaement Review and Licensina Action 4

a. The responsible supervisor shall ensure that all examinatiori results and l

documentation are complete. The supervisor shall evaluate the written > l examination results, ensure that the required quality reviews were completed, work with the chief examiner and the facility licensee (as necessary) to resolve any grading problems, and then sign and date Form ES-403-1 to document approval of the process. l Every written examination shall have at least two levels of NRC review. l Therefore, the NRC examiner who performed the regional quality review is l disqualified from also performing the supervisory review. i l b. The responsible supervisor will also independently review the operating test l results, check either the " Issue License" block or the " Deny License" block in the

               ' License Recommendation" section of each applicant's Form ES 303-1, and sign l

l and date each form. Under no circumstances will all three levels of

recommendation on Form ES-303-1 (i.e., operating test administrator, chief examiner, and NRC supervisor) be signed by the same individual.

If the responsible supervisor (or licensing official) does not believe that the operating test documentation supports the final recommendation, he or she shall - consult the NRC examiner of record and the chief examiner to discuss and resolve any disagreements. If a recommendation is overturned during the regional management review, the responsible supervisor will line out and initial the affected summary evaluations.

 .A            The supervisor will then enter the new summary evaluation in the appropriate 5 of 22                     NUREG-1021, Revision 8 l

m

r I I ES-501 l block, and explain the change on Form ES 303-2, " Operating Test Comments," l and attach it to the applicant's Form ES-303-1. l

c. After making the licensing recommendations, the responsible supervisor will i have the operator licensing assistant prepare a license, denial, or notification letter for each examined applicant and forward the examination package to the regional licensing official. Applicants who withdrew before taking any part of the )

license examination shall not be sent a denialletter. Attachments 3 and 4 provide sample RO and SRO (conditional) license letters and a sample denial letter. Attachment 5 is a sample letter for use in notifying applicants that they passed , the examination, but that their licensing action will be delayed. For example, if l an applicant was granted a waiver (refer to ES-202 and ES-204) and allowed to j take the examination before completing all of the training and experience requirements, the regional office shall normally not issue a license to the applicant until the facility licensee has certified in writing that the applicant has completed all of the waived items. Likewise, if any of the applicants failed the written examination, the regional office shall analyze the question-by-question performance of those applicants who scored 81 percent or lower on the examination to ensure that any question deletions or changes will not affect the licensing decision. And, if necessary, the regional office shall delay issuing licenses to those applicants until any written examination appeals have been reviewed for impact on the licensing decisions. Before issuing a license in either instance, the regional office shall ensure that: (1) the applicant has been determined to be medically fit within the past 24 l months; (2) the applicant has not developed any permanent physical or mental ) condition that would be reportable under 10 CFR 55.25; and (3) the applicant is l up-to-date in the requalification training program. Moreover, the regional office shall advise the facility licensee to properly activate the individual's license in accordance with 10 CFR 55.53(f) if more than three months have passed since the examination results were issued. If a licensing action is delayed for any reason, the effective date of the license i will be the date on which it is issued; licenses will not be backdated. l

d. The final licensing decision is made by the NRC regional administrator or a designated attemate, who must be at or above the level of branch chief; short-term alternates shall not rnake licensing decisions. The licensing official will consider all recommendations, make changes as described above, and sign each applicant's license, denial, or notification letter, as applicable.

NUREG-1021, Revision 8 6 of 22 1

ES-501 \ E. EXAMINATION FOLLOW-UP

1. Notifv Facility Licensee of Results The NRC regional office will notify the facility licensee and applicants of the examination results (as described below) only after they rae reviewed and approved by the licensing official.

a. The regional office should normally notify the facility licensee's designated representative of the examination results by telephone, and may confirm the results by mailing a copy of Form ES 501-2 under a separate cover letter. For each applicant that failed or had significant deficiencies that warrant further evaluation and retraining by the facility licensee, the regional office will also send to the facility licensee a copy of the applicant's Form ES-303-1 and written ext,mination answer sheet. These form (s) shall not be placed in the public docu-ment room or distributed with the final examination report.

If the written examinations were administered much before the operating tests I and management has approved the results of those examinations, the regional office may notify the facility licensee of those results rather than waiting until the operating tests are completed.

b. After the licensing official has signed the license, denial, and notification letters, the regional office shall send each applicant's letter along with the following materiald:

a copy of Forms ES-303-1, ES-303-2, and ES-D-1 (and Forms ES-D-2 if the applicant failed Category C of the operating test) reflecting the "as run" scenario conditions but without any rough examiner notes regarding the applicant's performance (pen-and-ink markups of the original, l approved scenarios are acceptable) l a copy of the applicant's written examination cover and answer sheets (as well as a copy of the master written examination and answer key if the applicant failed the written examination)

c. The regional office shall send a copy of Form ES-501-2 to the NRR operator licensing program office. If any of the examinations are later regraded in response to an applicant's request for review (refer to ES 502), the original Form ES-501-2 on file in the regional office shall be corrected by lining out the old grade, entering the new grade, and initiating the change. Whenever a change is made, the regional office shall mail a copy of the revised form to the program office.

7 of 22 NUREG-1021, Revision 8

ES-501

2. Retum the Facility Reference Material If desired by the facility licensee, the NRC chief examiner shall ensure that the reference materials provided for NRC examiners to prepare for the examinations are returned as soon as possible. Il none of the applicants failed the examination, the materials should be returned as soon as the licenses are issued. If any applicant was denied a license based on an examination failure, the reference materials should be retained pending expiration of the 20-day period during which the applicant may request a regrade. If an applicant requests a regrade in accordance with ES 502, the chlef examiner shall deter-mine what reference materials need to be retained and should return all unnecessary materials. All reference materials should be returned to the facility licensee within 30 days following the resolution of any appeals.

3. Preoare the Examination Reoort The NRC chief examiner shall prepare the final examination report when all portions of the examination have been graded and documented. If the regional office delays some licensing actions in accordance with Section D.3, it should issue and later amend the examination report. The examinor shall use NRC Manual Chapter 0610," Inspection Reports," when preparing the report.

a. The final examination report shall document the following:

a factual description of the test item changes, including the type and number of psychometric enhancements made and agreed upon between the facility licensee and NRC examiners as a result of the joint NRC and facility licensee examination review process. Conclusions regarding the adequacy of the facility proposed examinations are not required and should only be considered if the Region has concluded that the finding has programmatic aspects, e.g., multiple examples of examination submittals with an excessive number of unacceptable test items. any delay in administering the examination and the reason for the delay, and any extensions of the written examination time beyond five hours the results of the examination, including any generic strengths and weaknesses noted while administering the operating tests or grading and reviewing the written examinations, and any significant grading deficiencies if the examinations were graded by the facility licensee any examination security issues and incidents or other matters requiring facility attention (consistent with NRC enforcement policy) any other issues discussed at the exit meeting NUREG-1021, Revision 8 8 of 22

                                                                                                )
                                                                                                )

1 l

f ) i l ! I

 /7                                                                                            ES-501 l        b. The report shall include the following items, as applicable:

l l - a copy of the master written examination (s) and answer key (s) 1 l a copy of the facility licensee's (and applicants') specific comments and recommended changes regarding the written examination that was i administered i i the specific NRC resolution, including a precise explanation for accepting or rejecting each facility comment, for each facility recommendation and l a specific justification for every additional item deletion or change (Refer to Attachment 1 for examples of facility comments and NRC resolutions.) a simulation facility report (as described below) i Generic comments submitted by the facility licensee about the examinations or j the administration process should also be included in the report; however, such comments do not necessarily require a regional response or resolution.

c. The simulation facility report shall document the NRC examiners' evaluation of 3 a

the performance or fidelity of the simulation facility during the preparation or conduct of the operating tests. A sample report is provided in Attachment 2. l (p) All simulator deficiencies encountered while preparing or conducting the operating tests should be describea in sufficient detail to allow screening and j classification during a simulation facilityfollowup. The NRC examiners may include in the simulation facility report any concerns about physical fidelity i (hardware or equipment discrepancies) or functional fidelity (performance of the simulation facility during normal, surveillance, abnormal, or emergency events). i Each deficiency should include a description of the cperation, event, or transient j that was in progress, and how the simulation facility failed to accurately model ' the expected performance of the reference plant. I

d. The applicants' names and specific grades (i.e., Form ES-501-2) shall not be j published in the examination report. f

e. The NRC regional office shall send the final examination report to the facility l licensee and ensure that a copy is made available to the public.

4. Perform Other Activities

a. If an applicant did not complete the SRO upgrade training program or failed the upgrade examination, regional management should ensure that the RO licensee l complies with the requirements of 10 CFR 55.53(e), (f), and (h) and 10 CFR 55.59(a) before resuming active duties as an RO. i i

f} d 9 of 22 NUREG-1021, Revision 8 I l I ___w

ES-501

b. The NRC regional office should also conduct a case-specific review of the SRO upgrade exemination to determine if the applicant failed as a result of significant deficiencies in RO knowledge or abilities. Pursuant to 10 CFR 55.7, the NRC may, by rule, regulation, or order, impose upon any licensee additional requirements deemed appropriate or necessary to protect public health and to minimize danger to life and preparty. If the SRO upgrade applicant's deficiencies pose such a threat, the NRC may require the facility licensee to provide remedial training and reevaluation and to submit evidence of its completion to the NRC.

c. Once the licensing decisions are complete, the NRC examiners should discard any marked-up documentation or rough notes for those applicants receiving licenses (except as noted below). In accordance with ES-502, NRC examiners should retain all applicable notes and documentation associated with proposed denials until the denials become final; this may include simulator operating test notes regarding crew members that passed the test if the notes contain information relevant to the failing applicant's performance. Examinerr are advised that such notes would be subject to disclosure if requested under the Freedom of Information Act.

d. Agency policy requires that all documents submitted to the NRC for review and approval be made available to the public. Therefore, the NRC regional office shall ensure that a clean copy of the examination outline, the draft written examination (s), the draft operating test (s), the associated quality checklists, and other documents (excluding those containing information protected under the Privacy Act and internal NRC docurrents developed during the review process) are routed to the NRC's document control system for public dissemination. The final written examination (s) and operating tests must also be made available to the public; however, the intermediate working copies of these documents need ,

not be released to the public unless the NRC regional office provided a copy to i the facility licensee to facilitate the communication of deficiencies and required changes. NRC Manual Chapter 0620," Inspection Documents and Records," q provides additional policies and guidance in this area. 1 F. NRC RECORD RETENTION

1. The NRC regional office shall ensure, for the last iritial examination at each facility, that i the original (whenever possible) or a copy of the following items are either retained in I

the facility's master examination fiis or are electronically available via the NRC's document access and management system. The italicized items should be retained or available for the last two examinations at each facility so examiners can verify compliance with the limits on test item repetition.

a. ES-201, Attachment 3, " Corporate Notification Letter"

b. ES-201, Attachment 4, " Examination Assignment Sheet,' with pen-and-ink ,

changes to identify the applicants that were actually examined i l NUREG 1021, Revision 8 10 of 22

l l ES-501 l

c. Form ES-201-1, " Examination Preparation Checklist"

d. the written examination and operating test outline (s), along with Form ES-201-2,

           " Examination Outline Quality Checklist"

e. the proposed NRC- or facility-developed written examination and operating tests (including comments made by the facility licensee or the NRC, as applicable)

f. the final written examination and answer key (enclosure to examination report) ;

with all changes incorporated (the pen-and-ink corrections made for the i applicants while the examination was administered may be changed to typewritten corrections; however, all changes shall be annotated in such a way that they are evident), Form ES-401-7, " Written Examination Quality Checklist," and Form ES-401-9," Written Examination Review Worksheet" I

g. the as-given scenarios including Forms ES-D-1, ' Scenario Outline,"and ES-D-2, l

           ' Operator Actions,' for each scenario set administered, as well as the as-given      \

walk-through tests including Forms ES-301-1, ' Administrative Topics Outline,' and ES-301-2, ' Control Room Systems and Facility Walk-Through Test Outline,' and the JPMs for each walk-through test (all record copies should have the required signatures and reflect the "as run' test conditions; pen-and-ink markups of the original, approved forms are acceptable)

h. for each operating test administered: Form ES-301-3, " Operating Test Quality Checklist," Form ES-301-4, " Simulator Scenario Quality Checklist," Form ES-301-5, " Transient and Event Checklist," and Form ES-301-6, " Competencies q Checklist"

i. Form ES-403-1, " Written Examination Grading Quality Checklist' J. Form ES-501-2, " Power Plant Examination Results Summary Sheet'

k. ES-501, Attachment 1, ' Examination Report," with all enclosures

1. Form ES 201-3," Examination Security Agreements *

2. The NRC regional office shall place the following items in each applicant's docket file:

a. Forms ES-303-1, " Individual Examination Report," ES-303-2, " Operating Test !

Cornments" (original copies, all pages, including strip charts and other attachments that support the licensing decision), and ES-D-1, " Scenario Outline," as well as Form (s) ES-D-2, " Operator Actions," if the applicant failed Category C of the operating test (all record copies should have the required signatures and reflect the "as run' test conditions; pen and-ink markups of the original, approved forms are acceptable)

b. all correspondence with the applicant i 11 of 22 NUREG-1021, Revision 8

1 ES-501

c. the applicant's original written examination cover and answer sheets I

l G. ATTACHMENTS / FORMS Attarhment 1, " Sample Facility Comments and NRC Resolutions" , " Sample Simulation Fidelity Report" , " Sample License Letters' , " Sample Proposed Denial Letter" , " Sample Notification Letter" Fo,m ES-501-1, " Post Examination Check Sheet" Form ES-501-2, " Power Plant Examination Results Summary" j l O 1 NUREG 1021, Revision 8 12 of 22

l ES-501 Sample Facility Comments and NRC Resolutions Attachment 1, Question #28 Comment: The question asks for the required method of securing a diesel generator and ensuring that an auto restart does not recur following auto initiation on receipt of a valid LOCA signal with off-site power still available to its associated emergency bus. The question is

recommended for deletion since the system operating piocedure directs that the diesel be unloaded, verifying that the 4KV bus auto transler annunciator is reset, and then secured by

       - placing the handswitch in pull to lock. Therefore, the key answer - ensure the "4KV AUTO TRANSFER INOP' annunciator is lit before placing the control switch in PULL TO LOCK - is incorrect.

NRC Resolution: Recommendation accepted. The question is deleted due to no corract answer. The intended answer specified that the annunciator be confirmed as " lit" when it should have specified ' reset" in accordance with system operating procedure No.123 Section 5.1 (Rev. 29). Question #81 i Comment: The question asks for a description of RHR Loop B outboard injection valve l operation if level rapidly decreases to 119.5 inches with the RHR loop B operating in the Shutdown Cooling Mode. The question is recommended for deletion since the outboard injection valve reopens automatically when the Group 4 isolation is reset, if a LPCI loop

       - selection is sealed-in. Therefore, the key answer - the operator must reset the shutdown             l cooling isolation and manually reopen the RHR Loop B outboard injection valve - is incorrect.        l

\ - NRC Resolution: Recommendation not accepted. The RHR Loop B outboard injection valvo L will not auto-open unless the operator manually resets the shutdown cooling isolation signal. Therefore, the use of the phrase " manually reopen' is correct, and the key answer is correct. The facility provided reference justification supports that manual action is required to open the injection valve. Question #96

        ' Comment: The question asks for the condition that will prevent the standby diesel generator upon restoration of power after a station blackout. The facility recommends acceptance of an additional answer - restore power to emergency bus 2B3 prior to restoration of power to the diesel - since the reference AOP indicates that if power is restored to bus 2B3 before 125 VDC

is restored to the standby diesel generator starting logic, then there would be no Loss of Off-i

        . site Power (LOOP) signal available for generation of an auto start signal,                          j NRC Resolution: Recommendation accepted; the question has two correct answers (a and c).
        . Figure 8 of system description No.123, Rev. 3 for Bus 283 shows that bus 2B3 cannot be              ;

restored without restoring voltage to the secondaries of the startup and standby transformers thereby removing the LOOP diesel auto start signal. The AOP reference cited by the facility l was not considered relevant since it did not state that the standby or startup transformers had to be energized in order to energize the 2B3 bus, ( 13 of 22 NUREG-1021, Revision 8 I i 4

ES-501 _ Sample Simulation Facility Report Attachtnent 2 Facility Licensee: (Facility name) . Facility Docket No.: _. (number) Operating Tests Administered on: (date) This forrn is to be used only to report observations. These observations do not constitute audit or inspection findings and, without further verification and review, are not indicative of noncompliance with 10 CFR 55.45(b). These observations do not affect NRC certification or approval of the simulation facility other than to provide information that may be used in future evaluations. No Ibensee action is required in response to these observations. I While conducting the simulator portion of the operating tests, examiners observed the following items (if none, so state): (EXAMPLES) IT_EM DESCRIPTION HPSI header F. The pressure instrument read mid-scale regardless of actual pressure pressure. (PI 301) Head bubble During a scenario that caused a rapid depressurization during natural , circulation, the vessel head level indication indicated a void (bubble). The confirming indications (i.e., pressurizer level and pressure) failed to verify or confirrn the bubble. Stearn The meter has been out of service for the last three generator A operating tests (approximately 18 months). wide range level . NUREG-1021, Revision 8 14 of 22 ; _ _______a

O, ES-501 Sample License Letters Attachment 3

  )                                                                                                       l NRC Letterhead (Date)

LICENSE (Acolicant's name) (Street address) (City. State. Zio code) Pursuant to the Atomic Energy Act of 1954, as amended; the Energy Reorganization Act of 1974, as amended; and Public Law 93-438, and subject to the conditions and limitations incorporated herein, the Nuclear Regulatory Commission hereby licenses you to manipulate all controls of the (Name of facilitv. facility license number). Your License No. is OP-(number). Your Docket No. is 55-(number). The effective date is (date). Unless sooner terminated, renewed, or upgraded, this license shall expire six years from the effective date. This license is subject to the provisions of Title 10, Section 55.53, of the Code of Federal Regulations, with the same force and effect as if fully set forth herein. ( While performing licensed duties, you shall observe the operating procedures and other Q] conditions specified in the facility license authorizing cperation of the facility. The issuance of this license is based upon examination of your qualifications, including the representations and information contained in your application for this license. A copy of this license has been made available to the facility licensee. For the Nuclear Regulatory Commission, (Name and title of licensina official) Docket No. 55-(number) cc: (Facility representative who sianed the aoolicant's NRC Form 398) O- 15 of 22 NUREG-1021, Revision 8

Es-501 Attachment 3 NRC Letterhead (Date) LICENSE LApolicant's name) (Street address) l (City. State. Zio code) Pursuant to the Atomic Energy Act of 1954, as amended; the Energy Reorganization Act of 1974, as amended; and Public Law 93-438, and subject to the conditions and limitations incorporated herein, the Nuclear Regulatory Commission hereby licenses you to direct the [ licensed] (( fuel handling)) activities of [ licensed] operators at, and to manipulate [all] (( fuel handling)) controls of the (Name of facility. facility license number). Your License No, is SOP-(number). Your Docket No. is 55-(number). The effective date is (date). Unless sooner terminated or renewed, this license shall expire six years from the effective date. This license is subject to the provisions of Title 10, Section 55.53, of the Code of Federal Regulations, with the same force and effect as if fully set forth herein. While performing licensed duties, you shall observe the operating procedures and other conditions specified in the facility license authorizing operation of the facility. You shall also comply with the following condition (s): You shall wear corrective lenses while performing the activities for which you are licensed. The issuance of this license is based upon examination of your qualifications, including the representations and information contained in your application for this license. A copy of this license has been made available to the facility licensee. For the Nuclear Regulatory Commission, (Name and title of licensina official) Docket No. 55-{ number) cc: { Facility reoresentative who sianed the aoolicant's NRC Form 398) () include only for unrestricted senior operators. (()) Include only for senior operators limited to fuel handling. NUREG 1021, Revision 8 16 of 22

ES-501 Sample Proposed Denial Letter Attachment 4' ( NRC Letterhead IDAtel (Apolicant's name) - (Street address) m (Citv. State. Zio code)

Dear (Name):

This is to inform you that your grade on the (written examination. operatino test. or both) taken on (date(s)), in connection with your application for a (reactor operator. senior reactor ooerator) license for the (facility name). Indicates that you did not pass that (examination, test. or both). As a result, it is proposed that your application be denied. Enclosed is a copy of the Iwritten examination. ooeratina test. or both) results indicating those areas in which you exhibited

       -- deficiencies. (A coov of the master answer key is also orovided.)

If you accept the proposed denial and decline to request either an informal NRC staff review or a hearing within 20 days, as discussed below, this proposed denial will become a final denial. You may then reapply for a license in accordance with Titie 10, Section 55.35, of the Code of Federal Regulations (10 CFR 55.35), subject to the following conditions:

a. Because you passed {g written examination. an ooeratina test) on (date), you may request a waiver of that portion.

(' - b. Because you did not pass the (written examination. ooeratina test) administered to you on (date), you will be required to retake that portion.

c. You may reapply for a license two months from the date of this letter.

a. Because this is your (second. subseouent) examination failure, you will be required to retake both the written examination and the operating test.

b. You may reapply for a license 16l4}4 months from the date of this letter.

a. Because you did not pass either the written examination or the operating test administered to you on (date(s)). you will be required to retake both the written examination and the operating test.

b. - You may reapply for a license (2. 6. 24) months from the date of this letter.

If you do not accept the proposed denial, you may, within 20 days of the date of thi.s letter, take either of the following actions: You may request an informal NRC staff review of the grading of your examination. Your written request must be sent to the Director, Division of Inspection Program Management, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. Your request must identify the portions of your examination that you believe were graded incorrectly or too severely. In addition, you ; must provide the basis, including supporting documentation (such as procedurcs, 17 of 22 NUREG-1021, Revision 8 L

[ES-501 2 Attachment 4 instructions, computer printouts, and chart traces), in as much detail as possible, to support your contention that certain of your responses were graded incorrectly or too severely. The NRC will review your contentions, reconsider your grading, and inform you of the results it the proposed denial is sustained, you will have the opportunity to request a hearing pursuant to 10 CFR 2.103(b)(2) at that time. You may request a hearing pursuant to 10 CFR 2.103(b)('2). Submit your request, )n writing, to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Associate General Counsel for Hearings, Enforcement, and Administration, Office of the General Counsel, at the same address. Pursuant to 10 CFR 55.35, you may not reapply for a license until your license has been finally denied. Failure on your part to exorcise either of the above options within 20 days constitutes a waiver of your opportunity for informal review and your right to demand a hearing. For the purpose of reapplication under 10 CFR 55.35, such waiver renders this letter a notice of final denial of your application, effective as of the date of this letter. If you have any questions, please contact (name) at (telephone number). Sincerely, (Name and title of licensina officiall Dockat No. 55-(number)

Enclosures:

As stated cc: (Facility representative who sig.ned the aoolicant's NRC Form 398) CERTIFIED MAIL, RETURN RECEIPT REQUESTED Use for initial RO or SRO license app!icants who passed either the written examination or the operating test but failed the other. l l l Use for second and subsequent retake applicants. Use for applicants who failed both the written and the operating test. l NUREG-1021, Revision 8 18 of 22

Ci ES-501 Sample Notification Letter Attachment 5

\}

NRC Letterhead (Date) (Apolicant's name) ! ) IStreet addraggl, (City. State. Zio code)

Dear (Namd:

This letter is to inform you that you passed the site-specific written examination and operating test administered to you duririg the week of (date) in connection with your application for a (reactor coerator. seniqtreactor operator) license for the {fgeility name). Copies of your operating test and your written examination answer sheets are enclosed. However, as explained in NUREG-1021," Operator Licensing Examination Standards for Power Reactors," Revision 8, Section ES-501, paragraph D.3.c, we will not be issuing your license hatjlypur_emolover certifies in writin2.that you have acauired all of the trainina and experience igr which you wera orevioujiv aranted a waiver.1 IIbecause any written exarnination with a paggina arade of 81 percent or below is normally hold for review until those aoolicants wpp i fa.iigd the examination have had an opp _ortunity to aoneal their license denials, ) 1 af_tgnqqqlvino ople,ntial chanaes from any acoeal. the NRC will iggug your license if your final arada_Lemains ap_Qve 80 percent. _S,hould chances result in_your fina.lgr,ade beina below 80 1 percent. the NPQ_wlil_ pend you a pr,gposed dental letter. which will outline vour re,sponse gg.tions.11 g, if you have any questions, please contact (name at {telechone number). Sincerely, (Name and title of licensina official) Docket No. 55-(number)

Enclosures:

As stated cc: LFacility representative who sicned thp_a_pglic. ant's NRC Fqrm 398J [] Use only for applicants who need to complete training or experience prior to licensing. (( )) Use only for applicants whose final licensing action is pending the resolution of wntten examination appeals. O

 \     l
  \/                                                     19 of 22                   NUREG-1021, Revision 8

ES-501 Post-Examination Check Sheet Form ES-501-1 i I l I l Task Description Date { Complete

1. Facility written exam comments or graded exams received and i verified complete {

l l

2. Facility written exam ccmments reviewed and incorporated and NRC grading completed,if necessary

3. Operating tests graded by NRC examiners

! 4. NRC Chief examiner review of written exam and operating test grading completed j

5. Responsible supervisor review completed

6. Management (licensing official) review completed

7. License and denialletters mailed

8. Facility notified of results

9. Examination report issued (refer to NRC MC 0610) l

10. Reference material returned after final resolution of any j appeals '

l I ( i 1 NUREG-1021, Revision 8 20 of 22 9! i

l I l O ES-501 Power Plant Examination Results Summary Form ES-501-2 N PRIVACY' ACT INFORMATION - FOR OFFICIAL USE ONLY POWER PLANT EXAMINATION RESULTS

SUMMARY

f Facility: Circle Plant Status: Hot / Cold Written Examination Date: Operating Test Date(s): NRC Examiners: OVERALL RESULTS Applicants: Total # # Passed % Passed # Failed % Failed l RO SRO l INDIVIDUAL RESULTS Name Docket # Type Written Operating Test (2) l} m 55-f ) (1) Grade f A B C

 's                                                                            __

I M-- NOTES: (1) 1 =RO; 2wSRO-1; 3=SRO-U; 4=RO-Retaka; S=sRO-l Retake; 6=sRO-U-Petake; 7=sRO-Fuel (2) P= Passed: F= Failed; W= Waived e PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY T 21 of 22 NUREG 1021, Revision 8

1 1 ES-501 2 Form ES-501-2 { l PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY l l POWER PLANT EXAMINATION RESULTS

SUMMARY

(CONTINUATION SHEET) _ Facility: l Examination Date-(s): i INDIVIDUAL RESULTS

  . . - -                                                                                                       I Name                        Docket #        Type        Written              Operating Test (2) 55-(       )     (1)         Grade                                        )

A B C !

O i

NOTES: (1) 1=RO: 2=SRO-l; 3=SRO-U; 4 RO-Retake; 5=SRO-i-Retane: 6=SRO-U Rotake; 7=SRO-Fuel (2) P= Pawed; F= Failed; W=Waivej PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY NUREG-1021, Revision 8 22 of 22

r O V ES-502 PROCESSING REQUESTS FOR ADMINISTRATIVE REVIEWS AND HEARINGS ! - AFTER INITIAL LICENSE DENIAL A. PURPOSE This standard describes the options and associated responsibilities regarding administrative reviews and hearings related to license application denials and license denials resulting from examination failures. This standard also addresses license reapplications after a denial becomes final. B. BACKGROUND Those operator license applicants who are denied the opportunity to take an NRC licensing - examination because they do not meet the eligibility requirements for a license pursuant to Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55) and those applicants who are denied a license because they failed a written examination or operating test administered pursuant to 10 CFR Part 55 are notified of their denials in writing. The proposed denial letters

      ' describe the nature of the deficiencies noted and inform the applicants of their available response options. Applicants may reapply pursuant to the provisions of 10 CFR 55.35.

However, the NRC will not accept a reapplication as iong as a request is pending for either an administrative NRC review or a hearing. C. RESPONSIBILITIES

1. Acolicant

a. An applicant who does not appear to meet the experience and training requirements for a license may be asked to provide additional information to the NRC regional office in accordance with ES-202. If the application is still deniod after providing the additional information requested by the NRC, the applicant may exercise one of the following options within 20 days after the date on the proposed denial lotter from the regional office:

(1)_ Do nothing. The proposed denial letter then becomes the final denial. The applicant may reapply after obtaining the requisite training or experience. (2) Request reconsideration of the application denial. Such requests must be submitted to the Operator Licensing Program, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. The applicant's submittal must clearly state the basis for the request. (3) . Request a hearing pursuant to 10 CFR 2.103(b)(2). Such requests must be submitted to the Secretary of the Commission, U.S. Nuclear > Regulatory Commission, Washington, D.C. 20555, with a copy to the p m

Associate General Counsel for Hearings, Enforcement, and Administration, Office of the General Counsel, at the same address.

b. If an applicant fails the operator licensing written examination or operating test (or both) and receives a proposed license denial letter issued by an NRC regional office in accordance with ES-501, the applicant has 20 days from the date on the letter to exercise one of the following three options:

(1) Do nothing. The proposed denial letter then becomes the final denial. The applicant may reapply, pursuant to 10 CFR 55.35, two months after the date on the denialletter. (2) Request that the NRC administratively regrade the written examination, the operating test, or both, in light of new information to be provided by the applicant. Such requests must be submitted to the Operator Licensing Program, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, if the applicant submits such a request, the NRC will not consider a reapplication pursuant to 10 CFR 55.35 until a denial is final. The applicant's request for administrative review must identify the item (s) for which additional review is requested and must include documentation supporting the item (s) in contention. The applicant is responsible for ensuring thtt the request and the supporting documentation is sent to the NRR operator licensing program office within 20 days after the date on the proposed denial letter, j

If the NRC administratively reviews a failure and determines that the l applicant did not provide sufficient basis to justify passing grades on all j sections of the licensing examination, a letter will be issued to the i applicant sustaining the proposed denial. The applicant may then request a hearing pursuant to 10 CFR 2.103(b)(2). A request for a hearing after an administrative review must be submitted by the applicant within 20 days after the date on the letter from the NRR operator licensing program office sustaining the proposed denial. The hearing request must be submitted in accordance with Section C.1.b(3). I If the applicant does not request a hearing when the proposed denialis sustained by the NRR operator licensing program office, then the l proposec denial becomes the final denial. The applicant may then { reapply for a license, pursuant to 10 CFR 55.35, two months after the date of the sustained denialletter. (3) Request a hearing as provided by 10 CFR 2.103(b)(2). The hearing request must be submitted to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Associate General Counsel for Hearings, Enforcement, and NUREG-1021, Revision 8 2 of 4 1

p l l l l ES-502 1 [Q Administration, Office of the General Counsel, at the same address if the applicant requests a hearing, the NRC will not consider a reapplication pursuant to 10 CFR 55.35 until the denial is final.

2. Facility Licensee The facility licensee is expected to provide reference materials and technical support as necessary for the NRC to evaluate and resolve any concerns raised by a license applicent who has requested the NRC to reconsider a proposed denial of an application -

or a license.

3. NRC i

a. The NRC will conduct administrative reviews of Part 55 license application denials based on eligibility as described in Section D.1 below.

b. The NRC will conduct administrative reviews of Part 55 license denials based on examination failures as described in Section D.2 below.

c. The NRC will conduct Part 55 operator licensing hearings in accordance with Subpart L," Administrative Hearing Procedures for Adjudications in Materials and Operator Licensing Proceedings," of 10 CFR Part 2," Rules of Practice for Domestic Licensing Proceedings and Issuance of Orders."

D. ADMINISTRATIVE REVIEW PROCEDURES

1. Apolication Denial !

If an applicant requests an administrative review in accordance with Section C.1.a, the NRR operator licensing program office will generally complete its review of the applicant's eligibility within 60 days of receiving the request. When the review is completed, the applicant will be notified in writing if he or she will be allowed to take the license examination. If the review results in the original denial being sustained, the applicant may request a hearing pursuant to 10 CFR 2.103(b)(2).

2. Examination Results if an applicant requests an administrative review in accordance with Section C.1.b, the review will generally be completed, as follows, within 60 days after the NRR operator licensing program office receives the request,

a. The NRR operator licensing program office will determine v.t. ether to: (1) review the appeal internally; (2) have the regional office review the appeal; or (3) convene a three-person board to review the applicant's documented contentions.

l The appeal board normally will be composed of a branch and two examiners or subject matter experts; it may include a representative from the affected region but no one who was involved with the applicant's licensing examination. I 3 of 4 NUREG 1021, Revision 8

ES-502 For written examinations, the review shall focus only on those questions that are being contested. The review shall evaluate the original grading of the applicant's examination, the reference material supplied by the facility licensee, and the contentions and supporting documentation provided by the applicant. For operating tests, the review shall evaluate the examiner's comments, the examination report, the test that was administered, and the contentions and supporting documentation provided for review by the applicant or facility licensee (e.g., plant system descriptions, operating procedures, logs, chart recorder traces, and process computer printouts).

b. Based on the findings and recommendations from the review, a decision will be made whether to sustain or overturn the applicant's license examination failure.

The NRR operator licensing program office will notify the applicant in writing of the results of the review,

c. When the NRR operator licensing program office has concurred in the results of the review, the NRC regional office will: (1) issue a license if the proposed denial was overturned; (2) review the examination results of the other applicants to determine if any of the licensing decisions are affected; and (3) update the master examination file to reflect any test item deletions or answer key changes.

O NUREG-1021, Revision 8 4 of 4

1

                                                      - ES-601 CONDUCTING NRC REQUALIFICATION EXAMINATIONS A.         PURPOSE Title 10, Section 55.59(a), of the Code of Federal Regulations requires licensed operators and senior operators to complete a requalification program developed by the facility licensee and to pass a comprehensive requalification written examination and an annual operating test. In lieu_

l of accepting the facility licensee's certification that the operator has passed the required examinations and tests administered within the facility licensee's Commi.ssion-approved

    . program, the NRC may administer a comprehensive requalification written examination and an            ,

i annual operating test. l This standard provides generalguidance and requirements for conducting NRC requalification examinations. In addition this standard provides guidance and procedures for evaluating the facility licensee's licensed operator requalification training program to ensure it is effectively . , maintaining the competency of the licensed operators. Specific guidance and requirements for I conducting the comprehensive requalification written examinations and the annual operating i tests (including both the plant walk-through and dynamic simulator sections) are provided in 1' ES-602 through ES-604. These standards are not a substitute for the operator licensing regulations and are subject to revision and other changes to the internal operator licensing program policy. B. BACKGROUND Section 306 of the Nuclear Waste Policy Act of 1982 (NWPA) authorized and directed the NRC to promulgate regulations, or other appropriate guidance, for training and qualifying nuclear power plant operators. Those regulations were to include requirements governing the administration of requalification examinations and operating tests at nuclear power plant simulators. The NRC's requalification evaluation program consists primarily of periodic, on-site q requalification inspections, supplemented with NRC examinations at facilities where the NRC believes that ineffective training is causing operators to commit errors. The NRC's Office of the General Counsel has concluded that this program satisfies the statutory requirements in Section 306 of the NWPA. The oversight program will require the NRC to actively oversee the facility licensees' requalification training programs, and the Commission's regulations will continue to contain legally binding requirements that apply to the conduct of operator requalification examinations by facility licensees. When determining the scope of the requalification intoection and examination activities at a

    ' facility, regional managers will consider overall facility performance (e.g., systematic assessment of licensee perfom1ance (SALP) ratings), the results of the NRC's inspection programs (e.g., requalification, emergency operating procedure, and resident), the results of routine initial and requalification examinations, and other facwrs. Generally, only the inspection requirements of Inspection Procedure (IP) 71001, " Licensed Operator Requalification Program Evaluation," will need to be met; however, augmented activities can be initiated in accordance with program office guidance when necessary to ensure safe plant operation. Those activities could include a training program inspection in accordance with IP 41500, " Training and O                                                       1 of 36                 NUREG-1021, Revision 8
                                                                                                           )

l 1

ES-601 Qualification Effectiveness," operational evaluations of on-shift crews, or NRC examinations conducted in accordance with this series of Examination Standards. The NRC will conduct requalification examinations only when it has lost confidence in the facility licensee's ability to conduct examinations, or when the staff believes that the inspection process will not provide the needed insight. Regional management should consider conducting requalification examinations or operational evaluations when any of the following conditions exist: requalification inspection results indicate an ineffective operator requalification program operator errors are a major contributor to operational problems a SALP Category 3 rating in plant operations is attributed to operator performance allegations have been raised regarding significant training program deficiencies The decision to conduct NRC examinations should be implemented through the normal resource planning system, such as the plant performance review (PPR) and master inspection planning (MIP) processes, since an inspection activity will be replaced with examinations that are more resource-intensive. Using the existing inspection planning process will ensure that the regional offico and the Office of Nuclear Reactor Regulation (NRR) consider the need to conduct examinations, as well as the alternative, expanded inspection tools, when allocating the required resources. Operational evaluations should be considered as a reactive effort based on immediate safety concerns. C. SCOPE The NRC-conducted requalification examinations will measure the effectiveness of a requalification program by evaluating the ability of the facility licensee to adequately prepare written examination questions, job performance measures (JPMs), and simulator scenarios, as well as its ability to properly evaluate its operators' performance. The examination procedures are based on a systems approach to training (SAT) program, as defined in 10 CFR 55.4. To the extent possible, these procedures rely on existing requalification program standards for developing and implementing the NRC examination. The SAT approach allows the NRC to conduct requalification examinations that are fundamentally consistent with existing facility-developed programs. As such, this approach reduces the impact on the facilities and improves the reliability of the NRC assessment of requalification training programs. The NRC-conducted requalification examination will normally be composed of three parts, including a two-section, open-reference written examination, a walk-through evaluation, and a dynamic simulator evaluation. The three examination parts are Nrther described in ES-602, 603, and 604, respectively. If the facility licensee's requalification program uses an examination structure or methodology different from that described herein, tha regional office should consult with the NRR operator licensing program office to determine tho appropriate examination procedure. NUREG-1021, Revision 8 2 of 36

ES-601 5 To the extent practical, the examination will be based on the facility licensee's requalification program and leaming objectives. The facility licensee is expected to use the plant-specific job and task analyses (JTAs) as the basis for developing the examination materials and ' substantiating the importance rating factors for each task. The facility licensee may also use NUREG-1122 or -1123, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors (or Boiling Water Reactors]," for additional guidance in identifying , job-specific importance rating factors. The use of a JTA will result in more technically sound I and operationally oriented examinations. ! An examination team, composed of NRC examiners and facility representatives, will develop, review, and conduct each requalification examination. Parallel evaluation of operator performance by NRC examiners and facility evaluators will enhance the ability of the NRC to : assess both individual and program performance. The administrative guidelines and procedures for conducting an NRC requalification examination are outlined in Attachment 1, " Examination Timetable." D. EXAMINATION PREPARATIONS

1. Communication

a. When the NRC determines that it is necessary to conduct requalification p examinations, the regional office will notify the facility licensee to be evaluated at

( least 90 but preferably 120 days before the examination start date, using the corporate notification letter shown in Attachment 2. If possible, the NRC will schedule the site visits to coincide with the requalification training cycle of the facility. Depending on the number of operators and crews at the facility, it may be necessary to conduct the examinations over a period of two or more weeks to attain the required sample size. The requalification training cycle, referenced here and throughout NUREG-1021, is that continuous period of time (not to exceed 24 months) within which the facility licensee conducts its operator requalification training program. ; l if the purpose of the examination is to retest operators who previously failed an ) NRC-conducted requalification examination, the regional office should modify the l corporate notification letter, as appropriate.

b. The facility licensee is expected to respond to the corporate notification letter at !

least 60 days before the evaluation by submitting the materials and information requested in the letter. The facility licensee may request that the NRC chief examiner or another NRC representative meet with the operators to be examined and with appropriate ! facility licensee managers. Such a meeting should be scheduled during the examination preparation week as discussed in Section D.S. 3 of 36 NUREG-1021, Revision 8

1 ES-601

c. At least 30 days in advance of the examination, the NRC will confirm with the facility licensee which operators have been selected to participate in the evaluation.

2. Selection of Operators

a. The NRC expects facility licensees to train and examine their operators in the same crew configurations with which they normally operate the plant. Generally, the NRC expects the crew to include no more than five operators, but it will consider larger crews on a case-by-case basis.

At times, to ensure an adequate sample size for the examination, the examination team may configure crews that do not routinely perform shift duties together. Mixed crews of shift and non-shift operators should not be configured unless the facility licensee routinely evaluates mixed crews in its requalification training program, or the facility licensee's normal crew size is so large that separating a normal crew is required for examination purposes.

b. All crew members for requalification dynamic simulator examinations must be currently licensed on the facility and up-to-date in the facility licensee's requalification program.

c. The selections will be made to minimize perturbation on the facility licensee's schedules and plant operations. Operating crew (s) in training will be given first priority during the examination week (s). If the facility's prograrn is being reevaluated after an unsatisfactory evaluation, the selection process should favor operators who either failed their previous NRC-conducted examination or were not previously examined.

d. During retake examinations, operators who have passed an NRC requalification examination may be included in the dynamic simulator crew evaluation. These operators will not be required to take the written or walk-through portions of that examination. The operators' performance on the simulator examination will be evaluated in accordance with the guidance of ES-604.

e. A shift technical advisor (STA) may be added to the crew if the facility normally uses an STA during requalification training. The NRC expects the STA's duties and responsibilities to be the sama as those assigned during requalification training and plant operations,

f. The NRC will review the list of crews and operators submitted by the facility licensee, and recommend any necessary changes.

3. Reference Material

a. The facility licensee is expected to supply the reference materials requested in the corporate notification letter (see Enclosure 1 to Attachment 2). The NRC will NUREG-1021, Revision 8 4 of 36 I

l 1

1 [3 ES-601 lV evaluate the facility's reference materials for adequacy in advance of the scheduled preparation week, using the " Evaluation Checklist for Facility Reference Material," Form ES-601-2. I

b. The NRC reserves the right to prepare the requalification examinations using the l NRC's examination question banks and facility background reference materials if the facility licensee's test items are inadequate for examination preparation. If the NRC prepares the examination, the NRC may require additional reference material comparable to that listed in ES-201, Attachment 2, " Reference Material Guidelines for Initial Operator Licensing Examinations."

c. The facility is expected to provide a sample plan that meets the guidelines of Attachment 3, " Examination Sample Plan," for the NRC's use in developing the examination.

4. Examination Team Selection l

a. The NRC will contribute no fewer than two examiners to the examination team.

The regional office should consider assigning additional examiners if the ) operating crews for the dynamic simulator examinations contain five or more operators. To promote consistency in requalification program administration, regional office management should try to assign an examiner who participated in a prior requalification inspection or examination at the facility to be part of the NRC examination team. In most cases, it is expected that the NRR operator licensing program office will send a representative to observe the examination process or an examiner to participate as an additional member of the examination team. The program office will work with the responsible regional supervisor to . Ike the necessary arrangements.

b. The facility licensee is expected to provide an employee to work with the NRC as part of the requalification examination team. The employee must be drawn from the operations staff, and must be an active senior reactor operator (SRO) as defined in 10 CFR 55.53(e) or (f). The NRC encourages the facility licensee to designate another employee from the training staff to be a member of the examination team. This employee should also be a licensed SRO, but may be a certified instructor. If the facility licensee desires, and the chief examiner agrees, additional employees from the operations or training staffs with qualifications comparable to the other facility examination team members may be included on the examination team.

The function of these examination team members is to provide facility-specific technical assistance to the NRC in developing and reviewing the written exami-nation items, plant walk-through topics, and dynamic simulator scenarios. If necessary, the facility representatives may participate as facility evaluators in conducting the operating test or written examination. However, the facility 5 of 36 NUREG-1021, Revision 8

i ES-601 representatives should only be used as evaluators if they routinely perform that function during the administration of the facility licensee's requalification program.

5. Examination Develooment The facility licensee may develop proposed written examinations and operating tests and forward them to the NRC as part of its reference material submittal (see Attachment 2). In accordance with 10 CFR 55.59(a)(2)(ii), the facility licensee must ensure that the operating tests require the operators to demonstrate an understanding of and the ability to perform the actions necesst.ry to accomplish a comprehensive sample of the items specified in 10 CFR 55.45(a)(2) through (13), inclusive, to the extent applicable to the facility. !

Approximately two weeks before the scheduled examinations, the NRC examiners will visit the facility to make final preparations for the examination. The written, walk-through, and dynamic simulator examinations will be developed in accordance with ES-602, ES-603, and ES-604, respectively. The examination should distinguish between reactor operator (RO) and SRO knowledge and abilities to the extent that the facility training materials allow the examiners to make these distinctions. The NRC examiners will rely upon the facility licensee's examination team members for site-specific technical assistance in developing, reviewing, and validating the written examination static scenarios and items, plant walk-through topics (JPMs), and dynamic simulator , scenarios. The chief examiner and the responsible regional supervisor will determine the required ! i length of time on site and the required number of examiners. This determination will be based on the experience of the examiners, the quality of the facility licensee's testing material, and the level of effort required to develop new test items. If requested by the facility licensee, the chief examiner will brief the operators and j managers about the requalification examination process. The examiner will use this ; time to explain the examination and grading processes and to respond to any questions that the operators may ask about the NRC's examination procedures. If the schedule i does not allow them to meet during the preparation week, they may meet at any l mutually agreeable time.

6. Examination Security To ensure examination security, each facility representative who acquires knowledge of the content of the NRC requalif; cation examination before it is administered will be subject to the security restrictions described below from the time he or she first acquires the specific knowledge until the examination exit meeting.

To the maximum extent possible, only the examination team members and a simulator operator should be given specific knowledge about the content of the examination. The facility evaluators should be given the package of simulator scenarios and JPMs the week before the examination to allow them to prepare for their evaluation, including l NUREG-1021, Revision 8 6 of 36 u

ES-601 coordinating the use of the simulator to pedorm JPMs and scenarios. If the facility ) licensee submits a proposed examination, those who participate in developing the examination become subject to the security restrictions when their involvement begins. Also, if facility representatives other than the examination team members are used to time validate the written examination, they too become subject to the security restrictions as soon as they are exposed to the examination questions. Facility representatives who acquire specific knowledge of the NRC examinations will i sign Form ES-601-1," Examination Security Agreement," or a reasonable facsimile before their examination involvement begins and again after the examination process is complete (i.e., following the exit meeting). E. PROGRAM AND OPERATOR EVALUATION PROCEDURES

1. Examination Administration

a. For each selected operator, conduct a requalification examination using ES-602, ES-603, and ES-604 for the written, walk-through, and simulator portions of the requalification examination, respectively. Document operator performance on Form ES-601-5.

b. The number of persons present during an operating test should be limited to ,

p ensure the integrity of the test and to minimize distractions to the operators. (' Under no circumstances will another operator be allowed to witness an operating test. Operating tests are not to be used by the facility licensee as training vehicles for future requalification examinations. Other examiners may observe an operating test as part of their training or to audit the performance of the examiner administering the operating test. The chief examiner may permit others (such as resident inspectors, regional personnel, researchers, or NRC supervisors) to observe an operating test if the applicant does not object to the observers' presence. Deviations from this policy must be approved, in advance, by the NRR operator Ibensing program office. Other non-NRC personnel (e.g., representatives from the Institute of Nuclear Power Operations (INPO) or the Nuclear Energy Institute (NEI)) may observe the operating tests with prior approval from the program office. The chief examiner will control the observers' activities in accordance with guidance provided by the program office.

2. Examination Gradina

a. The facility licensee is expected to grade the written examinations and operating tests in parallel with the NRC examiners.

b. The facility evaluators are expected to provide preliminary pass / fail results for the simulator and walk-through portions of the examination by the end of each day, 7 of 36 NUREG-1021, Revision 8

ES-601 and the final results before the exit briefing or at the end of each examination , week for multi week examinations. l l

c. The NRC will notify the facility licensee immediately if any operator's I I

performance on the examination is sufficiently poor to require immediate removal from licensed activity. The NRC will also notify the facility licensee of the results of the examination in accordance with ES-605.

d. The facility licensee will provide the NRC with the final results of the written examinations and an overall summary of the examination results within two weeks after the exit meeting.

3. Evaluation of Reaualification Procrams A requalification program evaluation requires a minimum sample size of 12 operators.

The sample size is determined by counting the number of operators taking the dynamic simulator examination. This total includes those operators who only participate in the simulator examination for the purpose of meeting crew composition requirements, but excludes those operators who are being reexamined after failing a previous NRC-conducted examination. In instances where fewer than one-half of the operators taking the dynamic simulator examination complete the entire examination, the regional supervisor will determine whether a valid program evaluation can be made. In these instances, the regional supervisor will contact the NRR operator licensing program office.

a. A satisfactory requalification program meets each of the following criteria:

(1) At least 75 percent of the operators must pass all portions of the examination in which they participate. The pass rate is determined by dividing the number of operators that pass all portions of the examination in which they participate by the total number of operators in the sample. In the event of a crew failure, only those operators who receive an unsatisfactory evaluation in the individual follow-up evaluation will be counted when calculating the operator pass rate. When calculating the pass rates, fractions should be rounded up to the next highest number. For example, if 15 operators are evaluated,75 percent passing would be 11.25; thus,11 of 15 passing would not meet the 75 percent requirement, but 12 would.

             '(2)     At least two-thirds (66 percent) of the crewa must pass the simulator examination.

For requalification examinations with more than three crews participating, three-of-four, or fcur-of-five crews must pass to satisfy this requirement. NUREG-1021, Revision 8 8 of 36

ES-601

b. - The following areas will be considered in the overall program evaluation and may be used to identify facility weaknesses that will be documented in the .

examination report: i (1) The facility evaluators do not concur with the NRC examiners on all unsatisfactory crew evaluations. (2) More than one facility evaluator is determined to be unsatisfactory. Section D of Appendix C provides guidance that examiners should use to assess evaluator competence. (3) The facility licensee failed to train and evaluate an operator in all positions permitted by the individual's license. (For instance, the facility is required to train and evaluate an SRO in the RO position as well as in directing operators.) An SRO will not be required to perform RO activities during the simulator portion of the operating test; however, his or her performance will be evaluated if the facility normally places the SRO in a shift RO position during the simulator examination. Otherwise, RO skills will be evaluated during the performance of JPMs. (4) - The facility licensee has insufficient administrative controls to preclude an RO or SRO with an inactive license from performing licensed duties. O Operators must meet the requirements of 10 CFR 55.53 to restore an inactive license to active status. (5) The facility licensee has insufficient quality control of its examination bank. The NRC will evaluate the facility's performance in this area if post-examination changes to facility-developed test items result in significant modifications or deletions of more than 10 percent of the questions on the written examination. (6) The number of test items duplicated from any past examination or combination of examinations administered during the current requalification training cycle (as described in 10 CFR 55.59(a)(1)), or the number of operating test items repeated on successive days of an examination period, is such that the discrimination validity and integrity of the examination could be affected. When test items are repeated, they should be selected in a distributed manner and approximately equal!y over all previous examinations to reduce predictability (if a large number of items were taken from the most recent examination). (7) The facility licensee's failure decisions are not as conservative as the NRC's. To ensure that the rationale for the evaluation is fully understood, the NRC will review with the facility managers any case where the facility licensee passes an operator failed by the NRC. In addition, the NRC will assess whether the facility licensee's evaluations are conducted in 9 of 36 NUREG-1021, Revision 8

ES-601 accordance with documented facility guidance and whether facility managers periodically assess their evaluation process. The NRC also expects the facility program to explicitly link an operator's examination failure with unsafe performance in this way, all facility ; failuras and NRC failures will agree, in certain instances, the facility licensee's prograra may have operator performsnce standards that are not explicitly linked to unsafe performance, and thus do not meet the threshold stated in these standards for the operator to fait the examination, in such instances, the facility licensee is expected to differentiate failures !n which the operator performed at an unsafe level from those in which the operator failed for reasons other than safety (i.e., not meeting higher facility-established performance standards). In these instances, operators identified as failing for safety reasons would a!so be considered NRC failures.

4. Evaluation of Operator Performance To pass the NRC-conducted requalification examination, the operator must pass a written examination and an operating test consisting of a walk-through examination and a dynamic simulator examination. These examinations are developed and administered in accordance with ES-602, ES-603, and ES-604, respectively, unless the NRR operator licensing program office authorizes the regional office to use the facility licensee's attemative examination methodology. To pass the operating test, the operator must also be a member of a crew that passes the dynamic simulator examination.

F. UNSATISFACTORY OPERATOR OR PROGRAM EVALUATION I

1. Actions Followina an Unsatisfactory Ooerator Evaluation i

in all cases, a facility licensee's administrative procedures should ensure that an j operator who fails a requalification examination is removed from licensed duties, given j remedial training, and reexamined before being allowed to return to licensed duties. l This also applies to an SRO who performs only RO-level duties at the facility when the failure is caused solely by activities involving SRO responsibility. ES-605 contains the ; procedure for notifying the operator about his or her performance on the requalification examination, as well as guidance about the actions to be perforrned for an operator to return to licensed duty. l The regulation (10 CFR 55.57(b)(2)(iv)) tMt required an operator to pass an NRC- ! administered requalification examination as a prerequisite for license renewal has been deleted. Nonetheless, it would be inappropriate to renew the license of any operator i who failed to pass any NRC-conducted requalification examination, without some level l of NRC involvement in the retesting process. The amount of NRC involvement may l include conducting the retest in accordance wi'le the appropriate Examination l Standard (s); inspecting the facility licensee in accordance with IP 71001, " Licensed Operator Requalification Program Evaluation," as it retests the operator; or reviewing NUREG-1021, Revision 8 10 of 36 Oli '

i O O the reexamination prepared by the facility licensee. The regional office, in consultation ES-601

{' with the NRR operator licensing program office, will determine the appropriate level of involvement on a case-by-case basis depending on the quality of the facility licensee's program, As long as the operator submits a timely renewal application, the term of the license will continue until the renewal requirerrents are satisfied or the operator fails three NRC-conducted examinations as discussed in ES-605. If an operator who failed a requalification examination is not prepared for a , reexamination after six months of remedial training, the regional office will request the l i following information from the facility licensee: confirmation that the facility licensee still has a need for the individual's license the expected completion date of the operator's remedial training and when the facility licensee will be ready to administer its retake examination assurance that the operator will not be retumed to licensed duties until he or she successfully retakes the examination (or portion thereof) administered by the facility licensee with a satisfactory requalification program or in accordance with the provisions of the confirmatory action letter (CAL) if the facility licensee has an unsatisfactory program and the NRC has not determined it to be " provisionally satisfactory."

     - The NRC will inform the facility licensee that a comprehensive requalification examination may         '

be necessary if the operator is not ready to take a retest within one year after failing the examination. I

2. Actions Followina an Unsatisfactory Heaualification Proaram Evaluation The following actions will be taken for al! requalification programs evaluated as unsatisfactory:

a. The facility licensee is expected to identify program deficiencies and corrective actions to improve operator performance. The NRC will use a CAL to establish a formal dialogue and to document the facility licensee's corrective action commitments.'

An operator who fails the requalification examir.ation, as determined by the NRC, wi!! be subject to an NRC-administered reexamination before resuming licensed duties. An operator whose performance does not meet facility standards, as determined by the facility licensee, is expected to be remediated and reevaluated by the facility licensee in accordance with the provisions of the facility licensee's requalification program. The NRC will review and/or monitor the reexamination to ensure the adequacy of the facility licensee's requalification program. p i k 11 of 36 NUREG-1021, Revision 8

I b 1 ES-601

b. The NRC will schedule a meeting with senior facility managers to review the examination results, the identified deficiencies and their root causes, the proposed corrective actions and the schedule for their implementation and the need for follow up inspections and examinations. (Refer to Section F.3 for additional guidance for conducting augmented inspections.)

The Regional Administrator will evaluate the examination and inspection resuits and make a decision regarding the continued operation of the facility and possible enforcement action against the f acility licensee. At a minimum, the Regiona! Administrator should consider the following factors when making this determination: , the results and corrective actions from previous program evaluations the significance of generic performance deficiencies identified during the program evaluation I recent SALP ratjngs 1 recent facility events that relate to licensed operator performance I recommendations by the NRC staff (including the resuits of any operational evaluations and inspections) ,

c. If the unsatisfactory program evaluation is caused by ocerator performance deficiencies, an operational evaluation is required. The operational evaluation is intended to help the Regional Administrator determine whether the facility's remaining operating crews are suitably qualified to continue to operate the -

facility. In this case, the facility licensee identifies the individual operators and shift crews it proposes to use to continue plant operations. The regional ofiice may choose not to evaluate those operators who passed their most recent NRC-conducted initial or requalification examination within the past 12 months. 1 However, the regional office will evaluate all other operators in those areas noted as operational deficiencies during the requalification examination, regardless of whether they have already passed or not yet taken the facility-administered requalification examination. The regional office will conduct the operational evaluations in accordance with the guidance in ES-603 and ES-604, as j applicable. ) If the facility licensee proposes to use a shift crew that is significantly different from its normal configuration, even though all of the operators may have recently passed an NRC-conducted examination, the regional of' ice may perform an operational evaluation of this crew. The regional office should schedule the operational evaluatico as soon as i possible after determining that the facility licensee's requalification program is unsatisfactory. The evaluation should not be delayed to accommodate the NUREG 1021, Revision 8 12 of 36

1 Y ES-601 facility's operating schedule, the completion of programmatic corrective actions, or the completion of remedial training for operators who failed the requalification 1' examination. The operational evaluation may identify further program deficiencies that may need to be reflected in the CAL discussed in Section F.2(a) or that may warrant additional inspection by the NRC. Additional operator . weaknesses that require remediation may also be identified. I

d. The NRC will review the corrective actions the facility is to perform, the expected follow-up actions by the NRC, and the schedule for sach.

As part of the follow-up activities, the NRC may conduct additional operational evaluations, requalification retake examinations, and augmented inspections, as necessary. Before these activities, the NRC will verify that the facility licensee has completed the applicable corrective actions, and will obtain a certification of . crew readiness from the facility managers. Regional managers should consider using a new chief examiner and having examiners from other regional offices participate on those operational evaluations and requalification retake examinations that have restart approval implications.

e. The Regional Administrator willincorporate into the decision concerning follow-up activities any extraordinary circumstances surrounding the examination activities that may have a bearing on the validity of the examination results.

s

  ) f. Once the NRC determines that a requalification program is unsatisfactory, the program will remain unsatisfactory until the facility licensee completes all identified corrective actions agreed upon by the NRC for restoring the program to satisfactory status, and the NRC completes all related follow-up activities.

For purposes of allowing facility examiners to perform reexamination functions, however, a facility may attain a status of " provisionally satisfactory" provided that the facility has completed to the NRC's satisfaction all short- and intermediate-term corrective actions agreed on with the NRC. Once the NRC determines that the facility licensee has satisfactorily implemented these corrective actions, the Regional Administrator or designee , will determine whether to permit the facility to reexamine all operators who failed j the NRC-conducted requalification examination for the purpose of returning the j operators to licensed duties. Any operator who fails the NRC-conducted examination still needs to pass a future NRC-administered (i.e., conducted, inspected, or approved, as appropriate) requalification examination to renew the license. Long-term corrective actions are expected to be completed before the NRC's next requalification program evaluation. l To attain a satisfactory rating following an unsatisfactory evaluation, the 1 subsequent requalification program evaluation, with a sample size of at least 12 operators, must satisfy the passing criteria in Section E.3. 13 of 36 NUREG-1021, Revision 8 11

ES-601 The Regional Administrator or designee may specify additional actions, as appropriate. The specific sequence of actions is not critical; however, this sequence of events corresponds to a typical regional response to an unsatisfactory program evaluation. The Regional Administrator or designee should defer determining whether a plant shutdown is required until he or she reviews all factors listed in Section F.2(b) above.

3. Auamented Inspection Guidelines t if it is determined that an augmented requalification program inspection is required, i regional management shall define its scope and depth based upon the nature of the !

deflciencies. The regional office should consider the following activities in addition to those specified in Section F.2.

a. The regional office may conduct augmented inspection coverage of all shifts.

The inspection procedures for shift coverage should be used as appropriate. Inspection activities should devote particular attention to the following areas: operator performance and attitude operator overtime management oversight shift staffing

b. The regional office may develop a long-term training program inspection plan based on Inspection Procedure (IP) 41500, ' Training and Qualification Effectiveness." Such an inspection plan may include the following activities:

ongoing status reviews of requalification training effectiveness, with an emphasis on known program deficiencies and implementation of short-term corrective actions an inspection to determine the root cause(s) for the unsatisfactory requalification program evaluation, and to verify that the facility licensee's proposed corrective action plan should preclude or minimize the probability of recurrence an inspection to evaluate the adequacy of the facility licensee's corrective actions, and to determine the effectiveness of the facility licensee's SAT-based requalification program

c. The regional office may convene an enforcement panel to determine if action is warranted on the basis of the requirements of 10 CFR 50.54, paragraph (i-1).

The potential exists that a requalification program rated unsatisfactory on two successive NRC evaluations does not meet the minimum requirements of 10 CFR 55.59(c) as required by 10 CFR 50.54(i-1). The basis for any proposed enforcement action will be the inadequate corrective action or requalification program element deficiencies (identified by the inspections related to Section , F.3(b)) which led to the successive requalification examination failures. NUREG-1021, Revision 8 14 of 36

r;

        -                                                                                                    ES-601 "Ge        REQUALIFICATION PROGRAM EVALUATION REPORT
          ' After the Regional Administrator or designee approves the requalification examination results, the regional office will prepare a final requalification program evaluation report. A copy of the written examination need only be included in the program evaluation report if the report addresses written examination problems. The regional office will issue the report within 30 days following receipt of the facility licensee's final results or the examination exit meeting, whichever   I is later, and will place a complete copy of the report in the facility's requalrfication file.          l The chief examinor is responsible for completing Forms ES-601-3 and ES-601-4, the

Power Plant Requalification Results Summary (and Continuation) Sheet (s)? The examiner will enter i each operator's scores in the appropriate columns. Under the column titled ? Simulator," the examiner will enter the results of the operator's individual follow-up evaluation. If the operator did not receive an individual follow up evaluation, the examiner will enter a passing score. lf an operator was a member of a crew that failed the dynamic simulator examination, but the operator passed or did not receive an individual follow up evaluation, the examiner will enter a pass in the simulator column for that operator. Crew failures will be summarized in the overall results at the top of Form ES-601-3.

The regional office will send a copy of the summary (and continuation) sheet (s) to the headquarters' operator licensing assistant (OLA). The NRR operator licensing program onice uses the results summary to verify the data in the operator licensing tracking system (OLTS) so j

   ,%       that statistics can be maintained on operator performance. As it contains Privacy Act
 /    \. information, the regional office will not include the results summary in the examination report.

N] l If a small number of operators are given retake examinations, the regional office may issue an addendum to the original requalification evaluation report instead of issuing a new report. If the reexaminations are conducted concurrently with initial examinations or inspected during a requalification program evaluation in accordance with IP 71001, the results may be reported as , part of the initial examination or inspection report. l i H. INDIVIDUAL REQUAllFICATION EXAMINATION REPORT ; After the regional office completes the requalification evaluation, it will keep a copy of each operator's NRC-conducted written. walk-through, and simubtor examination results and return the original documents to the facility licensee. The facility licensee is required by 10 CFR 55.59 to maintain records of these examination results along with a copy of the written examination until the operator's license is renewed or for two years after the license expires. The NRC chief examiner will ensure that Form ES-601-5, ' Individual Requalification Examination Report," is completed for each operator who takes an NRC conducted requalification examination. Tho report will include the following information for each individual: 1 written examination grade crew evaluation from the dynamic simulator examination, the individual follow-up results (P or F) from the dynamic simulator examination i the number (and percentage) of JPMs performed correctly,if conducted

? ,a ;

i b .' 15 of 36 NUREG-1021, Revision 8

ES-601 The regional office will send a copy of this report to the facility's training manager and to the operator with a letter notifying the operator of the examination results in accordance with ES-605. The regional office will also file a copy in the operator's docket file.

1. OPERATOR LICENSE RENEWAL POLICY ,

l Operators are not required to take an NRC+ conducted requalification examirtation in order to ? renew their licenses. However, if an operator takes but fails to pass an NRC-conducted exarnination, the NRC 'will not renew the license until the operator passes a retake examination conducted by the NRC, passes a retake examination administered by the facility licensee and inspected by the NRC in accordance 'with IP 71001, or passes an examination approved by the NRC. The regional office, in consultation with the NRR operator licensing program office, will determine the appropriate level of involvement on a case-by-case basis depending on the , I quality of the facility licensee's requalification program ES-605 contains the specific procedures to follow for an operator v;ho f ails one or more NRC- l administered requalification examinations, as well as the procedure for processing license renewal applications. J. RECORD RETENTION l

4

1. Easlity Reaualification Exarnination File j The operator licensing statt in each re0 i onal office will maintain a facility requalification file for each facility in its region and will retain those files consistent with regional office guidelines for retention of facility initial examination files. The regional office will place O j j

j the following items in the file for each requalification examination: j

a. Examination Standard attachments and forms: i l

ES 201, Attachment 4, " Examination Assignment Shee!" !

                   +        Form ES-403-1, " Written Examination Grading Quality Cnecklist"                 I ES-601, Attachment 2, " Corporate Notification Letter"                          {

Form ES-601-1," Examination Security Agreernent* l

                   -        Form ES-601-3, "Pcwer Plant Requalification Results Summary Sheet" and Form ES-601-4, " Power Plant Requalification Results Continuation

. Sheet"(if applicable) ;

                   -        Form ES-604-2, " Simulator Crew Evaluation"                                     ]

b. a master list of all JPMs administered and the operators to which they were l administered 1 i

c. a master list of all scenarios conducted and operators to which they were ,

administered (facility-generated forms or Form ES-D-1, " Scenario Outline," may l be used to meet this requirement) l 1

d. a copy of the written examination and answer key j NUREG 1021, Revisicn 8 16 of 36 1

1 O\j ES-601 e, a copy of the requalification examination report The regional office may require that additional documents be retained in the facility

                        ,         requalification examination file.

2. Operator Docket Files -

The regional office will retain the following records in each operator's docket file until the license is renewed, or for two years after the license expires or is terminated:

a. Form ES-601-5, " Individual Requalification Examination Report"

b. ES-605, Attachment 1,' 2, or 3, "Results Notification Letter"

c. a copy of all failed portions of the NRC-graded examination

3. Qther Files The regional office will retain reference materials used to develop each examination until the NRC has resolved with the facility licensee all failures associated with the examination and has sent a notification letter to each operator.

K. REQUALIFICATION STRESS FEEDBACK

               - 'T       The level of stress perceived by operators and facility personnel can affect their overall
                 ')'      performance on the requalification examination. Therefore, the NRR operator licensing program oMice is interested in monitoring the stress of operators and facility personnel participating in the requalification examination. Regional examiners and other personnel who are involved with an NRC requalification examination should assume the following responsibilities:

Monitor the level of stress in operators and facility representatives, and be alert for examination techniques that may be causing examination stress. Recommend to the program office any changes to NUREG-1021 that would further a5eviate operator stress. Recommendations should be documented and forwarded to i headquarters using " Report on interaction" (ROI) forms. j L. ATTACHMENTS / FORMS .;

                          ' Attachment 1,                                                                                                      I
                                                   " Examination Timetable" Attachment 2,i -        " Sample Corporate Notification Letter" Attachment 3,           " Examination Sample Plan"                                                                  ;

Form ES-601-1, " Examination Security Agreement" Form ES-601-2, " Evaluation Checklist for Facility Reference Material" Form ES-601-3, " Power Plant Requalification Results Summary Sheet" Form ES-601-4, " Power Plant Requalification Results Continuation Sheet" Form ES-601-5, " Individual Requalification Examination Report" k(}N 17 of 36 NUREG-1021, Revision 8

  .a mi..ms.a-   .  .
                                                                                         .    .i             ..          _ _ _ . . _ . _ _ -

ES-601 Examination Timetable Attachment 1 Qa_1g* Activity -120/90 The NRC notifies the facility licensee. -60 The facility licensee sends the NRC the materials requested for developing the examination (including written examination questions, simulator scenario banks, and JPMs). The facility licensee proposes composition of the crews to be evaluated and identifies facility examination team members. The facility licensee may request that the NRC chief examiner review the examination process with operators and managers. -45 The facility licensee submits its proposed requalification written examination and operating test. -30 The NRC concurs on the operating crews to be evaluated. -14 The NRC examiners visit the facility to review the JPMs to be administered, observe the static and dynamic simulator examinations, and validate the test items as needed. The chief examiner and the regional branch chief determine the length of time on-site and the number of examiners required, on the basis of the examiners' experience and the quality of the facility licensee's testing materials. The facility licensee designates a simulator operator. If requested, the chief examiner briefs the operators and managers about the j requalification examination process. !

-7           The facility examination team members finalize the examinations based on preparation week activities. Evaluators review reference material to prepare for the JPMs and simulator scenarios.

O The NRC administers the examinations to selected crews and operators. Facility licensee notifies the NRC of its final results for crews and individuals at the end of each examination week.

+7           The NRC finalizes the examination results.                                             ,
+14          The facility licensee transmits the written examination grades and a final             l summary to the NRC.                                                                    l
+30#         The NRC issues operator results and the final requalification examination report.

Number of days before or after the examination, except as noted

#      Number of days after receipt of facility results NUREG-1021, Revision 8                        18 of 36 O I

1 l ES-601 Sample Corporate Notification Letter Attachment 2 (_/ l NRC Letterhead ! (Date) (Names_ Title (Name of facility) (Street address) IQity Sigte. Zio code)

SUBJECT:

REQUALIFICATION PROGRAM EVALUATION

Dear (Namd:

In a telephone conversation on (date), (Name. title) and (Name, title) arranged to evaluate the requalification program and licensed personnel at the (facility name). The evaluation is scheduled for the week of (date). NRC examiners and evaluators from your facility will conduct requalification examinations, and the NRC will evaluate your requalification program in accordance with Sections ES-601 through ES-604 of NUREG-1021," Operator Licensing Examination Standards for Power Reactors," Revision 8. You are encouraged to ensure that your training staff and proposed examinees are familiar with these standards. For the NRC to adequately prepare for this evaluation, the facility licensee will need to furnish the NRC with the approved items listed in Enclosure 1, " Reference Material Guidelines." You are also requested to submit, at your option, a proposed examination for use during the ( examination week. However, if you do submit a proposed examination, the personnel Q] participating in its development will become subject to the security restrictions described in this letter. Please review the guidance promulgated in Revision 8 of NUREG 1021 concerning the content and scope of simulator examination scenarios. The scenario ex;.nination bank should cover the entire spectrum of emergency operating procedures (EOPs), including alternative decision paths within the EOPs, and it should incorporate a range of failures with various degrees of severity for the same type of event. Each scenario should contain simultaneous events that require the senior reactor operators (SROs) to prioritize their actions and to assign particular tasks to other crew members. Each scenario should also require the SROs to decide when to make the transition between EOPs and which actions to take within EOPs. You are requested to designate at least one employee to be a member of a joint NRC-facility exarnination team. That employee is expected to be an active SRO (as defined by 10 CFR 55.53(e) or (f)) from the (facility name) operations department. You are encouraged to designate a second employee from the training staff to be a member of the examination team. This employee should also be a licensed SRO, but may be a certified instructor. If desired au agreed to by the chief examiner, you may designate one additional employee from the training staff with appropriate qualifications to be a member of the examination team. In addition to these individuals, you will need to designate a simulator operator for scenario preview and validation during the on-site examination preparation week. In some cases, you may need to designate a simulator operator during the test item review period. All of these individuals will be subject to the examination security agreement. 19 of 36 NUREG-1021, Revision 8

ES-601 2 Attachment 2 The NRC restricts any facility licensee representatives under the security agreement from knowingly communicating by any means the content or scope of the examination to l unauthorized persons and from participating in any facility licensee programs such as instruction, examination, or tutoring in which an identified requalification examinee will be present. These restrictions apply from the day that the facility licensee representative signs the examination security agreement indicating that the representative understands that he or she has specialized knowledge of the examination. The chief examiner will determine when a facility licensee representative has received specialized knowledge conceming the examination ! and will execute an examination security agreement. In most cases, the examination team members will not be required to enter into an examination security agreement more than 60 days before the examination week. The simulator operator will normally become subject to the security restrictions during the examination preparation and validation week; however, this may occur as much as 45 days before the examination week. Sixty days before the examination administration date, please provide the NRC regional office with a proposed list of operators, including crew composition, for the examination. The list should include at least 12 operators, comprising three or more crews, and the current mailing address for each proposed operator, if different from that listed on the most recent Form 398 submitted " the NRC. Your training staff should send this information directly to the NRC chief examir wuring that each operator's address is sent in a manner to ensure privacy. The facility licensee may request that the NRC chief examiner or another NRC representative meet with the operators to be examined and the licensee managers during the examination preparation week, normally two weeks before the examination. However,if the schedule does not allow them to meet during the preparation week, they may meet at any mutually agreeable time. The NRC examiner will explain the examination and grading processes and will respond to any questions that the operators may have about the NRC's examination procedures. If such a meeting is desired, your training staff should schedule it with the NRC chief examiner. The facility licensee staff is responsible for providing adequate space and accornmodations to { properly develop and conduct the examinations. Enclosure 2, " Administration of Requalification j Examinations," describes our requirements for developing and conducting the examinations. 1 Also, a facility operations management representative above a shift supervisor !cvel should observe the simulator examination process at the site. This letter contains information collections that are subject to the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq.). These information collections were approved by the Office of Management and Budget, approval number 3150-0101, which expires on September 30,2000. l The public reporting burden for this collection of information is estimated to average 25 hours per response, including the time for reviewing instructions, gathering and maintaining the data , needed, and completing and reviewing the collection of information. Send comments on any i aspect of this collection of information, including suggestions for reducing the burden, to the l Information and Records Management Branch (T-6 F33), U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001, or by Internet electronic mail at bjs1 @nrc. gov; and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0101), Office of Management and Budget, Washington, D.C. 20503. 4 NUREG-1021, Revision 8 20 of 36 l l i l

r 1 l

       ' ES-601                                            3            ,

Attachment 2 The NRC may not conduct or sponsor, and a person is not required to respond to, a collection i of information unless it displays a currently valid OMB control number. Thank you for your cooperation in this matter. (Name) has been advised of the NRC guidelines and policies addressed in this letter. If you have any questions on the evaluation process, please contact (Name. reaional section chief) at (teleohone number). Sincerely, (Appropriate Regional Title) Docket No..: 50-(Number)

Enclosures:

1. Reference Material Guidelines

2. Administration of Requalification Exarninations DISTRIBUTION:

Public NRC Document Control System

 /N      Regional Office Distribution l

l 6

 \                                                                               NUREG-1021, Revision 8 21 of 36                                          I I

1 l l ES-601 4 Attachment 2 Enclosure 1 I Reference Material Guidelines

1. Sixty days before the examination date, the facility licensee should provide test items to i support all aspects of the requalification examination to the NRC. l

2. The facility licensee is expected to submit the following reference materials for all NRC-conducted requalification examinations:

an examination sample plan that meets the requirements of ES-601, 1 Attachment 3 ) l the facility's examination banks (written, simulator, and JPM) and associated i reference materials (including, at a minimum, technical specifications, abnormal l and emergency operating procedures, and emergency plan procedures utilized 1 in requalification training) additional reference materials requested by the NRC chief examiner

3. The facility licensee's examination banks are expected to contain the following information:

a minimum of 700 test items equally divided for use in the two sections of the written examination and covering all safety-related elements of the facility job-task analysis (JTA). The facility licensee is expected to maintain a dynamic bank by reviewing, revising, or generating at least 150 questions a year. New ques-tions should cover equipment and system modifications, as well as recent industry and licensee events and procedural changes. JPMs that meet the criteria in ES-603 for evaluating each reactor operator (RO) and senior reactor operator (SRO) safety-related task identified in the facility JTA. The JPM bank should expand at a rate of at least ten JPMs per year until this goalis reached. It is estimated that 125 to 150 JPMs will be the final result. a bank of at least 30 simulator scenarios reflecting all abnormal and emergency situations to which an operator is expected to respond or control. At least five scenarios per year should be generated until all aspects of the emergency operating procedures are covered with sufficient variation in the type and scope of initiating events and level of degradation. Emphasis should be placed on scenarios that include applicable industry events. NUREG-1021, Revision 8 22 of 36

~ l l l l l l '[] ES-60T 5 Attachment 2 ! U Enclosure 2 Administrat;on of Requalification Examinations

1. The NRC must evaluate at least 12 operators to perform a program evaluation. The guidelines on crew comiwsition in the simulator are described in ES-601, Section D.2, and ES-604. I

2. The simulator and simulator operators need to be available for examination I development. The chief examiner and the facility representatives will agree on the dates l and amount of time needed to develop the examinations.

3. The chief examiner will review the reference materials used in the simulator. The NRC will not authorize for use during the simulator test any reference material that is not normally used for plant operation in the control room. i 1

4. The facility licensee will provide a single room for completing Section B of the written examination. The examination room and the supporting restroom facilities will be located to prevent the examinees from having contact with any other facility or contractor personnel during the examination.

5. The chief examiner will inspect the examination room to see that it meets the minimum

(~'N standard that will ensure examination integdty. The minimum spacing standard consists

 \          of one examinee per table and a three-foot spacc between tables. No wall charts, models, or other training materials are allowed in the examination room.

6. The facility licensee is expected to provide a copy of each reference document for each examinee for Section B of the written examination. The material should include documents that are normally available to the operators in the control room (such as the technical specifications, operating and abnormal procedures, administrative procedures, and emergency plans). The chief examiner will review the reference materials before the examination begins.

7. The NRC requalification examination will attempt to distinguish between RO and SRO knowledge and abilities to the extent that the facility training materials allow the developers to make these distinctions.

8. Prudent scheduling of examination week activities is important to help alleviate undue stress on the operators. The facility training staff and the NRC chief examiner should attempt to formulate a schedule that will minimize delays while conducting the examination. The following suggestions will help to structure the examination activities to achieve this objective:

Consider allowing operators to stay at hom .til their scheduled examination times. V 23 of 36 NUREG 1021, Revision 8

ES-601 , 6 Attachment 2 2 Enclosure 2 Segregate the group of operators completing their examination, instead of the group of operators that are scheduled to start their examination. Following simulator scenarios, the facility evaluators and NRC cxaminers should quickly determine whether follow-up questioning is required to that the crew members may be released to talk among themselves about the scenario. Ensure that time validation of JPMs, particularly those performed in the l simulator, is accurate. Establish a reasonable schedule to prevent operators i from waiting for sirrulater availability to complete their JPMs. ,

9. The NRC does not require the facility licensee to videotape dynamic simulator l examinations. Il the facility licenses requests to videotape the examination, any use of the tape must be completed before the NRC leaves the ' ite at the end of the l.

examination. If a disagreement over the grading of an operator still exists at the end of the examination week, the facility licensee may retain the tape for the purpose of submitting it to support a request for regrade by the NRC. During the regrade, the NRC will review only the portion of the videotape under contention. After all requalification examination grades are finalized, including the review of e.ny regrade requests, the facility licensee is expected to erase all video tapes made during the examination. O NUREG-1021, Revision 8 24 of 36

l ES 601 Examination Sample Plan Attachment 3

 \                   .

A. INTRODUCTION An examination sample plan provides a systematic approach to selecting and developing test items to determine if a student has mastered the knowledge, skills, and abilities (KSAs) to be

    . covered in _a particular training program. The sample plan should provide an explicit, documented link between the leaming objectives associated with the training program and the test items used to perform the evaluation and to verify tne relevance to the job task analysis (JTA) associated with the operator's position.

ES-401 gives explicit guidance for developing a sample plan for initial examinations using NUREG-1122 or -1123, the " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized (or Boiling) Water Reactors." A similar methodology may be applied to any training program. With respect to a requalification program, the scope of topics is necessarily limited because the amount of material that is covered during a requalification - program is less than that covered in an initial licensing training program. However, the NRC

     . permits and encourages reserving 10 to 20 percent of test items for topics that have high importance ratings and contain K/As that operators should retain because of their safety significance, but were not necessarily covered during the requalification cycle.

B. REQUALIFICATION TEST OUTLINE D The facility licensee is expected to develop a test outline for all NRC-administered requalification examinations.' At least 80 percent of the test outline must reflect the training

curriculum of the most recent requalification cycle in a manner consistent with the distribution of emphasis in the curriculum. The curriculum of the requalification training cycle for which the examination is being developed should identify the following information:

1. requalification lecture / classroom topics indicating the percent of the cycle devoted to each

2. concentration of training exercises using the simulation facility, including the type of scenarios trained (e.g., accident, abnormal, normal) and the number of times each scenario was run

3. special focus of the training such as plant modifications, licensee event reports (LERs),

                  ~ and major changes to operating practices or policy .
      . 4.         practical training such as operation of individual systems or components for requalification training purposes, using either the simulation facility, mockups, or actual systems and components I

O-I 25 of 36 NUREG-1021, Revision 8

l 1 i ES-601 2 Attachment 3 The format of the sample plan is a matter of training department preference as long as the plan results in a thorough and accurate assessment of the facility's training program and its intended j objectives. The sample plan is expected to contain the following information for use in developing or selecting the test items to be used in the requalification examination:

1. Identification of the subjects to be evaluated (system, component, procedure, or other l training subject)

2. the preferred testing media for evaluating each subject (written, simulator, or walk-through examination); more than one testing method may be used to evaluate a subject

3. the learning objectives intended to be evaluated

4. a list of references used to develop the test items

5. the specific K/A topic or facility JTA KSAs that are closely linked to the learning objectives for each subject and the importance factors for each (the facility licensec may use a site-specific K/A if it exists)

6. all test items used in the examination should have a K/A value of 3 or greater; the facility licensee may propose the use of test items with NRC K/A values less than 3 with appropriate justification

7. the percentage or number of points of the examination that should be devoted to the topic area (e.g.,3 points for technical specification interpretation, or five percent on reactor coolant pumps)

8. whether the subject is identified as safety-related in the facility JTA

9. whether the subject was covered in the cycle for which the examination is being developed

10. the identification code or number for previously developed test items that evaluate the subject

11. recent safety-related issues and events (e.g., relevant LERs) i I

l NUREG 1021, Revision 8 26 of 36 i 1

ES-601 Examination Security Agreement Form ES-601-1

1. Pre Examination I acknowledge that I have acquired specialized knowledge about the requalification examination scheduled for the week (s) indicated in this agreement as of the date of my signature. I agree that I will not knowingly divulge any information about this examination to any unauthorized persons. An unauthorized person is any individual who has not been approved by the NRC chief examiner to receive specialized knowledge of ,

the examination. I understand that I am not to participate in any instruction, tutoring, or I examination involving those operators scheduled to be administered this requalification examination from this date until completion of examination administration. I further understand that violation of the conditions of this agreement may result in cancellation of the examination and/or an enforcement action against me or the facility licensee by j which I am employed or which I represent. l l

2. Post-Examination To the best of my knowledge, I did not divulge to any unauthorized persons any information concerning the examination (s) administered during the indicated week (s).

From the date that I entered into this security agreement until the completion of j examination administration, I did not participate in instructing those operators who were administered this requalification examination (s). Examination Period to d Printed Name Pre-Examination Post-Examination Certification (1) Certification (2) Date Date Date Date Date _ Date Date Date Date Date Date Date Date Date ) Date Date l l Date Date Date Date Date Date Date Date i Date Date Date Date l Date Date Date Date l L F k 27 of 36 NUREG-1021, Revision 8 l

1 ES-601 Evaluation Checklist for Facility Form ES-601-2 l Reference Material The following checklist represents the minimum content of facility-generated reference material. Items marked " optional" should be checked if requested from the f acility licensee by the chief examiner. The chief examiner or designee may use this checklist to make a quick, general evaluation of the completeness and adequacy of the facility licensee's r ferences. The chief examiner may resolve any specific questions about the references wit

the facility staff as necessary.

I. Quantity , Actual l Reference Material Recuired Minimum Submitted j l A. Open-reference written 350 per section; bank i examination items is to be dynamic, with at least 150 revised, reviewed, or newly generated questions per year I J' B. Simulator scenarios 25; + 5 per year following the initial requalification exam until at least 30 scenarios covering all aspects ] of the EOPs are developed C. Job performance 95; + 10 per year following measures the initial requalification exam until the JTA is fully covered D. Technical specifications 1 copy E. Applicable plant 1 set (optional) procedures F. Emergency plan 1 copy G. Applicable administrative 1 copy (optional) procedures H. Sample plan 1 copy

1. Requalification cycle 1 set (optional) training reference material (lesson plans, handouts, etc.)

J. Appropriate sections 1 set (optional) of JTA or facility-specific K/A Catalog Reviewed by: Date: NUREG-1021, Revision 8 28 of 36

I Ch . ES-601 2 .' 5

                                                                                       '01-2 II. Usability Q::'u one A. The reference materialis legible                                          yes no B. The reference material is properly arranged and labeled for its function                                                          yes no C. The reference materialindicates a SAT program                             yes no D. Reference material is available to verify that test items are appropriate, job relevant, and technically accurate                                                                  yes no E. Reference material is available to adequately support the examination topics                                            yes no Comments O

Reviewed by: Date: O'. 29 of 36 NUREG-1021, Revision 8

r l l ES-601 111. Quality 3 Form ES-601-2 e{! i Exam Section Reauired Standards Comments A. Sample Plan Subjects covered in requalification cycle are identified The test outline incorporates the following: time spent on topic relative importance frequency of performance job level (RO or SRO) K/As (or facility equivalent) of sufficient importance are identified in the test outline Plant-specific priorities are identified (LERs, procedure changes, system modifications, risk-dominant accident scenarios, risk-important systems and operator actions' identified in the facilly licensee's PRA/IPE, etc.) Appropriate testing methods are indicated for each K/A (i.e., JPM, written exam, and/or simulator) Applicable learning objectives are associated with K/As Methodology exists to tie test items to a learning objective and a K/A ; Sample plan includes important topics not covered in the requalification cycle Test areas appropriate to ROs and SROs only are identified Reviewed by: Date: j i

           ' Chapter 13 of NUREG-1560," Individual Plant Examination Program: Perspectives on Reactor safety and Plant  ,

Performance,' identifies a number of important human actions that may be appropriate for evaluation. j NUREG-1021, Revision 8 30 of 36

r; l I J l ES-601 4 Form ES-601-2 lll. Quality (cont.) l Exam Section Reauired Standards Comments i B. Written At least 10 percent of all test items shall be reviewed

l. using Form ES-602-1

                                  . Test items are important l

to safety Test items are clearly written Test items are appropriate to license level l The criteria for open reference examinations are met l/ Test items are associated

                                 ' with K/As of 3 or greater and are adequate discriminators A leaming objective and -

applicable reference are identified for each test item The facility has identified SRO-level questions for both sections of the exam 1 if the above criteria are not adequately met, the NRC will conduct further review of the examination bank utilizing ES-602, Attachment 1, " Guidelines for the Development and Review of Open-Reference Examinations," and Form ES-602-1, "NRC Checklist for Open-Reference Test items." Reviewed by: Date: l_

 .O 31 of 36               NUREG-1021, Revision 8

ES-601 Form ES-601-2 llt, Quality (cont.) Exam Section Reauired Standards Comments C. Walk-Through At least 10 percent of the JPM bank reviewed, using Form ES-C-2 Applicable plant systems identified by test outline: systems covered in requalification cycle new or recently modified systems systems in recent facility LERs or vendor notices PRA-identified risk dominant systems systems in NRC generic communications Tasks / abilities for identified systems are applicable to the facility l are at the AO/RO/SRO level ! have a K/A value of 3 or greater include JPMs pertinent only to SROs Some JPMs are performed under low-power or shutdown operating conditions Some JPMs require the operator to implement attemative paths within the facility licensee's procedures Facility JPMs contain the information found on Form ES-C-1 Reviewed by: Date: NUREG-1021. Revision 8 32 of 36

O)

%a ES-601                                                       6                                         Form ES-601-2 111.      Q_uality (cont.)

Exam Section Recuired Standards Comments D. Simulator At least 10 percent of the scenarios reviewed, using Form ES-604-1 Scenarios are an appropriate measure of the material covered in the sample plan Scenarios are based on lessons covered in the requal cycle recent industry events LERs emergency and abnormal procedures design and procedural changes Scenarios exercise crew's ability to use facility procedures in accident prevention and mitigation Scenario events have a K/A of 3 or greater ( Some scenarios are based on low power8 operations Some scenarios are based on the dominant accident sequences (DAS) for the facility as determined by PRA/IPE Scenario identifies critical tasks that meet the criteria of Appendix D Proposed examination scenarios that were used for training during _ the most recent training cycle have been reviewed by the NRC and ! replaced or modified,if appropriate, to ensure the validity of the examination and to minimize the potential for examination compromise Reviewed by: Date: . l 2 .{% NUREG 1449, *NRc staff Evaluation of shutdown and Low-Power operation,' defines low power to include the range

   ' from enticality to 5 percent power.

33 of 36 NUREG-1021, Revision 8

l ,i i d O t. 6- T

S d le % W E m ia/ F # r ) o F F r r o t o P ( la d u e s m s s /

                     %                               t l

u i S a s P # e R n t e it r l t a W t e o # e T h y n e

B % % % % % % % % % % % % % % S l n t y r O it r & a e W A ( m s m U . u l . s S a i M lt s i c ~- : r : s P % u f f s - o - r J s O t t a o w e r l u r t a e r R o s ._ e p r C 6 e e r 3 F p e C C C C C C C O R . O g C C C C C C C fo d iv - l r O in a R A R A R A R A R A R A R A 4 t a n l a . to r t r N F N F N F N F N F N F N F ic o r e c io a ra l: G 3 f i t v a n e i la a . e e o p t u m O R S T O i a r e f o t

R n e I k t n y c c o la a D ( P v 5 r i r 5 e P w o P r t o 8 a r n e o p is

O iv s

r e e R n 1 te im 2 a a 0 y D xE 1 it G 1 0 l mC E a xa R O ic R E

o. F E N ' U N

l l I

{ l llill 4 8

 -                                                                                       n d

w T is o

         -                 /                                                           i S                                                                                   v E                     W                                                             e R

m r 1 o ) 2 F F i v 0 r r d 1 o o n - P t la I G ( u E s R lt m w U u i S e N s r e C R n t e t i r t W e e h y S l n n e B

                            )       % % % % % % % % % % % % % % % % % % %            %

y r O t it & a e r m m U s WA ( u l S i a s c s lt u i f f M% P s e O J r 6 R o 3 7 F r f / D - d e C C C C C C C C C C C C C C C C C C C C 5o t a n a R A R A R A R A R A R A R A R A R A R A 3 ic o r N F N F N F N F N F N F N F N F N F N F f i l i t a G a m u r a n o ) f A i n t e t k n y c la c o ( P a D 5 r i v 5 e r w P o P r t o a r e p O 1 0 5

   'v E

lll

l ES-601 Individual Recualification Examination Report Form ES-601,-5 U.S. Nuclear Regulatory Commission Individual Requalification Examination Report (Privacy Information - For Official Use Only) J Frcility- Operator's Name: Docket No: 55- Ucense No: Expiration Date: _ Exam Type: RO / SRO Retake: 1st / 2nd / No Date of Last Exam: 2 Whitten Examination Results ~ Date(s)of Exam: NRC Examiner (Print): Facility Evaluator (Print): NRC Grading Facility Grading Section A (Points) of of Section B (Points) of of Overall Score (%) % % (

                                        ' Simulator Examination Results                                          l Date(s) of Exam:                 NRC Examiner (s) (Print):          Facility Evaluator (s) (Print):

Crew Evaluation Pass / Fail Pass / Fail Individual Follow-up Pass / Fail / NA Pass / Fail / NA Walk-Through Examination Results Date(s) of Exam: NRC Examiner (s) (Print): Facility Evaluator (s) (Print): 1 No. of Successful JPMs of 5 of 5 Exam Results (%) % % NRC Examiner Recommendations Category Results Signature /Date Written Pass / Fail Simulator Pass / Fail Walk-Through Pass / Fail NRC Supervisor Review Date: Pass / Fail Sianature: 1 NUREG-1021, Revision 8 36 of 36 ' I

r O_ ES-602 REQUALIFICATION WRITTEN EXAMINATIONS A. PURPOSE The NRC staff conducts written requalification examinations, using this standard in accordance

    . with Title 10, Section 55.59(a)(2)(iii), of the Code of Federa/ Regulations (CFR). NRC examiners are to follow this standard to prepare and administer all NRC-conducted written requalification examinations. Moreover, NRC examiners are to use this standard in conjunction with ES-601, which explains how to conduct requalification program evaluations.

B. SCOPE. The written examination is useful for evaluating those knowledge, skills, and abilities (KSAs) of licensed operators that are difficult to infer from behavior alone but that can be readily tested through written responses to questions that value interpretation and allow the examinee to use references. Additionally, determining an individual's knowledge of factual information and ability to perform " paper-and-pencil" tasks are best evaluated by a wntten examination. The written examination consists of two sections for which the examinee may refer to references (i.e., "open-reference examinations"). Section A, " Plant and Control Systems," is administered using a static simulator. Section B, " Administrative Controls / Procedural Limits," may be administered in a classroom. Each section should be designed to last a minimum of one hour, including time for the operator to review his or her work. Combined, the two sections

 /'   of the written examination will be designed to last three hours. The exact number of questions and time allowed to complete each section will be determined by the facility licensee on the basis of the requalification sample plan and the license level of the operators taking the examination (reactor operator (RO) or senior reactor operator (SRO)).

Even though the examination is designed so that examinees may use references, an examinee should not expect to have time to complete the examination by consulting references to determine each answer. A good mix of test items will contain some questions that evaluate the operators' ability to determine a correct response without delving into reference material and

     -others that require the use of reference material to select the correct response. By combining test items that require using references with those that do not, the written examination can test a broader sample of operator l<nowledge within a given period.

On both sections of the written examination, certain questions will test the knowledge and abilities (K/As) of an RO while others will test those of an SRO. During the development of the examination, the examiner should consult the facility job task analysis (JTA) and NUREG-1122 or -1123 to help identify the most suitable topics for an RO and an SRO, respectively. Additionally,10 CFR 55.41 and 55.43 give further guidance on item selection for RO and SRO written examinations, respectively. , l

1. .Section A. " Plant and Control Systems" (Static Simulator)

This section of the written examination is designed for using the simulator as a reference tool to provide realistic information visually and to give the operators an 1 of 27 NUREG-1021, Revision 8

ES-602 environment as close as possible to their normal control room. While administering this section, the simulator will be frozen in the middle of an evolution, transient, or accident. In developing the test items for this section, allow the use of references and relate them to plant systems and components, control room indications, instrumentation and controls, and technical specification (TS) limiting conditions for operation (LCOs). Section A i designed to evaluate the operators' knowledge of plant systems, integrated plant operations, and instruments and controls. In addition, it evaluates the operators' j ability to recognize TS LCOs and to determine the effects of postulated events. The NRC encourages facility licensees to include questions that test the ability of the operators to use their facility curves and charts. J While administering Section A, the examination team will use one " frozen" simulator condition or set-up. The condition places the simulator in a " snapshot" of the plant j following a major transient that resulted in an engineered safeguard feature (ESF) l initiation or in a steady-state situation at power. Some equipment should be frozen in an abnormal or failed condition to provide adequate material for test items.

2. Section B. " Administrative Controls /Procedurai Limits" Section B of the written examination is designed to evaluate the ability of the operators to analyze a given set of conditions and determine the proper procedural and adminis-trative guidance to use. This section may include theory related questions appropriate to sample the topics listed in 10 CFR 55.41 and 10 CFR 55.43, as long as they are operational in nature or test unique facility characteristics.

Section B is designed to test the operators' knowledge and use of plant procedures and administrative controls, while allowing the use of refernces. The NRC uses I administrative, operating, normal, abnormal, and em < p' Acy preadures, TS, and the I emergency plan as sources of test items for this sectl6n of the exarrination. The test items focus on how direction, guidance, and information found in thec e procedures are used or laterpreted, rather than focusing on finding the procedure in which the necessary information is located. Additionally, the test items for Section B of the SRO l examination examine the operators' understanding of the reasons and bases for procedural requirements. The use of graphs, charts, tables, and drawings is appropriate. The simulator may be made available to the examinees to make the examination more operationally oriented. C. EXAMINATION DEVELOPMENT

1. Facility Examination Team Members' Responsibilities

a. The facility is expected to provide a bank of test items developed by using the guidance in Attachment 1 and Appendix B. The number of test items should meet the submittal guidelines of ES-601, Attachment 2, Enclosure 1, "Requalification Examination Reference Material Requirements." Form ES-601-2, " Evaluation Checklist for Facility Reference Material," provides information NUREG-1021, Revision 8 2 of 27 l

l

f ES-602 that facility personnel may use to evaluate reference material sets before submitting them to the NRC. ) The facility licensee shall maintain its examination question bank up-to-date by reviewing, modifying, or creating at least 150 questions each year to expand the bank, reflect procedure or system changes, new lesson plans, and recent licensee and industry events. If the facility question bank contains at least 700 items that meet the format guidance of Attachment 1, the facility may release the bank to its operators for review.

b. The following items should be provided for'each test question:

applicable K/A reference and values (RO/SRO) reference JTA (if applicable) estimated time to answer appropriate leaming objectives applicable reference (e.g., lesson plan, EOP)

c. The facility is expected to provide a sample plan that meets the guidelines of ES-601, Attachment 3, " Examination Sample Plan," and may submit a proposed examination that conforms to the facility sample plan. The proposed examination should contain a total of 30 to 40 test items, depending on the time C' validation (maximum time of three hours) of the individual questions selected.

Sections A and B should each contain 15 to 20 questions. Each section must be designed to last a minimum of one hour, with the total examination designed to last three hours. The facility licensee will determine the number of questions in each section, basing the decision on the requalification sample plan and the license level of the , operators taking the examination (RO or SRO), and subject to the quantitative constraints of the previous paragraph. Plant systems questions not directly related to the static scenar',o can be included in Section A to meet the facility's sample plan and 10 CFR 55 requirements. In addition, up to 20 percent of the test items may be from topics outside the sample plan, as long as the information stated in Section C.1.b. of this standard is provided, if the facility licensee submits a proposed examination, those individuals involved in its development become subject to the security restrictions of ES-601 once examination development commences. These restrictions remain in effect until the NRC examination is given. If, after developing a proposed examination, the facility decides not to submit it for use in the NRC-conducted examination, the developers are released from the security restrictions of ES-601.

d. After the NRC has reviewed the facility's examination bank and commented on the test items selected for the examination, the facility team members are 3 of 27 NUREG-1021, Revision 8

i I ES-602 expected to psepare the examination for final NRC review and approval. The examination may be finalized before or during the preparation week.

e. The facility team representative will evaluate each test item that the NRC j

revised, in order to assess the following criteria: I l appropriateness time required to answer, given the operational context technical accuracy clarity K/A and objective references Following this evaluation, the facility examination team representatives and the ! chief examiner need to agree on the final form of the examination. They also l need to complete a time validation of the proposed examination. A variety of methodologies have proven effective in accomplishing this task; Attachment 1 provides further information in this area. Any individual involved in time validating the examination is required to sign the security agreement, Form ES-601-1. The examination team may add or delete items from the examination based on the results of this time validation, and their experiences. If any test items are added, the entire examination need not be time validated again as long as a subject matter expert (SME) has reviewed the added questions, indicating the approximate time an operator should take to answer each question.

f. The facility licensee is expected to provide enough copies of each reference so that each examinee can use the references during the examination and, immediately upon completion of the examination, compile the examinations and reproduce sufficient copies for their own use and that of the NRC.

g. To help relieve the burden of providing a complete set of references to each operator, the examination may be assembled so that a different sequence of questions appears on each operator's examination. Altematively, handouts of relevant information (e.g., plant curves, blank forms, etc.) may be provided with the test.

2. NRC Examination Team Members' Resoonsibilities

a. The NRC will begin its evaluation of the sample plan, the bank of test items, and the proposed examination as soon as possible after receipt of the facility's materials. The NRC will promptly evaluate the materials to allow sufficient time l for the NRC or the facility to develop the test items and for the facility to revise them to meet NRC standards, if required. The NRC examiners should review the proposed test items using Form ES-602-1, "NRC Checklist for Open-l Reference Test !tems," to ensure appropriateness, clarity, and importance to l

t safety, as described in Attachment 1. NUREG-1021, Revision 8 4 of 27 e l L___.___________ . . . . . .

[- ES-602 If the facility licensee intends to administer both sections of the examination during a single three-hour period as noted in Section D.1.c, the examination team members must review the examination as a whole to ensure that the items in either section do not compromise those in the other.

b. A minimum of 80 percent of the test items will be chosen in accordance with the sample plcn. The remaining 20 percent may be substituted by the examination i

team, using facility examination bank questions or new questions the exam team develops. Should it be necessary to develop additional items to satisfy the sample plan, the NRC will request that the facility do so.

c. If, after reviewing at least 75 percent of the bank, insufficient test items exist to develop an NRC examination that meets the sample plan, the NRC staff will declare the bank of test items inadequate, in that event, the regional managers may either cancel the scheduled examination or administer an examination using NRC-developed test items without consideration for the 20 percent substitution constraints.

I

d. If the sample plan does not include testing of topics from outside of the requalification cycle, then the examination team should consider incorporating 10 to 20 percent non-requalification cycle specific test items.

e. If a test item does not have a clear tie to the JTA, the examiner will discuss the

   ,            applicability of the test item with the facility representatives.
   \

D. EXAMINATION ADMINISTRATION AND EVALUATIONS

1. Written Examination Conduct

a. An NRC examiner or a knowledgeable facility representative who has signed the security agreement will proctor each section of the examination. As a minimum, an NRC examiner will observe the examination briefing as the operators begin l the examination to ensure that all administrative aspects of the examination are adhered to. If an NRC examiner does not continuously proctor the examination, an examiner will periodically visit the examination room to ensure that the proctor appropriately addresses questions on the content or administration of the examination that may have arisen.

b. Section A is administered on the facility's simulator or an approved simulation facility.

c. Section B may be administered in the simulator or in a classroom setting as the facility staff and the chief examiner deem appropriate.

If both sections of the examination are administered in the simulator during a single three-hour period, operators may return to a section of the examination that they already completed or retain both sections of the examination until the

 ,              time has expired.

l 5 of 27 NUREG 1021, Revision 8 l L

1 1 ES-602 )

d. For Section A of the examination, the facility licerisee is responsible for providing !

the group of examinees at least one copy of all controlled reference material j available in the control room. Examination reference material will notinclude j material that is intended for training use only. The licensee controls all reference material in accordance with the licensee's 10 CFR 50, Appendix B, procedure revision control program. This material should be authorized for use in operating the power plant, agreed upon by the facility and the chief examiner, and in effect at the time of the examination validation (i.e., the preparation week),

e. During the administration of Section B, each examinee will have available for use the following material (complete, current issue):

I technical specifications l plant procedures (EOP/AOP/OP, etc.) l emergency plan (as available in the control room) administrative procedures applicable to operations other controlled plant reference material normally available in the control room (e.g., curves and data book, forms, plant drawings, flow charts, etc.) and authorized for use in operating the plant Note: "Non-controlled" reference material, such as the Emergency Operating Procedure Owner's Group Basis Documents will not be provided unless these documents are authorized to be used by the control room operators during plant operations.

2. Examination Administration Procedures The written examinations will begin only after the chief examiner has verified the adequacy of the examination facilities and made arrangements for continuous proctoring of the examination as discussed in Section D.1.a of this standard. An NRC examiner i may act ao proctor during this examination. However, the chief examiner is responsible for ensuring the actions of D.2.b through D.2.1 below are complete.

Each section of the written examination will be administered as follows:

a. An NRC examiner will verify each examinee's identity and examination level against the examination assignment sheet (ES-201, Attachment 4). If possible, the ROs and SROs should be seated at alternate tables. Any errors or no shows will be resolved with the facility staff and the assignment sheet will be updated as required.

b. The proctor will remind the operators that they ma 'use calculators to complete the examination and that no other reference material other than that provided is j allowed in the examination area. The proctor will define the examination area for !

the examinees. ! l NUREG-1021, Revision 8 6 of 27

G

       ;4 ES-602
                  . c.      The proctoiwill pass out the examinations and answer sheets and instruct the       <

l . examinees not to turn over the examination until told to do so. The examinees will be informed that pads of scrap paper are available upon request from the

                                                                                                               )
        '1 proctor,                                                                          l

d. The proctor will brief the examinees regarding the rules and guidelines in effect l during the written examination using Parts A and B of Appendix E, " Policies and )

Guidelines for Taking NRC Examinations." The proctor should inform the - examinees that they may refer to the instructions directly beneath their l examination cover sheet. The proctor will read the indicated policies verbatim. , 1

e. The proctor will ask the examinees to verify the completeness of their copies by l checking each page of the examination. The proctor should also have the examinees check to ensure that an equation sheet has been included in their examination,if required. l

f. After answering any questions that the examinees may have about the examination policies, the proctor will start the examination and record the time,

g. The proctor will periodically advise the examinees of the time that remains to complete the examination. Normally, a chalk board or white board i.s available for this purpose. 1 i

G h. As the examinees complete the examination, the proctor will ensure that they sign the examination cover sheet and staple it on top of their answer sheets. The proctor will collect the examination packages, including the questions and answer sheets, any references used with the examination, and all scrap and unused paper. The NRC examiner will keep the cover and answer sheets, dispose of the scrap paper, and give the packages of questions to the facility licensee for subsequent use.

i. The proctor will remind the examinees to leave the examination area, as defined by the examination team.

3. Written Examinaf8mfatuatioris.

Using the examination and answer key, the facility and NRC willindependently grade each section of the written examination and will complete the grading of all written examinations within 10 working days of the examination administration date. NRC examiners will record the grades on the written examination cover sheet (Form ES-602-

2) and complete Form ES-4031, " Written Examination Grading Quality Checklist."

An individual's grade will be obtained by summing the points credited to the examinee on both sections of the examination, and dividing by the total points available (Le., compensatory grading methodology.)

   ?'      .5
   .t' b:                                                       7 of 27                   NUREG-1021, Revision 8

ES-602 To pass the written portion of the examination, operators must achieve an overall score of 80 percent on the written examination. i--

4. Test item Evaluation if a number of test items require significant modification during the grading of the l examination (e.g., more than 10 percent of the items are deleted or the answer is changed from the original key) the NRC will determine the root cause and reflect it in the examination report. As discussed in ES 601, if significant deficiencies exist in the ;

facility's quality control of their examination bank, the NRC will consider them as part of the program evaluation. If technical flaws that have some degree of safety significance are found in procedures while analyzing the answers to the written examination, the facility may institute an i l immediate procedural change and inform all operators of the change. E ATTACHMENTS / FORMS 1 Attachment 1, " Guidelines for the Development and Review of Open Reference Examinations" i "NRC Checklist for Open-Reference Test items" ) Form ES-602-1, Form ES-602-2, " Written Examination Cover Sheet' O\ l I i NUREG 1021, Revision 8 8 of 2'7

ES-602 Guidelines for the Development Attachment 1 l 1 - and Review of Open-Reference Examinations i A. INTRODUCTION i The following guidelines are intended for those who are involved in developing or reviewing test j ltems for the written portion of the NRC requalification exam.ination. As described in ES-601, l

          ' Administration of NRC Requalification Program Evaluations," the written examination consists          l cf two sections. Section A utilizes a static simulator to provide the context for questions on          l
         ' plant and control systems, and Section B focuses on plant procedures and administrative controis. Examinees may use references, including simulator displays, for both sections.

Open-reference written examinations are used for two reasons:

1. Examination Vafidity By permitting the use of references that are available to the control room operators, the conditions and requirements of the written examination more closely approximate those of the actual job. The information provided to the operators in the test items should closely parallel the information typically available to them, while the responses elicited by the questions should be related to the decisions, solutions, and actions required for effective job performance. In other words, consulting references more closely correlates job demands and test demands - a cornerstone of examination validity. ,

i

 -1 O /

2. Levelof Kncvy),edgg !

Use of references enhances examination validity by elevating the level of knowledge of the test iterns. As described later in these guidelines, operator access to references precludes the use of questions that test for the mere recall of facts and specifics. Instead, open-reference test items require test takers to demonstrate that they can find, apply, analyze, evaluate, or otherwise use knowledge to handle the problems and issues ; encountered on the job. i l B. OPEN REFERENCE GUIDELINES i Most principles for effective test item construction apply equally to all types of written questions, regardless of format. Therefore, those who develop and review open reference test items a should consult Appendices A and B of this NUREG in addition to the following guidelines. ; I

1. Selection of Test Topics Select test item topics for the NRC requalification exarnination, using the following cnitoria. ,

l

a. Requalification Training Program Curriculum j i

Base the test topics on the curriculum of the most recent operator requalification , program training cycle. However, the NRC may substitute up to 20 percent of ) >p.

'('

NUREG-1021, Revision 8 9 of 27

l 1

l ES-602 2 Attachment 1 the examination topics selected by the facility with subjects not emphasized during the requalification cycle. Emphasize knowledges that are of high l importance in terms of safety significance in these test items,

b. Performance Basis Like the requalification program itself, draw the test topics from a job-task l analysis (JTA) for an RO a.1d an SRO. The facility licensee should validate each I test item by demonstrating a link between each item and the following JTA l products: (

important operator tasks as identified by the JTA ) 1 important K/As (rated 3.0 or higher) as identified in NUREG 1122/1123 or a facility-specific K/A catalog facility leaming objectives identified as important to safety ! I

c. Adequacy of Test Coverage The f acility's proposed sample plan (or test outline) should be checked to ensure that it provides balanced, comprehensive coverage of the requalification training cycle topics. The distribution of proposed facility test items on the examination may be revised if the topics under or over-represent the material covered in the requalification program. Recent safety related issues and events (e.g., those in relevant licensee event reports (LERs)) should be addressed in the sample plan.

ES-601, Attachment 3, " Examination Sample Plan" provides further information on sample plan development.

2. General Guidelines for Sectiong_A and B Use the following guidelines to construct and review test items for both parts of the written examination. These guidelines are inter'ded to supplement, rather than replace, the good practices stated in Appendices A and B.

a. Operational Orientation As discussed earlier, examination validity is enhanced to the extent that the demands of the test match the demands of the job. Therefore, in addition to being derived from important K/Ac and testing objectives, the context and stipulations of test items should mirror the situations encountered in the work NUREG-102.1, Revision 8 10 of 27 a

ES-602 3 Attachment 1 & setting. The following example illustrates effective and ineffective ways to design test items from K/As and learning objectives: KIA: Knowledge of the design attributes of the turbine-driven auxiliary feedwater pump (TDAFWP) ditferentialpressure controller \ l Task: Operate the TDAFWP controls during allmodes of plant operation. Learning Objective: The student will be able to operate the TDAFWP differentialpressure controller without error during a loss-of-feedwater event. Enabling Objective: After completing this lesson, the student willbe able to explain the operation of the TDAFWP differentialpressure controller. Poor Test item: State the parameters used by the TDAFWP differentialpressure controller. Better Test item: Before isolating the 'C' steam generator (per ( EPP11) an operator noted that the transducer-fed Q) auxiliary feed flow indicators for the 'C' steam generator were reading greater than the flow indicators to the 'A' and 'B' steam generators. Who ? the reason for this flow deviation? Notice that the second test item requires the operator to demonstrate mastery of the knowledge by applying it to an actual job situation. In developing items, ask oneself "Why is the K/A important to satisfactory job performance?" and "In what situation will the operator need this K/A?" The answers to these questions can provide a basis and context for a test item.

b. Levelof Knowledge The operational orientation required of test items on the open-reference examination, as well as the operators' access to controlled documents, precludes the use of questions that test for mere recall or memorization. Rather than requiring operators to simply recognize or recall facts and specifics, open- ,

reference test items have the operators demonstrate understanding by using the knowledge to address real-life situations and problems. A test item at the higher level of knowledge requires operators to determine or identify the appropriate fact, rulo, or principle to a novel situation and then correctly apply it., A descrip-tion of each level of knowledge is found in Appendix B. That Appendix and

11 of 27 NUREG-1021, Revision 8 q

             'A                                                                                       ___. _ __d

ES-602 4 Attachment 1 Table 1 at the end of this Attachment provide sample questions that illustrate the various levels of knowledge. i i

c. Realistic Context To additionally ensure examination validity, make the situation or problem posed in the open-reference test item as similar as possible to the actual situations that operators encounter on the job. Situations described in the questions should not only be realistic, but should also be free of common " context" problems, including " backwards logic" and " window dressing."

Backwards-logic questions provide operators with information they normally have to produce, while asking them for information they normally receive. For example: N/A: Ability to calculate shutdown margins Backward Logic item: If the shutdown margin is 5.5 percent, how long has the unit been shut down? Better item: The unit has been shut down forx hours. Which of the following is the shutdown margin? Questions with window dressing have additional, unnecessary information, typi-cally in an attempt to make a memory level item more operationally oriented, as in the following example: The plant has tripped from the effects of a tomado crossing the site boundary. You, as Shift Supervisor, direct the phone talker to complete the 15-minute notification. He informs you that the normalnotification networkis inoperable. Which of the following do you direct him to use for complet-ing the 15-minute notification? Better item: If the normal notification network is inoperable, \ which of the following methods do you use to complete the 15-minute notification after the plant has tripped? Another common problem when constructing a question with a realistic context is that often "real world" situations have more than one correct solution or response. Carefully check the question and references to ensure that each test item has only one correct answer. i NUREG-1021, Revision 8 12 of 27 9i I

i: L ES-602 5 Attachment 1

d. Question Novelty One of the most effective ways to ensure that an operator has a high level of knowledge is to present novel situations and require that the operator both realize what information is relevant and how to apply it. If a test question does not contain unique or varied circumstances different from those presented in training, the item will be reduced to eliciting simple recall. j l

When candidates are able to memorize test items and answers (in their static I state) to respond to test items, then we do not really know if they can truly solve the problems or if they have only memorized the answers. Once a test item and ; its answer has been seen and rehearsed, then it ceases to be a viable e i discriminator of safe operator performance, it is no longer challenging or testing , problem-solving ability; rather, it is simply testing recall. Therefore, test items ' must be dynamic, replacing or substituting items of like kind and difficulty to preserve integrity in the test discrimination process. Because an infinite number of combinations of plant or equipment parameters and malfunctions may exist at any given time, a true test will compensate for this variation and will become dynamic so that the test can adapt to the infinite number of combinations and still test the same kinds of responses, but to different situations, j Review the training material to ensure that questions do not include overly familiar conditions. Keep in mind, however, that all conditions and situations should be reasonable, realistic, and safety related.

e. The Relationship of Open-Reference Examinations and Direct Look-up Questions Direct.look-up questions are associated with open-reference examinations. The key phrase here, " direct look-up,* conveys the meaning that little mental activity is involved other than simply copying an answer, readily available in a reference,

                 . to a question (i.e., simple recall of where to find the information). Merely omitting from the -stem of the question any mention of where to find the answer does not make it an acceptable open-reference question.

Do not use direct look-up questions for two reasons. First, these items only test memory, in that the information is readily available; this is an inefficient and less valid means of testing candidate knowledge. Second, other than demonstrating that a candidate knows where to find information, this type of question does not

                 . test the understanding or analysis of the information that can be applied on the job. Consequently, this type of question will not discriminate the safe from the unsafe operator.

13 of 2'7 NUREG-1021, Revision 8 i

f ES-602 6 Attachment 1 The other option is an "open reference" question. Use an open-reference examinatior, to test candidate knowledge for the following purposes: Which reference to use and where to find it? How to apply the information in the reference to the problem? For an open-reference question, the kind and amount of information required to solve the problem would exceed that which could normally be committed to memoly, in other words, the NRC does not expect candidates to remember the information needed to solve the problem. in regular closed-reference questions, we expect the candidate to know and understand how systems operate to answer a question with the infonnation provided in the stem of the question. For a closed-reference question, the candidate would not need '.o consult a reference (i.e., the NRC expects the candidate to know and vaderstand how the systems work to solve the problem, given various conditions set in the problem). Whether an exaraination is open- or closed-reference, we sn, 4d, to the extent possible, be testing problem solving or decision making, for at .t is more complex level of thought, we more closely approximate the job and achie.e a valid assessment. i Memory types - Understanding how memory operates is intenelated to understanding why an open-reference question is preferable. Obviously, all that we know or do involves memory. Operationally, however, we can look at memory as falling into two categories: simple memory complex memory Simple memory can be viewed as the recall or recognition of simple bits and chunks of information. Simple memory may still be involved even when the volume of information increases, i.e., the amount of information is large but the process is basically simple memorization of more bits of information. Visualize the type of memory required to memorize five letters of the alphabet versus 26 letters, or the recitation of a short vice a long poem, procedure, and so forth. This memorization process does not involve analysis, integration of facts, or problem solving. Rather the process requires repetition, practice, and rehearsal. The difference lies in the amount of information to be recalled and not in the level of mental processing. NUREG-1021, Revision 8 14 of 27 G1

1 I ES-602 7 Attachment 1 By contrast, complex memory, as the term suggests, involves a higher level of cognitive processing.~ The bits and chunks of information must now be combined or integrated to create something new, solve a problem, predict a response, or

            . make a decision. Therefore, both the amount of information and what is to be done with it makes the cognitive mental processes complex. Naturally, too, some questions will involve greater complexity than others, but the mental processes will be the same: integrating bits of information, combining them, sorting through and distinguishing the relevant from the irrelevant to arrive at an answer to the question. This is the essence of an analysis / synthesis process.

As stated earlier, the NRC should evaluate candidates at this complex level, for this level of thought processing most closely approximates that needed on the job. The complex, problem solving level subsumes knowledge of the bits and l chunks of information frequently tested at the simple memory level. Therefore, by testing at the complex level, we are also implicitly testing at the simple memory level. As a prerequisite to solving the problem, the candidate recalls l and integrates these bits and chunks of information. Therefore, testing at the ; analysis level and is more efficient than testing at the simple memory level. 1 A Final Note - Undue emphasis is placed on the term "immediately" in the definition of a direct look-up. Speed of knowing where to locate a reference is irrelevant to direct look-up. The NRC expects candidates who have been trained to quickly locate the appropriate reference. The speed issue is relevant to s whether the stem of the question contains unnecessary cues to the candidate about where to find the reference. If the intent of the open-reference question is to assess whether the candidate knows where to find the information, then a knowledge where cue should not be in the stem. Part of the value of an open reference exam is to test the candidate's evaluative knowledge of where to look. If the stem provides unnecessary cues to the reference, a candidate can immediately go to the reference and a value of the open-reference test a lost. ; Now, speed in answering the question proper is a function of the level of difficulty and the thought processes / steps required to answer the question. Obviously, if the question is a direct look-up, then by definition, it assesses only simple memory and will be quickly and easily answered. This type of question should rever have been asked in the first place. References should be considered " tools" that operators use to solve problems. The correct use of these " tools" is what should be tested during the open-

             ' reference examination, not just the recall of facts and specifics. As stated eatiier, " direct look-up" questions should not be included in the examination; rather, questions should be structured to determine if operators can identify, locate, or select correct reference information to produce organized responses and satisfactory solutions to job-related problems and issues. An example of a look-up question, which should generally be avoided, follows:

A kV) 15 of 27 NUREG-1021, Revision 8

ES-602 8 Attachment 1 , Based on the ' Alarm Response Procedure' 12Z-040-3, what is the set-point of the high-high containment pressure alarm (PK25) on VB37

a. 10psig

b. 15 psig

c. 20psig

d. 25psig This question should be rejected because a candidate can easily find the set-point in the alarm response procedure (ARP). Some may argue that knowing how to look up this data in the ARP makes the item valid, however, no higher order cognitive skills requiring analysis or synthesis of information were required to determine the correct response. Avoid similar questions on precautions or prerequisites which are listed in procedures. A better question using reference material would be as follows:

Using the current plant conditions (assume ECCS and CS flow rates REMAIN CONSTANT), how much time is available before switch-over to containment recirculation?

a. 3.6 hours

b. 4.2 hours

c. 4.8 hours

d. 5.2 hours This is a " lock up' question in a sense, but it certainly requires gathering data from the control boards (e.g. ECCS flow, CS flow, and RWST level) and then identifying the correct emergency procedure and locating and selecting the correct graph to determine how much time is left before a specific level is reached in the RWST. It requires use of both the simulator and the plant i procedures as references. j i

Another appropriate question using facility referencea is: l l Following a LOCA, automatic actions have occurred as follows: j the reactor has tripped and is shut down 1 AFW has actuated and steam generator pressure is being controlled at 1005 psig, using steam dumps to the condenser Containtnent pressure has risen to 15 psig and no additional automatic actions have occurred. Which of the following Functional Recovery Procedures should be implementedIMMEDIATELY?

a. FR-C1

b. FR-Z1

c. FR-P1 1

d. FR-11 l NUREG-1021, Revision 8 16 of 27 9ll

__________________m

E l ES-602 9 Attachment 1 This question requires identifying which systems should have actuated based on the ESFAS set-points and which critical safety functions are compromised. The operator should refer to the functional recovery procedures to verify which critical safety functions have been compromised. Knowing where to look and what to look for are required to answer this question in a reasonable time. The item could also be used in the simulator section by requiring the operator to look at the control board in the " frozen" simulator to determine the plant condi-tions and deduce what critical safety functions were not met. Naturally, the more ) integration and evaluation required, the more time must be given to answer the question. Another question that makes effective use of reference material: i While operating at 100 percent power, VCT and Pressurizer alarms and indications show decreasing pressurizer level with two charging pumps operating. Also, the blow-down and main steam radiation monitors have alarmed. While following the appropriate Abnormaland Emergency , procedures, you, as the Shift Supervisor must evaluate the existing l conditions. What emergency classification wouldyou declare on the basis of this information? I

a. Notification of UnusualEvent \

 \                      b.      Aled                                                                i

c. Site Area Emergency

d. GeneralEmergency This question requires the operator to consult references to classify an event. It requires analyzing the situation, finding the correct part of the EPIPs, and selecting the appropriate classification.

f. Difficulty Level Versus Discriminatory Value Test developers sometimes believe, erroneously, that open-reference questions should be more difficult to compensate for the operators' access to reference material. Frequently, this increased difficulty is in the form of requiring knowledge of more obscure or otherwise unnecessary information. Both open-and closed-reference questions should have the same standard of difficulty; that is, difficulty should be based on the job demands and responsibilities of operators. A question should be constructed so that it effectively discriminates a competent operator from one who is not. A high K/A value should not be confused with the difficulty or discriminating ability of a question.

sD 17 of 27 NUREG-1021, Revision 8

I ES-602 10 Attachment 1 Operators take considerably longer to answer open-reference questions than i closed. (Weaker operators especially have been found to spend an inordinate amount of time consulting references versus writing responses). Provide the operators ample time to complete the examination, although not so much time that less than-competent operators have the opportunity to locate answers without prior familiarization with the topic. The following guidelines should be used to determine the appropriate length of the examination: (1) A competent operator should complete the combination of Section A and Section B in three hours. Give the operators an appropriate amount of time to review Sec.tiens A and B based on the number of questions assigned to that section. For example, if Section A has 15 questions and is validated for 45 minutes, operators should be allowed 15 minutes for review. Likewise, if 20 questions are selected for Section B and time validated for 90 minutes,30 minutes should be allowed for review. The time allocated to review Sections A and B must be included in the three hour time limit. (2) Questions should be developed so that Sections A and B each have approximately 15 to 20 points, for a total test value of 30 to 40 points. The examination sample plan should be used to determine the exact number of questions to be asked in each section. As noted in Appendix B, multiple-choice questions are preferred, but other formats are acceptable. No question will be worth more than two points. l I (3) in an open-reference examination every answer need not require the ! operator to use a reference. The individual developing the questions ! should make a reasonable estimate of the time required to answer each j question and identify any references used to obtain a response. Whether and to what extent references are needed affect what constitutes a reasonable amount of time to develop a response. For ] example, if the static scenario is set up for an abnormal plant transient that requires relatively rapid operator analysis or response, then the time allowed to respond to the question should be similar to that required to react to the transient. The NRC does not expect the operator to answer the question as quickly as he would react in the plant, but does expect that an operator would consult few references. Conversely, questions involving scenarios for which an operator would i have time to consult many references would allow similar time to develop l a response to the question. NUREG-1021, Revision 8 18 of 27 l

ES-602 11 Attachment 1 v (4) Each proposed examination is expected to be time validated. The best method would result in the examination being taken in near-test conditions by a representative cross-section of plant operators. Then, by taking the average of the time it took for each individual to answer each question, a reasonable time may be established for the test. However, if a large deviation occurs among test takers on particular questions, they should be asked why they took either an excessive or relatively short amount of time to answer the question from that anticipated. Responses i may lead to eliminating certain operators' times from the averaging j process and, thereby, eliminate anomalies associated with the individual vice the test item. However, logistics dictate that sometimes only one or a few individuals can participate in validating the time to complete the test. In any case, the results need to be evaluated carefully for any unanticipated deviations from the amount of time anticipated to complete each item.

Facility managers responsible for validating the examination are expected l to validate the time for each question similarly. When performing time validation of the examination, these expectations should be made clear to the facility representatives validating the examination so that a reasonable estimate can be obtained.

p. h. Correct Mode of Measurement No matter how high their importance ratings or operational relevance, certain operator knowledges, skills, and abilities are not amenable to written testing, as in the following example:

Arrange the major steps in the proper sequence to start, parallel, and load DG-2: use govemor control to increase DG-2 KW raise DG speed to 900 RPM match voltage with bus IA2 voltage close breaker 1AD2 Despite its operational orientation, the underlying skill addressed in this test item would be better assessed by having the operator simulate or perform the steps during either the simulator or walk through portions of the operating examination.

3. Specific Guidelines for Section A. " Plant and Control Systems" The following guidelines are specific to the Plant and Control Systems section of the written examination as performed on a static simulator.

These guidelines are divided into two sections, namely " Question Development" and

         " Simulator Setup."

O 19 of 27 NUREG-1021, Revision 8

E ES-602 12 Attachment 1 l

a. Question Development To ensure that the operators' knowledge of systems and integrated plant response is adequately evaluated, Section A of the written examination should incorporate the behavior of systems and controls in normal, abnormal, and emergency plant conditions. Questions should require the operators, to the i extent possible, to refer to control room indications in formulating their responses, as in the following example:

Which one of the following describes the location of the steam break?

a. inside containment, upstream of the steam line flow transmitters

b. inside containment, downstream of the steam line flow transmitters

c. outside containment, between 'C' MSIV and "C" main steam line check valve

d. outside containment, between 'C' MSIV and 'C' main steam line containmentpenetration The scenario used should put the plant at some point in a major plant transient (e.g., LOCA, SGTR, loss of all AC) with several passive or active failures incorporated. However, the number of malfunctions or failures included in the scenario should be limited. No more than four minor failures should be used (e.g., failure of a safety-related pump to start, failed pressurizer pressure indication, nuclear instrumentation failure). Four will provide sufficient effects to test a wide range of objectives. Such a scenario would provide sufficient visual cues to develop a good percentage (at least 50 percent) of questions directly related to the existing plant conditions.

Questions may be used that do not relate to the transient but use the simulator as a frame of reference only, provided the operators are aware of this lack of relationship to the transient. Carefully ensure that multiple questions stemming from one event do not give each other away. The operator should be able to understand and correctly answer each question, based only on the information given in the question, rather than on the answer to a previous question. Use of plant diagnostic questions for which the examinee attempts to determine i what transient has occurred are general!y not suitable, given the purpose of this I section of the examination. Having the operator attempt to identify what took l place may limit the number of questions you may ask about the transient. i Indicate what symptoms or events have occurred, which procedure has been implemented, and the point in the procedure that was reached at the time the simulator was " frozen." l NUREG-1021, Revision 8 20 of 27 O 1 1

1 l t 1 (') ES-602 13 Attachment 1 h The operator's response should either determine the root cause of the actual system or component failure, or by using "what if" questions, propose a future event and ask for the expected response.

b. Simulator Setup Before the test, advance the simulator recorders to provide clean readings and ,

check the recorders for proper operation. Check all indications (e.g., bulbs, l meters, manual loader indications) to ensure they are in proper working order. I When the simulator has been frozen, secure the chart recorder drive power, if necessary. Before adm8 .:stering the test, verify the simulator indications to ensure they are l what is expected to arrive at the correct answer. Freeze any "first-cut" annunciators that would normally blink to announce first-out conditions and provide them to the operators. If a transient is stabilized by use of plant procedures, note the step at which the ; simulator is frozen and record this information on the simulator operations l summary sheet. Give the examinees, as necessary, the progress of the procedure step in effect. V 4. Ideas for Open Reference Formats Table 2 provides a list of sample formats to assist individuals who are developing performance based, open-reference questions. Table 3 provides additiona; guidance on the process for developing open-reference questions. t V 21 of 27 NUREG-1021, Revision 8

ES-602 14 Attachment 1 EXAMPLES OF DIFFERENT TYPES OF OUESTIONS Table 1

1. Memory level questions are not to be used on open reference examinations.

2. Comorehension level questions would require the operator to demonstrate an understanding of a knowledge without necessarily relating it to other material, or fully comprehending it in depth.

A spurious safety injection (SI) signal resulted in HHSI flow to the loop cold legs when the plant was in mode 4. After completing corrective actions for the inadvertent SIinitiation, you must

a. stroke test the coldleg motor-operated stop valves within 24 hours.

b. test the cold leg injection check valves for leakage within 48 hours.

c. stroke test the cold leg motor-operated stop valves before entering mode 3.

d. test the cold leg injection valves for leakage before entering mode 2.

3. Analysis. synthesis, and aoolication level questions require higher-order cognitive thought processes,

a. Acolication questions may require the operator to apply the knowledge to various concrete situations.

Given the following conditions: 1 both CFPTs tripped k CA automatically started \ CA valves reset to control steam generator water level j CA suction pressure decreases to seven psig ) Which ONE of the following describes CA pump response for the given conditions?

a. the pump suction will automatically shift to RN

b. the pump suction will automatically shift to UST

c. the pump will trip when suction pressure decreases to 5 psig

d. the pump will trip after a 6-second delay NUREG-1021, Revision 8 22 of 27 Gi l

                                                                                                 --________________J

F-' l O ES-602 15 Attachment 1 2 Table 1 l

b. Analysis questions require the operator to mentally integrate a number of conditions, analyze their interrelationships, sort through and discriminate among distractors, and finally choose the correct answer.

Which answer below correctly indicates the posting required for a room using the results of the following radiologicalsurvey? SURVEY RESULTS: 1 AIRBORNE ACTIVITY: 6.34 E-9 uci/cc (Co-60) FLOOR SMEAR: Beta-610 dpm/cm "; Alpha-4 dpm/cm ' EQUIPMENT SMEAR: Beta-1800 dpm/cm ": Alpha-16 dpm/cm ' GENERAL RADIATION LEVEL: 110 mr/hr

a. Radiation Area, Airbome Area and Full Anti-Cs

b. High Radiation Area, Airbome Area and Full Anti-Cs

c. High Radiation Area, Full Anti-Cs

d. Locked High Radiation Area, Airbome Area, Double Anti-Cs '

c. Problem solvino questions require putting together elements to demonstrate an understanding of the underlying knowledge.

The plant is operating at 100 percentpower when a LOCA occurs. The reactor trips automatically, but fast transfer fails and buses 1A1 and 1A2 become de-energized. PPLS and CPHS initiate and all equipment operates as designed. Which ONE of the following is the expected system response?

a. OPLS initiates load shed and starts both emergency diesel generators

b. OPLS does NOTactuate; the emergency diesel generators start and re-energize buses 1A1 and 1A2

c. OPLS does NOTactuate; the emergency diesel generators do NOT start and safeguards motors are started by the sequencers

d. OPLS does NOT actuate; the emergency diesel generators run at \

idle speed and safeguards motors are started by the sequencers l l 23 of 27 NUREG-1021, Revision 8 '

l ES-602 _ 16 Attachment 1 EXAMPLE FORMATG j OR OPEN-REFERENCE QUEST!ONS Table 2

1. Given Plant / System / Component Condition /Problern I

diagnose cause of the problem l identify location of problem j predict effect on plant / system / component l identify precipitating events / actions classify / indicate if conditions meet specified criteria indicate & utilize proper procedures / references identify appropriate recuperative actions

2. Given Plant Conditions and Operator Actions / Procedural Steps implemented indicate purpose / reason behind taking these actions determine if the correct actions were taken given available cues indicate what further actions are required to achieve a desired effect

3. Given a Proposed or Hypothetical Course of Action / Recommendation determine the appropriateness or acceptability predict expected plant / system / component response predict effect on other systems / components

4. Given Data Regarding Plant Conditions or Parameters compute or determine status or change in other parameters utili7e charts, curves, graphs, etc., to perform calculations or estimations NUREG-1021, Revision 8 24 of 27

O ES-602 17 Attachment 1 DEVELOPING OPEN-REFERENCE TEST ITEMS Table 3 Here are the decision steps and mental model for developing analysis-level open-reference questions:

1. Determine the purpose of the test. Do you want to test knowledge where and knowledge what?

2. Determine the information needed to respond to the question, is the volume and kind of information such that you would not normally expect candidates to recall the information from memory to answer the question?

3. If the answer is yes to both of these questions, develop an open-reference question.

4. Construct the question as two tiers:

Tier / Purpose Process Criteria Outcome

1. Knowledge where Evaluate reference Avoid clues Locate ref.

sources in stem sources

2. Knowledge Integrate multiple Info. volume Identify ,

IO what/how variables, events and detail high (not in memory) correct answer j l OUESTION STEM I bits, chunks of stem information (conditions, set points, components, etc.) l l l Mental Processes Intearative Processes 1

Analyzing a. Answer
Integrate mental i
Sorting b. Distractor processes with !
Eliminating < > c. Distractor < > stem information,
Differentiating d. Distractor reference data,
Evaluating and distractors l

Determine Answer 25 of 27 NUREG-1021, Revision 8 j

l ES-602 NRC Checklist for Form ES-602-1 j Open-Reference Test items . Test item Level

1. Does each test itern have a documented link to important operator tasks, K/As, and/or f acility learning objectives? i i

2. Is each question operationally oriented (i.e., is there a correlation between job i demands and test demands)?

3. Is the question at least at the comprehension-level of knowledge?

4. Is the context of the questions realistic and free of window dressing and !

backwards logic? l S. Does the item require an appropriate use of references (i.e., use of analysis skills or synthesis of information either to discem what procedures were applicable or to consult the procedures to obtain the answer)?

6. Is the question a " direct look-up" question, or does one question on the examination compromise another? A " direct look-up question" is defined as a question that only requires the examinee to recall where to find the answer.

7. Does the question possess a high K/A importance factor (3.0 or greater) for the job position?

8. Does the question discriminate a competent operator from one who is not?

9. Is the question appropriate for the written examination and the selected format (e.g., short answer or multiple choice)?

10. Do questions in Section A take advantage of the simulator control room setting?

11. Does any question have the potential of being a " double-jeopardy" question?

12. Is the question clear, precise, and easy to read and understand?

13. Is there only one correct answer to the question?

14. Does the question pose situations and problems other than those presented during training?

15. Does the question have a reasonable estimated response time?

NUREG-1021, Revision 8 26 of 27

( ES-602 Written Requalification Examination Form ES-602-2 (, Cover Sheet U. S. Nuclear Regulatory Commission Written Requalification Examination i Operator Information Name: Date: Region: I /11/ lli / IV Facility / Unit: Reactor Type: W / CE / BW / GE Start Time: Stop Time: Instructions Use the answer sheets provided to document your answers. Staple this cover sheet on top of the answer sheets. Points for each question are indicated in parentheses after the i question. The passing grade is 80.00 percent. You have a total of three hours to complete both sections of the examination. U Operator Certification All work done on this examination is my own. I have neither given nor received aid. Operator's Signature Results l l Test Value (Points) Section A: Section B: TOTt. Operator's Score (Points) Section A: Section B: , TOTAL: Operator's Grade (Combined) Percent b) 27 of 27 NUREG-1021, Revision 8

g 1 l 1 O O Y.

l ES-603 ( REQUAliFICATION WALK-THROUGH EXAMINATIONS A. PURPOSE l NRC examiners, working with facility evaluators, follow this standard to administer walk-through requalification examinations as authorized by Title 10, Section 55.59(a)(2)(iii), of the Code of Federal Regulations (10 CFR 55.59(a)(2)(iii)). The walk-through examination is an effective tool for evaluating the ability of a licensed operator to manipulate system components and controls, interpret references, use administrative procedures, and demonstrate knowledge of componsnt locations. B. SCOPE This standard provides specific guidance and requirements to NRC examiners for preparing, reviewing, and administering walk-through requalification examinations. In the walk-through examination, each operator performs five job performance measures (JPMs). Each operator's examination is designed to test the operator on plant systems that are important to the safe operation of the reactor. NRC examiners and facility evaluators jointly approve the JPMs for each examination. Each JPM consists of several steps, one or more of them is designated '

  " critical." An operator has to properly complete each critical step to pass the JPM.

1 The examination team will agree on five JPMs so that a minimum of two are conducted in the simulator (or the control room) and at least two are conducted in the plant. When operators ,Q perform JPMs in the plant or the control room, they will be cautioned not to manipulate the V reactor controls. To successfully complete those JPMs, operators will demonstrate to the examiners the steps or actions they would take to complete the task. To the maximum extent practical, control room JPMs will be conducted using the simulator. C. JPM DEVELOPMENT

1. Facility Exam Team Members' Responsibilities

a. The facility licensee is expected to identify the plant systems that are critical to protecting the public health and safety. The systems that are selected for the examination should meet the following criteria:

systems covered during the current requalification cycle; the facility's sample plan (see ES-601, Attachment 3) should identify the systems and appropriate learning objectives new or recently modified systems systems that are the subject of recent facility licensee event reports (LERs) or vendor notices 1 of 7 NUREG-1021, Revision 8

ES-603 probacilistic risk assessment (PRA) identified risk-important systems, components, and operator actions' for plant or vendor generic systems systems that are the subject of NRC Information Notices systems that are important to safety during low power or shutdown operations

b. For those systems identified as being important to safety, the facility representatives are expected to review the job task analysis (JTA), learning objectives, and NUREG-1122 or -1123, the " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized [or Boiling] Water Reactors." The facility representative should highlight for use as JPMs the tasks, abilities, and learning objectives that fulfill the following criteria:

apply to the facility are at the appropriate level for the operator being examined (reactor operator (RO) is responsible for auxiliary operator (AO)/RO tasks, and the senior reactor operator (SRO) is responsible for AO/RO/SRO tasks) have a K/A rating of 3.0 or higher (Tasks and abilities may be selected for use that have ratings below 3.0 if proper facility justification exists for these ratings. ) If a facility-specific knowledge and ability (K/A) is used in lieu of NUREG-1122 or

                     -1123, the importance ratings must be based on protecting the public health and safety.

c. Many tasks that are important to safety are unique to a specific plant and are not referenced in NUREG-1122 or -1123. Each facility is expected to maintain a list of these plant-specific tasks and develop JPMs that test the operators' knowledge and ability in these areas. The facility is responsible for ensuring that the tasks are appropriate to the applicable license level and have a safety importance rating of at least 3.0, before submitting them to the NRC for review.

l t

d. JPMs should meet the guidelines of Appendix C and Form ES-C-2, " Job l Performance Measure Quality Checklist." The JPMs should indicate which steps l are " critical" to successful completion of the task. Critical steps are those which j when not performed in proper sequence, not performed at the proper time, or not !

performed correctly will prevent the system from functioning properly or preclude i the successful completion of the task. Form ES-C-1," Job Performance Measure l l

    ' Chapter 13 of NUREG-1560. Individual Plant Examination Program: Perspectives on Reactor safety and Plant                      I Performance, identifies a number of important human actions that may oe appropriate for the operating test. In determining important opetator actions, do not overlook actions that are relied upon or result in specific events being driven to low risir contribution. This wi8 helpidentify those human actions, assumed to be very reliable, that might otherwise not show up on a list of  ,

risk deminant actions. NUREG-1021, Revision 8 2 of 7 l i I l l l l

l ES-603 Workshect," or an equivalent facility form should bc used to construct and format the JPMs. i The majority of the JPMs selected for the examination will cover topics from the most recent requalification training cycle, in addition, the facility is expected to create at least ten new JPMs each year until they have a JPM bank that is representative of Sections C.1.a and C.1.b of this standard. The NRC anticipates that the number of JPMs in a facility's bank will be approximately 125 to 150; however, the exact number will depend on the facility's JTA. New JPMs should generally be based on recent requalification training, industry events, facility changes, and tasks for safety-significant systems.

e. The NRC expects the facility to develop " time-critical" JPMs to evaluate time critical-tasks identified in the facility JTA for each licensed position. To facilitate the selection of time critical JPMs for the requalification examination, the facility licensee is expected to uniquely identify these JPMs. To successfully complete a time-critical JPM, the operator must perform the " time-critical" steps within a pre-specified time period, in addition to successfully performing all of the critical steps that are not time-critical. The time period identified in the time-critical JPM should be based on a regulatory requirement or a facility commitment with the NRC.

f. The facility is expected to develop "altemate-path JPMs" and include them in their JPM bank. To facilitate the selection of alternate-path JPMs for the requalification examination, the facility licensee is expected to uniquely identify these JPMs. Appendix C gives guidance on the development of these JPMs.

2. NRC Examination Team Members' Regoonsibilities

a. The NRC team will review and approve the JPMs selected by the facility. The majority of the JPMs selected should be based on the systems covered during the most recent requalification cycle. However, the facility should also select JPMs in systems that are important to safety regardless of when they were reviewed in requalification training. NRC examiners will review the JPMs submitted by the facility to ensure that 20 percent of the JPMs selected were not included in the topics in the most recent training cycle, as this examination is intended to sample skills and abilities that operators should always be able to display. In general, examiners should select systems in Groups I and 11 of the appropriate written examination model in ES-401, with Group I comprising at least 50 percent of the selected systems.

b. The NRC staff will discuss with the facility representatives the JPMs selected that are not in NUREG-1122 or -1123 to ensure the system or task meets the site-specific importance criteria. Any modifications to the selection of JPMs will also be discussed with the facility representatives. The NRC may substitute up to 20 percent of the facility proposed JPMs with NRC-developed JPMs. The NRC will give facility representatives sufficient time to review any substituted p JPMs.

k 3 of 7 NUREG-1021, Revision 8 I L- _ . _ _ _ _ _ _ _ . . _

ES-603

c. The chief examiner has the authority to decide the content of each exarnination set. NRC examiners should review the proposed JPMs using the criteria in Appendix C and the " Job Performance Measure Quality Checklist" (Form ES-C-2).

d. The chief examiner will ensure that a sufficient number of different JPMs are scheduled during the examination week to avoid compromising the examination.

e. The chief examiner will ensure that the time validation of each JPM is reasonable and will verify that each JPM is identified as " time-critical' or not ' time-critical."

D. EXAM ADMINISTRATION

1. Conductina JPM Walk-Throuah Examinations

a. The facility evaluator is responsible for conducting the walk-through examination while the NRC examiner observes. The NRC examiner and the facility evaluator may ask the operator questions to clarify his or her performance of the JPM after the JPM is completed. In most instances, the NRC examiner will ask the facility evaluator to question the operator about the appropriateness of an action or a response that does not follow the actions specified in the JPM.

b. The facility evaluator will brief the operator, using Parts A, C, and D cf Appendix E. If desired, the evaluator may brief the operators as a group befors starting ,

the walk-through examination.

c. Operators should not be informed of the expected completion time before commencing the JPM. Informing operators of the expected completion time may increase tension as they approach the time limit. However, the evaluator may inform the operator that a JPM is time critical, if it is normal practics to do so at that facility.

d. Time should be a!iotted during the operating test for evaluating the performance of five JPMs. ,

Each walk-through examination should last approximately two hours. This time includes the validated times associated with each JPM that is planned, plus any administrative tasks required to conduct the examination. Administrative tasks include the following examples- ; transit time to and from the plant site time spent complying with facility security and radiological administrative requirements (unless this is part of the JPM being performed) transit time within the plant after a JPM is completed to get to a location where the initiating cue for the next JPM is to be given NUREG-1021, Revision 8 4 of 7 y 1

p i h O ES-603 Note: The JPM sample size will be constrained to the requirements of this standard for NRC-conducted examinations. The facility may perform additional evaluation of its operators outside the time frame designated for the NRC l examination. However, any additional evaluation by the facility will not be ,

                  - factored into the final requalification evaluation of the operator by the NRC. The criteria for the determination of requalification program status remains the same.

e. JPMs that are directly related to the operators' job function are preferable, j particularly for SROs. For example, if an SRO will not perform an emergency l action level (EAL) classification during the dynamic simulator or written examinations, then the examination team may choose to have the operator perform one JPM that involves classifying an emergency.

f. The NRC examiner will ensure that the facility evaluator conducts an appropriate examination. Appendix C provides examples of good evaluation techniques to look for during the walk-through examination. If the NRC examiner observes improper evaluation techniques that may render the examination invalid, the NRC examiner will stop the walk-through and counsel the facility evaluator. If the facility evaluator continues to display poor evaluation techniques, then the NRC examiner will stop the examination and request that another facility evaluator continue the examination. If it is necessary, the NRC examiner may conduct the walk-through with the original facility evaluator observing and co-evaluating,

g. !f an evaluator believes that follow-up questioning is required and is not sure how to phrase the question, he or she should consult the NRC examiner. This will avcid inadvertent prompting of the operator and enhance communicelions between the facility evaluatos end the NRC examiner. i j

h. The examiner will document the operator's performance using the applicable l portions of a JPM worksheet, Form ES-C-1, or the facility equivalent for each i JPM. Document any questions asked to clarify the operator's performance. Also i fill out the'JPM summary matrix, Form ES-603-1, to maintain operators' scores !'

during the examination, document which JPM each operator performed, and to meet the requirements of ES-601, Section J.1.b.

i. After completing an oocrator's JPM set, the NRC and facility evaluators shall discuss and resolve any outstanding issues that may result in the operator failing the wa'k-through examination or any individual JPM. Many times a discussion of what was observed will serve to correct a difference of opinion. Unresolved diffe:ences should be brought to the attention of the chief examiner. j

2. _G_rgfina the Examination -

t

a. To pass the walk 4hrough examination, each operator has to successfully f complete at least four of five JPMs. To successfully complete a JPM, the operator has to complete all critical steps and satisfy the completion criteria T

g). specified in each JPM. (V 5 of 7 NUREG-1021, Revision 8

7.. ES-603

b. Each JPM is expected to be completed within the validated time period. For a I JPM that is not time-critical, the validated time may be exceeded if the facility evaluator and the NRC examiner agree that the operator is making acceptable progress toward completing the JPM.

For time-critical JPMs, the facility should identify a period that they consider to be I the absolute maximum time in which they would expect an operator to perform this task (e.g., locally opening reactor trip breakers on an ATWS or locally starting an AFW pump on a loss of all feedwater). An operator that fails to meet the time criteria will receive an unsatisfactory evaluation on that JPM. E. ATTACHMENTS / FORMS Form ES-603-1, "JPM Summary Matrix" I i

l 1 i Oll 1 l I NUREG-1021, Revision 8 6 of 7 9! l

  %)   1 ii :                         iI 8

3 n f 0 o 6- i .!I is i S v - E e - R I m r 1 o 2 F i I I 0 _ 1 G _ i E _ R _ U N I I ,I

                                   ,                                                     !                                            l s
                                                    !i i

x t r a M _

  ')
  \  ,

y r l 7

 ~       a                                                                                                                                   f o
 ~(L     m m

u 7 _ S _ il Ii iI M P J n it o _

                      >        ip r
                      >          c
                      >          s s          e e        D                                                                                                                    _

f m e . a i r _ _ N B _ i s r r _ o e - t a b . r e m p u _ N O

              -                M P                                                                                                                  _

J _ 3 0 . . . . . 6- 5. l'\U 0 2 3 4 _

                                                .        .   .     .      .      .    .        .               1 S                                    1        2   3     4      5     6     i        8          1      1    1     1       1     1 E             gI! l            !s i                                       li.                                  .h f                                                                                                                   li

(_) DYNAM;C SIMULATOR REQUA1.lFICATION EXAMINATIONS A. PURPOSE NRC examiners use this standard ir, preparing and administering dynamic simulator requalification operating tests in accordance with the provision,s of Title 10, Section 55.59(a)(2)(iii), of the Code of Fedem/ Regulations (10 CFR 55.59(a)(2)(iii)). By simulating actual plant operation, the dynamic simulator test provides a comprehendve evaluation of the integrated plant knowledge and skills required of operating crews. It is effective in evaluatira a crew's communication skills and team behavior and in identifying any areas in which the licensed operators should be retrained to improve their knowledge and abilities (K/As) in accordance with the provisions of the requalification program developed by the facility liconsee. B. SCOPE The dynamic simulator test consists of two scenarios. Each scenario is constructed to last approximately 45 to 60 minutes. The actua; time needed to complete the scenarios will depend upon the specific events within the scenarios but should allow the crew the time necessary to perform the actions required to respond to each event. To successfully complete this portion of the operating test the crew must demonstrate the ability to operate effectively as a team while completing a series of critical tasks (cts) that measure the crew's ability to safely operate the plant during normal, abnormal, and emergency situations. I g/ v The NRC examiners evaluate the performance of each crew, using standard competency rating scales. Each competency is rated according to the crew's ability to satisfactorily complete the tasks that have been designated as " critical" within that crew's scenario set. Critical means "necessary to place and maintain the reactor in a safe operational or shutdown condition." Each valid CT rnust meet the criteria specified in Section D of Appendix D. If the crew fails to correctly perform a CT, that failure would indicate a significant deficiency in the knowledge, skill, or ability of that crew to demonstrate team behavior and will be evaluated, using the behavioral anchors on the " Simulator Crew Evaluation Form," Form ES-604-2. The facility evaluators will evaluate the performance of the operators during the dynamic simulator test. Because the primary purpose of the dynamic simulator test is to evaluato crews, eact1 individual is not required to perform a specific number of cts or necessarily receive an individual evaluation by an NRC examiner. However, NRC examiners will follow up on significant individual performance deficiencies on cts observed during the simulator test in a manner and setting compatible with the deficiency. A significant performance deficiency is the omission of or the inability to complete a critical task, or the demonstration of a significant lack of knowledge or ability while performing a critical task. This follow-up evaluation will be graded as a component of the individual's operating test. To meet the requirements of 10 CFR 55.59(a)(2), it is the f acility licensee's responsibility to conduct its annual operator performance evaluations on the dynamic simulator in accordance with the requirements of its requalification program. The NRC-conducted operating test may be used by the facility licensee to meet this requirement if the conditions of 10 CFR 55.59(a)(2)(ii) are satisfied (i.e., every individual ) operating test includes a comprehensive sample of the items specified in 10 CFR55.45(a)). I

1 of 19 NUREG 1021, Revision 8

ES-604 ' i If an operator demonstrates significant performance deficiencies linked to the execution of cts dunng the dynamic simulator portion of the operating test, then the facility and NRC examination taarn members should discuss those deficiencies at the end of the dynamic . simulator test. If the operating crew performs satisfactorily and NRC examiners observe no significant indNidual performance deficiencies linked to cts, the individual would pass the dynamic simulator test. In the case of operators with significant deficiencies identified while performing cts, the facility evaluators and NRC examiners will decide whether the operator would pass or 1 fail by asking the operator follow-up questions about his or her performance to determine the extent of the knowledge or ability deficiency demonstrated. The number and scope of follow-up questions to be asked will be agreed to by the NRC examiners and facility evaluators and will be based on the individual's demonstrated knowledge or ability weakness identified during the performance of cts. The follow-up questions and individual's answers will be documented cr'd used, along with the individual's performance, as the basis for a pass or fail decision. Section l E.2 of this standard describes the method for evaluating and documenting individual performance. l 1 in the rare event that the only way to evaluate the scope and depth of the individual's performance deficiency is by conducting another scenario to gain additional information, then , the examination team (NRC and facility) will determine the content, critical tasks, operator ] actions, and crew position rotation necessary to complete the evaluation of the individual's j performance. Conducting another scenario is time consuming and may have an adverse effect l on the examination process. If an individual operator exhibits only minor deficiencies in l performance and completes the testing requirements of 10 CFR 55.59(a) satisfactorily, i remedial retraining and reevaluation will be conducted in accordance with the facility licenseo l requalification program. O. EXAMINATION DEVELOPMENT l Developing the NRC dynamic simulator requalification examination is a combined effort l between the facility representatives and the NRC examiners on the examination team. The responsibilities of the members of the examination team are outlined below.

1. Facility Team Member Responsibilities i

a. The facility licensee develops the dynamic simulator scenarios with identified cts that meet the guidance specified in Appendix D and Form ES-604-1,

                   " Simulator Scenario Review Checklist." The facility licensee submits each proposed dynamic simulator test to the chief examiner 45 days before the scheduled examination.

b. The tacility licensee is expected to provide a qualified simulator operatcr to assist in developing and administering the simulator examinations. The simulator operator must be available to support the examination team during the examination preparation week, normally 2 weeks before tne examination. The simulater operator will be expected to sign a security agreement at the time that NUREG-1021, Revision 8 2 of 19

w 7 , u

                                                                                      ..                                                      1 ti ;f-                                                                  5 j
             ~,                           o y                                                                                                                                       3 Hj%                            '

ES-604 3 >- L\

  ,       ,,                          the chief examiner determines that he or she has access to specialized
 .Y                                 ' knowledge of any part of the examination.

l 1 c. The scenarios should be based on the training that was conducted during the j roqualification cycle, recent industry events, licensee event reports (LERs), 1 emergency and abnormal procedufes, and design and procedural changes. The

scenarios should demonstrate the crew's ability to use facility procedures for preventing and mitigating accidents. , Some scenarios should be based on the dominant accident sequences (DAS) for the facility or actual events that have

u occurred at that or a similar facilidf DAS are those sequences which contribute significantly to the frequency of core damage as determined by the facdity licensee's probabilistic risk assessment (PRA) or individual plant examination (IPE). The PRA/IPE should also be used to identify risk-important opesator actions'. In' determining those actions, do not overlook actions that are relied y upon or result in specific events being driven to low risk contribution. This will help identify those human actions, assumed to be very reliable, that might otherwise not show up in a list of risk-dominant actions. J tt .The facility representatives on the examination team will be given the opportunity to review any modifications the NRC made to the scenarios. The representatives may recommend changes to events that are critical to plant { safety but must substantiate the reason for those changes. The examination { team has to agree on the validity and content of each scenario before the p3 examination. 1 f N e. The NRC encourages the utility to have their management discuss with the NRC l 1 any problems with examination complexity. Utility managers engaged in the examination review will be subject to signing a security agreement. Responsibility rests with the utility to reso!ve any issues befors administering the a examination. This review is to ensure that the final scenarios are (1) consistent ) with the facility's requalification requirements for operatora licensed at the facility, J (2) within the capability of the simulation facility, and (3) within the scope of the facility's procedures. This utility senior manager or representative should communicate any significant ; concems about scenario validity to the chief examiner. If adequate resolution as not reached, the concems should be brought to the attention of regional , managers and then,if necessary, to managers at the Office of Nuclear Reactor Regulation (NRR), to resolve these concerns.

2. N_B, Gleam MerrAar Responsibilities

a. At least two weeks before the preparation week, the chief examiner or a j i

_ . _ _ . . 1 I Chapter 13 of MUREG 1560, 'IndMdual Plant Examination Prograrn: Perspectives on Reat, tor Safoty and Plant l ' Pmformanco.* ide4fies a numoer et important human actions that may be appopriate for evaluation on the dynamic simulator operatirg test.

       \                                                                    3 of 19                           NUREG-1021, Revision 8 l

B

ES-604 designee completes a draft of Form ES-604-1, " Simulator Scenario Review Checklist," for each scenario proposed by the facility to use during the examination, along with any proposed changes to be validated during preparation week. During the review of each scenario that the facility selected for the examination, the chief examiner or designee will consider the quantitative and qualitative factors described in Appendix D, as summarized on Form ES-604-1

b. If the proposed scenarios require major changes to meet the guidance provided on Form ES-604-1, the chief examiner informs regional managers and determines the appropriate course of action to take. The NRC may revise the scenarios, as appropriate, or develop scenarios to add to the facility's scenarios, if required. All changes to the scenarios will be communicated to the appropriate facility representative in sufficient time prior to the preparation week to allow for scenario validation. Minor changes needed to ensure the scenario objectives are properly accomplished may be made during the preparation week. The NRC staff reviews the final scenarios with the facility's examination team representatives before the examination is administered. The NRC has the final authority to decido the content of the scenarios and determine whether a task is critical for evaluating the competency of the crew.

c. A key element of the examination team's tesolutx .)f concems with scope, depth, and complexity of simulator scenarios inv 9s the observation of the I proposed examination scenarios by a senior utility nanager (subject to signing I an appropriate examination security agreement) during examination preparation. !

This executive would, if necessary, raise specific concerns to appropriate NRC l regional management for resolution before the examination is administered. ! D. EXAMINATION ADMINISTRATION

1. Administrative Reouirements

a. A facility manager or representative with responsibdities for conducting plant operations (as a minimum, a manager at the first level above shift supervisor) should be present while the simulator examinations are administered. The NRC chief examiner is the principal point of contact between the facility manager and the NRC. The chief examiner or a designee is present during the administration of each dynamic simulator examination.

b. The examination team briefs the operating crews before the start of the simulator scenarios, using the information in Par +.s A, C, and E of Appendix E.

c. Crews should be given adequate time to respond to all planned and unplanned ever,ts. A scenario's contact time should be approximately 45 to 60 minutes.

I Contact time means the actual time the operators spend in tne scenario but does not include time spent on briefings, simulator setup, or investigating simulator performance problems. NUREG-1021, Revision 8 4 of 19 91

I [ ES-604 V d. Under no circumstance will the sequence of events and transients be modified l by any member of the exan:ination team during the scenario. If the scenario is l not properly administered due to a simulator operator error or an unexpected I simulator response, the examination team will confer immediately after the { scenario set to detennine if the crew has performed a sufficient number of I transients and events to justify an evaluation of the required competencies, if ] necessary, the examination team can run an additional scenario to ensure that i I the required competencies are covered.

e. Crew rotation practices shall be discussed and agreed to during the preparation week and any problems resolved before the administration of the operating test.

f. The members of the operating crew should maintain the sarne operating positions as during facility requalification evaluations. The crew rnembers should rotate between positions in the manner identical to the facility's rotation practices for evaluations specified in the facility's requalification program. l

g. Senior reactor operators (SROs) must be evaluated in at least one scenario in an j SRO-licensed crew position. More than two simulator scenarios may be required i to examino crews that consist of more than four SROs. l

2. Post-Scenario Activities

} 'f(/-n) a. If the NRC examiners and facility evaluators observa actions that are unclear during the simulator scenario, they should question the crew members as ' necessary to obtain complete documentation on the performance of events during the scenario. Questions should be factual and should clarify performance related to observations.

b. If an examiner observes an individual who demonstrates significant deficiencies performing a critical task, the NRC examinor and the facility evaluator will discuss those deficiencies at the completion of the scenario. If they determine that the operator's perforrnance deficiency could not be assessed due to a lack of information, the examination team has the option to conduct an additional scenario or a job performance measure (JPM) to obtain this information.

During the post-scenario discussion, the facility evaluator is expected to describe the operator's deficiencies to the NRC examiner and suggest a series of follow-up questions designed to idenbfy the cause of the deficiency. The NRC examiner will assess the facility evaluator's ability to diagnose the operator's deficiency and document it in the examination report, if applicable. The NRC examiner has the option to augment the follow-up questions proposed by the facility evaluator, if necessary. The examination team should minimize the time it takes to conduct this review of crew and individual performance to minimize the impact on the operators. However, it is the examination team's responsibility to ensure the review is thorough and complete p) 5of19 NUREG-10.21, Revision 8

ES-604 The facility evaluator will conduct an individual evaluation of the operator in accordance with Section E.2 of this standard. The NRC examiner has ttle option to ask additional follow-up questions.

c. Upon completing any follow-up questioning, NRC examiners and the facility evaluators dismiss the crew to await the next scenario and inform the crew that they may discuss the completed scenario among themselves.

d. The NRC examiners and facility evaluators meet separately to compare observations and determine if any cts were omitted or incorrectly performed by the crew.

e. The facility evaluators discuss the crew's performance with the NRC examiners after each scenario to clarify any performance deficiencies that have been noted.

The examination team determines if the as-run scenario has invalidated any predesignated cts or if any new cts should be designated for evaluating unpredicted events or actions taken by the crew during the scenario. The examination team revalidates the cts in each scenario, using the methodology contained in Appendix D.

f. After the crew cornpletes the last scenario, the NRC examiners and the facility evaluators independently complete a " Simulator Crew Evaluation Form," Form ES 604-2, as discussed in Section E. The facility evaluators also evaluate individual operator performance in accordance with their requalification program requirements and Section E2. The NRC examiners review the facility's evaluations of individual operator performance after completing each crew evaluation.

E. PERFORMANCE EVALUATIONS Two separate evaluations are conducted based on the information obtained during the dynamic simulator examination. First, a crew simulator evaluation is performed. Next, individual simulator performance is used by the examination tearr to determine whether follow-up questioning of the operator is necessary. The exarniriation team may conclude that, after observing the operator's perfonnance in the dynamic simulator and evaluating the response to follow-up questions, additional performance information about the operator must be obtained to make an individual evaluation. In this case, an additional scenario or JPM would be conducted. The individual follow-up would then be documented along with the individual's crew evaluation on Form ES-601-5. The operator is subject to failure based on a competency evaluation of the operator's performance on the dynamic simulator and the required follow-up evaluation,if deficient performance was exhibited by the operator executing a crew critical task.

1. Crew Simulator Evaluations After administering the dynamic simulator scenario set as escussed in Section D, the NRC examiners and facility evaluators independently evaluate the crew's performance NUREG-1021, Revision 8 6 of 19

s >

'}

ES-604 - by completing a copy of Form ES-604-2. The facility.is expected to provide its final crew J evaluations to the NRC examiners before the crew members retum to licensed duties or i the end of the examination week, whichever is sooner. Specific guidance for completing ) Form ES-604-2 is on the first page of the form. - The results of the crew eva!uations are factored into each individual's examination results and the facility requalification program evaluation. Members of a crew that receive an unsatisfactory crew evaluation are expected to receive remedial training from , the facility licensee and be reevaluated in accordance with the facility licensee's NRC- .l approved requalification program before retuming to licensed duties. Although operators are not required to take an NRC-conducted requalification examination for purposes of license renewal, those that fail to pas's (individually or as a member of a l crew) an examination conducted by the NRC must be reevaluated by the NRC before their license will be renewed. The level of NRC involvement during the reevaluation will be determined on a case-by-case basis (refer to Section F.1 of ES-601). i

NRC examiners will document the results of each operator's crew performance on Form ES-601-5 in the " Simulator Examination Results" section.

2. Individual Operatina Evaluations individual operating evaluations on the dynamic simulator examination and the resulting O remedial training are primarily the responsibility of the facility licensee- Unsatisfactory Q operator performance of a crew critical task will be followed up after the simulator scenario and documented on Form ES-601-5.

The facility evaluators are expected to document anu grade individual operator performance during the dynamic simulator examination in accordance with the requirements of the facilitylicenseo's requalification program. The NRC expects that the facility's grading methodology will identify operator deficiencies and that the facility evaluators will discuss those deficiencies with the NRC examiners during the meetings following the scenarios as described in Section D. The facility evaluators are expected to document deficiencies and remediate and retest the operators for the identified deficiencies in accordance with the facility licensee's requalification training program. The facility evaluators are expected to, at a minimum, identify any operator on the crew l who was directly responsible for the omi.ssion or incorrect performance of validated cts. ; 1 Individual fo! low-up is conducted if an operator has significant performance deficiencies linked to a CT. The NRC examiner will assist in the development and administration of follow.up questions specific to the deficiencies displayed by thu operator performing the CT as described in Section D.2.b of this standard. The exarnination team will determine 1 the number and scope of the follow-up questions that will be asked based on a review of the operator's deficiencies at the cornpletion of the scenario. The examination team has the option to gather additional information about an operator who displays performance deficiencies attemptir'g critical tasks by either running an additional scenario or by using JPMs, if the dynamic simulator examination and follow-up questioning are inconclu'sive. [b NUREG-1021, Revision 8 7 of 19

1 ES-604 Upon completion of the individual follow-up questions the NRC examiner will complete the evaluation using the appropriate competency grading worksheet from ES-303 (ES-303-3 for reactor operators and ES-303-4 for senior reactor operators), or a facility's equivalent form. It is only necessary to use those competency grading worksheets that apply to the knowledge or ability deficiency being evaluated. Normally, to determine if-an individual passes or fails, the examiner uses the weighted average of al! the rating factors on the form. Here, only those competencies that deal with the operator's performance deficiencies should be filled out, and should not be weighted. If the NRC examiner gives the operator a rating factor score of "1" in either of the following cases, the individual fails this portion of the examination: any two rating factors in any one competency any one rating factor in any one competency if, in the judgement of the examination team, the operator's performance deficiency jeopardizes the safety of the piant or has significant safety impact on the public (NRC management will make the final decision on all operator failures resulting from a single rating factor evaluation of "1.") When conducting the evaluation described herein, rating factor scores of "1" will not be assigned based solely on performance in the dynamic simulator. Follow-up questions are asked and the operator's responses are recorded to evaluate and document the knowledge or ability deficiency linked to the performance of a critical task. The NRC examiner will then apply the individual's responses to the questions asked to evaluate and justify individual performance deficiencies that warrant a rating factor score , of "1." The examiners will document and include with the completed Individual j Competency Grading Worksheets (Form ES 303-3 or ~4) the follow-up questions asked ) and the responses given by the operator. Written comments describing the operator's l performance and the as-run simulator scenario set are to be included with the results of l the operator's simulator examination. l The NRC examiner will document the pass or fail determination for each operator's l individual follow up on Form ES-601-5, " Individual Requalification Examination Report," l' in the "Si.mulator Examination Results" section under " Individual Follow-up." If an operator demonstrates no performance deficiencies and therefore does not require ) any additional follow-up questioning, regardless of whether the crew passes or fails the ! dynamic simulator examination, the NRC examiner will record an "N/A" for " Individual Follow-up" in the " Simulator Examination Results" section of Form ES-601-5. l F. ATTACHMENTS / FORMS Form ES-604-1, " Simulator Scenario Review Checklist" Form ES-604-2, " Simulator Crew Evaluation" NUREG-1021, Revision 8 8 of 19 91l

rt f n NY $8-604

Simulator Scenario Review Checklist Form EB-UG4-1

      ')  Note: Attach a separate copy of this form to each scenario reviewed. This form is used as guidance for the examination team as they conduct their review for the proposed scenarios.

SCENARIO IDENTIFIER: REVIEWER: Qgalitative Attributeg - _ 1. The scenario summary clearly states the objectives of the scenario, i _ 2. The initial conditions are realistic, in that some equipment and/or instrumentation may j be out of service, but it does not cue the crew into expected events. 4 _ 3. The scenario consists mostly of related events. , i 1 _ 4. Each event description consists of ; I the point in the scenario when it is to be initiated l

                             ~ the malfunction (s) that are entered to initiate the event the symptoms / cues that will be visible to the crew
                             ' the expected oporator actions (by shift position) the event termination point (O
   \. )

_ 5. No more than one non-mechanistic failure (e.g., pipe break) is incorporated into the scenario without a credible preceding incident such as a seismic event. { _ 6. The events are valid with regard to physics and thermodynamics. j l _ 7. Sequencing / timing of events is reasonable, and allows for the examination team to j obtain complete evaluation results commensurate with the scenario objectives, j _ 8. If time compression techniques are used, scenario summary clearly so indicates. Operators have sufficient time to carry out expected activities without undue time constraints. Cues are given. _ 9. The simulator modeling is not altered. _10. All crew competencies can be evaluated. l l _11. The scenario has been validated. j i _12. If the sampling plan indicates that the scenario was used for training during the requalification cycle, evaluate the need to modify or replace the scenario. I i (A t 1 9 of 19 NUREG-1021, Revision 8

ES-604 2 Form ES-604-1 SIMULATOR SCENARIO REVIEW CHECKLIST (CONTINUED) Note: The following criteria list scenario traits that are numerical in nature. A second set of numbers indicates a range to be met for a set of two scenarios. Therefore, to complete

this part of the review, the set of scenarios must be available. This page should be l completed once per scenario set.

I SCENARIO SET CONSISTS OF SCENARIO AND SCENARIO Quantitative Attributes I' _13. total malfunctions inserted: 4 to 8/10 to 14 _14. malfunctions that occur after EOP entry: 1 to 4/3 to 6

15. abnormal events: 1 to 2/2 to 3 i

_16. major transients: 1 to 2/2 to 3 _17. EOPs used beyond primary scram response EOP: 1 to 3/3 to 5 _18. EOP contingency procedures used: 0 to 3/1 to 3 _19. approximate scenario run time: 45 to 60 minutes (one scenario may approach 90 minutes) _20. EOP run time: 40 to 70 percent of scenario run time _21. crew critical tasks: 2 to 5/5 to 8

 ,,_. 22. technical specifications are exerci. sed during the test COMMENT & ,_

1 NUREG-1021, Revision 8 10 of 19

t l ES-604 Simulator Crew Evaluation Form Form ES-604-2 l The examination team should use this evaluation form during the dynamic simulator component of the requalification examination. The rating scales on this form are for evaluating the crew as ! a whole rather than the individual operators. Use the following instructions when rating team l performance on the simulator examination:

1. Review the rating scales before the simulator examination so that you are familiar with each competency to be evaluated.

I 2. Use the " Operator Actions" Form (ES-D 2), or an equivalent facility form to make notes during the examination, as described in Appendix D and ES-302.

3. . Complete this form immediately after the simulator examination. Evaluate the crew's performance on each applicable rating factor by comparing the actions of the crew q against the associated behavioral anchors and selecting the appropriate grade. The l tasks planned and performed during the crew's scenario set may not permit you to l evaluate every rating factor for every crew. Annotate those rating factors that are not i I

used in the evaluation. The examination team should pay particular attention to the completion of tasks that they identified as critical to plant safety. The crew may compensate for actions performed incorrectly by individual operators, as long as the critical task was completed satisfactorily. Other less significant deficiencies should also be accounted for in the O rating factor evaluations to provide a source of information for crew remedial training Q during subsequent requalification training.

4. Justify all rating factor grades of "1" and document each justification in the space for

              ' Comments" on the form. Rating factor grades of "1" must be linked to the performance of at least one critical task.

5. Complete the examination summary sheet, recording for each scenario, the scenario name (or identifier), and the critical tasks performed by the crew. Annotate whether the critical task was performed satisfactory or unsatisfactory. Complete the crew's overall evaluation using the criteria listed in the next paragraph. Space is provided for additional comments about the crew's performance.

6. The threshold for failing the simulator portion of the examination is to receive a I (behavioral anchor) score of "1" in either of the following: l

a. any two rating factors in any one competency

b. any one rating factor in any one competency if, in the judgement of the l examination team, the crew's performance deficiency Jeopardizes the safety of the plant or has significant safety impact on the public (NRC management will j make the final decision on all crew failures resulting from a single rating factor l evaluation of "1.")

l.

  \                                                    11 of 19                   NUREG-1021, Revision 8

ES-604 2 Form ES-604-2 SIMULATOR EXAMINATION

SUMMARY

SHEET Facility: Examination Date: OVERALL DYNAMIC SIMULATOR CREW EVALUATION: SAT or UNSAT Crew Members Docket No. Scenario #1 Scenario #2 Position Position

1. 55-2, 55-

3. 55-

4. 55-

5. 55-

6. 55-1 Scenario #1: [ Enter scenario descriptor]

Crew Critical Tasks SAT UNSAT

1. [ Enter critical task descriptor]

2. 3. 4. 5. Scenario #2: Crew Critical Tasks SAT UNSAT 1. 2. 3. l 4. 5. Comments: NUREG-1021, Revision 8 12 of 19

  . ES-604                                            3-                               Form ES-604-2 V

DIAGNOSIS OF EVENTS AND CONDITIONS BASED ON SIGNALS OR READINGS Did the crew-- - (a)' recognize off-normal trends and status? 3 2 1 Recognized status and Recognized the status and Did not recognize adverse trends quickly and trends at the time of, but status and trends, even - accurately, not before, exceeding after alarms and established limits. annunciators sounded. l (b) use information and reference material (prints, books, charts, emergency plan implementation procedures) to aid in diagnosing and classifying events and conditions? l 3 2 1 Made accurate diagnosis Committed minor errors in Failed to use, or misused, by using information and using or interpreting or misinterpreted i reference material information and reference information or reference correctly and in a timely material. material that resulted in manner. improper diagnosis. (c) correctly diagnose plant conditions based on control room indications? l 3 2 1 Performed timely and Committed minor errors or Made incorrect diagnosis, accurate diagnosis. had minor difficulties in which resulted in incorrect making diagnosis. manipulation of any safety , control. Grade for diagnosis of events and conditions based on signals and readings: SAT or UNSAT Comments: ' NUREG-1021, Revision 8 13 of 19

y ES-604 4 Form ES-604-2 UNDERSTANDING OF PLANT AND SYSTEM RESPONSES Did the crew-- (a) locate and interpret control room indicators correctly and efficiently to ascertain and verify the status / operation of plant systems? 3 2 1 Each crew member Some crew members The crew members made located and interpreted committed minor errors in serious omissions, delays, instruments accurately and locating or interpreting or errors in interpreting efficiently, instruments or displays. safety related parameters. Some crew members required assistance. (b) demonstrate an unrierstanding of the manner in which the plant, systems, and components operate, including setpoints, interlocks, and automatic actions? 3 2 1 Crew members The crew committed minor inadequate knowledge of demonstrated thorough errors because of safety system or understanding of how incomplete knowledge of component operation systems and components the operation of the resulted in serious operate. system or component's mistakes or in plant operation. Some crew degradation. members required assistance. (c) demonstrate an understanding of how their acti^ns (or inaction) affected systems and plant conditions? 3 2 1 All members unoerstood Actions or directives The crew appeared to act the effect that actions or indicated minor without knowledge of or directives had on the plant inaccuracies in with disregard for the and systems. understanding by effects on plant safety. individuals, but the crew corrected the actions. Grade on understanding of the response of plant and systems: SAT or UNSAT Comments: l NUREG-1021, Revision 8 14 of 19 9

ES-604 5 Form ES-604-2 n ADHERENCE TO AND USE OF PROCEDURES Did the crew- i (a) refer to the appropriate procedures in a timely manner? 3 2 1 l l The crew used procedures The crew committed minor The crew failed to correctly as required and knew what failures to refer to refer to procedure (s) when conditions were covered procedures without required, resulting in faulty by procedures and where prompting,'which affected safety system operation. to find them. the plant's status. (b) correctly implement procedures, including following procedural steps in correct sequence, abiding by cautions and limitations, selecting correct paths on decision blocks, and transitioning between procedures when required? 3 2 1 The crew followed the The crew misapplied The crew failed to follow procedural steps procedures in minor procedures correctly, accurately and in a timely instances but made which impeded recovery p manner, demonstrating a thorough understanding of corrections in sufficient time to avoid adverse from events or caused unnecessary degradation the procedural purposes effects, in the safety of the plant. and bases. (c) recognize EOP entry conditions and perform appropriate actions without the aid of references or other forms of assistance? 3 2 1 The crew recognized The crew had minor The crew failed to

   - plant conditions and               lapses or errors. Individual       accurately recognize          j implemented EOPs                   crew members needed                degraded plant                I consistently, accurately,          assistance from others to          condition (s) or execute and in a timely manner.            implement procedures.              efficient mitigating          I action (s), even with the     )

use of aids. Grade on adherence to and use of procedures: SAT or UNSAT 1 Comments: A v 15 of 19 NUREG-1021, Revision 8

ES-604 6 Form ES-604-2 ! CONTROL BOARD OPERATIONS Did the crew-(a) locate controls efficiently and accurately? 3 2 1 Individual operators One or more operators The crew failed to locate located controls and hesitated or had difficulty control (s), which Indicators without in locating controls. jeopardized system (s) hesitation. important to safety. (b) manipulate controls in an accurate and timely manner? 3 2 1 The crew manipulated The crew demonstrated The crew made mistakes plant controls smoothly minor shortcomings in manipulating control (s) and maintained manipulating controls, but that caused safety system parameters within recovered from errors transients and related specified bounds, without causing problems. problems. (c) take manual control of automatic functions, when appropriate? 3 2 1 All operators took control Some operators delayed The crew failed to j and smoothly operated or required prompting manually control automatic l automatic systems before overriding or systems important to { manually, without operating automatic safety, even when ample assistance, thereby functions, but avoided time and indications averting adverse events, plant transients where existed. possible. Grade on control board operations: SAT or UNSAT Comments: i i l NUREG-1021, Revision 8 16 of 19 O: J

t p L ES-604 7 Form ES-604-2 CREW OPERATIONS

Did the crew members-(a) maintain a command role? -

l 3 2 1 The crew took early in minor instances, the The crew failed to take remedial action when crew failed to take action timely action, which ' necessary. within a reasonable period resulted in the of time. deterioration of plant conditions. (b) provide timely, well planned directions to each other that facilitated their performance and demonstrated appropriate concern for the safety of the plant, staff, and public? 3 2 1 Supervisor's directives The supervisors,in minor The supervisor's allowed for safe and instances, gave orders directive (s) inhibited safe integrated performance by that were incorrect, trivial, crew performance. Crew all crew members. or difficult to implement. members had to explain I L 1 O why order (s) could not or should not be followed. (c) maintain control during the scenario with an appropriate amount of direction and guidance from the crew's supervisors? 3 2- 1 Crew members stayed Crew members had to Crew members had to involved without creating a solicit assistance from repeatedly request distraction, the crew supervisors or each other, guidance. The crew failed members anticipated each interfering with their ability to verify successful other's needs, and the to carry out critical accomplishment of orders. supervisors provided action (s). guidance when necessa.y. L CREW OPERATIONS CONTINUED ON NEXT PAGE i

17 of 19 NUREG-1021, Revision 8 l

1 i j ES-604 8 Form ES-604-2 CREW OPERATIONS (Continued) Did the crew members-- (d) use a team approach to problem solving and decision making by soliciting and incorporating relevant information from all crew member? 3 2 1 Crew members were At times, crew members The crew was not involved involved in the problem failed to get involved in the in making decision (s). solving process and the decision making process The crew was divided over decision making process when they should have, the scenario's progress for effective team decision detracting from the team and this behavior was making. oriented approach, counter-productive. Grade on crew operations: SAT or UNSAT Comments: 0 t NUREG-1021, Revision 8 O 18 of 19

ES-604 9 Form ES-604-2 .- r% COMMUNICATIONS Did the crew-(a) exchange complete and relevant information in a clear, accurate and attentive manner? 3 2 1 Crew members provided Crew communications Crew members did not relevant and accurate were generally complete inform each other of information to each other. and accurate, but abnormalindication(s) or sometimes needed action (s). Crew members prompting, or the crew were inattentive when failed to acknowledge the important information was completion of evolutions, requested. or to respond to information from others. (b) keep key personnel outside the control room informed of plant status? 3 2 1 Crew members provided Minor instances of needing Failed to provide needed (~ key personnel outside the control room with to be prompted for information; some information. ( accurate, relevant incomplete / inaccurate information throughout the information provided. scenarios. (c) ensure receipt of clear, easily understood communications from the crew and others? 3 2 1 Requests information/ Minor instances of failing Failed to request needed clarification when to require or acknowledge information, or inattentive necessary; understands information from others. when information is communications from provided; serious others, misunderstandings among crew members. Grade on communications: SAT or UNSAT Comments:

v 19 of 19 NUREG-1021, Revision 8

.)

i C ES-605 ( License Maintenance, License Renewal Applications, and Requests for Administrative Reviews and Hearings l A. PURPOSE This standard describes the requirements for maintaining an NRC operator's license and the procedures for processing license renewal applications, requests for administrative reviews and 1 hearings by licensed operators in connection with failures of NRC-conducted requalification I examinations, and denials of applications for license renewal.

l B. BACKGROUND l The renewal license application differs in some respects from the initial license application. The staff developed this standard to establish the procedures for processing operators' renewal applications and requests for administrative reviews and hearings regarding the denial of ; renewal applications as a result of failures on an NRC-conducted requalification examination. { C. LICENSE MAINTENANCE ! I i

1. Reaualification Trainina l A licensed operator must, in accordance with 10 CFR 55.53(h), complete a requalification training program as described by 10 CFR 55.59. The facility licensee O may request in writing that an operator temporarily suspend participation in the facility

() licensee's requalification training program. The NRC regional office may authorize the operator to temporarily suspend participation in the requalification training program if it finds that:

a. the operator will be reassigned to full-time, career-enhancing duties at another location, making it impractical to participate in the training program (e.g.,

assignment to the Institute of Nuclear Power Operations or a foreign interchange program; college attendance)

b. the duration of the assignment will not excced 24 months (If the assignment extends beyond the date of license expiration, the operator may apply for timely license renewal in accordance with 10 CFR 55.55(b) and 10 CFR 55.57(a).)

c. the facility licensee's plan for ensuring the operator's qualifications and status is acceptable (i.e., the operator must be retrained, tested, reactivated, and medically fit for duty)

If the region approves the temporary suspension, the region will amend the operator's license to prohibit the performance of licensed duties during the reassignment. The region will also confirm its expectations regarding the operator's return to licensed duties and the need for the facility licensee to certify when the actions have been completed. The expectations will be documented in a letter to the facility licensee with a copy to the operator. ,CT 1 of 6 NURF.G-1021, Revision 8

ES-605 The regional office shall refer situations outside the specified parameters to the NRR operator licensing program office for evaluation.

2. Proficiency Watches in accordance with 10 CFR 55.53(e), licensed operators are required to maintain their proficiency by actively performing the functions of an operator or senior operator on at least seven 8-hour or five 12-hour shifts per calendar quarter. This requirement may be completed with a combination of complete S- and 12-hour shifts (in a position required by the plant's technical specifications) at sites having a mixed shift schedule, and ;

watches shall not be truncated when the minimum quarterly requirement (56 hours) is satisfied. Overtime may be credited if the overtime work is in a position required by the plant's technical specifications. Overtime as an extra " helper" after the official watch has been tumed over to another watchstander does not count toward proficiency time.

3. Medical Standards

a. If an operator is temporarily unable to meet medical standards but is expected to meet those standards again in the future, the facility licensee may administratively classify that operators license as " inactive" until the operator is once again certified to rneet all medical standards by the facility licensee. The facility licensee need not notify the NRC nor request a conditional license for the temporary disability provided the operator is administratively prevented from performing licensed duties during the period of his or her temporary disability. If the disability extends beyond the da'e of license expiration, the operator may apply for timely license renewal in accordance with 10 CFR 55.55(b) and 10 CFR 55.57(a). The facility licensee should document the nature of the operators temporary disability on the medical certificate and submit a revised certificate to the NRC after the physician determines that the operator meets the requirements of 10 CFR 55.33(a)(1). The NRC will not renew the operators license untilit finds that all of the conditions specified in 10 CFR 55.57(b) are satisfied,

b. If the facility licensee determines that an operators medical condition is permanentlydisqualifying in accordance with ANSI /ANS 3.4, " Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants," the facility licensee shall notify the NRC within 30 days of learning of the diagnosis (see 10 CFR 50.74 and 55.25).

D. LICENSE RENEWAL

1. An operator wishing to renew a license must comply with the requirements of 10 CFR 55.57(a) as follows:

a. The operator will complete NRC Form 398, including the operators experience under the current license, the approximate number of hours the operator spent on operating shifts, and the date and results of the applicant's most recent NUREG-1021, Revision 8 2 of 6

O)

   %                                                                                                                               ES-605 requalification examination. The senior management representative on site shall provide evidence that the operator has safely and competently discharged his or her license responsibilities and satisfactorily completed the facility's approved requalification program by checking item 19.c and signing in the designated space on Form 398.

b. The facility licensee must certify on NRC Form 396 that a physician has performed a medical examination within the previous two years as required by 10 CFR 55.21 and submit that form along with NRC Form 398.

c. The operator must submit NRC Forms 396 and 398 not less than 30 days before the expiration date of the license, in accordance with 10 CFR 55.55(b), if the operator files a proper application for renewal at least 30 days before the date of expiration, the license shall not expire until the application for renewal has been denied or a new license has been issued. If an operator is waiting to be given a reexamination after failing a requalification examination, the operator should stillmake timely application for license renewal under the provisions of 10 CFR 55.55(b). The NRC regional office may allow for transit time and accept a license renewal application received 25 days before the license expiration date, provided all [^

 \                                                signatures on NRC Forms 398 and 396 are dated before the 30-day timely renewal cutoff date. The submittal will not be considered timely if it is received less than 25 days before the date of license expiration unless positive evidence of receipt (e.g., postmark or docketing stamp) by the U.S. Postal Service or the NRC is available if the application is received less than 25 days before the date of license expiration and too late for processing in the regional office, the license shall expire on the expiration date. The regional office may issue a new license when it has completed processing the application.

d. After reviewing the renewal application, the NRC regional office may request the operator or facility to provide supplementalinformation. The supplemental information must be forwarded to the regional office within 20 days. If an applicant for renewalis asked to provide supplementalinformation and declines to provide it or if the regional office concludes, after reviewing any additional information supplied by the operator, that the application is still inadequate for license renewal, the regional office will notify the operator in writing that the renewal application is denied. The operator may exercise one of the following options within 20 days after the date of the proposed denial letter from the regional office: (1) Do nothing. The denial will become final 20 day,s after the date of issuance, and the regional office will inform the facility licensee and the operator in writing that.the license has been terminated. m 3 of 6 NUREG-1021, Revision 8 i r

                                                                                ~~mi?'N-

ES-605 (2) Request reconsideration of the application denial. Such a request should be sent in writinp to the Operator Licensing Program, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. Requests for informal reviews by the NRC shall list the items for which additional review is being requested and include documentation supporting the contentions made by the operator. The package containing the review request and supporting documentation must be mailed or delivered with!n 20 days of the date of the denial. (3) Request a hearing pursuant to 10 CFR 2.103(b)(2). The hearing request must be submitted in writing to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Associate General Counsel for Hearings, Enforcement, and Administration, Office of the General Counsel, at the same address.

2. Upon receipt of a renewal application, the NRC regional office will perform the following:

a. Review the application and issue the license renewal if it finds that the conditions in 10 CFR 55.57(b) are satisfied. There is no rninimum number of hours that the operator has to operate the facility in order to qualify for license renewal (i.e.,

inactive licenses are also renewable). However, the regional office should take the applicant's operating history into consideration as an additional piece of information if any of the requirements of 10 CFR 55.57(b) are not met.

b. If the renewal applicant does not meet the requirements of 10 CFR 55.57, the regional office shall inform the facility licensee of the deficiencies and request any supplemental information that might be required to make a reticensing decision. If, after evaluating the supplemental information, the regional office still concludes that the applicant does not meet the requirements for license renewal, it will issue a proposed denialletter to the operator (with a copy to the facility licensee),

c. If the operator requests informal reconsideration of the application denial or a hearing, the regional office will review the operator's request as directed by the NRR operator licensing program office.

The NRR operator licensing program office will inform the operator, in writing, of the outcome of the review. E. NRC-CONDUCTED REQUALIFICATION EXAMINATION RESULTS

1. Passina an NRC-Conducted Recualification Examination An operator who passed all portions of the requalification examination, including being a member of a cre'n that passed the dynamic simulator examination, will receive written notification from the NRC regional office.

NUREG-1021, Revision 8 4 of 6 l

b I' j i

   .                                                                                                1
     -[                                                                                     ES-605 i V       requalification examination. The senior management representative on site shall

{ l provide evidence that the operator has safely and competently discharged his or her license responsibilities and satisfactorily completed the facility's approved j requalification program by checking item 19.c and signing in the designated space on Form 398.

b. The facility licensee must certify on NRC Form 396 that a physician has performed a medical examination within the previous two years as required by 10 CFR 55.21 and submit that fcrm along with NRC Form 398.

l c. The operator must submit NRC Forms 396 and 398 not less than 30 days before the expiration date of the license. in accordance with 10 CFR 55.55(b), if the i operator files a proper application for renewal at least 30 days before the date of 1 expiration, the licensa shall not expire until the application for renewal has been denied or a new license has been issued. If an operator is waiting to be given a reexamination after tailing a requalification examination, the operator should stillmake timely application for license renewal , under the provisions of 10 CFR 55.55(b). ! The NRC regional office may allow for transit time and accept a license renewal application received 25 days before the license expiration date, provided all , signatures on NRC Forms 398 and 396 are dated before the 30-day timely ( renewal cutoff date. The submittal will not be considered timely if it is received ! less than 25 days before the date of license expiration unless positive evidence of receipt (e.g., postmark or docketing stamp) by the U.S. Postal Service or the NRC is available. If the application is received less 'han 25 days before the date of license expiration and too late for processing in the regional office, the license shall expire on the expiration date. The regional office may issue a new license when it has completed processing the application.

d. After reviewing the renewal application, the NRC regional office may request the operator or facility to provide supplemental information. The supplemental information must be forwarded to the regional office within 20 days.

If an applicant for renewal is asked to provide supplemental information and declines to provide it or if the regional office concludes, after reviewing any additional information supplied by the operator, that the application is still inadequate for license renewal, the regional office will notify the operator in writing that the renewal application is denied. The operator may exercise one of the following options within 20 days after the date of the proposed denialletter from the regional office: (1) Do nothing. The denial will become final 20 days after the date of issuance, and the regional office will inform the facility licensee and the operator in writing that the license has been terminated. i t [mD) 3 of 6 NUREG 1021, Revision 8

ES-605 (2) Request reconsideration of the application denial. Such a request should be sent in writin0 to the Operator Licensing Program, U.S. Nuclear Regulatory Corr

mission, Washington, D.C. 20555. Requests for informal reviews by the NRC sha;l list the items for which additional review is being requested and include documentation supporting the contentions made by the operator. The package containing the review request and supporting documentation must be mailed or delivered within 20 days of the date of the denial.

(3) Request a hearing pursuant to 10 CFR 2.103(b)(2). The hearing request must be submitted in writing to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Associate General Counsel for Hearings, Enforcement, and Administration, Office of the General Counsel, at the same address.

2. Upon receipt of a renewal application, the NRC regional office will perform the following:

a. Review the application and issue the license renewal if it finds that the conditions in 10 CFR 55.57(b) are satisfied. There is no minimum number of hours that the operator has to operate the facility in order to qualify for license renewal (i.e.,

inactive licenses are also renewable). However, the regional office should tal<e the applicant's operating history into consideration as an additional piece of information if any of the requirements of 10 CFR 55.57(b) are not met.

b. If the renewal applicant does not meet the requirements of 10 CFR 55.57, the regional office shall inform the facility licensee of the deficiencies and request any supplemental information that might be required to make a relicensing decision. If, after evaluating the supplementalinformation, the regional office still concludes that the applicant does not meet the requirements for license renewal, it will issue a proposed denial letter to the operator (with a copy to the facility licensee).

c. If the operator requests informal reconsideration of the application denial or a hearing, the regional office will review the operator's request as direct 6d by the NRR operator licensing program office.

The NRR operator licensing program office will inform the operator, in writing, of the outcome of the review. E. NRC-CONDUCTED REQUALIFICATION EXAMINATION RESULTS

1. Passino an NRC-Conducted Reoualification Examination An operator who passed all portions of the requalification examination, including being a member of a crew that passed the dynamic simulator examination, will receive written notification from the NRC regional office.

NUREG-1021, Revision 8 4 of 6 l

 /                                                                                             ES-605 e

2. Failin2 an NRC-Conducted Reaualification Examinati.qq l

a. The NRC regional office will notify the operator in writing of a failure on the requaliCcation examination. On receiving the failure notification, the operator ,

I can request an informal review of the portion of the examination that was failed. The request must be made as described (for reconsideration of application denials) in Section D.1.d (2) above.

b. If an operator fails any part of an NRC-conducted requalification examination, the facility licensee is expected to remove the operator from licensed duty and take corrective action consistent with the provisions of its requaliiication program before retuming the operator to licensed duty. If the facility licensee's I requalification program is unsatisfactory, refer to Section F.2 of ES-601 for a list of other recommended actions to be taken, including those actions the facility i licensee is expected to complete before attaining a " provisionally satisfactory" requalification program status.

c. Although the regulation (10 CFR 55.57(b)(2)(iv)) that required operators to pass an NRC-administered requalification examination as a prerequisite for license l renewal was deleted effective March 11,1994, the license of any operator who i failed to pass any NRC-conducted requalification examination will not be renewed without some level of NRC involvement in the retesting process. The i amount of NRC involvement may include conducting the retest in accordance O., with the applicable Examination Standard (s); inspecting the facility licensee in ;

accordance with Inspection Procedure (IP) 71001, " Licensed Operator ! Requalification Program Evaluation," as it retests the operator; or reviewing an examination prepared by the facility licensee. The NRC regional office, in consultation with the NRR operator licensing program office, will determine the appropriate level of involvement on a case-by-case basis depending on the quality of the facility licensee's program. As long as the operator subrnits a timely renewal application, the term of the license will continue until the renewal requirements are satisfied or the operator fails three NRC-conducted examinations as discussed in Section E.2.e.

d. The NRC will normally administer a second (first retake) examination approximately six months after issuing the first failure nctification in accordance with Section E.2.a of this standard. That examination will concentrate ori the areas in which the operator exhibited deficiencies.

e. The NRC will normally administer a third (second retake) examination approximately six months after issuing the second failure notification in accordance with Section E.2.a of this standard. The third examination will be a comprehensive requalification examination.

Regardless of the status of the facility licensee's requalification program,if an operator fails a third requalification examination, the NRC will thoroughty review the operator's examination performance and may conduct a complete review of O 5 of 6 NUREG-1021, Revision 8

i ES-605 the facility licensee's training program. The third failure may be grounds for suspending or revoking the operator's license. If an operator has an application pending for license renewal with the NRC at the time of a third requalification failure, that f ailure will provide the basis for denying the application. Notification of the operator will be handled on a case-by-case basis and coordinated through the NRR operator licensing program office. I l; O 4 i { l NUREG-1021, Revision 8 6 of 6

(o

  'k/)

ES-701 ADMINISTRNDON OF INITIAL EXAMINATIONS

                            'FOR SENIOR OPERATORS LIMITED TO FUEL HANDLING A.      . PURPOSE This standard provides specific instructions for the preparation, administration, grading, and documentation of initial examinations for senior operators who are limited to fuel handling (LSROs).

8. BACKGROUND Except as noted herein, the guidance in Examination Standaros (ES) 201,202,204,301,302, 303, 401, 402, 403, 501, and 502 for administering unrestricted initial licensing examinations at power reactors is also applicable to the LSRO exarnination. The generic fundamentals examination (GFE) program described in ES-205 does not apply to LSRO applicants.

C. RESPONSIBILITIES

1. Facility Licensee The facility licensee is responsible for the same activities specified in the unrestricted ES with the following exceptions and roodifications:

Q i

a. As an execption to ES-202, the facility licensee may request LSRO licenses that are valid for more than one site. The facility licensee shall provido documentation that deceribes the differences in the design, procedures, technical data, and administrative controls of the separate facilities for which the licerise is being sought.

b. The scope and centent of the written examination and operating test shall be as described in Sections D and E below,

c. The wdtten examinations shall be administered in accordance with the additional instructions in Section D.

2. NRC Hg,gional Offic_e.

The NRC regional office is responsible for the same activities specified in the unrestricted ES with the following exceptions and modifications:

a. The regional office should generally conduct the LSRO examinations during a time when the fuel hand!ing equipment will be available for the operating tests,

b. With the concurrence of the NRR operator licensing program office, the regional office may issue LSRO licenses that are valid for units at more than one site provided the units are manufactured by the same vendor and are of similar design. The applicant must pass an examination that addresses the differences in the design, procedures, technical data, and administrative controls of the

+ separate facilities for which the license is being sought. 1 of 8 NUREG-1021, Revision 8

l ES-701 { l

c. The scope and content of the written examination and operating test shall be as {

described in Sections D and E below. l l

     ' d.      The NRC regional office shall ensure that the examinations are administered and         !

I documented in accordance with the instructions in Sections D and E. D. WRITTEN EXAMINATION INSTRUCTIONS

1. Content and Preppration LSRO written examinations shall consist of one section containing 50 questions, and it should be constructed so that a competent applicant can complete the examination in i

2.5 hours. The applicants will be allowed three hours to complete and review the examination. Develop an examination model or sample plan using the appropriate knowledge and abilities (K/A) catalog: NUREG-1122 for pressurized water reactors (PWRs) or NUREG-1123 for boiling water reactors (BWRs). K/As from the following sections of the K/A catalog may be selected: appropriate generic knowledge and abilities, plant systems associated with fuel handling operations, refueling equipment, and emergency and abnormal plant evolutions related to refueling, if the facility licensee has completed a job task analysis (JTA) for fuel handlers, that analysis would provide an excellent source of information for developing the test outline and test items. The examination points should be distributed among the following subject areas according to the percentages (%) noted: Samole Subject Area 15% Reactor and fuel characteristics and physical aspects of core construction that are important to fuel handling or shutdown activities. For example, this could include a question on how a task performed during fuel handling activities affects the shutdown margin but should not include a question on power defects. Since LSRO applicants are not required to take the GFE, questions on basic reactor theory and thermodynamic topics that apply to fuel handling operations are also appropriate and may be selected from prior GFE examinations. 30 % System equipment and instruments that are important to plant safety and are located near or used during fuel handling activities or during altemate shutdown procedures. This includes the knowledge and use of radiation monitors, spent fuel pool cooling, and residual heat removal (RHR) systems and the technical specifications associated with those systems. 40% Normal, abnormal, emergency operating and administrative procedures related to fuel handling activities, core safety, and accident mitigation, including general facility events such as a loss of forced circulation or a release of radioactive effluent. The questions should evaluate the NUREG 1021, Revision 8 2 of 8 _ 1

w l (% ES-701 applicant's knowledge of the control room operator's response to events i only as it relates to fuel handling activities and the general response expected of employees. 15 % . Health physics and radiation protection for fuel handFng activities and general employee responsibilities. These questions may include administrative procedures associated with radiation protection. The actual percentages may vary from those listed above by up to five percent (plus or minus), if the examination will be used to license the app!icants at more than one ; facility, ensure that it adequately covers all cf the applicable units. An examination developed for the purpose of cross-qualifying a licensed LSRO at another similar facility may focus exclusively on the differences between the facilities. The LSRO written examination should otherwise meet all the guidelines and requirements for question construction, quality, and facility ruisws specified in ES-401 j and Appendix B. l Form ES-701-1 shall be used as an examination cover sheet, i

2. Administration The written examination for LJROs shall be administered in accordance with ES-402 I g with the following modifications:

a. The examination may be administered concurrently in the same room with full-scope, initial license examinations. The proctor should minimize any disturbance to those applicants taking the longer initial licensing examination.

b. The time for the applicant to complete the examrat:on is three hours.

E. OPERATING TEST INSTRUCTIONS j The LSRO operating test shall generally be prepared, administered, and documented in accordance with ES-301, ES-302, and ES-303. However, the sampling requirements will be less than those required for an SRO applicant. c The operating test shall be performance-based to the maximum extent possible. Therefore, the facility 'icensee should be encouraged to permit the actual use of equipment to handle dummy - fuel elements, assemblies, or modules during the operating test whenever feasible. This may require careful coordination with the facility licensee to establish a schedule and to make sure that a licensed SRO is available,if needed. When actual equipment is not available or inaccessible (e.g., because of high radiation), the examiner shall administer the test using walk-through methods near the actual equipment or by using mockup equipment. If the facility licenseo has a refueling machine simulator, the examiner should use it to the extent possible during the administration of Categories B and C of the operating test.

       . The operating test shall assess the applicant's ability to execute normal, abnormal, and 3 of 8                    NUREG-1021, Revision 8 1

fL

ES-701 , emergency procedures associated with fuel handling. Each applicant will be required to simulate or parform tasks related to fuel handling and to answer questions associated with the refueling equipment and associated systems. The operating test will also datermino if the applicant has the ability to supervise the operation of equipment and systems to safely conduct fuel handling operations. The applicant shall not be held accountable for duties that are performed exclusively by the control room staff or shift supervisor. The operating test should normally take between four and six hours, depending on whether or not refueling equipment is actually operated. The following additional guidelines clarify the expectations for Categories A, B, and C of the LSRO operating test.

1. Adminiqtrative Tooics (Cateaory A,)

Topic A.1, " Conduct of Operations

The subjects under this topic shall be evaluated as they pertain to refueling activities at that f acility. Some subjects (e.g., reactor plant startup requirements) may not be applicable, however, most can be adapted for use during the LSRO operating test.

Topic A.2, ' Equipment Control" These subjects alllend themselves to evaluating the required refueling maintenance and surveillanco actions that the LSRO should be able to supervise or perform. Topic A.3," Radiation Control' These subjects arc all applicable to refueling operations and should be evaluated on a sampling basis. Topic A.4, " Emergency Plan" This topic shall be evaluated to the extent that the applicant is required to respond to a declared event und the knowledge required of a radiation worker.

2. gbtstems (Categrn BJ Instead of distributing systems among the safety function groupings in the K/A catalog as discussed in ES-301, this category of the LSRO operating test shall evaluate the I applicant on 5 systems selected from Attachment 1 as follows: one from decay heat removal (DHR) or auxiliary (AUX), one from instrument and control (IC) or radiation monitoring (RM), and three from fuel handling equipment (FHE). No distinction between I control room and facility systems is required, because most, if not all, of the systems will be covered outside the control rocm.

In accordance with ES 301, each system shall be evaluated by simulating or actually performing a JPM whenever possible. If a system does not lend itself to the conduct of NUREG-1021, Revision 8 4 of 8

L l I i m 1 l ES-701 [V I I a JPM, then the responsible regional supentisor may authorize the use of prescripted { questions to evaluate two K/As associated with the system.. j The examiner may conduct part of the operating test in the control room so that those controls, instruments, and other materials or equipment related to fuel handiing (e.g., i procedures and diagrams) are available for reference. Although LSROs will not operate any systerns from the control room, they must be aware of the effects (e.g.., alarms) that . fuel handling operations will have in the control room. They must also be familiar with I ' the methods and requirements for communicating with the control room staff and shift supervisor. When documenting this category of the operating test on Fonn ES-303-1, disregard Category B.2 and anter the appropriate information under control room systems. If the j system was evaluated using prescripted questions and the applicant's understanding of I both prescripted K/A topics was satisfactory, enter an "S" in the "Evah;ation" column for ' tnat system on page 2 of Form ES-303-1. If the applicant's understanding of only one of the prescripted K/A topics was satisf actory, enter an "S" unless the knowledge or ability that was missed is of such safety significance that an unsatisfactory system grade is justified; then enter a "U" and explain the safety significance in the documentation. If i the applicant's understanding of both prescripted K/A topics is unsatisfactory, enter a l "U."

3. Intearats,i Plant (Refuelina Eauioment) 0.peration (Categoty QJ V

e'\ i To adequately evaluate the applicant in this area, prepare two ' discussion" scenarios to evaluate the applicant's competency in much the same way that it would be done using a simulation facility. Use Forms ES-D-1 and ES-D-2 to develop the scenario events and expected operator actions. Each scenario sha!!, at a minimum, simulate an equipment or instrument rnalfunction requiring operator actions and an event requiring the applicant to follow emergency response procedures associated with fuel handling activities. The malfunctions and events shall not duplicate activities tested under Category B of the l operating test. The scenarios should require the appUcant to demonstrate adequate ' knowledge and ability in areas that are important to fuel handling safety, including an understanding of the associated technical specifications. When documenting the applicant's pedormance, the examiner will substituto page 3 of Form ES-303-1 with Form ES-701-2, "LSRO Competency Grading Sheet," and evaluate the applicant on as many of the rating factors as possible. The examiner should evaluate the applicant on rating factor S.b only if actual fuel handling equipment ' manipulations were performed. Those rating factors against which the applicant is not evaluated will have their weight equally distributed among those that were usad in evaluating that competency. F. ATTACHMENTS / FORMS Attachment 1, "LSRO Operating Test Topics and Systems (BWR/PWR)" , Form ES-701-1, "LSRO Written Examination Cover Sheet" Form ES-701-2, "LSRO Competency Grading Sheet' (A) V NUREG 1021, Revision 8 5 of 8

I i

ES-701 L'SRO Operating Test Tipics Attachment 1 andyystems (BWR/PWR) _

1. Decay Heat Removel Systems residual heat removal shutdown cooling service water core coofi"O -

normal and emergency AC power reactor coolant system l 2., Auxiliary Systems CVCS-makeuo/ letdown (in Mode 6) CVCS-boration/ dilution (in Mode 6) RBCCW/CCW I containment HVAC fuel pool / spent fuel pit coo!!ng/ cleanup local compressed air system local fire protection system reactor vesselintomals temporary and special equipment (seats, dams, etc.) i I

3. Instrument and Controls Systems

source range detection startup channels , traveling in-core probe j vessel / loop level indicating system ] I

4. Radiation Monitoring 1 l

containment and fuel handling process monitoring l containment and fuel handling area monitoring temporary and special radiation monitoring (e.g., neutron sources)

5. Fuel Handling Equipment new and spent fuel shipping, receiving, and storage fue! element / assembly / module design features to accommodate handling fuel sipping fuel transfer system j video mapping system !

spent fuel pool and fuel building cranes refueling otatform and associated equipment fuel handling tools refueling machine ! manipulator crane l 1 Note: Some items apply to specific vendors i NUREG-1021, Revision 8 6 of 8 l

      ,m                                                                       ~                                 '

i ') ES-701 LSRO Written Examination Form ES-701-1

     '\_,,/                                         _

Cove- Sheet _ _ ,, U. S. NUCLEAR REGULATORY COMMISSION t LSRO WRITTEN EXAMINATION

                                          .w                               .--                                        ,

APPLICANT INFORMATION Name: _ Date: Region: I / ll /111/ IV _ Facility / Unit: Reactor Type: W / CE / BW / GE Start Time: _Stop Time: INSTRUCTIONS Use the answer sheets provided to docurnent your answers. Staple this cover sheet on top

   /^N!

of the answer sheets. The passing grade requires a final grade of at least 80.00 percent. i Examination papers will be picked up three hours after the examination starts. APPLICANT CERTIFICATION All work done on this examination is my own. I have neither given nor received aid. i Applicant's Signature i. RESULTS Test Value Points Applicant's Score Points Applicant's Grade Percent ( ) , 7 of 8 NUREG 1021, Revision 8

ES 701 LSRO Competency Grading Sheet Form ES 701-2 A Page of J plicant Docket Numoer SS- _C. LSRO Integrated Plant Operations Grading Summary Competencies / Comment Rating Factora (1) Weight 3.0 2.0 1.0 Total Page #

1. Alarms / Annunciators

a. Prioritize 0.30 0.90 0.60 0.30

b. Interpret 0.35 1.05 0.70 0.35

c. Verify 0.35 1.05 0.70 0.35 _ _ _ _ _ . _

2. Diagnosis

a. Recognize 0.33 1.0 0.66 0.33

b. Accuracy 0.33 1.0 0.66 0.33

c. Diagnose 0.33 1.0 0.66 0.33 ___ _

3. System Response

a. Interpret 0.35 1.05 0.70 0.35

b. Attenth/e 0.20 0.60 0.40 0.20

c. Plant Effects 0.45 1.35 0.90 0.45

4. Procedures

a. Reference 0.33 1.0 0.66 0.33

b. Correct Use 0.66 2.0 1.33 0.66

5. Control Board Operations

a. Locate 0.25 0.75 0.50 0.25

b. Manipulate (2) 0.25 0.75 0.50 0.25

c. Response 0.25 0.75 0.50 0.25

d. Manual Control 0.25 0.75 0.50 0.25 __

6. Communications

a. Clarity 0.70 2.10 1.40 0.70

b. Receive Information 0.30 0.90 0.60 0.30 __.

7. Directing Operations

a. Tirnely Action 0.25 0.75 0.50 0.25

b. Safe Directions 0.50 1.50 1.00 0.50

c. Oversight 0.25 0.75 0.50 0.25 ____ ___

8. Technical Specifications

a. Recognize 0.40 1.20 0.80 0.40

b. Locate 0.20 0.60 0.40 0.20

c. Compliance 0.40 1.20 0.80 , 0.40 . _ _ _

(1) Evaluate the applicant on as many rating factors as possible. If a rating factor is not evaluated, redistribute its weight among the remaining factors under that competency. (2) Only evaluate this rating factor if actual fuel handling equipment manipulations were < performed. NUREG-1021, Revision 8 8 of 8

O, ES-702 ADMINISTRATION OF REQUALIFICATION EXAMINATIONS FOR SENIOR REACTOR OPERATORS LIMITED TO FUEL HANDLING A. PURPOSE The NRC requalification examinations for senior reactor operators limited to fuel handling (LSROs) are administered under this standard according to the provisions of Title 10, Section 55.59(a)(2)(iii), of the Code of Federa/ Regulations (10 CFR), B. BACKGROUND This standard, in conjunction with ES-601 through ES-603, providas general guidance to facility licensees and requirements to NRC examiners for preparing, administering, grading, and documenting NRC requalification examinations for LSROs. Except as noted herein, the methodology and guidance in ES-601 through ES-603 for administering full-scope requalificatic,i examinations at power reactors is also applicable to LSRO requalification examinations. C, GENERAL DIFFERENCES The LSRO examinations will be conducted in accordance with the methodology outlined in ES-601, with the following exceptions. Specific exceptions related to each category of the examination are discussed in Section D. Any questions regarding the program office's y p expectations regarding the conduct of LSRO requalification examinations shall be referred to the NRR operator licensing program office, for resolution.

1. The NRC will coordinate with the facility licensee to schedule the NRC's LSRO examinations concurrent with the facility licensee's LSRO requalification examination schedule, if practical, the examination team will conduct the LSRO examination shortly before or after an outage to facilitate access to refueling equipment because some of the equipment is not accessible during plant operations.

The NRC may administer LSRO requalification examinations concurrent with full-scope initial license or operator requalification examinations.

2. The facility licensee's LSRO requalification program, LSRO job task analysis, and associated learning objectives will provide the basis for the examination if they are of sufficient scope and depth. The items in 10 CFR 55.43 and 55.45 will be samoled as appropriate to the LSRO's limited responsibilities.

3. The LSRO requalification examination will consist of a written examination and a walk-through operating test, which is administered and evaluated individually. References to the crew-based dynamic simulator test and the associated crew evaluation criteria and forms do not apply to LSROs.

4. Whenever possible, the facility licensee should include an LSRO on the examination team.

? O 1 of 6 NUREG 1021, Revision 8 Y ..

l { l ES-70'2 I l

5. The requirement to examine at least 12 operators to arrive at a program evaluation is not applicable. The region and the NRR operator licensing program office wiu determine the appropriate sample size based on the nurober of LSROs licensed at the facility. j

6. The ' Corporate Notification Letter" (Attachment 2 of ES-601) shall be revised as necessary to reflect the examination arrangements and to specify a modified list of reference material requirements associated with LSRO fuel handling activities. The region will review the reference material using the applicable portions of Form ES-601-2.

7. The facility licensee is expected to maintain JPM and written examination banks for )

evaluating LSROs. These examination banks should be periodically updated to reflect i area., of emphasis in training and to ensure that all applicable knowledge and skills are I I represented. There is no minimum threshold or ceiling for these banks.

8. The region will document the NRC's LSRO requalification examination results using l Forms ES-702-1, "LSRO Requalification Examination Report,' and ES-702-2, "LSRO Requalification Results Summary,"instead of Forms ES-601-3,4, and 5.

9. This standard does NOT provide for a formal LSRO requalification program evaluation, however if more than one-third of the examined LSROs at a facility fail, the NRC may need to inspect the LSRO requalification program. The regional staff is responsible for l determining if such an inspection should be conducted. If an inspection is performed, the staff should assess at least the following: i

a. the content of the training prograrn, the development of examination materials, and quality controls

b. the administrative controls for maintaining training material current with procedural revisions and design changes

c. the training and evaluation techniques of the facility licensee's evaluators

d. the evaluation techniques that the facility licensee uses to determine if it has implemented and assessed its training effectively -

e. the frequency, scope, and depth of the training provided to the operators D. EXAMINATION DIFFERENCES

1. Written Examination The written examination will be developed, administered, and evaluated as described in ES-602 with the following exceptions:

a. The written examination will be open reference and will contain a minimum of 25 points in a single section; static simulator scenarios do not apply to the LSRO examination. The time limit for completing the examination shall be two hours, NUREG-1021, Revision 8 2 of 6 l

i t ES-702

       )               but the examination should be constructed so that a competent LSRO can complete it in 1.5 hours. The examination should emphasize refueling procedures, administrative controls, and abnormal and emergency procedures.

The examination should include questions associated with industry and licensee event reports (LERs) and recent plant modifications affecting refueling j operations and systems that apply to the facility.

b. Form ES-702-3 will be used as a cover sheet rather than Form ES 602-1.

2. Walk throuah Ooeratina Test The walk-through test will be developed, administerod, and evaluated as described in ES 603 with the folk; wing exceptions:

a. Each LSRO will be adrninistered an operating test consisting of five tasks /JPMs.

Whenever possible, these tasks should include the use of the refueling equipment to manipulate dummy fuelonlyor the use of a refueling machine simulator if one is available at the facility, il dummy fuel manipulation or the use of a simulator are not possible, the refueling tasks should be simulated. The requirement. to conduct a minimum number of JPMs in the control room / simulator is not applicable,

b. Each JPM will consist of a task normally performed by fuel handling personnel O and will include tasks performed before and after refueling and for maintenance, surveillance or testing of systems or equipment. The examination team may evaluate the LSRO's ability to perform normal fuel handling administrative tasks including the documentation of clearances, maintenance activities, and surveillances. The operating test should also evaluate the LSRO's response to abnormal or emergency events associated with fuel handling.

c. If sufficient facility-developed JPMs are not available, the NRC can conduct a walk through examination of the type administered to an initial LSRO applicant, as discussed la ES 701.

3. Dynam c Simulator Ooeratina Test The dynamic simulator operating test described in ES-604 is not applicable to the LSRO roqualification examination.

E. ATTACHMENTS / FORMS Form ES 702-1 " Individual LSRO Requalification Examination Report' Form ES-702-2, " Power Plant LSRO Requalification Results Summary' Form ES-702 3, 'LSRO Written Requalification Examination Cover Sheet' 3 of 6 NUREG-1021, Revision 8

ES 702 individual LSRO Requalification Form ES-702-1 Examination Report PRIVACY ACT INFORM ATION - FO'R OFFICIAL USE ONLY U.S. Nuclear Regulatory Commission Individual LSRO Requalification' Examination Report Operator's Name: Facility: Docket No.: 55- Retake Exam: 1st/2nd/# Date of Last Exam: License No.: SOP-d, Expiration Date: j i Written Examination Results I i Date of Exam: NRC Examiner: Facility Evaluator: { NRC Facility Overall Grade (%) -> Operating Test Results Date of Test: NRC Examiner: Facility Evaluator: No. JPMs Correct of of Final Grade % % NRC Examiner Recommendations l Category Results Signature Written Pass / Fail Operating Pass / Fail NRC Supervisor Review Date: Pass / Fail PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY NUREG-1021, Revision 8 4 of 6

/] 've ES-702 Power Plant LSRO Requalification Results Summary Form ES-702-2 l PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY Power Plant LSRO Requalification Results Summary _= Facility: Exam Date: Examiners: Total # of Passed Failed i Overall Operators (# / %) (# / %) l Results l Individual Results Operator's Docket Grader JPM % Written Results(P/F) Name 55- Overall (%) Written Operating () NRC Fac NRC Fac NRC Fac NRC Fac NRC Fac NRC Fac NRC Fac PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY l'\ \U) S of 6 NUREG-1021, Revision 8

ES-702 LSRO Written Requalification Form ES-702-3 Examination Cover Sheet U. S. Nuclear Regulatory Commission LSRO Written Requalification Examination Operator information Name: Date: Region: I / II / Ill / IV Facility / Unit: Reactor Type: W / CE / BW / GE Start Time: Stop Time: Instructions Use the answer sheets provided to document your answers. Stap: - ,ver sheet on top of the answer sheets. Points for each question are indicated in parer .,ses after the question. The passing grade requires a final grade of at least 80.00 percent. Examination papers will be picked up two hours after the examination starts. Operator Certification All work done on this examination is my own. I have neither given nor received aid. Operator's Signature Results Test Value Points Operator's Score Points Operator's Grade Percent i NUREG-1021, Revision 8 6 of 6 i i I l l

1 1 'O APPENDIX A OVERVIEW OF GENERIC EXAMINATION CONCEPTS A. PURPOSE 1 The purpose of this Appendix is to provide an overview of two fundamental examination concepts, validity and reliability, as they apply to the development of NRC operator licensing and requalification examinations. The following topics ah? discussed: 1 the rationale for providing guidance for the construction, review, and approval of NRC , examinations (Section B) the various aspects of validity and how the NRC establishes the validity of its examinations (Section C) the concept of reliability and how it is maintained on NRC examinations (Section D) . l B. BACKGROUND The ' fact that the NRC's operator licensing examinations are prepared and administered by many different individuals working in various locations makes it imperative that a defir'ed set of administrative structures and protocols be establiched and followed to ensure that the i examinations are administered successfully and consistently. External attributes such as the number and kind of items, the length of the examination, security procedures, proctoring ' O instructions, and other administrative details are essential to the orderly conduct of an h examination. These factors have a significant effect on the reliability and validity of an examination, the comerstones that allow the NRC to make confident licensin] decisions. The internal attributes of the examination, such as its level of knowledge, level of difficulty, and

  ' the use of item banks, also impact the operational and discriminatory validity of the examination, which, in turn, can affect its consistency and reliability. If the intemal and external attributes of examinations are allowed to vary significantly, the uniform conditions that are required by Section 107 of the Atomic Energy Act of 1954, as uuended, and the basis upon which the NRC's licensing decisions rest arc challenged. The NRC nust reasonably control and structure the examination processes to ensure the integrity of the licenses it lasues.

Acceptable levels of examination consistency, uniformity, and fairness would be impossible to achieve without quantitative ,e A qualitative acceptance criteria. The Examination Standards ideatify many of the quantitative criteria necessary fer a well-balanced and consistent examination. Althougis the NRC's Knowledge and Abilities Catalogs (NUREG-1122 and -1123) have brought a degree of consistency to the qualitative issue of safety-significance, there is no comparable mechanism to aid in determining an examination's level of knowledge or difficulty before it is given. In the end, the validity and consistency of NRC examinations depend in Ic.ge part on the individual and collective judgments of the people who write and review the examinations. The discussions herein clarify the intent of the NRC's examination criteria, thereby decreasing the likelihood that inconsistencies among examinations, particularly with regard to the level of knowledge and difficulty, will jeopardize the validity of the NRC's licensing decisions. O 1 of 8 NUREG-1021, Revision 8

Appendix A C. VAllDITY For a test to be considered valid, it must be shown to measure that which it is intended to measure. In the case of the NRC examinations, the intent is to measure the examinee's knowledge and ability such that those who pass will be able to perform the duties of the RO and SRO to ensure the safe operation of the plant. The three principal facets of test validity and the techniques that are used to establish the validity of NRC examinations are outlined below.

1. Content Validity

a. Establish a Unk to Job Duties in order to develop valid examinations, the knowledge and abilities (K/As) selected for testing must be linked to and based upon a description of the most important job duties. This is accomplished through the conduct of a job / task analysis (JTA), focusing on the delineation of essential K/As.

This approach to the development of content valid lic - sing examinations was endorsed by the testing industry in the 1985 revis.c the " Standards 'or Educational and Psychological'iasting" published , iha American Educational Research Association, the American Psychologican Association, and the National Council on Measurement in Education. The standards treat licensing examinations in a separate section in recognition of their importance and uniqteness. Accordingly, those seeking additional technical guidance are encouraged to consult Chapter 11 of the document for further clarification. For the purposes of validating the content of the NRC examinations, the JTA performed on the licensed operator and senior operator positions by the Institute of Nuclear Power Operations (INPO) served as the initial source of information. The INPO JTA identified more than 28,000 K/As and nearly 800 tasks. The extensive number of tasks and K/A statements is due, in part, to the specific purpose of the analysis, which was to provide an information base to be used in developing training programs that would be applicable to all PWR and BWR facilities. Accordingly, many of the indivHual statements were too specific and/or too elementary for use as the basis for the development of NRC examinations. The job content of special interest to the NRC is that subset of K/As that are required for the safe operation of the nuclear plant. Although safe performance and efficient performance may have considerable overlap, any K/A that contributes te efficiency but not to safety is an inappropriate focus for the NRC examination. NUREG-1122," Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors," and NUREG-1123, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors," provide the basis for the development of content valid examinations for ROs and SROs consistent with the testing incustry standards described above. The fact NUREG-1021, Revision 8 2 of 8

Appendix A that the K/As from which test items are developed are drawn or sampled from the same universe regardless of who develops the examination ensures that the examinations a.c "msistently content-velid. Furthermore, developing the examinations using tht. appropriate K/A catalog in conjunction with the applicable Examination Standard.s and related Appendices will ensure that the examinations cover a representative sample of the topics listed under Title 10, Part 55, of the Code of FederalRegulations. ; The NRC's K/A catalogs were developed based on the industry JTA and were reviewed by licensed ROs and SROs as well as NRC license examiners. These experts reviewed the K/A statements for acuuracy and completeness and then rated each statement with respect to its irnportance to safe plant operation. Further explanation of the content of the K/A catalogs is provided in Section 1 of each catalog. $ in addition to the NRC's K/A catalogs, learning objectives from the facility licensee's training program often provide a supportive reference for test items to be included in the NRC examination. Since facility learning objectives are specific to the job requirements at that site, they should provide an excellent basis for test item development. However, because they are not always stated at the comprehension or analysis levels of knowledge (the preferred focus for NRC examinations) they should be referenced only to the extent that they support a test item that is being developed.

b. Use a Sample Plan Once the essential K/As have been identified through the conduct of the JTA, ;

test specifications must be developed. The test specifications consist of a l content outline or sample plan indicating what proportion of items or questions shall deal with each K/A. Because a single test cannot measure every < knowledge or competency required to be a licensed operator, it must sample the required knowledge or performance in a manner that allows inferences to be made regarding the examinees' performance on the broader population of knowledge even though it was not tested. The sample must be evenly distributed and soundly based so inat the NRC can confidently assume that the untested knowledge is proportionately known or not known in relation to the score on the sample. In other words, by testing performance on the sample, it is possible to make inferences upon the broader area of knowledge not tested. This is referred to as a validity inference. The sample plan is at the heart of making a validity inference. Research indicates that when samples are not chosen systematically and according to the sample plan, the sample is said to be biased which reduces validity. When the sample is biased or skewed in a particular direction, it introduces sampling error that makes it impractical to infer or generalize that the examinees have mastered the larger population of untested knowledge from which the sample was drawn. (' b 3 of 8 NUREG-1021, Revision 8

Appendix A Test items selected for inclusion in an NRC examination should be based on K/As contained in the appropriate K/A catalog. Testing outside the documented K/As can jeopardize the content validity of the examination. Content validity can also be reduced if important K/As are omitted from the examination. The.efore, the sample of K/As that are tested st ould cover all the K/A categories in the catalog in a fashion that is consistent with their contribution to the public protection function of the examination. Not all categories are equalin this regard. This conclusion is based on the analysis of ratings on importance and testing emphasis collected from licensed SROs and NRC license examiners. The spa.wic Examination Standards provide additional guidance on how to develop tast outlines that will ensure adequate content coverage. it is important to note that there is a difference in the testing demands for an l initial examinmon versus a requalification examination. The requalification ) examination is based upon the plant's systems approach to training during the requalification cycle and will more closely parallel the training received in the ! requalification program. Consequently, the instructional and testing processes I are more closely linked. The initial examination, on the other hand, covers all l instruction related to safety-significant K/As that either were or should have been i taught during the training program. The Examination Standards ensure that the l K/As are sampled in a relatively uniform process that would likely include content and instruction that occui s,d from the beginning to the end of the program and not be focused upon any particular segment of instruction. I

2. Ooerational Validity The second facet of validity is operational va!idity. To the extent possible, test items should address an actual or conceivable mental or psychomotor activity performed on .

the job. In this regard, the more operationally oriented a test item is, the more valid the l test item. Since operationally valid items involve skills central to job performance (i.e., analysis, predictions of events or system responses, or problem-solving) the items should be written at the comprehension or analysis levels rather than simple fundamental knowledge. The theoretical level of knowledge classification system upon I which the NRC bases its operational validity estimates is B'oom's Taxonomy. l Bloom's taxonomy suggests that testing knowledge at higher cognitive levels (i.e., comprehension and analysis) is more efficient and operationally valid because testing at those higher ievels includes the fundamental knowledge required in part to answer the higher level question. Furthermore, the higher the level tested in the test item, generally i I the more operationally valid that test item will be since it is at the higher levels that questions invoke problem-solving, diagnosis, prediction, and analysis of conditions, evente, and responses. Designing test items that test the application of knowledge in different content situations (i.e., process testing) is at the heart of designing good, discriminatnry test items. Just I as the mathematics teacher would not ask multiplication questions that were identical to practice questions, so too should the' examination author minimize asking questions that NUREG-1021, Revision 8 4 of 8 l

r j Appendix A are identical to those previously rehearsed or tested. Test items should attempt to assess similar knowledge applications in different contexts, thereby assessing the problem solving skills of students in new and different applications. These applications should be item substitutions of comparable difficulty, neither harder nor less difficult than those practiced. This practice provides assurance that the examination is valid and discriminatory, since the process rather than the specific content is primarily measured. l The NRC cannot make confident and comistent validity inferences (i.e., licensing decisions) if one examination assesses knowledge at lower levels and another examination assesses knowledge at higher cognitive levels (greater depth). 'Nhile each examination may meet sample plan coverage, the examinations are testing different levels of knowledge and consequently, they are different and inconsistent measuring instruments. Therefore, the validity inferences to be drawn regarding minimally safo operator performance are different in each case. Refer to Section D for a more detailed discussicn of consistency and reliability and to Appendix B for a more detailed discussion of the various levels of knowledge as they relate to the develooment of written test questions.

3. Discrimination Validity The third facet of validity concems the examination's ability to discriminate, or to make some distinction along a continuum of examinee performance. In that regard, the primary objective of the NRC examinations is to determine whether or not the O examinees have sufficiently " mastered" the knowledge, skills, abilities, and other attributes to perform the job of reactor operator (RC.) or senior reactor operator (SRO) at a specific plant. The NRC examinations are not intended to distinguish among levels of competency or to identify the most qualified individuals, but to make reliable and valid distinctions at the minimum level of competency that the agency has selected in the interests of public protection. j l

a. Criterion-referenced Testing l The NRC's initial and requalification examinations, like most licensing l examinations, are criterion- rather than norm-referenced tests. This means that I there is a pass-fail or minimal cut score or grade that the examinee must achieve i to demonstrate sufficient knowledge and ability to safely operate the power plant.

If the examination does not intend to discriminate at an agreed-upon minimal measure of knowledge or performance, then there is little reason to give the ; examination. For a criterion-referenced test to be effective, both the individual l test items and the examination in total must discriminate between applicants who j have and have not mastered the required knowledge, skills, and abilities.

b. Cut Scores i For NRC examinations, the cut scores (on the written examination and JPMs) are fixed at 80 percent; it is the content of the examination which varies from ;

1 occasion to occasion because of the plant-specific character of the test material. (. As is discussed below, there are several reasons why the cut scoro must be 5 of 8 NUREG-1021, Revision 8

Appendix A fixed, including the uniqueness of each examination, consistency, and public confidence. In the writing, reviewing, setting of scoring standards, and grading of any particular NRC examination, both the examination author and the reviewer are well aware of the NRC-established passing score of 80 percent. They may also have knowledge of how prior examinees have done on questions similar to the ones being used on the examination under construction and expectations as to how a qualified or an unqualified applicant should do on the examination. They must use this knowledge to control the nature and difficulty of the examination such that an examinee who is deemed to be qualified scores above the passing grade and an examinee who is deemed to be unqualified scores below this grade. The traditional cut score on the examination should not be viewed as arbitrary. l Rather, it reflects a point on the test at which author and reviewer judgment separates the qualified from the unqualified. Nonetheless, the judgment is probably similar to other methodologies for determining passing test scores. For j example, rather than explicitlyjudging the probability that a rninimally qualified applicant will pass an item, the author is implicitly being asked to write an examination on which, in the author'sjudgment, the minimally qualified applicant will obtain a score of at least 80 percent. Achieving this objective requires the author and reviewer to integrate their content and process skills.

c. Cut Scores and the Level of Difficulty For the cut score of 80 percent to be meaningful requires that individual test items be written "near" that level. A target range of 70 to 90 percent level of difficulty is recommended for individual test iterns. Test items that are so difficult l

that few if any of the examinees are expected to answer correctly do not discriminate and should not be used on an NRC examination. Test items that I are so easy or fundamental that even those examinees who are known to have performance problems will be able to answer correctly should be used with discretion. It is expected that every examination will contain some test items that . all or most of the examinees will answer correctly or incorrectly. This does not l necessarily mean that the test items or the examination are invalid. j lt should be stressed that the intent is not for everyone to get a score of 80 percent. In fact, historically about 90 percent of examinees score 80 percent or above on the NRC examinations. A score of 80 percent is the minimal pass score that the author and reviewer must keep in mind as a functional level of discrimination for setting item difficulty. In order to achieve this, the test author must keep in mind and integrate the following concepts: the level of knowledge required of the examinees taking the examination NUREG-1021, Revision 8 6 of 8 __._________...______...____J

Appendix A the operationally validity of the questions (i.e., are they expressed as a conceivable job behavior?) the ability of the distractors to distract the examinees the examinees' past performance on items of similar difficulty

d. Use of item Banks

     - Test item banks are a valuable resource for learning and represent one fundamental basis for training and testing. However, it would be ineooropriate to copy all or a significant portion of the items for an examination directly from the bank if the same items were previously used for testing or training. Test !*em banks must be used properly to maintain the validity, reliability, and consistency of the examinations. Previously administered test items reduce examination integrity because examination discrimination is reduced.

Discrimination is reduced because the cognitive level at which the examinees are tested could decrease to the simple recognition level if the item bank is small and available for the examinees to study. The comprehension and analysis levels of knowledge may not be assessable because mental thought has been reduced to a recognition level, and decision-making is absent because test /9 items, JPMs, or scenario events have been rehearsed and are anticipated. In Q short, challenge and mental analysis are lost and the examinees are tested at a rote-style rehearsal level. An examination cannot assess higher cognitive and analytical abilities if a significant portion of the items within the examination have already been seen. Furthermore, when the bank of items from which the examination is drawn is known to the examinees prior to the examination, then the examination is said to be highly predictable. Predictable examinations tend not to discriminate because what is being tested is simple recognition of the answer. Although studying past examinations can have a positive learning value, total predictability of examination coverage through over-reliance upon examination banks reduces examination integrity. When the examinees know the precise and limited pool from which test items will be drawn, they will tend only to study from that pool (i.e., studying to the test) and may likely exclude from study the larger domain of job knowledge. When this occurs, it decreases the confidence in the validity inferences that are made from performance on the test to that of the larger realm of knowledge or skill to be mastered. Therefore, the NRC has placed limits on the use of facility item banks or other such available banks or resources that have been published, reviewed, or used as the basis for training; the specific limits are discussed in the Examination Standards. The NRC appreciates the amount of resources required to develop new test items that are appropriate for use on an NRC examination, and it 7 of 8 NUREG-1021, Revision 8

Appendix A realizes that existing test items are a valuable resource that should not be wasted. Therefore, the NRC has elected to strike a balance in setting limits on the mix of previously used bank items, modified bank items, and newly developed (i.e., not previously seen) items. Additional limits have been placed on the repetition of test items from prior quizzes and examinations given at the facility. D. RELIABILITY Reliability is the second fundamental testing concept that has played a decisive role in the development of the NRC's initial and requalification examination programs. Whereas the notion of validity emphasizes the appropriateness of the content of the NRC examinations, reliability , stresses consistency, repeatability, and the degree of confidence that the examination process I will result in valid pass / fail decisions. The reliability of an examination is as important as its l validity; if it is not reliable, then it cannot be valid. j The importance of examination consistency (reliability) cannot be overstated. Test reliability, in fact, represents the consistency among examinations which, in tum, gives the NRC the confidence that all examinations are valid measures from which to make confident and valid licensing decisions. The combined effects of item bank use, the level of knowledge tested in the individual test items, and the expected discriminatory (difficulty) level of the items play an important role in determining the reliability of the examination. The higher the reliability of a test, the fewer errors will be made in determining whether the examinees have mastered the job requirements. Examinations should differ only in the specific content covered, not in their developmental processes, manner of sampling, item construction I criteria, level of item bank use, or their levels of knowledge and difficulty. The standardization I of the process creates consistency of measurement. Ideally, any two examinations that are written in accordance with these procedures and guidelines and given to the same group of examinees should produce comparable results; likewise, the results of any examination given to different but similarly trained and qualified examinees should also be comparable. The standardized examination development, administration, and grading procedures described in this NUREG have evolved over a period of years in an effort to enhance the reliability and, hence, the validity of the NRC's licensing decisions. The importance of having these procedures and complying with their intent has grown in proportion with the number of individuals and organizations that have become involved with the examination process. Section 107 of the Atomic Energy Act of 1954, as amended, requires the Commission to prescribe uniform licensing conditions for operators. Therefore, facility licensees are expected to develop and submit their proposed examinations based on the guidelines and instructions contained herein. The NRC discourages facility licensees from using testing methodologies that do not conform to the policies, procedures, and practices defined in this NUREG. Nevertheless, facility licensees may propose alternatives to specific guidance in NUF EG-1021, and the NRC will review and rule on the acceptability of the alternatives. l NUREG-1021, Revision 8 8 of 8 I u

I i APPENDIX B b WRITTEN EXAMINATION GUIDELINES A. PURPOSE The purpose of this appendix is to provide a background understanding of the principles and practices for developing multiple-choice written test questions for NRC initial and requalification examinations. Examination authors and reviewers should use the guidance herein when selecting, constructing, and reviewing questions for use on NRC written examinations. The following topics are covered: written examination background (Section B) the basic psychometric principles (e.g., low level of knowledge, low operational validity, low discriminatory validity, implausible distractors, confusing language or ambiguous questions, confusing or inappropriate negatives, collection of trua/ false statements, backward logic) and other guidelines applicable to the question development process (Section C) a checklist for reviewing multiple-choice questions (Attachment 1) examples of questions that illustrate the psychometric principles (Attachment 2). a list of references that provide additionalinformation on written examination development (Attachment 3) For a discussion of the specific written examination criteria applicable to the initial and requalification examinations, refer to ES-401 and ES-602, respectively. B. BACKGROUND

1. The Imoortance of the Written Examination Written examinations have been institutionalized into our society as an accepted and important facet of performance testing. A written examination is routinely used as an integral component of measurir,g human performance in nearly every field of study.

                                                . Educational institutions from elementary through graduate school use a written examination, in part or in whole, to measure intended competencies. Moreover, many fields of business, including the legal, medical, educational, and accounting professions, use written examinations for licensing and credentialing activities.

The importance of knowledge testing should not be underestimated since knowledge is the underpinning of professional performance. The objectives of knowledge testing are varied; they may include assessment of fundamental understandings as well as testing more advarved levels of expertise. The most effective tests of knowledge include questions and test items that measure applications of knowledge directly related to the job. In the case of the NRC operator licensing examination, the written examination yields a key measure that allows a confident decision to be made on the safety significant performance of the individual seeking a license.

   .t
    \

1 of 26 NUREG-1021, Revision 8 I

Appendix B When knowledge testing is deemphasized or sidestepped through careless or simplistic testing processes or if it is treated secondarily to other portions of the examination that are more operationally oriented, then subsequent job performance could be affected. The locs of attention and focus on testing the individual operator's cognitive abilities (i.e., comprehension, problem-solving, and decision-making) or paying insufficient attention to the operator's fundamental understanding of job content (e.g., systems, components, and procedures) ultimately may place job performance at risk of gradual degradation. When the demand for disciplined learning and study declines or the level of knowledge (depth of application) required for the job is reduced, it could lead to less time spent in training preparation, less mental review and practice, more forgetting of factual details, less reinforcement and application of job concepts, and a gradual decline in performance. Moreover, without a solid fundamental knowledge base, operators may not perform acceptably in situations that are not specifically addressed in procedures. Since every performance has an underlying knowledge component, that knowledge and its depth needs development and assessment to ensure the operators' competence on the job. Recent studies assessing mental performance in cognitively demanding emergencies point out that higher level cognitive thought such as event diagnosis and response planning are important in responding to safety related events.

2. Objective Versus Subjective Test items Traditionally, questions that require the examinee to supply an answer (e.g., short answer and essay) have been considered " subjective," while questions requiring the I examinee to select an answer (e.g., multiple-choice) have been considered " objective."

The names arose from the scoring of the items. If graders require subiect matter expertise to interpret the answers the question has been considered subjective. if the examination can be scored by verifying a single letter or number, it has Deen considered objective. Multiple-choice items are the most common and most popular of the select-type items. , For reasons of consistency and reliability, they are currently the only type of items ! acceptable for use on NRC initiallicensing examinations. Although multiple-choice { items are not as easy to construct as other forms, they are very versatile, can be used to i test for alllevels and types of knowledge, and minimize the likelihood of the examinee I obtaining the correct answer by guessing. Scoring multiple-choice examinations is also J considerably more reliable and less time consuming than scoring open-ended response items. Furthermore, since each item requires less time to answer, more items can be used to test a larger sample of K/As. This provides better content coverage, which also increases test reliability. , I l For purposes of NRC requalification examinations and initial operating tests, the l definition of " objective" is different than the traditional one described above. An cbjective tes' item is one for which (1) there is only one correct answer, and (2) all qualified graders would agree on the amount of credit allowed for any answer. NUREG-1021, Revision 8 2 of 26 l

i i l I O Appendix B U Therefore, all questions on NRC examinations shall be objectively gradable regardless

          . of the item format. Questions with no single correct answer or for which the credit given can vary, depending on who graded it or when it was graded, have no place on an NRC examination.

1 C. QU STION DEVELOPMENT Examination authors and reviewers should observe the following generic principles and question construction guidelines when preparing NRC written examinations. The guidance is based upon psychometrics, the process of applying sound qualitative processes to mental measurements. Tha generic principles apply to all question formats, including multiple-choice, while the guidelines in Section C.2 apply strictly to the multiple-choice format. It is important to minimize the number of psychometric errors in NRC examinations because test items that are free of psychometric errors yield greater measurement validity. The following principles and guidelines are summarized on Attachment 1, which can be used as l a desk-reference during the question development and review processes. The list appears to be long, but with practice, the concepts become internalized, and the process becomes less difficult. Many of the principles are accompanied by examples that illustrate the psychometric errors that should be avoided. Additional examples are provided in Attachment 2.

1. Generic Princioles 1

)            a.      Ensure that the concept being measured has a direct, important relationship to the ability to perform the job.

Although the importance of relevant knowledge and abilities (K/As) and testing objectives was stressed in Appendix A, it is equally impodant that construction of I the question itself clearly reflects the importance of the topic. Word the question i so that is has " face validity" as well as underlying content validity. That is, make sure that the question would be considered reasonable to other subject matter experts utilizing the same reference materialc. It is not always necessary to establish a direct, word-for-word match between a question and a facility learning objective. A broadly stated learning objective may support any number of related questions. Similarly, the absence of a facility learning objective does not preclude the development of a valid, K/A-based question. This is consistent with the concept that the NRC examination as a check and balance on the facility licensee's training program and alerts the licensee that it may need to develop such a learning objective. Altnough it is appropriate to develop questions regarding knowledge that is embedded in or covered by procedures, the knowledge tested should not be trivial in nature. 3 of 26 NUREG-1021, Revision 8

Appendix B

b. Make sure that the question matches the intent of the K/A.

l It is very easy to end up with a question that tests a relatively trivial aspect of an important K/A topic. When reviewing your draft question, ask yourself whether it , is likely that someone could answer the question correctly and still not meet the l objective or intent of the K/A or perform the responsibilities or tasks for which the i K/A is needed. l I If you are having difficulty translating a K/A into a test question, ask yourself the following questions to help you generate ideas for potential test questions: (1) What are the common misconceptions about this topic? (2) Why is this topic important to satisfactory job performance? I (3) in what sort of circumstances might it be important to understand this topic? (4) What might the individual do who does not understand this topic? (5) What might be the consequences of a lack of knowledge about this topic? (6) How can the individual demonstrate the knowledge?

c. State the question unambiguously and precisely. State the question as concisely as possible, but provide all necessary information.

Often the individuals who develop a question assume that certain stipulations or conditions are inherent in the question when, in fact, they are not. It is very difficult for the person who wrote the question to review it impartially or through the eyes of a new reader. Therefore, it is very important to have others review your questions to ensure that all necessary information is included, and that all extraneous or superfluous information is deleted. For example, it is not , necessary to provide a status for each and every annunciated parameter that is l in its normal (non-alarming) state. Refer to Section C.3 for additional guidance regarding examination reviews and to Part B of Appendix E concerning the instructions provided to applicants regarding question clarity and assumptions. l l However, as discussed in Appendix A, keep in mind that the key purpose of any i test item is to assess important knowledge and abilities at a level that distinguiches between safe and unsafe applicants. A test item's ability to make that distinction is referred to as its discrimination validity. For a question to discriminate at the appropriate level, the test author must exercise judgement in establishing the initial conditions posed in the stem of the question Providing too much information may " lead the applicant to the answer" and decrease the discrimination validity of the question because the answer is obvious to all the applicants. NUREG-1021, Revision 8 4 of 26

l l l 1 h G Appendix B

d. Write the question at the highest level of knowledge reflected in the testino objective. ,

One of the most challenging aspects of question development is attaining the ) appropriate level of knowledge. The reference benchmark that the NRC uses to ) classify the levels of knowledge of test items is Bloom's Taxonomy, a { classification scheme that permits the grouping of items by the level (depth) of 1 mental thought and performance required to answer the items. (Refer to Attachment 3 for additional references for further reading on Bloom's Taxonomy). Although Bloom's Taxonomy is most pertinent to written { examination questions it can also be applied to simulator scenarios and JPM 4 items. In ascending order, the three levels are defined below; refer to Section A ! of Attachment 2 for examples of each level-Level 1 (i.e., fundamental knowledge or simple memory) tests the recaK or recognition of discrete bits of information; examples include knowledge ! of terminology, definitions, set points, patterns, structures, procedural l steps and cautions, or other specific facts. Level 2 (i.e., comprehension) involves the mental process of understanding the material through relating it to its own parts or to some other material. Examples can include rephrasing information in different words, describing or recognizing relationships, showing similarities and Q differences among parts or wholes, and recognizing how systems interact, including consequences or implications. Level 3 (i.e., analysis, synthesis, or application) testing is a more active and product-oriented testing which involves the multi-part mental process of assembling, soding, or integrating the parts (information bits W Yeir relationships) to predict an event or outcome, solve a problem, or creau something new. This level requires mentally using the knowledge and its meaning to solve problems. Although test questions should be written to reflect the level of knowlec ge that is most appropriate for a specific K/A, it is best to ovoid high percentages of fundamental knowledge-level questions on the examination. (Refer tr tis-401 for specific limits.) When there is a choice between two levels of knowledge, try to write the question to reflec't the higher level. In general, test items at the comprehension and analysis levels are the most operationally oriented and, therefore, tend to be the most valid and discriminatory measure of operator i f knowledge and safe performance. Questions that require only memorization or recall are not acceptable for use on open-reference examinations. e-. Avoid questions that are unnecessarily difficult or irrelevaM. As discussed conceptually in Appendix A, both the level of knowledge and item difficulty are at the heart of examination discrimination. Examination authors

 '                                          5 of 26                     NUREG-1021, Revision 8

Appendix B l should develop examinations that are estimated to center around the 80 percent cut score level, with individual item difficulty estimated to fall in the 70 to 90 percent difficulty range. These parameters should not be viewed as precise benchmarks, but rather as approximate end points. Examination authors should consider the results of past examinations when preparing a new one. Past performance on individual test questions may provide a basis for generating new { questions and for estimating the level of difficulty of the examination. For example, questions that everyone got wrong may be an indicator that the topic was not given sufficient emphasis in training or that the item was poorly worded. Conversely, questions that everyone got correct may indicate that the item was written at too low a level or that the distractors were not very plausible. Since item difficulty can usually be decreased or increased by revision, the , examination author need not be overly preoccupied with difficulty when writing the items. The author should focus on achieving a valid measure of the concept he is attempting to evaluate. When attempting to determine the appropriate level of difficulty, it may be helpful to think of two groups of individuals, one composed of experienced operators and the other of typical applicants, and evaluate the likelihood that each group of individuals will be able to answer the question. If at least 80 percent of the job incumbents or license applicants should be able to answer the question as written based on the expected knowledge levels for the position (operator or senior operator), then the item is likely written at an appropriate discriminatory level. Examination authors and reviewers may also ask themselves the following questions in an effort to identify questions that are unnecessarily difficult or irrelevant: Could someone do the job safely and effectively without being able to answer the question? If so, is it because the content is inappropriate, because the wording is unclear, or because the level of understanding is too great? What aspects of the item or option might cause the most difficulty? Has the item been made artificially difficult? Can a person understand the principle being tested and still miss the item? Estimates of difficulty made by the examination author and reviewers may vary somewhat but should not vary widely. Unless there is some reason to doubt the estimates of some reviewers, the average estimate may be taken as a basis for judging the suitability of item difficulty for the examination. Items should be revised if estimates fall well below or above the 70 to 90 percent target range. Research has shown that when authors write test items in their own area of specialization, there is a tendency to underestimate the difficulty of a concept or principle being tested. This tendency can manifest itself in two ways: (1) the i NUREG-1021, Revision 0 6 of 26 9, l i

l (,,} Appencix B _ V author will view items of average difficulty as easy, or (2) in an effort to get plausible misleads among distractors in a multiple-choice test item, the author may make the item even more difficult. For this reason, an estimate of item difficulty made by the reviewers will probably be more accurate than one made ; by the author of the item. ; Examination authors should take care not to develop an examination with wide swings of individual item difficulty. For example, writing half the items at a 60 percent difficulty level with the other half at a 100 percent level would yield an average of 80 percent, however, the flaws in this approach are numerous. The items at the 100 percent level, by design, would be meaningless since they would fail to discriminato at any level because the expectation is that nearly everyone would get the answer correct. On the other hand, those written at the 60 percent difficulty level, by design, would also not discriminate and likewise be l unfair because those items would be expected not to be answerable by 40 , percent of the examinees. l

f. Limit the question to one concept or topic, unless a synthesis of concepts is being tested. l There is a common misconception that testing for multiple K/A topics in onc l question is a time-efficient way to examine. Questions containing a variety of ;

Ch/ ( topics and issues only serve to confuse the examinee about the purpose of the question and what is expected in terms of a correct response. Each individual l l question should test one K/A topic, and that topic, as well as the intent of the question, should be clear to both the reviewer and the examinee.

g. Avoid copying text directly from training or other reference material. !

Another common tendency among examination developers is to copy sentences directly from reference material and turn them into test questions. Unfortunately, ques 6Dns written in this way generally encourage rote memorization. Further, copying from reference material can cause ambiguity or ceficiency in questions because the material lifted often draws its meaning and importance from its surrounding context. Therefore, important assumptions or stipulations stated elsewhere in the material are often omitted from the test question. Finally, such questions can frequently be answered correctly by examinees who do not really understand the concept, but do remember the specific wording on a page of reference material. Conversely, examinees who understand the topic, but not in the exact way it was written in the material, may miss the question because of unstated assumptions or other missir.g information.

h. Avoid " backward logic" questions that ask for what should be provided in the question, and provide what should be required in the examinee's response.

O t 1 'd' 7 of 26 NUREG-1021, Revision 8

7 I Appendix B in addition to testing on valid topics, it is important to examine on those topics in a way that is consistent with how the K/A should be remembered and used. Do not test on the topic in a backwards way. Section G of Attachment 1 provides examples of backward togic questions.

2. Other Question Construction Guidelines The following principles and guidelines apply specifica!!y to multiple-choice questions.

a. Use four answer options.

The four-distractor multiple-choice item with only one correct answer is the only style acceptable for NRC examinations. The use of test items with multiple correct answers from which the examinee mLst select the "most correct" answer is not acceptable because it would significantly reduce the reliability of the examination results by increasing the effect of examiner subjectivity in the examination development and grading process. The five answer option contributes nothing to the question, and any forrnat with fewer than four distractors makes guessing correctly more probable. The following tour basic models are acceptable and may be used in combination with one another. Model A: a. correct answer

b. Incorrect answer

c. incorrect answer

d. incorrect answer This model depicts the traditional multiple-choice design format with one correct single word / phrase answer followed by three incorrect single word / phrases options. Notice that the lengths of all the options are similar.

Model B: a. correct answer

b. plausible misconception i

c. incorrect answer !

l

d. incorrect answer This model, in which a plausible misconception is used as an incorrect answer, is ,

a variation of Model A. Notice again that the lengths of all options are similar. { Model C: a. correct answer with correct condition (e.g., because, since, i when, if, etc.) l b, correct answer with incorrect condition I

c. incorrect answer with incorrect condition

d. incorrect answer with incorrect condition NUREG-1021, Revision 8 8 of 26

l i

h - . . - Appendix B l j Model C depicts an acceptable design that uses answers with conditions (i.e., a setting, event, cause/effect) that may make the answer correct or incorrect. Notice that Model C shows only one correct answer with its y correct condition and that all options are uniform in length. ] I Model D: a. correct answer i

b. Incorrect answer

c. correct answer with incorrect condition

d. incorrect answer with incorrect condition Model D is useful when it is not possible to create all options in uniform iength. I This model shows paired lengths (two long and two short options) which avoids i any single option from standing apart (either too long or too short) from the remaining options.

b. Do not use "none of the above" or "all of the above."

               "All of the above" questions provide inadvertent clues to the exeminee. When the "all of te above" option is the correct response, tne examinee need only          ,

recognize that two of the options are correct to answer the questica correctly. { When the 'all of the above" option is used as a distractor, the examince needs ! only to be able to determine that one option is incorrect in order to eliminate this I

/^

t option. "None of the above" responses should not be used with "best answer" multiple-choice questions, since it may always be defensible as a response, l

c. Do not present a collection of true false (T/F) staternents as a multiple-choice item. l As discussed earlier, each item should focus on ene K/A topic determined by the stem of the question. A question containing answer options related to many separate issues does not increase the efficiency of the question. On the l contrary, questions with multiple topics only confuse the examinee about the meaning and purpose of the question.

A way of determining if you have a test item that is a collection of T-F statements is to check whether the answer can be determined or the distractors can be rejected without the information contained in the stem. If they can, then you likely have a question that is a T/F collection. Refer to Section F of Attachment 2 for sample c,uestions that illustrate this , I psychometric deficiency.

d. Dofine the question, task, or problem in the stem of the question.

In designing multiple-choice questions that are operationally based and require an application /use scenario, one suggestion is to provide the conditions in tne y 9 of 26 NUREG-1021, Revision 8 4

                        ~                                                                        ~

Appendix B first part of the question separated by a double space from the body of the question, and blocked to the left column with each condition bulleted, as in the following example:

                   "Given the following conditions:

Both main feed pumps tripped AFW automatically started AFW valves reset to control steam generator water level APN suction pressum decreases to seven (7) psig Which ONE of the following describes the AFW pump response for the given conditions?

a. suction will automatically shift to the nuclear service water system

b. suction will automatically shift to the upper surge tank

c. trip when suction pressure decreases to five (5) psig

d. trip after a six (6) second time delay" include as much necessary information as possible about the problem or situation in the stem, leaving only the solution, action or effect for the answer options. Consider the following " poor" and "better" examples:

(Poor) "At 50% power:

a. the equilibrium xenon reactivity worth is approximately equal to the equilibrium xenon worth at 100% power b, the equilibrium xenon reactivity worth is approximately one-half the equilibrium xenon worth as 100% power

c. the equilibrium xenon reactivity worth is approximately two-thirds the equilibrium xenon worth at 100% power

d. the equilibrium xenon reactivity worth is approximately three-fourths the equinbrium xenon worth at 10u% power" (Better) "How does the 50% power equilibrium xenon reactivity worth compare to the 100% power equilibrium xenon reactivity worth?

a. equal to

b. one-half

c. two-thirds

d. three-fourths"

e. When possible, avoid using negatively stated stems. If a negative stem is necessary, highlight the negative word (e.g., not, never, least).

i lt is very tempting to write negatively stated questions, since they can be constructed by picking three true statements out of the reference material and

                                                                                                    '1 NUREG-1021, Revision 8                          10 of 26                                             <

o

r. i

  /]                                          Appendix B

() changing a fourth statement to make ;t false. However, studies have shown that examinees do not do as well on negatively stated questions, either because they overlook the negative word and/or because negatively stated questions require examinees to pick an answer that is not true or characteristic, which can be , somewhat confusing. In addition, these questions tend to emphasize negative j leaming. For example, consider the following stem of a multiple-choice question: l l

                       'During 100% power operation, the feedwater heater 2A high level dump     l valve opens inadvertently. The condensate pumps will not do which of the following:"

This stem can be made to read positively: l "During 100% power operation, the feedwater heater 2A high level dump valve opens inadvertently. The condensate pumps will:

a. increase flow to maintain feedwater flow rate i b, trip due to a runout condition l

c. have no response

d. trip due to low suction pressure" There are times when a negatively stated question is unavoidable. However, never use a negatively stated stem with a negatively stated answer option, as illustrated by example E.3 in Attachment 2.

f. Provide sufficient counterbalance in questions with multi-part answers.

Multiple-choice questions can legitimately contain multi-part answer options. However, if the answers contain too many parts and/or too many options for each part, cues indicating the correct answer may be unavoidable. Consider the l following example:

                        "The RCS is in hot standby with no reactor coolant pumps running. If OTSG pressure is decreased, according to the plant verification procedure, which of the following temperature responses indicates the presence of natural circulation?

a. T-H increases, T-C remains the same

b. T-H increases, T-C decreases

c. T-H decreases, T-C decreases

d. T-H remains the same, T-C decreases" The examinee could choose the correct answer (c) without knowing about the T-C temperature response in this situation, since "T-H decreases" only occurs in option "c".

Notice that two-part answers, with each part containing a two-option response, provide complete counterbalance, since all contingencies can be covered in four f] responses, as in the following example: N ) NUREG-1021, Revision 8

    "                                           11 of 26

p I I Appendix B l l l i "Which of the following is a definition of quadrant power tilt ratio (OPTR)? l

a. minimum upper detector output divided by average upper detector output l

b. maximum upper detector output divided by average upper i detector output

c. minimum upper detector output divided by average lower detector output

d. maximum upper detector output divided by average lower detector output" A multi-part question format which is highly recommended is one in which the two-part answer options consist of a two-level response (e.g., yes/no; off/on) and a reason, as in the following example:

                    "Which of the following describes the behavior of equilibrium xenen reactivity over core life?

a. it decreases because of the increased fuel burnup

b. it decreases because of the decrease in plutonium- xenon yield

c. it increases because of the increase in thermal flux

d. it increases because of the decrease in boron concentration" l

Sometimes, in an effort to improve their plausibility, distractors may include I secondary pieces of information that have lower relative importance and i discriminatory value than the key point of the distractor. However, those secondary pieces of information are not irrelevant; the value of the question should be considered as a whole and not discounted because the answer choices contain information of lower importance.

g. When possible, include common misconceptions as distractors. Since the j purpose of the examination is to differentiate between competent and l

             .ss-than-competent examinees, a good source of questions involves topics in          l which there are common misconceptions about important K/A topics. For                 i example, the following question was based upon a common misconception about loss of subcooling margin:
                     "During a small break LOCA with a resultant loss of subcooling margin, why are the reactor coolant pumps (RCPs) secured?                             )

a. to prevent pump damage resulting from operation under l two-phase conditions to prevent core damage resulting from rapid phase separation b.

upon subsequent loss of RCS flow l c. to reduce RCS pressure by removing the pressure head developed by the RCPs

d. to remove the heat being added to the RCS by the operating RCPs" NUREG-1021, Revision 8 12 of 26 9il j

l l i

O Appendix B

h. Make all answer options homogeneous and higilly piausible.

Consider the following " poor" and "better" examples:

                                                            "On a loss of condenser circulating water intake canal, the upper surge tank, hotwell, and condensate storage tank will supply sufficient feedwater to allow decay heat removal for approximately:

Poor BeMer

a. 15 minutes 8 hours

b. 8 hours 24 hours

c. 48 hours 48 hours

d. 3 months 72 hours' Notice how one method of changing the difficulty level of a question is to vary the similarity of the answer options. The distractors should be similar enough to be chosen 'oy those who do not meet the testing objective, yet different enough so they do not test trivialissues or distinctions. A!so note how the answer options in each case have been listed in order of magnitude,

l. If the answer options have a logical sequence, put them in order (as in "h" above).

O

j. Avoid overlapping answer options, as in the following example:

                                                            "The SPND uses rhodium which decays with a half-life of 42 seconds.

How long will it take for a detector to indicate approximately 95% of an instantaneous power level change? Poor Better

a. 2 to 4 minutes 1 to 2 minutes

b. 4 to 6 minutes 3 to 4 minutes

c. 6 to 8 minutes 5 to 6 minutes

d. 8 to 10 minutes 7 to 8 minutes'

k. Do not include trivial distractors with more important distractors.

In the search for distractors, it is very tempting to include relatively trivial facts along with options focused on more important issues or concepts, as in the following example:

                                                           'Which of the following is true concerning the turbine?

a. The turbine is rotated at low speed when shut down in order to prevent distortion of the turbine casing.

b. Turbine eccentricity is the measure of turbine speed.

 ,/                                                        c.       The turbine blades are cooled by hydrogen gas.

t b 13 of 26 NUREG 1021, Revision 8 l

Appendix B

d. Technical specifications require at least one turbine overspeed protection system be operable in Mode 2."

Relative to the other options, "c" could be considered a trivial distractor. Even if included as a wrong answer, the inclusion of relatively unimportant information jeopardizes the content validity of the question. Also note that this question consists of a collection of true/ false statements as described in Section C.2.c. I. Vary the location of the correct answer; avoid a pattern. Make sure the position of the correct answer is randomized throughout the examination. This means that the "a", "b", "c" and "d" options should be correct about an equa! number of times, but in no specific order,

m. Avoid " specific determiners" that give clues as to the correct answer. Specific determiners include the following:

(1) distractors that do not fo%w grammatically hm the stem, as in the following example:

                         "During 100% normal power operation a single steam flow              j element in the steam generator feedwater control system fails        l high. This will cause:

a. the feedwater valves to increase steam generator level .

slightly before retuming the level to normal l

b. before retuming the level to slightly above normal, the i feedwater valves to increase the steam generator level significantly ;

c. the feedwater valves to increase the steam generator level l to the level of a reactor trip

d. the feedwater valves to increase the steam generator level slightly anc* maintain the increased level" l i

Note the improvement when distractor "b" is reworded as follows:

                          "b. the feedwater valves to increase the steam generator level  !

significantly before returning the level to slightly above I normal" (2) options that can be judged correct or incorrect without reading the stem (3) equivalent and/or synonymous options, which rule out both options for an j examinee who recognizes the equivalence j (4) an option which includes another option (for example: a) less than 5; b) less than 3...) NUREG-1021. Revision 8 14 of 26

i p Appendix .B (5) Implausible distractors (6) a correct answer which is longer than the distractors (7) qualifiers in the correct answer (e.g., probably and ordinarily) unlecs they are also used in the distn.: tors (8) words such as "never" or "always" which suggest a wrong option l (9) a correct option that differs frcm the distractors in favorableness, style, or terminology, as in the following example:

                     "Which action or occurrence is likely to cause water hammer?

a. maintaining the discharge line from an auto starting pump filled with fluid

b. water collecting in a steamline

c. pre-warming of steam lines

d. slowly closing the discharge valve of an operating pump" In the above question, all options except for "b" (the correct answer) describe preventiva actions, while option "b" describes a condition that occurs as a result of negligence or oversiqht. A test-wise examinee need

{ s only know that water hammer is not a desired occurrence to determine that "b" is the least favorable and therefore the correct answer.

n. When appropriate, use distractors that are generically correct statements but do not correctly answer the question, as illustrated in the following example:

Preparations are being made for refueling and the following plant conditions exist: The refueling cavity is filled with the transfer tube gate valve open. The SFP LO LEVEL and CTMT SUMP HI LEVEL annunciators are in alar n. Which ONE of the. following is the required IMMEDIATE ACTION in response to these conditions?

a. Verify alarms by checking the containment sump level recorder and spent fuellevelindication.

b. Sound the containment evacuation alarm.

c. Initiate containment ventilation isolation.

d. Initiate control room ventilation isolation.

Answer "a" is a generic good practice, but it is not responsive to the conditions i specified in the stem of the question. It is not a required immediate action, nor is it an appropriate response in light of the mutually confirmatory annunciators that are in alarm. 15 of 26 NUREG-1021, Revision 8 l

3. Reviewiw Test items Examination reviewers can assist examination authors by performing technical content, level of difficulty, psychometric, and editorial checks. There is an advantage in considering each of these four areas separately and in this order. If there is a need to revise an item on the basis of one stage of the review, the changes should be made before going further because the changes at each stage could well affect the reviews which follow. For example, a criticism which appears to affect only one distractor may ultimately lead to changes in other parts of the item, so time spent reviewing the item for grammar and punctuation may be wasted.

There are some advantages in having someone not familiar with the area being tested review the questions for clarity, grammar, expression, spelling and punctuation. Such a reviewer can determine whether the item is answerable by a person without knowledge of the field. The examination author and reviewers should ask themselves the following types of questions: Will the examinees clearly know what they are expected to do? Do they have all the information they need to work with? Does answering the question depend on certain assumptions that must be stated? A more thorough list of suggestions for examination authors and reviewers to check is included in Attachment 1. D. ATTACHMENTS / FORMS , " Question Development Checklist" ; , " Examples" , " References" l NUREG-1021, Revision 8 16 of 26

    - Appendix B                       Ouestion Development Checklist                      Attachment 1

1. Does the concept being measured have a direct, important relationship to the ability to perform the job?

2. Does the question match the testing objective and intent of the K/A?

3. Is the question clear, concise, and easy to read? Could it be stated more simply and still provide the necessary information? Should it be reworded or split up into more than one question?

4. Is each quest lJn stated positively, unless the intent is to test knowledge of what not to do?

5. Does the question provide all necessary information, stipulations, and assumptions needed for a correct response? Is as much information as possible Mcluded in the stem?

6. Is the question written at the highest appropriate level of knowledge or ability for the job position of the person being tested?

7. Is the question free of unnecessary difficulty, trickiness, or irrelevancy?

8. Is the question limited to one concept or topic, making it something other than a

,O collection of true-false items? \ )

9. Does the question have face validity?

10. Are key points underlined or highlighted?

11. Is each question separate and independent of all other questions?

12. Are the answer options isomogeneous and highly plausible? Are common misconceptions used as distractors? Is the question free of trivial distractors?

13. Are "none of the above" and "all of the above" avoided?

14. Are there four answer options for each question?

15. Are the answer options of the questions ordered sequentially?

16. Is the question free of " specific determiners" (e.g., logical or grammatical inconsistencies, incorrect answers which are consistently different, verbal associations between the stem and the answer options)?

( \ 17 of 23 NUREG-1021, Revision 8

Appendix B Examples Attachment 2 A. LEVELS OF KNOWLEDGE The first three examples illustrate how the level of knowledge tested can vary among a series of questions that focus on the same pair of K/As. Even though the K/A statements use verbs (identify, define) that elicit a fundamental or simple memory level of knowledge, the item writer can increase the question's operational validity by testing at a higher cognitise level. 191004K101 (PWR) or 291004K101 (BWR): Identification, symptoms, and consequences of cavitation 193006K111 (PWR) or 293006K109 (BWR): Define or explain cavitation

1. Fundamental Knowledae/Simole Memorv "Which one of the following describes pump cavitation?

a. Vapor bubbles are formed when the enthalpy difference between pump discharge and pump suction exceeds the latent hoat of vaporization.

b. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.

c. Vapor bubbles are produced when the localized pressure exceeds the vapor ]

pressure at the existing temperature,

d. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer." ,

This question simply asks for a description of cavitation and as such, is a low cognitive order question. There is no understanding, analysis, or problem-solving involved. The examinee only needs to reccgnize the correct description (b); the other options appear plausible but are, nonetheless, incorrect.

2. Comprehension

                   " Cavitation in an operating pump may be caused by:
                                                                                                                   ]

a. lowering the pump suction temperature

b. throttling the pump suction valve i

c. increasing the pump backpressure !

d. increasing the pump suction pressure" This example requires the examinee to determine causation, which requires an understanding of the correct answer and that the incorrect answers are indeed, incorrect. The quality of the item, as with any item, is determined by the distractibility of j the incorrect options. !

NUREG-1021, Revision 8 18 of 26 Ol

Appendix B 2 Attachment 2

3. Analysis "While on surveillance rounds, an operator notices that a centrifugal pump is making a great deal of noise (like marbles rattling inside the pump casing) and the dischatge pressure is fluctuating. This set of conditions indicates pump:

a. runout

      - k.      cavitation

c. bearing deterioration

d. packing deterioration" This example requires the candidate to analyze multipte abnormal indications (multiple effects) for an operating centrifugal pump and determine the cause (complex cause-effect). All the distractors are initially plausible in that they have face validity (i.e., they have reasonable connections to centrifugal pump operation).

4. The following four examples illustrate " low level of soowledge" questions that should be used judiciously on NRC examinations.

        "Which one of the following is powered from 4160 VAC bus 1 A7 l

g. RHR pump A j

b. RHR pump B l

c. RHR pump C

d. RHR pump D' i

        " Select the full core display indication of a drifting control rod.

a. red light

b. white light

c. blue light

d. amber light" Although the above items have a high K/A value, they are written at a low level of j knowledge and also have low operational validity and low discriminatory value. The i following question tests at a low level of knowledge because it doesn't test the l examinee's ability to recognize what class a fire is or select the correct extinguisher. All !

the examineo has to know is that water is for class A fires. l I i 4 g

r Appendix B 3 Attachment 2 "Conceming use of water as a fire extinguishing agent, select the correct statement from the following:

a. It is the primary agent for extinguishing Class A fires and also effective on Class B and C fires.

b. It is the primary agsnt for extinguishing Class B fires and also effective on Class A and C fires.

c. It is the primary agent for extinguishing Class A and B fires but not effective on Class C fires.

d. It is the primary agent for extinguishing Class B and C fires but not effective on Class A fires."

The next question might be considered a fundamental knowledge level question that errs in the opposite direction: It could be too difficult unless the operators are expected to memorize the correct time requirement in order to preclude damage to equipment. Moreover, this item may also have low discriminatory validity unless at least 80 percent of the examinees are expected to know the answer from memory.

          "RCP 2A tripped after running for 50 minutes. The RCP was restarted, but tripped again within 4 seconds. Which ONE of the following is the minimum required interval before the next attempt to start RCP 2A7

a. 15 minutes

b. 30 minutes

c. 45 minutes

d. 60 minutes' B. LOW OPERATIONAL VALIDITY l The next three questions illustrate another common psychometric deficiency, low operational validity, that should be avoided on NRC examinations.

1. "Under which one of the following conditions should the Shift Supervisor inform the shop l steward? l

a. initiation of a directed overtime request

b. disciplinary action against a supervisory employee

c. medicalinjury of a contractor employee

d. personnel error by a bargaining unit member" While this question may be related to a shift supervisor's job, it has nothing to do with nuclear safety and should not be included on an NRC examination.

l l l NUREG-1021, Revision 8 20 of 26 l l t

f: O Appendix B 4 Attachment 2

2. "Which one of the following main steam line components is designed to limit the l

differential pressure across the steam dryer assembly? I

a. main steam line flow elbows

b. main steam isolation valves

c. main steam shutoff valves

d. main steam line flow restrictors" l Knowing the purpose of a flow restrictor is not a good indicator of the operator's ability to operate the plant. Knowing the answer to this question is not clearly job related.

3. "Given that all components controlled by the ' Locked Valve, Breaker, and Component Control' administrative procedure must be properly sealed and tagged, which one of the l following is the correct location for the "XXXX-XXXX" tag for an electrical breaker? '

a. wired to the breaker handle

k. glued to the breaker cubicle

c. attached to the breaker cubicle with a magnetic clip

d. wired to the breaker cabinet door" l 1

This question is likely unrelated to the reactor operator's job function and would therefore be unacceptable. I

Q C. LOW DISCRIMINATORY VALIDITY The next three questions illustrate another common psychometric deficiency, low discriminatory validity, that should be avoided on NRC examinations.

1. "Which one of the following reactor water levels will initiate the RHR pumps?

a. level 1 only

b. level 1 and 2 only

c. level 1 and 2 and 3 only

d. level 6 only" This information in this question should be known by all operators at all times. The question has low discriminatory value and also tests at a low level of knowledge.

i

                                                     21 of 26                    NUREG-1021, Revision 8 i

l

Appendix B 5 Attachment 2

2. "The plant is recovering from a scram due to a spurious Group i isolation. The cause of the isolation has been repaired and preparations are being made to reopen the MSIVs.

Reactor pressure is currently 825 psig and the main steam lines are being pressurized. WHICH ONE (1) of the following is the LOWEST main steam line pressure that will allow the MSIVs to be opened per the procedure?

a. 625 psig

b. 675 psig

c. 725 psig

d. 775 psig" This question does not discriminate and has low operational validity because in real life l the applicant may not be expected to have memorized the procedure. j l

3. "S.G. (corrected) = S.G. (uncorrected) + (T - 77 decrees F)(.001) + (Level Mark)(.003)

I Based on the above information, the specific gravity (SG) is ? which ? meet the Technical Specification Category A limit. Note: This question requires the use of TS 3.8.2.3. ,

a. 1.198, does NOT

b. 1.195, does NOT ,

c. 1.207, does l si. 1.201, does* l l

This question might appears to test the examinees' ability to understand and apply l battery parameters to the determination of TS operability. However, the question really l only tests their ability to substitute certain parameters into a given equation and perform an arithmetic calculation. Reference to the TS noted in the question is not required based on the three different values of SG (corrected) supplied as distractors. Therefore, the question has a low discriminatory value since any individual possessing adequate arithmetic knowledge will arrive at the correct answer. D. IMPLAUSIBLE DISTRACTORS The next two questions illustrate the concept of implausible distractors, another common psychometric deficiency that should be avoided on NRC examinations. l l l NUREG-1021, Revision 8 22 of 26

['v - ppendix B 6 Attachment 2 1, 'Which of the following will cause the RHR pumps to start during a design basis LOCA? 4

a. Iow drywell pressure

b. high reactor water level

s. high drywell pressure

d. MSIVs in the NOT OPEN position" Distractors 'a," "b," and "d" are implausible, considering minimal knowledge of the plant response to a loss of coolant accident.

2. "Which ONE of the following ccnditions will NOT result in a shutdown of the SBGT System?

a. manual shutdown

b. high temperature 225 degree F charcoal bed

c. high temperature 180 degree F heater inlet

d. overloads in the local control panel' Distractor "a" is very implausible, and distractor "d" is subjective. The question is also written from a negative perspective.

E. CONFUSING LANGUAGE ) The following questions illustrate how confusing language and inappropriate negatives in the stem of the question can mislead examinees. Such questions should be avoided on NRC examinations. I

1. 'Which one of the following parameters will start HPCI, RCIC and SBGTS7

a. Iow reactor water level

k. high primary containment pressure

c. high reactor building exhaust radiation -

d. Iow reactor building differential pressure" This question could result in 4 correct answers since the question could be interpreted individually or collectively.

l I l l l l

   /O U                                                     23 of 26                    NUREG-1021, Revision 8

Appendix B 7 Attachment 2

2. "Which ONE of the following most accurately describes the response to a static inverter failing.

a. The power supply will automatically transfer to the alternate 600 V Bus 2C / Vital AC Transformer 2A.

b. The 125 VDC battery will maintain power to the Vital AC Cabinet for up to 5 hours.

c. The power supply can be manually transferred to the alternate 600 V Bus 2C /

Alternate Static Inverter by depressing a transfer pushbutton. d_. The power supply can be manually transferred to the alternate 600 V Bus 2C / Vital AC Transformer 2A by positioning the transfer switch to ALTERNATE." This question implies an automatic response, but the listed correct answer and one distractor are operator actions, not responses to the loss of the static inverter.

3. "Regarding temporary plant alterations (TPA), technical reviews are NOT required for -

a. a TPA NOT installed using an approved procedure

b. TPAs installed on BOP systems but are required for safety related systems

c. a TPA that has NOT been directed by the shift supervisor to be an emergency TPA

d. all TPAs directed by the shift supervisor.

This question contains multiple problems: (1) While negative questions can be used, they should be used for good reason; there appears to be no good basis for asking this question negatively. (2) Two of the distractors ("a" and "c") also contain a negative, creating a double negative and readability confusion, a violation of good item writing practice. The question should more appropriately ask the conditions under which technical reviews are required, thereby eliminating the negative in the stem. F. COLLECTIONS OF TRUE/ FALSE STATEMENTS Collections of true/ false statements typically only test simple rote memory; the examinee needs only to recall a definition or condition. The questions elicit no comprehension or problem-solving; hence, they lack operational validity. This type of question allows an examinee to answer the question without referring to the stem of the question and should be avoided on NRC examinations. l

1. "Which ONE of the following is true?

I ! a. High drywell pressure will auto start the emergency diesel generates.

b. Low reactor water level will trip the main turbine.

c. High reactor pressure willinitiate RCIC.

d. High reactor power with the mode switch in startup will NOT close the MSIVs."

NUREG-1021, Revision 8 24 of 26

Appendix B 8 Attachment 2

2. "Which one of the following describes pump cavitation?

a. Vapor bubbles are formed when the enthalpy difference between pump

                    - discharge and a pump suction exceeds the latent heat of vaporization.

h. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump,

c. Vapor bubbles are produced when the localized pressure exceeds the vapor pressure at the existing temperature.

d. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer."

G. BACKWARD LOGIC Backward logic questions ask the examinee for information normally received, and provide the examinee with information he/she normally has to supply. In an operational setting, operators are faced with conditions and required to know what procedure to use. These questions ask them to do just the opposite and should be avoided on NRC examinations.

1. "Which of the following parameters will simultaneousiv start HPCI, RCIC and SBGTS?

a. high RPV water level i

h. high drywell pressure O c.

d. Iow RPV water level Iow drywell pressure" It would be better to select a parameter and then request the expected system response because that is more operationally relevant.

2. "If it takes 12.5 cubic feet of concrete to build a square loading pad 6 inches thick, what is the length of one side of the pad?"

This question gives the examinees information they should be asked to calculate, while it requires them to provide information they would be supplied in an actual work situation. 25 of 26 NUREG-1021, Revision 8

l Appendix B References Attachment 3

1. Bloom, Benjamin (1956) Taxonomy of Instructional Objectives: The Classification of Educational Goals, Handbook I: Cognitive Domain, New York, McCay

2. Isaac, Stephen (1990) Handbook in Research and Evaluation,2nd ed., San Diego, EDITS

3. Novac, J. and Gowin, D. (1993) Leaming How to team, Cambridge University Press i

l l I O\l l l l

NUREG-1021, Revision 8 26 of 26 r

1 l J 1 O APPENDIX C V JOB PERFORMANCE MEASURE GUIDELINES A. PURPOSE This Appendix provides a framework for preparing and evaluating job performance measures

   - (JPMs) to ensure they are of appropriate substance and format for initial operator licensing and     i requalification examinations. The following elements are discussed in detail or attached for information:

a basic procedure for developing new JPMs (Section B), including forms to document the JPM and to assess the quality of the product (Form ES-C-1 and ES-C-2) guidelines for the development and use of alternate-path JPMs (Section C) a discussion of walk-through evaluation techniques (Section D) Adhering.to the concepts and guidelines discussed herein, in association with the specific operating test criteria cited in ES-301 or ES-603, as applicable, will enhance the consistency and validity of the walk-through tests. B. DEVELOPING AND REVIEWING JPMs The major JPM components and instructions for their development are summarized below. The instructions apply to both the initial and the requalification examination programs, except as tq- noted. Although they are written from the perspective of new JPM development, the V instructions should also be referenced, as necessary, when moditying existing JPMs for reuse and reviewing proposed JPMs for quality. Select the systems and tasks to be evaluated during the walk-through portion of the operating test in accordance with the specific initial and requalification examination criteria in ES-301 and ES-603, respectively. If a JPM already exists for the selected task, it should be reviewed against the guidelines and criteria discussed below to ensure that it is acceptable for use, if a new JPM is required to evaluate the selected system or task, prepare the JPM in accordance i with the following basic steps and document the JPM using Form ES-C-1, " Job Performance i Measure Worksheet," or equivalent. Form ES-C-2," Job Performance Measure Quality l Checklist," can be used to verify that the relevant criteria are satisfied. l

1. Soecifv initial Conditions Determine those system and plant conditions that would permh tne task to be performed realistically. They should provide sufficient information regarding the status of the plant and system to facilitate task performance, without coaching the examinee. If the task is t

ended to be performed on the simulator, it is worthwhile to differentiate those specific

             .,iitial conditions and system realignments that are necessary for the task to be performed as planned from those other general conditions that add realism and set the stage for performing the task but have no real bearing on the successful execution of        ,

the task. Breaking down the initial conditions in such a manner will simplify the simultaneous administration of diffeient tasks by two or more examinees. t 1 of 10 NUREG 1021, Revision 8

f Appendix C If the JPM is intended to evaluate the examinee's ability to implement an alternate path (refer to Section C) within the facility licensee's procedural guidance, the initiating equipment or instrument failure should be reflected in the simulator initial condition specifications. The JPM shall also include an initiating cue that provides the stimulus for the examinee to begin the task performance. When appropriate, the cue should clearly specify the desired endpoint for the task. For example, if it is desired for the examinee to start and load the emergency diesel generator, the cue should state the load at which the task will be considered complete. Alternate path tasks, as described in Section C, may have an actual endpoint different from that stated in the initiating cue. The initial conditions and initiating cue may be duplicated on a separate sheet of paper so that they can be handed to the examinee. This is particularly helpful for tasks with detailed initial conditions or those that will be performed in high-noise areas. Take care to ensure that the initial conditions and initiating cue do not reveal the nature of any attemate path JPMs that are planned. 1

2. Identifv References and Tools l The JPM shall identify those plant procedures that require task performance and the procedures that provide guidance, directions, or standards for performing the task.

When reviewing JPMs selected from the facility licensee's bank, it is important to ensure that the procedures identified in the JPM are still current. The JPM shall also identify any special tools or equipment (e.g., a stop watch, wrench, fuse puller, or spool piece) that the examinee will need to perform the task. It is helpful to the examiner who will be giving the test if the location in which these items may be found is stated in the JPM. It is expected that any required tools will be readily available to the plant operators; they should not be staged specifically for the examination.

3. Develoo Performance Criteria The JPM should have meaningful performance requirements that will provide a legitimate basis for evaluating the examinee's ability to safely operate the system or the plant. Artificially subdividing existing tasks to generate new ones may dilute the value of the JPMs to a point where they become meaningless.

The JPM shall identify specific performance standards, or check points, that will permit the examiner to evaluate successful progress toward completing the task in accordance with the procedural references. Detailed control and indication nomenclature and criteria (e.g., switch positions and meter readings) should be identified whenever possible, even if these criteria are not specified in the procedural step. The JPM should also note any important observations that should be made by the examinee while performing the task. NUREG-1021, Revision 8 2 of 10

1 L Appendix C The JPM must clearly identify the task standarct, i.e., the predetermined outcome

       . (qualitative and/or quantitative) against which task performance will be measured.

j Every procedural step that the examinee must perform correctly (i.e., accurately, in the l proper sequence, and at the proper time) in order to accomplish the task standard shall be identified as a criticalstep and shall have an associated performance standard, if there are any specific procedural restrictions on the sequence in which the steps are l performed they shall be clearly noted in the'JPM.

4. Develoo Examiner Cues The JPM shall identify appropriate system response cues so that the examiner can provide the examinee with specific feedback regarding the component and system reactions to the examinee's manipulations, especially those procedural steps identified as critical to task completion. The response cues are particularly important in the following situations:

for in-plant tasks that will be simulated because the examinee will not have available the normal indications (e.g., alarms, flow rates, temperatures, and pressures) that would be observed during actual task performance l for attemate path JPMs that require the examinee to perform auxiliary i procedures when equipment or instrumentation fails during use ,\

       - System response cues may not be necessary for those tasks that will be performed on the simulator, To the extent that it is possible to anticipate incorrect actions that the examinees might take, it is beneficial to note the expected system response cues in the JPM as an aid to the exan iner who will be administering and evaluating the task.

The JPM shall also identify any additional cues orinstructions that the examiner might need to provide to the examinee in response to procedural steps for which the examinee will not be held accountable; i.e., those steps that have either already been performed or will be performed by other personnel in remote locations. Sc Develoo a Time Standard Every JPM shall identify an estimated average time for completing the task. The time should be measured from the moment that the examinee is read the initiating cue at the plant location in which an operator would normally be given the order to perform the specified task. i JPMs that are considered time-critical (i.e., those having a task standard that must be completed within a time period specified in a regulation or a facility commitment to the NRC) shall be uniquely identified and specifically validated. The facility licensee must N 3 of 10 NUREG-1021, Revision 8 1 4 5

Appendix C agree that a failure to complete the task within the time specified will justify a failure of l the JPM. ! 1 C. DEVELOPMENT AND USE OF ALTERNATE PATH JPMs JPMs are intended to be tasks that an operator must be able to perform that are related to their particular job task analysis (JTA). Often times, operators are challenged to perform auxiliary procedures when equipment or instrumentation fails during use. Therefore, examinees are expected to be able to use alternative methods to perform tasks. Alternative paths are evaluated during an examination by incorporating malfunctions to instrumentation or components that require the examinee to perform actions other than those performed when a system responds normally. JPMs where malfunctions occur are used to provide a methodology to evaluate whether an examinee has the skills and knowledge at the level needed to safely operate the system. This type of JPM, called " alternate-path," provides an excellent opportunity to observe the examinees execute alternative paths within the wide spectrum of procedures under their cognizance that would not otherwise be examined. All alternate-path JPMs should include the following five characteristics:

1. Success Path - Each JPM should have a valid, facility endorsed success path. This path may require analyzing initial conditions to determine an alternative method for completing the task, mitigating a system-related problem that occurs during the task, or realigning the system. ,

2. Procedurally Driven - For each JPM, a procedure should address the actions that are required (i.e., if the JPM requires an alMmative method to complete the task, the procedure would have an exit step that aLects the use of this alternative method). The examinee may be required to use some common practices endorsed by the facility that are addressed through generic administra ve procedures or policies (e.g., shifting controls to manual).

3. Loaical Seauence - The sequence of procedurally driven actions should be logical. For example, an examinee performing a normal evaluation when a malfunction occurs should not be expected to enter emergency operating procedures (EOPs). More realistically, the examinee would attempt to correct the problem by referring to an annunciator response procedure (ARP) or abnormal operating procedure (AOP).

However, an examinee performing a normal evolution may encounter a situation requiring a reactor trip. The JPM should not contain a cascading sequence of malfunctions, for which several procedures must be used simultaneously, that occur while performing a task. This type of activity is better tested in the dynamic simulator portion of the examination.

4. Indeoendent of Crew Dynamics - Each JPM should allow the examinee to complete the task or mitigate a problem that occurs during a task without having to rely on the actions of other control room operators. This provision does not prohibit simulator operators NUREG-1021, Revision 8 4 of 10

O Appendix C V from acknowledging non-pertinent alarms or unexpected reactions of other systems not associated with the task. Also, the JPMs may still require the examinee to use the simulator operator to perform needed manipulations in the plant.

5. Validated in Advance - Each JPM should be validated before the examination begins and not changed after it begins. The JPM should not be a surprise to the examiners or the simulator operators. Each JPM should be validated as early as possible before the examination is to be administered to allow time for changes to be made.

D. WALK-THROUGH EVALUATION TECHNIQUES This guidance is intended to assist NRC examiners and facility evaluators in administering JPMs by illustrating good and bad examples of walk-through examination techniques.

1. Providina Cues Cuing refers to the information provided to an examinee by an examiner when conducting a JPM. When conducting JPMs on the simulator, the simulator provides most of the required cues. However, when conducting JPMs outside of the simulator, the examiner must provide realistic and timely information to the examinee.

a. Verbal Cues l

(~N \ () Many times verbal cues are required to provide relevant system information, such as valve position, meter deflection, or indicating light status. The examiner . has to be careful to provide the examinee with the indications that should be readily observed (9.g., "the red light just illuminated," "the valve position indicator does not move"). An excminer can give too much information or inappropriate information (e.g., providing indications that are not visible or audible to the examinee) that could invalidate the JPM. The examiner must keep in mind what the examinee would see and hear while performing the JPM, and provide consistent cues.

b. Non-Verbal Cues Maintaining a " poker face" when an examinee provides an ir correct response or performs the wrong procedural step is important. Voice inflections indicating something has been performed incorrectly, or changing the manner in which cues are given (e.g., talking more methodically, or rapidly) are examples of non-verbal communications that should be avoided.

Thorough preparation and familiarity with the JPM is vital to providing proper cuing. Knowledge of what indications will be available and how they will respond to the examinee's actions allow an examiner to give accurate and timely cues when an examinee is incorrectly performing the task.

5 of 10 NUREG-1021, Revision 8 I

Appendix C

2. Evaluation Skills When evaluating an examinee, an examiner must have the ability to differentiate between what he knows or believes to be true about an examinee's ability and how the examinee actually performs on the JPM. As previously discussed, an examiner must be familiar with the JPM to be able to accurately evaluate performance. Errors made by the examinee performing the JPM may not be seen or pertinent questions may not be asked if the examiner has not prepared for the examination.

An examiner must remain attentive to the examinee's actions at all times. This will ensure the examiner provides timely cues and detects errors in performance.

3. Exam Administration While conducting the walk-through examination, the examiner must be aware of conduct that is appropriate for a trainer, but is inappropriate for an examiner. As a trainer, interacting with the examinee during the performance of the JPM to gain insight into what the examinee is thinking is a good practice. However as an examiner, this is distracting to the examinee and may inadvertently result in prompting or leading the examinee.

When conducting JPMs in the simulator, examiners should not manipulate any controls or silence / acknowledge alarms. The examiner must take a " hands off" approach to maintain the proper testing environment. i The examiner must be careful to shield any notes or grading from the examinee to prevent giving an indication of performance, which may either provide a false sense of security or increase stress levels. If an examinee's actions are not clear, the examiner must be prepared to ask appropriate follow-up or clarifying questions. Documenting these questions and the subsequent answers is important as they may have a bearing on an examinee's overall grade. E. ATTACHMENTS / FORMS Form ES-C-1, " Job Performa1ce Measure Worksheet" Form ES-C-2, " Job Perforrrance Measure Quality Checklist" NUREG-1021, Revision 8 6 of 10 9,

E O Appendix C Job Performance Measure Form ES-C-i V Worksheet Facility: Task No: 1 Task

Title:

Job Performance Measure No: K/A

Reference:

j Examinee: NRC Examiner: Facility Evaluator: Date: Method of testina: Simulated Performance Actual Performance Classroom Simulator Plant READ TO THE EXAMINEE I will explain the initial conditions, which steps to simulate or discuss, and provide initiating cues. When you complete the task successfully, the objective for this job performance measure will be satisfied. Initial Conditions: Task Standard: Required Materials: General

References:

l Initiating Cue: Time Critical Task: YES/NO Validation Time: 7 of 10 NUREG-1021, Revision 8

Appendix C 2 Form ES-C-1 PERFORMANCE INFORMATION (Denote critical steps with a check mark) Performance step: Standard: Comment: Performance step: Standard: Comment: Performance step: O l Standard: I Comment: j I Terminating cue: l NUREG-1021, Revision 8 6 of 10 _ _ ___ o

     \ Appendix C                                3                      Form ES-C-1 VERIFICATION OF COMPLETION Job Performance Measure No.

Examinee's Name: . Examiner's Name: Date performed: Facility Evaluator: Number of attempts: Time to complete: ( Question Documentation: Question:

Response

Result: SAT or UNSAT I l

l Examiner's signature and date: l l i v 9 of 10 NUREG-1021, Revisico 8

Appendix C Job Performance Measure Form ES-C-2 Quality Checklist Every JPM should:

1. be supported by facility licensee's job task analysis.

2. be operationally important (meets NRC K/A Catalog threshold criterion of 2.5 (3 for requalification exams) or as determined by the facility and agreed to by the NRC).

3. be designed as either SRO only, RO/SRO or AO/RO/SRO.

4. include the following, as applicable:

a. initial conditions

b. initiating cues

c. references and tools, including associated procedures

d. _ validated time limits (average time allowed for completion) and specific designation of those JPMs that are deemed to be time-critical by the facility operations department

e. specific performance criteria that include:

(1) expected actions with exact control and indication nomenclature and criteria (switch position, meter reading), even if these criteria are not specified in the procedural step (2) system response and other cues that are complete and correct so that the examiner can properly cue the examinee, if asked (3) statements describing important observations that should be made by the examinee (4) criteria for successful completion of the task (5) identification of those steps that are considered critical (6) restrictions on the sequence of steps NUREG-1021, Revision 8 10 of 10

[ APPENDIX D SIMULATOR TESTING GUIDELINES A. PURPOSE This Appendix provides a framework for preparing and evaluating simulator scenarios to ensure they are of appropriate scope, depth, and complexity for initial operator licensing and requalification examinations. The following elements are discussed in detail or attached for information: a basic procedure for developing new simulator scenarios (Section B), including a description of the associated qualitative and quantitative attributes (Section C) and the critical task methodology (Section D) the competencies in which reactor operators (ROs) and senior reactor operators (SROs) are expected to be proficient (Section E) the simulator security considerations that should be kept in mind during scenario validation and administration (Section F) selected examples of initial and requalification scenarios (Attachments 1 and 2) Adhering to the concepts and guidelines discussed herein, in association with the specific criteria cited in ES-301 or ES-604, as applicable, will enhance the consistency and validity of the dynamic simulator operating tests. B. INTEGRATED SCENARIO DEVELOPMENT The major activities applicable to the development of dynamic simulator scenarios are summarized below. The instructions apply to both the initial and the requalification examination programs, except as noted. Although they are written from the perspective of new scenario development, the instructions should also be referenced, as nacessary, when modifying existing scenarios for reuse and while reviewing proposed scencrios for quality.

1. Identify Scenario Obiectives A scenario should explicitly identify its objectives. For a requalification examination, these should come, in part, from the facility's requalification training program objectives.

However,10 CFR Part 55 requires that the initial licensing and the annual requalification operating tests be a comprehensive sampling of items (2) through (13) listed in 10 CFR 55.45. Therefore, both tests should sample from all the operating skills and abilities required of an operator and the operating crew. Limiting the requalification examination to topics covered in the requalification cycle is not sufficient. The basic objective of a scenario should be to evaluate the operators' ability to respond to events that are most appropriately tested in a dynamic simulator environment. These are events that require the operators to demonstrate their knowledge of integrated plant operations, diagnose abnormal plant conditions, and demonstrate their ability to work 1 of 40 NUREG-1021, Revision 8

l l l Appendix D together and to mitigate plant transients that exercise their knowledge and use of AOPs and EOPs. Additionally, the scenario should require the operators (usually the SROs) to utilize technical specifications (TS) and, for requalification examinations, to implement the emergency plan. The full range of competencies in which the operators must demonstrate proficiency during the simulator test are described in Section E of this Appendix. Briefly describe the objectives in the space provided at the top of Form ES-D-1,

    " Simulator Outline," cr equivalent. Also enter an identifying number for the scenario and the name of the plant after which the simulator is modeled.                                       j i

2. Select Initial Conditions initial conditions must be established that will allow the scenario to commence i realistically. The initial conditions should be representative of a typical plant status, with i various components, instruments and annunciators out of service. To have maintenance or surveillance activities in progress is realistic. All, some, or even none of these initial conditions may have a bearing on subsequent scenario events. Initial conditions should be frequently changed, to prevent predictability of future events. The initial conditions should be varied among the scenarios and should include startup, low-power, and full-power situations.

Briefly describe the initial conditions, including any items that should be addressed , during the shift turnover, in the space provided at the top of Form ES-D-1, or equivalent.

3. Select and Document Events Once the initial conditions are established, select a sequence of events designed to attain the stated objectives. Section C discusses a number of qualitative and quantitative criteria that should be considered when selecting events. The specific requirements for each quantitative criterion are enumerated in ES-301 and ES-604, as applicable.

Each event should have or contribute to an objective, whether it is to evaluate the operators' knowledge of a recent system modification, evaluate their ability to respond to a safety-significant event, or assess their use of TS for a particular safety-related component. Uncomplicated events that require no operator action beyond the acknowledgment of alarms and verification of automatic actions provide little basis for evaluating the operators' competence and should not be included on the operating test unless they are necessary to set the stage for subsequent events. The scenarios should be developed so that a variety of systems is affected within each type of event (i.e., normal evolutions, instrument failures, component failures, and major plant transients). Having one equipment failure cause or exacerbate another can be used to evaluate the operators' understanding of system and component interactions. Balancing the severity of events and the demands they place on each operating position NUREG-1021, Revision 8 2 of 40

Appendix D (e.g., RO and BOP) will allow each operator to demonstrate his or her competence across a range of conditions. All events do not have to be linked, that is, one event need not occur for the next event to logically occur, although in many instances, such a relationship adds to the credibility of the scenario. However, the scenario should not consist of a series of totally unrelated events. A well-crafted scenario should flow from event to event, giving the operators sufficient time in each event to analyze what has happened, evaluate the consequences of their action (or inaction), assign a priority to the event given the existing plant conditions, and determine a course of action. Exercise care that one event does not fully mask the symptoms of another because the operators could overlook the malfunction and cause the event or competency coverage for the scenario set to be deficient. Record each planned operation, malfunction, and transient on Form ES-D-1 and number them sequentially.' Cross-reference each event to a simulator malfunction number, if applicable, or briefly describe the simulator instructions that must be entered. For each event listed on Form ES-D-1, prepare a Form ES-D-2, " Operator Actions," (or equivalent) by entering the scenario, event, and page numbers and a brief description of i the event at the top of the form. Each event description should include when it is to be l initiated, whether by signal of the lead examiner / evaluator, time line, or plant parameter. l The form shall also identify the symptoms or cues that the operators will be provided, the expected actions to be taken, communications to be made, the references to be used by each operating position (e.g., the SRO, RO, and BOP operators) on the crew, and the event terminus (i.e., the anticipated point at which the examiners or evaluators will have enough information on operator performance to move on to the next event). l Every expected operator action should be included on the form, particularly the critical i tasks (refer to Section D, " Critical Task Methodology") and other verifiable actions and

behaviors that will provide a useful basis for evaluating the operators' competence.

Critical tasks (cts) shall be flagged in a manner that makes them apparent to the individuals who will be administering the operating test (e.g., by using underlines, asterisks, or bold type) and the measurable performance indicators shall be identified. When possible, set points and other parameters should be included to provide an objective method for evaluating the operators' performance. Statementa such as

    " Performs actions in accordance with Procedure XXXXX" generally do not provide j    sufficient guidance and are inadequate. However, the statement " Performs actions of l    steps XXX of Procedure XXX (attached)" is acceptable.

1 Although the expected actions should, to the extent possible, be listed in chronological order, certain actions may be required throughout the event (for instance, if a safety or relief valve fails open, the operators should continually monitor pressure and water level). Flag these actions to show that they are continuous. i 3 of 40 NUREG-1021, Revision 8

l Appendix D The expected actions on Form ES-D-2 should be widely spaced to leave room for notes l to document the operator's performance during the simulator test. The far-left column of the form should also be left blank so that it may be used to record the actual time at which key actions occurred while giving the test. i

4. Determine the Scenario Endpoint j The last cperator action sheet (Form ES-D-2) in the scenario should specify the {'

endpoint of the scenario by identifying a particular plant condition, procedural step, or other point that is clearly recognizable. The scenario should not be terminated until the stated objectives have been achieved.

5. Validate the Scenario l Every scenario should be validated to ensure that it will run as intended. If a previously validated scenario is being modified slightly, real-time validation may not be necessary.

However, if there are major changes or if someone questions the validity, revalidation in real-time is recommended. C. SCENARIO ATTRIBUTES All valid scenarios contain common elements that make them useful as evaluation tools. A properly constructed scenario provides for an accurate test of each individual operator's skills and abilities as well as an opportunity to evaluate the crew members' team-dependent skills and abilities. The scenario should be of sufficient scope and complexity to demonstrate the difference between competent operators and crews and those that are not performing at an acceptable level. It also should require that the crew demonstrate its ability as a team to adequately protect the public health and safety in emergency conditions, using the facility's emc gency operating procedures (EOPs). Scenario attributes can be characterized as both qualitative and quantitative. No single qualitative of quantitative attribute or group of attributes can be used to determine the . acceptability of a scenario. However, a trained examiner should be able to assess of the adequacy of a sce: irio or develop a new scenario, using both sets of attributes. This assessment, combined with validation of the scenario on a real-time basis, should be sufficient to determina if a scenario provides an accentable tool to measure the competency of a crew and/or its irdividual members.

1. Quahtative Attributes

a. Realism / Credibility introducing unrealistic or incredible events into a scenario can affect the validity of the scenario and provide negative training. Piping, component, and instrument failures often occur in such a way that deterioration can be tracked over a discrete time period (e.g., a small leak that propagates over time or a NUREG-1021, Revision 8 4 of 40 G],

p l. l L i Appendix D pump failure preceded by a high vibration condition). Including such precursors ) into scenarios is important, where appropriate. A great deal of evaluative 4 feedback can be obtained by obsenting how an operator and crew responds to a l. gradually worsening condition. A good technique inserts an event precursor ) (e.g., small steam generator tube leak) and maintains the plant at a slightly degraded condition to observe how the crew incorporates that condition into its L conduct of subsequent plant operations. Although faults that occur with little or no warning (e.g., valve operators fail, fires occur in breakers or transformers, undetected pipe erosion results in piping ! failures) may be included in scenarios, they often provide minimal evaluative benefits because they happen so suddenly that operators have little to do but watch the event unfold.- These events are most useful when trying to establish a plant condition for subsequent evaluation goals or to assess an operator's or a l crew's ability to use procedures in a symptom-based rather than an event-based mode. I i Mechanistic component failures are well documented events that occur each year, many times in multiple numbers. However, non-mechanistic failures (e.g., pipe breaks) generally occur singularly; therefore, unless there is a connective precursor, such as a seismic event, it would not be realistic or credible to have L several piping systems fail during any one scenario. Simulated events that appear to violate the laws of physics and thermodynamics l contribute to negative training and are to be avoided. Time compression techniques, which are discussed later, may contribute to negative training. However, if the intent of a scenario is to evaluate a crew's ability to execute l procedural steps that may take a long time to reach during an event (e.g., ! hydrogen generation during a core uncover event), such a technique may be useful. However, the scenario must contain a cue that, when the indications for l such events are detected by the crew, the crew is informed that the parameters are not responding as expected for the actual plant and that time is being compressed. This cue should be presented at the first opportunity that does not distract the crew from responding to available indications and before the crew l challenges the validity of the indications. For example, in the first PWR scenario (Attachment 1), the cue should be given following the crew's determination that a o reactor coolant system (RCS) feed and bleed may be necessary (per FR-H.1) ) but prior to steam generator levels requiring initiation. I l

j. b. Event Sequencing l

The sequence of events has a major effect in establishing the complexity of a simulator scenario. The pace at which malfunctions are entered can adversely affect the way an operator or a crew responds. O,N NUREG-1021, Revision 8 5 of 40

Appendix D Malfunctions may be entered simultaneously at separate control panel locations if each event can be handled by an individual applicant and does not require extensive assistance. Too short a time between malfunctions may mask the effects of a particular malfunction and divert the operators' attention. This cuts short the obt.ervers' ability to evaluate the operators' response to the prior malfunction and may be prejudicial to a fair evaluation. Conversely, extending the time between malfunctions so that no operator activity is in progress may cause undue stress. During an examination, the operators expect something to occur; too much time between events should be avoided. Therefore, insertion of malfunctions in the scenario should be carefully timed. Rigorously following a planned time sequence of events is often less valid than initiating malfunctions on the basis of plant parameters or operator actions. The appropriate sequencing of events relates directly to the objectives of the scenario. Event sequencing may involve time compression to speed up the response of key parameters so that the scenario can proceed to the next event within a reasonable time. Time compression may be accomplished by adjusting parameter indications or accelerating plant behavior characteristics so that an event is triggered by plant indications more quickly than would typically occur in reality (e.g., opening a drain path from a steam generator that is not noticeable to the operator so that the entry conditions for a loss of heat sink are reached.) This method is acceptable as long as the time compression allows the operators time to perform tasks that they would typically perform during the period in which time is compressed. To avoid wasting the operators' time determining the validity of their indications, the crew should be informed before the scenario begins that time compression may be used during an event and debriefed after the scena;io to minimize the potential for negative training. Frequently, important evaluative benefit in terms of safety significance is gained by having key components or instruments fail after entering the EOPs. This process compels the operators to respond immediately to a safety-related situation by taking alternate actions to mitigate the event. This process also allows for a better evaluation of the operators' overall knowledge of plant procedures and systems because the event must be incorporated into the mitigation strategy for the remainder of the scenario. Conversely, instrument and component failures that are initiated after the major transient sometimes require little action and may provide little insight to the operators' competence.

c. Simulater Modeling Despite the certification of simulators to a set standard (ANSI /ANS-3.5), not all simulators are equally capable of performing major transients. The scenario should not exceed the limits of the facility licensee's configuration management system by altering a simulator model to obtain a desired effect. For example, NUREG-1021, Revision 8 6 of 40 O1 l

l

 , - ~ . .

Appendix D increasing the post-trip decay heat input in order to maximize intemal core temperatures during a loss of cooling event is not appropriate; the simulator model should be allowed to perform as designed. The scenario may simulate s events for which a simulator malfunction does not exist by using overrides or ) remote functions for local operator actions. An example would be failing indicators to simulate an inoperable component.

d. Evaluating Competencies Each scenario est shall ensure that all of the rating factors within each competency can be evaluated and that, if the crew performs poorly, their performance could cause plant degradation or threaten the public health and safety. Therefore, events must be incorporated into the scenario that will allow an unsatisfactory evaluation of an operator or crew in a particular rating factor if they perform poorly. Scenarios that require little analysis or problem-solving and few operator actions may not provide an adequate basis to evaluate the required rating factors.

The individual competencies applicable to the RO and SRO license levels during initial and requalification examinations are described in Section E; the rating factors within each competency are identified in ES-303 (specifically, on Forms ES 3013 and ES-303-4 for RO and SRO applicants, respectively). The crew compancies applicable to requalification examinations only are identified in ES-604~

e. Levelof Difficulty The dynamic simulator operating test must discriminate between those examinees who have and those who have not adequately mastered the knowledge, skills, and abilities required to be licensed operators. Simulator scenarios that are either too easy or difficult are not effective discriminators.

In general, as the quantitative attributes, such as the number of malfunctions or critical tasks (discussed below), of a simulator scenario increase, the level of difficulty of the scenario willincrease as well. However, counting the number of scenario quantitative attributes is not always indicative of the scenario's level of difficulty; two scenarios having the same quantitative attributes can vary significantly in level of difficulty. There are no definitive minimum or maximum attribute values that can be used to identify inappropriate scenarios that will not discriminate because they are too easy or difficult. The two most important detenninants of the level of difficulty of a simulator scenario are the amount of analysis and problem-solving and the number of operator actions required to mitigate the events in the scenario. Malfunctions that require analysis or problem solving increase the level of difficulty because they require the examinees to integrate a number of system conditions, evaluate n their interrelationships, and take actions that demonstrate an understanding of 7 of 40 NUREG-1021, Revision 8

Appendix D the underlying knowledge. Scenarios that consist of a number of unrelated malfunctions that require little or no operator analysis or response are generally less challenging.

2. Quantitative Attributes Those traits discussed in the previous section provide for a qualitative assessment of the complexity of a simulator scenario. Hoviever, there are some characteristics of a scenario that can be quantified and that generally have a bearing on the complexity and level of difficulty of the scenario. These characteristics are described below, and a target range for sach trait that is applicable to the initial and requalification examination is enumerated in ES-301 and ES-604, respectively. The ranges are not absolute limitations; some scenarios may be an excellent evaluation tool but may not fit within the ranges. A scenario that does not fit into these ranges should be evaluated to ensure the ,

( scenario is appropriate.

a. Normal Evolutions i

Normal evolutions include activities such as a feed pump sartup, turbine loading, generator synchronization, and reactivity manipulations, which include evolutions such as a reactor startup or changing power with boron concentration, control rods, or core flow. Reactivity manipulations are considered significant if they l produce a clearly observable plant response, such as bringing the reactor critical from a substantially subcritical state, raising power to ti'e point that reactivity r feedback from nuclear heat addition is noticeable and a heatup rate is established, or changing reactor power manually with control rods or recirculation flow. Normal evolutions can be used as a backdrop on which to stage the emergency or abnormal situations. For example, a main feedwater control valve may fait passively (i.e., as is) before the operators conduct a normal power change. Time consuming normal evolutions such as a power escalation from low power can provide an opportunity to evaluate the SRO's supervisory or resource management skills. Events such as component or instrument failures may be added to challenge the operators while continuing the power escalation. Short surveillances (e.g., exercising safety rods or paralleling the emergency diesel generator with the grid) may be used to examine the operators' dexterity on the control panels or to involve operators who are not engaged in other activities.

b. Total Malfunctions l Total malfunctions are the number of instrument (e.g., nuclear, control, or

) process) and component failures (e.g., pump, motor, valve, or pipe) used to NUREG-1021, Revision 8 8 of 40 O1 _______-_-_-__--_______-______-___________________--______________________________-________________-____-_________m

Appendix D o initiate the events that constitute a scenario, including those initiated after EOP entry (see item C.2.c below). To count as a separate malfunction, they must involve a significant system response and require operator action to correct. For example, an anticipated transient without scram or trip (ATWS/ATWT) is a single malfunction, regardless of how many instructions a simulator operator must program to produce it. Components that are placed out of service at the beginning of a scenario as part of the shift tumover conditions, and which the crew is made aware of, are not considered malfunctions. Component or instrument failures that require no operator actions or response do not count toward the recommended total number of malfunctione.

c. Malfunctions After EOP Entry Some malfunctions should result in vital instruments or components failing after the EOPs have been entered (these may have been inoperable at the beginning (

of the scenario or before EOP entry) and influence the operators' choice of I mitigation strategy. For example, failing a high head safety injection (SI) pump to start on a large-break loss-of-coolant accident (LOCA) does not affect the mitigation strategy; however, this would have an effect if it were the only available high head SI pump on a small-break LOCA. I' d. Abnormal Events Each scenario should evaluate the operators' ability to implement abnormal operating procedures (AOPs). An abnormal event may or may not be a precursor to the major transient (see item C.2.e below), although it can add to the credibility of a scenario, such as preceding a total loss of feed water with a single feed pump trip. However, certain events may cue the operators about subsequent events. Therefore, if a scenario is derived from the facility licensee's bank, it is wise to vary or modify the precursor events that lead to the major transient. It is also good to insert abnormal events that are not always predictive of the same major transient (e.g., a steam generator tube leak does not always lead to a subsequent tube rupture). Some abnormal events for each scenario should require that the operators recognize and interpret technical specifications. This recognition and interpretation can also be incorporated into the scenario by designating TS-related equipment that is out of service at the start of the scenario. Components or instrument failures that occur following EOP entry do not count toward the recommended total number of abnormal events. I O 9 of 40 NUREG-1021, Revision 8 l i

Appendix D

e. Major Transients A major transient is one that has a significant effect on plant safety and that leads to an automatic (or manual, if initiated by an operator) protective system actuation, such as a reactor trip or an engineered safety system actuation. A single major transient that actuates more than one automatic protective system actuation will be counted as a single major transient. Examples include loss of offsite power, LOCA, steam or feed line break, steam generator tube rupture, and loss of feed water. A major transient should normally involve activation of the facility's emergency plan.

f. EOPs Used A scenario that requires the operators to refer to many different EOPs may not be as complex as a scenario for which only one EOP is used, but which requires use of attemative decision paths and prioritization of actions within the EOP to deal with the situation. Therefore, this attribute should reflect the EOPs that have measurable actions that the crew must take. Moreover, the primary scram response procedure that serves as the entry point for the EOPs is not counted.

For BWRs, the number of "EOPs Used" should be counted consistent with the following four top level guidelines of the Emergency Procedures Guidelines:(1) RPV Control, (2) Primary Containment Control, (3) Secondary Containment Control, and Radioactivity Release Control. Use of multiple control sections of the above listed guidelines do not count separately as "EOPs Used." For example, use of RPV level control and RPV pressure control should be counted as one EOP Used - RPV Control.

g. EOP Contingency Procedures Used Contingency procedures are used when there is a challenge to a critical safety function or if plant conditions have become severely degraded. Therefore, using them in a scenario provides an opportunity to observe the operators attempt to execute a mitigation strategy that clearly has safety significance to the plant and tha public health and safety. Each scenario set should require the operators to enter and perform safety-related tasks within an EOP contingency procedure at least once.

The following list of contingency procedures is not unique or all-inclusive. Scenario developers and reviewers should consider it as a set of general guides that may not fully apply to all scenarios. I (1) Westinahouse Optimal Recovery Procedures designated as Emergency Contingency Action (ECAs) procedures: NUREG-1021, Revision 8 10 of 40

m Appendix D Loss of All AC Power With or Without SI Required Loss of Emergency Coolant Recirculation LOCA Outside Containment Uncontrolled Depressurization of All Steam Generators Steam Generator Tube Rupture (SGTR) With Loss of Reactor Coolant-Subcooled Recovery SGTR With Loss of Reactor Coolant-Saturated Recovery SGTR Without Pressurizer Pressure Control Functional Recovery Procedures entered as a result of RED or ORANGE conditions on the Critical Safety Function Status Trees: Response to Nuclear Power Generation /ATWS Response to inadequate Core Cooling Re.sponse to Degraded Core Cooling Response to Loss of Secondary Heat Sink Response to Imminent Pressurized Thermal Shock Conditions Response to High Containment Pressure Response to Containment Flooding (2) Combustion Enaineerina Entry into Functional Recovery Procedures (FRPs) O - Transition among Functional Recovery Safety Function success paths Transition from one safety function to another within the FRPs 1 (3) Babcock and Wilcox (B&W)

The B&W EOP structure does not identify procedures that can be easily i recognized as contingency procedures. However, use of the descriptions l given above for Westinghouse contingency procedures provides guidance on the type of events to be considered. (4) General Electric Attemate Level Control Emergency Reactor Pressure Vessel (RPV) Depressurization Primary Containment Flooding Level / Power Control RPV Flooding Steam Cooling

h. Simulator Run Time A scenario should be designed to run approximately 60 to 90 minutes. However, this does not preclude scenarios taking more or less time. The nominal run time of 60 minutes may not provide sufficient time to conduct a scenario that U 11 of 40 NUREG-1021, Revision 8

I l Appendix D progresses through several EOPs or that requires performance of fairly involved procedural steps. It is possible to conduct very meaningful and involved scenarios in less time, but care should be taken not to place an undue burden on the operators by initiating malfunctions at too rapid a pace. This parameter is one of many that should be considered in assessing the overall quality of a scenario, and as long as the scenario meets the other criteria stated in this guidance, the scenario run time is a secondary concern.

1. EOP Run Time The time during which the operators are involved in EOPs has a strong relationship to the complexity of the scenario because most critical tasks occur in the EOPs and the actions the operators take have the most potential for affecting the health and safety of the public. Therefore, a significant percentage of the time a scenario is progressing should be spent in the EOPs. Usually, more time is required when contingency procedures are in effect, because it generally takes some time for the plant to degrade to a point where critical safety functions are jeopardized. However, operators should be evaluted in EOP activities beyond the point at which an event is diagnosed and initial mitigation actions are taken. Many of the actions taken to stabilize the plant and recover from a transient are safety significant. Therefore, scenarios should be allowed to progress so that these operations can be observed.

Scenarios should not be just EOP oriented. Valuable assessments can be made within AOPs with the plant at power because of the level of safety significance associated with transients in these conditions. J. Critical Tasks Critical tasks range between fairly simplistic but safety-signifcant tasks (starting the standby liquid control system on an anticipated transient without scram condition or tripping a reactor coolant pump during a small-break LOCA) and other tasks that require a much higher level of skill involving several crew members (executing a rapid cooldown within predefined limits using steam generator power-operated relief valves or using low pressure injection systems to maintain the vessel level while cooling the suppression pool). Therefore, the difficulty level must be considered to judge the appropriateness of the number of cts in a scenario or scenario set. Refer to Section D for a detailed explanation of the critical task methodology. D. CRITICAL TASK METHODOLOGY The requalification examination uses critical tasks (cts) for evaluating crew performance on tasks that have safety significance to the plant or the public. The cts are objective measures for determining whether an individual's or a crew's performance is satisfactory or unsatisfactory. NUREG-1021, Revision 8 12 of 40 I l

l I 1 Appendix D j h On the initial licensing examinations, cts are used to provide a basis for the individual operator i competency evaluations because they help the examiner focus on those tasks that have a ] significant impact on the safety of the plant or the public. Refer to ES-303 and ES-604 for i specific instructions on the use of cts in grading initial and requalification examinations.

1. Identification of Critical Tasks A critical task must include the following elements:

a. Safety Significance in reviewing each proposed CT, assess the task to ensure that it is essential to safety. A task is essential to safety if the improper performance or omission of this task by an operator will result in direct adverse consequences or in signiixant degradation in the mitigative capability of the plant.

l If an automatically actuated plant system would have been required to mitigate the consequences of an individual's incorrect performance or the performance necessitates the crew taking compensatory action that would complicate the event mitigation strategy, the task is safety significant. Examples of cts invoMng essential safety actions include those for which operation or correct performance prevents the following: degradation of any barrier to fission product release degraded emergency core cooling system (ECCS) or emergency power capacity a violation of a safety limit a violation of the facility license condition incorrect reactivity control (such as failure to initiate emergency boration or standby liquid control, or manually insert control rods) a significant reduction of safety margin beyond that irreparably introduced by the scenario

                       ' Examples of cts involving essential safety actions include those for which a crew demonstrates the following abilities:

effectively direct or manipulate engineered safety feature (ESF) controls that would prevent any condition described in the previous paragraph recognize a failure or an incorrect automatic actuation of an ESF system ; or component i O- ! Q 13 of 40 NUREG-1021, Revision 8 l I C

Appendix D take one or more actions that would prevent a challenge to plant safety prevent inappropriate actions that create a challenge to plant safety (such as an unintentional reactor protection system (RPS) or ESF actuation)

b. Cueing For a CT to be valid, an external stimulus prompts at least one operator to perform the task. A cue prompts the operators to respond by taking certain actions and provides the initial conditions. The cue need not indicate the task as

          " critical."

Appropriate cues include the following examples: verbal direction by or reports from other crew members procedural steps, such as satisfying entry conditions, flow chart decision points, and " response not obtained" columns indication of a system or a component malfunction (including passive failures) by meters or alarming devices

c. Measurable Performance Indicators A measurable performance indicator consists of positive actions that an observer can objectively identify taken by at least one member of the crew.

The NRC and facility licensee should review each critical task to ensure it is objective. For example, "If pressure falls below 1400 psi, start pump xyz," is a performance measure that is not objective. The operator performing this task could conceivably start the pump when pressure reaches zero psi and still not violate the performance measure stated in the procedure, even though the facility licensee expects the operator to start the pump sooner. The NRC and facility licensee should agree in writing that the limits for each CT are acceptable before the requalification examination begins. For the example given above, adding an acceptable pressure tolerance (e.g., within 200 psi) would clarify the standard of performance that is expected. Measurable performance indicators include the following examples:

           -         actions taken as the result of transitioning in emery 9 operating procedures (EOPs), for example transitioning to ano pdorming the actions required in FR-S.1 if the reactor does not trip (Westinghouse), or performing an automatic depressurization after confirming indications of   i high suppression pool temperature (General Electric) l NUREG-1021, Revision 8                       14 of 40

Appendix D

control manipulations such as a manual reactor trip or the start of an ECCS pump verbal reports or notifications of abnormal parameters or conditions such as "all control rods are not inserted" or " containment pressure is greater than 2 psi" The following are examples of performance indicators that cannot be measured objectively during a simulator scenario: understanding, such as of the significance of a certain plant response verification that an expected response has occurred - passive observations such as monitoring the performance of a system

d. Performance Feedback Each CT must provide at least one member of the crew with performance feedback. The feedback provides the crew member with information about the effect of the crew's actions or inaction on the CT. This requirement must be met for all cts.

t 2. Critical Tasks as " Generic' Safety Tasks Avoid assigning the 'CT' designation to generic tasks that have safety significance but that do not meet all of the criteria required to identify a critical task. Although a crew is not performing optimally if it fails to anticipate an automatic action given sufficient time to assess plant behavior, crew members are not required to anticipate an automatic action. A crew member may, at any time, take manual action in advance of an automatic action if, in the crew member's judgement, manual action is i needeu + r the reactor in a safe condition. If an operator takes an action that the examiners did not expect, the examiners must further evaluate the individual's rationale for taking those actions. This preemptive action may indicate a misunderstanding of plant conditions or a weakness in integrated plant knowledge that should be clarified with follow-up questions. , l Taking manual control of an automatic safety system qualifies as a CT only if the auto-initiation feature fails to work. It is then safety significant for the crew to take manual actions, as plant conditions clearly indicate that an automatic action should have : occurred and did not. Moreover, during scenario development and validation, ! identification of cts is based on those actions which, if performed incorrectly or omitted,

       - degrade the mitigation strategy needed in the scenario. If the manual system has also failed and no action will be effective, this should not be identified as a CT. However, if an operator or the crew significantly deviate from or fail to follow procedures affecting 15 of 40                     NUREG-1021, Revision 8

Appendix D assurance of basic safety functions, those actions may form the basis of a CT identified in the post-scenario review. Experience has shown emergency event classification to be an important evaluation area, but generally not a critical task. The argument is made that an incorrect classification could adversely affect the public health and safety if the appropriate instructions are not given to public service agencies in a timely manner. If a misclassification occurs, the emphasis for corrective action is placed on the facility licensee and an appropriate period allotted for implementation of the corrective actions. Therefore, although emergency classification is still an area that is to be evaluated, it should not receive the weight of a critical task. If a misclassification occurs, the examiners should determine the rationale used to establish the classification in order to determine if the crew understood the status of the plant and incorporate into the program evaluation those pertinent corrective actions deemed appropriate. If a widespread problem during a program evaluation is observed, the examiner should share this information with other inspection program managers. E. COMPETENCY DESCRIPTIONS

1. Reactor Operator

a. Understand and Interpret Annunciators and Alarm Signals This competency involves the ability to notice and acknowledge alarms. It includes the abilities to prioritize one's attention in keeping with the severity and importance of annunciators and alarm signals and to correctly interpret and verify that signals are consistent with plant and system conditions (with the use of alarm response procedures, as appropriate). This competency deals strictly with the understanding and interpretation of annunciators and alarm signals and, therefore, does not include knowledge of, or the ability to diagnose, overall plant and system status on the basis of other indications.

b. Diagnose Events and Conditions Based on Signals and Readings This competency involves the ability to accurately and promptly recognize and analyze off-normal tren is and diagnose plant conditions to guard against and mitigate conditions that are out of specification. It includes the use of control room reference materials, such as prints, books, and charts, to aid in the diagnosis and classification of events and conditions. It does notinclude knowledge of system operation, such as set points, interlocks, or automatic actions, or the understanding of how one's actions affect the plant and system conditions.

NUREG-1021, Revision 8 16 of 40

A Appendix D

c. Understand Plant and System Response This competency involves knowledge of system operation, including set points, interlocb and automatic actions. It includes the abilities to locate andinterpret plant an 1 ' u%m instruments and to understand how one's actions affect plant and syst - snditions. It does not include the ability to notice or attend to annuncia . , and alarm signals or the ability to diagnose or classify events and conditions on the basis of control room indications.

d. Comply With and Use Procedures and Technical Specifications This competency involves the ability to refer to and comply with normal, abnormal, emergency, and administrative procedures in a timely manner (i.e., in sufficient time to avoid adverse impacts on plant status). It includes the ability to recognize emergency operating procedure entry conditions, cany out immediate actions without assistance, and recognize and comply with required ilmiting conditions for operation and action statements.

e. Operate the Control Boards This competency involves the ability to locate and manipulate controls to attain a desired plant and system response or condition. It includes the ability to take manualcontrolof automatic functions, when appropriate.

f. Communicate and Interact With Other Crew Members This competency involves the ability to provide and receive pertinent information, both oral and written (e.g., log entries). It includes the ability to cany out l supervisory instructions and to interact with other crew members with respect to conditions affecting safe plant operation regardless of which applicant's control board is directly affected. ,

l I

2. Senior Reactor Operator

a. Understand and Interpret Annunciators and Alarm Signals This competency involves the ability to notice and attend to alarms. It inJ ; des the ability to prioritize one's attention in keeping with the severity and importance of the annunciators and alarms and the ability to correctly interpret the significance of each alarm and verifythat it is consistentwith plant and system conditions (with the use of alarm response procedures, as appropriate). This competency deals strictly with the understanding and interpretation of annunciators and alarm signals and, therefore, does notinclude knowledge of, or the ability to diagnose, overall plant and system status on the basis of other indications.

17 of 40 NUREG-1021, Revision 8 l 1 l

f l Appendix D

b. Diagnose Events and Conditions Based on Signals and Readings l

This competency involves the ability to diagnose plant conditions to guard ) against and mitigato conditions that do not meet specifications. It includes the i ability of both the supervisor and the crew to recognize and analyze off-normal trends in an accurate and timely manner and to use control room reference i materials, such as prints, books, and charts, to aid in diagnosing and classifying events and conditions. It does notinclude knowledge of system operation, such as set points, interlocks, or automatic actions, or the understanding of how one's actions affect the plant and system conditions.

c. Understand Plant and System Response This competency involves knowledge of system operation, including set points, interlocks, and automatic actions. It includes the ability to locate and remain attentive to control room indicators, interpret those indicators to verify the status and operation of systems, and understandhow one's actions and directives affect plant and system conditions. It does not include the ability to notice or attend to annunciator and alarm signals or to diagnose or classify events and conditions on the basis of control room indications.

d. Comply With and Use Procedures This competency involves ihe ability to refer to and comply with normal, abnormal, emergency, and administrative procedures in a timely manner (i.e., in sufficient time to avoid adverse impacts on plant status). It includes the ability to use proceoures correctly and ensure correct implementation by the crew.

e. Operate the Control Boards This competency involves the ability to locate and manipulate controls to attain a desired plant and system response or condition. It includes the ability to take manualcontrolof automatic functions, when appropriate.

f. Communicate and Interact With the Crew and Other Personnel This competency involves the ability to provide and receive pertinent information in a clear, easily understood manner. It includes the ability to keep crew members andpersonneloutside the control room informedof plant status.

g. Direct Shif' Operations This competency involves the ability to take timely and decisive actions in response to problems during both normal and off-normal situations. It includes the ability to provide timely and well thought out directions that indicate concem for safety; to encourage a team approach to problem solving and decision NUREG-1021, Revision 8 18 of 40

c Appendix D making by soliciting andincorporating feedback from members of the crew; and to remain in a position of oversight to maintain the " big picture."

h. Comply With and Use Technical Specifications This competency involves the ability to recognize when conditions are covered by technical specifications, it includes the ability to locate the appropriate technical specification and ensure correct compliance with any limiting conditions for operation and action statements. ,

i F. SECURITY CONSIDERATIONS FOR SIMULATOR OPERATING TESTS Simulators present a unique set of integrity concems during the development and administration of operating tests. NRC examiners and facility licensees should be aware of the j simulator's vulnerabilities and take appropriate measures to ensure that operating test security ' is maintained in three areas: the instructor station, the programmers' tools, and the external interconnections. Because facility licensees are more familiar with their simulator's unique _ capabilities, limitations, and vulnerabilities than the NRC examiners are, it is expected that the licensees will take responsibility for determining and implementing whatever measures might be necessary to ensure the integrity of the operating tests. Most of the instructor station features can be checked through the tableau or graphic interface n provided at the instructor's console. The programmers' tools and the extemal interconnections are not generally apparent to the instructor or the examiner. The simulator staff should be (V) consulted to determine the status of those items.

1. Instructor Station Features Snapshots - All simulators have snapshot capability. Initial conditions (ICs) are recorded for future recall.

Backtrack- Backtrack files are snapshots that are automatically recorded at pre-determined intervals, usually up to 1 hour of operation at intervals as frequent as 1 minute. Backtrack files are usually only accessible through the BACKTRACK feature. The files typically cannot be erased, only overwritten by real-time operation. i Replay / Playback- The replay / playback feature steps through a' series of l snapshots and displays the I/O status (lights, meters, etc.) for each sequentially. ; Often, the replay feature uses the backtrack files, although separate replay file storage may be provided. Scripts / Computer Assisted Exercises - Many simulators have a feature that allows pre-programmed implementation of malfunctions and remote functions based on time and/or logical conditions. Scripts may be used by the simulator staff to facilitate scenario administration. Scripts can typically be stored fcc future use. Stored scripts can also be selected for review and editing from the instructor station. 19 of 40 NUREG-1021, Revision 8

Appendix D initial Conditions Summary- Snapshots are usually labeled on the instructor station IC menu with date/ time recorded, pertinent plant parameter status, and instructor comments. Even if the comment field has been changed to indicate that a snapshot is available for re-use, the data (scenario initialization) may still be representative of test conditions until the snapshot is actually overwritten or updated. Malfunction Summary- Malfunction summary menus display the status of selected malfunctions, both active and inactive. The malfunction summary is usually IC dependent and therefore depicts the malfunctions that were active or staged when an IC, such as a scenario validation, was stored. i i Monitored Parameters - Instructors are afforded the capability to define individual or groups of parameters for display or printout. The monitored parameter group assignments can be recalled for review and editing. If used to facilitate scenario validation or examination administration, the monitored parameters can provide insight into the focus of the examination. Trend Recording- Groups of parameters can be defined and assigned to trend recorders. The recorders may be, but is not necessarily, located at the instructor station. The recording may also be in file format for presentation on instructor station screens. Recording sessions are typically activated or de-activated at the instructor station. Student Performance Monitoring- Special groups of parameters and simulated plant operating conditions can often be assigned to a tracking and recording function that plots an individual student's performance during training exercises. Recording sessions are typically activated or de-activated at the instructor station. Video and Audio Recording- Many simulators are equipped with video and audio recording capability in the control room. Video and audio controls are typically located at the instructor station.

2. Proarammers' Tools Software Terminals- Simulator engineers have access to real-time monitoring and control of simulator and model conditions through software support terminals. These terminals may be located in the computer facility or at the engineer's desk.

Independent Executives - The conditions for scenarios can sometimes be replicated off-line using independent executive programs. These programs should not be in communication with the I/O. Independent executives and their associated initialization files may provide an indication of planned exercises if they have been used to resolve problems during scenario validation. i NUREG-1021, Revision 8 20 of 40 9i

c Appendix D Graphical User interfaces - Instructor station graphical user interfaces often display simulated plant conditions and performance in real-time. At remote locations, such as a programmer's desk, the GUI could display the full scenario.

3. ExtemalInterconnections -

ESF Feeds - Many simulators have data links to the ESF and the operations management offices for emergency planning drills. These links can display simulated plant conditlon to observers outside the simulated control room during scenario validation or examinations. Remote Plant Process Computer and Instructor Station Screens - Repeater screens in the training area can display scenarios in real tirne to observers outside the simulated control room. Modems and Remote Simulator Support Systems - Many simulators are equipped with modems from the instructor station or simulation computers for outside monitoring and control of simulator status and activities by parties off site. E. ATTACHMENTS / FORMS C Attachment 1, Attachment 2,

                         " Example initial Dynamic Simulator Scenarios"
                         " Example Requalification Dynamic Simulator Scenarios" Form ES-D-1,          " Scenario Outline" Form ES-D-2,          " Operator Actions" l

21 of 40 NUREG-1021, Revision 8

Appendix D Example initial Dynamic Simulator Scenarios Attachment 1 l l Facility: PWR Scenario No.: 1_. Op-Test No.: __.1__ Examiners: Operators: Objectives: To evaluate the applicants' ability to implement the AOPs for a SG tube leak and a loss of charging, to reduce power per GOP-3, and to execute the EOPs for a loss of all AC power (ECA 0.0), loss of coolant (E-1), and a SG tube ructure (E-3), with a success path via ES-3.1, post-SGTR 1 cooldown using backfill. Initial Conditions: 1C-38; 100% power, middle of life; CCP 'B' is running Unit 2 is in Mode 5. Tumover: The following equipment is out of service: DG "A" (6 hrs); CCW pump "A" (2 aays); VCT level transmitter LT-185; the block valve for PORV 456 is inoperable with power removed; MFP "A;' and AFW pump "A" (30 hrs). All required surveillances have been done. A severe thunderstomi waming is in effect. I Event Malf. Event Event No. No. Type

Description 1 XXX, C(RO) 70 gpm tube leak on "A" SG (ramped over 5 min) with running CCP XXX N(BOP) trip and failure of standby pump to start; requires power reduction R(RO) 2 XXX l(RO) pressurizer level instrument L-459 fails low 3 XXX C(ALL) instrument bus 112 iriverter failure 4 XXX, M(ALL) 450 gpm tube rupture on 'A' SG (ramped over 3 min) with an "A" SG XXX l(BOP) pressure transmitter failure causing the PORV to open 5 XXX, M(ALL) concurrent failures of the station auxiliary transformer and the "B" DG XXX, result in a loss of all AC power; power remains available through Unit 2 XXX TDAFW pump trips on overspeed (can be reset)

C(BOP)

  * (N)ormal, (R)eactivity, (1)nstrument, (C)omponent, (M)ajor Note: The scenarios in this attachment are individual examples; they are not intended to represent complete scenario sets / operating tests.

For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event and detailed actions required by the applicable plant procedures (e.g., normal, abnormal, emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event on the next page. NUREG-1021, Revision 8 22 of 40

p L I I I t , l O Annendix D 2 Attachment 1 Ld Op-Test No.: __1_. . Scenario No.: _1_ Event No.: _1_ Page _1_ of _5_ Event

Description:

A 70 ppm tube leak on the "A" SG (ramped over 5 minutes), combined with a trip of the running CCP and a failure of the backup CCP to start, forces a reduction in power because RCS leakage exceeds TS limits. Time Position ' Applicant's Actions or Behavior ! RO/SRO/ BOP Recognize indications of the tube leak on the 'A' SG -

                                  - air ejector off gas radiation monitor
                                  - steam line radiation monitor
                                  - charging / letdown mismatch                                                 {
                                  - SG blowdown radiation monitor                                             j l

SRO Direct RO/ BOP actions per AOP-1.2 - J L

                                  - monitor and control pzr level & pressure                                   {
                                  - monitor and control VCT level                                               1
                                  - verify leakage greater than TS limit                                        !
                                  - announce possible h'gh   i  radiation in turbine bidg
                                  - verify tube leak with SG samples
                                  - have health physit:s verify release calculation                             j
                                  - commence unit shutdown
                                  - notify NRC
                                  - minimize secor dary contamination                                           !
                                  - classify the event per the EPIPs (unusual event)                            l RO/ BOP         Execute AOP actions per SRO directions p

i SRO/RO Recog'aze running CCP tripped -

                                  - no r.harging flow
                                  - pcop inpped light
                                  - Various charging / letdown annunciators SRO             M ay direct RO/ BOP per AOP-1.3 -
                                  - isolate letdown
                                  - monitor pressurizer level and pressure start the standby CCP
                                  - reestablish letdown
                                  - refer to TS 3.8.1
                                  - initiate repairs SRO              Supervise / coordinate power reduction -
                                  - review precautions in GOP-3
                                  - ensure delta-l maintained within limits

! - verify load reduction rate RO Coordinate with BOP to initiate

                                  - review GOP-3 precautions power reduction -
                                  - calculate / estimate boration required for shutdown                         i
                                   - contact load dispatcher                                                    l l                                   - borates and/or inserts rods to maintain T-ave within SF of T-ref and       '

maintains delta-1 within limits l BOP Coordinate with RO to initiate power reduction -

                                   - review GOP-3 precautions l                                   - operate turbine controls to maintain unloading rate 1

1 1 1 23 of 40 NUREG-1021, Revision 8

Appendix D 3 Attachment 1 Facility: PWR Scenario No.: 2__. Op-Test No.: __2_. Examiners: Operators: Objectives: To evaluate the applicants' ability to implement the AOPs for a loss of CCW to the RCPs, loss of reactor makeup control, and the loss of an emergency bus; to initiate a normal plant shutdown; and to execute the EOPs for a steam line break in containment with an ATWS (FR-S.1) and a subsequent loss of all feed flow (FR-H.1) requiring bleed and feed operations. Initial Conditions: IC-20; approximately 100% power,218 ppm boron (F.0L), equilibrium xenon; bank "D" rods are at step 216 Tumover: The operations department is making preparations to shut down the plant due to equipment problems. Train "B" CSS logic failed an actuation test last shift; the LCO for TS 3.3.2 was entered 2 hrs ago; l&C is working on the problem MDAFW pump *B"is out of service to repair an oil leak and should be back in about 45 min. The block valve for PORV 445A is closed and deenergized for leakage control. Event Malf. Event Event No. No. Type

Description 1 XXX, l(BOP) spurious containment spray actuation, phase "B" isolation, and CSS XXX pump "A" failure to auto start (reset malf. to allow equipment restoration '

and before required stop of RCPs) 2 N/A N(BOP) begin normal shutdown due to CS problems R(RO) 3 XXX C(RO) boric acid filter plugged (100% in 1 min) at start of boration; when asked, filter d/p is 80# (remove when backflushed) 4 XXX l(RO) narrow range RCS temperature detector fails high 5 XXX, C(BOP) emergency bus 1 A-SA normal feeder breaker trips and DG 'A' breaker XXX trips 2 min later 6 XXX, M(ALL) "A" SG line break in containment with auto Sl on high containment XXX, C(BOP) pressure but failure of reactor and turbine trip; the local manual breaker XXX, C(RO) is operable and the turbine will follow; TDAFW pump overspeed on Sl; XXX PORV "B" failure to open in auto or manual

(N)ormal, (R)eactivity, (I)nstrument, (C)omponent, (M)ajor For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event and detailed actions required by the applicable plant procedures (e.g., normal, abnormal, emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event for the first PWR scenario (page 2 of this Attachment).

NUREG-1021, Revision 8 24 of 40

f) Appendix D 4 Attachment 1 Nl Facility: BWR Scenario No.: 1 Op-Test No.: __1__ l Examiners: Operators: Obiectives: To evaluate the applicants' ability to raise and subsequently lower reactor power in response to a TS 3.0.5 assessment; to implement the AOP for a loss of UPS; to execute the EOPs for a turbine trip on high vibration with a failure of all control rods to insert and a loss of RPIS; to determine power by attemate means due to APRM failure; and to control pressure with SRVs due to ; the inoperability of the bypass valves. initial Conditions: 10-11; approximately 90% reactor power at dispatcher request; at power for 28 )' days, beginning of cycle; core spray pump 2A is out of service to replace a breaker closing coil; APRM F failed downscale last shift and is bypassed l i Tumover: Raise power to 100% when contacted by dispatcher; test core spray pump 2A when the l clearance is lifted (imminent) l Event Malf. Event Event l No. No. Type

Description l

l 1 N/A R(RO) raise reactor power to 100% upon load dispatcher's request O 2 XXX N(BOP) test core spray pump 2A starting at step 7.9.2 of PT-07.2.4a and ( C(BOP) respond to the motor overload l 3 XXX C(SRO) individual bus breaker failure (MCC DGD), requiring DG #4 to be declared inoperable and a plant shutdown per TS 3.0.5 l 4 XXX l(RO) UPS inverter 2A malfunction and loss of UPS (no APRMs, rod C(BOP) positions, or rod control) l 5 XXX C(BOP) turbine bearing #3 vibration alarm l I l 6 XXX, M(ALL) turbine trip and reactor scram with very few rods inserted (SLC pump l XXX, 2A will trip after initiation and the scram discharge volume vents and XXX, drains fail to reopen when RPS is reset) XXX C(ALL) bypass valves fait closed after turbine coasts down (no UPS) l

               * (N)ormal, (R)eactivity, (1)nstrument, (C)omponent, (M)ajor For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event and                  i I

detailed actions required by the applicable plant procedures (e.g., normal, abnormal, emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event for the first PWR scenario (page 2 of this Attachment). , l 25 of 40 NUREG-1021, Revision 8

Appendix D 5 Attachment 1 i Facility: BWR Scenario No.: 2_ Op-Test No.: _2_ Examiners: Operators: Objectives: To evaluate the applicants' ability to lower reactor power in accordance with plant procedures; to respond to an SSW pump failure with TS implications (EDG operability); diagnose and respond to a feedwater master controller failure with rising RPV level; respond to a loss of an ESF bus and the running CRD pump; and implement the EOPs and an emergency depressurization in response to a loss of service transformers, failure of the second EDG to start, and a recirculation loop break. Initial Conditions: IC-17; 100% reactor power; 8 CRD pump is in service Tumover: The load dispatcher has asked that power be lowered to 70%, and chemistry requests an SSW surveillance to be run at the beginning of the shift. Event Malf. Event Event No. No. Type

Description 1 N/A R(RO) decrease power to 70%

2 XXX N(BOP) perform SSW surveillance per chemistry request; SSW pump B will C(BOP) trip shortly after start 3 XXX f(RO) feedwater master controller fails as is 4 XXX C(BOP) loss of power to Division 2 ESF bus 5 XXX, M(ALL) 1.5 minutes after event 4, the service transformers lock out, tne XXX, C(BOP) Division 1 EDG fails to start, and a 5% recirculation loop break XXX M(ALL) develops in the drywell C(BOP) 30 seconds after initiating, the high pressure core spray pump trips I

(N)oimal, (R)eactivity, (1)nstrument, (C)omponent, (M)ajor For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event and detailed actions required by the applicable plant procedures (e.g., normal, abnormal, emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event for the first PWR scenario (page 2 of this Attachment).

NUREG-1021, Revision 8 26 of 40

' rO Appendix D Example Requalification Attachment 2 V Dynamic Simulator Scenatios The following are two PWR and two BWR simulator scenario outlines that can be used for reference when developing or reviewing requalification examinations. , l PWR SCENARIO ONE - LOSS OF HEAT SINK l Scenario Objectives Evaluate the operators in their use of the " Loss of Heat Sink" procedure, FR-H.1. Evaluate the crew in performing a " bleed-and-feed" sequence, using reactor head vents and pressurizer vents. S_genario Summary Initial Conditions: 75 percent power "B" auxiliary feedwater pump inoperable I One PORV (A) leaking and isolated I l Events: Feed pump control problem that will eventually trip causing a partial loss of feed Totalloss of main feedwater Loss of all feedwater Scenario Seauence "A" feedwater purnp hydraulic control unit problems prompt the crew to reduce power. l During power reduction, the 'A' feedwater pump trips, causing a plant runback. Feedline break occurs causing a reactor trip. Auxiliary feedwater pumps fail over several minutes, causing a loss of all feedwater, and prompting the crew to initiate a bleed-and-feed procedure. Event one - malfunction / loss of feed oumo Crew responds to a problem with the "A" feed pump, which eventually trips causing a runback. Malfunctions required: 2 (RFP "A" HCU failure and RFP "A" Trip) Objectives: Evaluate the crew in using normal operating procedures to reduce power when the feed pump starts to fail. Evaluate the crew in using abnormal operating procedures (AOPs) to respond to a partialloss of feed, f% a 27 of 40 NUREG-1021, L'evision 8 } m

i

         , , Appendix D                                        2                                   Attachment 2 Success Path:

Use the normal operating procedures to reduce power when initial problems occur with the feedwater pump. Use the AOPs to respond to the partial loss of feed water and stabilize the plant to avoid a reactor trip. Event two - feedline rupture / reactor trio Crew responds to a total loss of feed flow with only the remaining motor-driven AFW pump availab!e. Malfunctions required: 1 (feedline rupture) Objective: Evaluate the crew's response to a loss of feed transient requiring a reactor trip by using the reactor trip response and reactor trip recovery EOPs. Success Path: Recognize the impending reactor trip, trip the reactor if time permits, and implement the apprcpriate immediate actions. Make the correct transition to the reactor trip recovery EOP upon completing the immediate and app:icable subsequent actions of the reactor trip EOP. Event three -loss of all AFW/PORV failure Crew responds to a total loss of feed flow, eventually implementing a bleed-and-feed procedure with a failed PORV. Evaluators inform the crew that time compression is being used to accelerate the decrease in steam generator level. Malfunctions required: 2 (failure of all AFW and "B" PORV fails to open) Objective: Evaluate the crew's ability to recognize that there is no longer a heat sink, and correctly { implement the applicable contingency procedure (loss of heat sink), including performing the bleed and-feed procedure. Success Path: i Implement the EOP for loss of heat sink. l Attempt to reestablish auxiliary feed flow; when SG levels become too low, initiate the ) bleed-and-feed procedure. l Recognize the failure of the available PORV and reenergize, unblock, and open the j leaking PORV; open both pressurizer and reactor head vents to ensure adequate bleed flow. NUREG-1021, Revision 8 28 of 40 4 I i u_____________._________________ __. I

f 1-m Appendix D 3 Attachment 2

      . Sggnario Recapitulation Total Malfunctions:     5                                                     'l Abnormal Events:        1                                                      i l-       Major Transients:       2 (loss of main feed and totalloss of feed)

EOPs Entered: 1 EOP Contingencies: 1 (loss of heat sink) l l i i I q l l l l ! l O i-l l l i I

29 of 40 NUREG-1021, Revision 8

l Appendix D 4 Attachment 2 PWR SCENARIO TWO - LOCA AND COLD LEG RECIRCULATION Scenario Obiectives Evaluate the crew's response to unidentified primary leakage. Evaluate the crew's response to a circulating water pump trip and a condenser tube leak. 1 Evaluate the crew in using the EOPs during a LOCA with adverse containment ) conditions. i Evaluate the crew's sensitivity to key parameters and ability to implement cold leg l recirculation. Scenario Summary Initial Conditions: 100 percent power Inoperable "A" diesel generator and "A" instrument air compressor Seismic event occurred during last shift Events: Primary leak increases to a point requiring a reactor trip. AFW pumps fail to automatically start on reactor trip. Leak leads to a safety injection (SI) and high pressure SI pumps fail to start automatically; LOCA occurs, RWST leak occurs, and crew must initiate cold leg recirculation. Scenario Seauence A small pressurizer steam space leak increases to a point requiring a reactor trip and eventually to the point of Sl initiation. The high pressure SI pumps fail to start automatically. A LOCA occurs as a result of the seismic event. When the SI pumps start, the thermal shock causes a LOCA in the RCS. The high pressure of the LOCA causes adverse containment conditions. An RWST leak will also occur concurrent with the Si that will eventually prompt the crew to initiate cold leg recirculation. RWST level will eventually drop to the point where the crew must initiate cold leg recirculation. Event one - Unidentified leakaae due to cressurizer steam space leak The crew reacts to unidentified primary leakage, eventually requiring a reactor trip. Malfunctions required: 1 (pressurizer steam space leak) NUREG-1021, Revision 8 30 of 40

n l Appendix D 5 Attachment 2

 . Objectives:

Evaluate the crew's use of AOPs and TS to respond to unidentified primary leakage. l Evaluate the crew's knowledge of parameters in the AOP that require a trip because of primary leakage. l Success Path: 1

1 Use the AOPs, increase reactor make-up and calculate a leak rate. 1 Use the NOPs to commence a reactor shutdown in accordance with TS. When leakage exceeds the AOP parameters, trip the reactor. ; Event two - reactor trio /AFW oumo fails to start automatically l The crew trips the reactor on excessive leakage per the AOP. The AFW pumps fail to start automatically, requiring manualinitiation. Malfunctions required: 1 (AFW failure to auto start) Objective: , Evaluate the crew's use of the EOPs following a reactor trip, with the complication of the AFW l pumps failing to start automatically. Success Path: Recognize that the AFW pumps failed to start automatically and manually start the pumps. l Correctly perform the reactor trip EOP and make the transition to the reactor tnp l recovery EOP once the immediate actions and applicable subsequent actions are l completed. Event three - increasina oressurizer leak /SI oumos fail to start The pressurizer leak increases causing a loss of pressurizer level / pressure requiring an SI. The charging pumps fail to automatically start requiring manual start. Malfunctions required: 2 (pressurizer leak increases and charging pumps fail to auto start) Objectives: Evaluate the crew's ability to monitor important parameters in the EOPs and initiate SI when required. Evaluate the crew's ability to manually start the charging pumps following a SI signal. 31 of 40 NUREG-1021, Revision 8

1 Appendix D 6 Attachment 2 Success Path: l l Initiate SI when pressurizer level and pressure decrease to the values stated in the EOPs. j Recognize the failure of charging pumps to automaticall;y start and manually start the I 2 required charging pumps to complete the Si initiation sequence. Event four - LO ,A/ adverse containment 1 A LOCA occurs as a result of the seismic event, which leads to adverse containment conditions. RWST level decreases to the point where the crew must enter the EOP for initiating cold leg recirculation. Evaluators inform the crew that time compression is being used to accelerate the decrease in RWST level. Malfunctions required: 2 (LOCA and RWST leak) l Objectives. . Evaluate the crew's use of the EOPs with adverse containment. Evaluate the crew's ability to recognize the need for and use the cold leg recirculation procedure. Success Path: Correctly enter and use the LOCA EOP and the containment functional recovery EOP using adverse containment criteria.

 -       When RWST levels reach the low-low alarm and the reactor sump levelis high enough,           ,

enter and implement the cold leg recirculation EOP. l l Scenario Recacitulation Total Malfunctions: 6 Abnormal Events: 2 Major Transients: 2 (leak requiring Si and LOCA with high containment pressure) EOPs Entered: 4 (enter LOCA EOP twice) EOP Contingencies: 1 (containment safety) l NUREG-1021, Revision 8 32 of 40 9i

1 4 Appendix D 7 Attachment 2 BWR SCENARIO ONE - LOSS OF OFFSITE POWER WITH A LOCA Scenario Objective l

Evaluate the operators in using the " Emergency Depressurization" and "RPV Flooding" EOP l

contingency procedures. j Scenario Summarv l i Initial Conditions: I l-98 percent power l

                "A" average power range monitor (APRM) failed and bypassed i                                                                                                                4 Events:

Reactor core isolation cooling (RCIC) becomes isolated during a RCIC flow surveillance. l - Loss of offsite power / division ill diesel generator fails to start, disabling the high pressure (HP) core spray. Small break LOCA occurs. l Adverse containment conditions make the reactor level instrumentation unusable. Scenano Seauence The RCIC becomes isolated during surveillance testing, rendering the RCIC system

    \           inoperable.

Faults in the 345 KV switchyard and the reserve auxiliary transformer result in a loss of offsite power and a reactor scram. The Division hl diesel generator fails to start and will not start manually, disabling the HP core spray system. The plant transient causes a recirculation line break resulting in a small break LOCA i that develops over several minutes. Reactor level instrumentation becomes erratic and unusable because of the rapid decrease in pressure and the elevated drywell temperature. Event one - RCIC isolation The crew responds to an isolation of the reactor core isolation cooling (RCIC) system during a full ficw test surveillance. Malfunctions required: 1 (RCIC isolation)- 1 Objective: Evaluate the crew in using technical specifications to determine that RCIC is inoperable. l C 33 of 40 NUREG-1021, Revision 8

8 Success Path: Use technical specifications to recognize that the RCIC system should be declared inoperable until the problem can be investigated and corrected. Event two - loss of offsite power with concurrent division lli diesel oenerator failure (HP core spray) The crew responds to the loss of offsite power, reactor scram and loss of high pressure injection sources. Malfunctions required: 2 (loss of offsite power and HPCS failure) Objective: l l Evaluate the crew's response to a plant transient that causes a reactor scram and a loss of high j pressure injection sources by using the reactor pressure vessel (RPV) and primary containment 4 control EOPs. I l Success Path: l l Maintain RPV pressure at less than 1065 psig using the main turbine bypass valves. l Manually control pressure with safety relief valves (SRVs) upon a loss of electro-hydraulic control (EHC) hydraulic pressure because of the lors of power to the EHC ; pumps. Initiate suppression pool cooling and pump down in accordance with EOPs if the temperature in the suppression pool exceeds 90 degrees or the level exceeds 18.5 feet. Event three - small break LOCA I The crew responds to a loss of vessel inventory and an inability to maintain a level greater than top of active fuel, eventually implementing emergency depressurization. Malfunctions required: 1 (LOCA) l Objective: Evaluate the crew's ability to recognize an inability to maintain reactor water level and correctly implement the applicable contingency procedures including emergency depressurization. Success Path: Execute RPV emergency depressurization so reactor pressure can be decreased to allow injection by the low pressure ECCS systems. , 1 l l l g l l

Appendix D 9 Attachment 2 [ Event four - reactor level instrumentation failure The crew recognizes a loss of reactor level instrumentation and responds in accordance with RPV flooding EOP. Malfunctions required: 1 (reactor level instrumentation failure) Objective: Evaluate the crew's ability to recognize failed reactor level instrumentation and correctly . Implement the applicable actions of the RPV flooding EOP to ensure adequate core cooling. Success Path: Reflood the RPV in accordance with the EOPs and establish adequate core cooling. Adequate core cooling will be ensured when reactor pressure can be maintained greater than 120 psig with at least 3 SRVs opened by manually controlling low pressure ECCS injection flow. Scenario Recao tulation Total Malfunctions: 5 Abnormal Events: 3 O Major Transients: 2 (emergency depressurization and RPV flooding)

     'Q EOPs Entered:

EOP Contingencies: 2 3 (altemate level control, emergency depressurization, and RPV flooding) I l l 35 of 40 NUREG-1021, Revision 8

Appendix D 10 Attachment 2 BWR SCENARIO TWO - POWER OSCILLATIONS WITH AN ATWS

Scenario Obiective Evaluate the operators in using the " Level / Power Control" and " Emergency Depressurization' l EOP contingency procedures. Scenario Summary initial Conditions: 75 percent reactor power High Pressure Core Spray pump out of service "B" recirculation pump flow control valve is locked Events: The "A" reactor recirculation pump trips, causing power oscillations, and an SRV fails open during the power oscillations. Anticipated transient without scram (ATWS) requiring lowering of level to control power. Feed system pumps will fail to restart and standby liquid control (SLC) pumps and RCIC pump trip during the transient, complicating recovery from the event. Scenario Seauence The 'A" recirculation pump trips resulting in power oscillations within 5 minutes. The reactor fails to manually scram. The safety relief valve (SRV) sticks open during power oscillations. Condensate booster and feedwater pumps fail to restart, and the SLC pumps trip after power is reduced less than 3 percent. RCIC pump trips after it is restarted by an operator. ._ ent one "A" recirc oumo trio resultina in oower oscillations The crew responds to a recirculation pump trip and a failure of the reactor scram system. Malfunctions required: 2 (recirculation pump trip and ATWS) Objectives: Evaluate the crew's use of AOPs and EOPs to respond to an ATWS and to restore the power and flow parameters to acceptable values. Evaluate the crew's use of TS that apply to single recirculation loop operation. NUREG-1021, Revision 8 36 of 40

i Appendix D 11 Attachment 2 Success Path: l Recognize power to be in region B or C of the power and flow map and initiate control rod insertion to reduce thermal power. Recognize symptoms of thermal hydraulic instability and attempt to manually scram. Use the EOP flow charts for RPV level, power, and pressure control. l Trip the "B" recirculation pump and initiate actions to achieve control rod insertion and to actuate the standby liquid control system in accordance with the EOPs. Event two - SRV sticks open durina oower oscillations The crew recognizes and responds to the stuck open SRV, eventually implementing the actions of the primary containment control EOP. Malfunctions required: 1 (SRV sticks open) Objective: Evaluate the crew's ability to recognize the failed open SRV and implement the applicable abnormal and emergency procedure actions. Success Path: l( - Initiate actions to close the SRV. .\ - Use EOPs to initiate suppression pool cooling and reduce the level. Terminate all injection into the RPV except for the control rod drive and SLC systems when suppression pool temperature exceeds 110 degrees with reactor power less than 3 percent. Event three - failure of iniections sources after control rod insertion The crew responds to a loss of vessel inventory and the inability to maintain level greater than top of active fuel by eventually implementing emergency depressurization. Malfunctions required: 3 (feedwater system failure, SLC pump trip, and RCIC fails to start) Objective: Evaluate the crew's use of EOPs to respond to an inability to maintain reactor water level and to initiate an emergency depressurization. l Success Path: 1 Execute RPV emergency depressurization to allow for injection by the low pressure ECCS

systems.

1 O 37 of 40 NUREG-1021, Revision 8 f L

Appendix D Attachment 2 Scenario Recapitulation Total Malfunctions: 6 Abnormal Events: 2 Major Transients: 2 (ATWS and emergency depressuritation) EOPs Entered: 2 EOP Contingencies: 3 (level and povier control, alternate level control, and emergency depressurization) O l NUREG-1021, Revision 8 38 of 40

T Appendix D Scenario Outline Form ES-D-1 j i Facility: Scenario No.: Op-Test No.. q Examiners: Operators: 1 Objectives: I initial Conditions: i Turnover; Event Malf. Event Event No. No. Type

Description

        * (N)ormal, (R)eactivity, (1)nstrument, (C)omponent, (M)ajor t

O 1 39 of 40 NUREG 1021, Revision 8

i ) l l l Appendix D Operator Actions Form ES-D-2 Op-Test No.: Scenario No.: Event No.: Page of Event

Description:

Time Position Applicant's Actions or Behavior O l NUREG-1021, Revision 8 40 of 40

APPENDIX E POLICIES AND GUIDELINES FOR TAKING NRC EXAMINATIONS Each examinee shall be briefed on the policies and guidelines applicable to the examination category (written and/or operating test) being administered. The applicants may be briefed individually or as a group. Facility licensees are encouraged to distribute a copy of this appendix to every examinee before the examinations begin. All items apply to both initial and requalification examinations, except as noted. PART A - GENERAL GUIDELINES 1, [ Read VerbatimJ Cheating on any part of the examination will result in a denial of your application and/or action against your license.

2. If you have any questions conceming the administration of any part of the examination, do not hesitate asking them before starting that part of the test.

4. SRO applicants will be tested at the level of responsibility of the senior licensed shift position (i.e., shift supervisor, senior shift supervisor, or whatever the title of the position may be).

5. You must pass every part of the examination to receive a license or to continue performing license duties. Applicants for an SRO-upgrade license may require remedial training in order to continue their RO duties if the examination reveals deficiencies in the required knowledge and abilities.

6. The NRC examiner is not allowed to reveal the results of any part of the examination until they have been reviewed and approved by NRC management. Grades provided by the facility licensee are preliminary until approved by the NRC. You will be informed of the official examination results about 30 days after all the examinations are complete.

PART B - WRITTEN EXAMINATION GUIDELINES

1. [ Read VerbatimJ After you complete the examination, sign the statement on the cover sheet indicating that the work is your own and you have not received or given assistance in completing the examination.

2. To pass the examination, you must achieve a grade of 80.00 percent or greater; grades will not be rounded up to achieve a passing score. Every question is worth one point.

3. For an initial examination, the time limit for completing the examination is five hours.

For a requalification examination, the time limit for completing both sections of the examination is three hours. If both sections are adminisicred in the simulator during a single three-hour period, you may retum to a section of the examination that was already completed or retain both sections of the examination until the allotted time has expired. 1 of 5 NUREG-1021, Revision 8

Appendix E l

4. You may bring pens, pencils, and calculators into the examination room. Use black ink to ensure legible copies; dark pencil shculd be used only if necessary to facilitate machine grading.

5. Print your name in the blank provided on the examination cover sheet and the answer sheet. You may be asked to provide the examiner with some form of positive identification.

6. Mark your answers on the answer sheet provided and do not leave any question blank.

Use only the paper provided and do not write on the back side of the pages. If you are using ink and decide to change your original answer, draw a single line through the error, enter the desired answer, and initial the change.

7. If you have any questions concerning the intent or the initial conditions of a question, do not hesitate asking them before answering the question. Ask questions of the NRC examiner or the designated facility instructor only. When answering a question, do not make assumptions regarding conditions that are not specified in the question unless {

they occur as a consequence of other conditions that are stated in the question. For 1 example, you should not assume that any alarm has activated unless the question so I states or the alarm is expected to activate as a result of the conditions that are stated in the question. I

8. Restroom trips are permitted, but only one applicant at a time will be allowed to leave. !

Avoid all contact with anyone outside the examination room to eliminate even the I appearance or possibility of cheating.

9. When you complete the examination, assemble a package including the examination questions, examination aids, answer sheets, and scrap paper and give it to the NRC examiner or proctor. Remember to sign the statement on the examination cover sheet indicating that the work is your own and that you have neither given nor received assistance in completing the examination. The scrap paper will be disposed of immediately after the examination.

10. After you have turned in your examination, leave the examination area as defined by the proctor or NRC examiner. If you are found in this area while the examination is stillin progress, your license may be denied or revoked.

11. Do you have any questions?

l PART C - GENERIC OPERATING TEST GUIDELINES (CATEGORIES A. B. AND C)

1. If you are asked a question or directed to perform a task that is unclear, you should not hesitate to ask for clarification.

2. The examiner will take notes throughout the test to document your performance, and l sometimes the examiner may take a short break for this reason. The amount of note-NUREG-1021, Revision 8 2 of 5

Appendix E taking does not reflect your level of performance. The examiner is required to document satisfactory as well as less than satisfactory performance.

3. The operating test is considered "open reference.' The reference material that is normally available to operators in the facility and control room (including calibration curves, previous log entries, piping and instrumentation diagrams, calculation sheets, and procedures) is also available to you during the operating test. However, you should know from memory certain automatic actions, set points, interlocks, operating characteristics, and the immediate actions of emergency and other procedures, as appropriate to the facility. If you desire to use a reference, you should ask the examiner if it is acceptable to do so for the task or question under consideration.

You may not solicit technica/information from other operators, engineers, or technical advisors.

4. You must not discuss any aspect of your operating test with any other examinee until after all the examinations are complete.

PART D - WALK-THROUGH TEST GUIDELINES (CATEGORIES A AND B)

1. .The walk-through test covers control room systems, local system operations, and administrative requirements. The examiner will evaluate these areas using a combination of job performance measures (JPMs) and spectric questions.

The initial walk-through consists of ten JPMs for RO and SRO(l) applicants and five for SRO(U) applicants. Seven of the JPMs (two or three for upgrade applicants) will be conducted in the control room or simulator and the remainder will be conducted in the plant. The requalification walk through consists of five JPMs total, with at least two in the control room / simulator and at least two in the plant. )

2. The examiner is a visitor at this facility. When you enter the plant, you may be expected to escort the examiner and ensure that he or she complies with safety, security, and radiation protection procedures.

3. You should not operate plant equipment without appropriate permission from the operating crew. Nothing the examiner says or asks will be intended to violate this l

principle. [

4. Eofore beginning each JPM, the examiner will describe the initial conditions, explain the task that is to be completed, indicate whether the task is time-critical, and explain which steps are to be simulated or discussed. You should perform or simulate the required actions as if directed by plant procedures or shift supervision. Do not assume that the examiner will accept an oral description of the required action unless the examiner indicates otherwise.

1

                                                                                                     )
 \                                                 3 of 5                  NUREG-1021, Revision 8 l

Appendix E

5. Time-critical JPMs have been validated by your facility and must be completed within the predetermined time interval in order to obtain a satisfactory grade for that JPM.. You will be permitted to take whatever time is necessary to complete those JPMs that are not time-critical, provided you are making reasonable progress toward achieving the task standard.

6. When performing JPMs, you are expected to make decisions and take actions based on the facility's procedural guidance and the indications available. Some of the tasks that the examiner asks you to perform will require the implementation of an alternative method directed by plant procedures.

7. As part of the examination, the examiner may ask questions to investigate your knowledge of an administrative topic, system, or task. Many of the questions will require you to use plant reference material, while others should be answered without the use of references. If you need to consult a reference to answer a question, ask the examiner if it is acceptable to do so. There is no specific time limit for any question, however, you may be evaluated as unsatisfactory on a question if you are unfamiliar with the subject or reference material and are unable to answer the question in a reasonable period of time. You will not be permitted to conduct unlimited searches of the plh.1t reference material during the examination.

8. To facilitate the examination and better enable the examiner to assess your level of understanding, please verbalize your actions and observations while performing the

' JPMs. Also, please inform the examiner when you consider your performance of each JPM and your answer to each question to be complete.

9. If you need a break during the test, you should ask the examiner.

10. Do you have any questions before we begin the walk-through test?

PART E - SIMULATOR TEST GUIDELINES (CATEGORY C)

1. Your primary responsibility is to coerate the simulator as if it were the actual plant. If you believe that the simulator is not responding properly, you should make decisions and recommendations on the basis of the indications available, unless directed otherwise by the examiner.

2. If the examiner asks you a question, you should answer it only if doing so will not interfere with simulation facility operations.

3. Teamwork and communications are evaluated. You can enhance the evaluation process by vocalizing your observations, analyses, and the bases for your actions.

Requalification examinations evaluate the crew's ability to safely operate the plant and the performance of both the individuals and the crew. NUREG-1021, Revision 8 4 of 5 1

n - O Appendix E

4. If you recognize but fail to correct an erroneous decision, response, answer, analysis, action, or interpretation made by the operating team or crew, the examiner may conclude that you agree with the incorrect item.

5. You should keep a rough log during each scenario that would be sufficient to complete necessary formallog entries.

6. A designated facility instructor (or an examiner) will act as the auxiliary operators, radiation health and chemistry technicians, maintenance supervisors, plant management, and anyone else needed outside the control room.

7. The facility instructor (or examiner) wi!! provide a shift turnover briefing before the scenario begins. The briefing will cover present plant conditions, power history, equipment out of service, abnormal conditions, surveillances due, and instructions for the shift. j 4

8. Control board switches may be purposely misaligned to enhance a scenario or transient I where appropriate. You will not be required to locate misaligned switches as part of the )

evaluation. If a switch is misaligned,it will be tagged or otherwise highlighted as appropriate to the faciH+y and will be noted during the turnover briefing. The examiners will not misalign switches during the scenario. ,

i

9. Time compression may be used to expedite the sequence of events in some scenarios, V but it will not preclude you from performing the actions that you would typicaKy be required to perform in response to the events. If time cornpression is used, you will be so informed during and after the scenario.

10. You will given sufficient time (normally about five minutes) to familiarize yourselves with plant conditions before starting each simulator scenario.

11. The initial test will normally consist of two or three scenarios lasting a total of three to four hours. The requalification test will normally consist of two scenarios lasting about one hour each. You will be given a short break between scenarios.

12. SRO upgrade applicants who fill the role of an RO or balance of plant (BOP) operator during a scenario will be evaluated on their ability to manipulate the controls even though an examiner may not be assigned to directly monitor their performance.

13. Do you have any questions before we begin the sunulator test?

I 5 of 5 NUREG-1021, Revision 8

r APPENDlX F GLOSSARY Achievement test: An instrument designed to measure a trainee's grasp of some body of knowledge or skill proficiency. Annual. In most instances, a period of time equal to 365 days reckoned from any point in a " calendar year to the same point in time in the following calendar year. However, annual requirements in successive years can reach a period of nearly two years. Annual could encompass a range extending to 729 days depending on when an event occurred in the first calendar year and viewing December 31 of the following calendar year as meeting the annual ' requirement. 1 Apclicant: Any individual who has submitted an NRC Form 398 in pursuit of an RO or SRO license. For purposes of this and the other Examination Standards, it is synonymous with

       " candidate."

Aoolicant license level: The level of operator license (i.e., RO or SRO) for which the applicant

     ' has applied.

Aptitude test: An instrument designed to assess an individual's potential for performing some task or skill area. Averaae: A score that provides an indication of the typical performance of a group of scores. The mean, median, and mode of a distribution of scores are all commonly used as averages. Biennial: ~ In most instances, a period of time equal to 730 days and synonymous with the term "two years." Biennial requirements can extend beyond 730 days if the requirement is met during the anniversary month of the second year. For example, a biennial medical exarnination last performed on January 10,1995, would be due again by January 31,1997. January is seen as the ar.niversary month, the period of time between the two examinations is longer than 730 days, but the biennial requirement is satisfied. Bloom's Taxonomy: A classification system that depicts knowledge and information processing of knowledge in a hierarchy from lowest to highest as follows: fundamental knowledge, comprehension, analysis, synthesis, and evaluation. Calendar auarter: One of four parts of a calendar year, each consisting of a 3-month segment. In any calendar year, the first quarter is from the first day of January to the last day of March, the second quarter is from the first day of April to the last day of June, the third quarter is from the first day of July to the last day of September, and the fourth quarter is from the first day of October to the last day of December. Cateaorv: One of 3 major subdivisions of related subjects on the operating test. Refer to Section D of ES-301 for a description of and detailed instructions for developing each operating test category. O U 1 of 6 NUREG-1021, Revision 8 i

Appendix F Central tendencv: A term referring to the most typical performance of a group of individuals; generally tne mean, median, or mode Coanitive: Aspects of a person or test level that refer to knowledge or understanding. Content validitv: The degree to which a test measures the specific objectives or content of that test. Correlation coefficient: A numerical value ranging from -1 to +1 that indicates the relationship between two sets of scores or other measures of each individualin a group. A value of O indicates no relationship; +1 or -1 indicates a perfect relationship, either positive or negative. Criterion: A characteristic or combination of characteristice C od as the basis for judging a performance. Criterion-referenced test: An examination based upon mastery of objectives of content that was or should have been tau;;ht and mastered and one that uses an established standard or cutoff score as a measure of acceptable performance. Cut score: The score at which a trainee is deemed to have met the criteria on an exam. Diaanostic test: An instrument that is designed to identify the strengths and weaknesses of an individual for a given content area. Difficulty index: A numerical index ranging from 0.00 to 1.00 that indicates the percentage of trainees who answer a test item correctly. An index of 0.00 indicates that no one answered the test item correctly while an index of 1.00 indicates that all individuals answered the item correctly. Discrimination index: A measure of a test item's ability to differentiate between good and poor trainees. A high discrimination index indicates that more high performers than low performers answered the item correctly (high and low are typically determined by overall test scores but may also be established by external criteria). Discrimination validitv: Setting the item difficulty at an estimated level around the cut score. Distractor: An incorrect alternative among the choices of a test item. Error of measurement: Any difference between an obtained score and a true score on a test is referred to as error of measurement. The actual error of measurenient can only be estimated since it is impossible to know what the true score is. Eauivalent forms: Two or more exams that test the same objectives using different test items or the same test items in a different sequence. NUREG 1021, Revision 8 2 of 6 i i

y l Appendix F l [v) Freccency distribution: A graphic display listing scores, or score intervals on one axis of a graph, and the number of trainees at that score or in that interval on the other. l l Item analysis: A set of procedures performed on examination items to determine their difficulty and discriminating power. Item bank: A group of test items covering a defined area. Items for a test can be chosen from this source, item stem: The part of a test item that presents the problem or situation to be solved. The stem may be a question requiring a response or a statement that is followed by the alternatives from which the trainee must choose the best answer. 3 Job oerformance measure (JPM): An evaluation tool that is based on tasks contained in the facility's job task analysis (JTA) or the applicable NRC Knowledge and Abilities Catalog (NUREG-1122 or 1123) and requires the applicant to perform (or simulate) a task applicable to the license level of the examination. Job task analysis (JTA): A systematic analysis of the knowledge, skills, and abilities required to perform a particular occupation. Leamina obiective: A statement of the behavior a trainee is expected to exhibit following l

 /'. instruction.

Masterv test: A term synonymous with criterion-referenced test, i.e., evaluating the expected behavior following instruction. Mean. An indication of central tendency; it usually refers to the arithmetic mean, which is computed by summing all the scores of a group and dividing that sum by the number of scores in the group. Median: A measure of central tendency; the point on a scale of scores that splits the scores in half; 50 percent of the scores are below this point, and 50 percent of the scores are above this point. Mode: The least reliable of the common measure of central tendency; the mode is the most j frequently occurring score in a distribution of scores. Multiple choice item: A test item composed of a stem and several alternatives from which the ; trainee must select the best answer. ! Normal distribution: A theoretical frequency distribution represented by a symmetrical j bell-shaped curve; sometimes referred to as the bell curve. Norm-referenc_e_d: A score interpretation based on the comparison of an individual's score with a cornparable reference group. 3 of 6 NUREG-1021, Revision 8 l

Appendix F Nuclear power olant experience: As defined in Section 2 of ANSI /ANS-3.1-1981, means experience acquired in preoperational and startup testing activities or operation of nuclear power plants. Experience in design, construction, and operational training may be considered applicable nuclear power plant experience and should be evaNated on a case-by-case basis. Refer to ANSI /ANS-3.1-1981 for additionalinformation regarding the equivalence of simulator and on-the-job training, and military and other experience. Nuclear oower olant fundamentals: Refer to Section 5.2.1.1 of ANSI /ANS-3.1-1981 for a list of topics to be covered in a fundamentals course. Obiective test: A test that can be scored without subjective judgment in the scoring. On-the-lob trainina: Participation in nuclear power plant startup, operation, maintenance, or technical services as a trainee under the direction of experienced personnel. Operatina test: That portion of the operator licensing examination based on direct interaction between an examiner and an applicant. It tests the applicants' knowledge of the design and operation of the reactor and its associated plant systems, both inside and outside the control room. It is administered in a plant walk-through and in a simulation facility. Operational validity: A test item that is 1) related to the operations of the job and appears reasonable to ask and 2) expressea in an operational context that requires the candidate to mentally or physically perform through understanding or analysis. Performance test: Any test that requires the trainee to demonstrate either mental performance through knowledge testing or skill by actual operation or manipulation of tools and equipment. Typically, perforrnance tests connote the meaning of skill testing. i Plant-referenced simulator: As defined in 10 CFR 55.4, means a simulator modeling the systems of the reference plant with which the operator interfaces in the control room, including i operating consoles, and which permits use of the reference plant's procedures. A plant- ] referenced simulator demonstrates expected plant response to operator input, and to normal, transient, and accident conditions to which the simulator has been designed to respond. Power olant experience: As defined in Section 2 of ANSI /ANS 3.1-1981, means experience acquired in the testing, operation, and maintenance of power generating facilities. Experience in design and construction may be considered applicable power plant experience and should be evaluated on a case-by-case basis. Predictive validity evidence: The ability of a test to forecast future performance on a subsequent measure. Psychomotor: The domain of human performance that relates to physical performance based on mental activity. NUREG-1021, Revision 8 4 of 6 ki

l i l i l O Appendix F l Range: The smallest intewal on a scale of scores that will include all scores, mathematically defined as the largest score minus the smallest score plus one. Raw score: The numerical score first assigned when scoring a test before conversion to a derived score. Reactor operator aoolicant: An unlicensed individual who is applying for an RO license. Reference olant: As defined in 10 CFR 55.4, means the specific nuclear power plant from l which a simulation facility's control room configuration, system control arrangements, and design data are derived.' Related technical trainina: As defined in Section 2 of ANSI /ANS-3.1-1981, means formal training beyond the high school level in technical subjects associated with the position in 3 question, such as acquired in training schools or programs conducted by the military, industry, ! utilities, universities, vocational schools, or others. Such training programs shall be of a scheduled and planned length and include text material and lectures. I Reliability: The consistency or repeatability of any measure as an indicator of confidence of

that measure. Responsible power olant exoerience: As defined in Section C.1.e of Regulatory Guide 1.8, j Revision 2, subject to the following clarifications. Time spent in related technical training, or academic work leading to a technical degree, qualifies as experience equivalent to responsible power plant experience, on a one-for-one l basis, up to a maximum of 2 years. i Up to 6 years of experience acquired at military, nonstationary, propulsion, test, research, or production nuclear plants may qualify as responsible power plant experience, on a two-for-one ' basis, up to a maximum of 3 years. Such experience must be acquired in a position equivalent to a licensed RO or SRO (e.g., propulsion plant watch officer, engineering watch supervisor, l engine room supervisor, reactor operator, chief reactor watch, engineering officer of the watch, j propulsion plant watch supewisor, and shutdown maneuvering area watch). Experience in other positions may be justified and authorized on a case-by-case basis. Scenario: An integrated group of events that simulate a set of plant malfunctions and evolutions at a simulation facility.

    $_Qegario e         set: A group of scenarios that constitutes a complete simulator test (i.e., Category C,
     " Integrated Plant Operations," of the operating test).

Score: A numericalindication of the performance an individual displays on a test. Senior reactor ooerator uoarade (SRO-U) aoolicant: A licensed RO who is applying for an SRO license on the same unit (s). ( NUREG-1021, Revision 8 5 of 6

Appendix F Senior reactor operator instant (SRO-1) aoolicant: An unlicensed individual who is applying for an SRO license. Simulation facility: As defined in 10 CFR 55.4, means one or more of the following components, alone or in combination, used for the partial conduct of operating tests for operators, senior operators, and applicants:

1. the plant

2. a plant-referenced simulator

3. another simulation device This definition provides flexibility in the conduct of the " Integrated Plant Operations" category of the operating test, as permitted in 10 CFR 55.45(b). It allows examiners to administer the operating test on the plant itself, a plant-referenced simulator, or some other type of NRC-approved simu:ation device, such as a part-task or basic-principles simulator.

Standard deviation: A measure of variability of a set of scores around the group mean. The SD is mathematically defined as the square root of the mean of the squared deviations of the scores from the mean of the distribution. Standard error of measurement: An estimate of the stradard deviation of the errors of measurement associated with the test scores in a given test. Standardized test: A test that has the directions, time limits, and conditions of administration made consistent for all offerings of the test; this test is usuelly norm-referenced. Statistic: A numerical value computed on a sample of data. Technical Specifications: A document that identifies the plant-specific safety limits, system operability and surveillance testing requirements, and administrative controls. Whether stated ! or not, references to the technical specifications in this NUREG include those administrative { controls that have been moved to other technical requirements documents. l Test: A measurement instrument; examination. True score: The ideal or correct score for an individual. Its value cannot be known, but it can be estimated when assumptions regarding error of measurement are made. Validitv: The degree to which a test measures what it purports to measure. l NUREG-1021, Revision 8 6 of 6 l

NRc FORM 33s u.u. NUCLEAR REGULATORY COnsntsslON 1. R

        $llMisoa.                                                                                                                                              Ron 22oi. 22o2
   ^

BIBLIOGRAPHIC DATA SHEET (See =_- --- on the .a,. ) NUREG-1021, Rev. 8 g (, TITLE AND SuBTnLE Operator Licensing Examination Standards for Power Reactors

3. DATE REPORT PuBUSHED MONTH YEAR l

April 1999

4. FIN OR GRANT NUMBER

5. AUTHOR (S) 6. TYPE OF REPORT Staff Technical

7. PERIOD COVERED paciume Dates)

8. PERFORMING ORGANIZATION - NAME AND ADDRESS prNRC. prowde Dwsm. Otsco w Rega, uS. Nudear Reputafory Commissa, and madng ackfress; #contrador prowse name and meeng acess}

Division of inspection Program Management Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (#f NRC, type 'same as afmet scontracsor, prows NRC Dwson, Orsco or Rega, u s Nudear Regurasory comnusamn, and maung actkess)

Same as above

10. SUPPLEMENTARY NOTES Supersedes Interim Revision 8

11. ABSTRACT (200 woros wless)

NUREG-1021 establishes the policies, procedures, and practices for examining licensees and applicants for reactor operator and senior reactor operator licenses at power reactor facilities pursuant to Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55). It is intended to assist NRC examiners and facility licensees to better understand the processes associated with initial and requalification examinations. The standards also ensure the equitable and consistent administration of examinations for all applicants. These standards are for guidance purposes and are not a substitute for the operator licensing regulations (i.e.,10 CFR Part 55), and they are subject to revision or other changes in intemal operator licensing policy. This revision implements an amendment to 10 CFR Part 55 that allows facility licensees to prepare the entire operator ] licensing examination and to proctor and grade the written portion of the examination. The NRC will prepare the examinations ! at least four times per year to maintain the proficiency of its examiners, as necessary to ensure quality, and upon written request by facility licensees consistent with NRC staff availability. This revision will become effective concurrent with the associated amendment to 10 CFR Part 55 or at an earlier date agreed upon by the facility licensee and its NRC Regional Office.

        '12. KEY WORDS/DESCRIPTORS (Ost worcs or phrases that m# assst researchers e ecateg the seport)                               13, AVAILA68UlY STAItMENI operator licensing, examination standards                                                                                            unlimited 14 SECURrrY CLASSIFICATK)N (Thus Page) unclassified (Th,s Reporo unclassified

15. NUMBER Of PAGES

16. price NHC f ORM 335 (24H

e i O, Printed on recycled paper Federal Recycling Program O - _ _ _ _ - - _ _ _ _ -}}

ML20206N219

Contents

Text

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

Dear (Name):

SUBJECT:

Dear (Name):

Dear (Name):

Enclosure:

Dear { blame):

Enclosures:

Dear (Name):

Enclosures:

Enclosure:

Dear (Name):

Enclosures:

Dear (Namd:

Enclosures:

SUMMARY

SUMMARY

SUBJECT:

Dear (Namd:

Enclosures:

SUMMARY

Title:

Reference:

References:

Response

Description:

Description:

Navigation menu

ML20206N219

Text

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

SUMMARY

Dear (Name):

SUBJECT:

Dear (Name):

Dear (Name):

Enclosure:

Dear { blame):

Enclosures:

Dear (Name):

Enclosures:

Enclosure:

Dear (Name):

Enclosures:

Dear (Namd:

Enclosures:

SUMMARY

SUMMARY

SUBJECT:

Dear (Namd:

Enclosures:

SUMMARY

Title:

Reference:

References:

Response

Description:

Description:

Navigation menu

Search