ML20135D571

Operator Licensing Examination Standards for Power Reactors

ML20135D571
Person / Time
Issue date:	01/31/1997
From:	Office of Nuclear Reactor Regulation
To:
References
NUREG-1021, NUREG-1021-INT, NUREG-1021-INT-R08, NUREG-1021-INT-R8, NUDOCS 9703050343
Download: ML20135D571 (460)

Text

{{#Wiki_filter:_ _ .. __ _ . . _ _ __ _ _ _ _ _ _ _ _. _ _ _ __ _ _ _ _ _ _ . _. _ _ _ -- ._.. i l

NUREG-1021 l Interim Rev. 8 Operator Licensing Examination l

Standard.s for Power Reactors 1 l 4 U.S. Nuclear Regulatory Commission l l Omce of Nuclear Reactor Regulation l Y 75R 1021 R 28Ri8 " '

• PDR

                                                                                                        .g og imppuppppilips                                      o\\

            ..  .. .._. .--   .- .             ..       . - .   .             . .. . .-. - . ._ _             . . . .~. -        .__               .    .

t

                                                                  .nur    .m.     <     a-%    ,

I l 1 NUREG-1921 OPERATOR LICENSING EXAMINATION STANDARDS l j JANUndY 1997 I terim Rev. 8 FOR POWER REACTORS 4

                                                                   ,_ ,,, m ,,_._ _ __            _ _ . . _ .

6 t i l i f i

l l 1

 =

NUREG-1021 Interim Rev. 8 l 1 i i Operator Licensing Examination l Standards for Power Reactors i i l i j l 1 i )  ! l Manuscript Completed: January 1997 j Date Published: January 1997 I a i i Division of Reactor Controls and Iluman Factors j Omce of Nuclear Reactor Regulation  ; 1 U.S. Nuclear Regulatory Commission Washington, DC 20555-0001 i )

         ,p*"'%,
 !     E
       -          i a
       *t.,...../

J i I i 1 t

l l l 1 I i AVAILABILITY NOTICE Availability of Reference Materials Cited in NRC Publications Most documents cited in NRC publications will be available from one of the following sources:

1. The NRC Public Document Room, 2120 L Street, NW., Lower Level, Washington, DC 20555-0001

2. The Superintendent of Documents U.S. Govemment Printing Office, P. O. Box 37082 Washington, DC 20402-9328

3. The National Technical Information Service, Springfield, VA 22161-0002 Although the listing that follows represents the majority of documents cited in NRC publica-tions, it is not intended to be exhaustive Referenced documents available for inspection and copying for a fee from the NRC Public Document Room include NRC correspondence and internal NRC memoranda; NRC bulletins, circulars, information notices, inspection and investigation notices; licensee event reports; vendor reports and correspondence; Commission papers; and applicant and licensee docu-ments and correspondence.

The following documents in the NUREG series are available for purchase from the Govemment Printing Office: formal NRC staff and contractor reports, NRC-sponsored conference pro-ceedings, international agreement reports, grantee reports, and NRC booklets and bro-chures. Also available are regulatory guides NRC regulations in the Code of Ferleral Regula-tions, and Nuclear Regulatory Commission issuances. Documents available from the National Technical information Service include NUREG-series reports and technical reports prepared by other Federal agencies and reports prepared by the Atomic Energy Commission, forerunner agency to the Nuclear Regulatory Commission. Documents available from public and special technical libraries include all open literature items, such as books, journal articles, and transactions. Federal Register notices, Federal and State legislation, and congressional reports can usually be obtained from these libraries. Documents such as theses, dissertations, foreign reports and translations, and non-NRC con-ferenc1 proceedings are available for purchase from the organization sponsoring the publica-tion cited. Single copies of NRC draft reports are available free, to the extent of supply, upon written request to the Office of Administration, Distribution and Mail Services Section, U.S. Nuclear 1 Regulatory Comr.lission, Washington DC 20555-0001. Copies of industry codes and standards used in a substantive manner in the NRC regulatory  ; process are maintained at the NRC Library, Two White Flint North,11545 Rockville Pike, Rock-ville, MD 20852-2738, for use by the public. Codes and standards are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from the American National Standards institute,1430 Broadway, New York, NY l 10018-3308. O I

At 5 TRACT NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," , establishes the policies, procedures, and practices for examining licensees and applicants for reactor operator and senior reactor operator licenses at power reactor facilities pursuant to Title 10, Part 55, of the Code of federal , Regulatfons (10 CFR Part 55). The related guidance that was previously published in the " Examiners' Handbook for Developing Operator Licensing Written Examinations" (NUREG/BR-0122, Rev. 5, dated March 1990) has been incorporated herein. NUREG/BR-0122 is no longer in effect. These examination standards are intended to assist NRC examiners and facility I licensees to better understand the processes associated with initial and ' requalification examinations. The standards also ensure the equitable and consistent administration of examinations for all applicants. These standards , , are for guidance purposes and are not a substitute for the operator licensing ' regulations (i.e.,10 CFR Part 55), and they are subject to revision or other changes in internal operator licensing policy.

~

This interim revision permits facility licensees to prepare their initial l operator licensing examinations on a voluntary basis pending an amendment to ' 10 CFR Part 55 that will require facility participation. The NRC intends to solicit comments on this revision during the rulemaking process and to issue a final Revision 8 in conjunction with the final rule. For examinations prepared by the NRC, this revision will become effective 60 days after its publication is noted in the federal Register. The corporate notification letters issued after the effective date will give facility licensees at least 120 days of advance notice that the examinations will be administered in accordance with the revised procedures. Facility licensees i that volunteered to prepare their examinations before the date of the federal l Register notice (FRN) are expected to prepare the examinations based on the guidance herein or the pilot examination guidance in Generic Letter 95-06,

       " Changes in the Operator Licensing Program," dated August 15, 1995.      Facility licensees that volunteer after the date of the FRN are expected to prepare the          i examinations based on the guidance herein.                                              i NUREG-1021                                 iii        Interim Rev. 8, January 1997
                                                                                               ]

i

I i

,a TABLE OF CONTENTS ABSTRACT                                                                         I TABLE OF CONTENTS EXECUTIVE 

SUMMARY

ABBREVIATIONS ES-101 Purpose and Format of Operator Licensing Examination Standards ES-102 Regulations and Publications Applicable to Operator Licensing i ES-201 Initial Operator Licensing Examination Process i ES-202 Preparing and Reviewing Operator License Applications  ! ES-203 (Deleted] ES-204 Processing Waivers Requested by Reactor Operator and Seni.or

,                      Reactor Operator Applicants ES-205            Procedure for Administering the Generic Fundamentals Examination Program l

ES-301 Preparing Initial Operati.ng Tests l ES-302 Administering Operating Tests to Initial License Applicants ES-303 Documenting and Grading Initial Operating Tests ES-304 [ Deleted) ES-401 Preparing Initial Site-Specific Written Examinations l ES-402 Administering Initial Written Examinations ES-403 Grading Initial Site-Specific Written Examinations ES-404 [ Deleted] ES-501 Initial Post-Examination Activities ES-502 Processing Requests for Administrative Reviews and Hearings After Initial License Denial m ES-601 Conducting NRC Requalification Examinations I \ O NUREG-1021 v Interim Rev. 8, January 1997

1 l l TABLE OF CONTENTS 1 ES-602 Requalification Written Examinations ES-603 Requalification Walk-Through Examinations ES-604 Dynamic Simulator Requalification Examinations ES-60S License Maintenance, License Renewal Applications, and Requests for Administrative Reviews and Hearings ES-701 Administration of Initial Examinations for Senior Operators Limited to Fuel Handling 1 ES-702 Administration of Requalification Examinations for Senior Reactor Operators Limited to Fuel Handling  ! 1 APPENDIX A Overview of Generic Examination Concepts ' APPENDIX B Written Examination Guidelines APPENDIX C Job Performance Measure Guidelines 1 APPENDIX D Simulator Testing Guidelines I APPENDIX E Policies and Guidelines for Taking NRC Examinations 1 APPENDIX F Glossary l l NUREG-1021 vi Intarim Rev. 8, January 1997 \

I I fm i EXECUTIVE

SUMMARY

Title 10, Part 55, of the Code of federal Regulations (10 CFR Part 55) i requires applicants for reactor operator (RO) and senior reactor operator (SRO) licenses to pass a written examination and an operating test that are l developed and administered in accordance with 10 CFR 55.41 cnd 55.45 or 55.43 l and 55.45, respectively. License examiners from the U.S. Nuclear Regulatory Commission (NRC) have historically prepared all of the licensing examinations using facility-provided reference materials. Nonetheless, the regulations do . not prohibit facility licensees from developing and submitting proposed I examinations for NRC review and approval. The NRC believes that facility  ; licensees have the expertise to prepare draft examinations that provide an i acceptable basis for initial operator licensing. The NRC bases this contention on the success of the licensed operator requalification examination program and the results of the pilot examinations that were administered in ' accordance with Generic letter (GL) 95-06, " Changes in the Operator Licensing Program," dated August 15, 1995. Fur @ rmore, the NRC expects that the facility licensees' greater familiarity with plant systems and procedures (relative to that of NRC license examiners) will enhance the content validity of the examinations and the efficiency of their development. Facility licensees are encouraged to develop the licensing examinations for their own R0 and SR0 applicants (including those limited to fuel handling); however, the NRC retains the authority to develop the examinations when deemed ( necessary. Facility licensees are expected to develop and submit their G proposed examinations based on the guidelines and instructions contained herein. Section 107 of the Atomic Energy Act of 1954, as amended, requires the Commission to prescribe uniform licensing conditions for operators. Therefore, the NRC discourages facility licensees from using testing methodologies that do not conform to the policies, procedures, and practices defined in this NUREG. Nevertheless, facility licensees may propose deviations from specific guidance in NUREG-1021, and the NRC will review and rule on the acceptability of the deviations. The NRC will make a reasonable attempt to administer all license examinations on the dates requested by the facility licensees. At times, however, resource limitations may compel the staff to prioritize its examination review and development activities based on need and safety considerations. Facility licensees are strongly encouraged to schedule their initial license examinations with their NRC regional office before commencing a license training class. For Revision 8, NUREG-1021 was reorganized to more clearly identify the various organizational responsibilities. It incorporates the methodology and lessons learned from the pilot examination program described in GL 95-06, as well as other improvements and clarifications recommended by industry groups, licensed operators, and NRC examiners and managers. This revision also formally implements Revision 1 of NUREGs-1122 and 1123, the " Knowledge and Abilities Catalogs" for pressurized and boiling water reactors, respectively.

  'U  NUREG-1021                                vii           Interim Rev. 8, January 1997

EXECUTIVE

SUMMARY

In addition, NUREG-1021 supersedes Revision 5 of NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations," dated March 1990. This document reflects the following significant changes: Abbreviations This list has been added to provide a central location for defining the acronyms and abbreviations used throughout this NUREG. ES-101 No significant changes. ES-102 No significant changes. ES-201 Resources permitting, the NRC will give each facility licensee an opportunity for one initial operator licensing examination per reactor type or site per fiscal year. Additional examinations may be scheduled as the workload permits. Facility licensees that write the examinations will conduct the following activities based on the guidance in NUREG-10?l: Observe various examination security and integrity criteria, including restrictions on which personnel can participate in developing the examinations, physical security considerations, and limits on the use of examination banks. At least 60 days before the scheduled examination date, prepare and submit for NRC review and comment an integrated examination outline, in accordance with ES-301, ES-401, and the associated quality assurance checklist. At least 30 days before the scheduled examination date, prepare and submit for NRC review and comment the complete , examination, in accordance with ES-301 and ES-401, along with a statement indicating the source of each test item proposed for use on the examination. Make examination changes as agreed upon with the NRC. Facility licensees shall designate a point of contact to work with the NRC chief examiner, as well as an authorized management representative to review and approve their submittals before sending them to the NRC for review and comment. The amount of reference material requested from the facility licensee will be adjusted based on the NRC's level of involvement in the examination development process. NUREG-1021 viii Interim Rev. 8, January 1997

EXECUTIVE

SUMMARY

i. 1 { ES-202 The eligibility criteria for senior reactor operators limited to

fuel handling (LSR0s) have been moved from ES-701 to ES-202.

1 To make the standard conform with 10 CFR 55 and current practice, ES-202 now includes a provision requiring that facility licensees j submit a written request to have a license examination administered to an applicant. i a The regions will verify that an applicant's name does not appear on l the " Restricted Individuals List" before accepting the application. 4 The requirement for five significant reactivity manipulations has

been clarified.

i' ES-204 The provision for LSR0s to be licensed at more than one site has been moved from ES-701 to ES-204. The regions may, under certain circumstances, waive the requirement l

for an examination for applicants that were previously licensed at 1 3

the same facility. ES-205 Facility licensees should notify the Operator Licensing Branch (OLB) i if they must modify their previously submitted registration letter j ( for the generic fundamentals examination by adding or deleting a person. ES-301 Dominant accident sequences, as determined by the facility

<                 licensee's probabilistic risk assessment or individual plant i                  examination, should be considered for sampling during the operating j                  test.

i i A site-specific task list may be used to supplement or override, on . a case-by-case basis, selected individual items in the NRC's ! knowledge and abilities catalog. The instructions for developing the operating test outline and the

final test items have been separated to facilitate their sequential

! preparation, review, and approval. Generic guidelines (i.e., those that apply to both initial and requalification examinations) for developing the walk-through and dynamic simulator tests have been relocated to Appendix C, " Job Performance Measure Guidelines," and Appendix D, " Simulator Testing Guidelines," respectively. No more than 80 percent of any applicant's walk-through test may be  ! taken directly from the facility licensee's item bank without O NUREG-1021 ix Interim Rev. 8, January 1997 I

1 l EXECUTIVE

SUMMARY

ES-301 significant modification, and no more than 30 percent of the walk-through may be repeated from the last NRC license examination at the facility. A quality assurance checklist has been included as an attachment to this standard to highlight various criteria and promote consistency. Each applicant's dynamic simulator test shall include at least one new or significantly modified scenario that the applicant has not had the opportunity to rehearse or practice. A quality assurance checklist has been included as an attachment to this standard to promote consistency by highlighting and suggesting target ranges for various criteria, including simulator critical tasks. The target ranges are based on a study of simulator scenarios used during past initial operator licensing examinations. No dynamic simulator scenarios will be repeated on successive days, and no more than 30 percent of the job performance measures taken from the bank will be repeated from one day to the next. ES-302 NRC examiners may use additional surrogate operators to augment the simulator crews if the technical specifications require the facility licensee to operate with more than two R0s in the control room. The facility licensee and NRC chief examiner should confirm that the simulator instructor's station, programmers' tools, and external interconnections do not compromise the integrity of the operating test. Appendix D briefly describes a number of vulnerabilities. The practice of conducting an exit briefing with the facility licensee after the operating tests are complete has been adopted as policy. The operating test briefing for the applicants has been moved to Appendix E, " Policies and Guidelines for Taking NRC Examinations." ES-303 The simulator operating test grading guidelines for errors having serious safety consequences (including critical tasks) have been clarified. Missing a critical task does not necessarily mean that an applicant will fail the simulator test, nor does success on every critical task prevent the examiner from recommending a failure if the applicant had other deficiencies that, in the aggregate, justify the failure based on the competency evaluations. The simulator operating test documentation requirements have been increased; examiners must now briefly describe the error that the applicant made to justify a grade of "2" on any rating factor. NUREG-1021 x Interim Rev. 8, January 1997

EXEttiTIVE

SUMMARY

ES-303 The applicants' responses to prescripted JPM follow-up questions will be evaluated based primarily on safety-significance. ES-401 This standard now includes instructions and forms for use in developing the written examination outline. References to NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations," have been deleted. References for guidance in developing multiple choice test items have been changed from NUREG/BR-0122 to Appendix B, " Written Examination Guidelines." Facility licersees shall submit an outline at least 60 days before the examination dete, followed by the " ready-to-use" examination at least 30 days before the examination date. A site-specific task list may be used to supplement or override, on a case-by-case basis, selected individual items in the NRC's knowledge and abilities catalog; a site-specific task list may not be used in place of the entire catalog. s This standard now includes several new criteria developed to ensure the integrity of examinations developed by facility licensees. .' These criteria include limits on the number of questions that can be taken directly from the facility licensee's item bank or can be repeated from earlier quizzes and examinations. In an effort to maintain examination quality and consistency, at least 50 percent of the questions on the examination shall test at the comprehension / analysis level. As a final check for technical acci; racy, facility licensees should consider administering the examination to one or more previously uninvolved licensed personnel (under security agreements). ES-402 Facility licensees will generally administer the written examinations after they are approved by the NRC. The facility licensees will document for subsequent review by the NRC any questions posed by and answers provided to the license applicants during the examination. If NRC examiners are on site, they may periodically monitor the administration process. The guidelines for briefing the applicants who will take the I examination have been moved to Appendix E.  ; ES-403 Facility licensees should collect and consider any questions and j comments made by the applicants after the examinations are j administered. Facility licensees that prepare and administer the O NUREG-1021 xi Interim Rev. 8, January 1997 l l

EXECUTIVE

SUMMARY

l ES-403 written examinations will grade the examinations, review the grading, evaluate the applicants' performance, and submit the results to the NRC for review and approval. The facility licensee shall justify all recommended question deletions and changes to the 1 answer key. l The discussion of examination grading quality assurance has been l moved to ES-501. ES-501 This standard summarizes the documentation that facility licensees are expected to provide to the NRC if they develop and administer , (in the case of the written) the license examinations. In addition, the standard now summarizes the post-examination quality assurance review process that was previously contained in ES-403. If a facility licensee recommends deleting or changing five percent or more of the questions on a written examination that it developed, it may be asked to explain why so many changes were necessary. The regional offices will delay issuing the licenses for applicants who pass the written examination with insufficient margin to ensure that the licensing decision will be sustained if additional questions are deleted or changed upon appeal. The examination report shall address any significant problems that the region or facility licensee encountered in developing the examination. The record keeping requirements, includinn the submittal of proposed examinations to the public document room (PDR), have been revised to reflect the new examination process. ES-502 For those denials that are sustained by the regional office on preliminary review, the Chief of the Operator Licensing Branch will determine whether to convene a panel or evaluate the appeal internally. Appeal panels, when required, will consist of three certified examiners, one of which will be designated chairperson. ES-601 The NRC will continue to monitor licensees for indications of undue stress during requalification examinations, however the stress feedback forms have been eliminated. ES-602 References to NUREG/BR-0122 have been changed because the guidance for developing multiple choice test questions is now in Appendix B. NUREG-1021 xii Interim Rev. 8, January 1997

'V                                     EXECUTIVE 

SUMMARY

ES-602 Because the guidelines previously documented in Attachment I to ES-602, " Policies and Guidelines for Taking NRC Written Examinations," are generally the same for initial and requalification examinations, they have been consolidated in Appendix E. ES-603 Attachment I to ES-603, " Guidelines for the Development and Use of Alternate Path JPMs"; Attachment 3, " Walk-Through Evaluation Guidelines"; Form ES-603-1, "JPM Quality Checklist"; and Form ES-603-2, "JPM Worksheet," have been moved to Appendix C because they apply to both initial and requalification examinations. Attachment 2 to ES-603, " Briefing Checklist - System Walk-Through," has been moved to Appendix E. ES-604 Attachment I to ES-604, " Critical Task Methodology,' and Attachment 3, " Quantitative and Qualitative Scenario Attributes," have been moved to Appendix D because they apply to both initial and requalification examinations. Attachment 2, " Dynamic Simulator Briefing Checklist," has been moved to Appendix E. ES-605 The policy on renewing inactive licenses has been clarified. The regions may, under certain circumstances, authorize an operator [3) (/ to temporarily suspend participation in the facility licensee's requalification training program. l ES-701 The eligibility criteria for LSR0s have been moved to ES-202, and the provision for LSR0s to be licensed at rare than one site is now discussed in ES-204. The standard has also been edited to clarify the differences between the full-scope SR0 and the LSR0 examinations. l The number of systems tested in Category B of the operating test has been decreased from six to five, and the requirement to test a normal evolution during each of the two discussion scenarios in Category C has been eliminated. The number of subject areas to be evaluated with questions when it is not practical to conduct or simulate a job performance measure has been decreased. ES-702 This standard has been edited to clarify the differences between the full-scope and the LSR0 requalification examinations. Whenever possible, the facility licensee should include an LSR0 on the requalification examination team. f ( \ NUREG-1021 xiii Interim Rev. 8, January 1997

EXECUTIVE

SUMMARY

l Appendix A This new appendix discusses the generic examination concepts that play a role in the operator licensing process. It includes much of the information that was previously contained in NUREG/BR-0122 as well as discussions of new topics that have a bearing on the level of difficulty of an examination. Appendix B This new appendix incorporates the guidance for developing written test questions that was previously contained in NUREG/BR-0122. It focuses primarily on multiple-choice ouestions, the only type currently permitted on the initial c'>erator licensing examination, and includes examples to illustrate various psychometric concepts. Appendix C This new appendix summarizes the guidelines concerning job perfomnea measures that apply to both initial and requalification examinations. Much of this information was previously contained in Attachments to ES-603. There are no significant policy changes. Appendix D This new appendix summarizes the dynamic simulator scenario guidelines that apply to both the initial and requalification examination programs. Much of the information was previously contained in ES-301 and Attachments to ES-604. Appendix 0 also describes a number of simulator security vulnerabilities (related to features of the instructor's station, programmers' tools, and external interconnections) that NRC examiners and facility personnel should consider when preparing and administering operating tests. Appendix E This new appendix summarizes all of the policies and guidelines applicable to examinees who will be taking an , initial or requalification examination. The information was l previously contained in ES-302, ES-402, ES-602, ES-603, and 1 ES-604; there are no significant policy changes. Appendix F This new appendix provides a central location for defining terms used throughout this NUREG. l l NUREG-1021 xiv Interim Rev. 8, January 1997 9 I

    - .- .- . -. .             . - ~ . .        . . - . -           - - - . - _ - - - - -                                  - . - - . - .

l s i i l i~ l ABBREVIATIONS l 1 i j AC alternating current automatic depressurization system d ADS

AFW auxiliary feedwater l ANSI /ANS American National Standards Institute /American Nuclear Society i A0 auxiliary operator

{ AOP abnormal operating procedure ! APRM average power range monitor ARP alarm (or annunciator) response procedure i ATWS[T] anticipated transient without scram [ trip] { B&W Babcock and Wilcox

;                BWR                 boiling water reactor j                 CAL                 confirmatory action letter

. CCW component cooling water l CFR Code of Federal Regulations l CRD control rod drive ! CRT criterion-referenced test CT critical task j CTMT containment

i. CVCS chemical and volume control system j DAS dominant accident sequence DC direct current DHR decay heat removal

.                DRCH                Division of Reactor Controls and Human Factors l                 EAL                 emergency action level
!                ECA                 emergency contingency action (procedure) l                 ECCS                emergency core cooling system 8

ECP estimated. critical position ! EDG emergency diesel generator i EHC electrohydraulic control i E0P emergency operating procedure i EPIP emergency plan implementing procedure i EQB examination question bank ! ES examination standard 1 ESF engineered safety feature FHE fuel handling equipment { FRP functional recovery procedure j FSAR final safety analysis report  ; j GFE generic fundamentals examination  ! 3 GL generic _ letter i t GUI graphic user interface ! HP health physics , HPCI high pressure coolant injection } HPCS high pressure core spray

 !              HVAC                 heating, ventilation, and air conditioning                                                                l l                 IC                  instrumentation and control                                                                               ;

i INP0 Institute of Nuclear Power Operations I l 4 i j NUREG-1021 xv Interim Rev. 8, January 1997 a i i

i ABBREVIATIONS IP inspection procedure IPF individual plant examination IPA intermediate range monitor JDM job performance measure JIA job task analysis K/A knowledge and ability KSA knowledge, skill, and ability LC0 limiting condition for operation LER licensee event report LOCA loss of coolant accident LPCI low pressure coolant injection LPCS low pressure core spray LPRM local power range monitor LSR0 limited senior reactor operator MIP master inspection plan MSIV main steam isolation valve NEI Nuclear Energy Institute NRC Nuclear Regulator Commission , N0P normal operating procedure NRR Office of Nuclear Reactor Regulation NRT norm-referenced test NWPA Nuclear Waste Policy Act (of 1982) OJT on-the-job training OLA operator licensing assistant OLB Operator Licensing Branch OLTS operator licensing tracking system OMB Office of Management and Budget  ! PCIS primary containment isolation system l POR public document room PORV power-operated relief valve i PPR plant performance review l PRA probabilistic risk assessment i PWR pressurized water reactor ' QA quality assurance RBM rod block monitor RCA radiologically controlled area l RCIC reactor core isolation cooling  ! RG Regulatory Guide RHR residual heat removal RMCS reactor manual control system R0 reactor operator ROI report on interaction RM radiation monitor RPIS rod position indicction system RPS reactor protection system RPV reactor pressure vessel RWST refueling water storage tank ' NUREG-Id21 xvi Interim Rev. 8, January 1997

i 4 ABBREVIATIONS S satisfactory SALP systematic assessment of licensee performance l SAT systems approach to training ) , SGTS standby gas treatment system ' i SD standard deviation > SGTR steam generator tube rupture  ! 1 SI safety injection  ! 2 SLC standby liquid control  ; 3 SME subject matter expert  ! i SR0 senior reactor operator ' i SRP Standard Review Plan i 1 SRV STA safety relief valve shift technical advisor ! TDAFW(P) turbine-driven AFW (pump) TS technical specification j U unsatisfactory UPS uninterruptible power supply W/T walk-through l NUREG-1021 xvii Interim Rev. 8, January 1997 j l I i

ES-101 PURPOSE AND FORN.T OF OPERATOR LICENSING EXAMINATION STANDARDS  : ( A. PURPOSE i Title 10, Part 55, of the Code of federal Regulations (10 CFR Part 55)  ! requires that applicants for reactor operator (RO) and senior reactor operator (SRO) licenses pass written examinations and operating tests (both-initially  ; and for requalification). Moreover, the regulations mandate that the license l examinations must be developed and administered in accordance with  ! 10 CFR 55.41 and 55.45 for R0s, or 10 CFR 55.43 and 55.45 for SR0s. The " Operator Licensing Examination. Standards for Power Reactors" (NUREG-1021)  ! establish the procedures and practices for administering the required initial ' and requalification written examinations and operating tests. These standards describe the provisions of the act and regulations on which the program is based. They also ensure the equitable and consistent administration of examinations to all applicants and licensed operators at all facilities subject to the regulations. B. FORMAT Each standard explains the rules, procedures, and practices for a particular aspect of.the program. For ease of reference, each examination standard (ES) is assigned a three-digit number, and related standards are grouped together O in the sense that standards beginning with the same digit apply to related aspects of the program, as follows: ES-Ixx - General ES-2xx - Initial pre-examination activities ES-3xx - Initial operating tests ES-4xx - Initial written examinations ES-5xx - Initial post-examination activities ES-6xx - Requalification examinations l ES-7xx - Fuel handling examinations l l O NUREG-1021 1 of 1 Interim Rev. 8, January 1997

i j . a ES-102 , REGULATIONS AND PUBLICATIONS APPLICABLE l i TO OPERATOR LICENSING A. PURPOSE i This standt.rd lists the U.S. statutes and the regulations of the U.S. Nuclear 1 Regulatory Commission (NRC) that establish the requirements for conducting i operator licensing examinations. It also identifies the regulatory guides and - } NUREG reports that establish the procedures for implementing the regulations and administering the examinations, as well as standards of the American  !

,                     National Standards Institute /American Nuclear Society (ANSI /ANS) that may                             ,

provide additional guidance. 4 Regulatory guides (RGs), NUREG reports, and industry standards are not

requirements, except as specified in Commission orders or as committed to by '

i the facility licensee. The appropriate revisions should be consulted as '

j. referenced in the facility's FSAR or approved training program. The following paragraphs summarize the latest revisions of these documents.

! B. STATUTES I

l. Atonic Energy Act of 1954 ,

i Section 107 of the Atomic Energy Act of 1954 (42 U.S.C. 2137), as

amended, requires that the NRC prescribe uniform conditions for l licensing individuals as operators of production and utilization  !

facilities, determining the qualifications of these individuals, and

issuing licenses to such individuals.

2. Nuclear Waste Policy Act of 1982 Section 306 of the Nuclear Faste Policy Act of 1982 (42 U.S.C. 10226, 96 Stat. 2201 at 2262 - 2263) directs the NRC to establish requirements i governing (1) simulator training for applicants for operator licenses ,

i and for operator requalification training programs, (2) NRC i j administration of requalification examinations, and (3) operating tests  ! i at civilian nuclear power plant simulators. i C. REGULATIONS l 1. 10 CFR Part 2. Rules of Practice 4 j The regulations in 10 CFR Part 2 govern the conduct of all proceedings

under the Atomic Energy Act of 1954, as amended, and the Energy l

Reorganization Act of 1974 with regard to (a) granting, suspending, i

revoking, amending, or taking other action with respect to any license; j (b) imposing civil penalties; and (c) public rulemaking.

i i i NUREG-1021 1 of 6 Interim Rev. 8, January 1997 l, a u v.g.,--- , ,, ,--e - ---

ES-102 10 CFR 2.103(b)(2) establishes the applicant's right to demand a review of a propc:ed license denial, and defines the applicant's appeal and hearing rights. Subpart G, " Rules of General Applicability," governs all adjudications initiated by the issuance of an order to show cause, an order designating the time and place of a hearing requested by a person charged with a violation, and a notice of hearing. Subpart L, " Informal Hearing Procedures for Adjudications in Materials and Operator Licensing Proceedings," governs proceedings for the issuance, renewal, or licensee-initiated amendment of an operator or senior operator license.

2. 10 CFR Part 9. Public Records The regulations in 10 CFR Part 9 prescribe the rules governing the NRC's public records that relate to any proceeding subject to 10 CFR Part 2.

Subparts A and B describe and implement the requirements for balancing the public's rights to information under the freedom of Information Act and the NRC's responsibility to protect personal information under the Privacy Act. Subparts C and D implement the provisions of the Sunshine Act, concerning the opening of Commission meetings to public observation. They .tiso describe the procedures governing the production of agency records, information, or testimony in response to subpoenas or demands of courts or other judicial authorities in State and Federal proceedings.

3. 10 CFR Part 20. Standards for Protection Aaainst Radiation The regulations in 10 CFR Part 20 establish standards for protection against radiation hazards arising from licensed activities. Some of the material is appropriate for inclusion in the examinations administered to candidates for R0 or SR0 licenses.

4. 1C CFR Part 50. Licensina of Production and Utilization Facilities ,

10 CFR 50.34(b)(8) requires that the final safety analysis report (FSAR) include a description of the operator requalification program. That description forms the basis for the inspection, audit, and approval of requalification programs. NUREG-1021 2 of 6 Interim Rev. 8, January 1997

ES-102 10 CFR 50.54(1-1) requires facility licensees to implement an operator requalification program that meets the requirements of 10 CFR 55.59(c) within 3 months after receiving a facility operating license. Notwithstanding the provisions of 10 CFR 50.59, the licensee may not decrease the scope of its approved requalification program without authorization from the Commission. 10 CFR 50.54(k) - (m) contain regulations restricting control manipulations to licensed operators. These regulations are conditions of all facility licenses issued under 10 CFR Part 50. 10 CFR 50.74 requires facility licensees to notify the Commission within 30 days if there is a change in the status of a licensed R0 or SRO.

5. 10 CFR Part 55. Operators' Licenses 10 CFR Part 55 is the implementing regulation that establishes the requirements and the regulatory basis for licensing and requalifying R0s and SR0s.

D. REGULATORY GUIDES

1. Reaulatory Guide 1.8. "0ualification and Trainino of Personnel for Nuclear Power Plants." Rev. 2. April 1987 i

( Section C.1 of this RG currently endorses, with exception, ANSI /ANS 3.1-1981, "American National Standard for Selection, Qualification, and Training of Personnel for Nuclear Power Plants" (effective March 31, 1988). The NRC is currently reviewing, and is expected to endorse, with exception, the 1993 version of ANSI /ANS 3.1. Section C.2 endorses, with exception, ANSI /ANS N18.1-1971, "American National Standard for Selection and Training of Nuclear Power Plant Personnel."

2. Reculatory Guide 1.33. "Ouality Assurance Proaram Reautrements -

Operations" Appendix A to this RG contains a list of typical procedures for pressurized water reactors and boiling water reactors.

3. Reaulatory Guide 1.114. " Guidance on Beina an Operator at the Controls of a Nuclear Power Plant" This RG describes a method acceptable to the NRC staff for complying with the Commission's regulations in 10 CFR 50.54(k) - (m), which require the presence of an R0 at the controls of a nuclear power unit and an SR0 in the control room from which the nuclear power unit is being operated.

NUREG-1021 3 of 6 Interim Rev. 8, January 1997

ES-102

4. Reaulatory Guide 1.134. " Medical Evaluation of Licensed Personnel for Nuclear Power Plants." Rev. 2. April 1987 This RG currently endorses ANSI /ANS 3.4-1983, " Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Pl ants. " The RG is being revised to endorse the 1996 version of the standard.

5. Reaulatory Guide 1.149. " Nuclear Power Plant Simulation Facilities for Use in Operator License Examinations." Rev. 2. April 1996 This RG currently endorses, with exception, ANSI /ANS 3.5-1993, " Nuclear Power Plant Simulators for Use in Operator Training and Examination."

E. NUREG REPORTS

1. NUREG-0660. Vol .1. "NRC Action Plan Developed as a Result of the TMI-2 Accident." May 1980 Item I.A.4.2 of this document describr.s the guidelines for long-term simulator upgrades.

2. F!UREG-0737. " Clarification of TMI Action Plan Reouirements." November 1R80 This document clarifies the following action plan items which are intended to upgrade the training, licensing, education, and experience ,

of operators on the basis of experience gained from the accident at Three Mile Island, Unit 2: Item I.A.2.1, "Immediate Upgrading of R0 and SR0 Training and Qualifications" Item 1.A.2.3, " Administration of Training Programs" Item 1.A.3.1, " Revised Scope and Criteria for Licensing Exams" Item II.B.4, " Training for Mitigating Core Damage"

3. NUREG-0800. " Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants. LWR Edition." July 1981 Section 13.2, " Reactor Operator Training," describes the training and licensing of operators and identifies information to be submitted by applicants for construction permits and operating licenses.

NUREG-1021 4 of 6 Interim Rev. 8, January 1997

( ES-102

4. NUREG-1122. "Knowledae and Abilities Cataloa for Nuclear Power Plant Operators: Pressurized Water Reactors." Rev. 1. Auaust 1995 This document provides the basis for developing content-valid licensing examinations for operators at pressurized water reactors (PWRs). It contains knowledge and ability (K/A) statements that have been rated for l their importance to ensuring that the plant is operated in a manner l consistent with the health and safety of plant personnel and the public. '

i

5. NUREG-1123. "Knowledae and Abilities Cataloa for Nuclear Power Plant Doerators: Boilina Water Reactors." Rev. 1. Auaust 1995 This document provides the basis for developing content-valid licensing examinations for operators at boiling water reactors (BWRs). It contains K/A statements that have been rated for their importance to ensuring that the plant is operated in a manner consistent with the health and safety of plant personnel and the public.

6. NUREG-1291. "BWR and PWR Off-Normal Event Descriptions." November 1987 The reactor event descriptions in this document provide a reliable, performance-based source of information that examiners may use to design simulator scenarios that will be a valid test of an applicant's ability g to safely and competently perform all licensed duties and

- i responsibilities. V 7. NUREG-1560. " Individual Plant Examination Proaram: Perspectives on Reactor Safety and Plant Performance" This report provides perspectives gained by reviewing 75 individual plant examination (IPE) submittals pertaining to 108 nuclear power plant units. Chapter 13, " Operational Perspectives," is of particular 1 interest because it identifies a number of important human actions that should be considered for evaluation on BWR and PWR licensing and requalification examinations.

8. NUREG/BR-0122. " Examiners' Handbook for Developina Operator Licensina Written Examinations." Rev. 5. March 1990 This document, which presented a procedure for systematically constructing content-valid licensing examinations for nuclear power plant operators, has been incorporated into the examination standards in NUREG-1021, Rev. 8. It may be used for historical perspective, but is no longer used for developing examinations.

O NUREG-1021 5 of 6 Interim Rev. 8, January 1997

ES-102 p, 1HDUSTRY STANDARDS

1. ANSI /ANS 3.1. "American National Standard for Selection. Qualification and Trainina of Personnel for Nuclear Power Plants" This standard provides criteria for selecting and training nuclear power plant employees performing a variety of functions at various levels of responsibility (e.g., managers, supervisors, operators, and technicians). RG 1.8, Revision 2, endorsed, with exceptions, the 1981 version of the standard; the 1987 version was never endorsed by the NRC; the 1993 version is currently under review and is expected to receive a qualified endorsement from the NRC.

2. ANS 3.2 (ANSI N18.7-1976). " Administrative Controls and 0A for the Operational Phase of Nuclear Power Plants" This standard provides guidance and recommendations for administrative rules of practice and related subjects and for preparing procedures and audit programs. See RG 1.33.

3. ANSI /ANS 3.4-1996. " Medical Certification and Monitorina of Personnel

• Reauirino Operator Licenses for Nuclear Power Plants" This standard is the basic document covering the general health and disqualifying conditions applicable to license applicants and licensed personnel. RG 1.134 currently endorses the 1983 version of the standard but is being revised to recognize the newer document.

4. ANSI /ANS 3.5-1993. " Nuclear Power Plant Simulators for Use in Operator Trainina" This standard establishes the minimum functional requirements and capabilities for nuclear power plant simulators for use in operator training. Revision 2 of RG 1.149 endorses this standard, with exceptions.

NUREG-1021 6 of 6 Interim Rev. 8 January 1997

                                                                                                                                          .____.__.____s i

4 l ES-201  ! j INITIAL OPERATOR LICENSING EXANINATION PROCESS 1 4 A. PURPOSE f l This standard describes the activities that must be completed to prepare for l l initial operator licensing examinations (including written examinations and j i operating tests) at power reactor facilities. It. includes instructions for - l scheduling and coordinating examination development, assigning NRC examiners and facility personnel, maintaining examination security, and obtaining j reference and examination paterials from the facility licensee. i B. BACKGROUND l I Title 10,.Part 55, of the Code of Federal Regulations (10 CFR Part 55) l requires that applicants for reactor operator (RO) and senior reactor operator-l (SRO) licenses pass a written examination and an operating test. Moreover, j the regulation requires that the license examinations be developed and administered in accordance with 10 CFR 55.41 and 55.45 for R0s or 10 CFR 55.43 i and 55.45 for SR0s. License examiners from the NRC have historically prepared i all of the licensing examinations using facility-provided reference materials.

Nonetheless, the regulations do not prohibit facility licensees from writing

! and submitting examinations for NRC review and approval. The NRC believes j that facility licensees now have the expertise to prepare examinations that

,                           provide an acceptable basis for initial operator licensing. The NRC bases

! this belief on the success of the licensed operator requalification j examination program and the results of the pilot examinations that were i administered in accordance with Generic Letter (GL) 95-06, " Changes in the Operator Licensing Program," dated August 15, 1995. Furthermore, the NRC expects that the facility licensees' greater familiarity with plant systems j and procedures (relative to that of NRC license examiners) will enhance the ! content validity of the examinations and the efficiency of their development. i Facility licensees that choose to participate shall develop and submit the examinations in accordance with the instructions contained herein. The NRC j retains the authority to develop the examinations if it loses confidence that l a-facility licensee will develop examinations upon which the NRC can base its

licensing decisions. The NRC will also prepare selected examinations on a

random basis to maintain examiner proficiency.

! The NRC's goal is to provide each facility with an opportunity for at least j one initial operator licensing examination per fiscal year. Additional

examinations may be scheduled on a case-by-case basis if NRC resources are j available. Examinations for fewer than three applicants should be scheduled only under extenuating circumstances such as a shortage of licensed R0s or 4 SR0s at the facility. If a facility licensee has fewer than three license applicants, the examinations may be delayed until more applicants are trained.

1 j Other pre-examination activities, such as submitting and reviewing license

;                           applications and eligibility waivers and administering the generic NUREG-1021                                                                 1 of 24          Interim Rev. 8, January 1997 p

4- . v . - --- _ - . - - . _ _ . . . . . . _- --s y, .,-,--.-,_,4_m - . . -

i l ES-201 fundamentals examination program, are addressed in ES-202, ES-204, and ES-205.  ! Specific instructions for developing, administering, and grading the written  ; examinations and operating tests are found in ES-401 through ES-403 and ES-301  ! through ES-303, respectively. Post-examination administrative activities, including management review of the examination results and preparation of  ; examination reports, are discussed in ES-501. Cross-references to each of these standards have been made where appropriate. 1 C. RESPONSIBILITIES Facility licensees and NRC staff should use Form ES-201-1, " Examination Preparation Checklist," to track the examination preparations. As noted on the form, the target due dates can be adjusted as necessary to accommodate a given situation. The NRC chief examiner will initial the items as they are completed and ensure that the original form is retained for the master examination file (refer to ES-501).

1. Facility Licensee

a. The facility licensee shall apprise its NRC regional office of changes in its examination requirements.

The facility licensee should respond in writing to the NRC's annual administ ative letter soliciting estimated operator licensing needs and notify its NRC regional office if its examination requirements change significantly from those stated in its response. Facility licensees are strongly encouraged to schedule their examinations with their NRC re commencing an initial license training class..gional office before If the NRC determines that a facility is unable to develop acceptable examinations, the examinations could be delayed significantly until sufficient NRC resources can be scheduled to develop and conduct the examinations, or until the facility licensee can develop an acceptable examination.

b. In accordance with 10 CFR 55.49, facility licensees and applicants shall not engage in any activity that compromises the integrity of any application, test, or examination required by 10 CFR Part 55.

Attachment 1 of this ES summarizes several examination security and integrity considerations. l

c. All facility personnel involved with an examination are subject to the restrictions stated in Section D of this ES.

The facility licensee shall designate a point of contact to work with the NRC chief examiner and assign additional personnel as required to ensure that the examinations are developed, reviewed, administered, and graded in accordance with the applicable examination standards. The facility licensee may use contractors NUREG-1021 2 of 24 Interim Rev. 8, January 1997

' I i ES-201 l or other outside assistance to develop the examinations, but the i licensee bears full responsibility for the product. I

                -d. The facility contact shall submit the required reference materials, examination outlines, and examinations, as applicable, I

based on the level of facility participation. Form ES-201-1 l 1 specifies target due dates for the various materials; the actual dates may be adjusted with prior agreement from the NRC regional office. All examination-specific materials (i.e., the examination outlines and examinations) shall be controlled and protected as sensitive information (refer to Attachment 1).

e. The examination outlines and the examinations shall be prepared in accordance with the guidelines in ES-301, ES-401, and ES-701, as applicable. The proposed outlines and examinations shall cover all portions of. the license examination (written, dynamic simulator, and walk-through) at all license levels relevant to the applicants'(RO, SRO, and limited SRO) to be tested.

An authorized facility representative (i.e., a supervisor or manager having authority to speak on behalf of the facility licensee) shall independently review and approve all examination-O specific materials before they are submitted to the NRC regional office for review and approval. Supervisors who approved the course materials but did not perform instructional activities for the license applicants may review and approve the examination materials. i In conducting this review, the authorized facility representative shall use Forms ES-201-2, " Examination Outline Quality Assurance Checklist"; ES-301-3, " Operating _ Test Quality Assurance Checklist"; ES-301-4, " Simulator Scenario Quality Assurance Checklist"; and ES-401-6, " Written Examination Quality Assurance Checklist. "

f. In its examination submittal to the NRC, the facility licensee ,

shall indicate the source of each test item proposed for use on l the written examination and the operating tests. The following i information shall be included for each item: State the source of each item (e.g., Is the item taken directly, without changes, from the facility licensee's bank, another facility's bank, the NRC's bank, or an old NRC exam; is the item a modified version of a bank item; or is the item new?). Items that the facility licensee has obtair.ed from another bank and deposited in its own bank may be treated as " bank" items provided they have an equal chance of being selected for use on the examination. Items j V NUREG-1021 3 of 24 Interim Rev. 8, January 1997 i i

ES-201 from another bank may be treated as new items if they have not been made available for review and study by the license applicants and there is no basis (e.g., historical precedent or reciprocal arrangements with the other facility licensee) for the applicants to predict their use on the examination. l For those items that are taken directly from the facility licensee's bank, state if and when the item was used on the i last two NRC license examinations at the facility or to evaluate the applicants' performance during their current  ; license training class. l For those items that are derived by modifying existing bank items, note the changes that were made or submit a copy of the item from which it originated. '

g. The facility licensee shall make its simulation facility available, as necessary, for NRC examiners to prepare for and administer the operating tests. The NRC will take reasonable efforts to minimize the impact on other training activities.

Before developing or administering an initial licensing ' examination, facility licensees are encouraged to review the simulator examination security considerations in Appendix D to NUREG-1021 for applicability to their facility. Because facility licensees are more familiar than the NRC examiners with the unique capabilities, limitations, and vulnerabilities of their simulators, it is expected that the licensees will take responsibility for determining and implementing whatever measures might be necessary to ensure the integrity of the operating tests.

h. The facility licensee shall meet with the NRC in the regional office or at the facility, as necessary and appropriate, to review the examinations and discuss potential changes.

1. If the facility licensee developed the examinations, it will generally make any necessary changes as agreed upon with the NRC; however, the NRC retains final authority to approve the examinations.

J. In accordance with ES-202, the facility licensee shall submit the license applict.tions along with a letter requesting that licensing examinations be administered.

2. NRC Reaional Manaaement. Supervision. and Desianees i

a. The regional office shall schedule the NRC's initial operator l licensing examinations and shall arrange for the development, l administration, and grading of those examinations as discussed i below. The regional office shall periodically review each '

NUREG-1021 4 of 24 Interim Rev. 8, January 1997

ES-201 facility licensee's examination requirements and shall negotiate with the facility licensee's training representatives as necessary to schedule specific examination dates as close as possible to.the dates assigned on the national examination schedule, consistent with operational safety requirements and NRC resource availability.

b. Approximately six months before each anticipated examination date, the regional office should contact the facility licensee and confirm the examination date(s) and the expected number of applicants to be examined. The regional office should use that information to estimate the required number of NRC examiners and to make preliminary work assignments.

c. The regional office should contact the facility licensee by telephone at least four months before the scheduled examinations to reconfirm the expected number of applicants and the examination dates, and to make other preliminary arrangements for developing the examinations. The person who contacts the facility licensee shall discuss the following examination arrangements:

the guidelines for ensuring examination integrity and security (refer to Attachment 1) the need to have the examination outlines delivered to the O - NRC at least 60 days before the scheduled examination date; the need to have the reference materials necessary for the NRC to develop the examination (if applicable; refer to Attachment 2) delivered to the regional offira at least 60, but preferably 90, days before the scheduled examination date the guidelines for developing, administering, and grading the written examinations, as applicable (ES-401, ES-402, and ES-403,respectively) the need to have the simulator available and the guidelines for developing and administering the operating tests (ES-301 and ES-302, respectively) the need to have the examinations and the supporting reference materials (refer to Attachment 2) delivered to the regional office at least 30 days before the scheduled examination date the requirements (refer to 10 CFR 55.31) and guidelines (refer to ES-202) for submitting the license applications

d. The regional office shall normally issue a letter confirming the '

verbal arrangements 120 days before the examination begins. The letter should be addressed to the person at the highest level of  ! corporate management who is responsible for plant operations 1 (e.g., Vice President of Nuclear Operations). Attachment 3 is an j example of such a letter; the exact wording may be modified as ] necessary to reflect the situation. NUREG-1021 5 of 24 Interim Rev. 8, January 1997 l l l i

! l l l ES-201 l e. Approximately four months before the scheduled examination, the regional office will assign the required number of examiners to develop, prepare for, and administer the examination as arranged i with the facility licensee. The regional office will also  ! designate a chief examiner to coordinate the examination project with the facility licensee and other examiners assigned to the examination. When making assignments, the region should consider each examiner's certification status, other examination commitments, possible conflicts of interest (as discussed in Section D of this ES) and general availability. The region should try to assign a sufficient number of examiners  ! so that no examiner will have to administer more than four operating tests per week.

f. The regional office will evaluate each examination assignment to determine if some or all of the assigned examiners should make a separate preparatory site visit. The purposes of such a visit may include providing examiner orientation, retrieving additional reference material, or reviewing and validating the examinations.

When making a decision, the region should carefully weigh the j costs and benefits associated with each additional trip to the facility. The region should also consider such factors as the experience of the assigned examiners, the quality of the facility licensee's examinations (if applicable), the number of written examinations and operating tests to be validated, and the status i of the simulation facility (e.g., Is it new or recently upgraded?). In addition, the region should consider the alternative of reviewing the written examination (s) and operating test (s) with the facility licensee via telephone (if the examination quality is high) or in the regional office, as well as the alternative of validating the operating test (s) on-site at the beginning of the examination week. I l

g. Upon receiving the preliminary license applications, approximately 30 days before the examination date, the regional office shall review the applications in accordance with ES-202. In addition, the regional office shall evaluate any waiver requests in accordance with ES-204 to determine if the applicants meet the eligibility criteria specified in 10 CFR 55.31.

After reviewing and approving the preliminary license applications and resolving all waiver requests, the region will prepare an examination assignment sheet (in the format of Attachment 4) as far in advance as possible, but at least two weeks before the scheduled examination date. The region will review and revise the assignment sheet as necessary after receiving and evaluating the i final license applications. NUREG-1021 6 of 24 Interim Rev. 8, January 1997

' fh d ES-201 The assignment sheet will identify the chief and other examiners by name and list the applicants by name, docket number, and type of examination (e.g., SR0 upgrade, R0 written only) to be administered. All applicants listed on the assignment sheet should be administered complete examinations (written and operating) as indicated under " Examination Type" unless waivers have been granted in accordance with ES-204. A copy of the assignment sheet will be distributed to all assigned examiners, the Operator Licensing Branch (0LB), and regional distribution.

h. The responsible regional supervisor will review the examination outlines and the draft examinations and evaluate any recommended changes and corrections noted during the chief examiner's review.

The supervisory review is not intended to be another detailed  ; review, but rather a check to ensure that all applicable administrative requirements have been implemented. If the outlines, examinations, and recommended changes are acceptable, the supervisor will authorize the chief examiner to resolve any noted deficiencies with the author or facility contact. If any of the facility-developed examination materials (written, walk-through, or simulator) require substantive changes and cannot i be made to conform with the examination standards at least five  : working days before the scheduled examination date, regional l management shall consult OLB and make a decision whether to 4 s proceed with the facility-developed , examinations or develop the examinations in-house. If the region does not have the resources I to ensure that acceptable examinations are prepared by the ' scheduled administration date, regional management shall negotiate with the facility licensee to reschedule the examinations as necessary. If examination problems are identified at the last i minute, the examinations should be postponed; substantive l examination changes will not be made during the examination week. ' The responsible supervisor will also ensure that any significant deficiencies and problems are addressed in the examination report in accordance with ES-501.

i. After the chief examiner has verified that the necessary changes and corrections have been made, the responsible supervisor will i review and approve the examinations for administration. Before signing the applicable quality assurance form (i.e., Form ES-301-3 and/or Form ES-401-6), the supervisor must be satisfied that the examination is acceptable for administration.

i j. If there is an indication that an examination may have been l compromised, the responsible supervisor will take action as l necessary to ensure and restore the integrity and security of the l examination process. Actions may include not giving the i examination, making additional changes to the examination, voiding V NUREG-1021 7 of 24 Interim Rev. 8, January 1997

l the results if the examination has already been given, and possibly imposing enforcement action. The supervisor shall keep regional management and OLB informed of any concerns regarding examination integrity.

3. Assioned NRC Examiners

a. When assigned to administer operating tests for the first time at a particular facility, the examiner should inform the chief examiner and the responsible supervisor so that arrangements can be made to conduct an orientation trip to the facility as described in Item C.2.f, if deemed appropriate.

b. NRC examiners must monitor and ensure the integrity of the j examination process. If they perceive that a compromise has i occurred, they must immediately report it to the responsible i regional supervisor so that the necessary actions can be taken to I restore the integrity of the examination. Attachment I summarizes several examination security and integrity considerations. )

c. The assigned examiners shall review and inventory the reference materials received from the facility licensee in response to the 120-day corporate notification letter. The purpose of this review is to determine if the materials are complete and adequate to enable the regional office to review or develop the examinations, as applicable. If it is not, the reviewer (s) shall inform the chief examiner and the responsible supervisor and request that the facility licensee send any additional materials that might be i required. If necessary, an examiner may review and select '

additional reference materials during a site orientation trip (refer to Item C.2.f). 1

d. The chief examiner will work with.the assigned examiners and the  ;

designated facility contact, as applicable, to ensure that the ' examination outlines and examinations are developed in accordance with the applicable examination standards. The chief examiner i should adapt the level of oversight and coordination based upon the experience of the individuals who are preparing the examinations. Facility employees are generally less familiar with the examination standards and will require more oversight to ensure that a quality examination is ready on time.

e. The chief examiner will review the examination outlines using Form ES-201-2, " Examination Outline Quality Assurance (QA) Checklist,"  ;

as a guide. A thorough and timely review (i.e., within 5 working l days) will minimize the potential for significant problems with the examinations. The chief examiner will note any necessary changes and forward the outlines to the responsible supervisor (or a designated alternate l NUREG-1021 8 of 24 Interim Rev. 8, January 1997

O ES-201 other than the chief examiner) for review and comment before + resolving any deficiencies with the author or facility contact. If the outlines are significantly deficient, refer to Item C.2.h for additional guidance.

f. The chief examiner will review the written examinations and operating tests for quality in accordance with the ap)11 cable QA checklists (refer to ES-301.and ES-401) forwarded witi the examination. If the chief examiner wrote the operating tests, another NRC examiner shall perform the independent review. The regional office may conduct additional reviews at its discretion if resources permit.

It is especially important that facility-developed examinations  ! and tests be reviewed promptly because of the extra time that may  ! be required if extensive changes are necessary. The QA reviews ) should be completed within two weeks after the examinations and i tests are received from the author or facility contact. l l l The chief examiner will note any necessary changes and forward the  ! examinations and tests to the res)onsible supervisor (or a i designated alternate other than t1e chief examiner) for review and I comment before reviewing the examinations with the author or  ; 1' facility contact. There are no minimum or maximum limits on the ' number or scope of changes the NRC may direct the facility  !

(. licensee to make to its proposed examinations, provided they are  !

( necessary to make the examinations conform with established acceptance criteria or to attain an appropriate level of examination difficulty. Chief examiners shall exercise their experience and judgement to ensure that the level of difficulty remains consistent with that expected on NRC-prepared i examinations. If the examinations are significantly deficient, refer to Item C.2.h for additional guidance.

g. Upon supervisory ap3roval, generally about two weeks before the examinations are scleduled to be given, the chief examiner will review the written examinations and operating tests with the i facility licensee. If the examinations were developed by the NRC, J

          .              the review shall be conducted in accordance with the instructions in Attachment 5.

The chief examiner may conduct the examination review via telephone, in the regional office, or at the facility, as appropriate to the circumstances, depending on the extent of the changes, and as approved by the responsible regional supervisor

(refer to Item C.2.f).

i If the facility reviewers have significant disagreements with the chief examiner, the chief examiner will inform the responsible regional supervisor so that the disagre'ements can be resolved before the examinations are administered. 4 A NUREG-1021 9 of 24 Interim Rev. 8, January 1997 t

h. After the examination corrections have been made, the chief examiner shall verify that the changes are a)propriate and route the examinations and the mark-up drafts to tie responsible supervisor for final approval.

1. As soon as possible after the responsible supervisor has approved the operating tests for administration, the chief examiner shall distribute copies of the scenarios, job performance measures (JPMs), and questions to the other assigned examiners so that they can familiarize themselves with those materials and be better prepared to probe the applicants' deficiencies if required.

J. The chief examiner should work with the designated facility contact to schedule the operating tests to optimize efficiency and the mix of R0 and SR0 applicants in the crews assembled for the simulator examinations. The number of applicants on a crew shall not exceed the number of assigned examiners (i.e., one-on-one evaluations are mandatory). However, if the facility licensee's technical specifications routinely require more than two R0s to be stationed in the control room, OLB may authorize the use of additional surrogates. Normally, for purposes of test 99aration and contindity, the same examiner should administer all three operating test categories to an applicant. However, under certain circumstances, the operating test may be divided among different examiners. Such division is ap)ropriate if a facility licensee's simulator is not located near tie plant, because of limitations in examiner resources or scheduling, or if a facility licensee requests examinations for an unusually large group of applicants. Refer to ES-302 for specific instructions regarding administration of the operating tests. Operating tests will normally be administered on regular work days. If weekend or shift work is required to administer the operating tests, the chief examiner will coordinate the arrangements with the assigned examiners and the facility licensee. As a general rule, the operating tests should be scheduled after the written examinations; however, other sequences are permissible if agreed to by the facility licensee. Normally, the written examinations should be administered no more than one week before the operating tests. However, under extenuating circumstances and with prior approval from OLB, the written examinations may be given as soon as the license applications are accepted, any applicable waiver requests are resolved, and the examinations are approved. If necessary, OLB may also authorize the written examinations to be delayed until after the operating tests are complete. NUREG-1021 10 of 24 Interim Rev. 8, January 1997 !

i ES-201 If the examination schedule has to be changed on short notice, the chief examiner will work with his or her supervisor and the designated facility contact to reschedule the examinations to a time when examiners are available and other examinations on the national examination schedule are not affected.

k. If the facility licensee will administer the written examinations, the chief examiner shall review the ES-402 requirements (e.g.,

proctoring and responding to applicant questions) and confirm the applicant's status on the assignment sheet (i.e., examination type and waivers) with the facility contact before the examinations are given. D. PERSONNEL RESTRICTIONS It is impossible to define criteria that anticipate every possible conflict-of-interest issue. Supervisors must apply sound judgment to the facts of each case. If any doubt exists regarding a particular case, the supervisor should consult with regional management and/or OLB to resolve the issue.

1. NRC Examiners

a. The regional office shall not assign an examiner who failed an-applicant on an operating test to administer any part of that applicant's retake operating test.

b. If an examiner was previously employed by a facilit licensee or one of its contractors) and was significantly invol ed in training the current license applicants, the regional office will not assign that examiner any direct responsibilities for developing or administering written examinations or operating tests at that facility. Regional management will control other in-office examination activities concerning the facility such as technical j consultation and quality assurance reviews of examinations. i

c. If an examiner is assigned to an examination that might appear to

                ) resent a conflict of interest, the examiner shall inform his or ier immediate supervisor of the potential conflict. Such notifications should include the following information:

the nature and extent of previous personal and professional relationships with the applicants anything that could affect the administration, performance, evaluation, or results of the examination anything that could create the appearance of a conflict of interest ,

2. Facility Personnel I

a. Facility employees who had any direct involvement in training the NUREG-1021 11 of 24 Interim Rev. 8, January 1997 l l

ES-201 license applicants shall not prepare the outlines for the written examinations or the operating tests. Only one person who provided more than 40 total hours of scheduled l classroom and simulator instruction and no one who provided 15 percent or more of the scheduled classroom instruction or 20 , percent or more of the total scheduled classroom and simulator l instruction may participate in developing the written examination l questions. Furthermore, participants may not develop written ' questions for the topics they taught. Simulator instructors may prepare the implementing documentation for the operating test once the outlines (i.e., simulator event sequences and walk-through task lists) are defined. If the facility licensee considers these restrictions too burdensome, it shall define and discuss with the NRC chief examiner the process it pro)oses to ensure that exam integrity is not compromised. The NRC clief examiner must approve the facility proposal before it is implemented, and the regional office shall inform OLB of any such situations before commencing examination development.

b. The facility licensee shall minimize the number of personnel who have detailed knowledge of the NRC licensing examination.

c. All )ersonnel who will receive detailed knowledge of any portion of tie NRC licensing examination, including the examination I outline, must acknowledge their responsibilities by reading and signing Form ES-201-3, Examination Security Agreement," before they obtain detailed knowledge and again after the examinations are complete. The facility licensee will provide a copy of the form (listing the expected signatories) to the NRC chief examiner at the time the examination arrangements are confirmed. The facility licensee shall also inform the chief examiner if additional personnel need to be added. The original forms must be submitted to the regional office for retention after the examinations are complete.  !

E. ATTACHMENTS / FORMS Attachment 1, " Examination Security and Integrity Considerations" Attachment 2, " Reference Material Guidelines for Initial Operator Licensing Examinations" Attachment 3, " Sample Corporate Notification Letter" Attachment 4, " Sample Examination Assignment Sheet" Attachment 5, " Guidelines for Facility Prereview of NRC-Developed Initial Licensing Examinations" Form ES-201-1, " Examination Preparation Checklist" Form ES-201-2, " Examination Outline Quality Assurance Checklist" Form ES-201-3, " Examination Security Agreement" NUREG-1021 12 of 24 Interim Rev. 8, January 1997

I ES-201 Examination Security and Attachment 1 Integrity Considerations NRC and facility licensee personnel must be attentive to examination security measures to ensure compliance with 10 CFR 55.49. At the time the examination arrangements are confirmed, an NRC examiner shall review with the facility licensee the following guidelines covering physical security and limitations on the use of examination banks as well as the facility licensee's specific plans for ensuring examination security. Additional restrictions covering the assignment of personnel are addressed in Section D of ES-201. Physical Security

1. The NRC expects that the facility licensee will exercise the same physical security precautions with the initial examinations as it does with its requalification examinations.

2. All examination-specific materials (i.e., the examination outlines and final examinations) shall be positively controlled and protected as sensitive information (i.e., under lock-and-key). Drafts, copies, and waste materials must also be controlled and disposed of properly.

The NRC expects that the examinations will NOT be developed and stored on a computer network to which the license applicants could gain access. O 3. The examination outlines, written examinations, and operating tests that are mailed to the NRC regional office shall be placed in a double l envelope. The inner envelope shall be conspicuously marked "FOR OFFICIAL USE ONLY" and "T0 BE OPENED BY ADDRESSEE ONLY." Furthermore, the cover letter forwarding the examination materials shall direct that the materials be withheld from public disclosure until after the examinations are complete. The examination outlines and examinations shall not be transmitted via non-secure electronic means (e.g., the Internet); they may be transmitted via the NRC's " AUTOS" local area network in the resident inspectors' office.

4. The facility licensee shall immediately report to the NRC chief examiner any indications or suggestions that examination security may have been compromised. The NRC will evaluate such situations on a case-by-case basis and determine the appropriate course of action.

An NRC examiner may inspect the facility licensee's administration of the written examinations. The NRC will also review the results of the examination to determine if there is any indication of compromise. O NUREG-1021 13 of 24 Interim Rev. 8, January 1997

ES-201 2 Attachment 1

5. I The facility licensee and the NRC should determine if examination '

security problems were noted in the past and ensure that corrective i actions have been taken to preclude recurrence. 1

6. The facility licensee and the chief examiner will review the simulator I security considerations in Appendix 0 to ensure that the instructor station features, programmers' tools, and external interconnections do {

not compromise examination integrity. The primary objective is to i ensure that the exam material cannot be read or recorded at other unsecured consoles, and that examination materials are either physically secured or electronically protected when not in use by individuals listed on the security agreement.  ! Examination Bank Limitations

1. The facility licensee and chief examiner shall ensure that written l examinations and operating tests conform with the guidelines in Eli-301 '

and ES-401 regarding the use of items taken directly from the bank, modified items, and new items. l

2. If the facility licensee has an open bank, it will not place any new or modified items to be used on the examination (written questions, job performance measures, or simulator scenarios) in its examination bank  ;

until after the last examination has been administered.  ! l \ l NUREG-1021 14 of 24 Interim Rev. 8, January 1997 O l

i e 4 ! I For Ini a pera r n E nations , This attachment discusses the reference materials that facility licensees are  ; expected to provide for each NRC initial licensing examination. The regional l office will customize the list of reference materials as required to support  ! the specific examination assignment; additional materials may be requested at ' i a later time if necessary to ensure the accuracy and validity of the i examinations.  ! s In. determining the need for reference materials, the regional office will  ! consider the facility licensee's level of participation in the examination  ;

development process. If the facility licensee will be preparing the '

examinations, it may be sufficient to obtain only those references necessary i to review and validate the items that appear on the examination, plus a set of I key procedures and other documents required to prepare for the operating tests. The regional office will duly consider the administrative burden it i places on facility licensees and request only those materials that are j actually necessary for the NRC examiners to prepare for the examinations. i i 4 All reference materials provided for the license examinations should be approved, final issues and should be so marked. If any of the material is expected to change before the scheduled examination date, the facility i licensee should reach agreement with the NRC chief examiner regarding changes before the examinations are administered. i l The reference materials may be submitted on computer diskettes (in a format j compatible with the NRC's word processing software), as hard copy, or a

combination as arranged with the NRC chief examiner. If the facility licensee

prepares the examinations, the hard-copy references should normally be limited

! to those materials required to validate the selected test items. All. i procedures and reference materials should be bound with appropriate indices or j tables of contents so that they can be used efficiently; a master table of l contents should be provided for all materials sent. Failure to provide i complete, properly bound, and indexed reference material may prompt the NRC to ! return the material to the person at the highest level of corporate management ! responsible for plant operations. The returned reference materials will be

accompanied by a cover letter explaining the deficiencies in the material and j the basis for postponing or cancelling the examinations.

4 ! Unless otherwise instructed by the NRC regional office, the facility licensee is expected to provide the following reference materials for each NRC initial 3 1 kansing examination: i j 1. Materials used by the facility licensee to ensure operator competency

a. The following types of materbls used to train applicants for i initial R0 and SR0 licensing, as necessary to support examination

{, development: i 4 1

 !         NUREG-1021                                                  15 of 24       Interim Rev. 8, January 1997 i
               --_ _ _                       . _ _ -              . _           .                          ~.

ES-201 2 Attachment 2 learning objectives, student handouts, and lesson plans system descriptions of all operationally relevant flow paths, components, controls, and instrumentation material used to clarify and strengthen understanding of normal, abnormal, and emergency operating procedures complete, operationally useful descriptions of all safety system interactions and, where available, balance-of-plant - system interactions under emergency and abnormal conditions, including consequences of anticipated operator errors, maintenance errors, and equipment failures, as well as plant-specific risk insights based on a probabilistic risk analysis (PRA) and individual plant examination (IPE) These materials should be complete, comprehensive, and of sufficient detail to support the development of accurate and valid examinations without being redundant.

b. Questions and answers specific to the facility training program that may be used in the written examinations or operating tests

c. Copies of facility-generated simulator scenarios that expose the applicants to abnormal and emergency conditions, including degraded pressure control, degraded heat removal capability, and containment challenges, during all modes of operation, including low-power conditions (A description of the scenarios used for the ,

training class may also be provided.)  !

d. All JPMs used to ascertain the comnetence of the operators in performingtaskswithinthecontroTroomcomplexandoutsidethe control room (i.e., local operations) as identified in the facility's job task analysis (JTA) (JPMs should evaluate operator j responsibilities during normal, abnormal, and emergency conditions '

and events and duri shutdown, low power,ng and all fullmodes power.)of operation including cold

2. Complete index of procedures (including all categories sent)

3. All administrative procedures applicable to reactor operation or safety

4. All integrated plant procedures (normal or general operating procedures)

5. All emergency procedures (emergency instructions, abnormal or special procedures) 6.

Standing the regularorders proce (dures)important orders that are safety-related and may modify NUREG-1021 16 of 24 Interim Rev. 8, January 1997

ES-201 3 Attachment 2

7. Surveillance procedures that are run frequently (i.e., weekly) or that can be run on the simulator

8. Fuel handling and core loading procedures (if SR0 applicants will be examined)

9. All annunciator and alarm procedures

10. Radiation protection manual (radiation control manual or procedures)

11. Emergency plan implementing procedures

12. Technical Specifications. (and interpretations, if available) for all units for which licenses are sought

13. System operating procedures

14. Technical data book and plant curve information used by operators as l well as the facility precautions, limitations, and set points document

15. The following information pertaining to the simulation facility:

! a. list of all initial conditions

b. list of all malfunctions with identification numbers and cause and effect information, including a concise description of the expected result or range of results that will occur upon i

initiation and an indication of which annunciators will be actuated as a result of the malfunction o

! c. a description of the simulator's failure capabilities for valves, breakers, indicators, and alarms

d. tr.2 range of severity of each variable malfunction (e.g., the size of a reactor coolant or steam leak, or the rate of a component failure such as a feed pump, turbine generator, or major valve)

d. a list of modeling conditions (e.g., simplifications, assumptions, and limits) and problems that may affect the examination

        -f.      a list of any known performance test discrepancies not yet corrected

g. a list of differences between the simulator and the reference plant's control room

h. simulator instructor's manual

  -16. Any additional plant-specific material that has been requested by the NRC examiners to develop examinations that meet the guidelines of these standards and the regulations NUREG-1021                             17 of 24         Interim Rev. 8, January 1997

ES-201 Sample Corporate Attachment 3 Notification letter NRC Letterhead (date) (Name. Title) (Name of facility) (Address) (City. State. Zio code)

Dear (Name):

In a telephone conversation on (date) between Mr./Ms. (Name. Title) and Mr./Ms. (Name. Title), arrangements were made for the administration of licensing examinations at (facility name) during the week (s) of (date). [As agreed during the telephone conversation, your staff]((The NRC)) will prepare the examinations based on the guidelines in Revision 8, of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors."[ The NRC regional office will discuss with your staff any changes that might be necessary before the examinations are administered.](( In accordance with the guidelines in Attachment 5 of ES-201, your staff will be given the opportunity to review the examinations during the week of (date).)) ' To meet the above schedule, it will be necessary for your staff to furnish the [ examination outlines by Idate). The written examinations, operating tests, and the supporting] reference materials identified in Attachment 2 of ES-201 [will be due] by (date). Any delay in receiving the required [ examination and] reference materials, or the submittal of inadequate or incomplete materials, may cause the examinations to be rescheduled. In order to conduct the requested written examinations and operating tests, it will be necessary for your staff to provide adequate space and accommodations in accordance with ES-402, and to make the simulation facility available on the dates noted above. In accordance with ES-302, your staff should retain the original simulator performance data (e.g., system pressures, temperatures, and levels) generated during the dynamic operating tests until the examination results are final. Appendix E of NUREG-1021 contains a number of NRC policies and guidelines that will be in effect while the written examinations and operating tests are being administered. To permit timely NRC review and evaluation, your staff should submit preliminary reactor operator and senior reactor operator license applications (Office of Management and Budget (0MB) approval number 3150-0090), medical certifications (0MB approval number 3150-0024), and waiver requests (if any) (OMB approval number 3150-0090) at least 30 days before the first examination date. If the applications are not received at least 30 days before the NUREG-1021 13 of 24 Interim Rev. 8, January 1997 l i

ES-201 2 Attachment 3 examination date, a postponement may be necessary. Signed applications certifying that all training has been completed should be submitted at least 14 days before the first examination date. This letter contains information collections that are subject to the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq.). These information collections were approved by the Office of Management and Budget, approval number 3150-0101, which expires on (date). The public reporting burden for this collection of information is estimated to average (number) hours per response, including the time for reviewing instructions, gathering and maintaining the data needed, [ writing the examinations, ]and completing and reviewing the collection of information. Send comments on any aspect of this collection of information, including suggestions for reducing the burden, to the Information and Records Management Branch (T-6 F33), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by Internet electronic mail at BJS19NRC. GOV; and to the Desk Officer, Office of Information and Regulatory Affairs, NE0B-10202, (3150-0101), Office of Management and Budget, Washington, DC 20503. The NRC may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Thank you for your cooperation in this matter. (Name) has been advised of the policies and guidelines referenced in this letter. If you have any questions regarding the NRC's examination procedures and guidelines, please contact (name of recional contact) at (telephone number), or (name of responsible reaional supervisorl at (telephone number). Sincerely, (Acoropriate reaional representative. Title) Docket No.: 50-(Number) Distribution: Public NRC Document Control System Regional Distribution [] Include only for examinations to be prepared by the facility licensee. (( )) Include only for examinations to be prepared by the NRC. O NUREG-1021 19 of 24 Interim Rev. 8, January 1997 o

l ES-201 Sample Examination Attachment 4 Assignment Sheet MEMORANDUM TO: (List NRC examiners by name) FROM: (Reaional Supervisor's Name. Title)

SUBJECT:

EXAMINATION ASSIGNMENTS APPLICANT DOCKET NO. EXAMINATION TYPE 1 1 O Facility and location Facility contact NRC chief examiner Written examinations to be prepared by (RO) (SRO) Dates of Examinations NRC Supervisor cc: Resident inspector Project manager (Standard regional distribution) OLB NUREG-1021 20 of 24 Interim Rev, 8, January 1997

!' I i i i ES-201 Guidelines for Facility Prereview Attachment 5 g of NRC-Developed Initial Licensing Examinations 1 i 1. The prereview of NRC-developed initial. licensing examinations may be  ! t conducted at the site or in the NRC regional office, as discussed in i Section C.3.g of this standard. The licensing examinations shall be  ! j reviewed and approved by the responsible regional supervisor before they i ! are formally prereviewed with the facility licensee. i 4 6

2. The facility reviewers will be given access to the examination materials  !

after they sign a security agreement (Form ES-201-3).

3. If the review takes more than one day, the NRC chief examiner should '

brief the responsible supervisor on the status of the review (i.e., , i- scope of changes and unresolved issues) at the end of each day. l i Regional management is expected to monitor the review process for- i

possible indications of inadequate facility reference material or ,

t insufficient quality assurance by-the NRC. l 4. The facility reviewers should make their recommendations and comments on , ! provided to l the themcopy ofNRC by the the written examination examiner. Simple editorial (s) and answer changes that key (s) do not change i 3 the intent of the question require no justification; however, every 1 j substantive change

or revising an answe(e.g., r) must deleting a question, be supported replacing by approved a distractor, facility reference  !

material. If the original reference material submittal does not support i

the change the reviewers are expected to provide the necessary l documentatlonbeforetheNRCwillchangetheexamination.  ;

4

5. The operating tests (simulator scenarios and job performance measures)  !

may be prereviewed and evaluated in the office or validated at the  ! ! facility if the responsible regional supervisor determines that a site  ;

visit is necessary. If the operating tests are not validated during the  !

e prereview, or if they undergo significant alteration as a result of the l

preliminary validation, they should receive final validation at the i beginning of the examination week (refer to ES-302).

i 6. An NRC examiner will be available throughout the review and validation process to discuss the examination materials and the reviewers' concerns. The facility reviewers should return the marked-u) { examination materials to the NRC chief examiner and ensure t1at he or i she understands their comments and recommendations before they conclude i the prereview meeting. If the facility reviewers have significant j concerns regarding the quality or difficulty of the examination materials, it is their responsibility to convey such concerns to NRC management for resolution. ! -7. The facility reviewers may retain a copy of the examination materials (written, simulator, or walk-through) as modified during the review

process.

'- 8. The facility licensee shall comply with the physical security J requirements discussed in Attachment I to ES-201. i 1 1 4 NUREG-1021 21 of 24 Interim Rev. 8, January 1997 i l l d - , . - - - .- - _ - , . . _ - . . . . . . - . . . , _ - .,..m- , , . _ _

1 l l l ES-201 Examination Preparation Checklist Form ES-201-1 1 Facility: Date of Examination: Examinations Developed by: Facility / NRC (circle one) Target Chief Date' Task Description / Reference Examiner's initials

        -180       1. Examination administration date confirmed (C.1.a; C.2.a & b)
        -120       2. NRC examiners and facility contact assigned (C.1.c; C.2.e)
        -120       3. Facility contact briefed on security & other requirements (C.2.c)
        -120       4. Corporate notification letter sent (C.2.d) 1

(-90) 15. Reference material due (C.1.d; C.3.c))

         -60       6. Integrated examination outline (s) due (C.1.d & e; C.3.d) i
         -55       7. Examination outline (s) reviewed by NRC and feedback provided                      '

to facility licensee (C.2.h; C.3.e)

         -30       8. Preliminary license applications due (C.1.j; C.2.g; ES-202)
         -30       9. Draft exa;ninations, supporting documentation, and reference materials due (C.1.d, e, & f; C.3.d)
         -14       10. Finallicense applications due and assignment sheet prepared (C.1.j; C.2.g: ES-202)
         -14       11. Examination approved by NRC supervisor for facility review (C.2.h; C.3.f)
         -14       12. Examinations reviewed with facility (C.1.h; C.2.f & h; C.3.g)
          -7       13. Written examinations and operating tests approved by NRC supervisor (C.2.i; C.3.h)
          -7       14. Final applications reviewed; assignment sheet updated; waiver letters sent (C.2.g)

15. Proctoring / written exam administration guidelines reviewed with

          -7            facility licensee and authorization granted to give written exams (if applicable) (C.3.k)
          -7       16. Approved scenarios, job performance measures, and questions distributed to NRC examiners (C.3.i)

Target dates are for planning purposes and may be adjusted on a case-by-case basis [] Applies only to examinations prepared by the NRC NUREG-1021 22 of 24 Interim Rev. 8, January 1997

                                                                                                                                   }

k I f ip) ES-201 Examination Outline Quality Assurance Checklist Form ES-201-2

• V -

Focilitv: Date of Examination: Initials item Task Description a b c

1. a. Verify that the outline (s) fit (s) the appropriate model per ES-401.

W R b. Assess whether all six knowledge and four ability categories are appropriately sampled. I T c. Assess whether the outline over-emphasizes any systems, evolutions, or generic topics. T E

d. Assess whether the repetition from previous examination outlines is excessive.

2. e. Using Form ES-301-5, verify that the proposed scenario sets cover the required number of normal evolutions. instrument and component f ailures, and major transients.

S I b. Assess whether there are enough scenario sets (and spares) to test the projected number M and mix of applicants in accordance with the expected crew composition and rotation schedule without compromising exam integrity; ensure each applicant can be tested using at least one new scenario and scenarios will not be repeated over successive days, i

c. To the extent possible. assess whether the outline (s) conform (s) with the qualitative and quantitative criteria specified on Form ES-301-4 and described in Appendix D.

3. a. Verify that the outline (s) contain(s) the required number of control room and in-plant tasks t

and verify that no more than 30% of the test material is repeated from the last NRC , W examination. i N / T b. Verify that the tasks are distributed among the safety function groupings as specified in i ES-301; one task shall require a low-power or shutdown condition, one or two shall require the applicant to implement an alternate path procedure, and one should require entry to the . RCA. j

c. Verify that the recited administrative topics are covered, with emphasis on performance-based activities, i

d. Determine if there are enough different outlines to test the projected number and mix of  :

applicants and ensure that no more than 30% of the items are duplicated on successive days. f

4. a. Assess whether plant-specific priorities (iticluding PRA and IPE insights) are covered in the appropriate exam section.

G j E b. Assess whether the 10 CFR 55.41/43 and 55.45 sampling is appropriate.  ! N I E c. Ensure that K/A importance ratincs (except for plant-specific prionties) are at least 2.5. R A d. Check for duplication and overlap among exam sections. L

e. Check the entire exam for balance of coverage.

f. Assess whether the exam fits the appropriate job level (RO or SRO).

Printed Name / Signature Date

a. Author

b. Facility Reviewer (*)

c. Chief Examiner

d. NRC Supervisor *

(') Not applicable for NRC-developed examinations O t

\

NUREG-1021 23 of 24 Interim Rev. 8, January 1997 i

                                                                                                                            %       I

ES-201 Examination Security Agreement Form ES-201-3 > 1. Pre-Examination I acknowledge that I have acquired specialized knowledge about the NRC licensing examinations scheduled for the week (s) of as of the date of my signature. I agree that I will not knowingly divulge any information about these examinations to any persons who have not been authorized by the NRC chief examiner. I understand that I am not to participate in any instruction, evaluation, or other training-related activities involving those applicants scheduled to be administered these licensing examinations from this date until completion of examination administration. I further understand that violation of the conditions of this agreement may result in cancellation of the examinations and/or an enforcement action against me or the facility licensee. I will immediately report to facility management or the NRC chief examiner any indications or suggestions that examination security may have been compromised.

2. Post-Examination To the best of my knowledge, I did not divulge to any unauthorized persons any information concerning the NRC licensing examinations administered during the week (s) of . From the date that I entered into this security agreement until the completion of examination administration, I did not participate in any training-related activities involving those applicants who were administered these licensing examinations.

PRINTED NAME JOB TITLE / RESPONSIBILITY SIGNATURE (1) DATE SIGNATURE (2) DATE 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.

15. ___

NUREG-1021 24 of 24 Interim Rev. 8, January 1997 O O O

j_ l ES-202 i PREPARING AND REVIEWING OPERATOR LICENSING APPLICATIONS l l A. PURPOSE i 4 This standard provides instructions for facility licensees and applicants to prepare and the NRC to review initial licensing applications. It also discusses the experience, training, education, and certification requirements i and guidelines that an applicant should satisfy before being allowed to take  ;

an NRC reactor operator (RO), senior reactor operator (SRO), or limited senior - '

reactor operator (LSRO) licensing examination. i + i i B. BACKGROUND i ! The Commission-approved licensed operator training programs at most power i reactor facilities are based on a systems approach to training (SAT) and use , simulation facilities that have been either certified by the facility licensee i t or determined to be acceptable by the Commission under 10 CFR 55.45(b). In accordance with 10 CFR 55.31(a)(4), these facilities are not required to i include details of the applicant's qualifications, experience, and training on j the NRC license application form. In lieu of these details, the Commission i will generally accept certification by an authorized representative of the facility licensee that the applicant has successfully completed the facility's

;              Commission-approved training program.

. If the facility licensee does not have a SAT-based licensed operator training

! program that uses a simulation facility acceptable to the Commission, the NRC i will not accept the license application unless it includes the details of the i applicant's qualifications and training. Detailed license eligibility l requirements and guidelines are derived from 10 CFR Part 55, Subpart D, i " Applications," and Regulatory Guide (RG) 1.8, Revision 2, " Qualification and , Training of Personnel for Nuclear Power Plants," respectively. With respect

to license applicants, RG 1.8 endorses, with exceptions, the guidance in American National Standards Institute /American Nuclear Society (ANSI /ANS)-3.1-1981, " Selection, Qualification and Training of Personnel for Nuclear Power Pl ants . " NRC examiners should refer to those documents.as necessary when

evaluating the eligibility of applicants at facilities that do not use an NRC-

! approved or facility-certified simulator as part of a SAT-based licensed j operator training program. j C. RESPONSIBILITIES l- The regulatory requirements associated with the license application process j are detailed in Subpart D, " Applications," of 10 CFR Part 55. The medical i requirements for license applicants and licensed operators appear in Subpart ! C, " Medical Requirements," of 10 CFR Part 55. These requirements should be referred to as necessary when preparing and reviewing license applications.

                                                                                                                            ~

i 3 ii i NUREG-1021 1 of 11 Interim Rev. 8, January 1997

i 4

                                 . - . - - - .                        -          -                       - _- . .-            b

l I

1. Aoolicant/ Facility Licensee

a. To apply for an R0 or SR0 license, an applicant must submit an NRC Form 398, " Personal Qualifications Statement - Licensee," and an NRC Form 396, " Certification of Medical Examination by Facility  ;

Licensee." (Computer-generated duplicates are acceptable.) The  ! application is not complete until both forms are filled out, signed by the appropriate personnel, and received by the NRC. ' Detailed instructions for completing NRC Form 398 are provided I with the form. If the applicant is reapplying after a license denial,10 CFR 55.35 applies, and the applicant must complete and submit new Forms 398 and 396. The applicant may file the second application two months after the date of the first final denial, a third i application six months after the date of the second final denial, and successive applications 2 years after the date of each subsequent denial. Each new Form 398 shall describe the extent of I the applicant's additional training since the denial and shall ' include a certification by the facility licensee that the l applicant is ready for reexamination. l If the applicant previously passed either the written examination  ; or the operating test, he or she may request a waiver of that i portion of the licensing examination. Such waivers are limited to  ! the first reapplication and must be requested within one year of the date of the failed examination. Refer to ES-204 for a more detailed discussion of this and other waiver criteria. The medical data in support of NRC Form 396 are normally good for six months from the date of the medical examination. However, if the applicant is reapplying after withdrawing a previous application or accepting a final license denial, he or she may request a waiver by checking Item 4.f.4 on NRC Form 398 and either providing an explanation in Item 17, " Comments," of the form or  ; submitting a separate letter with the application, j

b. Each applicant (except those applying for an LSR0 license) must satisfactorily complete the NRC's generic fundamentals examination (GFE) section of the written operator licensing examination for i the applicable vendor. Refer to ES-205 for more information on l the GFE program.

Applicants do not need to take the GFE if they were previously l issued an R0 or SR0 license or an instructor certificate based on a site-specific written examination that was administered between February 1982 and November 1989 and included the material covered by the GFE. Enter the date of the examination in Item 4 9 on NRC Form 398 and an explanation in Item 17; a waiver is not required. NUREG-1021 2 of 11 Interim Rev. 8, January 1997 G,

I ES-202 J c. As noted in ES-201, the facility licensee should submit preliminary, uncertified license applications and medical certifications for review by the NRC regional office at least 30 days before the examination date. This will permit the NRC to make preliminary eligibility determinations, process the medical ' certifications, evaluate any waivers that might be appropriate, and obtain additional information, if necessary, while allowing the facility licensee to finish training the applicants before the certified application:; are due.

d. The facility licensee's senior management representative on site must certify when an applicant has completed all of the facility licensee's requirements for the desired license level (i.e.,

experience, control manipulations, training, and medical). Such certification involves placing a check in Item 19.b of NRC Form 398, signing the form, and submitting it to the NRC regional office at least 14 days before the examination date. The facility must also submit a written request that the written examination and operating test be administered to the applicant.

e. When the NRC regional office denies a license application, the applicant may not accept the proposed denial. In such instances, the applicant may request that the Director, Division of Reactor O Controls and Human Factors (DRCH), Office of Nuclear Reactor Regulation (NRR), review the application denial or request a hearing in accordance with 10 CFR 2.103(b)(2). Further action  !

will be taken in accordance with ES-502.

2. NRC Reaional Office

a. The NRC regional office shall review the preliminary applications as soon as possible after they are received. In that way, the regional office can process the medical certifications, evaluate and resolve any waiver requests in accordance with ES-204, and obtain from the facility licensee any additional information that might be necessary in order to support the final eligibility ,

determinations. I With regard to the medical certifications, the regional office shall forward the applicant's NRC Form 396 and the supporting medical evidence to the NRC physician at the Headquarters Health Unit for evaluation any time the examining physician has recommended that the applicant be issued a restricted license or that an existing restriction be changed (by checking block A.4 or A.5 on Form 396).  ! l The NRC will not process a retake application if the applicant's j request for reconsideration or a hearing on the previous license denial is still outstanding (refer to ES-502). O NUREG-1021 3 of 11 Interim Rev. 8, January 1997 l

l ES-202 Before entering the applicants' data in the operator licensing i tracking system (0LTS), the region shall verify that none of the applicants' names appear on the " Restricted Individuals List" found under the Agency-Wide applications of the NRC's local area network. The region shall check with the appropriate contact in the Office of Enforcement by telephone or electronic mail to verify that the information on the subject individual is current i before using the information on the list to deny a licensing 1 action,

b. The regional office will verify that the applicant has i successfully passed the GFE, if required, and review the data on NRC Form 398 to ensure that it is complete.

Affirmative responses to Items 12.a and 12.b of NRC Form 398, indicate that the applicant has successfully completed a Commission-approved, SAT-based training program that uses a simulation facility acceptable to the Commission under 10 CFR 55.45(b). If the facility licensee checks "yes" in response to  : these items, the licensee need not complete Items 13, " Training," , 14, " Experience," and 15, " Experience Details," of NRC Form 398. l The region may accept the application without further review ' unless there is reason to request further information concerning the applicant's qualifications. , Occasionally, a facility licensee completes Items 13,14, and 15 even though they are not required as explained above. In such instances, the region may review the information provided against the eligibility guidelines in Section 0 for the requested license level and resolve any deviations with the facility licensee. New applications must still include the number of significant control manipulations in Item 13.3; at least five are required on ' the facility for which the license is sought. This requirement can only be waived or deferred under the conditions specified in 10 CFR 55.31(a)(5); situations other than those specified would require an exemption in accordance with 10 CFR 55.11. If an applicant checks "no" in response to Items 12.a and 12.b on Form 398, the region sh(1 review the application against the specific R0, SRO, or LSR0 eligibility guidelines described in Section D. If the applicant is reapplying after a previous examination failure and license denial, the region shall evaluate the applicant's additional training to determine if the facility licensee made a reasonable effort to remediate the deficiencies that caused the applicant to fail the previous examination. NUREG-1021 4 of 11 Interim Rev. 8, January 1997

I ES-202

c. The region may determine that the preliminary application is  ;

incomplete, that more information is necessary to make a waiver  ; determination, or that the applicant does not meet the (' requirements in 10 CFR 55.31. In such instances, the region will note the deficiencies and request that the facility licensee i supply additional information when it submits the final, certified I license application (or sooner if possible).  ! Conversely, the region may determine that the preliminary application is complete, and the applicant meets the eligibility requirements or is expected to meet the requirements pending the receipt of additional information. In such instances, the region shall enter the applicant's name, docket number, and examination requirements on the examination' assignment sheet in accordance with ES-201.

d. Upon receiving the final, certified license application, the reviewer shall evaluate any new information to ensure that the eligibility criteria are satisfied. If so, the reviewer shall check the " meets requirements" block at the bottom of Form 398, sign and date the form. If necessary, the reviewer shall add the applicant's name and other data to the examination assignment sheet in accordance with ES-201. The reviewer shall also ensure that the assignment sheet accurately reflects any examination waivers that may have been granted in accordance with ES-204.

If the region determines that the applicant still does not meet the eligibility requirements, the regional licensing authority will discuss its decision with the Operater Licensing Branch (OLB) and notify the applicant in writing that the ao)11 cation is being denied and identify the deficiencies on which ue denial is based (Attachment 1). The responsible regional supervisor, or designee, shall check the "does not meet requirements" block at the bottom of Form 398, and shall sign and date the form. The applicant's name shall be stricken from the examination assignment sheet; the applicant shall not be pennitted to take the licensing examination until the region determines that he or she meets the eligibility criteria. With prior approval from OLB in accordance with ES-204, the region j may administer a license examination to an applicant who has not  ! satisfied the applicable training or experience requirements at i the time of the examination, but is expected to complete them ] shortly thereafter. Assuming that the applicant passes the examination, the region shall not issue the applicant's license until the facility licensee certifies that all of the requirements have been completed.  ! l l O NUREG-1021 5 of 11 Interim Rev. 8, January 1997

D. LICENSE ELIGIBILITY GUIDELINES  ; Through its final safety analysis report (FSAR), technical specifications I (TS), or quality assurance (QA) program, each facility licensee has committed to specific guidelines for nuclear power plant worker qualifications (e.g., ANSI 18.1-1971 or ANSI /ANS 3.1-1981). When reviewing the applicants' eligibility, NRC examiners must ensure that the proper facility licensee commitment is used. The NRC regional office should refer all questions . regarding license eligibility to the Chief, OLB, for resolution. l Revision 2 of Regulatory Guide (RG) 1.8, " Qualification and Training of Personnel for Nuclear Power Plants," describes a method acceptable to the NRC staff for complying with the Commission's regulations with regard to the i training and qualifications of nuclear power plant personnel. For the i positions of shift supervisor, senior operator, and licensed operator, this RG generally endorses the guidelines contained in ANSI /ANS-3.1-1981; specific clarifications, additions, and exceptions are noted in Section C, " Regulatory Position," of RG 1.8. The license eligibility guidelines in RG 1.8, Revision 2, and ANSI /ANS-3.1-1981 are summarized below. These guidelines may not be applicable to all facility licensees and shall not be construed as 1 requirements. I

1. Reactor Operator

a. Experience (1) The applicant should have a minimum of three years of power plant experience, at least one of which is spent at the nuclear power plant for which the license is sought (preferably in the performance of nonlicensed operator duties).

(2) The applicant should spend at least six months performing plant operational duties as a nonlicensed operator at the nuclear power plant for which the license is sought.

b. Training (1) The applicant should complete at least 13 weeks as an extra person on shift in training for the R0 position. This training should include all phases of day-to-day operations and be conducted under the supervision of licensed personnel.

(2) The applicant should be trained in nuclear power plant fundamentals and plant systems, use of those systems to control or mitigate an accident during which the core is severely damaged, and operating practice. NUREG-1021 6 of 11 Interim Rev. 8, January 1997

I h) i ES-202 c (3) The applicant should complete at least 500 hours of lectures 4 on the principles of reactor operation, design features and

<                                    general operating characteristics of the nuclear power plant 4
  .,                                 involved, instrumentation and control (IC) systems, safety                      ,

i and emergency systems, standard and emergency operating procedures, and radiation control and safety procedures. , (4) The applicant should satisfactorily complete an NRC-approved training program involving at least one week at a nuclear

- power plant simulator. The simulator training center should

certify the applicant's ability during a reactor startup to i manipulate the controls, keep the reactor under control,

! predict instrument responses, use instrumentation, follow

procedures, and explain annunciator alarms that occur during l

operation. 4 (5) The applicant must manipulate the controls of the reactor (not simulator) during five significant' changes in i reactivity or power level (refer to 10 CFR 55.31(a)(5)). j Every effort should be made to diversify the reactivity and power changes for each applicant. Startups, shutdowns, j large load changes, and changes in rod programming are some j oxamples; these changes could be accomplished manually using such systems as rod control, chemical shim control, and recirculation flow control. j c. Education The applicant should have a high school diploma or equivalent. l 2. Senior Reactor Operator ?

a. Experience (1) The applicant should have a minimum of four years of responsible power plant experience, as defined in RG 1.8.

At least two of those four years should be nuclear power plant experience. (2) The applicant should have actively performed licensed R0 duties for at least one year at the facility for which the SR0 license is sought. .The NRC may accept any one or more of the following education or experience qualifications to , satisfy this requirement provided that the applicant  ! supplies sufficient details in the license application for l the staff to make a judgement regarding equivalence: A four-year degree in engineering or the equivalent (e.g., a degree in engineering technology or the physical sciences that includes course work in

     \          NUREG-1021                               7 of 11        Interim Rev. 8, January 1997                 l

l l ES-202 physics, mathematics, or engineering; a professional engineer's (PE) license obtained by passing the PE ) examination). l At least one year as an active licensed R0 at a comparable facility (same vendor, similar vintage) or 18 months as an R0 at a noncomparable commercial power reactor. At least two years in a position equivalent to a licensed R0 at a military reactor. Experience obtained in licensed positions (or their equivalent) on other large-scale reactors will be evaluated on a case-by-case basis. Applicants must also submit a waiver request in accordance with ES-204 if they want this experience to apply toward the requirement. (3) At least six months of the applicant's nuclear power plant experience should be at the site for which the license is sought. (4) During the two years of nuclear power plant experience, the applicant should participate in reactor operator activities at power levels greater than 20 percent for at least six weeks.

b. Training (1) The applicant should complete at least 13 weeks as an extra person on shift in training for the SR0 position. This training should include all phases of day-to-day operations and be conducted under the supervision of licensed personnel. Any portion of the 13 weeks that is spent at or above 20 percent power may also be used to satisfy the experience guideline in Section D.2.a(4).

(2) If the applicant has not held an R0 license at the facility and one of the qualifications specified in Section D.2.a(2) is substituted for that experience, the training guidelines of Sections D.I.b(4) and D.I.b(5) should be met. The applicant should satisfactorily complete a training program that is comprehensive in its coverage of both R0 and SR0 knowledge, skills, and abilities and must take an SRO-instant license examination. NUREG-1021 8 of 11 Interim Rev. 8, January 1997 l l

(3) The applicant should be trained in nuclear power plant fundamentals and plant systems, use of those systems to control or mitigate an accident during which the core is severely damaged, and operating practice. (4) The applicant should also complete the additional instruction specified in Section 5.2.1.6 of ANSI /ANS-3.1-1981 in subjects related to the duties of an SRO.

c. Education The applicant should have a high school diploma or equivalent.

3. Limited Senior Reactor Operator

a. Experience The applicant should have three years of nuclear power experience that includes active participation in at least one refueling outage at the site for which the license is sought or at a similar facility. Six months of the nuclear power plant experience must be at the site for which the LSR0 license is sought or at a similar facility owned by the same facility licensee.

b. Training The applicant is expected to have satisfactorily completed a training program that ensures that he or she is qualified to supervise fuel handling operations. The program should be based on a systems approach to training and is expected to include i instruction in at least the following areas:  ;

1 (1) nuclear power plant and health physics fundamentals and the l principles of reactor theory and thermodynamics  ; (2) design features of the nuclear power plant pertaining to fuel handling activities, including plant systems and equipment associated with fuel handling operations, pertinent IC systems, and features of the emergency core cooling systems (ECCSs) associated with the refueling mode of operation (3) the use of installed plant systems to control or mitigate an accident in which the core is dr.maged during refueling operations (4) operating practices and procedures that pertain to refueling, including administrative, operational, surveillance, emergency, radiation control, and safety p procedures; the technical specifications applicable to NUREG-1021 9 of 11 Interim Rev. 8, January 1997

ES-202 refueling; and the requirements concerning communications and interfaces with the main control room The applicant should also complete a minimum of 80 hours of on-the-job training (0JT) in refueling activities, including manipulation of the refueling bridge or similar refueling equipment.

c. Education The applicant should have a high school diploma or equivalent.

4. Cold License Eliaibility Cold examinations are those administered before the unit completes preoperational testing and the initial startup test program as described in the FSAR.

a. Each applicant must satisfactorily complete the training programs described in Section 13.2 of the FSAR and approved by the NRC.

The NRC's review and approval are based on information contained in Section 13.2.1 of the Standard Review Plan (SRP) (NUREG-0800). Note: These NRC-approved training programs typically requirc ten startups on a research reactor. This requirement may be waived if the applicant has completed a plant-referenced simulator , training program accredited by the Institute of Nuclear Power Operations (INP0).

b. In lieu of the control manipulations on the facility for which the license is soug'at (per 10 CFR 55.31(a)(5)), the Commission may accept evidence of satisfactory performance of simulated control manipulations as part of a Commission-approved training program on l a simulation facility acceptable to the Commission under 10 CFR 55.45(b). i E. ATTACHMENTS / FORMS Attachment 1, " Sample Initial Application Denial from Region" l

l ! I NUREG-1021 10 of 11 Interim Rev. 8, January 1997 l

i 1 l ES-202 Sample Initial Application Attachment 1 Denial from Region l l , NRC Letterhead l l (date) i (Apolicant's name) (Street address) (City. State. Zio code)

Dear (Name):

l l This is to inform you that your application for a (reactor operator. senior reactor operator) license submitted in connection with the (facility name) is hereby denied. (Reaion to discuss deficiencies and which cart of 10 CFR 55.31. ES-202. NRC- ! aooroved facility trainina oroaram. or Reaulatory Guide 1.8 was involved.) When you have met the requirements of 10 CFR 55.31, you may submit another application. l If you do not accept this denial, you may, within 20 days of the date of this i letter, take one of the following actions: i (

• You may request that the NRC reconsider the denial of your application i O by writing to the Director, Division of Reactor Controls and Human V Factors, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, DC 20555. Your request must include specific reasons for your belief that your application was improperly denied. If the NRC determines that the denial of your application remains appropriate, you still have the right to request a hearing pursuant to l 10 CFR 2.103(b)(2), as described below.

You may request a hearing in accordance with 10 CFR 2.103(b)(2). Submit your request, in writing, to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, DC 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the

General Counsel, at the same address. 1 If you have any questions, please contact (name) at (teleohone number).

Sincerely, j (Regional branch chief or above) Docket No. 55-(number) cc: (Facility reoresentative who sianed the applicant's NRC Form 398) \ CERTIFIED MAIL - RETURN RECEIPT REQUESTED NUREG-1021 11 of 11 Interim Rev. 8, January 1997

t l n , . ES-204 PROCESSING WAIVERS REQUESTED BY REACTOR OPERATOR AND SENIOR REACTOR OPERATOR APPLICANTS A. PURPOSE l This standard provides guidance concerning the processing of waivers requested by re..ctor operator (RO) and senior reactor operator (SRO) license applicants at power reactor facilities. ! B. BACKGROUND In accordance with 10 CFR 55.35, "Reapplications," and 10 CFR 55.47, " Waiver of Examination and Test Requirements," an applicant may request to be excused from a written examination or an operating test. The NRC may waive any or all of the examination requirements if it determines that the applicant has l presented sufficient justification. In an effort to expedite the resolution of applicant requests, the NRC Operator Licensing Branch (OLB) has delegated the authority to grant routine waivers of certain operator licensing requirements to the NRC regional offices. C. RESPONSIBILITIES

1. Anolicant/ Facility Licensee ,

! \ a. An applicant may request a waiver of a license requirement by I checking the appropriate block in Item 4.f on NRC Form 398,

                   " Personal Qualifications Statement - Licensee." The applicant should also explain the basis for requesting the waiver in l

Item 17, " Comments." l b. The facility licensee's senior management representative on site must certify the final license application, thereby substantiating the basis for the applicant's waiver request. I

c. Facility licensees having units designed by the same nuclear steam supply system vendor and operated at approximately the same power level may request dual licensing for their operators. Similarly, if the units of a multi-unit facility are nearly identical, the facility licensee may request a waiver of the examination requirements for the second and subsequent units.

In either case, the facility licensee must justify to the NRC that i the differences between the units are not so significant that they l could affect the operator's ability to operate each unit safely l and competently. Further, the facility licensee must submit for NRC review the details of the training and certification program. The analysis and summary of the differences on which the applicants must be trained will include the following, as

applicable

NUREG-1021 1 of 5 Interim Rev. 8, January 1997 l l

ES-204 l facility design and systems relevant to control room ! personnel technical specifications l - procedures (primarily abnormal and emergency operating) l control room design and instrument location operational characteristics administrative procedures related to conduct of operations I at a multi-unit site (e.g., shift manning and response to accidents and fires) the expected method of rotating personnel between units and the refamiliarization training to be conducted before an ' operator assumes responsibility on a new unit

2. NRC Reaional Office

a. The regional office will evaluate waiver requests on a case-by-case basis against the waiver criteria discussed in Section D of this ES.

b. The regional office may grant routine waivers identified in Section D.1 without first obtaining OLB concurrence.

However, waivers of experience requirements, completion of train- l ing, or completion of examinations (e.g., the generic fundamentals examination) not specifically identified in Section D.1 must be approved by OLB. The regional office should evaluate the waiver l request and forward its approval recommendation to the Chief, OLB, for concurrence. The region does not require written OLB concurrence to deny an applicant's waiver request, but it should discuss its decision with OLB before informing the applicant; formal concurrence may be desirable in some cases.

c. If additional information is required to reach a decision on the waiver request, the region shall generally request the necessary information from the facility licensee in accordance with ES-202.

d. Upon deciding to grant or deny a waiver, the regional office shall  !

promptly notify the applicant in writing concerning the I disposition of the request. If time is too short to notify the applicant in writing before the examination date, the regional office shall notify the facility training representative by telephone concerning the disposition of the waiver request and provide a follow-up written response to the applicant. The regional office shall include the OLB Branch Chief on distribution for all waiver disposition letters.

e. The region shall document the disposition of every waiver request, whether granted or denied, by completing the block designated "For NUREG-1021 2 of 5 Interim Rev. 8, January 1997

h ES-204 NRC Use" on the applicant's NRC Form 398 and by entering the data in the operator licensing tracking system (0LTS).

f. NRC examiners assigned to a particular examination will be notified of approved waivers by the appropriate regional supervisor and by an entry on the examination assignment sheet (ES-202, Attachment 4).

g. If the applicant is determined to be ineligible to take the licensing examination, the regional office shall issue a denial letter in accordance with ES-202.

D. WAIVER CRITERIA

1. Routine Waivers

a. If an applicant fails only the written examination or one category of the operating test, the region may waive those examination areas (categories) that were passed. This is only applicable for the first retake examination and only if it takes place within one year of the examination that the applicant failed.

b. The region may waive training requirements specified in the final safety analysis report (FSAR) when the FSAR authorizes waiver of those specific requirements and the applicant otherwise meets NRC C requirements (e.g., waiver of some training requirements for applicants previously licensed at a comparable facility).

c. The medical data in support of NRC Form 396 are good for six months from the date of the medical examination for a person applying for an R0 or an SRO instant license. For reapplications following a license denial or withdrawal of an application, waivers extending the six-month period may be granted if the date s of the criginal medical examination is within one year of the scheduled reexamination. For renewal and SR0 upgrade applicants, the medical examination documented on NRC Form 396 is good for two years from the date of the medical examination,

d. Substitutions allowed by Regulatory Guide 1.8, Revision 2, are not considered to be waivers and, therefore, do not require approval.

For example, substitution of related technical training for up to two years of experience for an SR0 or up to one year for an R0 is not a waiver. However, training for the examination applied for may not be counted as related technical training.

e. If the facility licensee certifies that tne applicant has successfully completed a training program accredited by the Institute of Nuclear Power Operations using an acceptable O

NUREG-1021 3 of 5 Interim Rev. 8, January 1997

l ES-204 simulation facility, the region may waive the requirement for ten startups on a research reactor typically required by NRC-approved cold license training programs.

f. For those facilities unable to meet the requirement for six weeks on shift at greater than 20 percent power (because of extended plant shutdowns or other extraordinary circumstances), this requirement may be waived upon application if the following criteria are satisfied:

(1) Facility training objectives for the desired licensed position have been developed using a properly validated job and task analysis (JTA). (2) The facility licensee's training program is based on a systems approach to training (SAT) using the five elements defined in 10 CFR 55.4. (3) The facility licensee can accomplish the training objectives required for plant operation at greater than 20 percent power using a plant-referenced or NRC-approved simulation facility.

g. If an operator was previously licensed at a facility and reapplies for a license at the same facility and license level, the region may, pursuant to 10 CFR 55.47, waive the requirement for the applicant to pass a written examination and an operating test if it finds that the applicant (1) previously discharged his or her responsibilities competently and safely and is capable of continuing to do so (2) terminated participation in the facility licensee's requalification program less than two years before the date of the license application (3) successfully completed " Additional Training," pursuant to 10 CFR 55.59(b), and a facility-prepared written examination and operating test which ensure that the applicant is up to-date in the licensed operator requalification training program (4) successfully completed at least 40 hours of shift functions under the direction of an operator or senior operator, as appropriate, and in the position to which the applicant will be assigned (see 10 CFR 55.53(f))

(5) complies with the requirements of 10 CFR 55.31 NUREG-1021 4 of 5 Interim Rev. 8, January 1997

ES-204

2. Examination Waivers for Previously Licensed Operators at Comparable Facilities Depending on the justification provided by the applicant and the facility licensee, OLB will consider examination waivers for operators who were previously licensed at a comparable facility. Pursuant to 10 CFR 55.47, the Commission may waive any or all requirements for a written examination and operating test.

3. Multi-Unit Examination Waivers

a. Generally, personnel will not be examined on or allowed to hold licenses for "different units" simultaneously. "Different units" owned or managed by a single facility licensee are defined for purposes of this standard as follows:

units having the same vendor but significantly different age and/or power level (e.g., Dresden Units 1 and 2) units having the same vendor and similar design but differ'ent locations (e.g., Sequoyah and Watts Bar, Byron and Braidwood) units having different vendors (PWR only) but located on the (" same site (e.g., Arkansas Units 1 and 2, Millstone Units 2 Q] and 3) OLB -may authorize a limited senior reactor operator (LSRO) to be licensed at multiple sites, provided that the units are manufactured by the same vendor and are of similar design. The applicant must pass an examination that addresses the differences in the designs, procedures, technical data, and administrative controls of the separate facilities for which the license is being sought,

b. With regard to the examination requirements for " identical" second or subsequent units at the same site, OLB may waive any or all requirements for a written examination and operating test if it finds that the applicant meets the criteria specified in 10 CFR 55.47, as noted in Item D.2 above. If the situation warrants, the Commission may impose other examination requirements, such as NRC-administered operating tests and uritten examinations concerning the plant differences.

O NUREG-1021 5 of 5 Interim Rev. 8, January 1997

__ __ ~ _ _ _ _ _ _ _ 4

e -  !

! ES-205 PROCEDURE FOR ADMINISTERING THE GENERIC FUNDAMENTALS EXAMINATION PROGRAM  !

j. A. PURPOSE i This standard describes the procedures and policies pertaining to {

l administration of the generic fundamentals examination (GFE) section of the j written operator licensing examination at power reactor facilities. It '; describes how the examinations are scheduled and constructed, how to solicit , facility licensees for applicants to take the examinations, and how to promulgate the examination results. B. BACKGROUND  ! Sections 55.41 and 55.43 of 10 CFR Part 55 require that the written operator licensing examinations for reactor operators (R0s) and senior reactor operators (SR0s) include questions on various mechanical components, principles of heat transfer, thermodynamics, and fluid mechanics. These regulations also require that the written examination address fundamentals of reactor theory, including the fission process, neutron multiplication, source effects, control rod effects, criticality indications, reactivity coefficients, and poison effects. The fundamental knowledge and abilities (K/As) required of an operator do not vary significantly between license levels or among facilities of the same O vendor type. As a result, the NRC implemented the GFE program to standardize the fundamental examination coverage for all applicants at pressurized and boiling water reactors (PWRs and BWRs). Having passed a GFE as an RO or an SRO applicant, an operator will not have to take another GFE unless he or she transfers to a facility of the other vendor type. The GFE program does not pertain to limited senior reactor operator (LSR0) license applicants. Applicants do not need to take the GFE (nor obtain a waiver) if they were previously issued an R0 or SR0 license or an instructor certificate based on a site-specific written examination (on the same type of facility) that was administered between February 1982 and November 1989 and included the material covered by the GFE. Applicants who were issued a license before 1982 will have to take the examination or apply for a waiver in accordance with ES-204. The GFE examinations for BWRs and PWRs are typically administered twice a year, on the Wednesday following the first Sunday in April and October. C. RESPONSIBILITIES j i

1. Facility Licensee

a. The facility licensee must certify that all individuals who plan to take the GFE are enrolled in a facility-sponsored training program that will satisfy the eligibility requirements for an R0 i or SR0 license. The operator trainees need not complete all of O NUREG-1021 1 of 13 Interim Rev. 8, January 1997 l

l l

i l l ES-205 the training required for the license before they take the GFE. The facility licensee may use the sample registration letter enclosed with the NRC notification letter (Attachment 1) or any similar format that contains the required information and certification. If the facility licensee must add or delete an individual after submitting its registration letter, it should inform the NRC Operator Licensing Branch (0LB) of the change, as l specified in the examination cover letter, before the examinations I are administered.

b. When the examinations are received from the GFE contractor, the l facility licensee shall reproduce and safeguard the examinations i as described in the examination cover letter. l l

c. On the designated examination day, the facility licensee shall administer and proctor the GFE in accordance with the instructions l

contained in the examination package. j The facility licensee will start and stop the GFE in accordance with the time zone map contained within the examination package. i Late arrivals will be allowed to take the examination; however, all examinees must hand in their examinations at the completion time designated in the proctor instructions enclosed with the examination cover letter (refer to Section C.2.d).

d. No later than the day after the GFE is administered, the facility licensee shall send the following items via overnight mail to the name and address designated in the examination package:

the original answer sheets the signed exam cover sheets the signed security statements

2. NRC Operator Licensino Branch and GFE Contractor

a. OLB will designate a coordinator to oversee the GrE activities with the regional offices, the GFE contractor, and the facility licensees.

b. The NRC will send a notification letter (Attachment 1) to each facility licensee 60 days before the GFE administration date. The letter will notify the facility licensee of the date of the examination and request a registration letter listing the licensed operator trainees to whom the facility licensee plans to administer the examination. A sample registration letter is enclosed with the notification letter.

c. The GFE contractor will prepare the examinations as described in Section D of this ES. The examiner assigned responsibility for NUREG-1021 2 of 13 Interim Rev. 8, January 1997

l ES-205 developing the GFE shall submit the examinations to the OLB GFE coordinator and any other designated reviewers at least 20 calendar days before the scheduled administration date. OLB will provide comments and recommended changes to the examination author as soon as possible. The final examinations should be ready at i least 14-days before the GFE administration date.

d. The GFE contractor will assemble the approved examination packages as described below, and mail the packages to the names and addresses designated by'the participating' facility licensees. The examinations should normally be mailed one week before the examinations are scheduled to be administered.

The examination packet will contain the following information, enclosures, and attachments: cover letter (Attachment 2 is a sample letter) proctor instructions security agreement single copies of appropriate exam, forms A and B exam time zone map sample answer sheet facility docket number sheet applicant docket number sheet O - appropriate number of answer sheets applicant answer sheet instructions

e. On the day that the GFE is administered, the OLB GFE coordinator and GFE contractor shall be available to answer questions from facility proctors if.the need arises.

f. When the examination answer sheets are received from the facility licensees, the GFE contractor shall score, grade, and tabulate the overall item statistics, and generate facility and regional grade reports for each GFE examination. The contractor shall forward the regional and facility grade reports, including individual scores and copies of individual answer sheets, and corrected answer keys to the applicable regional office for distribution.

The GFE contractor shall develop individual item statistics on all questions used on the GFE examinations. Questions with acceptable statistical characteristics shall be moved into the " validated" GFE question bank. The contractor will provide copies of all grade reports to the OLB GFE coordinator, along with the following additional items: exam-wide item statistics (PWR and BWR) analysis reports of specific items deleted or answers O changed NUREG-1021 3 of 13 Interim Rev. 8, January 1997

ES-205 corrected answer keys original answer sheets original signed exam cover sheets signed security statements

g. The OLB licensing assistant will ensure that copies of the final master BWR and PWR examinations are placed in the NRC's Public ,

Document Room.

3. NRC Reaional Office

a. Regional management should assign an individual to coordinate GFE administration in the region.

b. The regional operator licensing assistant (0LA) shall assign a docket number to each individual identified in the facility licensee's registration letter. The OLA shall forward the list of names and docket numbers for each facility to the GFE contractor, with a copy to the OLB GFE coordinator, no later than 20 days before the examination administration date.

c. The regional GFE coordinator should keep the OLB GFE coordinator informed of any changes in the number of applicants scheduled to take the GFE at any facility. <

d. The regional office shall distribute the GFE examinations to their ,

respective facility licensees. Sample cover letters for facility J licensees that did and did not participate in the examination are provided in Attachment 3.

e. The regional OLA shall update the applicants' status (pass or  !

fail) in the operator licensing tracking system (0LTS) and ensure that a hard copy of the GFE results is placed in each applicant's docket file. D. EXAMINATION SCOPE AND STRUCTURE Each GFE shall contain 100 questions covering the " Components" and " Theory" (including reactor theory and thermodynamics) sections of NUREG-1122, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors," or NUREG-1123, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors." The passing grade for the GFE is 80 percent. The knowledge and ability (K/A) topics applicable to the GFE for PWRs and BWRs have been categorized into various component, rc:tetor theory, and thermodynamics groups as shown in Attachment 4. The attachment also identifies the number of test questions required to evaluate each topic. The NRC will use Institute of Nuclear Power Operations (INPO) Academy NUREG-1021 4 of 13 Inter b Rev. 8, January 4.

l i FS-205

Documents 89-003, " Generic Fundamentals Test Item Catalog - PWR Operator," and 89-004, " Generic Fundamentals Test Item Catalog - BWR Operator," as the

, primary sources of test questions when developing the GFE. The ratio of

previously used (i.e., " validated") test questions to new or unvalidated test questions will be adjusted as the size of the validated question bank increases.  !

t

The questions used on the GFE examination shall conform with the applicable )

construction and style guidelines in Appendix B.

1 E. ATTACHMENTS / FORMS

Attachment 1, " Sample Notification Letter" Attachment 2, " Sample Examination Cover Letter"

! Attachment 3, " Sample Results Letter"

Attachment 4, "GFE Test Item Distribution" l

i i i i,

        /'~                                                                                                                              '
         '--       NUREG-1021                                                           5 of 13            Interim Rev. 8, January 1997

i i l 1 l ES-205 Sample Notification Letter Attachment 1 NRC Letterhead (Date) (Name. Title) (Facility name) l (Street addressi (City. State Zio code)

Dear (Name):

i The NRC plans to administer the generic fundamentals examination (GFE) section  ; of the written operator licensing examination on (date), 1 To register personnel to take the GFE, an authorized representative of your facility must submit a letter to the appropriate regional administrator with a copy addressed as follows: i Chief, Operator Licensing Branch Mail Stop OWFN 9 D25 U.S. Nuclear Regulatory Commission , Washington, DC. 20555  ! l Your letter should identify the individuals who will take the examination, and it should certify that they are enrolled in a facility licensee-sponsored 1 program leading to NRC operator or senior operator licensing and that they  : will have completed their fundamentals training by the date of the l examination. The letter should also identify the personnel who will have access to the examinations before they are administered (e.g., proctors) and the address to which the examinations are to be sent. To allow the NRC to assign docket numbers, the letter should be received by both the NRC regional l administrator and the Chief, Operator Licensing Branch, 30 days before the l examination date. A sample registration letter is enclosed. Sincerely, (Anoropriate reaional representative) Docket No. 50-fNumber)

Enclosure:

As stated NUREG-1021 6 of 13 Interim Rev. 8, January 1997

(s_,/ ES-205 2 Attachment 1 Enclosure (Name) Regional Administrator U.S. Nuclear Regulatory Commission Region (Number)

                                                                                                    )

(Street address) l (City. State Zio code)

Dear (Name),

(Facility name) requests to have the following (number) individuals take the (BWR or PWR) generic fundamentals examination (GFE) section of the written operator licensing examination to be administered on (date): Hamg Date of Birth Previous Docket No. (Insert the name. date of birth. and orevious 10 CFR Part 55 Docket Number (if aDDlicable) for each Derson.) All of the listed personnel are enrolled in the (facility name) operator licensing training program and will have completed the generic fundamentals portion of the program by the examination date. O. The following personnel will have access to the examinations before they are administered: Hama Title (Insert the name and title of each oerson who will have access to the examinations before they are administered (e.a.. oroctors). Please address the examinations as follows: Name. Title Street address City. State Zio code If you have any questions, please contact (facility contact name) at 1 (telenhone number). l Sincerely, Name. title cc: Chief, Operator Licensing Branch  ; NUREG-1021 7 of 13 Interim Rev. 8, January 1997 l l

i l l l 1 ES-205 Sample Examination Cover Letter Attachment 2 9! l (Date) (Name. Title of desianated addressee) , (Facility name) l (Street addrers) 1 (City. State Zio code) I

Dear (Name):

Your facility is scheduled to administer the generic fundamentals examination j (GFE) section of the NRC's written operator licensing examination on (date). (Name of contractor) is authorized under contract to support the NRC in the administration of GFE activities. , 1 Note: For sacurity reasons, please open the sealed envelope now and page-check the examination using the enclosed checklist. Then contact immediately l and no later than (date) one of the persons listed below informing (him or l b_e1_1 that you have received this package and noting any discrepancies: j (Name), (TeleDhone Number) l (Name), (Telephone Number) l This letter and its enclosures provide the instructions and guidelines for administering the GFE and returning the completed exams and related materials to (Name of contractor). Please read this letter and follow the directions in the accompanying enclosures now. l Enclosure 1. Security Acreement. Please refer to the enclosed NRC Security l Agreement. A copy of this agreement must be completed by each and every exam administrator and/or proctor seeing or having knowledge of the GFE contents. For security reasons, the number of persons seeing or having knowledge of this exam's contents before the exam must be limited to three persons who have a need to know. The top portion of the security agreement is expected to be completed now, and , the bottom portion immediately after the exam has been completed. Fill in the  : spaces for each individual's name and the name of the facility for both portions, and have the individual (s) sign the form (s). Please note: The signed security agreements must be returned to (Name of contractor) along with the completed exam answer sheets before any scoring will be performed. , NUREG-1021 8 of 13 Interim Rev. 8, January 1997

1 l l ES-205 2 Attachment 2 Enclosure 2. Exam Copies. Two single copies of Forms A and B of the exam are provided. These alternative forms are identical in content; however, for security purposes, the test item sequence on each form is different to reduce i , the possibility of an applicant copying any answers from a nearby test answer sheet. (See the separate Proctor Instructions in Enclosure 3 for further exam administration instructions.) . You are responsible for reproducing the number of exam copies required for the ^ number of individuals taking the exam. Prior to the exam, store the original copies in a locked cabinet or safe and reproduce the necessary number of

,         copies only on the day immediately preceding the exam; in this case, copies should be made on (date). Please note: your total number of copies should consist of one half Form A and one half Form B. After making the necessary number of copies, secure the original and all copies from view of unauthorized persons, storing them in a locked cabinet or safe until the exam date.

Each individual taking the exam must sign the security statement on the exam i cover page. This page must be removed from the exam copy and mailed to (Name of contractor) along with the answer. sheets and administrator / proctor security agreements.

   ,m     After the exam has been given, the exam copies become public knowledge and no I   i   longer need security. Exam copies, therefore, may be kept or disposed of as V       desired.

Enclosure 3. Proctor Instructions. The proctor instructions detail the

guidelines for administering the exam. Please note that the specific 4

instructions presented are designed to be adhered to and followed identically by each proctor at all facilities. This process will ensure uniform administration and equity of results nationwide. As noted in the Proctor Instructions, all GFE exams will be administered at the same time in accordance with the local time zone in which the facility is located. Enclosure 4. Exam Answer Sheets. The appropriate number of answer sheets (extra copies included) is enclosed for the number of applicants you identified to take the exam. All applicants must use the original enclosed answer sheets for recording answers during the exam. Susanary of Items to be Returned to (Name of contractor) i The following items are to be mailed via Overnight Delivery Service to (Name of contractor) and postmarked no later than (date). l ) - completed answer sheets applicant-signed exam cover sheets administrator / proctor-signed security statement (s) A I NUREG-1021 9 of 13 Interim Rev. 8, January 1997 - c

l I ES-205 3 Attachment 2 ! Mail all of the above exam-related materials addressed as follows: 1 (Name) LName of contractor 1  ! (Street addressl ) (City. State Zio code) ' For further questions regarding the specifics of this exam, please contact (Name) at (teleobone number). For questions regarding the GFE in general, please contact (Name), NRC, at (telephone number). For matters regarding candic,.te withdrawals or cancellations, contact either (Name) or (Name) at (teleohone number) for specific guidance. (Name), Chief Operator Licensing Branch  ! Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation l

Enclosures:

As stated Distribution: w/o enclosures  ! Director, DRCH Chief, OLB OLB GFE Coordinator Project Manager  ! Public OLB R/F l l l l 1 NUREG-1021 10 of 13 Interim Rev. 8, January 1997

\ ES-205 Sample Results Letter Attachment 3 NRC Letterhead

                                                                                             \

(Date) (Name. Title) l (Facility r ame) l (Street adc ress)  : (C1tv. State Zio code) l

Dear (Name):

(*) (GFE) On (date), the NRC administered the generic fundamentals examinationsection of your facility. Enclosed with this letter are copies of both forms of the examination, including answer keys, the grading results for your facility, and copies of the individual ansrer sheets for each of your employees. Please forward the results to the in11viduals along with the copies of their respective answar sheets. A "P" in the column labeled RESULTS indicates that the individual achieved a passing grade of 80 percent or better on the GFE. Those individuals having an "F" in the RESULTS column failed the examination. (**) On (date), the NRC administered the generic fundamentals examination (GFE) section of the written operator licensing examination. O d (**) Your facility did not participate in this examination. However, a copy of the master (BWR or PWR.1 examination, with the answer key, is enclosed for your information. If you haw any cuestions concerning this examination, please contact (Name of the OLB GFE coorc inator) at (ohone number). Sincerely, j (Acoropriate reoional representative) Docket No. 50-(Number) l (*)

Enclosures:

l

1. Examination Form "A" and "B" with answers

2. Examination Results Summary for Facility Name i

3. Individual Answer Sheets

(**) Encloure: As stated [ Paragraphs marked (*) apply only to those facility licensees that participated 'in the examination, while paragraphs marked (**) apply only to those facility licensees that did not participate in the examination.] NUREG-1021 11 of 13 Interim Rev. 8, January 1997

I ES-205 GFE Test Item Distribution Attachment 4 Pressurized Water Reactors No. of K/A Topic Items Group I Comoonents 191001 Valves 4 l 191002 Sensors and Detectors 10 191003 Controllers and Positioners 5 191004 Pumps 7 191006 Heat Exchangers and Condensers 3 191008 Breakers, Relays, and Disconnects 7 Group II Components 191005 Motors and Generators 5 191007 Demineralizers and Ion Exchangers 3 Group I Reactor Theory 192004 Reactivity Coefficients 4 192005 Control Rods 4 192008 Reactor Operational Physics 8 Group II Reactor Theory 192003 Reactor Kinetics and Neutron Sources 2 192006 Fission Product Poisons 6 Group III Reactor Theory 192001 Neutrons 1 192002 Neutron Life Cycle 2 192007 Fuel Depletion and Burnable Poisons 1 Group I Thermodynamics 193009 Core Thermal Limits 2 193010 Brittle Fracture and Vessel Thermal Stress 5 Group 11 Thermodynamics 193003 Steam 2 193007 Heat Transfer 2 193008 Thermal Hydraulics 8 Group III Thermodynamics 193001 Thermodynamic Units and Properties 1 193004 Thermodynamic Processes 2 193005 Thermodynamic Cycles 1 193006 Fluid Statics and Dynamics 5 j Total Items 100 l NUREG-1021 12 of 13 Interim Rev. 8, January 1997 l

f-'s ( ,) ES-205 2 Attachment 4 Boiling Water Reactors No. of K/A Topic Items Components 291001 Valves 5 291002 Sensors and Detectors 9 291003 Controllers and Positioners 3 291004 Pumps 8 291005 Motors and Generators 5 291006 Heat Exchangers and Condensers 6 291007 Demineralizers and Ion Exchangers 3 291008 Breakers, Relays, and Disconnects 5 Group I Reactor Theory 292004 Reactivity Coefficients 2 292005 Control Rods 4 292008 Reactor Operational Physics 8 Group II Reactor Theory 292001 Neutrons 2 r~'s 292002 Neutron Life Cycle 2 292003 Reactor Kinetics and Neutron Sources (2) x 292006 Fission Product Poisons 3 6 292007 Fuel Depletion and Burnable Poisons 1 Group I Thermodynamics 293007 Heat Transfer and Heat Exchangers 3 293009 Core Thermal Limits 7 Group II Thermodynamics 293003 Steam 2 293004 Thermodynamic Processes 2 293008 Thermal Hydraulics 7 293010 Brittle Fracture and Vessel Thermal Stress 2 Group III Thermodynamics 293001 Thermodynamic Units and Properties 1 293005 Thermodynamic Cycles 1 293006 Fluid Statics 3 Total Items 100 I NUREG-1021 13 of 13 Interim Rev. 8, January 1997

(,/ ES-301 PREPARING INITIAL OPERATING TESTS A. PURPOSE All applicants for reactor operator (RO) and senior reactor operator (SRO) licenses at power reactor facilities are required to take an operating test, unless it has been waived in accordance with 10 CFR 55.47 (refer to ES-204). The specific content of the operating test depends on the type of license for which the applicant has applied. l This standard describes the procedure for developing operating tests that meet  ! the requirements of 10 CFR 55.45, including the use of reactor plant i simulation facilities and the conduct of multi-unit evaluations. B. BACKGROUND To the extent applicable, the operating test will require the applicant to demonstrate an understanding of, and the ability to perform, the actions  ; necessary to accomplish a representative sampling from the 13 items identified l in 10 CFR 55.45(a). In addition, the content of the operating test will be j identified, in part, from learning objectives contained in the facility licensee's training program and from information in the final safety analysis  ! report, system description manuals and operating procedures, the facility n license and license amendments, licensee event reports, and other materials ( I requested from the facility licensee by the Commission. V The structure of the operating test is dictated, in part, by 10 CFR 55.45(b)(1). It states that the test will be administered in a plant walk-through and in either a simulation facility (as defined in 10 CFR 55.4) that the Commission has approved or a simulation facility consisting solely of a plant-referenced simulator (also as defined in 10 CFR 55.4) that has been certified to the Commission by the facility licensee. The walk-through portion of the operating test consists of two categories, each focusing on specific knowledge and abilities (K/As) required for licensed operators to safely discharge their assigned duties and responsibilities. A third category of the operating test is administered on an NRC-approved or facility-certified simulation facility. Unless specifically waived in accordance with ES-204 and documented on the Examination Assignment Sheet (Form ES-201, Attachment 4), all three categories must be completed for every license applicant. Each category of the operating test is briefly described below. Section D of this standard provides detailed instructions for developing each category. Procedures for administering and grading the operating test are contained in ES-302 and ES-303, respectively. O NUREG-1021 1 of 26 Interim Rev. 8, January 1997

I. Cateoorv A. aAdministrative ToDics" This category of the operating test covers K/As that are generally associated with the administrative control of the plant. It implements items 9 through 12 of 10 CFR 55.45(a) and is divided into four administrative topics, as described below. The depth of coverage required in each topic is based on the applicant's license level. The applicant's competence in each topic is evaluated by administering job performance measures (JPMs) or by asking specific questions. Topic A.1, " Conduct of Operations," evaluates the applicant's knowledge of the daily operation of the facility. The following subjects are examples of the types of information that should be evaluated under this topic: shift turnover shift staffing requirements temporary modifications of peccedure: reactor plant startup req'Arements mode changes plant parameter verification (estimated critical position (ECP), 1 heat balance, etc.) 1 short-term information (e.g., night and standing orders) key control security (awareness and familiarityi fuel handling Topic A.2, " Equipment Control," addresses the administrative requirements associated with managing and controlling plant systems and equipment. The following subjects exemplify the types of information that should be evaluated under this topic: surveillance testing , maintenance ' tagging and clearances temporary modification of systems familiarity with and use of piping and instrument drawings Topic A.3, " Radiation Control," evaluates the applicant's knowledge and abilities with respect to radiation hazards and protection (of plant personnel and the public). The following subjects exemplify the types of information that should be evaluated under this topic: use and function of portable radiation and contamination survey instruments and personnel monitoring equipment knowledge of significant radiation hazards the ability to perform procedures to reduce excessive levels of radiation and to guard against personnel exposure radiation exposure limits and contamination control, including permissible levels in excess of those authorized NUREG-1021 2 of 26 Interim Rev. 8, January 1997

i ES-301 radiation work permits control of radiation releases l Topic A.4 " Emergency Plan," evaluates the applicant's knowledge of the 4 emergency plan for the facility, including, as appropriate, the responsibility of the RO or SRO to decide whether the plcn should be executed and the duties assigned under the plan. The following subjects are examples of the types of information that should be evaluated under this topic: lines of authority during an emergency j emergency action levels and classifications emergency facilities emergency ccumunications emergency protective action recommendations Category A is administered in a one-on-one, walk-through format and the four topic areas are graded collectively (refer to ES-302 and ES-303).

2. Cateaory B. " Control Room Systems and Facility Walk-throuah"-

This category of the operating test is used to determine if the applicant's knowledge in the area of plant system design is adequate and j to determine if the applicant is able to safely operate those systems. As such, this category implements the requirements of items 3, 4, 7, 8, O- and 9 identified in 10 CFR 55.45(a). It also encompasses several types of systems, including primary coolant, emergency coolant, decay heat removal, auxiliary, radiation monitoring, and instrumentation and control. Category B is divided into two subcategories. The first and larger subcategory (B.1, " Control Room Systems") focuses on those systems with which licensed operators are most involved (i.e., those having controls and indications in the main control room). The second subcategory (B.2,

                " Facility Walk-Through") ensures that the applicant is familiar with the design and operation of systems located outside the main control room.                    j The applicant's knowledge and abilities relative-to each system are                       i evaluated by administering JPMs and specific follow-up questions.                         I Subcategories B.1 and B.2 are administered in a one-on-one, walk-through format and are graded collectively (refer to ES-302 and ES-303).

3. Cateoorv C. "Intearated Plant Operations" This category of the operating test implements items 1 through 8 and 11 through 13 identified in 10 CFR 55.45(a). This is the most performance-based category of the operating test and is used to evaluate the applicant's ability to safely operate the plant's systems under dynamic, integrated conditions.

NUREG-1021 3 of 26 Interim Rev. 8, January 1997  !

ES-301 The simulator test is administered in a team format with up to three  ! applicants (or surrogates) filling the R0 and SR0 license positions (as I appropriate) on an operating crew. (Refer to ES-201 for additional l guidance on crew composition and ES-302 for test administration instructions.) This format enables the examiner to evaluate each applicant's ability to function within the control room team as appropriate to the assigned position, in such a way that the facility licensee's procedures are adhered to and that the limitations in its license and amendments are not violated (refer to 10 CFR 55.45(a)(13)). Each team or crew of applicants is administered a set of scenarios l designed so that the examiners can individually evaluate each applicant l on a range of competencies applicable to the applicant's license level. Appendix D describes those competencies, and Forms ES-303-3 and ES-303-4, the " Integrated Plant Operations Competency Grading Worksheets" for R0s and SR0s, break down each competency into a number of specific rating factors to be considered during the grading process , (refer to ES-303). ) Each applicant must demonstrate proficiency on every competency applicable to his or her license level. The only exception is that SRO Competency Number 5, " Control Board Operations," is optional for SRO-upgrade applicants. C. RESPONSIBILITIES

1. Facility Licensee The facility licensee is resaonsible for the following activities, as applicable, depending upon tie examination arrangements confirmed with the NRC regional office in accordante with ES-201 approximately four months before the scheduled examination date:

a. Prepare proposed examination outlines in accordance with Section D and submit them to the NRC regional office for review and approval in accordance with ES-201,

b. Submit the reference materials necessary for the NRC regional office to prepare and/or review the requested examination (s)

(refer to ES-201, Attachment 2).

c. Prepare and review the final operating tests in accordance with the previously approved examination outline (s) and the instructions in Sections D and E, and submit the tests to the NRC regional office in accordance with ES-201.

d. Make the simulation facility available, as necessary, for NRC examiners to prepare for the operating tests.

l %UREG-1021 4 of 26 Interim Rev. 8, January 1997  ;

ES-301

e. Meet with the NRC in the regional office or at the facility, when and as necessary, to review the proposed operating tests and discuss potential changes (refer to ES-201).

f. Revise the operating test outlines and the final tests as applicable and as agreed upon by the NRC regional office (refer to ES-201). The NRC retains final authority to approve the operating tests.

2. NRC Reaional Office The NRC regional office is responsible for the following activities:

i a. Ensure that the operating tests are developed in accordance with

Section D.

b. Ensure that the operating tests are reviewed for quality in accordance with Section E.

c. Meet with the facility licensee, when and as appropriate, to prereview the operating tests in accordance with ES-201.

I D. INSTRUCTIONS C Prepare each category of the operating test in accordance with the following general guidelines and specific instructions: l 1. General Guidelines

a. In an effort to reduce examination preparation effort, the same operating test may be used to examine multiple applicants and simulator crews. Depending on the number and license level of the applicants being examined, it might be possible to use the same set of JPMs and scenarios to examine all of the applicants if the operating test is administered in multiple segments (e.g., single scenarios or two-four JPMs) each of which can be given to all of the applicants in a single day. The facility licensee and the NRC chief examiner shall discuss the options and reach agreement on

,                the process before developing the operating tests.

To minimize predictability and maintain test integrity, varied subjects, systems, and operations shall be evaluated with applicants that are not being examined at the same time, unless measures are taken to preclude interaction among the applicants. Overlap or duplication between successive operating tests shall be minimized; simulator scenarios shall not be repeated on successive days, and no more than three (one in the case of upgrade SR0s) JPMs taken directly from the bank may be repeated day-to-day (i.e., new JPMs may not be repeated on successive days).

 . [

NUREG-1021 5 of 26 Interim Rev. 8, January 1997

ES-301 Operating tests written by the facility licensee may not repeat test items (simulator scenarios or JPMs) from the applicants' audit test given at or near the end of the license training class. Simulator events that are similar to events that were tested on the audit examination are permitted provided the actions required to mitigate the transient are significantly different from those required during the audit examination. Sufficient operating test materials shall be developed to ensure that all applicants can be tested with the available personnel according to the schedule agreed upon by the NRC regional office and the facility licensee (refer to ES-201).

b. To the extent permitted for each category of the operating test, select and modify testing materials (i.e., JPMs, questions, and simulator scenarios) from the facility's examination banks. Every selected test item must satisfy the qualitative and quantitative criteria specified for the applicable section of the operating test or be modified accordingly.

c. Consider the K/As associated with normal, abnormal, and emergency tasks and evolutions as a source of topics for use in evaluating applicant competency in each category of the operating test.

The knowledge and abilities associated with the tas's k and questions planned for the operating test should have importance factors of at least 2.5. Tasks with importance factors of less than 2.5 may be used if there is a substantive reason for including them (e.g., a recent licensee event or a significant system modification). The K/As should be appropriate to the plant-specific requirements for the applicant's license level. Refer to the facility's job and task analysis (if available), learning objectives, and other reference material to confirm that the operating test is correctly oriented to the facility and the applicant's license level. The facility licensee's site-specific task list may be used to supplement or override, on a case-by-case basis, selected individual items in the NRC's K/A catalogs. In order to maintain examination consistency, the site-specific task list shall not be used in place of the entire K/A catalog.

d. When selecting and developing materials (JPMs, scenarios, and questions) for the operating test, ensure that the materials contribute to the test's overall capacity to differentiate between those applicants who are competent to safely operate the plant and those who are not. Any test items that, when missed, would raise questions regarding adequate justification for denying the applicant's license should not be included on the operating test.

NUREG-1021 6 of 26 Interim Rev. 8, January 1997 O

i l ES-301

e. SR0 applicants, whether upgrade or instant, will be examined for the highest on-shift position for which the SR0's license is applicable (e.g., shift supervisor), regardless of the position to be assigned when licensed. SRO applicants should demonstrate their supervisory abilities and an attitude of responsibility for safe operation, and are expected to assume a management role during plant transients and upset conditions while taking Category C of the operating test. The operating test briefing, discussed in Appendix E, ensures that the applicants are advised of this policy.

Differences in administrative controls and facility design will affect the SRO's responsibilities, but, in general, the following guidelines should be used to differentiate the SR0 operating test from that of an R0. In directing licensed activities, the SR0 must evaluate plant performance and make operational judgments accordingly. SR0 applicants should, therefore, be more knowledgeable in areas such as operating characteristics, reactor behavior, and instrument interpretation. In airecting licensed activities, the SR0 must have a

broader and more thorough knowledge of facility ,

administrative controls and methods, including limitations ' Q imposed by the regulations and the facility's technical specifications and their bases. The SR0 may be assigned responsibilities for auxiliary systems that are outside the control room (e.g., waste disposal and handling systems) and are not normally operated by licensed operators. Because the SR0 may have these additional responsibilities, the SR0 license applicant should demonstrate knowledge of the designs of such systems as they relate to maximum permissible concentrations, effluent release rates, and other radiological considerations.

f. Incorporate facility-specific and industry-generic operating i experience into the operating test whenever possible. l Documentation such as licensee event reports, significant event  !

reports, and service information letters are readily available i sources of operationally oriented plant anomalics. Evaluate the dominant accident sequences (DASs) for the facility i to determine if they are suitable for testing, on a sampling basis, during the dynamic simulator or walk-through tests. DASs are those sequences that contribute significantly to the frequency i of core damage as determined by the facility licensee's probabilistic risk assessment (PRA) or individual plant examination (IPE). NUREG-1021 7 of 26 Interim Rev. 8, January 1997

i ES-301 The PRA/IPE should also be used to identify risk-important operator actions. Chapter 13, " Operational Perspectives," of , l NUREG-1560, " Individual Plant Examination Program: Perspectives on Reactor Safety and Plant Performance," identifies a number of ' important human actions that may he appropriate for evaluation on the operating test. In determining what actions to evaluate, do not overlook actions that are relied upon or result in specific events being driven to low risk contribution. This will help l identify those human actions, assumed to be very reliable, that might otherwise not show up in a list of risk-dominant actions. l

g. If the applicants at a facility qualify for dual or multi-unit i licenses, the operating tests should evaluate their knowledge of the design, procedural, and operational differences between the units.

Divide the operating test coverage among the units and do not , become predictable by conducting the walk-through tests on only one unit. Different applicants may be examined on different units, or each applicant may be asked to explain or demonstrate his or her understanding of variations in control board layouts, systems, instrumentation, and procedural actions between the units at the facility. Most dual- or multi-unit stations have a simulator that is modeled  ! after only one of the units. Therefore, ensure that the _ 1 applicants are properly tested on the different systems, control board layouts, and any other differences between the units during the walk-through portion of the operating test. For example, after administering Category C of the operating test on Browns Ferry Unit 1, the control room systems portion of Category B of the operating test could be administered on Unit 2 or Unit 3 or both.

h. The operating test should examine a broad range of knowledge and abilities, systems and components, and operations and events. The three categories of the test should not be redundant, nor should they duplicate material that is covered on the written examination. It is particularly important that Categories B and C be developed and reviewed as a package to preclude the same tasks and events from appearing on both parts of t'ne test.

i. Every facet of the operating test, including the questions and answers, JPMs, and simulator scenarios, should be planned, researched, validated, and documented to the maximum extent possible before the test is administered.

j. Examiners who will be administering the operating tests but were not involved in their development are expected to research and study the topics and systems to be examined on the operating test NUREG-1021 8 of 26 Interim Rev. 8, January 1997 O

i

                                                                                                 \

i i ES-301 i so that they are prepared to ask whatever follow-up questions might be necessary to determine if the applicant is competent in  ;

those areas. Examination team members are strongly encouraged to 1 i meet as a group with the chief examiner to review the examination l i materials after they have been approved for administration by the responsible supervisor. The discussions should focus on those test items that might require extensive cueing by the examiner and i those that are unique to the facility and require a response j different from what the examiner might expect based on past

- experience.

i j k. JPMs should include the elements identified in Appendix C (e.g., j initiating and terminating cues, critical steps, and performance i criteria). The guidelines and forms (or equivalents) in that ! appendix should be used when developing new JPMs. Facility i procedures may be adapted for use as JPMs by identifying critical

steps and entering comments on how to. execute particular steps.

i

1. The prescripted questions for Categories A and B may include a i combination of open- and closed-reference items. Open-reference

items that require applicants to apply their knowledge of the

plant to postulated normal, abnormal, and emergency situations are i preferred. ' Closed-reference items may be used to evaluate the i immediate actions of emergency and other procedures, certain j automatic actions, operating characteristics, interlocks, and set i points, as appropriate to the facility. Minimize the use of

closed-reference questions that rely solely on memory, and avoid l the use of open-reference questions whose answer can simply be

looked up if the applicant can find the correct document.

}

Refer to Section B.6 of Appendix C for more guidance repr6g the

! development and use of open reference questions for the walk-l through test. f

m. If it becomes necessary to deviate from a test outline that has been approved by the NRC chief examiner in accordance with ES-201, discuss the proposed deviation with the chief examiner and obtain concurrence before proceeding with the changes. Be prepared to explain why the original proposal could not be implemented and why  !

the proposed replacement is considered an acceptable substitute.

2. Soecific Instructions for Cateaory A. " Administrative Topics" Although the administrative topics may be examined separately, it is preferable, whenever possible, to link, associate, or integrate them with tasks and events conducted during Categories B and C. However, it is important to keep in mind that the applicant's proficiency in the l i

l: NUREG-1021 9 of 26 Interim Rev. 8, January 1997 ! l l

ES-301 administrative topics should be deliberately evaluated and not inferred solely from observations made during the simulator portion of the operating test.

a. For each of the administrative topics listed below, select the required number of subjects to be evaluated during the operating test. Section B.1 provides examples of the types of subjects that should be evaluated under each topic; the lists are not all-inclusive.

Topic Number of Sub.iects A.1, " Conduct or' 0perations" 2 A.2, " Equipment Control" 1 A.3, " Radiation Control" 1 A.4, " Emergency Plan" I

b. For each administrative subject, determine the best method for evaluating the applicant's knowledge or ability in that area.

Although a performance-based evaluation, using a single administrative JPM is generally preferred, two prescripted questions may be used to conduct the evaluation in each specific subject area selected for evaluation. The questions may be associated with Category B JPMs (as additional questions) or they may be administered separately.

c. In general, SR0s have more administrative responsibilities than R0s, so SR0 applicants should be evaluated in greater depth on the administrative topics. R0 applicants need only understand the mechanics and intent of the related subjects, as they pertain to tasks at the facility.

d. The following specific guidelines should be applied when selecting or developing questions or JPMs to confirm the applicant's competence with regard to each topic:

Topic A.I. " Conduct of Operations", Many of these subjects can be covered within the framework of a shift turnover or by integrating them into other discussions, as they apply, throughout the examination. The subject of fuel handling can be covered in the control room, but attempt to cover this subject in the fuel handling areas of the plant whenever possible. The R0 applicant should be aware of his or her duties in the control room during fuel handling. These duties include monitoring instrumentation and responding to alarms from the fuel handling area, communicating with the fuel handling and storage facility, and operating systems from the control room in support of (re) fueling operations. For the SR0 applicant, NUREG-1021 10 of 26 Interim Rev. 8, January 1997

ES-301 J evaluate topics such as core alterations, new and spent fuel storage and movement, the design of the fuel handling area, use of the fuel handling tools, and fuel handling casualties. The applicant's security awareness should be evaluated by observing his or her behavior during the operating test. However, passive observations, in and of themselves, are insufficient to justify an evaluation in that subject area. It is appropriate to question an SR0 applicant on applicable aspects of the facility's security plan and the operating crew's interactions with the security shift supervisor. Topic A.2. "Eauipment Control" These subjects can be evaluated within the framework of a normal maintenance evolution. For example, ask the applicant to demonstrate how he or she would take a failed system or component out of service, initiate maintenance on the system, and test the system before placing it back in service. Tooic A.3. " Radiation Control" This topic is best covered in conjunction with the JPMs and g questions prepared for Category B.2 of the walk-through (i.e., local systems and operations). It is most appropriate to evaluate these subjects during an entry into the radiologically controlled area (RCA). The levels of knowledge expected of R0 and SRO applicants in some radiation control subjects are significantly different. The R0's duties generally require knowledge of radiation worker responsibilities and operation of plant systems associated with liquid and gaseous waste releases. Therefore, the depth to which R0 applicants are evaluated should be limited to their responsibilities and the monitoring requirements before, during, and after the release. The SRO, however, may be involved in

 ,            reviewing and approving release permits and should be cognizant of the requirements associated with those releases, as well as their potential effect on the health and safety of the public. The SR0 applicants may be asked to discuss or simulate (i.e., with a JPM) a planned release (e.g., liquid, gaseous, or containment purge) when examining these topics.

Topic A.4. "Emeroency Plan" There are significant differences between the knowledge required of R0 and SR0 applicants in this area. R0 applicants should be familiar with the emergency plan and with their plant-specific responsibilities under the emergency plan implementing procedures (EPIPs). 3R0 applicants, however, must demonstrate additional O NUREG-1021 11 of 26 Interim Rev. 8, January 1997

I l ES-301 l knowledge based upon their responsibility to direct and manage the l l implementation of the EPIPs during the initial phases of an  ! l emergency. Because of this, SRO applicants should have a more l detailed understanding of the EPIPs, in general, and be familiar  ! with event classification procedures, protective action recommendations, and communication requirements and methods. This topic is best evaluated by integrating it into a discussion of a Category C transient that requires implementation of the 1 emergency plan, or by conducting a JPM requiring use of the , emergency plan. Such a JPH can be conducted immediately following i a simulator scenario or during the walk-through (Category A or B)  ! examination. l

e. The planned administrative subjects should normally take no mor than I hour and 1.5 hours to administer to R0 and SR0 applicants,  !

respectively. '

f. On Form ES-301-1, " Administrative Topics Outline," briefly i

describe the administrative subjects selected for evaluation and  ; the method (s) by which each subject will be evaluated. The method ' of evaluation should include the title of any planned JPMs and a brief summary of the proposed questions.

g. Forward the completed outline to the NRC chief examiner so that it is received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed; the outline is normally due at least 60 days before the scheduled l examination date. Refer to ES-201 for additional instructions regarding the review and submittal of the examination outline.

The NRC chief examiner and responsible supervisor shall review the test outline coverage as soon as possible in accordance with ES-201 and forward any comments to the originator for resolution.

h. After the NRC chief examiner approves the operating test outline, prepare the final Category A test materials (i.e., the JPMs, questions, and answers) in accordance with the general operating test guidelines in Section D.1, the open-reference question guidelines in Appendix B, and the JPN guidelines in Appendix C.

i. When the materials are complete, review the quality of the final Category A walk-through test using Form ES-301-3, " Operating Test Quality Assurance Checklist." This review shall be performed in conjunction with the associated Category B walk-through and the dynamic simulator operating test as noted in Sections D.3 and D.4.

Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner, as appropriate, for review and approval in accordance with Section E. The test must NUREG-1021 12 of 26 Interim Rev. 8, January 1997

ES-301

 %./

be received by the NRC chief examiner at least 30 days before the scheduled administration date, unless other arrangements have been made.

3. Specific Instructions for Cateaory B. " Control Room Systems and Facility Wal k-Throuah" This category of the operating test evaluates the applicant's systems-related K/As by having the applicant perform selected tasks and probing his or her knowledge of the task and its associated system with l

specific, prescripted, follow-up questions. The Category B tasks and - questions are in addition to and should be different from the events and evolutions conducted during Category C, "Ir'.egrated Plant Operations."

a. Refer to Section 1.9 of the K/A c e Mg applicable to the type of reactor for which the applicant ,eeking a license (i.e., NUREG-1122 for W Rs and NUREG-1123 for BWRs). From the nine safety l function groupings identified in the catalog, select the )

appropriate nunber of systems (see the table below) to be evaluated for each subcategory of the test based on the applicant's license level. The emergency and abnormal plant evolutions (E/ APES) listed in Section 1.10 of the appropriate NUREG may also be used to evaluate the applicable safety function m (as specified for each E/ APE in the first tier of the written examination outlines attached to ES-401). License level Subcateaory B.1 Subcateaory B.2 Total R0 7 3 10 l SRO-instant (I) 7 3 10 l SRO-upgrade (U) 2 or 3 3 or 2 5 j l The 10 systems and evolutions selected for R0 and SRO-I applicants should evaluate at least 7 different safety functions. All of the systems and evolutions in each subcategory of the test should be selected from different safety function lists, and the same system or evolution should not be used to evaluate more than one safety function in each subcategory. The 5 systems and evolutions selected for an SRO-O upplicant should evaluate at least 5 different safety functions. One of the control room systems or evolutions (Subcategory B.1) must be an engineered safety feature, and the same system or evolution should not be used to evaluate more than one safety function. Keep in mind that the systems and evolutions selected for evaluation in Subcategories B.1 and B.2 must be oriented toward control room operations and local operations, respectively. O D) ( NUREG-1021 13 of 26 Interim Rev. 8, January 1997

ES-301

b. For each system selected for evaluation, select from the applicable K/A catalog or the facility licensee's site-specific task list one task for which a JPN exists or can be developed.

Review the associated simulator outline if it has already been prepared (refer to Section D.4), and avoid those tasks that have already been selected for evaluation on the dynamic simult. tor test. In order to protect the integrity and security of the examination process, no more than 80 percent of any applicant's walk-through test (i.e, 8 out of 10 or 4 out of 5 JPMs, as applicable) may be taken directly from the facility's testing materials without significant modification. A significant modification means that at least one condition has been substantively changed in a manner that alters the course of action of the JPM. Additionally, no more than 30 percent of the walk-through test may be repeated from the last NRC licensing examination at the facility. At least one of the tasks shall be related to a shutdown or low-  ; power condition, and one or two of the tasks shall require the l applicant to execute alternate paths within the facility's ' operating procedures. In addition, at least one of the tasks conducted in the plant (i.e., Subcategory B.2) shall evaluate the applicant's ability to implement actions required during an emergency or abnormal condition, and another should require the applicant to enter the RCA. This provides an excellent opportunity for the applicant to discuss or demonstrate the radiation control subjects described in Administrative Topic A.3. If it is not possible to develop or locate a suitable task /JPM for each of the selected systems, return to Step (a), above, and select a different system or evolution. After identifying a JPM for each system, list it and its associated safety function number on Form ES-301-2, " Individual Walk-Through Test Outline." Also indicate the type of JPM by entering the applicable code (s) identified at the bottom of the form.

c. For each system and evolution selected for evaluation, refer to the applicable section in the K/A catalog and select two system-specific or generic K/As to be evaluated with prescripted l questions. The prescripted questions should be diversified among i the different K/A categories associated with each system or evolution. List the selected K/A numbers, their importance factors, and a brief description of the topic on Form ES-301-2.

d. Forward the completed walk-through test outline to the NRC chief examiner so that it is received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed; the outlines are normally due at least 60 days before the scheduled examination date. Refer to ES-201 for additional NUREG-1021 14 of 26 Interim Rev. 8, January 1997

O ES-301 instructions regarding the review and submittal of examination outlines. The NRC chief examiner and responsible supervisor shall review the , test outline in accordance with ES-201 and forward any comments to the originator for resolution.

e. After the NRC chief examiner approves the operating test outline, prepare the final Category 8 test materials (i.e., the JPMs, questions, and answers) in accordance with the general guidance in Section D.1 and the JPM guidelines in Appendix C.

f. When the materials are complete, review the completed walk-through test for quality using Form ES-301-3, " Operating Test Quality Assurance Checklist," and make any changes that might be necessary. To minimize duplication, this review shall be performed in conjunction with the associated administrative topics and the simulator operating test (refer to Sections D.2 and D.4).

Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner, as appropriate, for review and approval in accordance with Section E. The test must be received by the NRC chief examiner at least 30 days before the q scheduled administration date, unless other arrangements have been made.

4. Specific Instructions for Cateoorv C. "Inteorated Plant Operations"

a. Based on the anticipated crew compositions, determine the number of scenarios and scenario sets necessary to rotate each R0 and SRO-I applicant into the lead reactor operator position so that he or she can perform a direct reactivity manipulation. For example, a crew consisting of two R0s and one SRO-I will normally require three scenarios to evaluate each applicant's performance on the i reactor controls; however, a surrogate SRO will have to fill the '

supervisory role while the SRO-I applicant is in the lead operator position. Similarly, the crews and scenarios will have to be planned so that every SR0 applicant (U and I) fills the supervisory role for at least one scenario. SRO-U applicants are given credit for their previous R0 license evaluation and experience and are normally not required to manipulate the controls. It may be possible to significantly reduce the number of simulator scenario sets required to examine a large group of applicants by administering the same set of scenarios on the same day to two (or more) different crews of applicants. However, provisions must be made to ensure that the crews remain out of contact until all crews have completed the set of scenarios (refer to ES-302). ' ' NUREG-1021 15 of 26 Interim Rev. 8, January 1997

ES-301 Additional or replacement scenarios should also be prepared and available while administering the operating tests in accordance with ES-302 in case one of the planned scenarios does not work as intended.

b. The simulator operating tests (i.e., scenario sets) will be constructed by selecting and modifying scenarios from existing facility licensee or NRC scenario banks and by developing new scenarios.

In order to maintain test integrity, every applicant shall be tested on at least one new or significantly modified scenario that he or she has not had the opportunity to rehearse or practice. A significant modification means that at least one condition or event has been substantively changed to alter the course of action in the scenario. Furthermore, any other scenarios that are extracted from the facility licensee's bank must be altered to the degree necessary to prevent the applicants from immediately recognizing the scenarios based on the initial conditions or other Cues.

c. The initial conditions, normal operations, malfunctions, and major transients should be varied among the scenarios and should include startup, low-power, and full-power situations. Review the associated walk-through outline if it has already been prepared (refer to Section D.3), and take care not to duplicate operatic.s that will be tested during the walk-through portion of the operating test.

d. In order to maximize the quality and consistency of the operating l tests, develop new scenarios in accordance with the instructions j in Appendix D. Modify existing scenarios, as necessary, to make  ;

them conform with the qualitative and quantitative attributes I described in that appendix and enumerated on Form ES-301-4,

          " Simulator Scenario Quality Assurance Checklist." The quantitative attribute target ranges that are specified on the form are not absolute limitations; some scenarios may be an excellent evaluation tool but may not fit within the ranges. A scenario that does not fit into these ranges shall be evaluated to ensure that the level of difficulty is appropriate. Whenever possible, the critical tasks should be distributed so that each applicant is required to respond.

Each scenario set must, at a minimum, exercise each applicant on the types of evolutions, failures, and transients in the quantities identified for the applicant's license level on Form ES-301-5, " Transient and Event Checklist." An applicant should only be given credit for those events that require the applicant to perform verifiable actions that provide insight to the applicant's competence. Each event should only be counted once NUREG-1021 16 of 26 Interim Rev. 8, January 1997 O

l i i i ES-301 ! per applicant; for example, a power change can be counted as a i normal evolution OR as a reactivity manipulation, and, similarly, i a component failure that immediately results in a major transient - l counts as one or the other, but not both. , Furthermore, each scenario set must also enable the examiner to evaluate the applicant's performance on each competency and rating factor germane to the applicant's license level. Use Form ES-301- !- '6, " Competencies Checklist," to verify that the competencies are adequately evaluated by entering the scenario and event numbers ! that are intended to assess each competency. Appendix D provides detailed instructions for completing Form ES-D-1, the " Scenario Outline," and Form ES-D-2, the expected !- " Operator Actions," that examiners will use to administer the i simulator operating tests. In order to minimize the amount of

rework that might be required as a result of changes in the

!- planned scenario events, Form ES-D-2 should be completed after the ! NRC chief examiner has had the opportunity to review and comment i on the proposed simulator operating test outlines (i.e., Form ES-D-1) in accordance with CS-201.

e. When the proposed simulator 3perating test outlines are complete, 4

forward them to the NRC chief examiner so they are received by the ! date agreed upon with the NRC regional office at the time the i examination arrangements were confirmed; the outlines are normally ! due at least 60 days before the scheduled examination date. Refer i i to ES-201 for additional instructions regarding the review and }: submittal of the examination outlines. The NRC chief examiner shall review the operating test outlines in I

accordance with ES-201, and forward any comments to the originator l

! for resolution. '

f. After the NRC chief examiner approves the operating test outlines, .

l prepare the final simulator test materials by revising Form (s) I

,                                                         ES-D-1 as requested by the NRC chief examiner and completing a                                      i detailed operator action form (ES-D-2) for each event. All                                         l substantive operator actions (e.g., opening, closing, and                                          i I_                                                         throttling valves; starting and stopping equipment; raising and                                     !

! lowering level, flow, and pressure; making decisions and giving

directions; not acknowledging alarms or verifying automatic )

actions) shall be documented, and critical tasks shall be i

identified. Events that do not require an operator to-take one or ,

more substantive actions will not count toward the minimum number ,

j of events required for each operator per Form ES-301-5.  ! 1 [ g. Review the completed simulator operating test for quality using i Form ES-301-4, " Simulator Scenario Quality Assurance Checklist," j and make any changes that might be necessary. This review shall NUREG-1021 17 of 26 Interim Rev. 8, January 1997 f 4

          .-  ,--4,.

7w .

                                                                   -._ # ,._7_ .,.,m.,,,y n.-  v.    . . . , = . , .. _. .       ._. ,        ,. ry -mw+

1 i ES-301 be performed in conjunction with the associated walk-through test l (refer to Sections D.2 and D.3) to minimize duplication. Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner, as appropriate, for review and approval in accordance with Section E. The test must be received by the NRC chief examiner at least 30 days before the scheduled administration date, unless other arrangements have been made. E. QUALITY ASSURANCE REVIEWS

1. Facility Manaaement Review If the operating test was prepared by the facility licensee, it shall be independently reviewed and approved by an authorized facility l representative (refer to ES-201) before it is submitted to the NRC l regional office for review and approval. The reviewer should evaluate l the examination using the criteria on Forms ES-301-3 and ES-301-4 and '

include the signed forms (for each different operating test) in the examination package submitted to the NRC in accordanca with ES-201.

2. NRC Examiner Review

a. The NRC chief examiner shall independently review each operating test for content, wording, operational validity, and level of difficulty. As a minimum, the chief examiner shall check the items listed on Forms ES-301-3 and ES-301-4, as applicable. The examiner should keep in mind that counting the number of scenario quantitative attributes is not always indicative of the scenario's level of difficulty. Although there are no definitive minimum or maximum attribute values that can be used to identify scenarios that will not discriminate because they are too easy or difficult, scenarios that fall outside the target ranges specified on Form ES-301-4 should be carefully evaluated to ensure they are appropriate. If the chief examiner wrote the operating test, another NRC examiner shall perform the independent review.

b. The chief examiner should review the operating tests as soon as possible after receipt so that supervisory approval can be obtained before the final review with the facility licensee, which is normally scheduled about two weeks before the administration date. It is especially important that the chief examiner promptly review tests prepared by a facility licensee because of the extra time that may be required if extensive changes are necessary. The l chief examiner shall consolidate the comments from other regional )

reviewers and submit one set of comments to the author.

c. If the facility licensee developed the operating test, it assumed  !

primary responsible for technical accuracy and compliance with the  ; NUREG-1021 18 of 26 Interim Rev. 8, January 1997 I i

   - ___     _ _ ~ -     . . . . . . _    --                .- - -                ._   _ _ - _ - .        -

ES-301 restrictions concerning the use of examination banks. However, the chief examiner is expected to use his or her best judgment and take reasonable measures, including selective review of reference materials and past tests, to verify these items.

d. The chief examiner will note any changes that need to be inade and forward the tests to the responsible supervisor (or a designated alternate other than the chief examiner) for review and comment in accordance with Section E.3 before reviewing the examinations with the author or facility contact. There are no minimum or maximum limits on the number or scope of changes the chief examiner may direct the author or facility contact to make to the proposed tests, provided that they are necessary to make the tests conform with established acceptance criteria. Refer to ES-201 for additional guidance regarding NRC response to facility-developed

examinations that are significantly deficient, t

e. Upon supervisory approval, and generally at least 14 days before the operating tests are scheduled to be given, the chief examiner will review the tests with the facility licensee in accordance 4 with ES-201.

3 Tests that were developed by the NRC shall be clean, properly formatted, and " ready-to-give" before they are reviewed with the ] O facility licensee. The region shall not rely on the facility licensee to ensure that the tests are of acceptable quality to administer. i

f. After reviewing the tests with the facility licensee, the chief examiner will ensure that any comments and recommendations are resolved and the tests are revised as necessary. If the facility licensee developed the tests, it will generally be expected to make whatever changes are recommended by the NRC.

g. After the necessary changes have been made and the chief examiner i is satisfied with the examination, he or she will sign Form (s) l ES-301-3 and forward the test package to the responsible '

supervisor for final approval. j

3. NRC Supervisorv Review l

a. Per ES-201, the responsible supervisor (or a designated alternate '

other than the NRC author or chief examiner) shall review and approve the operating tests before the facility prereview. The supervisory review is not intended to be another detailed review, but rather a general assessment of test quality, including a review of the changes recommended by the chief examiner, and a check to ensure that all of the applicable administrative requirements have been implemented. O NUREG-1021 19 of 26 Interim Rev. 8, January 1997

l ES-301

b. The responsible supervisor should ensure that any significant i deficiencies in the original operating tests submitted by a '

facility licensee are evaluated in accordance with ES-201 to < determine the appropriate course of action. At a minimum, the ' supervisor should ensure that they are addressed in the final examination report in accordance with ES-501.

c. Following the facility review, the responsible supervisor should again review the tests to ensure that the concerns expressed by '

the facility licensee and the chief examiner have been  ; appropriately addressed. The supervisor shall not sign Form (s)  !' ES-301-3 until he or she is ratisfied that the examination is acceptable to be administereo. F. ATTACHMENTS / FORMS Form ES-301-1, " Administrative Topics Outline" l Form ES-301-2, " Individual Walk-Through Test Outline"  ! Form ES-301-3, " Operating Test Quality Assurance Checklist" l Form ES-301-4, " Simulator Scenario Quality Assurance Checklist" Form ES-301-5, " Transient and Event Checklist" Form ES-301-6, " Competencies Checklist" O NUREG-1021 20 of 26 Interim Rev. 8, January 1997

ES-301 Administrative Topics Outline Form ES-301-1 Facility: Date of Examination: Examination Level (circle one): R0 / SR0 Operating Test Number: s Administrative Describe method of evaluation: Topic / Subject 1. ONE Administrative JPM, OR Description 2. TWO Administrative Questions 4 A.1 i A.2 i i .l A.3 1 i a A.4 E d 1 i

   \'

NUREG-1021 21 of 26 Interim Rev. 8, January 1997

l ES-301 Individual Walk-Through Test Outline Form ES-301-2 Facility: Date of Examination: 1 Exam Level (circle one): R0 / SR0(I) / SR0(U) Operating Test No.: System / JPM Title / Type Safety Planned Follow-up Questions: Codes

• Function K/A/G - Importance - Description

1. a.

b.

                                                          ~

2. a.

b.

3. a.

b.

4. a.

b.

5. a, b.

6. a.

b.

7. a.

b.

8. a.

b.

9. a.

b.

10. a.

b.

• Type Codes: (D)irect from bank, (M)odified from bank, (N)ew, (A)lternate path, (C)ontrol room, (S)imulator, (L)ow-Power, (P)lant, (R)CA NUREG-1021 22 of 26 Interim Rev. 8, January 1997

1 i  ! 1 N j , , ES-301 ODeratino Test Quality Assurance Checklist Form ES-301-3 I l j Facility: Date of Examination: Operating Test Number: 4 i initials

1. GENERAL CRITERLA l

a b c j a. The operating test conforms with the previously approved outline; changes are consistent , with sampling requirements (e.g.,10 CFR 55.45, operational irnportance, safety function i distribution). . b. Repetition from operating tests used during previous licensing examinations is within ! acceptable limits (30% for the walk-through) and should not compromise test integrity.

c. Day-to day repetition between this and other operating tests to be administered !? within

, acceptable limits (30% of bank JPMs for the walk-through; none for the simulator).

d. Overlap with the written examination and between operating test categories is within ecceptable limits.

e, it appears that the operating test will differentiate between competent and less-than- , competent applicants at the designated license level, j

2. WALK-THROUGH l CATEGORY A & B) CRffERIA - - -

e. Each JPM includes the following, es applicable:

e initial conditions e initiating cues e references and tools, including aseconsted procedures a validated time limits (average time allowed for completion) and specific designation if deemed to be time critical by the facility licensee e specific performance criteria that include:

                                         - detailed expected actions with exact criteria and nomenclature
                                         - system response and other examiner cues statements describing important observations to be made by the applicant criteria for successful completion of the task identification of critical steps and their associated performance standards restrictions on the sequence of steps if applicable

b. Prescripted (Administrative and JPM follow-up) questions are predominently open reference and meet the criteria in Appendix C.

c. There are no direct look-up questions; memory level questions do not permit the use of references.

d. At least 20 percent of the JPMs and questions on each test are new or significantly modified.

3. foiMULATOR (CATEGORY Cl CRITERIA - - -

e. The associated simulator operating tests (scenario sets) have been reviewed in accordance with Form ES-301-4 and a copy is attached.

Printed Name / Signature Date

a. Author

b. Facility Reviewer (*)

c. NRC Chief Examiner (')

d. NRC Supervisor (*)

(*) The facility signature is not applicable for NRC-developed tests; two independent NRC reviews are required. NUREG-1021 23 of 26 Interim Rev. 8, January 1997 l l

l I i ES-301 Simulator Scenario Quality Assurance Checklist Form ES-301-4 Facility: Date of Exam: Scenario Numbers: / / Operating Test No.: I l QUALITATIVE ATTRIBUTES initials a b c

1. Tha. scenarios have clearly stated objectives in the scenario summaries.

2. The initial conditions are realistic, in that some equipment and/or instrumentation may be l out of service, but it does not cue the operators into expected events.

3. The scenarios consist mostly of related events. ,

1

4. Each event description consists of j

• the point in the scenario when it is to be initiated a the malfunction (s) that are entered to initiate the event l
• the symptoms / cues that will be visible to the crew
• the expected operator actions (by shift position) l
• t5e event termination point (if applicable) l l

5. No more than one non-mechanistic failure (e.g.. pipe break)is incorporated into the '

scenario without a credible preceding incident such as a seismic event. l l

6. The events are valid with regard to physics and thermodynamics. l l

7. Sequencing and timing of events is reasonaWe, and allows the examination team to obtain j complete evaluation results commensurate with the ecenario objectives. l S. If time compression techniques are used, the scenario summary clearly so indicates.

Operators have sufficient time to carry out expected activities without undue time constraints. Cues are given.

9. The simulator modeling is not altered.

10. The scenarios have been vahdated.

11. Every operator will be evaluated using at least one new scenario. All other scenarios have been modified in accordance with Section D.4 of ES-301.

12. All individual operator competencies can be evaluated, as verified using Form ES-301-6 (submit the form along with the simulator scenarios).

13. Each applicant will be significantly involved in the minimum number of transients and events specified on Form ES-301-5 (submit the form along with the simulator scenarios).

14. The level of difficu!ty is appropriate to support licensing decisions for each crew position.

TARGET QUANTITATIVE ATTRIBUTES (PER SCENARIO) Actual Attributes - - -

1. Total malfunctions (5-8) / /

2. Malfunctions after EOP entry (1-2) / /

3. Abnormal events (2-4) / /

4. Major transients (12) / /

5. EOPs entered / requiring substantive actions (1-2) / /

6. EOP contingencies requiring substantive actions (0-2) / /

7. Critical tasks (2-3) / /

NUREG-1021 24 of 26 Interim Rev. 8, January 1997

4 ES-301 Transient and Event Checklist Form ES-301-5 ) OPERATING TEST N0.:

Applicant Evolution Minimum Scenario Number i Type Type Number 1 2 3 4 Reactivity 1

Normal 1 1 R0 Instrument 2

) Component 2 , Major 1 i Reactivity 1 i Normal 0 As R0 Instrument 1 Component 1 l Ma_ior 1 i ( SRO-I Reactivity 0 l

Normal 1 As SR0 Instrument 1

. Component 1 i Major 1 Reactivity 0 5 - Normal 1 SRO-U Instrument 1

Component 1

} Major 1 2 l Instructions: (1) Enter the operating test number and Form ES-D-1 event numbers for each evolution type 1 (2) Reactivity manipulations must be significant as defined in Appendix D.

!      Author:

Chief Examiner: NUREG-1021 25 of 26 Interim Rev. 8, January 1997 1

ES-301 Competencies Checklist Form ES-301-6 l Applicant #1 Applicant #2 Applicant #3 R0/SRO-I/SRO-U R0/SRO-I/SRO-U R0/SRO-I/SRO-U Competencies SCENARIO SCENARIO SCENARIO I 1 2 3 4 1 2 3 4 1 2 3 4  ! Understand and Interpret Annunciators and Alarms Diagnose Events and Conditions Understand Plant and System Response Comply With and l Use Procedures (1) Operate Control Boards (2) Communicate and Interact With the Cred Demonstrate Supervist ry Ability (3) Comply With and Use Tech. Specs. (3) Notes: (1) Includes Technical Specification compliance for an R0. (2) Optional for an SRO-U. (3) Only applicable to SR0s. Instructions: Circle the applicant's license type and enter the event numbers that test the , competency for each scenario in the set. Author: Chief Examiner: NUREG-1021 26 of 26 Interim Rev. 8, January 1997 O. l

                                                                                                  +

l , i ES-302  ! , ADMINISTERING OPERATING TESTS TO INITIAL LICENSE APPLICANTS  ! i i, A. PURPOSE ! This standard describes how to administer operating tests to initial license i applicants in accordance with the requirements of 10 CFR 55.45.- It includes  ;

j. policies and guidelines for administering both the walk-through and the  !

- integrated plant operations categories of the operating test. It is assumed l l

that the operating test was prepared in accordance with ES-301. j j B. BACKGROUND , As noted in ES-201, facility licensees will generally prepare proposed  ! operating. tests in accordance with ES-301-and submit them to the responsible l NRC regional office for review and approval. Regardless of whether it was l prepared by the facility licensee or the NRC, every operating test will be  ; independently administered and graded by an NRC licensing examiner in i accordance with the instructions contained herein and in ES-303.

• C. RESPONSIBILITIES

1. Facility Licensee l

The facility-licensee is responsible for the following activities: l

a. Make the plant and simulation facility available, as necessary, j for validating and administering Category B (control room knd in- i plant systems) and Category C (integrated plant operations) of the operating tests.

b. Safeguard the integrity and security of the operating tests in accordance with ES-201.

c. Provide administrative and logistics support (e.g., personnel to opert.te the simulation facility, surrogate operators, etc.) to far.iiitate the administration of the operating tests-in accordance with Section D.

d. Inform the NRC regional office in writing if an applicant withdraws from the examination process before it is complete.

2. NRC Reaional Office The NRC regional office is responsibb for the following activities:

a. Work with the facility contact to coordinate the operating test administration schedule in a manner that maximizes efficiency and maintains security (refer to ES-201)'.  !

l NUREG-1021 1 of 9 Interim Rev. 8, January 1997 i

ES-302

b. Administer the operating tests in accordance with Section D. I D. TEST ADMINISTRATION INSTRUCTIONS AND POLICIES

1. General

a. Before beginning the operating test, an examiner shall brief the applicant (s) using Parts A, C, D, and E of Apnendix F. To save time, it is recommended that the examiner (s) briaf the applicants as a group.

b. If an applicant requests to withdraw during any part of the examination process, the examiner shall inform the applicant that this will result in automatic license denial and that he or she may reapply in accordance with 10 CFR 55.35. The chief examiner will request the facility licensee to document the applicant's withdrawal in a letter to the NRC regional administrator.

c. Each applicant listed on the examination assignment sheet (see ES-201, Attachment 4) shall be administered an operating te:it as indicated under " Examination Type."

d. For purposes of test integration and continuity, the chief examiner should generally schedule the same examiner to administer all three operating test categories to an applicant. However, under certain circumstances, such as when a licensee's simulation facility is not located near the plant or ' licensee requests examinations for an unusually large gro- glicants, the responsible regional supervisor may autk he chief examiner to divide the operating test categories a x abcategories among different examiners. The chief examiner will be responsible for ensuring that each applicant gets a complete operating test and that the tests are thoroughly and accurately documented.

e. The examiner is expected to administer the planned operating test l in accordance with the prepared and approved walk-through test outlines (Forms ES-301-1, " Administrative Topics Outline," and ES-301-2, " Individual Walk-Through Test Outline") and simulator scenarios (Forms ES-D-1, " Scenario Outline," and ES-D-2, " Operator l Actions"). Examiners shall document every significant aspect of -

each applicant's performance for later evaluation, but they shall not use the applicant's unplanned actions and statements to displace any part of the planned operating test. Normally, examiners should substitute or replace planned operating test materials only if it is determined that an item is invalid or I impossible to perform or simulate because of unanticipated access restrictions or equipment failures. NUREG-1021 2 of 9 Interim Rev. 8, January 1997

O ES-302 V f. Exaniiners may administer the same operating test (walk-through and simulator) to consecutive applicants and crews on the same day, , but they must ensure that the security of the operating test is ' maintained. The same simulator scenarios shall not be repeated . during successive days. i If previously agreed upon by the facility licensee, examiners may

also administer the same operating test (walk-through and simulator) by dividing the test into segments that can be administered to all of the applicants on the same day. This will 1 minimize the amount of effort required to develop different '

operating tests but will complicate the scheduling process.

g. The examiner should normally administer Categories B and C of the operating test first and attempt to concurrently evaluate as many of the planned administrative subjects in Category A as possible.

The remaining administrative subjects should then be evaluated in accordance with the approved outline.

h. The examiner must take sufficient notes to facilitate the thorough I documentation of any and all applicant deficiencies in accordance with ES-303. The examiner must be able to cross-reference each comment to a specific JPM, simulator event, or question.

O i. The making of videotapes during the administration of operating tests is not authorized. J l J. The number of persons present during an operating test should be limited to ensure the integrity of the test and to minimize distractions to the applicants. Except for the simulation facility operators, no other member of the facility's staff shall be allowed to observe an operating test without the chief examiner's permission. Facility management and other personnel deemed necessary by the facility licensee should generally be allowed access to the examination (under security agreements, as appropriate), provided the simulation facility can accommodate them and there is no impact on the applicants. Although the simulation facility operator will normally assume the role of the other personnel that the applicants direct or notify regarding plant operations, the chief examiner may permit other members of the facility training or operations staff to augment the operating shift team if necessary. The chief examiner shall fully brief those individuals regarding their responsibilities, reporting requirements, duties, and level of participation before the operating test begins. The examiners must not restrict the surrogate operators' activities to such an extent that the

 \ NUREG-1021                            3 of 9          Interim Rev. 8, January 1997

ES-302 applicants being evaluated are required to assume responsibilities beyond the scope of their position. The surrogate operators will be expected to assume the full responsibilities of the roles they take in the operating test. If the facility licensee normally operates with and is required by its technical specifications to have more than two reactor operators (R0s) in the control room, OLB may authorize the use of additional surrogates to fill out the crews. In such cases, examiners must take care that the presence of additional operators does not dilute the examiners' ability to evaluate each applicant on every competency and rating factor. Examiners shall not hesitate to run additional scenarios, as necessary, to ensure that every applicant is given the opportunity to demonstrate his or her competence. Under no circumstances will another applicant be allowed to witness an operating test. Operating tests are not to be used as training vehicles for future applicants. Other examiners may observe an operating test as part of their training or to audit the performance of the examiner - administering the operating test. The chief examiner may permit other NRC employees, such as resident inspectors, regional personnel, researchers, or NRC supervisors, to observe an operating test. Personnel who are not NRC employees (e.g., representatives from the Institute of Nuclear Power Operations (INPO)) may observe the operating tests with prior approval from the Chief, OLB. The chief examiner will control the observer's activities in accordance with guidance provided by OLB. The examiner should also give the applicant the opportunity to object to the presence of observers.

k. The chief examiner should confirm with the facility licensee that the simulator instructor's station, programmers' tools, and external interconnections do not compromise operating test security while conducting examinations. The primary objective is to ensure that the exam material cannot be read or recorded at other unsecured consoles and that examination material is either physically secured or electronically protected when not in use by individuals listed on the security agreement.

1. The chief examiner should arrange for any NRC examiners who are not familiar with the facility to obtain a tour before they administer any operating tests. The tours shall not be conducted or observed by any of the applicants. In addition, the tours NUREG-1021 4 of 9 Interim Rev. 8, January 1997

7 i-e l ES-302 l should concentrate on areas of the plant that will be used during i the examination process, such as the control room, the simulation facility, and planned walk-through locations. i m. The chief examiner will conduct an exit briefing with the facility

licensee after the. operating tests are complete. The briefing should address any generic weaknesses noted during the operating i tests and any other significant issues (e.g., problems with the

reference material, the simulation facility, or the plant) that j might be addressed in the examination report. The individual-operating test results are predecisional until approved by NRC

!. management in accordance with ES-501 and shall not be shared with 5 the facility licensee during the exit briefing.

2. Walk-Throuah (Cateaories A and B)

a. The examiner should validate any JPMs.that were not previously validated by the facility licensee or by the NRC during a preparatory site visit. This is particularly important for complex JPMs and those that require the applicant to implement an alternative method directed by plant procedures.

b. To the extent possible, the examiner should have the applicant perform the control room JPMs on the simulator, rather than asking O the applicant to describe how he or she would accomplish the task.

The chief examiner is expected to coordinate the administration of the JPMs to maximize the use of the simulator. To increase efficiency, different JPMs may be administered simultaneously to multiple applicants, but the examiners must ensure that mutual interference is minimized and test integrity is not compromised. When JPMs or discussions are conducted in the control room, the l examiners shall make every effort to accommodate and not interfere l with normal shift operations. The chief aaminer should request  ! that the facility training manager notify the shift supervisor when the NRC will be conducting examination activities in the control room. If the number of persons or the noise level in the i control room is excessive, the examiner should, if possible, move j to a quieter location, modify the sequence of the JPMs and return l when the level of activity in the control room has abated, or ask l the facility training manager to address the issue.

c. The examiner should encourage the applicant to sketch diagrams, flow paths, or other illustrations to aid in answering the 1 examiner's questions. In all cases, the examiner shall collect the supporting material because it provides additional documentation to support a pass or fail decision (refer to i ES-303). To facilitate copying, the applicant's drawings should be restricted to one side of separate sheets of 8.5-inch by NUREG-1021, 5 of 9 Interim Rev. 8, January 1997 j

l ES-302 11-inch paper; the back of Form ES-303-1 or its attachments shall not be used for this purpose.

d. The examiner should encourage the applicant to use such material as facility forms, schedules, and procedures if they are relevant to the questions asked.

e. The examiner should be careful not to infer excessive system and administrative knowledge from observations made during Category C.

The examiner should keep in mind that the applicant's proficiency in every administrative topic and each control room and in-plant system should be deliberately evaluated in accordance with the I operating test that was prepared in accordance with ES-301. i

f. The examiner should ask unplanned follow-up questions only as necessary to clarify or confirm the applicant's understanding of a preplanned task or prescripted question.

3. Simulator Test (Cateaory C)

a. Before administering the test (s), the examiners will validate each scenario cr. the sit.dator to ensure that it will run as intended.

Scenarios that were adapted from previous NRC examinations at the facility or from the facility licensee's bank may not require real-time validation. At a minimum, the examiners will " dry run" I those events having variable inputs and questionable outcomes and I discuss the remainder of the scenario with the facility's l simulator instructor to ensure that it will run as planned. l In some cases, the scenarios can be validated while the applicants are taking the written examination. However, it may be beneficial l to validate the scenarios during a preparatory site visit as determined by NRC regional management (refer to ES-201).

b. The examiners will take precautions to prevent the scenarios from i being revealed to the applicants before the tests begin. If significant portions of the scenarios are dry run or otherwise reviewed with the simulator instructor (s), the chief examiner shall ask the instructor (s) to sign a security agreement (Form ES-201-3) to protect the integrity of the simulator test,

c. The examiners should revise all copies of Forms ES-D-1 and ES-D-2 to reflect any changes made to the scenario events or the expected operator actions as a result of the scenario validation runs and reviews. These revisions should be neatly written in ink so that the forms can be used in the final write-up of the simulator test, as discussed in ES-303.

NUREG-1021 6 of 9 Interim Rev. 8, January 1997

1 d ES-302 , d. The examiners should review the scenarios together and discuss the required procedures, technical specifications, special circumstances, and so forth, related to the scenarios.

e. Immediately before beginning the simulator tests, the examiners should review the scenario events with the simulator operator and provide him or her with a copy of Form ES-D-1. This review should familiarize the operator with the sequence of events to ensure that they will proceed as planned. This is particularly important ,

if the simulator operator during the test is not the same j individual who assisted in validating the scenarios. l f. The examiners should identify important plant parameters to be monitored during each simulator scenario. The chief examiner i should ask the simulator operator to record selected parameters, ' if possible, on the facility's safety parameter display system (s). Parameter readings should be collected at meaningful intervals, l depending on the parameter, the nature of the event, and the ' capability of the simulation facility. The chief examiner should - retain the recordings as backup documentation to augment the notes ) taken by the examiners during the simulator test.

g. The examiner in charge of each scenario should arrange a suitable communication system with the simulator operator so that he or she )

S can be prompted to insert the malfunctions without cuing the s applicants. Malfunctions may be planned for a predetermined time l or power level so that the examiners and the facility operator are , aware of the event that is occurring or pending. ' If necessary, the examiners may use time compression to speed up the response of key parame'.ers so that the scenario can proceed to the next event within a reasonable time. Time compression is acceptable as long as it is used judiciously and the operators are given sufficient time to perform the tasks that they would typically perform in real time. If the examiners intend to use time compression, they should inform the applicants of that fact

      ,               during the operating test briefing (refer to Section D.I.a). The                  ,

examiners should also mitigate the potential for negative training i by debriefing the applicants after any scenario in which time compression was used.

h. Before beginning each scenario set, the examiners should advance any control room strip chart recorders that may prove useful in recreating the sequence of events. The charts should be clearly marked with the date, time, and examiner's initials so that they can be accurately matched with the correct operating crew,

i. The chief examiner should ensure that the simulator operator (or examiner) playing the role of other D

NUREG-1021 7 of 9 Interim Rev. 8, January 1997

l l ES-302 i information. For example, fast-time could be specified for l auxiliary operator checks or lineups to prevent long delays in l simulated operations, while maintenance and chemistry sample information can be provided with normal time delays to present the applicants with the same analysis problems that they will face as operators. J. Before the simulator test begins, the examiners shall caution the simulator operator to provide only information that is specifically requested by the applicants. When the simulator operator is briefing the applicants or communicating with them on the telephone, the examiners should monitor the conversation to ensure that the information provided is appropriate and does not cue the applicants.

k. Each examiner should use the expected actions and behaviors listed on Form ES-D-2 as a guide while administering the simulator tests.

If an applicant performs as expected, the examiner may simply note in the left-hand column of the form the time when the expected actions occurred. However, if an applicant does not perform as expected, the examiner should note the applicant's actions (or lack of actions) next to or below the expected action and follow up with appropriate questions after the simulator scenario is completed (refer to Section D.3.1). Each examiner must determine the best way to document the applicant's actions. Some examiners record a minute-by-minute account of all key plant events and the applicant's actions as they occur; other examiners only record the applicant's significant actions. Each individual examiner should develop his or her own examination documentation technique; however, the documentation must provide an adequate basis for a licensing decision. In addition, the examiner's notes must provide sufficient information to allow the ex; miner to confidently judge the applicant's performance on the competencies described in Appendix D.

1. Examiners shall limit discussions with the applicants during the scenarios both to maintain realism and to avoid distracting the applicants from operating the plant. The examiners' questions during the scenarios should be limited to those that are necessary to assess the applicants' understanding of plant conditions and the required operator actions. Whenever possible, the examiner shall defer questioning the applicant until a time when the applicant is not operating or closely monitoring the plant (preferably after the simulator has been placed in " freeze"). The examiner's follow-up questions or concerns can generally be addressed during a brief question and answer period after each l

NUREG-1021 8 of 9 Interim Rev. 8, January 1997

1 ES-302  ! scenario or during the control room systems and facility walk-through portion of the operating test (i.e., Category B) if it,is performed after the simulator test.

m. The examiners who administer the simulator test shall confer l immediately after completing the scenario set to compare notes and l to verify that.each examiner observed his or her applicant perform l the required number of transients and events in a manner '  !

sufficient to justify an evaluation of all the required i competencies. If necessary, the examiners shall run an additional ) scenario to ensure that the required evolutions and competencies  ! are covered. All scenarios will be planned and documented in  ! accordance with Section D of ES-301. i The chief examiner shall ensure that the examiners' observations are consistent and that their findings are mutually supportive. If a performance deficiency is " shared" by more than one applicant, it should be noted by both evaluating examiners. l Ideally, this cross-check should be accomplished as soon as possible after running the scenarios while still at the facility. The cross-check must be accomplished before finalizing the examination results in accordance with ES-303.

n. If the applicants did not perform as expected, the examiner shall O ask the simulator operator to provide copies of the logs, charts, and other materials that may be required after leaving the facility to evaluate and document the applicants' performance.

i i The examiner of record shall retain all documentation related to 1 l any operating test failure until the proposed denial becomes final-  ! l or a license is issued. The chief examiner should also ask the simulator operator to retain copies of the same materials until all applicants are licensed or all appeals are settled, as suggested in the sample j corporate notification letter shown in ES-201, Attachment 3. l o. If the simulation facility should become inoperable and cause excessive delay of the operating tests, the chief examiner should discuss the situation with the facility licensee and the responsible regional supervisor so that management can make a decision regarding the conduct of the operating tests. It may be necessary to reschedule the simulator examinations for a later date. i 4 i NUREG-1021 9 of 9 Interim Rev. 8, January 1997 i l ~ =_ - . - __ _

i f ES-303 l DOCUMENTING AND GRADING INITIAL OPERATING TESTS A. i PURPOSE

                                                                                                    )

i This standard describes the procedures for documenting all categories of the l l operating test, collating the data to arrive at a pass or fail recommendation, i and reviewing the documentation to ensure quality. l l l B. BACKGROUND i This standard assumes that the operating test was prepared and administered in l accordance with ES-301 and ES-302, respectively. In addition, the procedures 1 contained herein require the examiner to evaluate each applicant's performance 1 l on the operating test and make a judgement as to whether.the applicant's level l of knowledge and understanding meet the minimum requirements to safely operate the facility for which the license is sought. The examiner should evaluate ! each noted deficiency in light of the total breadth of knowledge and ability demonstrated by the applicant in that subject area.

When used to evaluate an applicant's performance on all or part of the

operating test as discussed herein, the terms " satisfactory" and j

! " unsatisfactory" are defined as follows:  ; S - Satisfactory Working Knowledge and Understanding l The applicant may have some slight or minor difficulty describing system j l interactions. Competence in the operation of equipment associated with t i the system is very good, although there may be some hesitation while  ! l discussing or performing some tasks. The applicant appears to be j familiar with the equipment and procedures. i l U - Unsatisfactory or Poor Working Knowledge and Understanding The applicant has difficulty 7.nswering questions in depth and describing the interactions of systems. Discussions or behavior while operating equipment show lack of familiarity with the equipment and procedures. The applicant is unable to anmer questions or provides incorrect or incomplete answers. The applicant is unfamiliar with the subject or system, as evidenced by heritant answers, inability to locate information, inability to locate control board indications or controls, l and lack of knowledge of systen operating procedures. C. RESPONSIBILITIES

1. Facility Licensee The facility licensee's responsibilities are limited to providing the NRC examiners with whatever additional reference materials and information the i examiner might require to evaluate the applicants' performance on the i

NUREG-1021 1 of 28 Interim Rev. 8, January 1997 l 1

i 1 l 1 operating tests. Such materials might include simulator strip chart recordings that document plant status during the simulator scenarios, and procedures that document the expected operator actions.

2. NRC Examiner of Record i

As soon as possible after administering the test, the examiner of record shall , review, evaluate, and finalize each applicant's operating test documentation I in accordance with the instructions in Section D.

3. NRC Chief Examiner I

a. The rhief examiner shall arrange a meeting of the NRC examination team members after the simulator scenarios are complete. Such l neetings enable the examiners to compare notes to ensure that the documentation for applicants on the same operating crew is consistent and mutually supportive.

b. The chief examiner shall work with the other examiners on the team to resolve any technical questions that might arise during the grading process, and communicate any additional reference material requirements to the facility contact.

c. The chief examiner or a management-approved designee will review the grading of each operating test to verify that the examiner's comments appropriately support his or her recommendation and to ensure that the operating test meets the requirements of ES-301.

If the chief examiner or designee does not agree with any of the examiner's recommendations, he or she shall confer with the examiner before overturning the recommendation. Such disagreements are not common and usually arise because an unsatisfactory grade is not adequately justified. It is, therefore, very important for examiners to be complete and accurate in their grading and documentation.

d. The chief examiner or designee shall make an independent pass-or-fail recommendation, sign the " Final Recommendation" block on Form ES-303-1, " Operator Licensing Examination Report," and forward the package to the responsible supervisor for review in accordance with ES-501. The supervisor must concur in any recommendation to overturn the examiner's results, and the specific reasons for this action must be explained on form ES-303-2, " Operating Test Comments."

D. GRADING AND DOCUMENTATION INSTRUCTIONS l 1. Review and Cateaorize Rouah Notes and Documentation l a. Review the job performance measures (JPMs) and simulator scenarios  : that were performed and the questions that were asked. Evaluate l NUREG-1021 2 of 28 Interim Rev. 8, January 1997 l l l

ES-303 all rough notes and documentation generated while administering the operating test to determine the areas in which the applicant was deficient. If the applicant generated or used any material (such as figures, drawings, flowchart.<, or forms) during the operating test, the material may be used to aid in documenting the applicant's performance. If it contributes to an unsatisfactory performance evaluation, the material shall be appropriately marked and cross-referenced to the applicable deficiency and attached to the examination package for retention.

b. Verify the validity and technical accuracy of any questions that were asked during the walk-through test but had not been prescripted, as well as any unexpected events or actions that occurred during the simulator operating test. If necessary, work through the chief examiner to obtain any additional reference material that might be required to resolve any technical .

questions,

c. On the rough notes and documentation, label or highlight every action, response, note, or comment that may constitute a performance deficiency. l 1

d. Label each deficiency related to the applicant's administrative s and plant system knowledge and abilities with the alphanumeric code of the administrative topic (e.g., A.1) or the control room V or plant system (e.g., B.I.3 or B.2.1) to which it applies.

e. Review each simulator operating test performance deficiency.

Using as a guide the competency and rating factor descriptions in Appendix D and on Form ES-303-3 (RO) or Form ES-303-4 (SRO), code each deficiency with the number and letter of the rating factor (s) it most accurately reflects (e.g., C.4.a). Whenever possible, attempt to identify the root cause of the applicant's deficiencies and code each deficiency with no more than two different rating factors. However, one significant deficiency may be coded with additional rating factors if the error can be shown, consistent with the criteria in Section D.3.b, to be relevant to each of the cited rating factors. As stated in ES-302, it is essential that the simulator operating test documentation is consistent and mutually supportive for all applicants in an operating crew. Operating errors that involved more than one applicant should be noted by each applicant's evaluating examiner. If the examination team members do not have the opportunity to discuss and compare their observations before leaving the site, the chief examiner shall schedule a conference call after the examiners return to their respective offices. v NUREG-1021 3 of 28 Interim Rev. 8, January 1997 l l I

2. Evaluate the ADolicants' Performance After categorizing and coding the rough notes, review, evaluate, and grade each applicant's performance in operating test Category A,

     " Administrative Topics," Category B, " Control Room Systems / Facility Walk-Through," and Category C, " Integrated Plant Operations," as follows:

a. Form ES-303-1, Category A Review the identified deficiencies and decide whether the applicant's knowledge and understanding of each administrative topic was satisfactory or unsatisfactory (refer to the discussion in Section B). Document the grade by placing an "S" or "U" in the appropriate block on page 2 of Form ES-303-1. Every unsatisfactory grade must be supported with detailed documentation as discussed in Section D.3.

After grading all of the topics in Category A, assess the applicant's topic grades and deficiencies and assign a single "S" or "U" grade for the category. If the applicant has a "U" in only one administrative topic, the examiner may fail the applicant in Category A depending on the importance of the identified deficiency. However, if the applicant has a "U" in two or more of the administrative topics, the examiner must assign a grade of "U" for Category A. Place the assigned grade in the " Administrative Topics" block of the " Operating Test Summary" on page 1 of Form ES-303-1. Enter "N/E" (not examined) if this category was waived in accordance with ES-204. 1

b. Form ES-303-1, Category B On page 2 of the applicant's Form ES-303-1, enter the names of the systems and JPMs examined during operating test Subcategories B.1,

            " Control Room Systems," and B.2, " Facility Walk-Through." Also enter the safety function number from the appropriate Knowledge and Abilities Catalog (NUREG-1022 for PWRs and NUREG-1023 for           l BWRs).

To determine if the JPMs listed on Form ES-303-1 were properly performed, evaluate each deficiency coded in the rough notes for Category B. If the following criteria are met, assign a satisfactory grade by placing an "S" in the "JPM Evaluation" column for that system; otherwise enter a "U": If the JPM was time-critical, it must be completed within the allotted time. All of the critical steps identified for a JPM must be completed correctly. NUREG-1021 4 of 28 Interim Rev. 8, January 1997

ES-303 (V) If the applicant initially missed a critical step, but later performed it correctly and accomplished the task standard without degrading the condition of the system or the plant, the applicant's performance on that JPM should be graded as satisfactory. Further evaluate the Category B documentation to determine a grade for the prescripted questions associated with each system listed on Form ES-303-1. It is permissible to allot partial credit, when appropriate, to determine whether or not the applicant's understanding of a particular K/A topic is satisfactory. If the applicant's understanding of both K/A topics is satisfactory, enter an "S" in the " Question Grade" column for that system on page 2 of Form ES-303-1. If the applicant's understanding of only one of the K/A topics is satisfactory, enter an "S" unless the knowledge or ability that was missed is of such safety significance that an unsatisfactory system grade is justified; in the latter case enter a "U." The K/A importance value will not be the only consideration in arriving at the unsatisfactory evaluation, and the safety significance must be explained in the documentation. If the applicant's understanding of both K/A topics is unsatisfactory, enter a "U" in the " Question

,m              Grade" column for the system. Every grade of "U" on a JPM or the

/ system questions must be supported with detailed documentation as V) discussed in Section D.3. For each Category B system listed on page 2 of form ES-303-1, enter a " System Grade" of "S" if the applicant's performance on both the JPM and the prescripted questions was satisfactory. If the applicant was assigned a "U" for either the JPM or the prescripted questions, then enter a "U" for the " System Grade." After grading the applicant's performance with respect to all the Category B systems, determine an overall grade for Category B by calculating the percentage of satisfactory system grades. If the

   ,            applicant has an "S" on at least 80 percent of the systems examined, the applicant passes Category B and receives an "S" overall. If the applicant has an "S" on fewer than 80 percent of the systems, the applicant fails Category B and receives a "U" overall.

Document the applicant's grade by placing an "S" or a "U" in block B, " Control Room Systems and Facility Walk-Through," in the

                " Operating Test Summary" on page 1 of Form ES-303-1. Enter "N/E" if this category was waived in accordance with ES-204.

( L NUREG-1021 5 of 28 Interim Rev. 8, January 1997

I l ES-303 c, Form ES-303-1, Category C Using Form ES-303-3 or ES-303-4, depending on the applicant's license level, evaluate any deficiencies coded for Category C. Circle the integral rating value (1 through 3) corresponding to the behavioral anchor that most accurately reflects the applicant's performance. A rating of "1" would be justified if the applicant missed a critical task (i.e., by omission or incorrect performance) or committed multiple errors of lesser significance that have a bearing on the rating factor. Missina one or more critical tasks does not necessarily mean that the aoolicant will fail the simulator test. nor does success on every critical task prevent the examiner from recommendina a failure if the apolicant had other deficiencies that. in the aaarecate.

          ,iustify the failure based on the competency evaluations. As discussed in ES-301, Competency 5 is optional for SRO upgrade applicants.

Multiply each integral rating value by its associated weighting factor to obtain a numerical measure of the applicant's performance on each rating factor. Then, circle the corresponding numbers on page 3 of the R0 or SR0 applicant's Form ES-303-1. For each rating factor, place check marks in the columns on page 3 of Form ES-303-1 corresponding to the numbers of the scenarios in which the rated behavior was observed. For each competency on page 3 of Form ES-303-1, sum the circled rating factor grades and enter the resulting competency grade in the " Total" column. (The grades should range between 1 and 3.) Using the following evaluation criteria, determine if the applicant's overall performance in Category C is satisfactory or unsatisfactory and document the grade by placing an "S" or a "U" in block C, " Integrated Plant Operations," in the "0rerating Test Summary" on page 1 of Form ES-303-1. Enter "N/E" if this category was waived in accordance with ES-204. If the " total" grade for a77 competencies is greater than 1.8, the applicant's performance is generally satisfactory. If the applicant made (an) error (s) with serious safety consequences for the plant or the public, the examiner can recommend a failure even if the failure can not be justified on the basis of the competency grades. In such circumstances, the region must obtain written concurrence from the Chief, OLB, before completing the licensing action. If the " total" grade for Competency 6, " Communications and Crew Interactions," is less than or equal to 1.8 but greater than 1.0, and the individual " total" grades for a77 other NUREG-1021 6 of 28 Interim Rev. 8, January 1997 1

O ES-303 V competencies are 2.0 or greater, the applicant's performance is satisfactory. If the " total" grade for Competency 6 is 1.0, or the " total" grade for any other competency is 1.8 or less, the applicant's performance is unsatisfactory. Note: Competency 5, " Control Board Operations," is optional for SR0 upgrade applicants. However, if it is evaluated, it shall be factored into the applicant's final grade. Justify in detail, as discussed in Section D.3, each rating factor that is assigned an integral rating value of 1, regardless of the

              " total" grade determined for the associated competency. If the applicant's overall performance in Category C is unsatisfactory, justify in detail every rating factor assigned an integral rating value of I or 2 for each competency that has an unsatisfactory score.

3. Finalize the Documentation

a. Review and finalize the simulator scenarios that were run for Category C of the operating test.

Complete Form ES-D-1, " Scenario Outline," by entering the s applicants' names, the positions they occupied during the scenario, and the facility's name on the top of the form. Also enter on Form ES-D-1 any scenario revisions made during the test, so that each form accurately shows all of the events that actually occurred during each scenario. Change the event numbers, malfunction numbers, malfunction types, and descriptions as necessary to reflect the "as run" conditions. These changes may  ; be made using pen-and-ink or by retyping the scenario, provided that the final form is clear and legible. Update each Form ES-D-2, " Operator Actions," to reflect the "as run" conditions. Discard or mark as "not used" any events that were not run, and fill out new forms for any events that were run but not originally planned. Neatly enter notes, comments, and additional actions in the spaces between the expected operator actions. The final Forms ES-D-1 and ES-D-2 must be a clear, legible, and c sequential record of the actual events and actions that occurred l during the simulator operating test. The forms sent to the applicant shall not contain any rough notes or irrelevant comments. Any events or malfunctions that did not function as expected or I were not useful in evaluating the applicants (e.g., a surveillance NUREG-1021 7 of 28 Interim Rev. 8, January 1997

ES-303 test that required a long time to perform) should be noted on the master copy of the scenarios to aid in future scenario preparation.

b. Review the applicant's Form ES-303-1 and the rough documentation.

Justify in detaff on Form ES-303-2, " Operating Test Comments," every knowledge or ability deficiency that contributed to the assignment of a "U" for any administrative topic in Category A or any system in Category B. Deficiencies that contributed to Category C integral rating factor grades of I and 2, as discussed in Section D.2.c, must also be justified in detaf f if they resulted in an unsatisfactory score in the associated competency. Deficiencies that do not contribute to an unsatisfactory grade anywhere on the operating test shall also be documented; however, a brief statement describing the error is ger.erally sufficient. Examiners should keep in mind that their licensing recommendation and the associated documentation will be subject to review by the chief examiner and regional management. Therefore, the documentation should contain sufficient detail so that the independent reviewer, responsible supervisor, and licensing official can make a logical decision in support of the examiner's recommendation to deny or issue the licer.se. It is expected that the documentation for an applicant who marginally passed the operating test would be very similar in scope and depth to that for an applicant who marginally failed. Provide the following specific information, as applicable, for each deficiency that contributed to an unsatisfactory evaluation: the question asked or task administered (i.e., describe the JPM or the simulator scenario and event, as well as the applicant's position on the operating crew) . 1 the applicant's incorrect answer or action and an indication of whether the action was a JPM critical step the lack of knowledge or ability that the applicant demonstrated the consequences of the applicant's incorrect answer or action the correct answer or action, with an appropriate facility reference (e.g., lesson plan, system description, procedure name and number) i the K/A number and its importance rating (as given in NUREG-1122 or NUREG-1123) and the facility's learning objective NUREG-1021 8 of 28 Interim Rev. 8, January 1997

ES-303 V the item from 10 CFR 55.45(a) that the applicant did not understand or was unable to perform s General statements (such as "did not know decay heat removal system") are inadequata. Whenever possible, tubstantiate comments with printouts or strip l chart recordings generated during the simulator operating test and i i drawings and illustrations generated by the applicant. Retain rough documentation until the chief examiner and management have reviewed the examiner's recommendations and concurred in the results (refer to ES-501).  ! ! c. Cross-reference each comment on Forn. ES-303-2 with the specific task, subject, or competency rating factor to which it applies on , the applicant's Form ES-303-1. Do this by entering the applicable i alphanumeric subject reference from Form ES-303-1 (e.g., A.2, B.I.3.C, C.4.B) in the left-hand column of Form ES-303-2, and entering the page number on which the comment is found in the appropriate block on Form ES-303-1. i

4. Make a Final Recommendation I G l

a. After grading and documenting the operating test, make an overall .

recommendation by checking the " Pass" or " Fail" (or " Waive" if the i entire operating test was waived in accordance with ES-204) block, and signing and dating the " Examiner Recommendations" section on l the applicant's Form ES-303-1. Make a " pass" recommendation only ' if all summary blocks of the operating test contain satisfactory (S) grades or the letters "N/E," indicating that the applicant was not examined in that area.

b. If the written examination was not waived and the written examination data have not yet been entered on Form ES-303-1, route the examination package to the written examination grader (or NRC reviewer if the examination was graded by the facility licensee) for processing in accordance with ES-403. If the written examination results have already been entered, forward the examination package to the chief examiner for review.

E. ATTACHMENTS / FORMS Form ES-303-1, " Operator Licensing Examination Report" Form ES-303-2, " Operating Test Comments" Form ES-303-3, "R0 Competency Grading Worksheet for Integrated Plant Operations" Form ES-303-4, "SR0 Competency Grading Worksheet for Integrated Plant Operations" NUREG-1021 9 of 28 Interim Rev. 8, January 1997

ES-303 Operator Licensing Examination Report Form ES-303-1 U.S. Nuclear Regulatory Commission Operator Licensing Examination Report Applicant's Name Docket Number 55-I R Examination Type (Initial or Retake) Facility Name Reactor Operator Hot Senior Reactor Operator (SRO) Instant Cold Facility SR0 Upgrade BWR Description SR0 Limited to Fuel Handling PWR Written Examination Sumary NRC Author / Reviewer Total Examination Points NRC Grader / Reviewer Total Applicant Points Date Administered Applicant Grade  % Operating Test Summary Administered by Date Administered

a. Administrative Topics

b. Control Room Systems and Facility Walk-Through

c. Integrated Plant Operations (Simulator Operating Test)

Examiner Recommendations Check Blocks Pass Fail Waive Signature Date  ! Written Examination Operating Test Final Recommendation License Recommendation Issue License Signature - Supervisor Date Deny License NUREG-1021 10 of 28 Interim Rev. 8, January 1997

ES-303 2 Form ES-303-1 Applicant Docket Number: 55- Page of A. Administrative Topics Evaluation Comment Page (S or U) Number

1. Conduct of Operations

2. Equipment Control

3. Radiation Control

4. Emergency P1an B.1 Control Room Systems Safety JPM Grade Function (S or U)

Question Grade (S or U) System Comment Grade Page ( System /JPM Title (S or U) Number 1. 2. 3. 4. 5. 6. 7. B.2 Facility Walk-Through 1. 2.

3. I j

i NUREG-1021 11 of 28 Interim Rev. 8, January 1997 l l l

                                                                       )

ES-303 3.a Form ES-303-1 Applicant Docket Number: 55- Page of C. Reactor Operator Integrated Plant Operations (Simulator Operating Testi Grading Summary Competencies / Scenarios Comment Rating Factors Weight 3.0 2.0 1.0 Total Observed Page No.

1. Alarms / Annunciators 1 2 3

a. Notice / Acknowledge 0.30 0.90 0.60 0.30 _ _ , ,,_

b. Interpret / verify 0.40 1.20 0.80 0.40 _ __,,. ._,_ _

c. Prioritize 0.30 0.90 0.60 0.30 , _ _,, _,,_,

2. Diagnosis 1 2 3

a. Recognize 0.40 1.20 0.80 0.40 _,,,,. ,,,,_

b. Use Reference Material 0.20 0.60 0.40 0.20 .,_. _ _,,._

c. Diagnose 0.40 1.20 0.80 0.40 . _ , _, , , , , , ,__

3. System Response 1 2 3

a. Locate / Interpret 0.33 1.00 0.67 0.33 _ _. ,_ ,_,

b. System Operation Knowledge 0.33 1.00 0.67 0.33 .,_,, . ., , , , , ___ _

c. Effect of Actions 0.33 1.00 0.67 0.33 _,,_ ,,,,,_ ,,,,,,,, _

4. Procedures / Tech specs 1 2 3

a. Reference 0.20 0.60 0.40 0.20 _ _ _ _

b. Eop Entry /Imediate Actions 0.40 1.20 0.80 0.40 _,,,, _,, __,, ,,_

c. Prnre.iare Compliance 0.20 0.60 0.40 0.20 ,,_ __, _ ,,,,_

d Tech spec Entry 0.20 0.60 0.40 0.20 ,,,, _ _ , ,

5. Control Board Operations 1 2 3

a. Locate 0.25 0.75 0.50 0.25 _ _ _, ,__

b. Manipulate 0.25 0.75 0.50 0.25 ,_ ,,_

c. Response 0.25 0.75 0.50 0.25 ,,__, _

d. Manual control 0.25 0.75 0.50 0.25 ,_ _ _,_ _,

6. Communications 1 2 3

s. Provide Information 0.33 1.00 0.67 0.33 ,_ _ .,,,_

b. Receive Information 0.33 1.00 0.67 0.33 _ _ _ _

c. Carry Out Instructions 0.33 1.00 0.67 0.33 ,,, __, _.,

NUREG-1021 12 of 28 Interim Rev. 8, January 1997

l l l t O ( ES-303 3.b Form ES-303-1 j Applicant Docket Number: 55- Pace of a C. Senior Reactor Operator Integrated Plant Operations (Simulator ODeratina Testi Gradino Summary  ; Coupetoneies/ ScenerIos Comunent Ratine Factors Weight 3.0 2.0 1.0 Total Observed Page No.

1. Alarms /Annmelators 1 2 3

e. Prioritize 0.30 0.90 0.60 0.30 _ _ _ _

b. Interpret 0.35 1.05 0.70 0.35 ,,_ _

4

c. Verify 0.35 1.05 0.70 0.35 ,_,,_ __ __

l 1

2. Diagnosis 1 2 3

a. Recognize 0.25 0.75 0.50 0.25 _ ,__, _ _,_,

b. Accuracy 0.25 0.75 0.50 0.25 .__ _,,_,. _ _ j

c. Diagnose 0.25 0.75 0.50 0.25 _ _ _ _

d. Crew Response 0.25 0.75 0.50 0.25 ,_ _ _ _,_

3. system Respons, 1 2 3

. a. Interpret 0.35 1.05 0.70 0.35 _,. _._ _, I

b. Attentive 0.20 0.60 0.40 0.20 _ _ _ _

c. Plant Effects 0.45 1.35 0.90 0.45 ,_ _ _ _

i . 4. ProcessJres 1 2 3 4 ( a. Reference

b. Correct use 0.25 0.50 0.75 1.50 0.50 1.00 0.25 0.50

c. Crew Implementation 0.25 0.75 0.50 0.25 __ ,_ _

j 5. Control Board operations 1 2 3

a. Locate 0.25 0.75 0.50 0.25 _,,,,,, . , _ , ,_.

b. Manipulate 0.25 0.75 0.50 0.25 _ _,,,_ _

c. Response 0.25 0.75 0.50 0.25 .__ .__ _ _

d. Manual control 0.25 0.75 0.50 0.25 ,_,_ _

i

6. Cosimmications 1 2 3

a. Clarity 0.45 1.35 0.90 0.45 ,,_. _. _ ,_

b. Crew Informed 0.35 1.05 0.70 0.35 .,_. _ _ _

c. Receive Information 0.20 0.60 0.40 0.20 _ _ _ _

7. Directing operations 1 2 3 4 s. Timely Action 0.20 0.60 0.40 0.20 _ _ _ _

b. Safe Directions 0.40 1.20 0.80 0.40 ,_ ,,_. _ _

c. Oversight 0.20 0.60 0.40 0.20 _ _ _ _

d. Crew Fee & ack 0.20 0.60 0.40 0.20 _._ __. _ _

8. Technical Specifications 1 2 3

a. Recognize 0.40 1.20 0.80 0.40 _ _ _ _

b. Locate 0.20 0.60 0.40 0.20 _ _ _ _

c. Compliance 0.40 1.20 0.80 0.40 _ _,,_, _ _

NUREG-1021 13 of 28 Interim Rev. 3, January 1997

I I ES-303 Operating Test Comments Form ES-303-2 Ol Applicant Docket Number: 55- Page of j Form ES-303-1 Comments Cross Reference l l l l l O 1 l l NUREG-1021 14 of 28 Interim Rev. 8, January 1997

O ES-303 R0 Competency Grading Worksheet for Integrated Plant Operations Form ES-303-3 l

1. UNDERSTAND AND INTERPRET ANNUNCIATORS AND ALARM SIGNALS DID THE APPLICANT:

(a) NOTICE and ACKNOWLEDGE alarms? l 3 2 1 Consistent and Minor difficulties Failed to notice and timely or lapses in awareness acknowledge important acknowledgement or response alarms; distracted by nuisance alarms; etc. [.30 X -

                                                                                                         )

(b) Correctly INTERPRET and VERIFY that annunciators and alarm signals were consistent with plant anu system conditions (including the use of alarm response procedures (ARPs), when necessary)? 3 2 1 Consistent and Minor inaccuracies Significant inaccur-efficient in interpreting acies resulted in interpretation and verifying signals plant degradation; and verification poor use of ARPs [.40 X -

                                                                                                         ]

(c) ATTEND to ANNUNCIATORS and ALARM SIGNALS in order of importance and i severity?  ! 3 2 1 Consistent Minor inaccuracies Did not prioritize attention in and oversights attention to signals; all cases inattentive to important alarms [.30 X =

                                                                                                         ] ;

NUREG-1021 15 of 28 Interim Rev. 8, January 1997

ES-303 2 Form ES-303-3

2. DIAGNOSE EVENTS AND CONDITIONS BASED ON SIGNALS AND READINGS DID THE APPLICANT:

(a) RECOGNIZE off-normal trends and status? i 3 2 1 Quick and accurate Some delays in Serious omissions, recognition recognizing off- delays, or normal conditions inaccuracies in recognizing events [.40 X =

                                                                                ]

l (b) Correctly USE REFERENCE MATERIAL (prints, books, charts) to aid in diagnosing and classifying events and conditions? l 3 2 1 Correctly used Minor errors in Did not use or references, using or relying incorrectly used when necessary on references references to diagnose events [.20 X =

                                                                                ]

(c) Correctly DIAGNOSE plant conditions based on control room indications? 3 2 1 Diagnoses were Minor errors or Faulty diagnoses accurate difficulties adversely affected in diagnoses plant status [.40 X =

                                                                                ]

NUREG-1021 16 of 28 Interim Rev. 8, January 1997

l l l ES-303 3 Form ES-303-3

3. UNDERSTAND PLANT AND SYSTEM RESPONSE DID THE APPLICANT.

(a) LOCATE and correctly INTERPRET relevant instruments and other indicators of plant and system response (s)? 3 2 1 Accurate and Minor errors in Serious omissions efficient location locating and inter- l or inaccuracies i and interpretation preting instruments in interpreting  ! of instruments and displays instruments  ! [.33 X =

                                                                                  ] l l

(b) Demonstrate KNOWLEDGE of SYSTEM OPERATION, including set points, interlocks, and automatic actions? 3 2 1 l O Demonstrated thorough under-standing of Minor instances of errors caused by in-adequate knowledge Inadequate knowledge resulted in plant degradation I system operations [.33 X =

                                                                                  ]

(c) Demonstrate an understanding of how his or her ACTIONS (or inaction) AFFECT PLANT and SYSTEM CONDITIONS?

.        3                               2                            1 Understood the               Minor misunder-              Appeared to act effect of actions            standing of effect           without knowledge on plant and                 of actions on plant          of or regard for        1 systems                      and systems                  effect on plant         i and systems             !

[.33 X =

                                                                                  ]  l 1

NUREG-1021 17 of 28 Interim Rev. 8, January 1997

l l ES-303 4 Form ES-303-3

4. COMPLY WITH AND USE PROCEDURES AND TECHNICAL SPECIFICATIONS DID THE APPLICANT:

(a) REFER T0 the appropriate procedure in a timely manner? j 3 2 1

                                                                                     )

1 Quickly located Minor difficulties and Problems and failures appropriate oversights in referring in referring to procedures to appropriate procedures in procedures important instances [.20 X =

                                                                                   ] ;

(b) RECOGNIZE E0P ENTRY CONDITIONS and carry out appropriate immediate l actions without the aid of references or other forms of assistance? I l 3 2 1 Consistent, accurate Minor lapses or errcrs, Did not accurately and timely but actions generally execute actions recognition appropriate [.40 X =

                                                                                   ] i (c)    COMPLY WITH procedures (including precautions and limitations) in an accurate and timely manner?

3 2 1 Accurate and Few errors; corrections Many significant timely compliance made in sufficient time errors; excessive to avoid adverse effect assistance required [.20 X =

                                                                                   ]

(d) RECOGNIZE plant conditions that are addressed in technical specifications? 3 2 1 Recognized and Minor assistance Did not recognize complied with required to recognize conditions and/or LCOs and action conditions and/or comply with LCOs and statements comply with LCOs and action statements action statements [.20 X =

                                                                                   ]

! NUREG-1021 18 of 28 Interim Rev. 8, January 1907

i l l l\ -

     'ES-303                                        5                       Form ES-303-3

5. OPERATE THE CONTROL BOARD l DID THE APPLICANT:

^ (a) LOCATE CONTROLS efficiently and accurately? l l 3 2 1 Promptly located Some minor hesitancy Unable to locate appropriate controls and difficulty in controls without in all instances locating controls assistance [.25 X -

                                                                                        ]

1 (b) MANIPULATE CONTROLS in an accurate and timely manner? 3 2 1 Control manipulations Minor shortcomings, Improper manipul-were consistently but efficiently ations resulted in accurate and mitigated any major system timely resulting consequences perturbations [.25X -

                                                                                        ]

(c) ACT appropriately in response to INSTRUMENT READINGS? 3 2 1 Responses were Generally adequate Failed to react appropriate and response; some appropriately to timely errors and lapses instrument readings without assistance 1 [.25 X -

                                                                                        ]

(d) Take MANUAL CONTROL of automatic functions when appropriate? 3 2 1 Took manual control Minor delays and some Depended on automatic  ! when appropriate prompting necessary actions; had to be before overriding prompted to take automatic functions manual control [.25 X -

                                                                                        ]

r

 \

NUREG-1021 19 of 28 Interim Rev. 8, January 1997

ES-303 6 Form ES-303-3

6. COMMUNICATE AND INTERACT WITH OTHER CREW MEMBERS DID THE APPLICANT:

l (a) PROVIDE clear and accurate INFORMATION on system status to others for I the performance of their jobs? 3 2 1 Provided others Minor instances of Failure to accurately with accurate and needing to be provide important pertinent information prompted for input; information to others some incomplete and jeopardized plant inaccurate information status [.33 X -

                                                                                 )

(b) Effectively RECEIVE INFORMATION from others (including requesting, l acknowledging, and attending to information)? l 3 2 1 Responded and Minor instances Inattentive to reacted appropriately of failure to information 1 to information acknowledge or provided by others from others respond to information j [.33 X -

                                                                                 ] 1 l

(c) CARRY OUT the INSTRUCTIONS of the supervisor successfully? . 3 2 1 Ably carried out Minor hesitancy and Failed to promptly all supervisory difficulty in following and accurately follow  ! objectives; dis- orders, but ultimately directions; blindly l cussed instructions complied successfully complied with when questionable erroneous orders [.33 X -

                                                                                 ]

l l

ES-303 SR0 Competency Grading Worksheets Form ES-303-4 for Integrated Plant Operations

1. UNDERSTAND AND INTERPRET ANNUNCIATORS AND ALARM SIGNALS

 .DID THE APPLICANT:
  -(a)    NOTICE and ATTEND to annunciator and alarm signals in order of their importance and severity?

3 2 1 Responded accurately Minor difficulties failed.to attend to and efficiently in in attending to signals or prioritize import-all instances or prioritizing attention ant alarms; responded slowly; distracted by nuisance alarms (.30 X -

                                                                                      ]

(b) Correctly INTERPRET the meaning and significance of alarms and annunciators (including the use of alam response procedures (ARPs), when necessary)? 3 2 1 O Understood and quickly determined Minor inaccuracies or delays in Misinterpretations, delays, or misuse of what failures alarms interpreting alarms ARPs resulted in were indicating plant degradation [.35 X =

                                                                                      ) j (c)    VERIFY that annunciator and alarm signals were consistent with plant and system conditions?

3 2 1 1 i Ensured proper Minor lapses in Failed to correctly l verification when alarm verification, verify signals on l necessary but no inappropriate important occasions; actions were taken as did not notice incon-a result of inadequate sistencies between verification alarms and plant conditions i [.35 X =

                                                                                      ] ;

i NUREG-1021 21 of 28 Interim Rev. 8, January 1997

l l 1 ES-303 2 Form ES-303-4

2. DIAGNOSE EVENTS AND CONDITIONS BASED ON SIGNALS AND READINGS DID THE APPLICANT:

(a) RECOGNIZE off-normal trends and status? 3 2 1 Quick and accurate Some delays in Serious omissions, recognition recognizing off- delays, or normal conditions inaccuracies in recognizing trends [.25 X =

                                                                                    )

(b) Ensure the collection of CORRECT, ACCURATE, and COMPLETE infremation and reference material on which to base diagnoses? 3 2 1 Ensured that all Minor instances of over- Serious instances of relevant indications looking, overrelying on, misusing or failing and references were or misinterpreting indic- to use important checked ations and/or references information or data [.25 X =

                                                                                    ]

(c) Correctly DIAGNOSE plant conditions based on control room indications? 3 2 1 Diagnoses of plant Minor errors or Faulty diagnoses conditions were difficulties in adversely affected accurate diagnosing conditions plant status [.25 X =

                                                                                    ]

(d) Ensure that CORRECT and TIMELY DIAGNOSTIC ACTIVITIES were carried out by the CREW 7 3 2 1 Ensured effective Minor errors or Faulty diagnostic diagnostic activities difficulties in activities by crew and diagnoses by crew diagnosing by crew adversely affected l plant status ' [.25 X =

                                                                                    ]

NUREG-1021 22 of 28 Interim Rev. 8, January 1997

4 ('M ES-303 3 Form ES-303-4

3. UNDERSTAND PLANT AND SYSTEM RESPONSE DID THE APPLICANT

(a) INTERPRET control room indicators correctly and efficiently to ascertain and verify the status and operation of plant systems? 3 2 1 Accurate and efficient Minor errors in Serious omissions, interpretation of interpreting delays, or inaccur-instruments and instruments and acies in interpreting displays displays instruments and displays [.35 X -

                                                                                               ]

(b) Remain ATTENTIVE to control room indications? 3 2 1 ' Regularly scanned Sporadically scanned Rarely scanned indi-indications; antici- indications; minor cations; failed to

  /       pated changes in plant       lapses in anticipating            anticipate predict-(       conditions due to events in progress predictable changes               able changes in plant status

[.20 X -

                                                                                               ]

(c) Demonstrate, through directives and actions, a thorough UNDERSTANDING of how the PLANT, SYSTEMS, and COMPONENTS OPERATE AND INTERACT (including set points, interlocks, and tutomatic actions)? 3 2 1 Demonstrated thorough Minor errors because of Inadequate knowledge understanding of how gaps in knowledge of of system and com-systems and components how systems and ponent operation operate and interact components operate resulted in serious mistakes or plant degradation [.45 X -

                                                                                               ]

NUREG-1021 23 of 28 Interim Rev. 8, January 1997

ES-303 4 Form ES-303-4

4. COMPLIANCE WITH AND USE OF PROCEDURES DID THE APPLICANT:

(a) REFER to correct procedures and procedural steps when appropriate? l 3 2 1 Requested or readily Minor lapses in Failed to correctly located all appro- referring to or refer to procedures priate procedures locating appropriate in important as necessary procedures instances , [.25 X =

                                                                                   ]

l (b) USE PROCEDURES CORRECTLY, including following procedural steps in 1 correct sequence, abiding by procedural cautions and limitations, selecting correct paths on decisions blocks, and correctly transitioning between procedures? 3 2 1 9 i Accurately and Minor errors, but Significant errors promptly executed made necessary impeded or slowed , procedural steps corrections in recovery or degraded a timely fashion plant unnecessarily [.50 X -

                                                                                   ]

(c) Ensure the safe, efficient IMPLEMENTATION of procedures BY THE CREW 7 3 2 1 Kept crew informed of Crew occasionally had Read procedures to procedural status; got to question SR0 him/herself; failed acknowledgment from regarding status; to coordinate or crew when reading allowed lapses in verify crew's use of procedures implementation by crew procedures  ; [.25 X =

                                                                                   ]

NUREG-1021 24 of 28 Interim Rev. 8, January 1997

ES-303 5 Form ES-303-4

5. OPERATE THE CONTROL BOARDS I DID THE APPLICANT:

(a) LOCATE CONTROLS efficiently and accurately? j 3 2 1 Promptly located Some minor hesitancy or Unable to locate appropriate controls difficulty in locating controls without in all instances controls assistance [.25 X =

                                                                                                   ]

(b) MANIPULATE CONTROLS in an accurate and timely manner? 3 2 1 Manipulations were Minor shortcomings, Improper manipu-consistently accurate but any resulting lations caused major i and timely consequences were system perturbations i readily mitigated [.25 X =

                                                                                                   ]

(c) ACT appropriately in response to INSTRUMENT READINGS? l l' 3 2 1 Responses were Generally responsive, Failed to react appropriate and but some minor appropriately to i timely errors and lapses instrument readings without assistance 1 [.25 X =

                                                                                                   ]

(d) Take MANUAL CONTROL of automatic functions when appropriate? 3 2 1 Took manual control Minor delays; some Depended on automatic as appropriate prompting necessary actions; required before overriding prompting to take automatic functions manual control [.25 X =

                                                                                                   ]

v NUREG-1021 25 of 28 Interim Rev. 8, January 1997

l ES-303 6 Form ES-303-4

6. COMMUNICATE AND INTERACT WITH THE CREW AND OTHER PERSONNEL DID THE APPLICANT:

(a) Communicate in a clear, easily-understood manner? 3 2 1 Communications were At times, communi- Communications were timely, clear, and cations were ill-timed, vague easy to hear and confusing, hard to or difficult to understand hear or understand hear or understand [.45 X =

                                                                                                                             ]

(b) Keep crew members and those outside the control room informed of plant status? 3 2 1 Provided others with Had to be prompted for Failed to provide accurate, pertinent information in some minor important information information throughout instances; gave some scenarios incomplete or inaccurate information [.35 X -

                                                                                                                             ]

(c) ENSURE RECEIPT of clear, easily-understood communications from crew and others? 3 2 1 Requested information Failed to require Failed to request or clarification when or acknowledge infor- needed information; necessary; understood mation from others inattentive when communications from information was others provided; failed to correct serious misunderstandings among crew members [.20 X =

                                                                                                                             ]

NUREG-1021 26 of 28 Interim Rev. 8, January 1997 O'

O

• ES-303 7 Form ES-303-4

7. DIRECT SHIFT OPERATIONS DID THE APPLICANT:

(a) Take TIMELY and DECISIVE ACTION when problems arose? 3 2 1 Took early remedial Minor instances of Failed to take timely j action when necessary failure to take action action; resulted in j within a reasonable deterioration of period of time plant conditions j [.20 X =

                                                                                           ]

i' (b) Provide TIMELY, WELL THOUGHT OUT DIRECTIONS that facilitated CREW PERFORMANCE and demonstrated appropriate CONCERN for the SAFETY of the plant, staff, and public? 3 2 1 Directives enabled Minor instances of Directives inhibited safe, integrated incorrect, trivial, safe performance-crew performance or difficult-to-carry- crew had to explain out orders why orders could not or should not be followed [.40 X -

                                                                                           ]

(c) Stay in a position of OVERSIGHT and provide an APPROPRIATE AMOUNT of DIRECTION and GUIDANCE? 3 2 1 Stayed involved but Crew occasionally had to Lost the big picture; not intrusive; reguest assistance, crew had to repeat-anticipated crew's which interfered edly request or needs and >rovided with their ability to guidance wien carry out actions provide failed toguidance; verify that necessary directives were correctly implemented [.20 X -

                                                                                           ]

(d) SOLICIT and INCORPORATE FEEDBACK from the crew to foster an effective, team-oriented approach to problem solving and decision making? 3 2 1 Involved crew in At times, failed to Made decisions with-problem-solving process involve crew in decision out crew participa-as appropriate, leading making when it would tion or consultation; to effective team have been appropriate, crew divisiveness was decision making detracting from team- counter productive oriented approach [.20 X =

                                                                                           ]

O NUREG-1021 27 of 28 Interim Rev. 8, January 1997

ES-303 8 Form ES-303-4

8. COMPLY WITH AND USE TECHNICAL SPECIFICATIONS DID THE APPLICANT:

(a) RECOGNIZE when conditions were covered by technical specifications (TS)? 3 2 1 Recognized TS Minor errors and Failed to correctly limiting conditions for misunderstandings recognize situations operation and action with respect to covered by TS and statements without use TS applications action statements of references [.40 X =

                                                                               ]

(b) LOCATE the appropriate TS quickly and efficiently? 3 2 1 Located applicable Had difficulty locating Could not locate TS quickly and TS; had to search appropriate TS accurately through index and body of document [.20 X =

                                                                               ] -

(c) Ensure correct COMPLIANCE with TS and LCO action statements? 3 2 1 Directives were based Needed some assistance Applied incorrect TS on correct under- from crew to ensure to situation; standing of TS compliance allowed crew to action statements violate TS [.40 X =

                                                                               ]

NUREG-1021 28 of 28 Interim Rev. 8, January 1997

O Q ES-401 PREPARING INITIAL SITE-SPECIFIC WRITTEN EXAMINATIONS A. PURPOSE This standard specifies the requirements, procedures, and guidelines for preparing site-specific written examinations for the initial licensing of reactor operator (RO) and senior reactor operator (SRO) applicants at power reactor facilities. B. BACKGROUND f The content of the written licensing examinations for R0s and SR0s is dictated by 10 CFR 55.41 and 55.43, respectively. Each examination shall contain a representative selection of questions concerning the knowledge, skills, and abilities (K/As) needed to perform duties at the desired license level. The written operator licensing examination is administered in two sections, including a generic fundamentals examination (GFE) and a site-specific examination. The GFE covers those K/As that do not vary significantly among reactors of the same type (i.e., pressurized or boiling water) and is generally administered early in the license training process (refer to ES-205 for a description of the program). The instructions in this standard apply only to the site-specific examination. Except as noted in Section D.I.b, the " Knowledge and Abilities Catalog [s] for Nuclear Power Plant Operators: Pressurized [and Boiling] Water Reactors," NUREG-1122 and -1123, respectively, provide the basis for developing content-valid licensing examinations. Each K/A stem statement has been linked to the applicable item number in 10 CFR 55.41 and/or 55.43. Preparing the license examination using the appropriate K/A catalog, in conjunction with the instructions in this NUREG, will ensure that the examination includes a representative sample of the items specified in the regulations. C. RESPONSIBILITIES

1. Facility Licensee The facility licensee is responsible for the following activities, as applicable, depending upon the examination arrangements confirmed with the NRC regional office (in accordance with ES-201) approximately four months before the scheduled examination date:

a. Prepare the proposed examination outline (s) in accordance with Section D.1, and submit the outline (s) to the NRC regional office for review and approval in accordance with ES-201.

b. Submit the reference materials necessary for the NRC regional office to prepare and/or validate the requested examination (s)

(refer to ES-201, Attachment 2). O 1 NUREG-1021 1 of 39 Interim Rev. 8, January 1997 1

i l l l ES-401

c. Prepare the proposed examination (s) in accordance with Sections D.2 through D.4, review the examination (s) in accordance with Section E, and submit the examination (s) to the NRC regional office in accordance with ES-201.

I

d. Meet with the NRC in the regional office or at the facility, when and as necessary, to review the proposed examination (s) and discuss potential changes (refer to ES-201).

e. Revise the proposed examination outline (s) and examination (s) as agreed upon with the NRC regional office; however, the NRC retains final authority to approve the examination.

2. NRC Reaional Office The NRC regional office is responsible for the following activities:

a. Ensure that the examinations are prepared in accordance with Section D.

b. Ensure that the examinations are reviewed for quality in accordance with Section E.

c. Meet with the facility licensee, when and as appropriate, to i prereview the examination (s) in accordance with ES-201.

D. EXAMINATION PREPARATION I

1. Develoo the Outline 1

Develop each written examination outline in accordance with the l l following general instructions: '

a. Select the appropriate examination outline model for the licensing examination being developed (i.e., R0 or SRO, BWR or PWR) from i Forms ES-401-1 through ES-401-4; Form ES-401-5, " Generic Knowledge i and Abilities Outline," applies to all examinations.

b. Identify plant-specific, high-priority items (e.g., operating events or problems, PRA-identified risk-important systems and operator actions , and recent technological developments) that l ' Chapter 13 of NUREG-1560, " Individual Plant Examination Program:

l Perspectives on Reactor Safety and Plant Performance," identifies a number of l important human actions that may be appropriate for examination. In determinis,i; i:::portant operator actions, do not overlook actions that are relied upon or result in specific events being driven to low risk contribution. This will help identify those human actions, assumed to be very reliable, that migit otherwise not show up in a list of risk-dominant actions. NUREG-1021 2 of 39 Interim Rev. 8, January 1997 l

i I' s k 1 i ES-401 f

should be tested in the written examination. Enter the applicable i

information (K/A number, topic, and importance rating) in the i

, appropriate tier (generic, plant systems, and emergency / abnormal plant evolutions) of the examination outline. If a plant-specific 1 topic does not fit any of the listed systems or evolutions, enter i it in a blank space provided at the bottom of the outline.  ! The facility licensee's site-specific K/A list may be used to - j supplement or override, on a case-by-case basis, selected i individual items in the NRC's K/A catalogs. In order to maintain examination consistency, the site-specific K/A list shall not be used in place of the entire K/A catalog. ,

c. Select additional K/As to complete each of the three tiers of the  !

examination outline. Select enough K/As from which to-generate  : 100, one-point, multiple choice questions distributed among the  ; three tiers as specified for the applicable outline. Select topics from as many different systems and evolutions as possible, and distribute the topics among the K/A categories (including the generic K/As, as applicable to the system), with at least one i topic from each category in every tier of the outline. Avoid  ! selecting more than two or three topics from a given system or  ; evolution unless they are related to a plant-specific priority. Enter the K/A numbers, a brief description of each topic, the O topics' importance ratings for the license level of the exam, and

                            -the point totals (system, category, group, and~ tier) on the outline.                                                                                              j i

i Pay particular attention to the importance of the selected K/As.  ! (Those below 2.5 should be justified on the basis of plant- , specific priorities.) Also note whether the selected K/As will  ; differentiate between competent and less-than-competent  ! applicants, whether they are appropriate for the job level being examined, and whether their operational orientation is suitable. Refer to Appendix B for additional discussion of these principles. Ensure that the outline used during successive audit and licensing examinations does not become repetitive and predictable. If a facility licensee proposes to use an outline that was previously used at another facility, it shall identify the source of the outline and explain what effect its reuse is expected to have on examination integrity,

e. After completing the outline, check the selected K/As for balance of coverage within and across the three tiers. Also check the overall balance of the entire licensing examination, including the walk-through and the dynamic simulator test, and make any necessary adjustments.

O NUREG-1021 3 of 39 Interim Rev. 8, January 1997 l l l l

i ES-401

f. Review and submit the completed outline to the NRC chief examiner for review and approval in accordance with ES-201. Facility-developed outlines shall be independently reviewed and approved by an authorized facility representative before being submitted to the NRC regional office. The NRC must receive the outlines by the date agreed upon when the examination arrangements were confirmed (normally at least 60 days before the scheduled examination date).

g. The NRC chief examiner will review the outline within five working days and provide comments and recommended changes, as appropriate.

2. Select and Develoo Ouestions

a. Prepare the site-specific written operator licensing examination using a combination of existing, modified, and new questions in accordance with the previously approved examination outline (refer to Item D.1 and ES-201) and the criteria summarized below.

If it becomes necessary to deviate from the previously approved examination outline, discuss the proposed deviations with the NRC chief examiner and obtain concurrence. Be prepared to explain why the original proposal could not be implemented and why the proposed replacement is considered an acceptable substitute.

b. Take care to ensure that each question is technically accurate and free of the following psychometric flaws that could diminish the validity of the examination:

low level of knowledge low operational validity low discriminatory validity implausible distractors confusing or ambiguous language confusing or inappropriate negatives collection of true/falso statements backward logic Appendix B provides a detailed discussion and examples of questions containing each of these errors. Appendices A and B contain more detailed instructions and guidelines for preparing and formatting content-valid examinations and should be referred to as necessary while preparing the examination.

c. Establish a level of difficulty that will enable an applicant who is capable of safely operating the plant to complete and review the examination within four hours and achieve a grade of 80 percent or greater. In order to maintain examination quality and consistency, at least 50 percent of the questions on the NUREG-1021 4 of 39 Interim Rev. 8, January 1997

ES-401 examination shall be written at the comprehension / analysis level. The cognitive level of any question drawn directly from a bank will be counted at its face value.

d. When both R0 and SR0 examinations are to be given at the same time, duplicate no more than 75 percent of the RO examination questions on the SRO examination. The remaining SR0 questions shall evaluate the additional knowledge and abilities required for the higher license level. 10 CFR 55.43(b) provides guidance on the additional topics to be considered for the SR0 examination.

All test questions shall be in the multiple choice format

                            ~

e. described in Appendix B, and each question shall be worth one point.

f. To avoid compromising the integrity and security'of the examination and to enhance consistency, observe the following l limits on question repetition and bank use when preparing the j examination:

Repeat no more than 25 percent of the questions on the examination from examinations, quizzes, or tests administered to the license applicants during their license training class or from the past two licensing examinations O at the facility. The facility test / quiz limit does not apply to NRC-developed examinations because those materials are generally not available to NRC examiners. Facility-written examinations shall repeat no questions directly from the applicants' audit examination or similar testing vehicle given at or near the end of the license training class, unless the two examinations are written independently (i.e., no interface between the examination authors). In such cases, up to five questions may be duplicated, and the facility licensee shall identify the duplicates. 4 Take no more than 50 percent of the questions for the  ! examination directly from the facility licensee's written i examination question bank without significant modification. Questions that the facility licensee has obtained from another bank and deposited in its own bank may be treated as

                    " bank" questions provided they have an equal chance of being        j selected for use on the examination.                                  '

Write at least 10 new questions at the comprehension and  ! analysis level, as described in Appendix B. Questions from another bank may be treated as new items if they have not been made available.for review and study by the license applicants and there is no basis (e.g., historical precedent NUREG-1021 5 of 39 Interim Rev. 8, January 1997 7e+

ES-401 or reciprocal arrangements with the other facility licensee) for the applicants to predict their use on the examination. Select the remaining questions for the examination from the facility licensee's bank, but significantly modify each question by changing the conditions in the stem and at least one distractor. The intent or objective of the question does not necessarily have to be changed.

g. Every question must be validated using the facility licensee's reference material provided in accordance with ES-201. A technical reference, including the learning objective used in the applicants' training program, and a cross-reference to the facility licensee's examination question bank, if applicable, shall be noted for every question. Refer to ES-201 for additional instructions regarding the documentation of the source of questions on facility-written examinations.

To facilitate the review process, examination authors should consider providing a brief explanation of why the answer is correct, and each of the distractors is plausible but incorrect. This optional practice increases the efficiency of the examination review process and promotes the detection and correction of problem questions before the examinations are administered. Reference materials such as diagrams, sketches, and portions of facility procedures may be used on a selective basis as attachments to the written examination. Ensure that any reference material used in the examination is easy to read and clearly marked, provides an effective and objective way for the applicant to demonstrate knowledge of the topic or concept, and does not give away the answers to o Ger questions on the examination.

3. Review and Submit the Examination

a. Review the entire examination to ensure that the criteria on Form ES-401-6, " Written Examination Quality Assurance [QA] Checklist,"

are satisfied.

b. Forward the examination package, including all proposed attachments, and the completed QA checklist to the first reviewer.

Section E provides instructions for conducting the QA reviews. Facility-developed examinations must be reviewed and approved by a designated facility representative before they are sent to the NRC regional office. Facility authors shall submit their examinations for management review in time to support their delivery to the NRC regional office at least 30 days before the scheduled examination date. NUREG-1021 6 of 39 Interim Rev. 8, January 1997

ES-401 G NRC examiners shall submit their examinations to the chief examiner for review at least one week before the scheduled prereview by the facility licensee (refer to ES-201).

4. Assemble the Examinations

a. Format the examinations using the one-question-per-page layout specified in Appendix B or by placing as many complete questions as possible on each page. .

i

b. Use a cover sheet in the format shown in Form ES-401-7 for all R0 l and SR0 written examinations. Fill out all items in the upper i section of the cover sheet, except the name of the applicant, when l preparing the examinations.

E. QUALITY ASSURANCE REVIEWS When reviewing questions, reviewers should try to put themselves in the position of the applicants by attempting to answer the questions without using reference material or referring to the answer key. Reviewers should ensure that the conditions and requirements posed in the question are complete and unambiguous, all necessary information is provided, all unnecessary  ; information is deleted, the intended answer clearly follows from what is asked i in the question, and the distractors are plausible. V 1. Facility Manaaement Review If the examination was prepared by the facility licensee, it shall be independently reviewed and approved by an authorized facility representative (refer to ES-201) before it is submitted to the NRC regional office for review and approval. The reviewer should evaluate the examination using the criteria on Form ES-401-6 and include the signed form in the examination package submitted to the NRC.

2. NRC Examiner Review

a. The NRC chief examiner shall independently review all examination questions for content, vording, operational validity, and level of difficulty. As a minimum, the chief examiner shall check the items listed on Form ES-401-6. If the chief examiner wrote the examination, another NRC examiner must perform the independent review.

b. The chief examiner should review the examination as soon as possible after receipt so that supervisory approval can be obtained before the final review with the facility licensee, which is normally scheduled about two weeks before the examination date.

It is especially important that the chief examiner promptly review examinations prepared by a facility licensee because of the extra time that may be required if extensive changes are necessary. The A) (V NUREG-1021 7 of 39 Interim Rev. 8, January 1997

i I l l ES-401 chief examiner shall consolidate the comments from other regional reviewers and submit one set of comments to the author. l

c. If the facility licensee develo?ed the examination, the licensee is primarily responsible for enfJring its technical accuracy and complying with the restrictions on question duplication and examination bank use (i.e., shaded items 1, 4, 5, and 6 on the QA l checklist) . However, the chief examiner is expected to use his or l her best judgment and take reasonable measures, including the selective review of reference materials, individual questions, and  !

past examinations, to verify these items when reviewing the examination; exclusive reliance on the facility author's and reviewer's initials is not adequate. Depending upon the expected i quality of the examination and the time available before the j scheduled review with the facility licensee, the chief or another examiner shall independently review and verify the technical accuracy of a sample of the written examination questions.

d. Before reviewing the examinations with the author or facility contact, the chief examiner will note any necessary changes and forward the examinations to the responsible supervisor (or a designated alternate other than the chief examiner) for review and comment in accordance with Section E.3. There are no minimum or maximum limits on the number or scope of changes the chief 1 examiner may direct the author or facility contact to make to the i proposed examinations, provided that they are necessary to make the examinations conform with established acceptance criteria.

Refer to ES-201 for additional guidance regarding the NRC response . to facility-developed examinations that are significantly I deficient. I

e. Upon supervisory approval, generally at least 14 days before the i examinations are scheduled to be given, the chief examiner will review the written examinations with the facility licensee in accordance with ES-201.

Examinations that are written by the NRC shall be clean, properly formatted, and " ready-to-give" before they are reviewed with the facility licensee. The region shall not rely on the facility licensee to ensure that the quality of the examination is acceptable for administration.

f. After reviewing the examination with the facility licensee, the chief examiner will ensure that any comments and recommendations are resolved and the examination is revised as necessary. If the facility licensee developed the examination, it will generally be expected to make whatever changes are recommended by the NRC.

g. After the necessary changes have been made and the chief examiner is satisfied with the examination, he or she will sign the QA NUREG-1021 8 of 39 Interim Rev. 8, January 1997

/ ES-401 checklist and forward the examination package to the responsible supervisor for final approval. If the examination was written by the facility licensee, the chief examiner should include a copy of the original submittal with the examination package.

3. NRC Supervisory Review

a. The responsible supervisor (or a designated alternate other than the NRC author or chief examiner) shall review and approve the examination and QA checklist before the facility prereview per ES-201. The supervisory review is not intended to be another technical review, but rather a general assessment of examination quality, including a review of the changes being recommended by the chief examiner, and a check to ensure that all the applicable administrative requirements have been implemented.

b. The responsible supervisor should ensure that any significant deficiencies in the original examinations submitted by a facility licensee are evaluated in accordance with ES-201 to determine the appropriate course of action. At a minimum, the supervisor should ensure that they are addressed in the final examination report in accordance with ES-501.

c. Following the facility review, the responsible supervisor should s) again review the examination after to ensure that the concerns j expressed by the facility licensee and the NRC have been appropriately addressed. The supervisor shall not sign Form ES-401-6 until he or she is satisfied that the examination is acceptable to be administered.

4. Peer Review As a final check of the examination's technical accuracy, facility management should consider administering the NRC-approved examination (under security agreements) to one or more licensed personnel who were previously uninvolved in developing the

   ,             examination. Any comments made and problems identified during the trial administration shall be discussed with the NRC chief examiner and resolved before the examination is administered to the license applicants.

F. ATTACHMENT ~i/ FORMS Form ES-401-1, "BWR SR0 Examination Outline" Form ES-401-2, "BWR R0 Examination Outline" Form ES-401-3, "PWR SR0 Examination Outline" Form ES-401-4, "PWR R0 Examination Outline" Form ES-401-5, " Generic Knowledge and Abilities Outline" Form ES-401-6, " Written Examination Quality Assurance Checklist" Form ES-401-7, " Site-Specific Written Examination Cover Sheet" O NUREG-1021 9 of 39 Interim Rev. 8, January 1997

i l i l l ES-401 BWR SR0 Examination Outline Form ES-401-1 Facility: Date of Exam: Exam Level: K/A Category Points Tier Group Point l K K K K K K A A A A G Total  ! 1 2 3 4 5 6 1 2 3 4

l. 1  ;

                                                        .~

i r;

                                                                . .' h '.     [j             [$,          26
                                                                                             ,n Emergency &
                                                                              . , ~ -..
                                                                                   ~'     '"

Abnormal 2 (i, 17

                                          , c >.
                                                              '3,g            ;;;         gy" Plant                               sw         , , . - ,,                                -

Evolutions Tier <f ;p gf 3- "1 , :fg! 43 i Total s . 4,, , S: . 5:'

                                                                              ~,.
                                                                                      !:  W' u, ,         t,,     's :,-.          e          , :< ,a :-
                                             ".           ~   w. 51            ,A,'        s     s 1                                                                                  23 2.

2 13 Plant Systems 3 4 Tier 40 Totals

3. Generic Knowledge and Cat 1 Cat 2 Cat 3 Cat 4 Abilities 17 Note: -

Attempt to distribute topics among all K/A categories; select at least one topic from every K/A category within each tier. Actual point totals must match those specified in the table. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities. Systems / evolutions within each group are identified on the associated outline. The shaded areas are not applicable to the category / tier. NUREG-1021 10 of 39 Interim Rev. 8, January 1997

_m.. - . . _ . _ . . . _ . . . - . . . . . . _ . . . .m.. . ._. _ .._ . . . . . . _ . . . _ - _ .. . .._

     %                                                                                               N                                                                     %

ES-401 BWR SRO Examination Outline Form ES-401-1 Emeroency and Abnormal Plant Evolutions - Tier 1/ Group 1 E/ APE # / Name / Safety Function K K K A A G K/A Topic (s) Imp. Points 1 2 3 1 2 295003 Pa. Mat or Complete loss of AC Pwe / VI 295006 SCR AM 18 . 295007 Hiah Reactor Dressure f til 295009 Low Reactor _ Water Levet /11 295010 Hiah Drywell Newre / V 295043 Hiah Suporession Pool Temo. I V t 295014 Inadvertent Reectivity Addition / I 295015 incomotete SCR AM / f 295016 Control Room Abandonment / Vit 295017 Hiah Off-site Release Rete /IX 295023 Refuelina Accidents Coolina Mode / Vlft 295024 Hiah Dryweit Pressure / V . 295025 Hiah Reactor Pressure /111 295026 Suppression Pool Hiah Water Temo. I V 295027 Hioh Containment Temperature / V 295030 Low Suppression Pool Water Level / V 295031 Reactor low Water level / Il 295037 SCRAM Condition Present and Power Above APRM Downscele or Unknown /I 295038 Hiah Off-site Release Rete /IX 500000 Hioh Containment Hydronen Conc. I V K/A Cateaory Totals: Group Point Totet: NUREG-1021 11 Of 39 Interim Rev. 8, January 1997

ES-401 BWR SRO Examination Outline Form ES-401 1 Emeroency and Abnormat Plant Evolutions - Tier 1/ Group 2 E/ APE # / Name / Safety Function K K K A A G K/A Topicts) Imp. Points 1 2 3 1 2 295001 Partial or Complete Loss of Forced Core Flow Circulation / I & IV 295002 Loss of Main Condenser Veevum / til 295004 Partiel or Total loss of DC Pwr / VI 295005 Mein Turbine Generator Trip / Itt 295008 Hia5 Reactor Water Level / Il 295011 Hia% Containment Temperature / V 295012 Hiah Dryweft Temperature / V 295018 Partial or Total loss of CCW / Viit 295019 Partial or Total Loss of inst. Air / Vi!! 295020 Inadvertent Cont. Isolation / V & Vit _ 295021 Loss of Shutdown Coolina / IV 295022 Loss of CRD Pumps /I 295028 Hiah Drywelt Temperature / V 295029 Hiah Suppression Pool Water levet / V 295032 High Secondary Containment Area Temperature / V 295033 High Secondary Containment Area Radiation levels / IX 295034 Secondary Containment Ventilation Hiah Radiation / IX 295035 Secondary Containment High Differential Pressure / V 295036 Secondary Containment High Sump / Ares Water Levet / V 600000 Plant Fire On Site / Vtil KIA Category Point Toteis: Group Point Total: NUREG-1021 12 Of 39 Interim Rev. 8, January 1997 O O O

3 O (J (v') V ES-401 BWR SRO Examination Outline Form ES-401-1 Plant Systems - Tier T/Grou > 1 System # / Name K K K K K K A A A A G K/A Topicts) Imp. Points 1 2 3 4 5 6 1 2 3 4 201005 RCIS 202002 Recirculation Flow Control 203000 RHR/LPCl: Injection Mode 206000 HPCI 207000 Isolation (Emera.) Condenser 209001 LPCS 209002 HPCS J11000 SLC 212000 RPS 215004 Source Renae Monitor 215005 APRM / LPRM 216000 Nuclear Boiler instrumentation 217000 RCIC , 218000 ADS 223001 Primary CTMT and Auxiliaries  ! 223002 PCIS/ Nuclear Steam Supolv Shutoff 226001 RHR/LPCI: CTMT Spray Mode 239002 SRVs 241000 Reactor / Turbine Pressure Reaulator 259002 Reactor Water Level Control 261000 SGTS 262001 AC Electrical Distribution 264000 EDGs 290001 Seconderv CTMT K/A Categnry Point Totals: Group Point Total: NUREG-1021 13 Of 39 Interim Rev. 8, January 1997

ES-401 BWR SRO Examination Outline Form ES-401-1 Plant Systems - Tier :'/Grou )2 System # / Name K K s K K K A A A A G K/A Topic (s) Imp. Points 1 2 3 4 5 6 1 2 3 4 201001 CRO Hvdroutic 201002 RMCS 201004 RSCS 201006 RWM 202001 Recirculation _ 204000 RWCU 205000 Shutdown CooFna 214000 RPfS 215002 RBM 215003 IRM 219000 RHR/LPCt: Torus / Pool Coolina Mode 230000 RHR/LPCI: Torus / Pool Sorav Mode 234000 Fuel Handlino Eauipment 239003 MStV Leeksoe Control 245000 Mein Turbine Gen. end Auxiliarise 259001 Reactor Feedwater 262002 UPS ( AC/CO) 263000 DC Electricef Distribution 271000 offaas 272000 Radiation Monitorino ~~ AOO Fire Protection 290003 Control Room HVAC 300000 Instrument Air 400000 Component Coolina Water K/A Catsoory Point Totals: Group Point Total: NUREG-1021 14 Of 39 Interim Rev. 8, January 1997 9 9 9

1 e

       }   1 t

n . 0 io 7 _ 4- P 9 _ f S 9 _ E 1 . m r y p o r _ F i m a . u _ n a J 8 _ v e _

                    )

R e = ( c i m ip o r . T e _ A t I K l: a n I t o

-                                                          T t

n . io P p - u - . o r G G e3 A4 in> lt u uo A3 OrG n/ ' io: t r 9

       }     ae     A2                                         3
  ~        iTni m-                                                f "f         ae                                                  o x

Eme A1 5 Ote 1 Rv SS K6 Rt Wn BP te K5 K4 K3 K2 K1 m is n p a u h n c a e le M e lsa e b C m e o d a n m v r n e e r a i r P a t t e N D e S a t  : r a t s n s

                     /    d     o    n  s  n     n  I ia
                      #     n   c- i e e      o  le      t e   n  lo  h  d     it    s      o m   d   I      o  e  n     a    s    T e     o   a  C   R   o    tit   e     t t

R d C e n V n 1 s y ine lo n r t s e r i o 2 lo r o e o e V o P S r e P t t 0 t c w t c y n v e le in a d n e r 1

    ,J                      o   r    u  e e  e  la    e      o   -

. C T F M R R P R g e G

   ~-                     3   1    0   1  0  0  0   2        t E 1

0 0 0 0 0 0 0 0 a R 0 0 0 0 0 0 0 0 0 C U 4

               -          1   5    3   9  6  8  8   0      A   N S             0   1    3   3  5  6  8   9      /

E 2 2 2 2 2 2 2 2 K

ES-401 BWR R0 Examination Outline Form ES-401-2 Facility: Date of Exam: Exam Level: K/A Category Points Tier Group Point K K K K K K A A A A G Total 1 2 3 4 5 6 1 2 3 4

1. 1 3 13 Emergency & '

2 d 19 Abnormal P1 ant 3 h j lh hk 4 Evolutions - q (y gg Totals k; pg d N$ 1 l l l l 28 I' Plant Systems 3 l l l l 4 Tier l l 51 Totals l l

3. Generic Knowledge and l Cat 1l Cat 2 Cat 3 Cat 4 Abilities 13 Note: -

Attempt to distribute topics among all K/A categories; select at least one topic from every K/A category within each tier. Actual point totals must match those specified in the table. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities. Systems / evolutions within each group are identified on the associated outline. The shaded areas are not applicable to the category / tier. NUREG-1021 16 of 39 Interim Rev. 8, January 1997

2 e . t _ 1 n 0 io 7 O). 4- P 9 9 S E 1 n . r r p y o m r F I a u n a J 8 . v e R i m r . e t n I

                  )

( s c ip o T A

                  /

1 K . p u o r G

              /

1 r ie l: a T t e- o in s T lt n t uo in Oitu o nlo P o iv p 9 tE u 3 a o r _ nt in G f f mal O aP x G 7 E :e.- 1 O T. R e R br A2 WA B M r e A1 y c n e K3 a r p m E K2 K1 r V e / w . I o e n / PI n io 1 1 1 n o t / o d/ C . c o l 1 n a n n n i it i 1 w e u ir l I d l I t no n F T l I le V d V i t le e n o y r o

                             /     v /   A       /   /     v  sk     r t

e t e r e e y 1 e e r e en dy a L r / r L u u PrUr t f a r r u i v u r H S e n s s e s s it M s s s s e no t

                  /      e     e  t a   e   c A     e   e t

a o n r r a R r r ie e

.                   e  G     P   W   P     e C   P   P   W    tl ie      m m    e     r         R   S        r      de      n a          o r

o l le l l e o w n s ia  : N in br I t c t c w t n e w t c o on is n

                  /

u

                           /

a a y e t e y e l Cw n t t 1 M e e r t r lp r e r o Mo o o 2

                  #    T A R   R   D     e v   m D   R    t AD C T   0 E    in    h       h         o h   h     c       h         y 1 P      a R

ia w ia d a c ia io a RM io r A C o n n e CR o G

                  /

E M S H L H I I H H R SP H o e E 5 6 7 9 0 4 5 4 5 1 7A 0 t a R Y 1 0 4-S 0 0 5 9 0 0 5 9 0 0 5 9 0 0 5 9 1 0 5 9 1 0 5 9 1 0 5 9 2 0 5 9 2 0 5 9 3 0 5 9 3e 0v 5o 9b 0 0 0 0 C A

                                                                           /

U N E 2 2 2 2 2 2 2 2 2 2 2A 5 K

ES-401 BWR Ro Emanaistion Outline Form ES-401-2 Emproency end Abnormet Plant Evolutione - Tier 1/ Group 2 E/ APE # / Name / Safety Function K K K A A G K/A Topic (e) Imp. Pointe 1 2 3 1 2 295001 Partial or Complete Loos of Forced Core Flow Circutetion / I & IV 295002 Loss of Mein Condenser Vacuum /111 295003 Partief or Complete Loos of AC Pwr / VI 295004 Partiet or Complete Loos of DC Pwr ! VI 295008 Hiah Reactor Water Levet /11 295011 Hioh CTMT Temperature / V 295012 Hiah Drywell Temooreture / V 295013 Hioh Suppression Pool Temp. / V 295016 Control Room Abendonment / Vtf 295017 Hiah Off-site Release Rota / IX 295018 Partial or Complete Loos of CCW / Vtti 295019 Partial or Complate Lose of inst. Air / Vitt 295020 Inadvertent Cont. leolation / V & Vil 295022 Loos of CRD Pumos / I 295026 Hiah Suppression Pool Water Temp. / V 295027 Hiah Containment Temperature / V 295028 Hiah Drywell Temperature / V 295029 Hiah Suppression Pool Water Levet / V 295030 Low Suppression Pool Water Level / V 295033 High Secondary Containment Area Radiation Levels / IX 295034 Secondary Containment Ventitetion Hiah Radiation / IX 295038 Hiah Off-site Reteese Rete / tX 600000 Plant Fire On Site / Vitt K/A Category Point Totals: Group Point Totet: NUREG-1021 18 of 39 Interim Rev. 8, January .1997 O O O

[ Ir k . 2- s t 1 n _

    ))v 0        io                                           7      _

4- P 9 - f( S E 9 1 m r p y o n r r F I a u n a J 8 v e R i m . r e t n I

                 )

( s icp _ o T A

                 /

3 K p u - o r G

            /

1 r e l: - Ti t a _ e-n o _ i s T _ lt n t uo n . Oitu io _ nlo P o p 9

    ))  iv tE a

nt u o r 3 in G f .. f' me aP o ._ x l 9 Ea G 1 Om r Ro Rnb A2 WA B d

.             n a  A1 v

vr e K3 a r e m E K2 K1 ~ a

-                  n              e r

o A it V c I t n h h . n / g ig u a e i F l m H H in l i n t t y t lo V ia n n _ e o / t e e f a C e n m m S n t n o n n iV e C ie a/ iss

                  /        w d    y     t    t e       o i c

r a nV nt o ev t o Co/e m dt a u c d Ce T A n yr yL N h o ru t

                  /      S     a  c      as r r ae          in  1 n                            o 2 SeV
                  #      f             ds    dt o li e         ne   na          P   0 E        s   u   e/    or cP o

cW y 1 P o f dre r A o e i v et ee o G

                  /

E L R Ht s Sis t Se r g e E 1 1 2 3 2 2r 3o 5n 3e 6A 3/ t a R 0 0 0 0o 0re 0o C U 4- 5 5 5m 5ff 5m N A S 9 9 9e 9i 9u / E 2 2 2T 2D 2S K

2- s t 1 n O-C i 7 4- o 9 P 9 S 1 E n y n p o m r F I a u n a J - 8 v e

        )

R s t c i m ip o r T e A t

        /                                                                            t:    n K                                                                             t s I o

T t n io P p u o r G G 1 A4 e n> iu lt uro A3 OG nN o 9 ir taie A2 3 nT im-as xm Ee Os A1 f 0 2 O O Ry S K6 Rt n We Bf P K5 K4 K3 K2 K1 f r f o t o t u e n h lu r o s e S c e e it i y to s a r R r to e n t ie lp p e t n r d e n f ix r o t n o d e u u e n u S s C m C o M o m u A s l a n C r m e r r e N o t d a e  : w ) s n e P t e s a

        /

lo it a n e l

        #     ic      F n

j c e n r e M I r e T S t r n ib d ele r t a o m tu o I m R l M e e r e t a T e e E P io T u e t - t s d r it ta I: C ( L B C tc T F W n 1 y y o P n / r y u / r r r io 2 S H io e r N o o o P 0 S s L I t S S M e a / s t t c t c S s C S I a C S c y 1 D R M I C ae R H C P lo C P C P C L S P M M R R P tc u I C S D i m r t C V R a e a e a e T G G D r o - O C R R R R H Is L H S R R I S A N R A P P S R R R G E g G e E 1 1 2 5 2 0 0 0 1 2 0 0 3 4 5 0 0 0 1 2 2 0 1 2 0 0 t a R 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C 4- 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 U 1 1 1 2 3 6 7 9 9 1 2 5 5 5 6 7 8 3 3 9 1 9 9 1 4 A N S 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 2 3 4 5 5 6 6 / E 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 K l

h. J V[b \ ES-401 BWR RO Examination Outline Form ES-401-2 Plant Systems - Tier P/Grou > 2 Syow:2. # / Name K K K K K K A A A A G K/A Topicts) Imp. Points 1 2 3 4 5 6 1 2 3 4 201003 Contnl Rod end Drive Mechenlem 201004 RSCS 201006 RWM 202001 Recirculation 204000 RWCU 205000 Shutdown Coofina 214000 RPtS 215002 RBM 219000 RHR/LPCI: Torus / Pool Coolina Mode 226001 RHR!LPCf: CTMT Sorav Mode 230000 RHR/LPCf: Torus / Pool Sprey Mode 239001 Main end Rehest Steam i 245000 Main Turbine Gen. and Auxiliaries 256000 Reactor Condensate 262001 AC Electrical Distribution 262002 UPS (AC/DC) 263000 DC Electrical Distribution 271000 Offoes 272000 Radiation Monitorina 286000 Fire Protection 290001 Seconderv CTMT 290003 Control Room HVAC 300000 instrument Air 400000 Component Coolina Water i K/A Category Point Totals: Grotip Point Total: NUREG-1021 21 Of 39 Interim Rev. 8, January 1997

l , 2- e t 1 n 7 0 9O o i 4- P S 9 E 1 m r

           .                                     y p

F o m r I a u n a J 8 v e

        )

R e l c i m ip o r T e A t

        /

K l: a n I t o T t n io P p u o r G G A4 e3 in> u lt o ur A3 OG / nP o ir 9 t e A2 3 O ai nT f im-ae o xm A1 Ee 2 2 - Os RvS K6 R t Wn Bl a P K5 K4 K3 K2 K1 e u n e o e l C t n e e b e ta o r d lo n m P n m r r a a ip t n e N e r a u o t n -s

        /      o    n   a C        n I c  i   E            o t     la lo    a   e          e    t o

n o a it e s s T m i in a e o C l k e ti t e t t s n lo d n e e t n e V n 1 y ie o a L s e V r o io 2 S r e P H P 0 v V w t n t c y 1 e le le t S d e a a e r r u u o M O G l T F F R P R g e E 1 1 0 0 3 0 0 2 t a R 0 0 0 0 0 0 0 0 C 4- 0 0 0 0 0 0 0 U 5 3 4 9 8 8 0 A N S 1 3 3 3 6 8 9 / E 2 2 2 2 2 2 2 K

ES-401 PWR SR0 Examination Outline Form ES-401-3 Facility: Date of Exam: Exam Level: l K/A Category Points Tier Group { Point I K K K K K K A A A A G 1 Total 1 2 3 4 5 6 1 2 3 4

1. I 24 j Emergency &

2 16 Abnomal

Plant 3 5 3 v u ns Tier  !! -

M 43 Totals $flhl$@kbh[ h fj$ , I)[ 3 l l l l l l 19 17 Plant , Systems 1 l l l l l l 4 Tier 40 Totals

3. Generic Knowledge and l Cat 1l Cat 2 Cat 3l Cat 4 Abilities 17 Note: -

Attempt to distribute topics among all K/A categories; select at least one topic from every K/A category within each tier. Actual point totals must match those specified in the table. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities. Systems / evolutions within each group are identified on the associated outline. The shaded areas are not applicable to the category / tier. NUREG-1021 23 of 39 interimRev.8, January 1997

ES-401 PWR SRO Eram nation Outline Form ES-4013 Emeroency and Abnormat Pfert Evolutions - Tier 1/ Group 1 E/ APE # / Name / Safety Function K K K A A G K/A Topicts) Imp. Points 1 2 3 1 2 000001 Continuous Rod Withdrawal / l 000003 Droceed Control Rod / I 000005 Inocerable/ Stuck Control Rod / I 000011 Larae Break LOCA /lil W/E04 LOCA Outside Containment / Ill WE02 St Termination / I?! 000015/17 RCP Malfunctions / IV BWE09: CE/A13: WE09&E10 Nature! Cire. / IV 000024 Emeroency Boretion / I 000026 Loss of Comoonent Coolina Water / V111 000029 Anticipated Transient w/o Scram / I 000040 (BW/E05: CEE05: W/E12) Steam Une Rupture - Excessive Heat Transfer / IV CE/All: W/E08 RCS Overuoolina - PTS / IV 000051 Loss of Condenser Vacuum / IV 000055 Station Blackout / VI 000057 Loss of Vital AC Elec. Inst. Bus / VI 000059 Accidental Llauid RedWeste Rel. / IX 000062 Loss of Nucteer Service Water / IV 000067 Plant Fire On-site / IX 000068 (BW/A06) Control Room Evac. I Vlti 000069 (W/E14) Loss of CTMT Inteority / V 000074 (W/E06&E07) Inad. Core Coolina / IV BW/E03 Inadeauste Subcoolino Meroin / fV 000076 Hiah Reactor Coolant Activity / IX BW/A02& A03 Loss of NNI-X/Y / VII K/A Category Tntats: Gmun Pnint Total: NUREG-1021 24 Of 39 Interim Rev. 8, January 1997 O O O

                                  - . .   .. -- -. -             . - . .        . - - - . _ - . - _        _   - . . _ - - . - . . - .                . _ _ ~ . . . - . - . .
                                                                                                    )

J J \ ES-401 PWR SRO Examination Outline Form ES-401-3 Eir..h . c e end Abno mal P'ent Evolutions - Tier 1/ Group 2 E/ APE # / Name / Safety Function K K K A A G K/A Topic (s) Imp. Points 1 2 3 1 2 000007 (BWE02&E10: CEE02) Reactor Trip - Stabilization - Recoverv / I BW/A01 Plant Runbeck /I BW/A04 Turbine Trio / IV 000008 Pressurizer Vapor Spece Accident / Iff 000009 Small Break LOCA / !!! BWE08: WE03 LOCA Cooldown - Dooress / IV WE11 Loss of Emeroency Coolent Rectre. / IV 000022 Loss of Reactor Coolent Makeup / It 000025 Loss of RHR Svetem / IV 000027 Pressurizer Pressure Control System Melfunction / It! 000032 Loss of Source Renae NI / Vt1 000033 Loss of Intermediate Renae N1/ Vft 000037 Steem Generator Tube Leak /111 000038 Steam Generater Tube Ruoture / lif 000054 (CEE06) Loss of Main Feedwater / IV BWE04: W/E05 Inedequate Heat Transfer - Loss of Secondary Heat Sink / IV 000058 Loss of DC Power / VI 000060 Accidentet Geseous Redweste Ret. / IX 000061 ARM System Atarms / Vit WE16 High Containment Radiation / IX 000065 Loss of Instrument Air / Vfil CEE09 Functional Recoverv KIA Category Point Totals: Group Point Total: NUREG-1021 25 Of 39 Interim Rev. 8, January 1997

3- s t 1 n O 0 7 4- io 9 P 9 S 1 E m r p y o m r F I a u n a J 8 v e R i m r e t n I

       )

l e c ip o T A

       /

3 K p u o r G

   /

1 r i e la eT t o tin- s T u n t t n Oio t io nlu P ioo p 9 t v u 3 aE O n or t G f imn a e o xP Et 6 a G 2 O Rm So n Rb A2 WA Pd n a A1 y c n e K3 o r e m E K2 K1 t t V

                 /

t V n l n I I o t

             /

e s V / it d e / e c n ic r u n r u n o c s o s u it A I V lo it s F c c a e y n a / n u f r V t u n r E t i

                                     /

o / e f i r e c r f a !e d w d A e e a M n o n n v n S e P a l e e O d i

s

       /

le H s s k e r o la e v l e e ie e o o t e e t D t l t l o is l m a L u - u i V L a r F T r F f R y S e t N e ) f P c / C n n t n r 8 O n a R e e 1

       /

d 0 O e m io 2

       #       u A f

o E o in s G P 0 s r d s in E s

                 /

s 4 e o e m e y 1 P A e r W B s o 1 E m l o c x a e t n r o - G O P L E F o g

       /         (
                         &           E    t e

E 8 6 6 3 5 7 S C t E 1 0 0 6 a R 0 2 3 5 1 1 3 5 C 4- 0 0 0 K A A A 1 1 U 0 0 0 / /

                                     /

E E A N S 0 0 0 W W W E /

                                                    /

E 0 0 0 B B B C W W K

3- s h) 1 0 t io n 7 9 T. 4-S E m P 9 1 y r p o r F I m a _ u n a J 8 v e R

                    )

e m ( icp i

 -                    o                                                                         r   _

T e t A

                    /                                                                     l:    n K                                                                     t a  I
 -                                                                                         o T         _

t _ n _ io P _ p u o r G - G _ e1 A4 ln> it u

 -           uo                                                                                     _

A3 OrG _ n! _ oT 9 _

      \     itr                                                                               3     _

ae A2 ni [G iT f m- o a xa Eme A1 _ 7 _ Oe 2 _

 .          Rv SS      K6                                                                              .

Rt n Wa l

 .          PP      K5 K4 K3 K2 K1 n

o it a t u r c e lo A r t a r t s o w e n e it d a m o r n e n n o C u n o e o N o t a n o M F i ir o  : e e o i t a y t u it lsa

                     /         m  m F  it    a   e    n                  b      la  n
                     #         u  u      a  t n

r u v c ir s o t y n i e P ic lo a o o v lo t e e t a o r o r e t is e M T m e i r t n V f d n m r C S e o D t p n t D a d a I u e p r e t a r e e e is o t n 1 s lo n S n t r t n s t n la D it i o 2 y d o a s m n a w m w a S o d io e e e e t s d E ic d e e P 0 R C la e r it in T m d m e e / y r e id y 1 r e s r n n n n e r t c R G a r i lo o ic e o e e ie o ia e F ia le e R o - r t n P e r n d g G t n c a m e tc o t n C t n d n l ix E iu t s e e E ia

     '       1 0

4-C o R e h C E n d R o N u I c-n C o e le C o o C ie M A u C D iq L W e A e r C t a R U 1 3 4 3 4 5 7 2 5 6 6 9 1 3 8 1 2 A N S 0 0 0 1 1 1 1 2 2 2 5 5 6 6 6 7 7 I E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 K 0_

ES-401 PWR SRO Exemination Outline Form ES-401-3 Pfent Svetems - Tier :'/Grou p 2 System # / Name K K K K K K A A A A G K/A Topic (e) Imp, Pointe 1 2 3 4 5 6 1 2 3 4 002 Reactor Cootent 006 Emeroency Core Coofina 010 Pressurizer Pressure Control 011 Pressurizer Levet Controf 012 Reector Protection 016 Non-nucteer instrumentation 027 Conteinment fodine Removal 028 Hydregen Recombiner and Purge Control 029 Containment Purae 033 Spent Fuel Pool Coofino 034 Fue s Hendrina Eauipment 035 Steem Generator 039 Mein and Rehest Steam 055 Condenser Air Removal 062 AC Electrical Distribution 064 Emeroency Diesci Generetor 073 Procese Radiation Monitorina 075 Circulatina Water 079 Station Air 086 Fire Protection 103 Conteinment K/A Cateoory Point Totals: Genop Point Total: NUREG-1021 28 of 39 Interim Rev. 8, January 1997 O O O

3- s

 \/   1 t

n 0 io 7 4- P 9 D S E 9 1 m r y p o r F I m a u - n a J 8 v e

             )                                                R e

l c m ip i o r _ T e _ A t _

             /

K l: a n t I o T t i n o P p u o r G G e3 A4 in> tt u uo OrG A3 n? o 1 I 9 itr ae A2 3 _ ni iT - m- f ( C a x Eme s A1 9 O O :s 2 Rv SS K6 Rt n Wa PP l K5 K4 K3 K2 K1 f _ o r t n e _

                        *r       C e         e                                _

T s e r s ._ m h e e _ o c t e v r N la n B o v e u W e f e s

             /

o Q a n r is _

             #     m             ib   e                   t e
                        /

f lin r n o m e o u e T e R li o T G t t e C / r ir t n 1 _ s a R o e e A i y e r t in t a o 2 S H e n m t P 0 e u br W n _ la izr n D u e y r 1 - u u s T e m o - d s - m icv u g G is s . a in r e E e e e a r e t s t 1 r m a R U t 0 R P C S M S n C

 '[   4-I U

5 7 8 1 5 6 8 A N S 0 0 0 4 4 7 7 / E 0 0 0 0 0 0 0 K

ES-401 PWR R0 Examination Outline Form ES-401-4 I Facility: Date of Exam: Exam Level: l K/A Category Points Tier Group Point l K K K K K K A A A A G Total 1 2 3 4 5 6 1 2 3 4 l g -_ _ _ _ _ _ . _

1. 1 l !En 16 Emergency &

Abnormal 2

                                                                   '  )               U P1 ant           3                 lI$$gd                  $5f;hjf               3 Evolutions                            nytagggg                qggg Totals               h hh                           k 1

l l l l 23 Plant Systems 3 8 l l l l Tier l 51 Totals l __

3. Generic Knowledge and l Cat 1l Cat 2 Cat 3 Cat 4 Abilities 13 Note: -

Attempt to distribute topics among all K/A categories; select at least one topic from every K/A category within each tier. Actual point totals must match those specified in the table. Select topics from many systems; avoid selecting more than two or three K/A topics from a given system unless they relate to plant-specific priorities. Systems / evolutions within each group are identified on the associated outline. The shaded areas are not applicable to the category / tier. NUREG-1021 30 of 39 Interim Rev. 8, January 1997

(gg l~g\ RJ (n

                                                                                                                                                                                                        ~

ES-401 PWR RO Examination Outline Form ES-401-4 E.. -a a and Abno mal P ent Evolutions - Tier 1/ Group 1 E/ APE # / Name / Safety Function K K K A A G K/A Topic (e) Imp. Points 1 2 3 1 2 000005 Inocerable/ Stuck Control Rod / I ! 000015/17 RCP Melfunctions /IV BWKO9: CE/A13: WM09&E10 Naturel Ciro. / IV 000024 Emeroenev Boretion /I 000026 Loss of Component Coolina Water / Vill 000027 Pressurizer Pressure Control System Melfunction / til 000040 (BW/EOS; CEE05; W,E12) Steam Une Rupture - Excessive Heat Trenefer / IV CE/All: WK08 RCS Overcoolina - PTS /IV , 000051 Lose of Condenser Vecuum / IV 000055 Station Blackout / VI 000057 Loos of Vital AC Elec. Inst. Bus / VI 000062 Lose of Nuclear Service Weter / IV 000067 Plant Fire On-site /IX 000068 IBW/A06) Control Room Evee. I Vill 000069 (W/E14) Loss of CTMT Intearity / V 000074 (WX06&E07) Ined. Core Coolina / IV BW/E03 Inadeauste Subcoolina Morain / IV 000076 Hioh Reactor Coolant Activity / IX BW/A02& A03 Loss of NNI-X/Y / Vil K/A Cateoorv Totale: Group Point Total: NUREG-1021 31 Of 39 Interim Rev. 8, January 1997

4- e t 1 n 0 io 4 - P S E m r p y o m r F I a u n a J 8 v e R i m r e t n I

         )

t s c ip o T A

         /

2 K n u o r G

   /

1 r e l: i T t a e- o in e T ltn t n uo Oitu io nlo P o n 9 iv u tE a nt m in G - ma aP x EJ G 2 O-. 3 Re , Rn b A2 WA Pd n a A1 v c n.

     -.. K3 r.

n i E K2 K1 s s X

                        -              f                 V     V                                        V    o                I p               i l

I

                                                         /

I t I L X / I ir I / f l l / - f T t n

                                                                   .        /       I       t V

i t r r I i. e n r n a e r n m / e e . R o o e a u a / l f e t f t e I i c f a s e X it c

              /

t c id l i t r o e e r c t 1 N / r u w n R t f n l a c I a R k a S i V k t a e s / a e c D e a o d r a n u R A t t M o / a a u e T t s F w / n n e a w t o

n. n i

y a I e a t V / 1 N a L R F t a d V it d r

                  /   2)                 c             .
                                                       . w   l o            n I w

R e e e I W e / ia t e ht 0 a fi l i t . o o a / t m b b in H V d R s d n t f d E o i d lo o e u u a a a a i o / S

                                           /   /

Jt l o C o m e e n t T T M eV

                                                                                                                    /

R s u m R y S W R E A A a a tI r r r  : Ct/ i r n o ri l C t is R i r r a e d o a e v s

         /

d l o / V o n C C o C c. i t r s v n a d e o o f o u/ w l u e l t n o ta e o r ;v k I a O O C i. / o S r a t a t a s qk o o s A e t R t n 0r c / L L e A n t T c m r r s en P a c o m a 1e a o V C

                                                                 -.           c R         r   r e   e     di i

L G m m e T N s u C o Ev

                      &o       b n

i r r a k a k a d i s O n. to i a e H d e u o t e n n e L o aS C l a l a e t in a R t o T e e L

                                                                 -i n          R   R       S     n   e         Int  a  D               s     !

s n 1

         /

u d 2ce u e iz r r t u E

a. t s I G G 1

6 t n t n y t n n io 2

         #          e 0 R      R     n r B   B   O     3 f   f io f   f               5e      f o   e     e S     o     P i

n n E t i u s l n O f o J, o o c o o m m 0 E 0H d d C it o 0 E t n o / n br s le o A T .

                                                                        . s   s       s   a   a   a  /  Ev        s i c

i c M c y 1 P o o W n- a u e r C s s s it n s m e e E /r s c c R h n r o - A r Bo l r m a W s T o o o o t t C Wa o A A ia u g

         /    C   D   (t r P   T   P   S   L   O             o   t      L   L   A   L   L   S   S   (

f

                                                                                                             ;- n L   A                 F     e E    1   3   7az      1   4   8   9   1 L     B L     S      2   5   9   2   3   7   8   4   4 e,    8   9     0   1 H        t 1

0 0 0 0 4 O 2 2 2 5 0c 5 5 6 6 6 9 a 0 0i 0 0 1 1 2 3 3 3 3 0 C 4- 0 0 0ib l A

                               /

A 0 0 0 O E 1 0 0 0 0 0 0 0 0 0 Ee

                                                                                                            /

0 0 0 0 1 E 0 0 0a / 0 0 0 M

                                                         /

E E 0 0 0 0 0 0 0 0 S 0 0 0 0 E / A S 0 0 0t W B W B 0 0 0 W W B W W

                                                                     /

0 0 0 0 0 0 0 0 0 0 0 0 Wf Bo 0 0 0 0 0 0 0 0

                                                                                                                                      /

W E C

                                                                                                                                              /

K E 0 0 0S 0 0 0 . 0 0 0 0

       )l 4-       s t

1 n 0 io 7 4- P 9 S 9 E 1 m r p y o m r F I a u n a J 8 v e R i m r e t n I

                 )

( s icp o T A

                 /

3 K . p u o r G

             /

1 r l: Tie t a e- o n T lis t n t uo n Oitu io _ nlo P o iv p 9 u h' t aE o 3 r nt r in G f mla O aP xl 3 Ee G . Om 3 Ror Rnb A2 WA Pd n a A1 _ y c n e K3 o r e m E K2 _ K1 ._ t t i V

                          /

t V n  ! n I I o I e e V / it

                      /

d e r I e c n ic u r

                                               .           r u

n o c t t s e e u it A I V i V o it s F c a e y n s. / / lc u f r V t u i r r n t f p e f  ;

                          ;       e  i   E    c      I r
                                                            - /

f le 4, w A d A e e a a n n v n M o t S . P n a te e O id

                 /    le  H            e  s   v      k e   r   o icn '

s v e m ie o e le it u le l e lo t m u s r u D l i L t a F n e L F - t R v V S r T N r e f f s e / e t n t

                          )

O n P n C n e in 1 I iz 8 0 I 0 e a R e o 2 r u f f E o n s G m P s A o o r i d s n 0 E s

                          /

s s 4 e o e m ia y 1 P e W s s 1 c a t r A r B o o E m lo x e n o - t/. / P ( L L E F E o g G E & S t C e E 8 6 6 5 3 5 7 6 t e, 1 2 3 5 6 1 0 0 3 5 a R 0 0 0 0 0 E A A 1 C U 4- 0 0 0 0 / / / A 1 E 1 E N

                                                     /           A S           0   0     0    0   W  W BW     E   /    /
                                                                 /

E 0 0 0 0 B B C W W K

4- e t 1 n 0 io 4- P S E m . y r o p r F m a I u n a J 8 v e

      )

R s t c i m ip o r T e t A

      /                                                        l:   n K                                                         a I t

o T t n io P p u o r G G A4 e1 ins u tturo A3 OG / n: o' 9 ir taie A2 nT im-ae xm A1 Ee 4 Os 3 RvS K6 R t n Wa Pl P K5 K4 K3 K2 K1 _ n - i o _ t a t u r c e l o A r t a t r e o w e n e i t d a m o r n e n o C t u n o e ir N o e a o M F o  :

      /        m       e it a   e     o            y         it le
      #        u   m u

F t r n c fe n a e P y n u i lo n s o t o v lo t e e t a o r e e o M T m p t e ir n V m r C e a t t D e f a u e r e t r e s e is n t n s to d n S r o t s a D o 1 _ y d o e t s m n n e w m w s it io 2 . S o d n e e e t d E d a ia P 0 R C la e r i T m d a s e / y e d y 1 r e r n n n e r R G a r lo o ic e a e ia o e F ia e R o r t c m in e r n d g t n e lc o t n C d n l ix iu t e a e o a e a u c o e o ia u a e e t 1 h n n

                                 -                           r  a 0

4-C R C E N i C Ic C M A i L W A C 1 3 4 3 5 7 2 5 6 9 1 9 1 2 A i S 0 0 0 1 1 1 2 2 5 5 6 6 7 7 / E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 K

s ~% s . v ES-401 PWR RO Examination Outline Form ES-401-4 Pfent Swe : ems - Tier T.Grou > 2 System # / Nome K K K K K K A A A A G K/A Topic (s) Imp. Points 1 2 3 4 5 6 1 2 3 4 002 Reactor Cootent 006 Emeroency Core Coolino 010 Pressurizer Pressure Controf 011 Pressurizer Level Control 012 Reactor Protection 014 Rod Position Indication 016 Non-nucteer Instrumentation 026 Containment Sorov 029 Containment Purae 033 Soent Fuel Pool Coolina 035 Steem Generator _,039 Mein and Reheet Steam 055 Condenser Air Removal 062 AC E?actrical Distribution 063 DC Efeetrical Distribution 064 Emeroency Diesef Generator 073 Process Radiation Monitorino 075 Circulatina Water 079 Station Air 086 Fire Protection K/A Category Point Totsis: Group Point Total: NUREG-1021 35 Of 39 Interim Rev. 8, January 1997

4- s t 1 n O 0 7 4- io 9 P S 9 E 1 m r p

        .                                                   y o

m r F I a u n a J 8 v e

     )

R s t c i m ip o r T e A t

     /

K l: a n t I o T t n io P n u o r G G A4 e3 in>u ltur o A3 O, nlT o ir 9 t e A2 3 O ai int f m-as o xm A1 Ee 6 t Os 3 Ry S K6 R t n We P t' 8 K5 K4 K3 K2 K1 lor t e n g o k n r C a u e

                      !e  P          s e      T     r   v            s e   o d          o m      h             n   t o        c  t a   m   a    n   v   r N    le    n       e   r    e B     o v    e u W   R     e    m   e  f e

s

     /     o        a   e   n    p   n   r            in
     #        Q     n   n i

b i e t m e 1 f i iuq br n o m R e fo id m u e T e i le o lo o E T G t t C c a / r e ir t n s s R t e n o e A t 1 y e r t n R i m n t a n io 2 S H e n e t u i br t n e P 0 e m n d n W e m y la izr n n e g e D u e n r 1 u u o T m o - p ia o H icv ie O d s r m n u g G s t n a r t n e E is e e m o o d le e ie r t s o t 1 r yl u e a R Hrto t 0 R P C C S M S n C F I C U 4- 5 7 8 7 8n 4 1 5 6 8 3 A N S 0 0 0 2 2o 3 4 4 7 7 0 / E 0 0 0 0 0C 0 0 0 0 0 1 K

a a p ES-401 Generic Knowledge and Abilities Outline (Tier 3) Form ES-401-5 4 Facility: Date of Exam: Exam Level: Category K/A # Topic Imp. Points i 2.1. 2.1. Conduct of 2.1. , Operations 2.1. 2.1. 2.1. Tot al 2.2. 2.2. l l 2.2. l 2.2. (j Total 2.3. 2.3. Radiat- on 2.3. Contro' p,3, 2.3. 2.3. Total 2.4. j 2.4. l Emergency p 4-Procedures and Plan 2,4, 2.4. 2.4. Total ' Tier 1 Target Point Total (R0/SRO) 13/17 NUREG-1021 37 of 39 Interim Rev. 8, January 1997

4 I ES-401 Written Examination Form ES-401-6 Quality Assurance checklist Facility: Date of Exam: Exam level: R0/SR0 Initial 2 Item Description a b c

1. Questions and answers technically accurate and applicable to facility

2. NRC K/As and learning objectives referenced for all ouestions

3. R0/SR0 overlap is no more than 751ercent, and SR0 ouestions are appropriate per 10 C R 55.43

4. Question duplication from [ practice exams, quizzes, and] the last two licensing exams is no more than 25 percent IN/A for NRC-developed examsl

5. No question duplication from the license  :

screenino/ audit exam ' 6. Bank use meets limits (no more than 50 percent from - the bank / at least 10 percent new)

7. At least 50 percent of the questions on the exam (including all new questions comprehension / analysis level) are written at the

8. References / handouts provided do not cive away answers

9. Question distribution meets previously approved examination outline: deviations are justified

10. Question )sychometric quality and format meet ES, Appendix L ouidelines

11. The exam contains 100, one-point, multiple choice items; the total is correct and corresponds to value on cover sheet Printed Name / Signature Date i

a. Author

b. Facility Reviewer (*)

c. NRC Chief Examiner (*)

d. NRC Regional Supervisor (*)

Note: (*) The facility reviewer's signature is not applicable for NRC-

developed examinations; two independent NRC reviews are required.

() See special instructions (Section E.2.c) for shaded boxes. NUREG-1021 38 of 39 Interim Rev. 8, January 1997

ES-401 Site-Specific Written Examination Form ES-401-7

, Cover Sheet i l 1 U.S. Nuclear Regulatory Commission

Site-Specific 4

Written Examination 4 Applicant Information Name: Region: I / II / III / IV Date: Facility / Unit: License Level: R0 / SR0 Reactor Type: W / CE / BW / GE Start Time: Finish Time: Instructions Use the answer sheets provided to document your answers. Staple this cover p sheet on top of the answer sheets. The passing grade requires a final grade of at least 80.00 percent. Examination papers will be collected four hours after the examination starts. , 4 Applicant Certification All work done on this examination is my own. I have neither given nor received aid. 1 Applicant's Signature Results Examination "alue Points

Applicant's Score Points j Applicant's Grade Percent 6

i 39 of 39 Interim Rev. 8, January 1997 NUREG-1021

5 , ES-402

                           . ADMINISTERING INITIAL WRITTEN EXAMINATIONS                          i A. PURPOSE This standard specifies the requirements and procedures for administering                    ;

ritten examinations for the initial licensing of reactor operator (RO) and j senior reactor operator (SRO) applicants at power reactor facilities. The 1 standard includes instructions for proctoring the examinations and conducting post-examination reviews of NRC-developed examinations. B. BACKGROUND As noted in ES-201, facility licensees will generally prepare the written operator licensing examinations, subject to review and approval by the NRC. Generally, examinations that are prepared by the facility licensee will also be administered by the facility licensee in accordance with the instructions contained herein. C. RESPONSIBILITIES

1. Facility Licensee

a. The facility licensee shall safeguard the integrity and security s of the examinations in accordance with ES-201.

b. The facility licensee shall provide a single room suitable for administering the written examination. To ensure examination integrity, the room shall be large enough so that there is only one applicant per table, with a 3-foot space between tables. .

The examination room and supporting restroom facilities (i.e., the i examination area) shall be located to prevent the applicants from ' having contact with all other facility and contractor personnel l during the written examination.

c. If desired and compatible with examination security requirements, the facility licensee may arrange for the applicants to have  ;

lunch, coffee, or other refreshments during the examination.

d. Before the scheduled examination date, the facility licensee should familiarize the applicants with the examination policies and guidelines contained in Appendix E.

e. The facility licensee shall provide the necessary copies of the approved examinations, answer sheets, and handouts (e.g., equation sheets, selected technical specifications, and steam tables) for each applicant, as directed and approved by the NRC chief examiner.

NUREG-1021 1 of 6 Interim Rev. 8, January 1997

ES-402 The facility licensee may use machine-gradable answer sheets if desired, but this is not required. -

f. If the facility licensee developed the examination, it shall also administer the examination to the applicants identified on the examination assignment sheet (Attachment 4 of ES-201) as arranged with the NRC chief examiner and in accordance with the specific instructions in Section D.

As a general rule, the facility licensee should give the written examinations the week before or during the same week that the NRC regional office administers the operating tests. However, under extenuating circumstances and with prior approval from the NRC Operator Licensing Branch (OLB), the examinations may be given up to 30 days before the operating tests, provided that the NRC has accepted the license applications, resolved any applicable waiver requests, and approved the examinations. If necessary, OLB may authorize the licensee to delay the examinations until after the operating tests are complete.

g. The facility licensee will send a letter to the NRC regional administrator to formally withdraw the applications of any individuals whose names appear on the examination assignment sheet (Attachment 4 of ES-201) but will not be taking the examination.

h. As discussed in Section E, the facility licensee should provide the NRC regional office with formal comments for consideration during the grading process (refer to ES-403). The facility licensee may also request an informal meeting with the NRC chief examiner to discuss the examination questions and resolve facility concerns.

2. NRC Reaional Office

a. The NRC regional office may administer the examination, at its discretion, in accordance with the specific instructions in Section D, even if the examination was developed by the facility licensee. However, the regional office will generally arrange for the facility licensee to administer the examination. (Refer to ES-201 for further instructions on examination scheduling.)

If the NRC developed the examination, the regional ofNee should normally arrange for an NRC examiner to administer the examination.

b. If the facility licensee will conduct the examinations while the NRC examiners are on-site, the chief examiner should inspect the examination facilities to ensure their adequacy. In addition, the NRC examiners should periodically monitor the exam to ensure that the proctor is appropriately addressing the applicants' questions.

NUREG-1021 2 of 6 Interim Rev. 8, January 1997 l l

1 i

/

ES-402 i i If this is r.ot feasible, the regional office should consider having an examiner check the facilities during the preparatory

site visit (if one is deemed necessary) or upon arriving at the j site for the operating tests.

! c. When the applicants have completed the written examination, the j chief examiner may conduct an examination review with the facility ! staff as described in Section E below. j D. . EXAMINATION ADMINISTRATION INSTRUCTIONS I l 1. Make Prenarations i l .a.- ' Arrange for the applicants to be proctored at all times while

-taking the written examination. Ensure that the proctor clearly 1 understands his or her responsibilities (refer to Section D.2) before the examinations are distributed.

I If the NRC will administer the examinations, the chief examiner ! should consider using the following resources to ensure adequate proctoring: NRC secretarial help i - another examiner l, - other NRC employees

The examiner may arrange for facility employees to proctor the examination for brief periods'if it is necessary for the examiner

to go to the restroom.

b. At least one individual who is familiar with the intent of the l questions (i.e., an NRC examiner or facility employee who took i part in the examination development) shall be available to clarify j examination questions for the applicants during the examination.

c. Remove from the examination area, or otherwise remove from the i , applicants' view, any wall charts, models, or other training

materials that might compromise examination integrity, s

! d. If applicable, the NRC examiner shall verify each applicant's j' identity and examination level against the examination assignment j sheet (Attachment 4 of ES-201) before beginning the examination. 1 Any errors or absences shall be resolved with the facility staff, and the assignment sheet shall be updated as required.

e. If possible, the R0 and SR0 applicants shall be seated at alternate tables. The proctor shall construct a chart illustrating the seating arrangement of the applicants during the examination.

NUREG-1021 3 of 6 Interim Rev. 8, January 1997

l l

2. Start the Examination

a. Remind the applicants that they may use calculators to complete the examination, and that only reference materials provided with the examination are allowed in the examination area (i.e., the examination room and supporting restroom facilities).

b. Pass out the examinations, blank answer sheets, and all required handouts as approved by the NRC chief examiner (e.g., steam ,

tables, equation sheets, and selected technical specifications). I Instruct the applicants not to review the examination until told to do so.

c. Provide each applicant with a copy of Appendix E, " Policies and i Guidelines for Taking NRC Examinations," and brief the applicants I on the rules and guidelines that will be in effect during the l written examination (i.e., review Parts A and B of the appendix).

If time permits and the operating tests have not yet been administered, review those policies and guidelines (i.e., Parts C, D, and E of Appendix E) as well; this will save time later and give the applicants greater opportunity to resolve any questions they may have.  !

d. Instruct the applicants to verify the completeness of their copies by checking each page of the examination.

e. Answer any questions that the applicants may have regarding the examination policies. Start the examination, and record the time.

3. Monitor the Examination

a. The proctor shall give full attention to the applicants taking the )

examination. The proctor shall not read procedures or other material, grade examinations, or engage in any other activities in a manner that may divert his or her attention from the applicants and possibly cause the examination to be compromised. l

b. Personnel responding to questions raised by the applicants during the examination must be extremely careful not to lead the applicants or give away answers when clarifying questions. If the proctor has any doubt about how to respond to an applicant's question, it is best to withhold additional guidance and instrtct i the applicant to do his or her best with the information that is provided.

i Any question changes or clarifications shall be made on a chalk board or white board, if available, and called to the atteation of all the applicants. Changes made to questions during the i examination should be made in ink on the NRC master copy and on a  ! copy that is retained by the facility staff after the examination NUREG-1021 4 of 6 Interim Rev. 8, January 1997

l 4

ES-402 N i is administered. Changes shall be reviewed and approved by the i NRC chief examiner as part of the grading process (refer to ES- , 403). All applicant q'estions regarding specific written examination  ; test items and ali ::tatements of clarification shall also be 1 i documented (verbatim if possible) for future review by the NRC  ! ! chief examiner and for reference in resolving grading conflicts. '

c. The proctor shall periodically advise the applicants of the time j that remains to complete the examination. Normally, a chalk board or white board is available and can be used for this purpose.

4. Complete the Examination

a. As the applicants complete the examination, ensure tha they sign

the examination cover sheet and staple it on top of their answer sheets. Collect the examination packages, including the questions j and answer sheets, and any reference material provided with the examination. Verify that all applicants have entered their names i

on both the answer and cover sheets, and record the official start time and the time at which each applicant completed the  ! examination in the space provided on the examination cover sheet. l i i b. Retain the cover and answer sheets for grading in accordance with i ES-403. The question books may be distributed to the applicants after the last examination has been collected.

c. Remind the applicants to leave the examination area, as previously defined. t

d. When four hours have elapsed, instruct the remaining applicants to I stop work, sign their examination cover sheets, and turn in their examinations.

e. Deliver the completed examination packages, the marked-up master examinations, the list of applicant questions and answers, and the seating chart to the NRC chief examiner or the appropriate facility representative, as applicable, for review and grading in accordance with ES-403.

E. POST-EXAMINATION REVIEWS

1. If the NRC administered the examination, the chief examiner shall ensure that the master copy of the examination reflects all changes made to questions during the administration of the examination. The chief examiner will then provide a copy of the master examination and answer key to the facility staff and answer any questions they may have regarding the NRC's examination review and comment process.

NUREG-1021 5 of 6 Interim Rev. 8, January 1997

ES-402

2. If the NRC developed the examination, the chief examiner will also provide the facility licensee with a copy of the examination as edited during the facility prereview. If the facility reviewers believe that the NRC did not adequately resolve the prereview comments, they should address those concerns in a formal comment letter.

3. The NRC chief examiner will request that the facility prereviewers confirm that they did not divulge any information about the examination (s) by having them sign the post-examination security statement (Form ES-201-3) after the examinations are completed.

4. The facility licensee should submit formal comments within five working days after the examination exit conference. However, the facility licensee may expedite the grading process by giving draft comments to the NRC chief examiner before he or she leaves the site. The NRC will consider comments not submitted within the requested time on a case-by-case basis; however, late comments may delay the examination grading process.

The facility licensee is also encouraged to collect and consider comments from the license applicants and include them in its submittal to the NRC.

5. The facility licensee should submit all comments in +he following format:

List the question, answer, and reference. State the comment and make a recommendation whether the answer should be changed or the question should be deleted. If the facility licensee does not support an applicant's comment, it should briefly explain the reason for its rejection. Support the comment with a reference, and provide a copy if it was not included in the original reference material submittal. (Note: The NRC will not change the examination without a reference te , support the facility's comment.)

6. Formal comments should be signed by the highest level of on-site corporate management responsible for plant operations (e.g., Vice President for Nuclear Operations) and addressed to the responsible NRC regional office, with a copy to the NRC chief examiner.

l NUREG-1021 6 of 6 Interim Rev. 8, January 1997

i , i - 1-4 ES-403  ! GRADING INITIAL SITE-SPECIFIC WRITTEN EXAMINATIONS  ! [-  ! A. PURPOSE i i This standard explains the requirements and procedures for grading the site-  !

specific written examinations for the initial licensing of reactor operator  ;

j (RO) and senior reactor operator (SRO) applicants at power reactor facilities. s 4 The standard includes instructions for evaluating and revising the examinations after they are administered, grading the examinations, and ' F conducting the first quality assurance (QA) review of the graded examinations. B. BACKGROUND l

As discussed in ES-201, facility licensees will generally develop and ,

i administer the initial operator licensing written examinations, subject to ' i review and approval by the NRC. Facility licensees will also be expected to grade the written examinations, evaluate the outcome, and submit the r . examination results to their NRC regional office for review, approval, and i j licensing action in accordance with ES-501. I 1 C. RESPONSIBILITIES ) i j 'l. Facility Licensee 1 , a. If the facility licensee developed and administered the written j examinations, the licensee is also expected to perform the j following grading activities, as described in Section D: 4 Review and resolve any questions and comments that arose

during and after the examination (refer to ES-402).

i L - Grade the examinations and review the grading using Form ES-

403-1, " Written . Examination Grading Quality Assurance Checkli st. "

l Evaluate the applicants' performance on the examination. ! Facility management will review the examination grading based on

- the guidance in ES-501 and forward the graded examinations and all

associated documentation to the NRC chief examiner so that it is 8

received, when practical, within five working days after the j examination was administered. i i b. If the NRC developed the examinations, the facility licensee's

responsibility is limited to providing the NRC chief examiner with ,

1 comments and recommendations regarding question deletions and i j answer key changes. Such comments and recommendations should , normally be received within five working days after the exit { meeting; any delay in submitting the comments will likely result l

 !-                                                                                                                                   I NUREG-1021                                                        1 of 5           Interim Rev. 8, January 1997 1

d

ES-403 in a comparable delay in the final licensing actions. (Refer to ES-402 for additional instructions regarding the post-examination review and comment process.)

2. NRC Reaional Office

a. If the facility licensee grades the examinations, the regional office shall provide guidance and assistance, as necessary, to ensure that the facility licensee complies with the instructions in Section D.

b. If the NRC developed the examinations, the regional office shall grade the examinations in accordance with Section D after receiving any comments and recommendations from the facility licensee (refer to ES-402).

c. After the examinations have been graded (by either the facility  !

licensee or an NRC examiner), the regional office shall review the ' grading, process the documentation, and complete the licensing , actions in accordance with ES-501, l D. GRADING INSTRUCTIONS The author of the examination should normally grade the examination; however, the examination may be graded by another equally qualified individual if the author is not available, the number of applicants is unusually large, or the NRC regional office or facility licensee wishes to expedite the grading process. The examinations shall be graded as expeditiously as possible, in accordance with the following instructions:

1. Evaluate Ouestions and Comments

a. Evaluate all questions posed by the applicants during the examination, any pen-and-ink changes made on the master examination during its administration, and any post-examination coments or recommendations received from the facility licensee and applicants after the examinations were administered.

Determine if any questions should be deleted from the examination, or if any answers need to be changed. Do not delete any question or change any answer unless there is a valid reference to support the change. If there is some doubt whether the NRC chief examiner will accept a proposed change, the grader is encouraged to discuss the matter with the chief examiner before proceeding with the grading process. This may help to minimize the need for grading corrections during the quality assurance reviews. For each comment and recommendation, the NRC chief examiner shall document the reason that the question was changed or the comment NUREG-1021 2 of 5 Interim Rev. 8, January 1997

i l u ES-403 was not accepted; this information will be included in the i examination report as discussed in ES-501.

b. .If it is determined that there are two correct answers, both  !

answers will be accepted as. correct. However, if three or more  ! answers could be considered correct or there is no correct answer, the question shall be deleted. Annotate the recommended changes on the master examination'and answer key and document the reason for every change or deletion.

c. Those applicant questions, facility comments, and recommendations that do not result in answer key changes or question deletions, should be evaluated to determine if the associated test questions l might benefit from editorial changes before they are used on

                 -another examination.

2. Grade the Examinations

a. On each applicant's answer sheet, indicate in red pen or penef!

which questions were answered incorrectly, note their correct answers, and ' indicate which questions (if any) were deleted. If the answer sheet is more than one page long, it is helpful to note the total number of incorrect answers on each page to aid in tabulating the _ final grade. O~ If the examinations are graded by machine, attach a copy of each

                 ' applicant's profile report to his or her answer sheet, or manually annotate the answer sheet as noted above.

b. If it is necessary to change a grade during the grading process, do so by lining out the original grade in such a way that it remains legible. Briefly explain the reason for the change on the applicant's answer sheet, and initial the change. Under no l'

circumstances will a grader use " white-out" or other methods that obscure the change.

c. After grading all the questions, enter the " Examination Value" 1 (i.e., the original test point total minus the point value of any  !

deleted questions), the " Applicant's Score," and the " Applicant's l Grade" (i.e., the Applicant's Score divided by the Examination i Value) in the "Results" section of the applicant's written  ! examination cover sheet. If a facility chooses to share its preliminary grades with the applicants, it should caution them that the outcome may change if the NRC does not accept all of the facility licensee's recommended changes to the examination answer key.  ; O l NUREG-1021 3 of 5 Interim Rev. 8, January 1997 i

ES-403 4

3. Evaluate and Review the Gradina .

1

a. Evaluate the applicants' performance on each examination question to identify any indications of a problem with the question or a j deficiency in the applicants' training program. A table that I summarizes the applicants' answers on each question, or a l computerized item analysis (if the examinations were graded by machine) may be used to identify items with which the applicants had problems.

If it appears that a test question was faulty, determine whether the question should be deleted, the answer key should be changed, and/or the question should be revised before reuse. Then regrade the examinations as necessary. If it appears that the training program was deficient, determine the need for remedial training and/or a program upgrade. , i

b. After evaluating the examinations, review the grading in detaf 7 ,

and complete Form ES-403-1, " Examination Grading Quality Assurance ' Checklist."

c. Forward the examination package (i.e., the master examination and answer key, justification for any examination changes, any item analysis that was performed, the applicant's examination cover and answer sheets, and Form ES-403-1) to the designated facility representative (if applicable) or to the NRC chief examiner for quality assurance review in accordance with ES-501.

E. ATTACHMENTS / FORMS Form ES-403-1, " Written Examination Grading Quality Assurance Checklist" l l l

                                                                                    )

l NUREG-1021 4 of 5 Interim Rev. 8, January 1997 9

ES-403 Written Examination Grading Form ES-403-1 Quality Assurance Checklist Facility: Date of Exam: Exam Level: R0/SR0 Initials Item Description a b c

1. Answer key changes and question deletions justified and documented

2. Applicants' scores checked for addition errors (reviewers spot check > 25% of examinations)

3. Grading for all borderline cases (80% +/- 2%)

reviewed in detail

4. All other failing examinations checked to ensure that grades are justified

5. Performance on missed questions checked for training deficiencies and wording problems; O

() evaluate validity of questions missed by half or more of the applicants Printed Name / Signature Date

a. Grader

b. Facility Reviewer (*)

c. NRC Chief Examiner (*)

 ,  d. NRC Supervisor (*)

(*) The facility reviewer's signature is not applicable for examinations graded by the NRC; two independent NRC reviews are required. NUREG-1021 5 of 5 Interim Rev. 8, January 1997 1

(~ ( ES-501 INITIAL POST-EXAMINATION ACTIVITIES A. PURPOSE This standard describes and coordinates the activities that must be completed after the written examinations and operating tests have been administered and graded in accordance with the ES-300 and ES-400 series. Specifically, the standard includes instructions for assembling and reviewing the examination package, notifying the facility licensee and applicants of the examination results, preparing the examination report, and retaining examination records. 1 B. BACKGROUND The goal of the Operator Licensing Branch (0LB) is to complete licensing or denial actions within 30 days after the facility licensee submits the graded examinations or its formal written examination comments to the NRC. The NRC and facility licensee staffs should establish their priorities and schedules ' to achieve this goal. Applicants must achieve a grade of 80 percent or greater on the written examination and a grade of " satisfactory" on all three categories of the operating test to qualify for a license. C. RESPONSIBILITIES

1. Facility Licensee

a. If the facility licensee participated in developing, administering, and grading the examination, the licensee shall forward the following examination documentation to the NRC chief examiner (" addressee only") so that it is received, when practical, within five working days after the examinations were administered:

the graded written examinations (i.e., each applicant's original answer and examination cover sheets) (ES-403) the master examination (s) and answer key (s), annotated to indicate any changes made while administering (ES-402) and grading the examination (s) (ES-403) any questions asked by and answers given to the applicants during the written examination (ES-402) any comments made by the applicants after the written examination with an explanation why the comment was accepted or rejected (ES-402)  ! the written examination seating chart (ES-402) ' a completed Form ES-403-1, " Written Examination Grading Quality Assurance Checklist" (ES-403 and Section D.1) the results of any written examination performance analysis that was performed (ES-403) (3 NUREG-1021 1 of 25 Interim Rev. 8, January 1997 l

ES-501 original Form (s) ES-201-3, " Examination Security Agreement," with a pre- ::id post-examination signature by every i individual who had detailed knowledge of any part of the i written examination or operating tests before they were  ! administered.  ! l Refer to the referenced Examination Standards for a more detailed  : discussion of each documentation requirement.

b. If the facility licensee did not participate in developing, administering, and grading the examination, the licensee should l submit comments and recommendations regarding the NRC-developed  !

written examination to the NRC regional office within five working days (when practical) after the exit meeting. The facility licensee should also include and consider comments made by the license applicants that took the examination. (Refar to ES-402 for more detailed instructions.)

2. NRC Reaional Office

a. The regional office shall ensure that the operating tests and written examinations are graded in accordance wit h ES-303 and ES-403, respectively,

b. The regional office shall ensure that the examination results and licensing recommendations receive the required giality assurance (QA) reviews and approvals in accordance with Sectio:~. D, that the associated administrative requirements are completed in accordance with Section E, and that the required records are retained in accordance with Section F.

The region may use Form ES-501-1, " Post-Examination Check Sheet," to track completion of the administrative items after the examinations are administered.

c. Regional management should also review the overall examination results and any generic findings, deficiencies, or issues to ,

determine if any follow-up action is required. I If the facility licensee recommends deleting or changing the answers to five percent or more of the questions on a written examination that it developed, the regional office should request that the facility licensee explain why so many post-examination changes were necessary and what actions will be taken to improve future license examinations. The region should also consult the OLB staff for further guidance. If ten percent or more of the examination questions are deleted during the grading process, the region shall evaluate the remaining examination to ensure that the test outline sampling NUREG-1021 2 of 25 Interim Rev. 8, January '997 4

f O ES-501 U requirements in ES-401 are still satisfied. The training and 1 assessment specialist on the OLB staff should be consulted if the validity of the examination is in question. If the content validity of the examination is affected (e.g., several knowledge and ability (K/A) topics are not covered, or the majority of the remaining K/As are associated with a small number of systems) as a result of deleting questions, OLB will make a , decision whether the examination should be voided. D. EXAMINATION REVIEWS AND LICENSING ACTION Except as noted below, the QA reviews generally constitute spot checks, or sampling, to follow up on the work performed by the written examination and operating test graders in accordance with ES-403 and ES-303, respectively. If the QA reviews indicate significant problems, additional detailed review will be necessary. Reviewers should discuss all grading discrepancies with the grader or previous i reviewer before making any changes. In addition, the reviewers shall document any changes by carefully lining out the original entry so that it remains I legible, entering the revision with a brief explanation, and initialling the change. Reviewers shall not use " white-out" or other methods that obscure the original entry. O 1. Facility Manaaement If the facility licensee graded the written exarninations, an authorized facility representative shall confirm the quality of the grading and sign the bottom of Form ES-403-1 before sending the examinations to the l NRC regional office. The NRC will consider the signed form to represent facility management concurrence with the individual and collective examination results, including the justification for any examination changes.

2. NRC Chief Examiner for desianated alternatgl The written examination grading shall be independently reviewed by at least two NRC personnel using Form ES-403-1 as a guide. If the examination was graded by the chief examiner, then another examiner i shall conduct the independent review. If the chief examiner conducts i the independent review, he or she may not perform the supervisory review i required by Section D.3.  !

a. If the facility licensee graded the written examinations, the chief examiner shall immediately inventory the examination package to ensure that all required materials have been submitted. The '

chief examiner shall inform the responsible supervisor of any NUREG-1021 3 of 25 Interim Rev. 8, January 1997

ES-501 obvious deficiencies, and shall contact the facility licensee to determine the status of any missing documentation,

b. The chief examiner shall independently analyze each examination and answer key change made or recommended by the facility licensee or a license applicant to determine whether it is justified. The chief examiner shall ensure that the reason for accepting or rejecting each change or recommendation is documented in the examination report. The report shall briefly state the region's basis for accepting or rejecting each facility comment; simply stating concurrence with no explanation is not sufficient. The chief examiner will not accept a change to the examination unless the facility licensee submits a valid reference to support its recommendation.

c. The chief examiner shall review the remaining items on Form ES-403-1. The chief examiner should apply his or her Judgment when reviewing the examination results and adjust the level of the review based on the performance of the applicants and the facility licensee (e.g., the number of questions changed or deleted, the average grade, the number of borderline or failing grades, etc.).

d. The chief examiner should review the written eyamination results and the facility licensee's performance analysts (if applicable) for indications of the following:

deficiencies in the applicants' training program, so that they may be addressed in the examination report poor question construction, su the applicants are not graded unfairly, any significant problems can be addressed in the examination report, and the questions are corrected before reuse any indications that the examination was compromised

e. When satisfied with the examination grading, the chief examiner shall sign and date Form ES-403-1 and pass it on to the responsible supervisor for management review (see Item D.2.h).

The chief examiner shall also ensure that the written examination results and the names of the NRC examiners who wrote, graded, or reviewed the examinations are recorded in the " Written Examination Summary" section of each applicant's " Operator Licensing Examination Report," Form ES-303-1.

f. The chief examiner shall also review, in detaf f, the other examiners' operating test documentation to ensure that the test (as given) and its grading meet the requirements in ES-301 and ES-303. In so doing, the chief examiner shall ensure that the NUREG-1021 4 of 25 Interim Rev. 8, January 1997

i N ES-501 [ other examiners' operating test comments support the pass or fail 1 ! recommendations and check for consistent documentation and grading l l among the operators tested on the same simulator crew, t If the documentation is accurate and complete, and the licensing

recommendation is appropriate, the chief examiner shall check l " Pass" or " Fail" and sign and date the " Final Recomendation" t

block on Form ES-303-1. If the licensing recomendation is not appropriate based on the documentation presented, the chief examiner shall discuss the examination findings with the NRC

examiner of record and resolve any disagreement.

j- If the chief examiner administered the operating test, the i responsible regional supervisor shall designate another examiner l to independently review the documentation and sign the " Final j Recomandation" block on page 1 of Form ES-303-1. l g. The chief examiner shall record the results of the written examinations and operating tests on Form ES-501-2, " Power Plant i j Examination Results Sumary."  ! j j h. The chief examiner shall ensure that the examination documentation

is complete and contains all of the items identified in Section F j before forwarding the entire package to the responsible supervisor

for review and approval in accordance with Section D.3.

3 If the written examinations were administered much before the i operating tests, the chief examiner should enter that data on the

form and forward it with the completed written examination package to the responsible supervisor for review and approval in advance j of the operating test results.

1 ' 3. NRC Manaaement Review and Licensina Action I a. The responsible supervisor (or a designated alternate) shall I j ensure that all examination results and documentation are ! complete. The supervisor shall evaluate the written examination

results, ensure that the required QA reviews were completed, work

with the chief examiner and the facility licensee (as necessary)

!- to resolve any grading problems, and then sign and date Form ES-403-1 to document approval of the process. 1 Every written examination shall have at least two levels of NRC i j review. Therefore, the NRC examiner who performed the regional QA i review is disqualified from also performing the supervisory i review.

b. The responsible supervisor will also independently review the operating test results, check either the " Issue License" block or the " Deny License" block in the " License Recomendation" section l NUREG-1021 5 of 25 Interim Rev. 8, January 1997 I

ES-501 of each applicant's Form ES-303-1, and sign and date each form. Under no circumstances will all three levels of recommendation on Form ES-303-1 (i.e., operating test administrator, chief examiner, and NRC supervisor) be signed by the same individual. If the responsible supervisor (or licensing official) does not believe that the operating test documentation supports the final recommendation, he or she shall consult the NRC examiner of record and the chief examiner to discuss and resolve any disagreements. If e recommendation is overturned during the regional management review, the responsible supervisor will line out and initial the affected summary evaluations. The supervisor will then enter the new summary evaluation in the appropriate block, and explain the change on Form ES-303-2, " Operating Test Comments," and attach it to the applicant's Form ES-303-1.

c. After making the licensing recommendations, the responsible supervisor will have the operator licensing assistant prepare a license or denial letter for each examined applicant and forward the examination package to the regional licensing official.

Applicants who withdrew before taking any part of the license examination shall not be sent a denial letter. Attachments 2 and 3 provide sample R0 and SR0 (conditional) license letters and a sample denial letter. In certain instances, an applicant may have been granted a waiver (refer to ES-202) and allowed to take the examination before completing all of the training and experience requirements. In such instances, the region shall not issue a license to the applicant until the facility licensee has certified in writing that the applicant has completed all of the waived items.

d. The final licensing dec'ision is made by the regional administrator or a designated alternate, who must be at or above the level of branch chief; short-term alternates shall not make licensing decisions. The licensing official will consider all recommendations, make changes as described above, and sign each applicant's license or denial letter, as applicable.

E. EXAMINATION FOLLOW-UP

1. Notify Facility Licensee of Results The NRC regional office will notify the facility licensee and applicants of the examination results (as described below) only after they are reviewed and approved by the licensing official. If any of the applicants failed the written examination, the region shall delay NUREG-1021 6 of 25 Interim Rev. 8, January 1997

/" ES-501 issuing licenses to those applicants that passed the written examination with a grade of 82 percent or below until any written examination appeals have been reviewed for impact on the licensing decisions,

a. The region should normally notify the facility licensee's designated representative by telephone, and confirm the results by mailing a copy of Form ES-501-2 under a separate cover letter.

For each applicant that failed or had significant deficiencies that warrant further evaluation and retraining by the facility licensee, the region will also send to the facility licensee a copy of the applicant's Form ES-303-1 and written examination answer sheet. These form (s) shall not be placed in the public document room or distributed with the final examination report. If the written examinations were administered much before the operating tests and management has approved the results of those examinations, the region may notify the facility licensee of those i results rather than waiting until the operating tests are completed. S. After the licensing official has signed the license and denial letters, the region shall send each applicant's letter along with the following materials: a copy of Forms ES-303-1, ES-303-2, and ES-D-1 (and Forms 1 ES-D-2 if the applicant failed Category C of the operating l test) reflecting the "as run" scenario conditions but l without any rough examiner notes regarding the applicant's  ! performance (pen-and-ink markups of the original, approved i scenarios are acceptable) 1 a copy of the applicant's written examination cover and answer sheets (as well. as a copy of the master written examination and answer key if the applicant failed the written examination)

   ,       c. The regional office shall also send a copy of Form ES-501-2 to OLB. If any of the examinations are later regraded in response to
                 - an applicant's request for review (refer to ES-502), the original Form ES-501-2 on file in the regional office shall be corrected by lining out the old grade, entering the new grade, and initia111ng the change. Whenever a change is made, the region shall mail a copy of the revised form to OLB.                                       4

2. Return the Facility Reference Material If desired by the facility licensee, the NRC chief examiner shall ensure i that the reference materials provided for NRC examiners to prepare for the examinations are returned as soon as possible. If none of the

 ~         applicants failed the examination, the materials should be returned as NUREG-1021                             7 of 25        Interim Rev. 8, January 1997

I 1 ES-501 soon as the licenses are issued. If any applicant was denied a license based on an examination failure, the reference materials should be retained pending expiration of the 20-day period during which the i applicant may request a regrade. If an applicant requests a regrade in  ! accordance with ES-502, the chief examiner shall determine what i reference materials need to be retained and should return all unnecessary materials. All reference materials should be returned to i the facility licensee within 30 days following the resolution of any i appeals.

3. Prepare the Examination Report The NRC chief examiner shall prepare the final examination report when all portions of the examination have been graded and documented. The sample examination report included as Attachment I should be used as a guide.

a. The final examination report shall document the following:

any significant problems encountered while developing, administering, and grading the examination (e.g., , significant deficiencies in the examinations submitted by ' the facility licensee, inadequate reference materials or examination banks, examination security concerns, grading deficiencies,etc.) the results of the examination (including any generic strengths and weaknesses noted while administering the operating tests or grading and reviewing the written examinations) any issues discussed at the exit meeting

b. The report shall include the following items, as applicable:

a copy of the master written examination (s) and answer key (s) a copy of the facility licensee's (and applicants') specific comments and recommended changes regarding the written examination that was administered the specific NRC resolution, including a precise explanation for accepting or rejecting each facility comment, for each facility recommendation and a specific justification for every additional item deletion or change a simulation facility report (as described below) NUREG-1021 8 of 25 Interim Rev. 8, January 1997

ES-501 Generic comments submitted by the facility licensee about the examinations or the administration process should also be included in the report; however, such comments do not necessarily require a regional response or resolution.

c. The simulation facility report shall document the NRC examiners' concerns about the performance or fidelity of the simulation facility during the preparation or conduct of the operating tests.

A sample report is provided as Enclosure 5 to Attachment 1. All. deficiencies should be described in sufficient detail to allow screening and classification during a simulation facility inspection. The NRC examiners may include in the simulation facility report any concerns about physical fidelity (hardware or equipment discrepancies) or functional fidelity (performance of the simulation facility during normal, surveillance, abnormal, or emergency events). Each deficiency should include a description of the operation, event, or transient that was in progress, and how the simulation facility failed to accurately model the expected performance of the reference plant.

d. The applicants' names and specific grades (i.e., Form ES-501-2) shall not be published in the examination report.

O e. The region shall send the final examination report to the facility licensee and copies to the public document room (PDR).

4. Perform Other Activities

a. If an applicant did not complete the SRO upgrade training program or failed the upgrade examination, regional management should ensure that the R0 licensee complies with the requirements of 10 CFR 55.53(e), (f), and (h) and 10 CFR 55.59(a) before resuming active duties as an R0.

b. The regional office should also conduct a case-specific review of the SR0 upgrade examination to determine if the applicant failed as a result of significant deficiencies in R0 knowledge or abilities. Pursuant to 10 CFR 55.7, the NRC may, by rule, regulation, or order, impose upon any licensee additional requirements deemed appropriate or necessary to protect public health and to minimize danger to life and property. If the SRO upgrade applicant's deficiencies pose such a threat, the NRC may require the facility licensee to provide remedial training and reevaluation and to submit evidence of its completion to the NRC.

c. Once the licensing decisions are complete, the NRC examiners should discard any marked-up documentation or rough notes for those applicants receiving licenses. In addition, the author of the written examination (s) should be instructed to upload the O NUREG-1021 9 of 25 Interim Rev. 8, January 1997

l ES-501 final version of the examination (s) to the examination question bank, as appropriate. In accordance with ES-502, NRC examiners should retain all notes and documentation associated with proposed denials until the denials become final.

d. Agency policy requires that all documents submitted to the NRC for review and approval be placed in the PDR. Therefore, the regional office shall send a clean copy of the examination outline, the i

draft written examination (s), and the operating tests to the PDR  ! in accordance with OLB guidance provided under separate cover. The final written examination (s) and operating tests must also be placed in the PDR; however, the intermediate working copies of l these documents need not be submitted or retained. F. NRC RECORD RETENTION I

1. The NRC regional office shall retain the original (whenever possible) or '

a copy of the following items in the facility's master examination file for two initial examination cycles (i.e., the file should contain the , last two examinations administered at the facility): l

a. ES-201, Attachment 3, " Corporate Notification Letter"

b. ES-201, Attachment 4, " Examination Assignment Sheet," with pen-l and-ink changes to identify the applicants that were actually examined

c. Form ES-201-1, " Examination Preparation Checklist"

d. the written examination and operating test outline (s) proposed by the facility licensee, along with Form ES-201-2, " Examination i Outline Quality Assurance Checklist" l

e. the proposed NRC- or facility-developed written examination and operating tests (including comments made by the facility licensee l or the NRC, as applicable) l f. the final written examination and answer key (enclosure to l

examination report) with all changes incorporated (the pen-and-ink l corrections made for the applicants while the examination was administered may be changed to typewritten corrections; however, all changes shall be annotated in such a way that they are evident) l

g. Form ES-401-6, " Written Examination Quality Assurance Checklist;"

Form ES-301-3, " Operating Test Quality Assurance Checklist," Form ES-301-4, " Simulator Scenario Quality Assurance Checklist," Form ES-301-5, " Transient and Event Checklist," and Form ES-301-6,

                " Competencies Checklist," for each operating test administered NVREG-1021                             10 of 25       Interim Rev. 8, January 1997

                   . _ .   .- .        . - . ~ - . -     - . _ _ _ .-       .-       .   -- - -        .-

ES-501

h. Forms ES-D-1, " Scenario Outline," and ES-D-2, " Operator Actions,"

for each scenario set administered, as well as Forms ES-301-1, r

                         " Administrative Topics Outline," and ES-301-2, " Individual Walk-Through Test Outline," for each walk-through test (all record copies should have the required signatures and reflect the "as run" test conditions; pen-and-ink markups of the original, 4

approved forms are acceptable) l

i. Form ES-403-1, " Written Examination Grading Quality Assurance Checklist" J. Form ES-501-2, " Power Plant Examination Results Summary Sheet"

k. ES-501, Attachment 1, " Examination Report," with all enclosures

1. Form ES-201-3, " Examination Security Agreements"

2. The NRC regional office shall place the following items in each applicant's docket file:

a. Forms ES-303-1, " Operator Licensing Examination Report," ES-303-2, i

                         " Operating Test Comments" (original copies, all pages), and                     l ES-D-1, " Scenario Outline," as well as Form (s) ES-D-2, " Operator m                  Actions," if the applicant failed Category C of the operating test (all record copies should have the required signatures and reflect

[V T the "as run" test conditions; pen-and-ink markups of the original, approved forms are acceptable)

b. all correspondence with the applicant

c. the applicant's original written examination cover and answer sheets i l

G. ATTACHMENTS / FORMS Attachment 1, " Sample Examination Report" Attachment 2, " Sample ( l NUREG-1021 11 of 25 Interim Rev. 8, January 1997 1

ES-501 Sample Examination Report Attachment 1 NRC Letterhead (Date) (Name) (Name u/ facility) (Street address) (City. State Zio code)

SUBJECT:

EXAMINATION REPORT NO. 50-(number)

Dear (Name):

On (date), the NRC administered examinations to employees of your company who had applied for licenses to operate (Name of facility). At the conclusion of the examinations, the NRC examiners discussed the examination questions and preliminary findings with those members of your staff identified in the enclosed report. In accordance with Title 10, Section 2.790 of the Code of federal Regulatfons (10 CFR 2.790), a copy of this letter and the enclosure (s) will be placed in the NRC Public Document Room. Should you have any questions concerning these examinations, please contact (Name) at (telechone number). Sincerely, (Name) (Title) , Docket No. 50-(number)

Enclosures:

1. Report Details

2. Examination (s) and Answer Key (s) (SRO/R0)

3. Facility Recommendations

4. NRC Resolution of Recommendations (Enclosures 3 and 4 may be combined]

5. Simulation Facility Report cc: Facility Training Manager (w/ enclosures)

Public (w/ enclosures) NRC Document Control System (w/ enclosures) Regional Distribution NUREG-1021 12 of 25 Interim Rev. 8, January 1997

ES-501 2 Attachment 1 Enclosure 1 REPORT DETAILS Report No: 50-(number) Facility Licensee: (Name) (Street address) (City. State Zio code) Facility: (Name_) Facility Docket No.: 50-(number) Facility License No.: NPF-(number) Examination Dates: (date) NRC Chief Examiner: (Name. Title) Date Accompanying Personnel: (Name. Title) I O\ (Name. Title) l Approved by: (Name. Title) Date l

SUMMARY

Written examinations and operating tests were administered to (number) applicants for senior reactor operator (SRO) licenses and (number) applicants for reactor operator (RO) licenses. A written examination was administered to (number) additional R0 applicant (s). (Number) SR0s and (number) R0s passed these examinations. All others failed. l NUREG-1021 13 of 25 Interim Rev. 8, January 1997

ES-501 3 Attachment 1 2 Enclosure I

1. INTRODUCTION

2. PRE-EXAMINATION ACTIVITIES ,

3. EXAMINATION RESULTS AND RELATED FINDINGS, OBSERVATIONS, AND CONCLUSIONS

4. EXIT MEETING At the conclusion of the site visit, the examiners met with representatives of the plant staff to discuss the results of the examinations. The examiners made the following observations concerning your training program:

a. Areas of generic weaknesses were found in (briefly state the areas of weakness identified durina the exit meetina). The facility committed to place more emphasis in these areas in future training programs (0 pen Item (number)).

b. Areas in which the examiners believe that the applicants exhibited good training and knowledge were (briefly state any strenaths identified durina the exit meetjngl.

P

5. FACILITY PERSONNEL CONTACTED 1 I

i NUREG-1021 14 of 25 Interim Rev. 8, January 1997 O,

I ES-501 4 Attachment 1 l Enclosure 2 i

                                                                                       )

WRITTEN EXAMINATION (S) AND ANSWER KEY (S) (SR0/RO) ) O l l O NUREG-1021 15 of 25 Interim Rev. 8, January 1997

i ES-501 5 Attachment 1 Enclosure 3 FACILITY REC 0WiENDATIONS Question #28: The question asks for the required method of securing a diesel generator and ensuring that an auto restart does not recur following auto initiation on receipt of a valid LOCA signal with off-site power still available to its associated emergency bus. The question is recommended for deletion since the system operating procedure directs that the diesel be unloaded, verifying that the 4KV bus auto transfer annunciator is reset, and then secured by placing the handswitch in pull to lock. Therefore, the key answer - ensure the "4KV AUTO TRANSFER IMOP" annunciator is lit before placing the control switch in PULL TO LOCK - is incorrect. Question #81: The question asks for a description of RHR Loop B outboard injection valve operation if level rapidly decreases to 119.5 inches with the RHR loop B operating in the Shutdown Cooling Mode. The question is recommended for deletion since the outboard injection valve reopens automatically when the Group 4 isolation is reset, if a LPCI loop selection is sealed-in. Therefore, the key answer - the operator must reset the shutdown cooling isolation and manually reopen the RHR Loop B outboard injection valve - is incorrect. Question #96: The question asks for the condition that will prevent the standby diesel generator upon restoration of power after a station blackout. The facility recommends acceptance of an additional answer - restore power to emergency bus 2B3 prior to restoration of power to the diesel - since the reference A0P indicates that if power is restored to bus 283 before 125 VDC is restored to the standby diesel generator starting logic, then there would be no Loss of Off-site Power (LOOP) signal available for generation of an auto start signal. NUREG-1021 16 of 25 Interim Rev. 8, January 1997 l

I ES-501 6 Attachment 1 Enclosure 4 NRC RESOLUTION OF FACILITY RECOMMENDATIONS Question #28: Recommendation accepted. The question is deleted due to no correct answer. The intended answer specified that the annunciator be confirmed as " lit" when it should have specified " reset" in accordance with system operating procedure No. 123 Section 5.1 (Rev. 29). Question #81: Recommendation not accepted. The RHR Loop B outboard injection valve will not auto-open unless the operator manually resets the shutdown cooling isolation signal. Therefor, the use of the phrase " manually reopen" is' correct, and the key answer is correct. The facility provided reference justification supports that manual action is required to open the injection valve. Question #96: Recommendation accepted; the question has two correct answers (a and c). Figure 8 of system description No.123, Rev. 3 for Bus 2B3 shows that bus 283 cannot be restored without restoring voltage to the secondaries of the startup and standby transformers thereby removing the LOOP diesel auto start signal. The A0P reference cited by the facility was not considered relevant since it did not state that the standby or startup transformers had t o be energized in order to energize the 283 bus. O l l l i i NUREG-1021 17 of 25 Interim Rev. 8, January 1997

ES-501 7 Attachment 1 Enclosure 5 SIMULATION FACILITY REPORT Facility Licensee: (Facility name) Facility Docket No.: (number) Operating Tests Administered on: (date) This form is to be used only to report observations. These observations do not constitute audit or inspection findings and, without further verification and review, are not indicative of noncompliance with 10 CFR 55.45(b). These observations do not affect NRC certification or approval of the simulation facility other than to provide information that may be used in future evaluations. No licensee action is required in response to these i observations. ' While conducting the simulator portion of the operating tests, examiners observed the following items (if none, so state): (EXAMPLES) , l IT.fH DESCRIPTION  : HPSI header B The pressure instrument read mid-scale regardless of actual pressure pressure. (PI-301) Head bubble During a scenario that caused a rapid depressurization during natural circulation, the vessel head level indication l indicated a void (bubble). The confirming indications i (i.e., pressurizer level and pressure) failed to verify or confirm the bubble. Steam The meter has been out of service for the last three generator A operating tests (approximately 18 months). wide range level NUREG-1021 18 of 25 Interim Rev. 8, January 1997

_ _ ...._ __ . _ _ _ . . _._..________._.-.____.m.._------ . ~ . - i I l ES-501 Sample License Letters Attachment 2 ? l NRC Letterhead ' Date LICENSE l (Anolicant's name) (Street address) (City. State. Zio code) Pursuant to the Atomic Energy Act of 1954, as amended; the Energy Reorganization Act of 1974, as amended; and Public Law 93-438, and subject to the conditions and limitations incorporated herein, the Nuclear Regulatory Commission hereby licenses you to manipulate all controls of the (Name of facility. facility license ninnher). Your License No. is OP-(nu=her). Your Docket No. is 55-(number). The effective date is (date). Unless sooner terminated, renewed, or upgraded, this license shall expire six years from the effective date. This license is subject to the provisions of Title 10, Section 55.53, of the Code of federal Regulations, with the same force and effect as if fully set forth herein. While performing licensed duties, you shall observe the operating procedures and other conditions specified in the facility license authorizing operation of the facility. The issuance of this license is based upon examination of your qualifications, including the representations and information contained in your application for this license. A copy of this license has been made available to the facility licensee. For the Nuclear Regulatory Commission, (Name and title of licensina official) ' Docket No. 55-(number) cc: (Facility representative who sianed the aoolicant's NRC Form 398) NUREG-1021 19 of 25 Interim Rev. 8, January 1997 l

ES-501 2 Attachment 2 NRC Letterhead G1R LICENSE l (Anolicant's name) (Street address) (City. State. Zio code) Pursuant to the Atomic Energy Act of 1954, as amended; the Energy j Reorganization Act of 1974, as amended; and Public Law 93-438, and subject to i the conditions and limitations incorporated herein, the Nuclear Regulatory l Commission hereby licenses you to direct the licensed activities of licensed operators at, and to manipulate all controls of the (Name of facility. facility license number). l Your License No. is SOP-(number). Your Docket No. is 55-(number). The effective date is (date). Unless sooner terminated or renewed, this license shall expire six years from the effective date. This license is subject to the provisions of Title 10, Section 55.53, of the l Code of federal Regulations, with the same force and effect as if fully set i forth herein. While performing licensed duties, you shall observe the operating procedures , and other conditions specified in the facility license authorizing operation l of the facility. You shall also comply with the following conditionisl: i You shall wear corrective lenses while performing the activities for which you are licensed. The issuance of this license is based upon exami> Mon of your qualifications, including the representations and information coMoned in your application for this license. A copy of this license has been made available to the facility licensee. For the Nuclear Regulatory Commission, (Name and title of licensina official) Docket No. 55-(number) cc: (Facility representative who sianed the anolicant's NRC Form 398) NUREr.~1021 20 of 25 Interim Rev. 8, January 1997

l 1 i ES-501 Proposed Denial Letter Attachment 3 NRC Letterhead '

;                                                                                       (date)

(Apolicant's name) (Street address 5 (C'tv. State. Z'o cQdt). I

Dear (Name):

This is to inform you that your grade on the Jwritten examination. operatina test. or both) taken on (date(s)), in connect 1on with your application for a

ireactor operator. senior reactor operator' license for the (facility name),

indicates that you did not pass that (exam- nation. test. or both). As a , result, it is proposed that your application be denied. Enclosed is a copy of the (written examination. operatino test. or both) results indicating those areas in which you exhibited deficiencies. (A copy of the master answer key

is also provided.)

If you accept the proposed denial and decline to request either an informal 4 NRC staff review or a hearing within 20 days, as discussed below this

proposed denial will become a final denial. You may then reapply for a license in accordance with Title 10, Section 55.35, of the Code of federal 4 Regulations (10 CFR 55.35), subject to the following conditions:

• a. Because you passed (a written examination. an operatina test) on (date), you may request a waiver of that portion.

b. Because you did not pass the (written examination. operatino test)  !

administered to you on (date), you will be required to retake that l portion. I i

• c. You may reapply for a license two months from the date of this l

letter.

  **      a. Because this is your (second. subseauenti examination failure, you will be required to retake both the written examination and the

operating test.

  **      b. You may reapply for a license (6. 24) months from the date of this letter.
  ***     a. Because you did not pass either the written examination or the operating test administered to you on (date(s)), you will be                              ,

required to retake both the written examination and the operating test.

  ***     b. You may reapply for a license (2. 6. 24) months from the date of this letter.

NUREG-1021 21 of 25 Interim Rev. 8, January 1997 i

ES-501 2 Attachment 3 If you do not accept the proposed denial, you may, within 20 days of the date of this letter, take either of the following actions: You may request an informal NRC staff review of the grading of your examination. Your written request must be sent to the Director, Division of Reactor Controls and Human Factors, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, D.C.- 20555. Your request must identify the portions of your examination that you believe were graded incorrectly or too severely. In addition, you must provide the basis, including supporting documentation (such as procedures, instructions, computer printouts, and chart traces), in as much detail as possible, to support your contention that certain of your responses were graded incorrectly or too severely. The NRC will review your contentions, reconsider your grading, and inform you of the results. If the proposed denial is sustained, you will have the op)ortunity to request a hearing pursuant to 10 CFR 2.103(b)(2) at t1at time. You may request a hearing pursuant to 10 CFR 2.103(b)(2). Submit your request, in writing, to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General Counsel, at the same address. Pursuant to 10 CFR 55.35, you may not reapply for a license until your license i has been finally denied. Failure on your part to exercise either of the above ' options within 20 days constitutes a waiver of your opportunity for informal review and your right to demand a hearing. For the purpose of reapplication under 10 CFR 55.35, such waiver renders this letter a notice of final denial of your application, effective as of the date of this letter. If you have any questions, please contact (name) at (teleobone number). Sincerely, (Name and title of licensino official) Docket No. 55-(number)

Enclosures:

As stated cc: (Facility reoresentative who sianed the applicant's NRC Form 398) CERTIFIED MAIL. RETURN RECEIPT REQUESTED

• Use for initial R0 or SR0 license applicants who passed either the written examination or the operating test but failed the other.

 **     Use for second and subsequent retake applicants.
 ***    Use for applicants who failed both the written and the operating test.

NUREG-1021 22 of 25 Interim Rev. 8, January 1997 9

ES-501 Post-Examination Check Sheet Form ES-501-1 1 1 , Facility: Date of Examination: 4 , Due Task Description Date Date(*) Complete 5(#) 1. Facility written exam comments or graded exams received and verified complete 10 2. Facility written exam comments reviewed and incorporated and NRC grading completed, if , necessary 15 3. Operating tests graded by NRC examiners 20 4. NRC Chief examiner review of written exam and operating test grading completed 7s '

   's        22     5. Responsible supervisor review completed                                            !

25 6. License and denial letters typed 28 7. Management (licensing official) review completed > 1 29 8. License and denial letters mailed 30 9. Facility notified of results 4 45 10. Examination report issued 45+ 11. Reference material returned after final resolution of any appeals J (*) Working days after the exit meeting, except as noted 1 (#) Working days after the written examination was administered NUREG-1021 2J of 25 Interim Rev. 8, January 1997

ES-501 Power Plant Examination Results Summary Form ES-501-2 PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY POWER PLANT EXAMINATION RESULTS

SUMMARY

Facility: Circle Plant Status: Hot / Cold Written Examination Date: Operating Test Date(s): NRC Examiners: OVERALL RESULTS Applicants: Total # # Passed  % Passad # Failed  % Failed R0 SR0 IleIVIDUAL RESULTS Name Docket # Type Written Operating Test (2) 55-( ) (1) Grade - A B C l NOTES: (1) 1 = RO; 2 = SRO-t; 3 = SRO-U; 4 = RO-Reteke; 5 = SRO-l-Retake; 6 = SRO-U-Retake; 7 = S RO-Fuel (2) P = Poseed; F = Failed; W = Waived PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY NUREG-1021 24 of 25 Interim Rev. 8, January 1997 O

ES-501 2 Form ES-St,i _ PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY 1 POWER PLANT EXAMINATION RESULTS

SUMMARY

(CONTINUATION SHEET) Facility: Examination Date(s): IlWIVIDUAL RESULTS Name Docket # Type Written Operating Test (2) 55-( ) (1) er a A B C J i

O NOTES:

(1) 1 = R0; 2 -SRO-t; 3 = SRO-U; 4 = RO-Retake; 5 = SRO-l-Retake; 6 = SRO-U-Retake; 7 = SRO-Fuel (2) P = Passed; F = Failed; W = Waived PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY b NUREG-1021 25 of 25 Interim Rev. 8, January 1997

i c t 1 (s

'   'j                                                ES-502                                           i PROCESSING REQUESTS FOR ADMINISTRATIVE REVIEWS AND HEARINGS AFTER INITIAL LICENSE DENIAL A. PURPOSE i

This standard specifies the NRC's policies and practices for processing informal staff reviews of initial license application denials, issuing final license application denials, and reapplying for a license after a final application denial is issued. B. BACKGROUND Operator license applicants who fail a written examination or operating test l administered pursuant to Title 10, Part 55, of the Code of Federal Regulations

           -(10 CFR Part 55) are notified of their failure in writing, including the                   ;

nature of the deficiencies noted. The notification letter constitutes a l proposed license denial and informs the applicant of his or her options as  ! specified in Section C below. One of these options is to seek reconsideration  ! of the proposed denial through the informal review process described in Section D below. If the proposed denial is sustained during the review, the applicant may request a hearing pursuant to 10 CFR 2.103(b)(2). The second option available to the applicant is to immediately request a hearing pursuant to 10 CFR 2.103(b)(2). t Applicants who fail a written examination or operating test may reapply

  \         pursuant to the provisions of 10 CFR 55.35, but only after a final denial has been issued with regard to their existing application. This may occur as a result of (1) the applicant's failure to respond to a proposed denial by requesting either an NRC staff review of the examination results or a hearing within the 20 days allowed, (2) the applicant's failure to request a hearing within 20 days of the letter from the NRC sustaining the initial proposed denial, or (3) the findings of a hearing requested by the applicant. The NRC will not accept a reapplication pursuant to 10 CFR 55.35 as long as a request is pending for either an informal NRC review or a hearing.

When the NRC denies an application for an operator license because the

      ,     applicant fails to meet the eligibility requirements in 10 CFR 55.31, the procedures that apply are similar to those for processing informal NRC reviews and license denials as a result of examination failures. The details of these procedures are described in Sections C.1 and D.1 below.

C. APPLICANT RESPONSIBILITIES

1. Proposed Aoolication Denial If an applicant still does not meet the eligibility requirements after providing any additional information requested by the NRC regional office in accordance with ES-202, he or she may then exercise either of b

V NUREG-1021 1 of 9 Interim Rev. 8, January 1997

ES-502 the following options within 20 days after the date on the proposed denial letter from the regional office:

a. Request reconsideration of the application denial. Such requests must be submitted to the Director, Division of Reactor Controls and Human Factors (DRCH), Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. The applicant's submittal must clearly state the basis for the request.

b. Request a hearing pursuant to 10 CFR 2.103(b)(2). Such requests must be submitted to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General Counsel, at the same address.

2. Proposed License Denial If an applicant receives a proposed license denial letter (refer to ES-501, Attachment 3) from an NRC regional office, he or she has 20 days from the date of the letter to exercise one of the following three options:

a. Do nothing. The proposed denial then constitutes a final denial, and the applicant may reapply pursuant to 10 CFR 50.35.

b. Submit a letter to the Director, Division of Reactor Controls and Human Factors, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, requesting that the NRC informally regrade the written examination, the operating test, or both in light of new information provided by the applicant. If the applicant submits such a request, the NRC will not consider a reapplication pursuant to 10 CFR 55.35 until a ,

final denial has been accepted. 1 The applicant's request for informal review must identify the item (s) for which additional review is requested and must include documentation supporting the item (s) in contention. The applicant must mail or deliver the request along with the supporting  ! documentation to the Director, DRCH, within 20 days after the date on the proposed denial letter. If the NRC determines that the applicant provided insufficient basis to justify passing grades on all sections of the licensing examination, the applicant may then request a hearing pursuant to 10 CFR 2.103(b)(2). Such a request must be submitted within 20 days after the date of the letter from the Director, DRCH, sustaining the proposed denial (refer to Section D.2.h). NUREG-1021 2 of 9 Interim Rev. 8, January 1997 Ol

4 ['

 \

ES-502

c. Request a hearing as provided by 10 CFR 2.103(b)(2). The hearing request must be submitted to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General Counsel, at the same address.

If the applicant requests a hearing, the NRC will not consider a reapplication pursuant to 10 CFR 55.35 until a final denial has been issued. If the applicant takes option "a" and declines the opportunity for an informal review or a hearing, or if the applicant takes option "b" but - does not request a hearing when the proposed denial is sustained by the Director, DRCH, the proposed denial becomes a final denial as of the date of the notification. The applicant may then reapply after the waiting period (specified in 10 CFR 55.35) following the date of the initial or sustaining notification, respectively. The NRC will not consider a reapplication pursuant to 10 CFR 55.35 if a request for an informal review or a hearing is outstanding. D. NRC RESPONSIBILITIES

1. Informal Review of ADolication Denial If the applicant requests an informal review in accordance with Section C.l.a, the Chief, Operator Licensing Branch (0LB), will evaluate the

 \.s        application and make a recommendation to the Director, DRCH, to sustain or overturn the denied application. The Chief, OLB, will prepare a letter (Attachment 1) for signature by the Director, DRCH. This letter          l will inform the applicant either that the eligibility requirements are           '

still not satisfied and he or she may request a hearing pursuant to 10 CFR 2.103(b)(2), or that the applicant will be allowed to take the license examination. <

2. Informal Review of Examination Results ,

1 If an applicant requests that the Director, DRCH, conduct an informal l review of his or her license examination in accordance with

Section C.2.b, the following actions should be completed within 45 days after receiving the applicant's request:

a. The Chief, OLB, will notify the appropriate NRC regional office and forward a complete copy of the review package to that region.

The Chief, OLB, will track the licensing action and prepare a letter for signature by the Director, DRCH, notifying the applicant that his or her examination is being reviewed.

b. Within five working days after receiving the review package, the NRC regional office shall evaluate the applicant's contentions.

If the contentions justify overturning the failure, the region

    .                 will inform the Chief, OLB, of its findings and issue the
 \    NUREG-1021                                 3 of 9       Interim Rev. 8, January 1997

ES-502 appropriate license (the license will not be backdated). OLB will then prepare a letter in the format of Attachment 2 to notify the applicant that the proposed denial was overturned and that a license will be issued.

c. If the NRC regional office sustains the original denial, it shall inform the Chief, OLB, of its decision within five working days.

The region shall also provide the Chief, OLB, with a written i summary and explanation of the grading changes that were made as a j result of its review. I

d. If the NRC regional office sustains the original denial, the Chief, OLB, will determine, based on the nature of the appeal, whether to review the appeal internally at OLB or to convene a three-person board to review the applicant's documented contentions. The board shall be impartial (i.e., it will not include anyone who was in any way involved with the applicant's I licensing examination) and will include three certified examiners, one of which will be designated chairperson. To promote objectivity, the three appeal board members will be drawn from different offices, whenever possible. The board may conduct its review in the appropriate regional office, at the facility where the operating test was administered, or by mail or telephone, depending on the extent of the applicant's contentions and the need for access to reference material (or the simulation facility '

in the case of operating test reviews). l

e. For written examinations, the board shall review the original grading of the applicant's examination, the reference material supplied by the facility licensee, and the contentions and supporting documentation provided by the applicant. The review shall focus on those portions of the examination that were contested by the applicant and were not regraded by the NRC regional office as part of the informal review process. i For operating tests, the review board shall evaluate the examiner's comments, the examination report, and the simulator scenarios that were administered. The board shall then review the applicant's contentions in light of the information and documentation provided for the review (e.g., plant system descriptions, operating procedures, logs, chart recorder traces, process computer printouts) to determine if the applicant's contentions have merit. The board should ensure that specific examples of unsatisfactory performance were used to document each unsatisfactory (U) rating and that all comments are technically and procedurally correct. '

The board will thoroughly document its findings and recommendations on each of the applicant's contentions. NUREG-1021 4 of 9 Interim Rev. 8, January 1997

ES-502

f. The review board should evaluate the results cf the NRC regional office's review conducted in accordance with Section D.2.b above.

If the board's findings and recommendations differ from those of the regional reviewer (s), the board chairperson shall discuss the matter with the original examiner and the regional office to determine the cause of the disparity. The board chairperson should also brief regional management on the board's preliminary findings and recommendations. Regional management may provide any additional information at that time, and the board will consider those concerns in its final recommendation.

g. The review board will submit its findings and a recommendation to sustain or overturn the license examination failure to the Chief, OLB. If the NRC regional office continues to have concerns, it should raise them to the Chief, OLB, who will make a final recommendation to the Director, DRCH.

h. The Director, DRCH, will consider the findings and recommendations

               .of the review board and make a decision whether to sustain or overturn the applicant's license examination failure. The Director, DRCH, will notify the applicant in writing that his or her proposed denial was overturned (Attachment 2, with direction that the NRC regional office should issue a license effective                         l immediately) or sustained (Attachment 3).

1. A branch chief or above will sign all letters informing an applicant of I an examination failure or an application denial. DRCH will include the appropriate regional licensing assistant on distribution for all correspondence that is generated in accordance with this standard. '

2. The facility licensee's authorized representative who signed the license applications shall be sent a copy of all external correspondence generated as a result of this standard.

3. The Chief, OLB, is responsible for keeping regional management informed

                                                                         ~

of review requests from license applicants.  ;

4. All correspondence referenced in this standard shall be sent to the l applicant via certified mail, return receipt requested.

F. ATTACHMENTS / FORMS

 ' Attachment 1,       " Initial Application Denial / Approval from DRCH" Attachment 2,       " Sample License Notification from DRCH" Attachment 3,        " Proposed License Denial from DRCH" NUREG-1021                                5 of 9         Interim Rev. 8, January 1997

ES-502 Initial Application Denial / Approval Attachment I from DRCH NRC Letterhead (Date) (Applicant's name) (Street address) (City. State. Zio code)

Dear (Name):

This letter responds to your request for reconsideration of the denial issued to you on (date), in connection with your application for a (reactor operator or senior reactor operatori license for the (facility name). Our review of your application indicates that you (* still do not meet) or (** have satisfied) the eligibility requirements of Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55). [* (Insert a discussion of the deficiencies and indicate which oart of 10 CFR 55.31. ES-202. NRC-aporoved facility trainino orocram. or Reaulatory Guide 1.8 is involved.) When you have met the requirements of 10 CFR 55.31, you may submit another application to the appropriate regional office.] [* If you do not accept this denial, you may, within 20 days of the date of this letter, request a hearing pursuant to 10 CFR 2.103(b)(2). Submit your request, in writing, to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General Counsel, at the same address.] [** Region (number) will add your name to the list of applicants to be examined for a (reactor operator or senior reactor operator) license for the (facility name) during the week of (date). If you have any questions, please contact (name.). at (telechone number). Sincerely, Director, Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation Docket No. 55-(number) cc: (Facility representative who sioned the aDolicant's NRC Form 398) CERTIFIED MAIL, RETURN RECEIPT REQUESTED [* / ** Applicable to sustained and overturned applications, respectively.] NUREG-1021 6 of 9 Interim Rev. 8, January 1997 l 1

   .= _ . ~ . _ . - - . . - .                - . - . - - . - . -                -      . - - . -                . . -    . . _ - .-.

I qb) ES-502 Sample License Notification from DRCH Attachment 2 i-NRC Letterhead I (Date) a . (Apolicant's name) i (Street address) (City. State. Zio code)

Dear (Name):

In response to your letter of (date), the staff of the U.S. Nuclear Regulatory Commission (NRL) has reviewed the grading of the (written examination and/or operatina test) administered to_you on (date(s)) and reconsidered the proposed denial issued to you on (date). In light of the additional information you supplied, the staff has determined that you passed the (written examination and/or operatina test) and satisfy the requirements of Title 10, Section 55.33(a), of the Code of Federal Regulatfons (10 CFR 55.33(a)) for approval of your license application. Region (number) will issue your (reactor operator or senior reactor operator) license pursuant to 10 CFR 55.51 and forward it to you under separate cover. [For your information, I am enclosing a copy of the staff's resolution of each of your (written examination and/or operatina test) comments.] If you have any questions, please contact (name) at (telephone number). Sincerely, Director, Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation Docket No. 55-(number)

Enclosure:

As stated [] Include only if the region does not forward discussion of comment I resolutions to applicant. NUREG-1021 7 of 9 Interim Rev. 8, January 1997

ES-502 Proposed License Denial Attachment 3 from DRCH NRC Letterhead (Date) (Apolicant's name) (Street tddress) (City. State. Zio code)

Dear (Name):

In response to your letter of (date), the staff of the U.S. Nuclear Regulatory Commission (NRC) has reconsidered the proposed denial issued to you on (date) and reviewed the grading of the (written examination and/or operatina test) administered to you on (date(s)), in light of the information you supplied. The staff finds that you did not pass the (examination and/or test). (The results of our review are enclosed.) Consequently, the proposed denial of your license application is sustained. If you accept the proposed denial and decline to request a hearing within 20 days as discussed below, the proposed denial will become a final denial. You may then reapply for a license in accordance with Title 10, Section 55.35, of the Code of federal Regulations (10 CFR 55.35), subject to the following conditions:

• a. Because you passed (a written examination. an operatina test) on (date), you may request a waiver of that portion.
• b. Because you did not pass the (written examination. operatina test) administered to you on (date), you will be required to retake that portion.
• c. You may reapply for a license two months from the date of this letter.
  • a. Because this is your (second. subseauent) examination failure, you will be required to retake both the written examination and the operating test.
  • b. You may reapply for a license (6. 24) months from the date of this letter.
    • a. Because you did not pass either the written examination or the operating test administered to you on (date), you will be required to retake both the written examination and the operating test.
    • b. You may reapply for a license (2. 6. 24) months from the date of this letter.

NUREG-1021 8 of 9 Interim Rev. 8, January 1997

I O h ES-502 2 Attachment 3 If you do not accept the proposed denial, you may, within 20 days of the date i of this letter, request a hearing in accordance with 10 CFR 2.103(b)(2).  ! Submit your request in writing to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General  ; Counsel, at the same address.  ; Failure on your part to request a hearing within 20 days constitutes a waiver of your right to demand a hearing. For the purpose of reapplication under 10 CFR 55.35, such waiver renders this letter a notice of final denial of your application, effective as of the date of this letter. If you have any questions, please contact (name) at (teleohone number). J Sincerely, t i Director, Division of Reactor Controls and Human Factors i Office of Nuclear Reactor Regulation Docket No. 55-(number)

Enclosure:

As stated ( cc: (Facility representative who sianed the aoolicant's NRC Form 398) i CERTIFIED MAIL, RETURN RECEIPT REQUESTED

• Use for initial R0 or SR0 license applicants who passed either the written examination or the operating test but failed the other.

     **     Use for second and subsequent retake applicants.                                       ,

l

     ***    Use for applicants who failed both the written examination and the operating test.

1 i ('s ' NUREG-1021 9 of 9 Interim Rev. 8, January 1997  !

j ES-601 1 CONDUCTING NRC REQUALIFICATION EXAMINATIONS A. PURPOSE

Title 10, Section 55.59(a), of the Code of Federal Regulatfons requires a

licensed operators and senior operators to complete a requalification program developed by the facility licensee and to pass a comprehensive requalification

written examination and an annual operating test. In lieu of accepting the i

facility licensee's certification that the o mrator has passed the required examinations and tests administered within tie facility licensee's Commission- , approved program, the NRC may administer a comprehensive requalification ! written examination and an annual operating test. i This standard provides general guidance and requirements for conducting NRC requalification examinations. In addition this standard provides guidance.and ] procedures for evaluating the facility licensee's licensed operator i requalification training program to ensure it is effectively maintaining the ! competency of the licensed operators. Specific guidance and requirements for ! conducting the comprehensive requalification written examinations and the j annual operating tests (including both the plant walk-through and dynamic l simulator sections) are provided in ES-602 through ES-604. These standards are not a substitute for the operator licensing regulations and are subject to j revision and other changes to the internal operator licensing program policy. l B. BACKGROUND

Section 306 of the Nuclear Waste Polfey Act of 1982 (NWPA) authorized and i

directed the NRC to promulgate regulations, or other appropriate guidance, for training and qualifying nuclear power plant operators. Those regulations were i to include requirements governing the administration of requalification  !

j. examinations and operating tests at nuclear power plant simulators. The NRC's i requalification evaluation program consists primarily of periodic, on-site l requalification inspections, supplemented with NRC examinations at fn.Wties

where the NRC believes that ineffective training is causing opers% e

! commit errors. The NRC's Office of the General Counsel has cons .onat i this program satisfies the statutory requirements in Section 306 ' the NWPA. The oversight program will requirt the NRC to actively oversee the facility ! licensees' requalification training programs, and the Commission's regulations s will continue to contain legally binding requirements that apply to the l i conduct of operator requalification examinations by facility licensees. l When determining the scope of the requalification inspection and examination activities at a facility, regional managers will consider overall facility performance (e.g., systematic assessment of licensee performance (SALP) , ratings), the results of the NRC's inspection programs (e.g., requalification, emergency operating procedure, and resident), the results of routine initial and requalification examinations, and other factors. Generally, only the inspection requirements of Inspection Procedure (IP) 71001, " Licensed Operator , Requalification Program Evaluation," will need to be met; however, augmented  ! NUREG-1021 1 of 38 Interim Rev. 8, January 1997

l l ES-601 activities can be initiated in accordance with program office guidance when necessary to ensure safe plant operation. Those activities could include a training program inspection in accordance with IP 41500, ' Training and Qualification Effectiveness," operational evaluations of on-shift crews, or NRC examinations conducted in accordance with this series of Examination Standards. The NRC will conduct requalification examinations only when it has lost confidence in the facility licensee's ability to conduct examinations, or when the staff believes that the inspection process will not provide the needed insight. Regional management should consider conducting requalification examinations or operational evaluations when any of the following conditions exist: requalification inspection results indicate an ineffective operator requalification program operator errors are a major contributor to operational problems a SALP Category 3 rating in plant operations is attributed to operator performance allegations have been raised regarding significant training program deficiencies The decision to conduct NRC examinations should be implemented through the normal resource planning system, such as the plant performance review (PPR) and master inspection planning (MIP) processes, since an inspection activity will be replaced with examinations that are more resource-intensive. Using the existing inspection planning process will ensure that the regional office and the Office of Nuclear Reactor Regulation (NRR) consider the need to conduct examinations, as well as the alternative, expanded inspection tools, when allocating the required resources. Operational evaluations should be considered as a reactive effort based on immediate safety concerns. i C. SCOPE The NRC-conducted requalification examinations will measure the effectiveness l of a requalification program by evaluating the ability of the facility  ; licensee to adequately prepare written examination questions, job performance measures (JPMs), and simulator scenarios, as well as its ability to properly evaluate its operators' performance. The examination procedures are based on a systems approach to training (SAT) program, as defined in 10 CFR 55.4. To the extent possible, these procedures rely on existing requalification program standards for developing and implementing the NRC examination. The SAT approach allows the NRC to conduct requalification examinations that are fundamentally consistent with existing facility-developed programs. As such, this approach reduces the impact un the facilities and improves the reliability of the NRC assessment of requalification training programs. NUREG-1021 2 of 38 Interim Rev. 8, January 1997

1 1 ES-601 The NRC-conducted requalification examination will normally be composed of three parts, including a two-section, open-reference written examination, a walk-through evaluation, and a dynamic simulator evaluation. The three examination parts are further described in ES-602, 603, and 604, respectively. If the facility licensee's requalification program uses an examination structure or methodology different from that described herein, the regional office should consult with the Operator Licensing Branch (0LB) to determine the appropriate examination procedure. To the extent practical, the examination will be based on the facility licensee's requalification program and learning objectives. The facility licensee is expected to use the plant-specific job and task analyses (JTAs) as the basis for developing the examination materials and substantiating the importance rating factors for each task. The facility licensee may also use NUREG-1122 or -1123, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors (or Boiling Water Reactors)," for additional guidance in identifying job-specific importance rating factors. The use of a JTA will result in more tichnically sound and operationally oriented examinations. An examination team, composed of NRC examiners and facility representatives, will develop, review, and conduct each requalification examination. Parallel evaluation of operator performance by NRC examiners and facility evaluators will enhance the ability of the NRC to assess both individual and program performance. The administrative guidelines and procedures for conducting an NRC requalification examination are outlined in Attachment 1, " Examination Timetable." ' D. EXAMINATION PREPARATIONS

1. Communication l

a. When the NRC determines that it is necessary to conduct requalification examinations, the regional office will notify the facility licensee to be evaluated at least 90 but preferably 120 days before the examination start date, using the corporate notification letter shown in Attachment 2. If possible, the NRC will schedule the site visits to coincide with the requalification training cycle of the facility. Depending on the number of operators and crews at the facility, it may be necessary to conduct the examinations over a period of two or more weeks to attain the required sample size. The requalification training cycle, referenced here and throughout NUREG-1021, is that continuous period of time (not to exceed 24 months) within which the facility licensee conducts its operator requalification training program.

V NUREG-1021 3 of 38 Interim Rev. 8, January 1997

ES-601 If the purpose of the examination is to retest operators who previously failed an NRC-conducted requalification examination, the regional office should modify the corporate notification letter, as appropriate.

b. The facility licensee is expected to respond to the corporate notification letter at least 60 days before the evaluation by submitting the materials and information requested in the letter.

The facility licensee may request that the NRC chief examiner or another NRC representative meet with the operators to be examined and with appropriate facility licensee managers. Such a meeting , should be scheduled during the examination preparation week as  ! discussed in Section D.5. J

c. At least 30 days in advance of the examination, the NRC will confirm with the facility licensee which operators have been j selected to participate in the evaluation.

l l

2. Selection of Operators

                                                                                 )

a. The NRC expects facility licensees to train and examine their operators in the same crew configurations with which they normally operate the plant. Generally, the NRC expects the crew to include ,

no more than five operators, but it will consider larger crews on a case-by-case basis. At times, to ensure an adequate sample size for the examination, the examination team may configure crews that do not routinely perform shift duties together. Mixed crews of shift and non-shift operators should not be configured unless the facility licensee routinely evaluates mixed crews in its requalification training program, or the facility licensee's normal crew size is so large that separating a normal crew is required for examination purposes.

b. All crew members for requalification dynamic simulator examinations must be currently licensed on the facility and up-to-date in the facility licensee's requalification program.

c. The selections will be made to minimize perturbation on the facility licensee's schedules and plant operations. Operating crew (s) in training will be given first priority during the examination week (s). If the facility's program is being reevaluated after an unsatisfactory evaluation, the selection process should favor operators who either failed their previous NRC-conducted examination or were not previously examined.

d. During retake examinations, operators who have passed an NRC requalification examination may be included in the dynamic NUREG-1021 4 of 38 Interim Rev. 8, January 1997

l l ES-601 simulator crew evaluation. These operators will not be required to take the written or walk-through portions of that examination. The operators' performance on the simulator examination will be evaluated in accordance with the guidance of ES 504.  ; I i ) e. A shift technical advisor (STA) may be added to the crew if the '

                   -facility normally uses an STA during requalification training.                       ,

The NRC expects the STA's duties and responsibilities to be the  ! l same as those assigned during requalification training and plant operations.

f. The NRC will review the list of crews and operators submitted by ,

the facility licensee, and recommend any necessary changes. '

3. Reference Material l

a. The facility licensee is expected to supply the reference materials requested in the corporate notification letter (see Enclosure 1 to Attachment 2). The NRC will evaluate the i facility's reference materials for adequacy in advance of the '

scheduled preparation week, using the " Evaluation Checklist for l Facility Reference Material," Form ES-601-2. l

b. The NRC reserves the right to prepare the requalification examinations using the NRC's examination question banks and Os facility background reference materials if the facility licensee's test items are inadequate for examination preparation. If the NRC prepares the examination, the NRC may require additional reference material comparable to that listed in ES-201, Attachment 2, l " Reference Material Guidelines for Initial Operator Licensing Examinations."

c. The facility is expected to provide a sample plan that meets the guidelines of Attachment 3, " Examination Sample Plan," for the RRC's use in developing the examination.

4. Examination Team Selection

a. The NRC wil? contribute no fewer than two examiners to the examination team. The regional office should consider assigning additional examiners if the operating crews for the dynamic  !

simulator examinations contain five or more operators. To promote consistency in requalification program administration, regional office management should try to assign an examiner who participated in a prior requalification inspection or examination l at the facility to be part of the NRC examination team. l . 4 In most cases, it is expected that OLB will send a representative to observe the examination process or an examiner to participate j as an additional member of the msination team. OLB will work NUREG-1021 5 of 38 Interim Rev. 8, January 1997

ES-601 with the responsible regional supervisor to make the necessary I arrangements.

b. The facility licensee is expected to provide an employee to work with the NRC as part of the requalification examination team. The employee must be drawn from the operations staff, and must be an active senior reactor operator (SRO) as defined in 10 CFR 55.53(e) or (f). The NRC encourages the facility licensee to designate ,

another employee from the training staff to be a member of the l examination team. This employee should also be a licensed SRO,  ! but may be a certified instructor. If the facility licensee l desires, and the chief examiner agrees, adoitional employees from l the operations or training staffs with qualifications comparable to the other facility examination team members may be included on the examination team. The function of these examination team members is to provide facility-specific technical assistance to the NRC in developing and reviewing the written examination items, plant walk-through topics, and dynamic simulator scenarios. If necessary, the facility representatives may participate as facility evaluators in conducting the operating test or written examination. However, the facility representatives should only be used as evaluators if they routinely perform that function during the administration of the facility licensee's requalification program.

5. Examination Development The facility licensee may develop proposed written examinations and operating tests and forward them to the NRC as part of its reference material submittal (see Attachment 2). In accordance with 10 CFR 55.59(a)(2)(ii), the facility licensee must ensure that the operating tests require the operators to demonstrate an understanding of and the ability to perform the actions necessary to accomplish a comprehensive sample of the items specified in 10 CFR 55.45(a)(2) through (13),

inclusive, to the extent applicable to the~ facility. Approximately two weeks before the scheduled examinations, the NRC examiners will visit the facility to make final preparations for the examination. The written, walk-through, and dynamic simulator examinations will be developed in accordance with ES-602, ES-603, and ES-604, respectively. The examination should distinguish between reactor operator (RO) and SRO knowledge and abilities to the extent that the facility training materials allow the examiners to make these distinctions. The NRC examiners will rely upon the facility licensee's examination team members for site-specific technical assistance in developing, reviewing, and validating the written examination static scenarios and items, plant walk-through topics (JPMs), and dynamic simulator scenarios. NUREG-1021 6 of 38 Interim Rev. 8, January 1997

l ES-601 The chief examiner and the responsible regional supervisor will determine the required length of time on site and the required number of examiners. This determination will be based on the experience of the examiners, the quality of the facility licensee's testing material, and the level of effort required to develop new test items. If requested by the facility licensee, the chief examiner will brief the operators and managers about the requalification examination process. The examiner will use this time to explain the examination and grading processes and to respond to any questions that the operators may ask about the NRC's examination procedures. If the schedule does not allow them to meet during the preparation week, they may meet at any mutually agreeable time.

6. Examination Security l

                                                                                                                           )

To ensure examination security, each facility representative who i acquires knowledge of the content of the NRC requalification examination before it is administered will be subject to the security restrictions described below from the time he or she first acquires the specific knowledge until the examination exit meeting.

                                                                                                                           )

To the maximum extent possible, only the examination team members and a  ! simulator operator should be given specific knowledge about the content of the examination. The facility evaluators should be given the package O. of simulator scenarios and JPMs the week before the examination to allow them to prepare for their evaluation, including coordinating the use of l the simulator to perform JPMs and scenarios. If the facility licensee i submits a proposed examination, those who participate in developing the i examination become subject to the security restrictions when their l involvement begins. Also, if facility representatives other than the ' examination team members are used to time validate the written l examination, they too become. subject to the security restrictions as soon as they are exposed to the examination questions. l Facility representatives who acquire specific knowledge of the NRC examinations will sign Form ES-601-1, " Examination Security Agreement," or a reasonable facsimile before their examination involvement begins and again after the examination process is complete (i.e., following the exit meeting). E. PROGRAM AND OPERATOR EVALUATION PROCEDURES

1. Examination Administration

a. For each selected operator, conduct a requalification examination j using ES-602, ES-603, and ES-604 for the written, walk-through, I and simulator portions of the requalification examination, respectively. Document operator performance on Form ES-601-5.

i NUREG-1021 7 of 38 Interim Rev. 8, January 1997 l

                  ~ -- -            ,                  - - , , - -         , , ,     .-    r- --,-  -
                                                                                                                , - - ,e

l l I l ES-601

b. The number of persons present during an operating test should be limited to ensure the integrity of the test and to minimize  ;

distractions to the operators. Under no circumstances will l another operator be allowed to witness an operating test. Operating tests are not to be used by the facility licensee as training vehicles for future requalification examinations.  :

                                                                                   )

Other examiners may observe an operating test as part of their  ! training or to audit the performance of the examiner administering 1 the operating test. The chief examiner may permit others (such as l resident inspectors, regional personnel, researchers, or NRC  ! supervisors) to observe an operating test if the applicant does j not object to the observers' presence. Deviations from this l policy must be approved, in advance, by the Chief, OLB. Other non-NRC personnel (e.g., representatives from the Institute of Nuclear Power Operations (INPO) or the Nuclear Energy Institute (NEI)) may observe the operating tests with prior approval from the Chief, OLB. The chief examiner will control the observers' activities in accordance with guidance provided by OLB.

2. Examination Gradino

a. The facility licensee is expected to grade the written l examinations and operating tests in parallel with the NRC  !

examiners.

b. The facility evaluators are expected to provide preliminary pass / fail results for the simulator and walk-through portions of the examination by the end of each day, and the final results before the exit briefing or at the end of each examination week for multi-week examinations.

c. The NRC will notify the facility licensee immediately if any operator's performance on the examination is sufficiently poor to require immediate removal from licensed activity. The NRC will also notify the facility licensee of the results of the examination in accordance with ES-605.

d. The facility licensee will provide the NRC with the final results j of the written examinations and an overall summary of the  !

examination results within two weeks after the exit meeting. l l

3. Evaluation of Reoualification proarams ,

1 A requalification program evaluation requires a minimum sample size of I 12 operators. The sample size is determined by counting the number of l operators taking the dynamic simulator examination. This total includes l those operators who only participate in the simulator examination for l the purpose of meeting crew composition requirements, but excludes those j i NUREG-1021 8 of 38 Interim Rev. 8, January 1997 1 a

l l i $ ES-601 operators who are b0 og recxemined a#ter falling a previous NRC-

conducted examinat4n.

. In instances where fewer than one-half of the operators taking the I dynamic simulator examination complete the entire examination, the regional supervisor will determine whether a valid program evaluation can be made. In these instances, the regional supervisor will contact the Chief, OLB.

a. A satisfactory requalification program meets each of the following criteria:

(1) At least 75 percent of the operators must pass all portions of the examination in which they participate. The pass rate is determined by dividing the number of operators that pass all portions of the examination in which they participate by  ; the total number of operators in the sample. i In the event of a crew failure, only those operators who receive an unsatisfactory evaluation in the individual follow-up evaluation will be counted when calculating the operator pass rate. When calculating the pass rates, fractions should be rounded .i O up to the next highest number. For example, if 15 operators are evaluated, 75 percent passing would be 11.25; thus,11 of 15 passing would not meet the 75 percent requirement, but i 12 would. (2) At'least two-thirds (66 percent) of the crews must pass the simulator examination. For requalification examinations with more than three crews participating, three-of-four, or four-of-five crews must pass to satisfy this requirement.

b. The following areas will be considered in the overall program evaluation and may be used to identify facility weaknesses that will be documented in the examination report:

(1) The facility evaluators do not concur with the NRC examiners on all unsatisfactory crew evaluations. (2) More than one facility evaluator is determined to be unsatisfactory. Section D of Appendix C provides guidance that examiners should use to assess evaluator competence. (3) The facility licensee failed to train and evaluate an operator in all positions permitted by the individual's license. (For instance, the facility is required to train O NUREG-1021 9 of 38 Interim Rev. 8, January 1997

ES-601 and evaluate an SRO in the RO position as well as in directing operators.) An SRO will not be required to perform R0 activities during the simulator portion of the operating test; however, his or her performance will be evaluated if the facility normally places the SR0 in a shift R0 position during the simulator examination. Otherwise, R0 skills wil) be evaluated during the performance of JPMs. (4) The facility licensee has insufficient administrative controls to preclude an R0 or SR0 with an inactive license from performing licensed duties. Operators must meet the requirements of 10 CFR 55.53 to restore an inactive license to active status. (5) The facility licensee has insufficient quality control of its examination bank. The NRC will evaluate the facility's performance in this area if post-examination changes to facility-developed test items result in significant modifications or deletions of more than 10 percent of the questions on the written examination. (6) The facility licensee's failure decisions are not as conservative as the NRC's. To ensure that the rationale for the evaluation is fully understood, the NRC will review with the facility managers any case where the facility licensee passes an operator failed by the NRC. In addition, the NRC will assess whether the facility licensee's evaluations are conducted in accordance with documented facility guidance and whether facility managers periodically assess their evaluation process. The NRC also expects the facility program to explicitly link an operator's examination failure with unsafe performance. In this way, all facility failures and NRC failures will agree. In certain instances, the facility licensee's program may have operator performance standards that are not explicitly linked to unsafe performance, and thus do not meet the threshold stated in these standards for the I operator to fail the examination. In such instances, the facility licensee is expected to differentiate iailures in which the operator performed at an unsafe level from those in which the operator failed for reasons other than safety (i.e., not meeting higher facility-established performance standards). In these instances, operators identified as failing for safety reasons would also be considered NRC failures. NUREG-1021 10 of 38 Interim Rev. 8, January 1997

l t l l l l

4. Evaluation of Ooerator Performance To pass the NRC-conducted requalification examination, the operator must pass a written examination and an operating test consisting of a walk-through examination and a dynamic simulator examination. These examinations are developed and administered in accordance with ES-602, ES-603, and ES-604, respectively, unless OLB authorizes the regional office to use the facility licensee's alternative examination methodology. To pass the operating test, the operator must also be a member of a crew that passes the dynamic simulator examination.

F. UNSATISFACTORY OPERATOR OR PROGRAM EVALUATION

1. Actions Followina an Unsatisfactorv Operator Evaluation In all cases, a facility licensee's administrative procedures should ensure that an operator who fails a requalification examination is removed from licensed duties, given remedial training, and reexamined before being allowed to return to licensed duties. This also applies to an SRO who performs only R0-level duties at the facility when the failure is caused solely by activities involving SR0 responsibility.

ES-605 contains the procedure for notifying the operator about his or her performance on the requalification examination, as well as guidance about the actions to be performed for an operator to return to licensed duty. ' The regulation (10 CFR 55.57(b)(2)(iv)) that required an operator to pass an NRC-administered requalification examination as a prerequisite for license renewal has been deleted. Nonetheless, it would be inappropriate to renew the license of any operator who failed to pass any NRC-conducted requalification examination, without some level of NRC involvement in the retesting process. The amount of NRC involvement may include conducting the retest in accordance with the appropriate Examiner Standard (s); inspecting the facility licensee in accordance with IP 71001, " Licensed Operator Requalification Program Evaluation," as it retests the operator; or reviewing the reexamination prepared by

 ,       the facility licensee. The regional office, in consultation with OLB, will determine the appropriate level of involvement on a case-by-case basis depending on the quality of the facility licensee's program. As long as the operator submits a timely renewal application, the term of the license will continue until the renewal requirements are satisfied or the operator fails three NRC-conducted examinations as discussed in ES-605.

If an operator who failed a requalification examination is not prepared i for a reexamination after :;ix months of remedial training, the regional l office will request the following information from the facility licensee: lO NUREG-1021 11 of 38 Interim Rev. 8, January 1997

ES-601 confirmation that the facility licensee still has a need for the individual's license l the expected completion date of the operator's remedial training and when the facility licensee will be ready to administer its retake examination assurance that the operator will not be returned to licensed duties until he or she successfully retakes the examination (or portion thereof) administered by the facility licensee with a satisfactory requalification program or in accordance with the provisions of the confirmatory action letter (CAL) if the facility licensee has an unsatisfactory program and the NRC has not determined it to be " provisionally satisfactory." The NRC will inform the facility licensee that a' comprehensive requalification examination may be necessary if the operator is not ready to take a retest within one year after failing the examination.

2. Actions Followina an Unsatisfactory Reaualification Proaram Evaluation The following actions will be taken for all requalification programs evaluated as unsatisfactory:

a. The facility licensee is expected to identify program deficiencies and corrective actions to improve operator performance. The NRC will use a CAL to establish a formal dialogue and to document the facility licensee's corrective action commitments.

An operator who fails the requalification examination, as determined by the NRC, will be subject to an NRC-administered reexamination before resuming licensed duties. An operator whose performance does not meet facility stand:rds, as determined by the facility licensee, is expected to be remediated and reevaluated by the facility licensee in accordance witL "' provisions of the facility licensee's requalification progi - The NRC will review and/or monitor the reexamination to ensure the adequacy of the facility licensee's requalification program.

b. The NRC will schedule a meeting with senior facility managers to review the examination results, the identified deficiencies and their root causes, the proposed corrective actions and the l schedule for their implementation and the need for follow-up inspections and examinations. (Refer to Section F.3 for '

additional guidance for conducting augmented inspections.) ) The Regional Administrator will evaluate the examination and I inspection results and make a decision regarding the continued l operation of the facility and possible enforcement action against ' NUREG-1021 12 of 38 Interim Rev. 8, January 1997 l

l l ES-601 l the facility licensee. At a minimum, the Regional Administrator j should consider the following factors when making this  ! determination the resuits and corrective actions from previous program l l evaluations the significance of generic performance deficiencies identified during the program evaluation recent SALP ratings recat facility events that relde to licensed operator performance recommendations by the NRC staff (including the results of any operational evaluations and inspections)

c. If the unsatisfactory program evaluation is caused by operator performance deficiencies, an operational evaluation is required.

The operational evaluation is intended to help the Regional Administrator determine whether the facility's remaining operating crews are suitably qualified to continue to operate the facility. In this case, the facility licensee identifies the individual operators and shift crews it proposes to use to continue plant i operations. The regional office may choose not to evaluate those operators who passed their most recent NRC-conducted initial or requalification examination within the past 12 months. However, the regional office will evaluate all other operators in those areas noted as operational deficiencies during the requalification examination, regardless of whether they have already passed or not yet taken the facility-administered requalification examination. I The regional office will conduct the operational evaluations in  ! accordance with the guidance in ES-603 and ES-604, as applicable. l If the facility licensee proposes to use a shift crew that is significantly different from its normal configuration, even though ' all of the operators may have recently passed an NRC-conducted examination, the regional office may perform an operational evaluation of this crew. l The regional office should schedule the operational evaluation as l soon as possible after determining that the facility licensee's  ; requalification program is unsatisfactory. The evaluation should not be delayed to accommodate the facility's operating schedule, - the completion of programmatic corrective actions, or the completion of remedial training for operators who failed the requalification examinatica. The operational evaluation may identify further program deficiencies that may need to be reflected in the CAL discussed in Section F.2(a) or that may NUREG-1021 13 of 38 . Interim Rev. 8, January 1997

ES-601 warrant additional inspection by the NRC. Additional operator weaknesses that require .emediation may also be identified.

d. The NRC will review the corrective actions the facility is to perform, the expected follow-up actions by the NRC, and the schedule for each.

As part of the follow-up activities, the NRC may conduct additional operational evaluations, requalification retake examinations, and augmented inspections, as necessary. Before l these activities, the NRC will verify that the facility licensee has completed the applicable corrective actions, and will obtain a certification of crew readiness from the facility managers. Regional managers should consider using a new chief examiner and i having examiners from other regional offices or OLB participate on those operational evaluations and requalification retake examinations that have restart approval implications.

e. The Regional Administrator will incorporate into the decision concerning follow-up activities any extraordinary circumstances surrounding the examination activities that may have a bearing on the validity of the examination results.

f. Once the NRC determines that a requalification program is unsatisfactory, the program will remain unsatisfactory until the facility licensee completes all identified corrective actions agreed upon by the NRC for restoring the program to satisfactory status, and the NRC completes all related follow-up activities.

For purposes of allowing facility examiners to perform reexamination functions, howeven, a facility my attain a status of " provisionally satisfactory" provided that tim facility has completed to the NRC's satisfaction all short- and intermediate-term corrective actions agreed on with the NRC. Once the NRC determines that the facility licensee has , satisfactorily implemented these corrective actions m. Regional Administrator or designee will determine whether to permit the facility to reexamine all operators who failed the NRC-conducted reoualification examination for the purpose of returning the operators to licensed duties. Any operator who fails the NRC-conducted exanination still needs to pass a future NRC-administered (i.e., conducted, inspected, or approved, as appropriate) requalification examination to renew the license. Long-term corrective actions are expected to be completed before the NRC's next requalification program evaluation. NUREG-1021 14 of 38 Interim Rev. 8, January 1997

l i i

                                                                                            \

ES-601 , To attain a satisfactory rating following ao unsatisfactory evaluation, the subsequent requalification program evaluation, , with a sample size of at least 12 operators, niust satisfy the l passing criteria in Section E.3. ' The Regional' Administrator or designee may specify additional actions, ' as appro however,priat;. The specific this sequence sequence of events of actions corresponds is not regional to a typical critical;  ; response to an unsatisfactory program evaluation. The Regional Administrator or designee should defer determining whether a plant

shutdown is required until he or she reviews all factors listed in i

Section F.2(b) above.

3. A m nted Insoection Guidelines 4

i If it is determined that an augmented requalification program inspection . is required, regional management shall define its scope and depth based i upon the nature of the deficiencies. The regional office should < , consider the following activities in addition to those specified in l Section F.2. l  ;

a. The regional office may conduct augmented inspection coverage of l all shifts. The inspection procedures for shift coverage should  ;

i be used as appropriate. Inspection activities should devote ' l O particular attention to the following areas: j O

• operator performance and attitude 1

l operator overtime  : management oversight l shift staffing i b. The regional office may develop a long-term training program i inspection' plan based on Inspection Procedure (IP) 41500, ! " Training and Qualification Effectiveness." Such an inspection j plan may include the following activities: ongoing status reviews of requalification training effectiveness, with an emphasis on known program deficiencies and implementation of short-term corrective actions an inspection to determine the root cause(s) for the unsatisfactory requalification program evaluation, and to verify that the facility licensee's proposed corrective action plan should preclude or minimize the probability of recurrence O NUREG-1021 15 of 38 Interim Rev. 8, January 1997 i

l ES-601 an inspection to evaluate the adequacy of the facility ) licensee's corrective actions, and to determine the effectiveness of the facility licensee's SAT-based requalification program l l

c. The regional office may convene an enforcement panel to determine '

if action is warranted on the basis of the requirements of 10 CFR 50.54, paragraph (1-1). The potential exists that a requalification program rated unsatisfactory on two successive NRC i evaluations does not meet the minimum requirements of 10 CFR l 55.59(c) as required by 10 CFR 50.54(1-1). The basis for any l proposed enforcement action will be the inadequate corrective l action or requalification program element deficiencies (identified ' by the inspections related to Section F.3(b)) which led to the successive requalification examination failures. l G. REQUALIFICATION PROGRAM EVALUATION REPORT After the Regional Administrator or designee approves the requalification examination results, the regional office will prepare a final requalification program evaluation report. A copy of the written examination need only be included in the program evaluation report if the report addresses written examination problems. The regional office will issue the report within 30 days following receipt of the facility licensee's final results or the examination exit meeting, whichever is later, and will place a complete copy of the report in the facility's requalification file. The chief examiner is responsible for completing Forms ES-601-3 and ES-601-4, the " Power Plant Requalification Results Summary (and Continuation) Sheet (s)." The examiner will enter each operator's scores in the appropriate columns. Under the column titled " Simulator," the examiner will enter the results of the operator's individual follow-up evaluation. If the operator did not receive an individual follow-up evaluation, the examiner will enter a passing score. If an operator was a member of a crew that failed the dynamic simulator examination, but the operator passed or did not receive an individual follow-up evaluation, the examiner will enter a pass in the simulator column for that operator. Crew failures will be summarized in the overall results at the top of Form ES-601-3. The regional office will send a copy of the summary (and continuation) sheet (s) to the operator licensing assistant (0LA), OLB. OLB uses the results summary to verify the data in the operator licensing tracking system (0LTS) so that statistics can be maintained on operator performance. As it contains Privacy Act information, the regional office will not include the results summary in the examination report. If a small number of operators are given retake examinations, the regional office may issue an addendum to the original requalification evaluation report instead of issuing a new report. If the reexaminations are conducted concurrently with initial examinations or inspected during a requalification NUREG-1021 16 of 38 Interim Rev. 8, January 1997

, ES-601 program evaluation in accordance with IP 71001, the results may be reported as , part of the initial examination or inspection report.  : H. INDIVIDUAL REQUALIFICATION EXANINATION REPORT After the regional office completes the requalification evaluation, it will keep a copy of each operator's NRC-conducted written, walk-through, and simulator examination result: and return the original documents to the facility licensee. The facility licensee is required by 10 CFR 55.59 to maintain records of these examination results along with a copy of the written examination until the operator's license is renewed or for two years after the license expires. - The NRC chief examiner will ensure that Form ES-601-5, " Individual Requalification Examination Report," is completed for each operator who takes  ! an NRC-conducted requalification examination. The report will include the following information for each individual: written examination grade  : crew evaluation t.on the dynamic simulator examination, the individual follow-up results (P or F) from the dynamic simulator examination the number (and percentage) of JPNs performed correctly, if conducted l The regional office will send a copy of this report to the facility's training manager and to the operator with a letter notifying the operator of the examination results in accordance with ES-605. The regional office will also file a copy in the operator's docket file. I. OPERATOR LICENSE RENEWAL POLICY Operators are not required to take an NRC-conducted requalification examination in order to renew their licenses. However, if an operator takes but fails to pus an NRC-conducted examination, the NRC will not renew the license until the operator passes a retake examination conducted by the NRC, passes a retake examination administered by the facility licensee and inspected by the NRC in accordance with IP 71001, or passes an examination approved by the NRC. The regional office, in consultation with OLB, will determine the appropriate level of involvement on a case-by-case basis depending on the quality of the facility licensee's requalification program. ES-605 contains the specific procedures to follow for an operator who fails one or more NRC-administered requalification examinations, as well as the procedure for processing license renewal applications. NUREG-1021 17 of 38 Interim Rev. 8, January 1997

ES-601 J. RECORD RETENTION

1. Facility Reaualification Examination File The operator licensing staff in each regional office will maintain a facility requalification file for each facility in its region and will retain those files consistent with regional office guidelines for retention of facility initial examination files. The regional office will place the following items in the file for each requalification examination:

a. Examination Standard attachments and forms:

ES-201, Attachment 4, " Examination Assignment Sheet" Form ES-403-1, " Written Examination Grading Quality Assurance Checklist" ES-601, Attachment 2, " Corporate Notification Letter" Form ES-601-1, " Examination Security Agreement" Form ES-601-3, " Power Plant Requalificction Results Summary Sheet" and Form ES-601-4, " Power Plant Requalification Results Continuation Sheet" (if applicable) Form ES-604-2, " Simulator Crew Evaluation"

b. a master list of all JPMs administered and the operators to which they were administered

c. a master list of all scenarios conducted and operators to which they were administered (facility-generated forms or Form ES-D-1,

                       " Scenario Outline," may be used to meet this requirement)

d. a copy of the written examination and answer key

e. a copy of the requalification examination report The regional office may require that additional documents be retained in the facility requalification examination file.

NUREG-1021 18 of 38 Interim Rev. 8, January 1997

  --     .- .        . . - .        -.      . -   . - . - . _ . . . .     - .    . . - . __~ _. _ .        _ - -   -

i i ES-601 i

2. Operator. Docket Files The regional' office will retain the following records in each operator's docket file until the license is renewed, or for two years after the license expires or is terminated 1

a. Form ES-601-5, " Individual Requalification Examination Report" '

, b. ES-605, Attachment 1, 2, or 3, "Results Notification Letter" l c. a copy of all failed portions of the NRC-graded examination  ! i 3. Other Files l The regional office will retain reference materials used to develop each l examination until the NRC has resolved with the facility licensee all failures associated with the examination and has sent a notification letter to each operator. K. REQUALIFICATION STRESS FEEDBACK l The level of stress perceived by operators and facility personnel can affect t their overall performance on the requalification examination. Therefore, OLB is interested in monitoring the stress of operators and facility personnel participating in the requalification examination. Regional examiners and other personnel who are involved with an NRC requalification examination should assume the following responsibilities: Monitor the level of stress in operators and facility representatives, and be alert for examination techniques that may be causing examination stress. Recommend to OLB any changes to NUREG-1021 that would further alleviate l operator stress. Recommendations should be documented and forwarded to , l headquarters using " Report on Interaction" (ROI) forms. i L. ATTACHMENTS / FORMS Attachment 1, " Examination Timetable"  ! Attachment 2, " Sample Corporate Notification Letter" '

Attachment 3, " Examination Sample Plan" Form ES-601-1, " Examination Security Agreement" l Form ES-601-2, " Evaluation Checklist for Facility Reference Material" '

Form ES-601-3, " Power Plant Requalification Results Summary Sheet" , ! Form ES-601-4, " Power Plant Requalification Results Continuation Sheet" ' Form ES-601-5, " Individual Requalification Examination Report" ' l I I ! NUREG-1021 19 of 38 Interim Rev. 8, January 1997 r

l l ES-601 Examination Timetable Attachment 1 Ditg* Activity -120/90 The NRC notifies the facility licensee. -60 T*.' facility licensee sends the NRC the materials requested for developing the examination (including written examination ques-tions, simulator scenario banks, and JPMs). The facility licensee proposes composition of the crews to be evaluated and identifies facility examination team members. The facility licensee may request that the NRC chief examiner review the examination process with operators and managers. -45 The facility licensee submits its proposed requalification written examination and operating test. -30 The NRC concurs on the operating crews to be evaluated. -14 The NRC examiners visit the facility to review the JPMs to be administered, observe the static and dynamic simulator I examinations, and validate the test items as needed. The chief examiner and the regional branch chief determine the length of time on-site and the number of examiners required, on the basis of the examiners' experience and the quality of the facility licensee's testing materials. The facility licensee designates a simulator operator. If requested, the chief examiner briefs the operators and managers about the requalification examination process. -7 The facility examination team members finalize the examinatiora based on preparation week activities. Evaluators review reference material to prepare for the JPMs and simulator scenarios. O The NRC administers the examinations to selected crews and operators. Facility licensee notifies the NRC of its final results for crews and individuals at the end of each examination week. +7 The NRC finalizes the examination results. +14 The facility licensee transmits the written examinatio grades and a final summary to the NRC. +30# The NRC issues operator results and the final requalification examination report.

• Number of days before or after the examination, except as noted

1. Number of days after receipt of facility results NUREG-1021 20 of 38 Interim Rev. 8, January 1997

l l l l ES-601 Sample Corporate Notification Letter Attachment 2 i NRC Letterhead (Date) (Name. "itlet l (Name of fac' lity) (Street address (C- tv. State. Z< o code)

SUBJECT:

REQUALIFICATION PROGRAM EVALUATION l

Dear (Name):

l In a telephone conversation on Jdate), (W. title) and (Name. title) arranged to evaluate the requaluf cation program and licensed personnel at the (fac itv n==) . The evaluation is scheduled for the week of (date. NRC l exam'nors and evaluators from your facility will conduct requalification l l examinations, and the NRC will evaluate your requalification program in accordance with Sections ES-601 through ES-604 of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," Revision 8. You are encouraged to ensure that your training staff and proposed examinees are familiar with these standards. l For the NRC to adequately prepare for this evaluation, the facility licensee l ' will need to furnish the NRC with the approved items listed in Enclosure 1, I " Reference Material Guidelines." You are also requested to submit, at your l option, a proposed examination for use during the examination week. However, if you do submit a proposed examination, the personnel participating in its development will become subject to the security restrictions described in this l letter. Please review the guidance promulgated in Revision 8 of NUREG-1021 concerning the content and scope of simulator examination scenarios. The scenario examination bank should cover the entire spectrum of emergency o)erating procedures (EOPs), including alternative decision paths within tie E0Ps and it should incorporate a range of failures with various degrees of severn.y for the same type of event. Each scenario should contain simultaneous events that require the senior reactor operators (SR0s) to prioritize their actions and to assign particular tasks to other crew members. Each scenario should also require the SR0s to decide when to make the transition between E0Ps and which ) actions to take within E0Ps. You are requested to designate at least one employee to be a member of a joint NRC-facility examination team. That employee is expected to be an active SR0 (as definedYou department. by 10 are CFR 55.53(e) encouraged or (f))ignate a second employee from thefrom the (fa to des training staff to be a member of the examination team. This employee should also be a licensed SRO, but may be a certified instructor. If desired and agreed to by the chief examiner,-you may designate one additional employee i from the training staff with appropriate qualifications to be a member of the examination team. In addition to these individuals, you'will need to designate a simulator operator for scenario preview and validation during the l NUREG-1021 21 of 38 Interim Rev. 8, January 1997 l i l

i i _ES-601 2 Attachment 2 on-site examination preparation week. In some cases, you may need to i designate a simulator o)erator during the test item review period. All of these individuals will >e subject to the examination security agreement. The NRC restricts any facility licensee representatives under the security l agreement from knowingly communicating by any means the content or scope of the examination to unauthorized persons and from participating in any facility licensee programs such as instruction, examination, or tutoring in which an identified requalification examinee will be present. These restrictions apply from the day that the facility licensee representative signs the examination security agreement indicating that the representative understands that he or  ! she has specialized knowledge of the examination. The chief examiner will l determine when a facility licensee representative has received specialized knowledge concerning the examination and will execute an examination security agreement. In most cases, the examination team members will not be required to enter into an examination security agreement more than 60 days before the examination week. The simulator operator will normally become subject to the security restrictions during the examination preparation and validation week; however, this n..y occur as much as 45 days before the examination week. Sixty days before the 9xamination administration date, please provide the NRC regional office with a prcacsed list of operators, including crew composition, I for the examination. TheIistshouldincludeatleast12 operators, comprising three or more crews, and the current mailing address for each proposed operator, if different from that listed on the most recent Form 398 , submitted to the NRC. Your trainin l directly to the NRC chief examiner,gensuring staff should send operator's that each this information address is l sent in a manner to ensure privacy.  ; The facility licensee may request that the NRC chief examiner or another NRC representative meet with the operators to be examined and the licensee managers during the examination preparation week, normally two weeks before the examination. However, if the schedule does not allow them to meet during the preparation week, they may meet at any mutually agreeable time. The NRC examiner will explain the examination and grading processes and will respond to any questions that the operators may have about the NRC's examination procedures. If such a meeting is desired, your training staff should schedule it with the NRC chief examiner. The facility licensee staff is responsible for providing adequate space and accommodations to properly develop and conduct the examinations. Enclosure 2,

 " Administration of Requalification Examinations," describes our requirements for developing and conducting the examinations. Also, a facility operations management representative above a shift supervisor level should observe the simulator examination process at the site.

This letter contains information collections that are subject to the Paperwork Reductfon Act of 1995 (44 U.S.C. 3501 et seg ). These information collections were ap) roved by the Office of Management and Budget, approval number 3150-0101, wiich expires on (date). NUREG-1021 22 of 38 Interim Rev. 8, January 1997

ES-601 3 Attachment 2 l The public reporting burden for this collection of information is estimated to ' average (nu=her) hours per response, including the time for reviewing instructions, gathering and maintaining the data needed, and completing and reviewing the collection of information Send comments on any aspect of this collection of information, including suggestions for reducing the burden, to the Information and Records Management Branch (T-6 F33), U.S. Nuclear Regulatory Commission, Washington, D.C. 2055520001, or by Internet electronic mail at BJS19NRC. GOV; and to the Desk Officer, Office of Information and Regulatory Affairs, NE08-10202, (3150-0101), Office of Management and Budget, Washington, D.C. 20503. , The NRC may not conduct or sponsor and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Thank you for your cooperation in this matter. (Name) has been advised of i.: HRC guidelines and policies addressed in this letter. If you have any questions on the evaluation process, please contact (Name. reaional section chief) at (teleohone number). Sincerely, O (Appropriate Regional Title) Q Docket No.: 50-(Number)

Enclosures:

1. Reference Material Guidelines

2. Administration of Requalification Examinations DISTRIBUTION:

Public NRC Document Control System Regional Office Distribution l NUREG-1021 23 of 38 Interim Rev. 8, January 1997

ES-601 4 Attachment 2 Enclosure 1 Reference Material Guidelines

1. Sixty days before the examination date, the facility licensee should provide test items to support all aspects of the requalification I examination to the NRC.

2. The facility licensee is expected to submit the following reference materials for all NRC-conducted requalification examinations:

1 an examination sample plan that meets the requirements of ES-601, i Attachment 3 l l the facility's examination banks (written, simulator, and JPM) and ' associated reference materials (including, at a minimum, technical specifications, abnormal and emergency operating procedures, and emergency plan procedures utilized in requalification training) additional reference materials requested by the NRC chief examiner

3. The facility licensee's examination banks are expected to contain the j following information: I a minimum of 700 test items equally divided for use in the two sections of the written examination and covering all safety-related elements of the facility job-task analysis (JTA). The -

facility licensee is expected to maintain a dynamic bank by l reviewing, revising, or generating at least 150 questions a year. New questions should cover equipment and system modifications, as well as recent industry and licensee events and procedural changes. JPMs that reaet the criteria in ES-603 for evaluating each reactor operator (RO) and senior reactor operator (SRO) safety-related task identified in the facility JTA. The JPM bank should expand at a rate of at least ten JPMs per year until this goal is reached. It is estimated that 125 to 150 JPMs will be the final result. a bank of at least 30 simulator scenarios reflecting all abnormal and emergency situations to which an operator is expected to respond or control. At least five scenarios per year should be generated until all aspects of the emergency operating procedures are covered with sufficient variation in the type and scope of initiating events and level of degradation. Emphasis should be placed on scenarios that include applicable industry events. ' NUREG-1021 24 of 38 Interim Rev. 8, January 1997

l l ES-601 5 Attachment 2 Enclosure 2 j Administration of Requalification Examinations

1. The NRC must evaluate at least 12 operators to perform a program evaluation. The guidelines on crew composition in the simulator are described in ES-601, Section D.2, and ES-604.

1

2. The simulator and simulator operators need to be available for  ;

examination development. The chief examiner and the facility ' representatives will agree on the dates and amount of time needed to develop the examinations. ' l 3. The chief examiner will review the reference materials used in the simulator. The NRC will not authorize for use during the simulator test any reference material that is not normally used for plant operation in the control room. 1

4. The facility licensee will provide a single room for completing Section B of the written examination. The examination room and the supporting restroom facilities will be located to prevent the examinees from having contact with any other facility or contractor personnel

, during the examination.

5. The chief examiner will inspect the examination room to see that it meets the minimum standard that will ensure examination integrity. The minimum spacing standard consists of one examinee per table and a three-foot space between tables. No wall charts, models, or other training materials are allowed in the examination room.

I 6. The facility licensee is expected to provide a copy of each reference l document for each examinee for Section B of the written examination. l The material should include documents that are normally available to the operators in the centrol room (such as the technical specifications,

operating and abnormal procedures, administrative procedures, and i

emergency plans). The chief examiner will review the reference j materials before the examination begins.

7. The NRC requalification examination will attempt to distinguish between R0 and SR0 knowledge and abilities to the extent that the facility l

training materials allow the developers to make these distinctions.

8. Prudent scheduling of examination week activities is important to help alleviate undue stress on the operators. The facility training staff and the NRC chief examiner should attempt to formulate a schedule that 1

4 O] t NUREG-1021 25 of 38 Interim Rev. 8, January 1997 1 l

i 1 l ES-601 6 Attachment 2 2 Enclosure 2 will minimize delays while conducting the examination. The following l suggestions will help to structure the examination activities to achieve  ! this objective:- ' Consider allowing operators to stay at home until their scheduled examination times. Segregate the group of operators completing their examination, instead of the group of operators that are scheduled to start their examination. Following simulator scenarios, the facility evaluators and NRC examiners should quickly determine whether follow-up questioning is required so that the crew members may be released to talk among themselves about the scenario. Ensure that time validation of JPMs, particularly those performed in the simulator, is accurate. Establish a reasonable schedule to i prevent operators from waiting for simulator availability to l complete their JPMs.

9. The NRC does not require the facility licensee to videotape dynamic I simulator examinations. If the facility licensee requests to videotape the examination, any use of the tape must be completed before the NRC leaves the site at the end of the examination. If a disagreement over the grading of an operator still exists at the end of the examination week, the facility licensee may retain the tape for the purpose of submitting it to support a request for regrade by the NRC. During the regrade, the NRC will review only the portion of the videotape under contention. After all requalification examination grades are finalized, including the review of any regrade requests, the facility licensee is expected to erase all video tapes made during the examination.

NUREG-1021 26 of 38 Interim Rev. 8, January 1997

    - . .   . - .          .    --. .         .. . .- - -                             _ - - -                     _ _   -=     _

l l ES-601 Examination Sample Plan Attachment 3 l' A. INTRODUCTION An examination sample plan provides a systematic approach to selecting and developing test items to determine if a student has mastered the knowledge, skills, and abilities (KSAs) to be covered in a particular training program. l The sample plan should provide an explicit, documented link between the  : l 1 earning objectives associated with the training program and the test items ' used to perform the evaluation and to verify the relevance to the job task i l analysis (JTA) associated with the operator's position. -

ES-401 gives explicit' guidance for developing a sample plan for initial examinations using NUREG-1122 or -1123, the " Knowledge and Abilities Catalog l for Nuclear Power Plant Operators

Pressurized [or Boiling] Water Reactors."

A similar methodology may be applied to any training program. With respect to . a requalification program, the scope of topics is necessarily limited because the amount of material that is covered during a requalification program is less than that covered in an initial licensing training program. However, the l NRC permits and encourages reserving 10 to 20 percent of test items for topics l that have high importance ratings and contain K/As that operators should i l retain because of their safety significance, but were not necessarily covered , l during the requalification cycle. B. REQUALIFICATION TEST OUTLINE l The facility licensee is expected to develop a test outline for all NRC-administered requalification examinations. At least 80 percent of the test outline must reflect the training curriculum of the most recent requalification cycle in a manner consistent with the distribution of emphasis l in the curriculum. l The curriculum of the requalification training cycle for which the examination is being developed should identify the following information:

1. requalification lecture / classroom topics indicating the percent of the cycle devoted to each

2. concentration of training exercises using the simulation facility, including the type of scenarios trained (e.g., accident, abnormal, normal) and the number of times each scenario was run

3. special focus of the training such as plant modifications, licensee event reports (LERs), and major changes to operating practices or policy

4. practical trainN such as operation of individual systems or components

, for requalification training purposes, using either the simulation facility, mockups, or actual systems and components 4 i NUREG-1021 27 of 38 Interim Rev. 8, January 1997

          -        ,         .,                 . . , . . _ . .       _-  , - . _ _ _                      , . - - -                 r

ES-601 2 Attachment 3 The format of the sample plan is a matter of training department preference as long as the plan results in a thorough and accurate assessment of the facility's training program and its intended objectives. The sample plan is expected to contain the following information for use in developing or selecting the test items to be used in the requalification examination:

1. identification of the subjects to be evaluated (system, component, procedure, or other training subject) 2, the preferred testing media for evaluating each subject (written, simulator, or walk-through examination); more than one testing method may be used to evaluate a subject j

3. the learning objectives intended to be evaluated  !

4. a list of references used to develop the test items

5. the specific K/A topic or facility JTA KSAs that are closely linked to 1 the learning objectives for each subject and the importance factors for 1 each (the facility licensee may use a site-specific K/A if it exists) l 4 . 6. all test items used in the examination should have a K/A value of 3 or greater; the facility licensee may propose the use of test items with NRC K/A values less than 3 with appropriate justification

7. the percentage or number of points of the examination that should be .

j devoted to the topic area (e.g., 3 points for technical specification l 2 interpretation, or five percent on reactor coolant pumps)

8. whether the subject is identified as safety-related in the facility JTA 4

9. whether the subject was covered in the cycle for which the examination is being developed

10. the identification code or number for previously developed test items that evaluate the subject

11. recent safety-related issues and events (e.g., relevant LERs)

NUREG-1021 28 of 38 Interim Rev. 8, January 1997

   . . - - - . - . - . - .                  ~ . - -     .   - . - - . - - .            . _ - . . - -            .  . . ..

l ES-601 Examination Security Agreement Form ES-601-1 3

1. Pre-Examination i

I acknowledge that I have acquired specialized knowledge about the requalification examination scheduled for the week (s) indicated in this

                               . agreement as of the date of my signature. I agree that I will not

knowingly divulge any information about this examination to any unauthorized persons. An unauthorized person is any individual who has not been approved by the NRC chief examiner to receive specialized knowledge of the examination. I understand that I am not to participate e in any instruction, tutoring, or examination involving those operators scheduled to be administered this requalification examination from this i date until completion of examination administration. I further understand that violation of the conditions of this agreement may result

, in cancellation of the examination and/or an enforcement action against j me or the facility licensee by which I am employed or which I represent.

2. Post-Examination f To the best of my knowledge, I did not divulge to any unauthorized persons any information concerning the. examination (s) administered during the indicated week (s). From the date that I entered into this

!                               security agreement until the completion of examination administration, I 1                                did not participate in instructing those operatcrs who were administered I                               this requalification examination (s).

l . l Examination Period to 1 ) Printed Name Pre-Examination Post-Examination Certification (1) Certification (2) I Date ~ Date Date Date. i Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date Date , Date Date Date Date i Date Date  ! O NUREG-1021 29 of 38 Interim Rev. 8, January 1997 j

ES-601 Evaluation Checklist for Facility Form ES-601-2 Reference Material The following checklist represents the minimum content of facility-generated reference material. Items marked " optional" should be checked if requested from the facility licensee by the chief examiner. The chief examiner or designee may use this checklist to make a quick, general evaluation of the completeness and adequacy of the facility licensee's references. The chief examiner may resolve any specific questions about the references with the facility staff as necessary. I. DDntity Actual Reference Material Beauired Minimum Submitted A. Open-reference written 350 per section; bank examination items is to be dynamic with atleast150 revised, reviewed, or newly generated questions per year B. Simulator scenarios 25; + 5 per year following the initial regualification exam until at least 30 scenarios covering all aspects of the E0Ps are developed C. Job performance 95; + 10 per year following measures the initial requalification exam until the JTA is fully covered D. Technical specifications I copy E. Applicable plant I set (optional) procedures F. Emergency plan 1 copy G. Applicable administrative I copy -ctional) procedures H. Sample plan I copy I. Requalification cycle 1 set (optional) training reference material (lesson plans, handouts,etc.) J. Appropriate sections I set (optional) of JTA or facility-specific K/A Catalog ' i Reviewed by: Date: NUREG-1021 30 of 38 Interim Rev. 8, January 1997

l l ES-601 2 Form ES-601-2 l II. Usability l Circle one  ! l A. The reference material is legible yes no l i B. The reference material is properly arranged and labeled j for its function yes no 1 1 C. The reference material indicates a SAT program yes no i I D. Reference material is available to verify that test l items are appropriate, job relevant, and technically  ! accurate yes no 1 E. Reference material is available to adequately I support the examination topics yes no 1 Comments O Reviewed by: Date: i ( NUREG-1021 31 of 38 Interim Rev. 8, January 1997 l

l ES-601 3 Form ES-601-2 1 III. Quality Exam Section Reauired Standards Comments A. Sample Plan Subjects covered in requalification cycle are identified The test outline incorporates the following: l time spent on topic relative importance frequency of performance job level (R0 or SRO) K/As (or facility equivalent) of sufficient importance are identified in the test outline Plant-specific priorities are identified (LERs, procedure changes, system modifications, risk-dominant accidentscenarios, risk-impprtant systems and operator actions identified , in the facility licensee's PRA/IPE, etc.) Appropriate testing methods are indicated for each K/A (i.e., JPM, written exam, and/or simulator) Applicable learning objectives are associated with K/As l Methodology exists to tie test items to a learning objective and a K/A Sample plan includes important topics not covered in the . requalification cycle Test areas appropriate to R0s and SR0s only are identified i Chapter 13 of NUREG-1560, " Individual Plant Examination Program: Perspectives on Reactor Safety and Plant Performance," identifies a number of important human actions that may be appropriate for evaluation. Reviewed by: Date: NUREG-1021 32 of 38 Interim Rev. 8, January 1997

  .    -    .   -       . .     ---          . _ -           . .- =_. - -       -         --

I ES-601 4 Form ES-601-2 III. Quality (cont.) Exam Section Reauired Standards Comments B. Written At least 10 percent of all test items shall be reviewed

using Form ES-602-1

Test items are important j to safety i

Test items are clearly written i Test items are appropriate to license level l The criteria for open d i reference examinations are met n Test items are associated ( with K/As of 3 or greater and . are adequate discriminators A learning objective and i applicable reference are i identified for each test item

The facility has identified SRO-level questions for both

sections of the exam l

. 1 If the above criteria are not adequately met, the NRC will conduct further j review of the examination bank utilizing ES-602, Attachment 1, " Guidelines for the Development and Review of Open-Reference Examinations," and Form ES-602-1, "NRC Checklist for Open-Reference Test Items." i 9 4 4 Reviewed by: Date: O NUREG-1021 33 of 38 Interim Rev. 8, January 1997

ES-601 5 Form ES-601-2 III. Quality (cont.) Exam Section Reauired Standards Comments C. Walk-Through At least 10 percent of the JPM bank reviewed, using Form ES-C-2 Applicable plant systems identified by test outline: systems covered in requalification cycle new or recently modified systems systems in recent facility LERs or vendor notices PRA-identified risk dominant systems systems in NRC generic communications Tasks / abilities for identified systems are applicable to the facility are at the A0/R0/SR0 level have a K/A value of 3 or greater include JPMs pertinent only to SR0s Some JPMs are performed under low-power or shutdown operating conditions Some JPMs require the operator to implement alternative paths within the facility licensee's procedures Facility JPMs contain the information found on Form ES-C-1 Reviewed by: Date: NUREG-1021 34 of 38 Interim Rev. 8, January 1997

ES-601 6 Form ES-601-2 III. Quality (cont.) Exas Section Reauired Standards Comments D. Simulator At least 10 percent of the scenarios reviewed, using Form ES-604-1 l Scenarios are an appropriate , measure of the material covered in the sample plan Scenarios are based on lessons covered in requalification cycle recent industry events LERs emergency and abnormal procedures design and procedural changes Scenarios exercise crew's ability to use facility procedures in g accident prevention and mitigation h Scenario events have a K/A of 3 or greater Somepcenariosarebasedonlow power operations Some scenarios are based on the dominant accident sequences (DAS) for the facility as determined by PRA/IPE Scenario identifies critical tasks that meet the criteria of Appendix D Proposed examination scenarios that were used for training during the most recent training cycle have been reviewed by the NRC and replaced or modified, if ap)ropriate, to ensure the validity of t1e examination and to minimize the potential for examination compromise i NUREG-1449, "NRC Staff Evaluation of Shutdown and Low-Power Operation," l defines low power to include the range from criticality to 5 percent power. l l Reviewed by: Date: ! NUREG-1021 35 of 38 Interim Rev. 8, January 1997

ES-601 Power Plant Requalification Results Summary Sheet Form ES-601-3 Privacy Information - For Official Use only Facility: Overall Results Total Passed Failed ' Exam Date: # #/% #/% NRC Examine- Reactor Operator: l Senior Operator: l l Total: l l Operating Crews: l t Results (P or F) Docket Grader JPMs Written Operator Simulator ' 55-( )  % (A & B) Written Crew Indiv NRC  % FAC  % , NRC  % FAC  % NRC  % FAC  % NRC  % FAC  % NRC  % FAC  % NRC  % FAC  % NRC  % FAC  % 9 g

91 4 7 _

      -                                                                  9      _

0 9 _ 6 T

                         /

1 y S E W r a u m n _ r J a _ o ) F F  ; 8 r r o t o a g v P l e ( u R s m l t i S m u s i r _ e t e _ R I n . n _ t e _ t . i t r _ e W e y h S l y n _ O )

     -            n         B     % % % % % % % % % % %  % % % % % % % %        _

e ta e n s t t m U i r u A S l a W ( s i t i c s l _ u f M _ s f P  % e O _ R J _ r 8 _ n o 3 . o F r f i t - d e C C C C C C C C C C C C C C C C C C C C o _ a c n a r R A R A R A R A R A R A R A R A R A R A 7 i o G N F N F N F N F N F N F N F N F N F N F 3 f i i t l a a m u r ) q o t e e f k R n c t I o ( n y D 5 a c 5 l a P v i r r e P w o P r o t a r ep O 1 2 0 1 1 G 0 E R S U E N

ES-601 Individual Requalification Examination Report Form ES-601-5 U.S. Nuclear Regulatory Cammiaion - Individual Requalification .Ernmination Report (Privacy Information - For Official Use Only) Facility: Operator's Name: Docket No: 55- IJcense No: Expiration Date: l Exam Type: RO / SRO Rede: 1st / 2nd / No Date of Last Exam: l

                               .            swista, vnnminniuResuits Date(s) of Exam:             NRC Fr==iaar (Print):                       Facility Evaluator (Print):

NRC Grading Facility Grading Section A (Points) of of Section B (Points) of of Overall Score (%)  %  % l Simulator Examination Results Date(s) of Fram- NRC Exammer(s) (Print): Facility Evaluator (s) (Print): j } Crew Evaluation Pass / Fail Pass / Fail Individual Follow-up Pass / Fail / NA Pass / Fail / NA LWalk-Drough Framination Results -  ! Date(s) of Exam: NRC Framiner(s) (Print): Facility Evaluator (s) (Print): No. of Successful JPMs of 5 of 5 Exam Results (%)  %  %

                                         ;NRC Examiner Recommendations -

Category Results Signature /Date W itten Pass / Fail Simulator Pass / Fail Walk 'Ihrough Pass / Fail

                                                ' NRC Supervisor Review -

l l I Date: Pass / Fail Signature: l l NUREG-1021 38 of 38 Interim Rev. 8, January 1997

ES-602 REQUALIFICATION WRITTEN EXAMINATIONS l A. PURPOSE l The NRC staff conducts written requalification examinations, using this

standard in accordance with Title 10, Section 55.59(a)(2)(iii), of the

, Code of Federal Regulations (10 CFR 55.59(a)(2)(iii)). NRC examiners are to l follow this standard to prepare and administer all NRC-conducted written requalification examinations. Moreover, NRC examiners are to use this f standard in conjunction with ES-601, which explains how to conduct requalification program evaluations. B. SCOPE The written examination is useful for evaluating those knowledge, skills, and abilities (KSAs) of licensed operators that are difficult to infer from behavior alone but that can be readily tested through written responses to questions that value interpretation and allow the examinee to use references. Additionally, determining an individual's knowledge of factual information and ability to perform " paper-and-pencil" tasks are best evaluated by a written examination. The written examination consists of two sections for which the examinee may refer to references (i.e., "open-reference examinations"). Section A, " Plant and Control Systems," is administered using a static simulator. Section B, O c"lassroom. Administrative EachControls section /should Procedural Limits," to be designed may lastbe administered a minimum in ahour, of one including time for the operator to review his or her work. Combined, the two sections of the written examination will be designed to last three hours. The exact number of questions and time allowed to complete each section will be detemined by the facility licensee on the basis of the requalification sample plan and the license level of the operators taking the examination (reactor . operator (RO) or senior reactor operator (SRO)). ' Even though the examination is designed so that examinees may use references,  : an examinee should not expect to have time to complete the examination by consulting references to determine each answer. A good mix of test items will contain some questions that evaluate the operators' ability to determine a correct response without delving into reference material and others that , require the use of reference material to select the correct response. By combining test items that require using references with those that do not, the written examination can test a broader sample of operator knowledge within a given period. On both sections of the written examination, certain questions will test the knowledge and abilities (K/As) of an R0 while others will test those of an SRO. During the development of the examination, the examiner should consult the facility job task analysis (JTA) and NUREG-1122 or -1123 to help identify the most suitable topics for an R0 and an SRO, respectively. Additionally, NUREG-1021 1 of 29 Interim Rev. 8, January 1997

ES-602 10 CFR 55.41 and 55.43 give further guidance on item selection for R0 and SR0 written examinations, respectively.

1. Section A. " Plant and Control Systems" (Static Simulator)

This section of the written examination is designed for using the simulator as a reference tool to provide realistic information visually i and to give the operators an environment as close as possible to their i normal control room. While administering this section, the simulator will be frozen in the middle of an evolution, transient, or accident. In developing the test items for this section, allow the use of references and relate them to plant systems and components, control roo:n indications, instrumentation and controls, and technical specification 1 (TS) limiting conditions for operation (LCOs).  ! Section A is designed to evaluate the operators' knowledge of plant systems, integrated plant oper?tions, and instruments and controls. In i addition, it evaluates the operators' ability to recognize TS LCOs and l to determine the effects of postulated events. The NRC encourages facility licensees to include questions that test the ability of the operators to use their facility curves and charts. While administering Section A, the examination team will use one

      " frozen" simulator condition or set-up. The condition places the simulator in a " snapshot" of the plant following a major transient that resulted in an engineered safeguard feature (ESF) initiation or in a steady-state situation at power. Sonie equipment should be frozen in an abnormal or failed condition to provide adequate material for test items.

2. Section B. " Administrative Controls / Procedural Limits" Section B of the written examination is designed to evaluate the ability l of the operators to analyze a given set of conditions and determine the proper procedural and administrative guidance to use. This section may include theory-related questions appropriate to sample the topics listed in 10 CFR 55.41 and 10 CFR 55.43, as long as they are operational in nature or test unique facility characteristics.

Section B is designed to test the operators' knowledge and use of plant procedures and administrative controls, while allowing the use of references. The NRC uses administrative, operating, normal, abnormal, and emergency procedures, TS, and the emergency plan as sources of test items for this section of the examination. The test items focus on how direction, guidance, and information found in these procedures are used or interpreted, rather than focusing on finding the procedure in which the necessary information is located. Additionally, the test items for Section B of the SR0 examination examine the operators' understanding of the reasons and bases for procedural requirements. The use of graphs, NUREG-1021 2 of 29 Interim Rev. 8, January 1997

l ES-602 charts, tables, and drawings is appropriate. The simulator may be made available to the examinees to make the examination more operationally oriented. C. EXAMINATION DEVELOPMENT

1. Facility Examination Team Members' Pesponsibilities

a. The facility is expected to prot'ide a bank of test items developed' by using the guidance in Attachse.it I and Appendix B. The number of test items should meet the submittal guidelines of ES-601, Attachment 2, Enclosure 1, "Requalification Examination Reference Material Requirements." Form ES-601-2, " Evaluation Checklist for Facility Reference Material," provides information that facility personnel may use to evaluate reference material sets before submitting them to the NRC.

The facility licensee shall maintain its examination question bank up-to-date by reviewing, modifying, or creating at least 150 questions each year to expand the bank, reflect procedure or system changes, new lesson plans, and recent licensee and industry events. If the facility question bank contains at least 700 items that O meet the format guidance of Attachment 1, the facility may release the bank to its operators for review.

b. The following items should be provided for each test question:

applicable K/A reference and values (R0/SRO) reference JTA (if applicable) estimated time to answer appropriate learning objectives applicable reference (e.g., lesson plah, E0P)

c. The facility is expected to provide a sample plan that meets the guidelines of ES-601, Attachment 3, " Examination Sample Plan," and may submit a proposed examination that conforms to the facility sample plan. The proposed examination should contain a total of 30 to 40 test items, depending on the time validation (maximum-time of three hours) of the individual questions selected.

Sections A and B should each contain 15 to 20 questions. Each section must be designed to last a minimum of one hour, with the total examination designed to last three hours. The facility licensee will determine the number of questions in each section, basing the decision on the requalification sample plan and the license level of the operators taking the examination (R0 or SRO), and subject to the quantitative constraints of the O NUREG-1021 3 of 29 Interim Rev. 8, January 1997

                                                                                 \

ES-602 previous paragraph. Plant systems questions not directly related to the static scenario can be included in Section A to meet the facility's sample plan and 10 CFR 55 requirements. In addition, up to 20 percent of the test items may be from topics outside the sample plan, as long as the information stated in Section C.I.b. of this standard is provided. If the facility licensee submits a proposed examination, those individuals involved in its development become subject to the security restrictions of ES-601 once examination development commences. These restrictions remain in effect until the NRC I examination is given. If, after developing a proposed examination, the facility decides not to submit it for use in the NRC-conducted examination, the developers are released from the l security restrictions of ES-601.

d. After the NRC has reviewed the facility's examination bank and commented on the test items selected for the examination, the facility team members are expected to prepare the examination for final NRC review and approval. The examination may be finalized before or during the preparation week.

e. The facility team representative will evaluate each test item that the NRC revised, in order to assess the following criteria:

appropriateness time required to answer, given the operational context technical accuracy clarity K/A and objective references Following this evaluation, the facility examination team representatives and the chief examiner need to agree on the final form of the examination. They also need to complete a time validation of the proposed examination. A variety of methodologies have proven effective in accomplishing this task; Attachment 1 provides further information in this area. Any individual involved in time validating the examination is required to sign the security agreement, Form ES-601-1. The examination team may add or delete items from the examination based on the results of this time validation, and their experiences. If any test items are added, the entire examination need not be time validated again as long as a subject matter expert (SME) has reviewed the added questions, indicating the approximate time an operator should take to answer each question.

f. The facility licensee is expected to provide enough copies of each reference so that each examinee can use the references during the examination and, immediately upon completion of the examination, NUREG-1021 4 of 29 Interim Rev. 8, January 1997

f - i ES-602 i compile the examinations and reproduce sufficient copies for their i own use and that of the NRC. I g. To help relieve the burden of providing a complete set of l references to each operator, the examination may be assembled so ! that a different sequence of questions appears on each operator's i examination. Alternatively, handouts of relevant information (e.g., plant curves, blank forms, etc.) may be provided with the j test. i l 2. NRC Examination Team Nembers' Resnonsibilities i a. The NRC will begin its evaluation of the sample plan, the bank of

test items, and the proposed examination as soon as possible after i receipt of the facility's materials. The NRC will promptly i evaluate the materials to allow sufficient time for the NRC or the i facility to develop the test items and for the facility to revise i them to meet NRC standards, if required. The NRC examiners should

review the proposed test items using Form ES-602-1, "NRC Checklist i for Open-Reference Test Items," to ensure appropriateness, i clarity, and importance to safety, as described in Attachment 1.

If the facility licensee intends to administer both sections of ] the examination during a single three-hour period as noted in Section D.I.c, the examination team members must review the i examination as a whole to ensure that the items in either section j do not compromise those in the other. l b. A minimum of 80 percent of the test items will be chosen in

accordance with the sample plan. The remaining 20 percent may be

! substituted by the examination team, using facility examination j bank questions or new questions the exam team develops. Should it

be necessary to develop additional items to satisfy the sample

plan, the NRC will request that the facility do so.

j c. If, after reviewing at least 75 percent of the bank, insufficient

test items exist to develop an NRC examination that meets the

! sample plan, the NRC staff will declare the bank of test items inadequate. In that event, the regional managers may either i cancel the scheduled examination or administer an examination i using NRC-developed test items without consideration for the 20 j percent substitution constraints. I d. If the sample plan does not include testing of topics from outside

of the requalification cycle, then the examination team should i consider incorporating 10 to 20 percent non-requalification cycle specific test items.

i f NUREG-1021 5 of 29 Interim Rev. 8, January 1997 2 1

ES-602

e. If a test item does not have a clear tie to the JTA, the examiner will discuss the applicability of the test item with the facility representatives.

D. EXAMINATION ADMINISTRATION AND EVALUATIONS

1. Written Examination Conduct

a. An NRC examiner or a knowledgeable facility representative who has signed the security agreement will proctor each section of the examination. As a minimum, an NRC examiner will observe the examination briefing as the operators begin the examination to ensure that all adrainistrative aspects of the examination are adhered to. If an NRC examiner does not continuously proctor the examination, an examiner will periodically visit the examination room to ensure that the proctor appro)riately addresses questions on the content or administration of tie examination that may have arisen.

b. Section A is administered on the facility's simulator or an approved simulation facility.

c. Section B may be administered in the simulator or in a classroom setting as the facility staff and the chief examiner deem appropriate.

If both sections of the examination are administered in the simulator during a single three-hour period, operators may return to a section of the examination that they already completed or retain both sections of the examination until the time has expired.

d. For Section A of the examination, the facility licensee is responsible for providing the group of examinees at least one copy of all controlled reference material available in the control room. Examination reference material will not include material that is intended for training use only. The licensee controls all reference material in accordance with the licensee's 10 CFR 50, A)pendix B, procedure revision control program. This material s1ould be authorized f.or use in operating the power plant, agreed upon by the facility and the chief examiner, and in effect at the time of the examination validation (i.e., the preparation week).

e. During the administration of Section B, each examinee will have available for use the following material (complete, current issue):

technical specifications plant procedures (E0P/A0P/0P, etc.) emergency plan (as available in the control room) NUREG-1021 6 of 99 Interim Rev. 8, January 1997

Y 4 !O ES-602 $U i - administrative procedures applicable to operations other controlled plant reference material normally available t in the control room plant drawings, flow (e.g., curves and data book, forms, charts, etc.) an j operating the plant Note: "Non-controlled" reference material, such as the Emergency l Operating Procedure Owner's Group Basis Documents will not be l provided unless these documents are authorized to be used by the control room operators during plant operations.  ;

2. Examination Administration Procedures l

l The written examinations will begin only after the chief examiner has verified the adequacy of the examination facilities and made , arrangements for continuous proctoring of the examination as discussed l in Section D.I.a of this standard. An NRC examiner may act as proctor ' during this examination. ' However, the chief examiner is responsible for ensuring the actions of D.2.b through D.2.1 below are complete. l Each section of the written examination will be administered as follows:

a. An NRC examiner will verify each examinee's identity and l

  #%                 examination level against the examination assignment sheet                                    l (ES-201, Attachment 4). If possible, the R0s and SR0s should be seated at alternate tables. Any errors or no shows will be resolved with the facility staff and the assignment sheet will be updated as required.

b. The proctor will remind the operators that they may use calculators to complete the examination and that no other reference material other than that provided is allowed in the examination area. The proctor will define the examination area for the examinees.

c. The proctor will pass out the examinations and answer sheets and instruct the examinees not to turn over the examination until told to do so. The examinees will be informed that pads of scrap paper are available upon request from the proctor.

d. The proctor will brief the examinees regarding the rules and guidelines in effect during the written examination using Parts A and B of Appendix E, " Policies and Guidelines for Taking NRC Examinations." The proctor should inform the examinees that they may refer to the instructions directly beneath their examination cover sheet. The proctor will read the indicated policies verbatin. '

NUREG-1021 7 of 29 Interim Rev. 8, January 1997

l ES-602 l

e. The proctor will ask the examinees to verify the completeness of their copies by checking each page of the examination. The proctor should also have the examinees check to ensure that an

equation sheet has been included in their examination, if required.

f. After answering any quest?ons that the examinees may have about the examination policies, Re proctor will start the examination and record the time.

g. The proctor will periodically advise the examinees of the time that remains to complete the examination. Normally, a chalk board

or white board is available for this purpose.

h. As the examinees complete the examination, the proctor will ensure that they sign the examination cover sheet and staple it on top of their answer sheets. The proctor will collect the examination packages, including the questions and answer sheets, any references used with the examination, and all scrap and unused paper. The NRC examiner will keep the cover and answer sheets, dispose of the scrap paper, and give the packages of questions to the facility licensee for subsequent use.

f. The proctor will remind the examinees to leave the examination area, as defined by the examination team. 1

3. Written Examination Evaluations Using the examination and answer key, the facility and NRC will independently grade each section of the written examination and will complete the grading of all written examinations within 10 working days of the examination adr.inistration date. NRC examiners will record the grades on the written examination cover sheet (Form ES-602-2) and complete Form ES-403-1, " Written Examination Grading Quality Assurance Checklist. "

An individual's grade will be obtained by summing the points credited to the examinee on both sections of the examination, and dividing by the total points available (i.e., compensatory grading methodology.) l To pass the written portion of the examination, operators must achieve an overall score of 80 percent on the written examination.

4. Test Item Evaluation If a number of test items require significant modification during the grading of the examination (e.g., more than 10 percent of the it es are deleted or the answer is changed from the original key) the NRC will determine the root cause and reflect it in the examination report. As NUREG-1021 8 of 29 Interim Rev. 8, January 1997

I ES-602 1 discussed in ES-60i, if significant deficiencies exist in the facility's quality control of their examination bank, the NRC will consider them as part of the program evaluation. If technical flaws that have some degree of safety significance are l found in procedures while analyzing the answers to the written 1 examination, the facility may institute an immediate procedural change and inform all operators of the change. E. ATTACHMENTS / FORMS Attachment 1, " Guidelines for the Development and Review of Open-Reference l Examir.ations" ' Form ES-602-1, "NRC Checklist for Open-Reference Test Items" j Form ES-602-2, " Written Examination Cover Shaet" l l i i I I l l NUREG-1021 9 of 29 Interim Rev. 8, January 1997

l ES-602 Guidelines for the Development Attachment I and Review of Open-Reference Examinations l A. INTRODUCTION The following guidelines are intended for those who are involved in developing or reviewing test items for the written portion of the NRC requalification examination. As described in ES-601, " Administration of NRC Requalification Program Evaluations," the written examination consists of two sections. Section A utilizes a static simulator to provide the context for questions on plant and control systems, and Section B focuses on plant procedures and administrative controls. Examinees may use references, including simulator displays, for both sections. Open-reference written examinations are used for two reasons:  ;

1. Examination Validity '

By permitting the use of referenras that are available to the control room operators, the conditions and regoirements of the written examination more closely approximate those of the actual job. The information provided to the operators in the test items should closely I parallel the information typically available to them, while the responses elicited by the questions should be related to the decisions, solutions, and actions required for effective job performance. In other { words, consulting references more closely correlates job demands and i test demands - a cornerstone of examination validity.

2. Level of Knowledae Use of references enhances examination validity by elevating the level of knowledge of the test items. As described later in these guidelines, ,

operator access to references precludes the use of questions that test I for the mere recall of facts and specifics. Instead, open-reference 1 test items require test takers to demonstrate that they can find, apply, analyze, evaluate, or otherwise use knowledge to handle the problems and issues encountered on the job. E. OPEN-REFERENCE GUIDELINES I Most principles for effective test item construction apply equally to all l types of written questions, regardless of format. Therefore, those who ' develop and review open-reference test items should consult Appendices A and B l of this NUREG in addition to the following guidelines. 1 NUREG-1021 10 of 29 Interim Rev. 8, January 1997

I ES-602 2 Attachment 1

1. Selection of Test Tonin >

Select test item topics for the NRC requalification examination, using the following criteria.

a. Requalification Training Program Curriculum Base the test topics on the curriculum of the most recent operator requalification program training cycle. However, the NRC may substitute up to 20 percent of the examination topics selected by  !

the facility with subjects not emphasized during the requalification cycle. Emphasize knowledges that are of high importance in terms of safety significance in these test items.

b. - Performance Basis Like the requalification program itself, draw the test topics from >

a job-task analysis (JTA) for an R0 and an SRO. The facility  ; licensee should validate each test item by demonstrating a link i between each item and the following JTA products:  : important operator tasks as identified by the JTA O important K/As (rated 3.0 or higher) as identified in NUREG 1122/1123 or a facility-specific K/A catalog facility learning objectives identified as important to safety

c. Adequacy of Test Coverage The facility's proposed sample plan (or test outline) should be checked to ensure that it provides balanced, comprehensive covenge of the requalification training cycle topics. The distribution of proposed facility test items on the examination may be revised if the topics under- or over-represent the material covered in the requalification program. Recent safety-related .

issues and events (e.g., those in relevant licensee event reports (LERs)) should be addressed in the sample plan. ES-601, Attachment 3, " Examination Sample Plan" provides further information on sample plan development.

2. General Guidelines for Sections A and B I

Use the following guidelines to construct and review test items for both  ! parts of the written examination. These guidelines are intended to I supplement, rather than replace, the good practices stated in Appendices l A and B. O NUREG-1021 11 of 29 Interim Rev. 8, January 1997

                                                                                          , - - - , , . , -              ,,e.   .     -,

ES-602 3 Attachment I

a. Operational Orientation As discussed earlier, examination validity is enhanced to the extent that the demands of the test match the demands of the job.

Therefore, in addition to being derived from important K/As and testing objectives, the context and stipulations of test items should mirror the situations encountered in the work setting. The following. example illustrates effective and ineffective ways to design test items from K/As and learning objectives: K/A: . Knowledge of the design attributes of the turbine-driven auxiliary feedwater pump (TDAFWP) differential pressure controller Task: Operate the TDAFWP controls during all modes of plant operation. Learning Objective: The student will be able to operate the TDAFWP differential pressure controller 1 without error during a loss-of-feedwater event. Enabling Objective: After completing this lesson, the student will be able to explain the operation of the TDAFWP differential pressure controller. Poor Test Item: State the parameters used by the TDAFWP differential pressure controller. Better Test Item: Before isolating the "C" steam generator (per EPP11) an operator noted that the transducer-fed auxiliary feed flow indicators for the "C" steam generator were reading gre'ter than the flow indicators to the "A" and "B" steam generators. What is the reason for this flow deviation? Notice that the second test item requires the operator to demonstrate mastery of the knowledge by applying it to an actual job situation. In developing items, ask oneself "Why is the K/A important to satisfactory job performance?" and "In what situation will the operator need this K/A?" The answers to these questions can provide a basis and context for a test item. I NUREG-1021 12 of 29 Interim Rev. 8, January 1997 Ol

 ._. _ _ __ _.___ _ _ _ _ . _ _ _ _ ..._ _ _ _ _ _~ _ _ _ _ _ _ _ _ _.- _ . _ _

i i ! ES-602 4 Attachment 1 4

b. Level of Knowledge

' The operational orientation required of test items on the open-reference examination,_as well as the operators' access to-i controlled documents, precludes the use of questions that test for

mere recall or memorization. Rather than requiring operators to i simply recognize or recall facts and specifics, open-reference test items have the operators demonstrate understanding by using the knowledge to address real-life situations and problems. A ,

j test item at the higher level of knowledge requires operators to [ determine or identify the appropriate fact, rule, or principle to

a novel situation and then correctly apply it. A description of i

each level of knowledge is found in Appendix B. That Appendix and j Table 1 at the end of this Attachment provide sample questions j that illustrate the various levels of knowledge.

c. Realistic Context

! To additionally ensure examination validity, make the situation or j problem posed in the open-reference test item as similar as

possible to the actual situations that operators encounter on the l Job. Situations described in the questions should not only be 4

realistic, but should also be free of common " context" problems, including " backwards logic" and " window dressing." Backwards-logic questions provide operators with information they nonsally have to produce, while asking them for information they normally receive. For example: K/A: Ability to calculate shutdown nargins Backward Logic Item: If the shutdown nargin is 5.5 percent, how long has the unit been shut down? Better Item: The unit has been shut down for x hours. Which of the following is the shutdown nargin? Questions with window dressing have additional, unnecessary information, typically in an attempt to make a memory level item more operationally oriented, as in the following example: The plant has tripped from the effects of a tornado crossing the site boundary. You, as Shift Supervisor, direct the phone talker to complete the 15-minute notificatfon. He informs you that the normal notification network is inoperable. NUREG-1021 13 of 29 Interim Rev. 8, January 1997

I l ES-602 5 Attachment 1 Which of the following do you direct him to use for completing the 15-minute notiffcatlon? Better Item: If the normal notification network is l inoperable, which of the following methods i do you use to complete the 15-minute \ notification after the plant has tripped? l Another common problem when constructing a question with a realistic context is that often "real world" situations have more than one correct solution or response. Carefully check the question and references to ensure that each test item has only one correct answer.

d. Question Novelty One of the most effective ways to ensure that an operator has a high level of knowledge is to present novel situations and require that the operator both realize what information is relevant and how to apply it. If a test question does not contain unique or varied circumstances different from those presented in training, the item will be reduced to eliciting simple recall.

When candidates are able to memorize test items and answers (in their static state) to respond to test items, then we do not really know if they can truly solve the problems or if they have only memorized the answers. Once a test item and its answer has been seen and rehearsed, then it ceases to be a viable discriminator of safe operator performance. It is no longer challenging or testing problem-solving ability; rather, it is simply testing recall. Therefore, test items must be dynamic, replacing or substituting items of like kind and difficulty to preserve integrity in the test discrimination process. Because an infinite number of combinations of plant or equipment parameters and malfunctions may exist at any given time, a true test will compensate for this variation and will become dynamic so that the test can adapt to the infinite number of combinations and still test the same kinds of responses, but to different situations. Review the training material to ensure that questions do not include overly familiar conditions. Keep in mind, however, that all conditions and situations should be reasonable, realistic, and safety-related. NUREG-1021 14 of 29 Interim Rev. 8, January 1997

i I J i ES-602 6 Attachment 1 ! e. The Relationship of Open-Reference Examinations and Direct Look-up Questions Direct look-up questions are associated with open-reference examinations. The key phrase here, " direct look-up," conveys the meaning that little mental activity is involved other than simply

copying an answer, readily available in a reference, to a question

! (i.e., simple recall of where to find the information). Merely

omitting from the stem of the question any mention of where to 4

find the answer does not make it an acceptable open-reference question. i

Do not use direct look-up questions for two reasons. First, these j items only test memory, in that the information is readily i available; this is an inefficient and less valid means of testing candidate knowledge. Second, other than demonstrating that a candidate knows where to find information, this type of question does not test the understanding or analysis of the information that can be applied on the job. Consequently, this type of question will not discriminate the safe from the unsafe operator.

The other option is an "open reference" question. Use an open-reference examination to test candidate knowledge for the following purposes: Which reference to use and where to find it? How to apply the information in the reference to the problem? For an open-reference question, the kind and amount of information required to solve the problem would exceed that which could normally be committed to memory, in other words, the NRC does not expect candidates to remember the information needed to solve the problem. In regular closed-reference questions, we expect the candidate to know and understand how systems operate to answer a question with the information provided in the stem of the question. For a closed-reference question, the candidate would not need to consult a reference (i.e., the NRC expects the candidate to know and understand how the systems work to solve the problem, given various conditions set in the problem). Whether an examination is open- or closed-reference, we should, to the extent possible, be testing problem solving or decision making, for at this more complex level of thought, we more closely approximate the job and achieve a valid assessment. NUREG-1021 15 of 29 Interim Rev. 8, January 1997

ES-602 7 Attachment 1 Memory types - Understanding how memory operates is interrelated to understanding why an open-reference question is preferable. Obviously, all that we know or do involves memory. Operationally, however, we can look at memory as falling it,to two categories: simple memory complex memory Simple memory can be viewed as the recall or recognition of simple bits and chunks of information. Simple memory may still be involved even when the volume of information increases, i.e., the amount of information is large but the process is basically simple memorization of more bits of information. Visualize the type of memory required to memorize five letters of the alphabet versus 26 letters, or the recitation of a short vice a long poem, procedure, and so forth. This memorization process does not involve analysis, integration of facts, or problem solving. Rather the process requires repetition, practice, and rehearsal. The difference lies in the amount of information to be recalled and not in the level of mental processing. By contrast, complex memory, as the term suggests, involves a higher level of cognitive processing. The bits and chunks of information must now be combined or integrated to create something I new, solve a problem, predict a response, or make a decision. Therefore, both the amount of information and what is to be done with it makes the cognitive mental processes complex. Naturally, too, some questions will involve greater complexity than others, but the mental processes will be the same: integrating bits of information, combining them, sorting through and distinguishing the relevant from the irrelevant to arrive at an answer to the question. This is the essence of an analysis / synthesis process. As stated earlier, the NRC should evaluate candidates at this complex level, for this level of thought processing most closely approximates that needed on the job. The complex, problem solving level subsumes knowledge of the bits and chunks of information frequently tested at the simple memory level. Therefore, by testing at the complex level, we are also implicitly testing at the simple memory level. As a prerequisite to solving the problem, the candidate recalls and integrates these bits and chunks of information. Therefore, testing at the analysis level and is more efficient than testing at the simple memory level. I NUREG-1021 16 of 29 Interim Rev. 8, January 1997

                                                                                                                                                -. y l

i i ( l ES-602 8 Attachment 1 ) i

A Final Note - Undue emphasis is placed on the term "immediately" in the definition of a direct look-up. Speed of knowing where to j loccte a reference is irrelevant to direct look-up. The NRC i expects candidates who have been trained to quickly locate the j appropriate reference. The speed issue is relevant to whether the j stem of the question contains unnecessary cues to the candidate about where to find the reference. If the intent of the open-

! reference question is to assess whether the candidate knows where

to find the information, then a knowledge where cue should not be in the stem. Part of the value of an open reference exam is to

test the candidate's evaluative knowledge of where to look. If the stem provides unnecessary cues to the reference, a candidate

! can immediately go to the reference and a value of the open-i reference test is lost. } Now, speed in answering the question proper is a function of the i level of difficulty and the thought processes / steps required to i answer the question. Obviously, if the question is a direct look-i up, then by definition, it assesses only simple memory and will be

quickly and easily answered. This type of question should never j have been asked in the first place.

References should be considered " tools" that operators use to solve problems. The correct use of these " tools" is what should . be tested during the open-reference examinatir.n, not just the

recall of facts and specifics. As stated earlier, " direct look-

! up" questions should not be included in the examination; rather, i questions should be structured to determine if operators can identify, locate, or select correct reference information to

produce organized responses and satisfactory solutions to job-

! related problems and issues. An example of a look-up question, i, which should generally be avoided, follows  ; i I Based on the "Alarn Response Procedure" 1ZZ-040-3, what is  ; j the set-point of the high-high containment pressure alarm l (PK25) on VB3? l a. 10 psig t b. 15 psig i ). c. 20 psig ,

d. 25 psig 4

i This question should be rejected because a candidate can easily j find the set-point in the alare response procedure (ARP). Some 4 may argue that knowing how to look up this data in the ARP makes the ites valid, however, no higher order cognitive skills 1 requiring analysis or synthesis of information were required to determine the correct response. Avoid similar questions on 3 NUREG-1021 17 of 29 Interim Rev. 8, January 1997 1 i _ _-_ _ __ _ . _ . _ . _ _ _ . -

ES-602 9 Attachment 1 precautions or prerequisites which are listed in procedures. A better question using reference material would be as follows: Using the current plant conditions (assume ECCS and CS flow rates REMAIN CONSTANT), how nuch time is available before switch-over to containment recirculation? l

a. 3.6 hours

b. 4.2 hours

c. 4.8 hours

d. 5.2 hours l This is a "look-up" question in a sense, but it certainly requires gathering data from the control boards (e.g. ECCS flow, CS flow, and RWST level) and then identifying the correct emergency procedure and locating and selecting the correct graph to determine how much time is left before a specific level is reached in the RWST. It requires use of both the simulator and the plant procedures as references.

Another appropriate question using facility references is: Following a LOCA, automatic actions have occurred as follows: the reactor has tripped and is shut down AFW has actuated and steam generator pressure is being controlled at 2005 psig, using steam dumps to the condenser Containment pressure has risen to 15 psig and no additional automatic actions have occurred. Which of the following Functional Recovery Procedures should be implemented IMMEDIATELY?

a. FR-C1

b. FR-Z1

c. FR-P1

d. FR-Il This question requires identifying which systems should have actuated based on the ESFAS set-points and which critical safety functions are compromised. The operator should refer to the functional recovery procedures to verify which critical safety functions have been compromised. Knowing where to look and what to look for are required to answer this question in a reasonable time.

NUREG-1021 18 of 29 Interim Rev. 8, January 1997 l

                                                                                                 )

i ! l

 '0 g      ES-602                                      10                           Attachment 1 I

The ites could also be used in the simulator section by requiring the operator to look at the control board in the " frozen" simulator to determine the plant conditions and deduce what , critical safety functions were not met. Naturally, the more integration and evaluation required, the more time must be given to answer the question. Another question that makes effective use of reference material: While operating at 100 percent power, VCT and Pressurizer alares and indications show decreasing pressurizer level with two charging pumps operating. Also, the blow-down and main steam radiation monitors have alarmed. While following the appropriate Abnormal and Energency procedures, you, as the Shift Supervisor must evaluate the existing conditions. What energency classification would you declare on the basis of this information?

a. Notification of Unusual Event

b. Alert

c. Site Area Energency

d. General Energency O This question requires the operator to consult references to d classify an event. It requires analyzing the situation, finding the correct part of the EPIPs, and selecting the appropriate classification.

I l

f. Difficulty Level Versus Discriminatory Value L

Test developers sometimes believe, erroneously, that open-reference questions should be more difficult to compensate for the , operators' access to reference material. Frequently, this ) increased difficulty is in the form of requiring knowledge of more I obscure or otherwise unnecessary information. Both open- and closed-reference questions should have the same standard of difficulty; that is, difficulty should be based on the job demands and responsibilities of operators. A question should be constructed so that it effectively discriminates a competent operator from one who is not. A high K/A value should not be confused witt, the difficulty or discriminating ability of a qtastion.

g. Time Limits Operators take considerably longer to answer open-reference questions than closed. (Weaker operators especially have been found to spend an inordinate amount of time consulting references

( NUREG-1021 19 of 79 Interim Rev. 8, January 1997

ES-602 11 Attachment I versus writing responses). Provide the operators ample time to complete the examination, although not so much time that less-than-competent operators have the opportunity to locate answers without prior familiarization with the topic. The following guidelines should be used to determine the appropriate length of the examination: (1) A competent operator should complete the combination of Section A and Section B in three hours. Give the operators an appropriate amount of time to review Sections A and B based on the number of questions assigned to that section. For example, if Section A has 15 questions and is validated for 45 minutes, operators should be allowed 15 minutes for review. Likewise, if 20 questions are selected for Section B and time validated for 90 minutes, 30 minutes should be allowed for review. The time allocated to review Sections A and B must be included in the three hour time limit. (2) Questions should be developed so that Sections A and B each have approximately 15 to 20 points, for a total test value of 30 to 40 points. The examination sample plan should be used to determine the exact number of questions to be asked in each section. As noted in Appendix B, multiple-choice questions are preferred, but other formats are acceptable. No question will be worth more than two points. (3) In an open-reference examination every answer need not require the operator to use a reference. The individual developing the questions should make a reasonable estimate of the time required to answer each question and identify any references used to obtain a response. Whether and to what extent references are needed affect what constitutes a reasonable amount of time to develop a , response. For example, if the static scenario is set up for l an abnormal plant transient that requires relatively rapid ' operator analysis or response, then the time allowed to respond to the question should be similar to that required to react to the transient. The NRC does not expect the operator to answer the question as quickly as he would react in the plant, but does expect that an operator would consult few references. Conversely, questions involving scenarios for which an operator would have time to consult many references would allow similar time to develop a response to the question. NUREG-1021 20 of 29 Interim Rev. 8, January 1997

4 l ES-602 12 Attachment 1 i (4) Each proposed examination is expected to be time validated.  ! The best method would result in the examination being taken 4 in near-test conditions by a representative cross-section of plant operators. Then, by taking the average of the time it took for each individual to answer each question, a ) . reasonable time may be established for the test. However, i if a large deviation occurs among test takers on particular questions, they should be asked why they took either an . excessive or relatively short amount of time to answer the question from that anticipated. Responses may lead to '

eliminating certain operators' times from the averaging i process and, thereby, eliminate anomalies associated with the individual vice the test item. However, logistics

dictate that sometimes only one or a few individuals can 4 participate in validating the time to complete the test. In l any case, the results need to be evaluated carefully for any <

! unanticipated deviations from the amount of time anticipated to complete each item. ! Facility managers responsible for validating the examination l are expected to validate the time for each question l i similarly. When performing time validation of the l examination, these expectations should be made clear to the l facility representatives validating the examination so that i a reasonable estimate can be obtained. q \ l h. Correct Mode of Measurement I No matter how high their importance ratings or operational relevance, certain operator knowledges, skills, and abilities are not amenable to written testing, as in the following example: 1 Arrange the major steps in the proper sequence to start, , parallel, and load DG-2: i use governor control to increase DG-2 KW 1 raise DG speed to 900 RPM 1 natch voltage with bus 1A2 voltage  ; ! close breaker 1AD2 l i i Despite its operational orientation, the underlying skill , addressed in this test item would be better assessed by having the j operator simulate or perfom the steps during either the simulator or walk-through portions of the operating examination. i NUREG-1021 21 of 29 Interim Rev. 8, January 1997  ! 1

l l l ES-602 13 Attachment 1

3. Specific Guidelines for Section A. " Plant and Control Systems" The following guidelines are specific to the Plant and Control Systems section of the written examination as performed on a static simulator.

These guidelines are divided into two sections, namely " Question Development" and " Simulator Setup."

a. Question Development To ensure that the operators' knowledge of systems and integrated I plant response is adequately evaluated, Section A of the written examination should incorporate the behavior of systems and controls in normal, abnormal, and emergency plant conditions. l Questions should require the operators, to the extent possible, to  !

refer to control room indications in formulating their responses, as in the following example: , Which one of the following describes the location of the steam break? l

a. Inside containment, upstream of the steam line flow transmitters

b. inside containment, downstream of the steam line flow transmitters

c. outside containment, between "C" MSIV and "C" main l steam line check valve

d. outside containment, between "C" MSIV and "C" aain stean line containment penetration The scenario used should put the plant at some point in a major

, plant transient (e.g., LOCA, SGTR, loss of all AC) with several passive or active failures incorporated. However, the number of  ! malfunctions or failures included in the scenario should be ' limited. No more than four minor failures should be used (e.g., failure of a safety-related pump to start, failed pressurizer

 ,              pressure indication, nuclear instrumentation failure). Four will provide sufficient effects to test a wide range of objectives.

Such a scenario would provide sufficient visual cues to develop a good percentage (at least 50 percent) of questions directly related to the existing plant conditions. Questions may be used that do not relate to the transient but use the simulator as a frame of reference only, provided the operators are aware of this lack of relationship to the transient. Carefully ensure that multiple questions stemming from one event do not give each other away. The operator should be able to NUREG-1021 22 of 29 Interim Rev. 8, January 1997

ES-602 14 Attachment I understand and correctly answer each question, based only on the information given in the question, rather than on the answer to a previous question. Use of-plant diagnostic questions for which the examinee attempts to detemine what transient has occurred are generally not suitable, given the purpose of this section of the examination. Having the operator attempt to identify what took place may limit the number of questions you may ask about the transient. Indicate what symptoms or events have occurred, which arocedure has been - implemented, and the point in the procedure t1at was reached at the time the simulator was " frozen." The operator's response should either determine the root cause of the actual system or component failure, or by using "what if" questions, propose a future event and ask for the expected response.

b. Simulator Setup Before the test, advance the simulator recorders to provide clean readings and check the recorders for proper operation.

Check all indications (e.g., bulbs, meters, manual loader indications) to ensure they are in proper working order. When the simulator has been frozen, secure the chart recorder drive power, if necessary. Before administering the test, verify the simulator indications to ensure they are what'is expected to arrive at the correct answer. Freeze any "first-out" annunciators that would normally blink to announce first-out conditions and provide them to the operators. If a transient is stabilized by use of plant procedures, note the step at which the simulator is frozen and record this information on the simulator operations summary sheet. Give the examinees, as necessary, the progress of the procedure step in effect.

4. Ideas for Open Reference Formats Table 2 provides a list of sample formats to assist individuals who are developing performance-based, open-reference questions.

Table 3 provides additional guidance on the process for developing open-reference questions. NUREG-1021 23 of 29 Interim Rev. 8, January 1997

l I i ES-6C2 15 Attachment 1 EXAMPLES OF DIFFERENT TYPES OF OVESTIONS Table 1

1. Memory level questions are not to be used on open reference examinations.

2. Comorehension level questions would require the operator to demonstrate an understanding of a knowledge without necessarily relating it to other material, or fully comprehending it in depth.

A spurious safety injection (SI) signal resulted in HHSI flow to the loop cold legs when the plant was in node 4. After completing corrective actions for the inadvertent SI initiation, you must

a. stroke test the cold leg motor-operated stop valves within 24 hours.

b. test the cold leg injection check valves for leakage within 48 hours.

c. stroke test the cold leg motor-operated stop valves before entering node 3.

d. test the cold leg injection valves for leakage before entering node 2.

3. Analysis. synthesis. and aoolication level questions require higher-order cognitive thought processes.

a. Aeolication questions may require the operator to apply the l knowledge to various concrete situations.

l Given the following conditions: both CFPTs tripped CA automatically started CA valves reset to control steam generator water level l CA suction pressure decreases to seven psig Which ONE of the following describes CA pump response for the given conditions? j

a. the pump suction will automatically shift to RN

b. the pump suction will automatically shift to UST

c. the pump will trip when suction pressure decreases to 5 psig

d. the pump will trip after a 6-second delay NUREG-1021 24 of 29 Interim Rev. 8, January 1997

i l I J ES-602 16 Attachment 1 2 Table 1 l

b. Analysis questions require the operator to mentally integrate a  ;

number of conditions, ualyze their interrelationships, sort through and discrimint.te among distractors, and finally choose the  ; correct answer.  ! Which answer below correctly indicates the posting required for a room using the results of the following radiological l survey? l SURVEY RESULTS: AIRBORNE ACTIVITY: FLOOR SNEAR: 6.34 Beta-610 E-9 ug;f/cc (Co-60) dpn/cn E EQUIPNENT SNEAR: Beta-1800 dpn/cmAy:ha-4dpn/cnAlpha-16 dpn/cn i GENERAL RADIATION LEVEL: 110 nr/hr l

a. Radiation Area, Airborne Area and Full Anti-Cs

b. High Radiation Area, Airborne Area and Full Anti-Cs

c. High Radiation Area, Full Anti-Cs

d. Locked High Radiation Area, Airborne Area, Double Antf-Cs

c. Problem solvina questions require putting together elements to demonstrate an understanding of the underlying knowledge.

The plant is operating at 100 percent power when a LOCA occurs. The reactor trips automatically, but fast transfer falls and buses 1A1 and 1A2 become de-energized. PPLS and CPHS initiate and all equipment operates as designed. Which ONE of the following is the expected systen response?

a. OPLS initiates load shed and starts both energency diesel generators

b. OPLS does NOT actuate; the energency diesel generators start and re-energize buses 1A1 and 1A2

c. OPLS does NOT actuate; the energency diesel generators do NOT start and safeguartis notors are started by the sequencers

d. OPLS does NOT actuate; the energency diesel generators run at idle speed and safeguards notors are started by the sequencers NUREG-1021 25 of 29 Interim Rev. 8, January 1997

i l { ES-602 17 Attachment 1 EXAMPLE FORMATS FOR OPEN-REFERENCE OVESTIONS Table 2

1. Given Plant / System / Component Condition / Problem diagnose cause of the problem identify location of problem predict effect on plant / system / component l

identify precipitating events / actions classify / indicate if conditions meet specified criteria indicate & utilize proper procedures / references identify appropriate recuperative actions l 2. Given Plant Conditions and Operator Actions / Procedural Steps l Implemented indicate purpose / reason behind taking these actions determine if the correct actions were taken given available cues irdicate what further actions are required to achieve a desired l effect

3. Given a Proposed or Hypothetical Course of Action / Recommendation determine the appropriateness or acceptability predict expected plant / system / component response predict effect on other systems / components

4. Given Data Regarding Plant Conditions or Parameters compute or determine status or change in other parameters ,

utilize charts, curves, graphs, etc., to perform calculations or l estimations I l l NUREG-1021 26 of 29 Interim Rev. 8, January 1997 l

ES-602 18 Attachment 1 DEVELOPING OPEN-REFERENCE TEST ITEMS Table 3 Here are the decision steps and mental model for developing analysis-level open-reference questions:

1. Determine the purpose of the test. Do you want to test knowledge where and knowleoye whatt

2. Determine the information needed to respond to the question. Is the volume and kind of information such that you would not normally expect candidates to recall the information from memory to answer the question 1

3. If the answer is yes to both of these questions, develop an open-reference question.

4. Construct the question as two tiers:

Tier / Purpose Process Criteria Outcome

1. Knowledge where Evaluate preference Avoid clues Locate ref.

sources in stem sources O 2. Knowledge what/how Integrate multiple variables, events Info. volume and detail high (not in memory) Identify correct answer M STION STEM bits, chunks of stem information (conditions, set points, components, etc.) 1 1 Mental Processes Inteorative Processes

• Analyzing a. Answer
• Integrate mental
• Sorting b. Distractor processes with
• Eliminating < > c. Distractor < > stem information,
• Differentiating d. Distractor reference data,
• Evaluating and distractors Determine Answer (b/ NUREG-1021 27 of 29 Interim Rev. 8, January 1997 j i

l m

l ES-602 NRC Checklist for Form ES-602-1 1 Open-Reference Test Items Test Item level i l

1. Does each test item have a documented link to important operator {

tasks, K/As, and/or facility learning objectives?

2. Is each question operationally oriented (i.e., is there a correlation between job demands and test demands)?

3. Is the question at least at the comprehension-level of knowledge?

4. Is the context of the questions realistic and free of window dressing and backwards logic?

5. Does the item require an appropriate use of references (i.e., use of analysis skills or synthesis of information either to discern what procedures were applicable or to consult the procedures to obtain the answer)?

6. Is the question a " direct look-up" question, or does one question )

on the examination compromise another? A " direct look-up question" is defined as a question that only requires the examinee to recall where to find the answer. i

7. Does the question possess a high K/A importance factor (3.0 or greater) for the job position? 9,

8. Does the question discriminate a competent operator from one who ,

is not? j

9. Is the question appropriate for the written examination and the selected format (e.g., short answer or multiple choice)?

10. Do questions in Section A take advantage of the simulator control l room setting? j i

11. Does any question have the potential of being a " double-jeopardy"  ;

question? 1

12. Is the question clear, precise, and easy to read and understand?

13. Is there only one correct answer to the question?

14. Does the question pose situations and problems other than those 4 presented during training? )

15. Does the question have a reasonable estimated response time?

NUREG-1021 28 of 29 Interim Rev. 8, January 199/ O!

l l i

                                                                                              )

ES-602 Written Requalification Examination Form ES-602-2 Cover Sheet 1

4 U. S. Nuclear Regulatory Commission

Written Requalification Examination i

Operator Information Name: Date: Region: I / II / III / IV Facility / Unit: Reactor Type: W / CE / BW / GE Start Time: Stop Time: Instructions Use the answer sheets provided to document your answers. Staple this cover sheet on top of the answer sheets. Points for each question are indicated in parentheses after the question. The passing grade is 80.00 percent.

 \         You have a total of three hours to complete both sections of the examination.

Operator Certification All work done on this examination is my own. I have neither given nor received aid. Operator's Signature Results Test Value (Points) Section A: Section B: TOTAL: Operator's Score (Points) Section A: Section B: TOTAL: Operator's Grade (Combined) Percent O b NUREG-102I 29 of 29 Interim Rev. 8, January 1997

ES-603 REQUALIFICATION WALK-THROUGH EXAMINATIONS A. PURPOSE NRC examiners, working with facility evaluators, follow this standard to administer walk-through requalification examinations as authorized by Title 10, Section 55.59(a)(2)(iii), of the Code of Federal Regulations (10 CFR 55.59(a)(2)(iii)). The walk-through examination is an effective tool for evaluating the ability of a licensed operator to manipulate system components and controls, interpret references, use administrative procedures, and demonstrate knowledge of component locations. B. SCOPE This standard provides specific guidance and requirements to NRC examiners for preparing, reviewing, and administering walk-through requalification examinations. -In the walk-through examination, each operator performs five job performance measures (JPMs). Each operator's examination is designed to test the operator on plant systems that are important to the safe operation of the reactor. NRC examiners and facility evaluators jointly approve the JPMs for each examination. Each JPM consists of several steps, one or more of them is designated " critical." An operator has to properly complete each critical step to pass the JPM. O The examination team will agree on five JPMs so that a minimum of two are conducted in the simulator (or the control room) and at least two are conducted in the plant. When operators perform JPMs in the plant or the control room, they will be cautioned not to manipulate the reactor controls. To successfully complete these JPMs, operators will demonstrate to the examiners the steps or actions they would take to complete the task. To the maximum extent practical, control room JPMs will be conducted using the simulator. C. JPM DEVELOPMENT

1. Facility Exam Team Members' Respons%ilities

a. The facility licensee is expected to identify the plant systems that are critical to protecting the public health and safety. The systems that are selected for the examination should meet the following criteria:

systems covered during the current requalification cycle; the facility's sample plan (see ES-601, Attachment 3) should identify the systems and appropriate learning objectives new or recently modified systems NUREG-1021 1 of 7 Interim Rev. 8, January 1997

I 1 ES-603 systems that are the subject of recent facility licensee i event reports (LERs) or vendor notices probabilistic risk assessment (PRA) identified risk-ir.portant systems, components, and operator actions' for plant or vendor generic systems systems that are the subject of NRC Information Notices systems that are important to safety during low power or shutdown operations

b. For those systems identified as being important to safety, the facility representatives are expected to review the job task analysis (JTA), learning objectives, and NUREG-1122 or -1123, the

             " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized [or Boiling] Water Reactors." The facility representative should highlight for use as JPMs the tasks, abilities, and learning objectives that fulfill the following criteria:

apply to the facility are at the appropriate level for the operator being examined (reactor operator (RO) is responsible for auxiliary operator (AO)/R0 tasks, and the senior reactor operator (SRO) is responsible for A0/R0/SR0 tasks) have a K/A rating of 3.0 or higher (Tasks and abilities may be selected for use that have ratings below 3.0 if proper facility justification exists for these ratings. ) If a facility-specific knowledge and ability (K/A) is used in lieu of NUREG-1122 or -1123, the importance ratings must be based on protecting the public health and safety.

c. Many tasks that are important to safety are unique to a specific plant and are not referenced in NUREG-1122 or -1123. Each facility is expected to maintain a list of these plant-specific tasks and develop JPMs that test the operators' knowledge and ability in these areas. The facility is responsible for ensuring 1

Chapter 13 of NUREG-1560, " Individual Plant Examination Program: l Perspectives on Reactor Safety and Plant Performance," identifies a number of ' important human actions that may be appropriate for the operating test. In determining important operator actions, do not overlook actions that are relied upon or result in specific events being driven to low risk contribution. This will help identify those human actions, assumed to be very , reliable, that might otherwise not show up on a list of risk-dominant actions. i NUREG-1021 2 of 7 Interim Rev. 8, January 1997 1 I l

f i p ES-603

,                                 that the tasks are appropriate to the applicable license level and
;                                 have a safety importance rating of at least 3.0, before submitting

them to the NRC for review.

l; d. JPMs should meet the guidelines of Appendix C and Form ES-C-2,  !

?
                                  " Job Performance Measure Quality Checklist." The JPMs should indicate which steps are " critical" to successful completion of                             i the task. Critical steps are those which when not performed in                               )

proper sequence, not perfomed at the proper time, or not performed correctly will prevent the system from functioning properly or preclude the successful completion of the task. Form ES-C-1, " Job Performance Measure Worksheet," or an equivalent facility form should be used to construct and format the JPMs. The majority of the JPMs selected for the examination will cover topics from the most recent requalification training cycle. In addition, the facility is expected to create at least ten new JPMs each year until they have a JPM bank that is representative of Sections C.I.a and C.I.b of this standard. The NRC anticipates that the number of JPMs in a facility's bank will be approximately 125 to 150; however, the exact number will depend on the facility's JTA. New JPMs should generally be based on recent requalification training, industry events, facility changes, and tasks for safety-significant systems.

e. The NRC expects the facility to develop " time-critical" JPMs to l evaluate time critical-tasks identified in the facility JTA for each licensed position. To facilitate the selection of time-critical JPMs for the requalification examination, the facility licensee is expected to uniquely identify these JPMs. To successfully complete a time-critical JPM, the operator must perform the " time-critical" steps within a pre-specified time period, in addition to successfully performing all of the critical steps that are not time-critical. The time period identified in the time-critical JPM should be based on a regulatory requirement or a facility commitment with the NRC.

f. The facility is expected to develop " alternate-path JPMs" and include them in their JPM bank. To facilitate the selection of '

alternate-path JPMs for the requalification examination, the facility licensee is expected to uniquely identify these JPMs. Appendix C gives guidance on the development of these JPMs.

2. NRC Examination Team Members' Responsibilities

a. The NRC team will review and approve the JPMs selected by the facility. The majority of the JPMs selected should be based on the systems covered during the most recent requalification cycle. 4 However, the facility should also select JPMs in systems that are important to safety regardless of when they were reviewed in NUREG-1021 3 of 7 Interim Rev. 8, January 1997

l l l l l ES-603  ! requalification training. NRC examiners will review the JPMs l submitted by the facility to ensure that 20 percent of the JPMs  ! selected were not included in the topics in the most recent  ! training cycle, as this examination is intended to sample skills ' and abilities that operators should always be able to display. In l general, examiners should select systems in Groups I and II of the , appropriate written examination model in ES-401, with Group I ' comprising at least 50 percent of the selected systems. I

b. The NRC staff will discuss with the facility representatives the JPMs selected that are not in NUREG-1122 or -1123 to ensure the  !

system or task meets the site-specific importance criteria. Any i modifications to the selection of JPMs will also be discussed with the facility representatives. The NRC may substitute up to 20 percent of the facility-proposed JPMs with NRC-developed JPMs. The NRC will give facility representatives sufficient time to review any substituted JPMs.

c. The chief examiner has the authority to decide the content of each .

examination set. NRC examiners should review the proposed JPMs  ! using the criteria in Appendix C and the " Job Performance Measure Quality Checklist" (Form ES-C-2).

d. The chief examiner will ensure that a sufficient number of different JPMs are scheduled during the examination week to avoid i compromising the examination.

e. The chief examiner will ensure that the time validation of each JPM is reasonable and will verify that each JPM is identified as

                               " time-critical" or not " time-critical."

D. EXAM ADMINISTRATION

1. Conductino JPM Walk-Throuah Examinations ,

a. The facility evaluator is responsible for conducting the walk- i through examination while the NRC examiner observes. The NRC  !

examiner and the facility evaluator may ask the operator questions to clarify his or her performance of the JPM after the JPM is completed. In most instances, the NRC examiner will ask the facility evaluator to question the operator about the appropriateness of an act ton or a response that does not follow the actions specified in the JPM.

b. The facility evaluator will brief the operator, using Parts A, C, and D of Appendix E. If desired, the evaluator may brief the operators as a group before starting the walk-through examination.

c. Operators should not be informed of the expected completion time before commencing the JPM. Informing operators of the expected NUREG-1021 4 of 7 Interim Rev. 8, January 1997

  - ~ - . . .   .    .. -.-.               - - . . .-.     -     - - - - _ - . _ - - - - - - . .

T ES-603 ! completion time may increase tension as they approach the time i limit. However, the evaluator may inform the operator that a JPM i is time-critical, if it is normal practice to do so at that facility. ! d. Time should be allotted during the operating test for evaluating j- the performance of five JPMs. 1 Each walk-through examination should last approximately two hours, t This time includes the validated times associated with each JPM , j that is planned, plus any administrative tasks required to conduct l l the examination. ' L Administrative tasks include the following examples: transit time to and from the plant site time spent complying with facility security and radiological

administrative requirements (unless this is part of the JPM being performed) .

transit time within the plant after a JPM is completed to  ! ! get to a location where the initiating cue for the next JPM

;                                      is to be given 1

l Note: The JPM sample size will be constrained to the requirements i of this standard for NRC-conducted examinations. The facility may l l perform additional evaluatien of its operators outside the time i frame designated for the NRC examination. However, any additional

evaluation by the facility will not be factored into the final  ;

requalification evaluation of the operator by the NRC. The l criteria for the determination of requalification program status

remains the same.

4 1 , e. JPMs that are directly related to the operators' job function are l preferable, particularly for SR0s. For example, if an SR0 will

not perform an emergency action level (EAL) classification during the dynamic simulator or written examinations, then the j examination team may choose to have the operator perform one JPM i that involves classifying an emergency.

f. The NRC examiner will ensure that the facility evaluator conducts

! an appropriate examination. Appendix C provides examples of good I evaluation techniques to look for during the walk-through ! examination. If the NRC examiner observes improper evaluation

techniques that may render the examination invalid, the NRC l examiner will stop the walk-through and counsel the facility j evaluator. If the facility evaluator continues to display poor evaluation techniques, then the NRC examiner will stop the l examination and request that another facility evaluator continue 3

i NUREG-1021 5 of 7 Interim Rev. 8, January 1997 i I . . . -

s ES-603 i the e.< amination. If it is necessary, the NRC examiner may conduct I the walk-through with the original facility evaluator observing and co-evaluating.

g. If an evaluator believes that follow-up questioning is required and is not sure how to phrase the question, he or she should consult the NRC examiner. This will avoid inadvertent prompting of the operator and enhance communications between the facility 4 evaluator and the NRC examiner.

1

h. The examinar will document the operator's performance using the applicable portions of a JPN worksheet, Form ES-C-1, or the facility equivalent for each JPM. Document any questions asked to clarify the operator's performance. Also fill out the JPM summary 1 matrix, Form ES-603-1, to maintain operators' scores during the examination, document which JPM ead operator performed, and to meet the requirements of ES-601, Section J.1.b.

i. After completing an operator's JPM set, the NRC and facility  !

evaluators shall discuss and resolve any outstanding issues that may result in the operator failing the walk-through examination or 1 any individual JPM. Many times a discussion of what was observed l will serve to correct a difference of opinion. Unresolved differences should be brought to the attention of the chief examiner.  ;

2. Gradina the Examination

a. To pass the walk-through examination, each operator has to I successfully complete at least four of five JPMs. To successfully complete a JPM, the operator has to complete all critical steps and satisfy the completion criteria specified in each JPM.

b. Each JPM is expected to be completed within the validated time period. For a JPM that is not time-critical, the validatei time may be exceeded if the facility evaluator and the NRC examiner agree that the operator is making acceptable progress toward completing the JPM.

For time-critical JPMs, the facility should identify a period that they consider to be the absolute maximum time in which they would expect an operator to perform this task (e.g., locally opening reactor trip breakers on an ATWS or locally starting an AFW pump on a loss of all feedwater). An operator that fails to meet the , time criteria will receive an unsatisfactory evaluation on that l JPM. E. ATTACHMENTS / FORMS l Form ES-603-1, "JPM Summary Matrix" NUREG-1021 6 of 7 Interim Rev. 8, January 1997 i

i ,- . -  ; i  !

1 7

       -                                                                                    9
 -O 3

0 6 S 9 1 y r E a u m n r a o J F 8 . v e R i m r e t _. n I i x r t a M 7 y r f a o

   '   m                                                                                    7 u

S M P J n i o t p n i r c s s e e D m a f N e i r

             's r

B o / t r a e r b e p m O u N M P J 1 2 0 1 3 - 0 G O 6 5 E

                           .     .    .     .   .   .   .   .     . 0   1   2   3   4        R S

1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 U E N

4 ! ES-604 !- DYNANIC SINULATOR REQUALIFICATION EXAMINATIONS i A. PURPOSE NRC examiners use this standard in preparing and administering dynamic simulator requalification operating tests in accordance with the provisions of Title 10, Section 55.59(a)(2)(iii), of the Code of federal Regulations (10 CFR 55.59(a)(2)(iii)). By simulating actual plant operation, the dynamic simulator test provides a comprehensive evaluation of the integrated plant knowledge and skills required of operating crews. It is effective in evaluating a crew's communication skills and team behavior and in identifying any areas in which the licensed operators should be retrained to improve their knowledge and abilities (K/As) in accordance with the provisions of the requalification program developed by the facility licensee. B. SCOPE The dynamic simulator test consists of two scenarios. Each scenario is constructed to last approximately 45 to 60 minutes. The actual time needed to complete the scenarios will depend upon the specific events within the scenarios but should allow the crew the time necessary to perform the actions required to respond to each event. To successfully complete this portion of O the operating test the crew must demonstrate the ability to operate effectively as a team while completing a series of critical tasks (cts) .that measure the crew's ability to safely operate the plant during normal, abnormal, and emergency situations. The NRC examiners evaluate the performance of each crew, using standard competency rating scales. Each competency is rated.according to the crew's ability to satisfactorily complete the tasks that have been designated as

                        " critical" within that crew's scenario set. Critical means "necessary to place and maintain the reactor in a safe operational or shutdown condition."

Each valid CT must meet the criteria specified in Section D of Appendix D. If the crew fails to correctly perform a CT, that failure would indicate a significant deficiency in the knowledge, skill, or ability of that crew to demonstrate team behavior and will be evaluated, using the behavioral anchors on the " Simulator Crew Evaluation Form," Form ES-604-2. The facility evaluators will evaluate the performance of the operators during the dynamic simulator test. Because the primary purpose of the dynamic simulator test is to evaluate crews, each individual is not required to perform a specific number of cts or necessarily receive an individual evaluation by an NRC examiner. However, NRC examiners will follow up on significant individual performance deficiencies on cts observed during the simulator test in a manner and setting compatible with the deficiency. A significant perfonmance deficiency is the omission of or the inability to complete a critical task, or the demonstration of a significant lack of O NUREG-1021 1 of 20 Interim Rev. 8 January 1997

knowledge or ability while performing a critical task. This follow-up evaluation will be graded as a component of the individual's operating test. To meet the requirements of 10 CFR 55.59(a)(2), it is the facility licensee's responsibility to conduct its annual operator performance evaluations on the dynamic simulator in accordance with the requirements of its requalification program. The NRC-conducted operating test may be used by the facility licensee to meet this requirement if the conditions of 10 CFR 55.59(a)(2)(ii) are satisfied. If an operator demonstrates significant performance deficiencies linked to the execution of cts during the dynamic simulator portion of the operating te t, then the facility and NRC examination team members should discuss those deficiencies at the end of the dynamic simulator test. If the operating n ew performs satisfactorily and NRC examiners observe no significant individual performance deficiencies linked to cts, the individual would pass the dynamic simulator test. In the case of operators with significant deficiencies identified while )erforming cts, the facility evaluators and NRC examiners will decide wiether the o)erator would pass or fail by asking the operator follow-up questions about 11s or her performance to determine the extent of the knowledge or ability deficiency demonstrated. The number and scope of follow-up questions to be asked will be agreed to by the NRC examiners and facility evaluators and will be based on the individual's demonstrated knowledge or ability weakness identified during the performance of cts. The follow-u) questions and individual's answers will be documented and used, along with t1e individual's performance, as the basis for a pass or fail decision. Section E.2 of this standard describes the method for evaluating and documenting individual performance. In the rare event thet the only way to evaluate the scope and depth of the individual's performance deficiency is by conducting another scenario to gain additional information, then the examination team (NRC and facility) will determine the content, critical tasks, operator actions, and crew position rotation necessary to complete the evaluation of the individual's performance. Conducting another scenario is time consuming and may have an adverse effect on the examination process. If an individual operator exhibits only minor deficiencies in performance and completes the testing requirements of 10 CFR 55.59(a) satisfactorily, remedial retraining and reevaluation will be conducted in accordance with the facility licensee requalification program. C. EXAMINATION DEVELOPMENT Developing the NRC dynamic simulator requalification examination is a combined effort between the facility representatives and the NRC examiners on the examination team. The responsibilities of the members of the examination team are outlined below.

1. Facility Team Member Responsibilities

a. The facility licensee develops the dynamic simulator scenarios with identified cts that meet the guidance specified in Appendix D NUREG-1021 2 of 20 Interim Rev. 8, January 1997

1

i

ES-604 '

i U and Form ES-604-1, " Simulator Scenario Review Checklist." The 4 facility licensee submits each proposed dynamic simulator test to l 4 the chief examiner 45 days before the scheduled examination. ) b. The facility licensee is expected to provide a qualified simulatur j operator to assist in developing and administering the simulator i examinations. The simulator operator must be available to support the examination team during the examination preparation week, ! normally 2 weeks before the examination. The simulator operator will be expected to sign a security agreement at the time that the

chief examiner determines that he or she has access to specialized

{ knowledge of any part of the examination.

c. The scenarios should be based on the training that was conducted during the requalification cycle, recent industry events, licensee event reports (LERs), emergency and abnormal procedures, and design and procedural changes. The scenarios should demonstrate  !

the crew's ability to use facility procedures for preventing and ' mitigating accidents. Some scenarios should be based on the dominant accident sequences (DAS) for the facility or actual events that have occurred at that or a similar facility. DAS are l those sequences which contribute significantly to the frequency of ' core damage as determined by the facility licensee's probabilistic risk assessment (PRA) or individual plant examination (IPE). The PRA/IPE,should also be used to identify risk-important operator (i actions . In determining those actions, do not overlook actions i that are relied upon or result in specific events being driven to  ; low risk contribution. This will help identify those human ' actions, assumed to be very reliable, that might otherwise not show up in a list of risk-dominant actions.

d. The facility representatives on the examination team will be given the opportunity to review any modifications the NRC made to the scenarios. The representatives may recommend changes to events that are critical to plant safety but must substantiate the reason 4 for those changes. The examination team has to agree on the '

validity and content of each scenario before the examination. 4

e. The NRC encourages the utility to have their management discuss with the NRC any problems with examination complexity. Utility managers engaged in the examination review will be subject to signing a security agreement. Responsibility rests with the utility to resolve any issues before administering the examination. This review is to ensure that the final scenarios Chapter 13 of NUREG-1560, " Individual Plant Examination Program:

Perspectives on Reactor Safety and Plant Performance," identifies a number of important human actions that may be appropriate for evaluation on the dynamic simulator operating test.

    - NUREG-1021                                   3 of 20   Interim Rev. 8, January 1997

ES-604 are (1) consistent with the facility's requalification requirements for operators licensed at the facility, (2) within the capability of the simulation facility, and (3) within the scope of the facility's procedures. This utility senior manager or representative should communicate any significant concerns about scenario validity to the chief examiner. If adequate resolution is not reached, the concerns should be brought to the attention of regional managers and then, if necessary, to managers at the Office of Nuclear Reactor , Regulation (NRR), to resolve these concerns. I

2. NRC Team Member Responsibilities j

a. At least two weeks before the preparation week, the chief examiner or a designee completes a draft of Form ES-604-1, " Simulator Scenario Review Checklist," for each scenario proposed by the facility to use during the examination, along with any proposed changes to be validated during preparation week. During the i review of each scenario that the facility selected for the l examination, the chief examiner or designee will consider the  !

quantitative and qualitative factors described in Appendix D, as surunarized on Form ES-604-1. l

b. If the proposed scenarios require major changes to meet the  !

guidance provided on Form ES-604-1, the chief examiner informs ' regional managers and determines the appropriate course of action to take. The NRC may revise the scenarios, as appropriate, or develop scenarios to add to the facility's scenarios, if required. All changes to the scenarios will be communicated to the appropriate facility representative in sufficient time prior to the preparation week to allow for scenario validation. Minor changes needed to ensure the scenario objectives are properly accomplished may be made during the preparation week. The NRC staff reviews the final scenarios with the facility's examination team representatives before the examination is administered. The NRC has the final authority to decide the content of the scenarios and determine whether a task is critical for evaluating the competency of the crew.

c. A key element of the examination team's resolution of concerns with scope, depth, and complexity of simulator scenarios involves the observation of the proposed examination scenarios by a senior utility manager (subject to signing an appropriate examination security agreement) during examination preparation. This executive would, if necessary, raise specific concerns to appropriate NRC regional management for resolution before the examination is administered.

NUREG-1021 4 of 20 Interim Rev. 8, January 1997

4 ES-604 i ,-) D. EXAMINATION ADMINISTRATION

1. Administrative Reauirements

a. A facility manager or representative with responsibilities for I conducting plant operations (as a minimum, a manager at the first level above shift supervisor) should be present while the

, simulator examinations are administered. The NRC chief examiner l 1s the principal point of contact between the facility manager and the NRC. The chief examiner or a designee is present during the administration of each dynamic simulator examination. i l

b. The examination team briefs the operating crews before the start '

of the simulator scenarios, using the information in Parts A, C, and E of Appendix E.

c. Crews :hould be given adequate time to respond to all planned and unplanned events. A scenario's contact time should be approximately 45 to 60 minutes. Contact time means the actual time the operators spend in the scenario but does not include time spent on briefings, simulator setup, or investigating simulator l performance problems. '

d. Under no circumstance will the sequence of events and transients O

V be modified by any member of the examination team during the scenario. If the scenario is not properly administered due to a ) simulator operator error or an unexpected simulator response, the i examination team will confer immediately after the scenario set to determine if the crew has performed a sufficient number of transients and events to justify an evaluation of the required l competencies. If necessary, the examination team can run an additional scenario to ensure that the required competencies are covered. l

e. Crew rotation practices shall be discussed and agreed to during the preparation week and any problems resolved before the administration of the operating test.

f. The members of the operating crew should maintain the same operating positions as during facility requalificatica evaluations. The crew members should rotate between positions in the manner identical to the facility's rotation practices for evaluations specified in the facility's requalification program.

g. Senior reactor operators (SR0s) must be evaluated in at least one scenario in an SRO-licensed crew position. More than two simulator scenarios may be required to examine crews that consist of more than four SR0s.

NUREG-1021 5 of 20 Interim Rev. 8, January 1997

l, ES-604 l l

2. Post-Scenario Activities

~

a. If the NRC examiners and facility evaluators observe actions that ,

are unclear during the simulator scenario, they should question I the crew members as necessary to obtain complete documentation on the performance of events during the scenario. Questions should be factual and should clarify performance related to observations.

b. If an examiner observes an individual who demonstrates significant deficiencies performing a critical task, the NRC examiner and the facility evaluator will discuss those deficiencies at the completion of the scenario. If they determine that the operator's performance deficiency could not be assessed due to a lack of information, the examination team has the option to conduct an ,

additional scenario or a job performance measure (JPM) to obtain I this information. During the post-scenario discussion, the facility evaluator is I expected to describe the operator's deficiencies to the NRC examiner and suggest a series of follow-up questions designed to identify the cause of the deficiency. The NRC examiner will assess the facility evaluator's ability to diagnose the operator's deficiency and document it in the examination report, if applicable. The NRC examiner has the option to augment the follow-up questions proposed by the facility evaluator, if necessary. The examination team should minimize the time it takes to conduct this review of crew and individual performance to minimize the impact on the operators. However, it is the examination team's responsibility to ensure the review is thorough and complete. The facility evaluator will conduct an individual evaluation of the operator in accordance witt. Section E.2 of this standard. The NRC examiner has the option to ask additional follow-up questions.

c. Upon completing any follow-up questioning, NRC examiners and the i facility evaluators dismiss the crew to await the next scenario i and inform the crew that they may discuss the completed scenario among themselves.

d. The NRC examiners and facility evaluators meet separately to compare observations and determine if any cts were omitted or 1 incorrectly performed by the crew. l

e. The facility evaluators discuss the crew's performance with the NRC examiners after each scenario to clarify any performance deficiencies that have been noted. The examination team determines if the as-run scenario has invalidated any j predesignated cts or if any new cts should be designated for i NUREG-1021 6 of 20 Interim Rev. 8, January 1997

ES-604 t

    \

evaluating unpredicted events or actions taker by the crew during the scenario. The examination team revalidates the cts in each scenario, using the methodology contained in Appendix D.

f. After the crew completes the last scenario, the NRC examiners and the facility evaluators independently complete a " Simulator Crew Evaluation Form," Form ES-604-2, as discussed in Section E. The facility evaluators also evaluate individual operator performance 1 in accordance with their requalification program requirements and Section E.2. The NRC examiners review the facility's evaluations i of individual operator performance after completing each crew evaluation.

E. PERFORMANCE EVALUATIONS Two separate evaluations are conducted based on the information obtained l during the dynamic simulator examination. First, a crew simulator evaluation is performed. Next, individual simulator performance is used by the examination team to determine whether follow-up questioning of the operator is necessary. The examination team may conclude that, after observing the operator's performance in the dynamic simulator and evaluating the response to follow-up questions, additional performance l information about the operator must be obtained to make an individual evaluation. In this case, an additional scenario or JPM would be conducted. The individual follow-up would then be documented along with  ; the individual's crew evaluation on Form ES-601-5. ' The operator is subject to failure based on a competency evaluation of the operator's performance on the dynamic simulator and the required follow-up evaluation, if deficient performance was exhibited by the l operator executing a crew critical task.

1. Crew Simulator Evaluations After administering the dynamic simulator scenario set as discussed in Section D, the NRC examiners and facility evaluators independently evaluate the crew's performance by completing a copy of Form ES-604-2. ,

The facility is expected to provide its final crew evaluations to the ' NRC examiners before the crew members return to licensed duties or the end of the examination week, whichever is sooner. Specific guidance for completing Fom ES-604-2 is on the first page of the form. The results of the crew evaluations are factored into each individual's examination results and the facility requalification program evaluation. , Members of a crew that receive an unsatisfactory crew evaluation are ! expected to receive remedial training from the facility licensee and be reevaluated in accordance with the facility licensee's NRC-approved requalification program before returning to licensed duties. Although operators are not required to take an NRC-conducted requalification examination for purposes of license renewal, those that fail to pass l NUREG-1021 7 of 20 Interim Rev. 8, January 1997 I I

ES-604 (individually or as a member of a crew) an examination conducted by the NRC must be reevaluated by the NRC before their license will be renewed. The level of NRC involvement during the reevaluation will be determined on a case-by-case basis (refer to Section F.1 of ES-601). NRC examiners will document the results of each operator's crew performance on Form ES-601-5 in the " Simulator Examination Results" section.

2. Individual Operatina Evaluations Individual operating evaluations on the dynamic simulator examination and the resulting remedial training are primarily the responsibility of the facility licensee. Unsatisfactory operator performance of a crew critical task will be followed up after the simulator scenario and documented on Form ES-601-5.

The facility evaluators are expected to document and grade individual operator perforstace during the dynamic simulator examination in , accordance with the requirements of the facility licensee's ' requalification program. The NRC expects that the facility's grading methodology will identify operator deficiencies and that the facility evaluators will discuss those deficiencies with the NRC examiners during the meetings following the scenarios as described in Section D. The facility evaluators are expected to document deficiencies and remediate and retest the operators for the identified deficiencies in accordance with the facility licensee's requalification training program. The facility evaluators are expected to, at a minimum, identify any operator on the crew who was directly responsible for the omission or incorrect perfornance of validated cts. Individual follow-up is conducted if an operator has significant performance deficiencies linked to a CT. The NRC examiner will assist in the development and administration of follow-up questions specific to the deficiencies displayed by the operator performing the CT as described in Section D.2.b of this standard. The examination team will determine the number and scope of the follow-up questions that will be asked based on a review of the operator's deficiencies at the completion of the scenario. The examination team has the option to gather additional information about an operator who displays performance deficiencies attempting critical tasks by either running an additional scenario or by using JPMs, if the dynamic simulator examination and follow-up questioning are inconclusive. Upon completion of the individual follow-up questions the NRC examiner will complete the evaluation using the appropriate competency grading worksheet from ES-303 (ES-303-3 for reactor operators and ES-303-4 for senior reactor operators), or a facility's equivalent form. It is only necessary to use those competency grading worksheets that apply to the knowledge or ability deficiency being evaluated. Normally, to determine l NUREG-1021 8 of 20 Interim Rev. 8, January 1997

4 i ES-604 1 j if'an individual passes or fails, the examiner uses the weighted average ]

of all the rating factors on the fom. Here, only those competencies 1 l that deal with the operator's perfomance deficiencies should be filled 1 i out, and should not-be weighted. If the NRC examiner gives the operator a rating factor score of "1" in either of the following cases, the individual fails this portion of the examination:

any two rating factors in any one competency l any one rating factor in any one competency if, in-the judgement 1

of the examination team, the operator's performance deficiency I j jeopardizes the safety of the plant or has significant safety

impact on the public (NRC management will make the final decision

on all operator failures resulting from a single rating factor evaluation of "1.")

When conducting the evaluation described herein, rating factor scores of j "1" will not be assigned based solely on performance in the dynamic

simulator. Follow-up questions are asked and the operator's responses i are recorded to evaluate and document the knowledge or ability j deficiency linked to the performance of a critical task.

l The NRC examiner will then apply the individual's responses to the

questions asked to evaluate and justify individual performance i deficiencies that warrant a rating factor score of "1." The examiners

! will document and include with the completed Individual Competency i Grading _Worksheets (Form ES-303-3 or -4) the follow-up questions asked 3 and the responses given by the operator. Written comments describing , the operator's performance and the as-run simulator scenario set are to l be included with the results of the operator's simulator examination. 1 j The NRC examiner will document the pass or fail determination for each ! operator's individual follow-up on Form ES-601-5, " Individual i Requalification Examination Report," in the " Simulator Examination

Results" section under " Individual Follow-up."

i , If an operator demonstrates no performance deficiencies and therefore l does not require any additional follow-up questioning, regardless of i whether the crew passes or fails the dynamic simulator examination, the

NRC examiner will record an "N/A" for " Individual Follow-up" in the

! " Simulator Examination Results" section of Form ES-601-5. ! F. ATTACHMENTS / FORMS ! Form ES-604-1, " Simulator Scenario Review Checklist" i Form ES-604-2, " Simulator Crew Evaluation" l I i 3 NUREG-1021 9 of 20 Interim Rev. P January 1997  ! i~

                        ,,,              _-                _.    ,              m.            -- - , - - _..

ES-604 Simulator Scenario Review Checklist Form ES-604-1 Note: Attach a separate copy of this form to each scenario reviewed. This form is used as guidance for the examination team as they conduct their review for the proposed scenarios. SCENARIO IDENTIFIER: REVIEWER: Qualitative Attributes

1. The scenario summary clearly states the objectives of the i scenario.

2. The initial conditions are realistic, in that some equipment and/or instrumentation may be out of service, but it does not cue  :

the crew into expected events. '

3. The scenario consists mostly of related events.

4. Each event description consists of the point in the scenario when it is to be initiated I the malfunction (s) that are entered to initiate the event the symptoms / cues that will be visible to the crew the expected operator actions (by shift position) the event termination point

5. No more than one non-mechanistic failure (e.g., pipe break) is incorporated into the scenario without a credible preceding incident such as a seismic event.

6. The events are valid with regard to physics and thermodynamics.

7. Sequencing / timing of events is reasonable, and allows for the examination team to obtain complete evaluation results commensurate with the scenario objectives.

8. If time compression techniques are used, scenario summary clearly so indicates. Operators have sufficient time to carry out expected activities without undue time constraints. Cues are '

given.

9. The simulator modeling is not altered.

10. All crew competencies can be evaluated.

11. The scenario has been validated.

12. If the sampling plan indicates that the scenario was used for training during the requalification cycle, evaluate the need to modify or replace the scenario. )

NUREG-1021 10 of 20 Interim Rev. 8, January 1997 i

1 l ES-604 2 Form ES-604-1 SIMULATOR SCENARIO REVIEW CHECKLIST (CONTINUED) ! Note: The following criteria list scenario traits that are numerical in nature. A second set of numbers indicates a range to be met for a set of two scenarios. Therefore, to complete this part of the review, the set of scenarios must be available. This page should be completed once per scenario set.

SCENARIO SET CONSISTS OF SCENARIO AND SCENARIO Quantitative Attributes

13. total malfunctions inserted: 4 to 8/10 to 14

14. malfunctions that occur after E0P entry: 1 to 4/3 to 6 l'

15. abnormal events: 1 to 2/2 to 3

16. major transients: 1 to 2/2 to 3

17. E0Ps used beyond primary scram response E0P: 1 to 3/3 to 5

18. E0P contingency procedures used: 0 to 3/1 to 3

19. approximate scenario run time: 45 to 60 minutes (one scenario may approach 90 minutes)

20. E0P run time: 40 to 70 percent of scenario run time

21. crew critical tasks: 2 to 5/5 to 8

22. technical specifications are exercised during the test ]

l I COMENTS:  ! l O h NUREG-1021 11 of 20 Interim Rev. 8, January 1997

ES-604 Simulator Crew Evaluation Form Form ES-604-2 The examination team should use this evaluation form during the dynamic simulator component of the requalification examination. The rating scales on this form are for evaluating the crew as a whole rather than the individual operators. Use the following instructions when rating team performance on the simulator examination: I

1. Review the rating scales before the simulator examination so that you are familiar with each competency to be evaluated.

2. Use the " Operator Actions" Form (ES-D-2), or an equivalent facility form to make notes during the examination, as described in Appendix D and ES-302.

3. Complete this form immediately after the simulator examination.

Evaluate the crew's performance on each applicable rating factor by com)aring the actions of the crew against the associated behavioral anc1 ors and selecting the appropriate grade. The tasks planned and performed during the crew's scenario set may not permit you to evaluate every rating factor for every crew. Annotate those rating factors that are not used in the evaluation. The examination team should pay particular attention to the completion of tasks that they identified as critical to plant safety. The crew may compensate for actions performed incorrectly by individual operators, as long as the critical task was completed satisfactorily. Other less significant deficiencies should also be accounted for in the rating factor evaluations to provide a source of information for crew remedial training during subsequent requalification training.

4. Justify all rating factor grades of "1" and document each justification in the space for " Comments" on the form. Rating factor grades of "1"

, must be linked to the performance of at least one critical task.

5. Complete the examination summary sheet, recording for each scenario, the scenario name (or identifier), and the critical tasks performed by the crew. Annotate whether the critical task was performed satisfactory or unsatisfactory. Complete the crew's overall evaluation using the criteria listed in the next paragraph. Space is provided for additional comments about the crew's performance.

6. The threshold for failing the simulator portion of the examination is to receive a (behavioral anchor) score of "1" in either of the following:

a. any two rating factors in any one competency

b. any one rating factor in any one competency if, in the judgement of the examination team, the crew's performance deficiency jeopardizes the safety of the plant or has significant safety impact on the public (NRC management will make the final decision on all crew failures resulting from a single rating factor evaluation of "1.")

NUREG-1021 12 of 20 Interim Rev. 8, January 1997

ES-604 2 Form ES-604-2 i SIMULATOR EXAMINATION

SUMMARY

SHEET 4 Facility: Examination Date: OVERALL DYNAMIC SIMULATOR CREW EVALUATION: SAT or UNSAT r Crew Members Docket No. Scenario #1 Scenario #2 Position Position

1. 55-4 2. 55-

3. 55-

4. 55-

5. 55-

6. 55-4 I

Scenario #1: [ Enter scenario descriptor] Crew Critical Tasks SAT UNSAT 4

1. [ Enter critical task descriptor]  !

2~ y 3. j 4. 5. i 4

Scenario #2: j

Crew Critical Tasks SAT UNSAT j

. 1. j 2. 1 3. 4. > 5. Comments: 4

,   NUREG-1021                              13 of 20        Interim Rev. 8, January 1997 s

l ES-604 3 Form ES-604-2 DIAGNOSIS OF EVENTS AND CONDITIONS BASED ON SIGNALS OR READINGS l Did the crew-- (a) recognize off-normal trends and status? 3 2 1 Recognized status and Recognized the status Did not recognize trends quickly and and trends at the time adverse status and accurately. of, but not before, trends, even after exceeding established alarms and limits. annunciators sounded. (b) use information and reference material (prints, books, charts, emergency plan implementation procedures) to aid in diagnosing and classifying events and conditions? 3 2 1 Made accurate Committed minor errors Failed to use, or diagnosis by using in using or misused, or information and interpreting misinterpreterj reference material information and information or correctly and in a reference material. reference material timely manner. that resulted in improper diagnosis. (c) correctly diagnose plant conditions based on control room indications? 3 2 1 Perfonned timely and Committed minor errors Made incorrect accurate diagnosis. or had minor diagnosis, which difficulties in making resulted in incorrect diagnosis. manipulation of any safety control. .c Grade for diagnosis of events and conditions based on signals and readings: SAT or UNSAT Comments: l NUREG-1021 14 of 20 Interim Rev. 8, January 1997

ES-604 4 Form ES-604-2 4 UNDERSTANDING OF PLANT AND SYSTEM RESPONSES Did the crew-- (a) locate and interpret control room indicators correctly and efficiently to j ascertain and verify the status / operation of plant systems?  ! 3 2 1 Each crew member Some crew members The crew members made located and committed minor errors serious omissions, interpreted in locating or delays, or errors in instruments accurately interpreting interpreting safety and efficiently. instruments or related parameters. i displays. Some crew 1 , members required i

assistance.

4 l (b) demonstrate an understanding of the manner in which the plant, systems, 3 and components operate, including setpoints, interlocks, and automatic ! actions? I 3 2 1 l Crew members The crew committed Inadequate knowledge ! demonstrated thorough minor errors.because of safety system or understanding of how of incomplete component operation systems and components knowledge of the resulted in serious i operate. operation of the mistakes or in plant

j. system or component's degradation.

l operation. Some crew i members required j assistance. i

(c) demonstrate an understanding of how their actions (or inaction) affected i systems and plant conditions?

i 3 2 1 All members understood zActions or. directives The crew appeared to i the effect that indicated minor act without knowledge j actions or directives inaccuracies in of or with disregard i had on the plant and understanding by for the effects on { systems. individuals, but the plant safety. ! crew corrected the ! actions. l Grade on understanding of the response of plant and systems; SAT or UNSAT Comments: i NUREG-1021 15 of 20 Interim Rev. 8, January 1997 4 l -. - l

d ES-604 5 Form ES-604-2 ADHERENCE TO AND USE OF PROCEDURES Did the crew-- (a) refer to the appropriate procedures in a timely m nner? l 3 2 1 The crew used The crew committed The crew failed to procedures as required minor failures to correctly refer to and knew what refer to procedures procedure (s) when conditions were without prompting, required, resulting in covered by procedures which affected the faulty safety system and where to find plant's status. operation. them. (b) correctly implement procedures, including following procedural steps in correct sequence, abiding by cautions and limitations, selecting correct paths on decision blocks, and transitioning between procedt:res when required? 3 2 1 The crew followed the The crew misapplied The crew failed to procedural steps procedures in minor follow procedures accurately and in a instances but made correctly, which timely manner, corrections in impeded recovery from demonstrating a sufficiant time to events or caused thorough understanding avoid adverse effects. unnecessary of the procedural degradation in the purposes and bases. safety of 1.ha plant. (c) recognize E0P entry conditions and perform appropriate actMs ;ithout the aid of references or other forms of assistance? 3 2 1 The crew recognized The crew had minor The crew failed to plant conditions and lapses or errors, accurately recognize implemented E0Ps Individual crew degraded plant consistently, members needed condition (s) or accurately, and in a assistance from others execute efficient timely manner. to implement mitigating action (s), procedures. even with the use of aids. Grade on adherence to and use of procedures: SAT or UNSAT Comments: O NUREG-10'l l 16 of 20 Interim Rev. 8, January 1997 l

)

l 1 j [E;-{C4 S F = ES E04 3 l lV i CONTROL BOARD OPERATIONS Did the crew-- ! (a) locate controls efficiently and accurately? i j 3 2 1 1 j Individual operators One or more operators The crew failed to j located controls ar.d hesitated or had locate control (s), indicators without difficulty in locating which jeopardized i hesitation. controls. system (s) important to j safety. 1 ! (b) manipulate controls in an accurate and timely manner? 3 2 1 t' i The crew manipulated The crew demonstrated The crew made mistakes l

plant controls minor shortcomings in manipulating j

smoothly and manipulating controls, control (s) that caused i i maintained parameters but recovered from safety system  !

5-within specified errors without causing transients and related i bounds. problems. problems. I 1

(c) take manual control of automatic functions, when appropriate?

3 2 1 i l i All operators took Some operators delayed Tlie crew failed to i control and smoothly or required prompting manually cont:ol l operated automatic before overriding or automatic systems j systems manually, operating automatic important to safety, 1 without assistance, functions, but avoided even when ample time i thereby averting plant transients where and indications

adverse events. possible. existed.

j Grade on control board operations: SAT or UNSAT i Comments: ? i

                         ~

l J j i i I NUREG-1021 17 of 20 Interim Rev. 8, January 1997 4 1

i l ES-604 7 Form ES-604-2 CREW OPERATIONS l Did the crew members-- (a) maintain a command role? 3 2 1 The crew took early In minor instances, The crew failed to remedial action when the crew failed to take timely action, necessary. take action within a which resulted in the reasonable period of deterioration of plant time. conditions. (b) provide timely, well planned directions to each other that facilitated their performance and demonstrated appropriate concern for the safety of the plant, staff, and public? 3 2 1 Supervisor's The supervisors, in The supervisor's directives allowed for minor instances, gave directive (s) inhibited safe and integrated orders that were safe crew performance. performance by all incorrect, trivial, or Crew members had to crew members. difficult to explain why order (s) implement. could not or should not be followed. (c) maintain control during the scenario with an appropriate amount of direction and guidance from the crew's supervisors? 3 2 1 Crew members stayed Crew members had to Crew members had to involved without solicit assistance repeatedly request creating a from supervisors or guidance. The crew distraction, the crew each other, failed to verify members anticipated interfering with their successful each other's needs, ability to carry out accomplishment of and the supervisors critical action (s). orders. provided guidance when necessary. CREW OPERATIONS CONTINUED ON NEXT PAGE NUREG-1021 18 of 20 Interim Rev. 8, January 1997

. , I ( ) () ES-604 8 CREW OPERATIONS Form ES-604-2 3 (Continued) i Did the crew members-- j (d) use a team approach to problem solving and decision making by soliciting . and incorporating relevant information from all crew member? 3 2 1

Crew members were At times, crew members The crew was not

involved in the failed to get involved involved in making

problem solving in the decision making decision (s). The crew process and the process when they was divided over the decision making should have, scenario's progress process for effective detracting from the and this behavior was team decision making. team oriented counter-productive.

approach. 4

Grade on crew operations

SAT or UNSAT  ;

l , Comments: i i s. , l l l 1 l l 3 d i f I .i O NUREG-1021 19 of 20 Interim Rev. 8, January 1997

ES-604 9 Form ES-604-2 COP 910NICATIONS Did the crew-- (a) exchange complete and relevant information in a clear, accurate and attentive manner? 3 2 1 Crew members provided Crew communications Crew meriebers did not relevant and accurate were generally inform each other of information to each complete and accurate, abnormal indication (s) other. but sometimes needed or action (s). Crew prompting, or the crew members were failed to acknowledge inattentive when the completion of important information evolutions, or to was requested. respond to information from others. (b) keep key personnel outside the control room informed of plant status? 3 2 1 Crew members provided Minor instances of Failed to provide key personnel outside needing to be prompted needed information. the control room with for information; some accurate, relevant incomplete / inaccurate information throughout d. formation provided. the scenarios. (c) ensure receipt of clear, easily understood communications from the crew and others? 3 2 1 Requests information/ Minor instances of Failed to request clarification when failing to require or needed information, or necessary; understands acknowledge inattentive when communications from information from information is others. others. provided; serious misunderstandings among crew members. Grade on communications: SAT or UNSAT Comments: NUREG-1021 20 of 20 Interim Rev. 8, January 1997

i 1 ' s I ES-605 !k License Naintenance, License Renewal Applications, and Requests for Administrative Reviews and Hearings i A. PURPOSE a l The NRC staff should use the procedures herein for maintaining licenses, the

processing license renewal applications, and handling requests for -

. administrative reviews and hearings by licensed operators in connection with

!      failures of NRC-conducted requalification examinations, and denials of

] applications for license renewal. ' , ! B. BACKGROUND i . 1 The renewal license application differs in some respects- from the initial i license application. To address these differences, the staff developed this j standard to establish the procedures for processing operators' renewal i applications and requests for administrative reviews and hearings regarding l the denial of renewal applications. i C. PROCEDURE FOR RESPONDING TO THE RESULTS OF NRC-CONDUCTED REQUALIFICATION i EXAMINATIONS i j 1. Passina an NRC-Conducted Reaualification Examination If an operator passes all portions of the requalification examination,

including being a member of a crew that passes the dynamic simulator i i examination, the regional office will so inform the operator using Attachment '

1, "Requalification Examination Pass Letter."

;      2.            Failina an NRC-Conducted Reaualification Examination i                                                                                                                                  ,

a. If an operator fails any part of an NRC-conducted requalification '

! examination, the facility licensee is expected to remove the

operator from licensed duty and take corrective action consistent

with the provisions of its requalification program before returning the operator to licensed duty. If the facility ,

t licensee's requalification program is unsatisfactory, refer to

Section.F.2 of ES-601 for a list of other recommended actions to

be taken, including those actions the facility licensee is

expected to complete before attaining a " provisionally

satisfactory" requalification program status.

! b. The regional office will inform the operator of the results of the l requalification examination using Attachment 2, "Requalification

Examination Failure Letter," or Attachment 3, "Requalification j Examination Second Failure Letter," as appropriate. On receiving j the failure notification, the operator has 20 days in which to 9 request an informal review of the portion of the examination that

!                                    was failed. The operator may submit the request for 1

4 j NUREG-1021 1 of 14 Interim Rev. 8, January 1997 i l1 w- . , -

                                                                                                             .,--m     - . - - %

ES-605 reconsideration to the Director, Division of Reactor Controls and Human Factors (DRCH), U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. Note 6 provides additional information on the requirements for supporting documentation.

c. Although the regulation (10 CFR 55.57(b)(2)(iv)) that required operators to pass an NRC-administered requalification examination as a prerequisite for license renewal was deleted effective March 11, 1994, the license of any operator who failed to pass any NRC-conducted requalification examination will not be renewed without some level of NRC involvement in the retesting process.

The amount of NRC involvement may include conducting the retest in accordance with the applicable Examination Standard (s); inspecting the facility licensee in accordance with Inspection Procedure (IP) 71001, " Licensed Operator Requalification Program Evaluation," as it retests the operator; or reviewing an examination prepared by the facility licensee. The regional office, in consultation with the Operator Licensing Branch (OLB), will determine the appropriate level of involvement on a case-by-case basis depending on the quality of the facility licensee's program. As long as the operator submits a timely renewal application, the term of the license will continue until the renewal requirements are satisfied or the operator fails three NRC-conducted examinations as discussed in Section C.2.e.

d. The NRC will nonnally administer a second (first retake) examination approximately six months after issuing the first failure notification in accordance with Section C.2.b of this standard, and will concentrate on the areas in which the operator exhibited deficiencies.

e. The NRC will nonaally administer a third (second retake) examination approximately six months after issuing the second failure notification in accordance with Section C.2.b of this standard. The third examination will be a comprehensive requalification examination.

Regardless of the status of the facility licensee's requalification program, if an operator fails a third requalification examination, the NRC will thoroughly review the operator's examination performance and may conduct a complete review of the facility licensee's training program. The third failure may be' grounds for suspending or revoking the operator's license. If an operator has an application pending for license renewal with the NRC at the time of a third requalification failure, that failure will provide the basis for denying the application. Notification of the operator will be handled on a , case-by-case basis and coordinated through OLB. l NUREG-1021 2 of 14 Interim Rev. 8, January 1997 Ol l I

1 t t l 4 ., l ES-605 j 4 i D. LICENSE MAINTENANCE  ! l i 1. Medical Standards

a. If an operator is temporarfly unable to meet medical standards but l is expected to meet those standards again in the future, the

facility licensec may administrative 1y classify that operator's i license as " inactive" until the operator is once again certified

? to meet all medical standards by the facility licensee. The i facility licensee need not notify the NRC nor request a

conditional license for the temporary disability provided the

operator is administrative 1y prevented from performing licensed i duties during the period of his or her temporary disability. If i the disability extends beyond the date of license expiration, the i operator may apply for timely license renewal. in accordance with 10 CFR 55.55(a) and 10 CFR 55.57(a). The facility licensee should j document the nature of the operator's temporary disability on the

medical certificate and submit a revised certificate to the NRC after the physician determines that the operator meets the i requirements of 10 CFR 55.33(a)(1). The NRC will not renew the i operator's license until it finds that all of the conditions <

! specified in 10 CFR 55.57(b) are satisfied. l 1 l i b. If the facility licensee determines that an operator's medical  ! j condition is permanently disqualifying in acco. '.ance with the most '

recent edition of ANSI /ANS 3.4, " Medical Certification and j Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants," endorsed by the NRC, the facility licensee shall l notify the NRC within 30 days of learning of the diagnosis (see j 10 CFR 50.74 and 55.25).

4 l 2. Participation in Reaualification Trainina

In accordance with 10 CFR 55.53(h), licensed operators are required to j complete a requalification program as described by 10 CFR 55.59.

j However, upon written request by the facGity licensee, the region may 1 authorize an operator to temporarily suspend participation in the j facility licensee's requalification training program if it finds that [ a. the operator will be reassigned to full-time, career-enhancing duties at another location, making it impractical to participate { in the training program (e.g., assignment to the Institute of Nuclear Power Operations or a foreign interchange program; college attendance) i b. the duration of the assignment will not exceed 24 months or the j remaining ters of the operator's license, whichever is s L rter s NUREG-1021 3 of 14 Interim Rev. 8, January 1997 i 4 4 5 w - - ,-- , , - ,. - -

c. the facility licensee's plan for ensuring the operator's qualifications and status is acceptable (i.e., the operator must be retrained, tested, reactivated, and medically fit for duty)

The region will amend the operator's license to prohibit the performance of licensed duties during the reassignment. The region will also confirm in a letter to the facility licensee (with a copy to the operator), its expectations regarding the operator's return to licensed duties and the need to certify when the actions have been completed. The region shall refer situations outside the specified parameters to OLB for evaluation. E. PROCESSING LICENSE RENEWAL APPLICATIONS If the operator wishes to renew the license, he or she must comply with the requirements of 10 CFR 55.57(a), as follows: The operator must submit NRC Forms 396 and 398 not less than 30 days before the date of expiration of the license. In accordance with 10 CFR 55.55(b), if the operator files a proper application for renewal at lea:,t 30 days before the date of expiration, the license shall not expire until the application for renewal has been denied or a new license has been issued. If an operator is waiting to be given a reexaminatior, after failing an NRC-administered requalification examination, the operator should still make timely application for , license renewal under the provisions of 10 CFR 55.55(b). The regional office may allow for transit time and accept a license , renewal application received 25 days before the license expiration date, provided all signatures on NRC Forms 398 and 396 are dated before the , 30-day timely renewal cutoff date. The submittal will not be considered timely if it is received less than 25 days before the date of license expiration unless positive evidence of receipt (e.g., postmark or docketing stamp) by the U.S. Postal Service or the NRC is available. If the application is received less than 25 days before the date of license expiration and too late for processing in the regional office, the license shall expire on the expiration date. The regional office may issue a new license when it has completed processing the application. The operator will complete NRC Form 398, including the operator's experience under the current license, the approximate number of hours the operator spent on operating shifts, and the date and results of the applicant's most recent NRC-administered requalification examination (if taken). The senior management representative on site shall provide evidence that the operator has safely and competently discharged his or her license responsibilities and satisfactorily completed the facility's approved requalification program by signing Item 19.c on Form 398. NUREG-1021 4 of 14 Interim Rev. 8, January 1997

I i ES-605 l The facility licensee must certify on NRC Form 396 that a physician has performed a medical examination within the previous two years as [ required by 10 CFR 55.21 and submit that form along with NRC Form 398. The regional office will review the application and issue the license renewal ^ if it finds that the conditions in 10 CFR 55.57(b) are satisfied. There is no minimum number of hours that the operator has to operate the facility in order to qualify for license renewal (i.e., inactive licenses are also renewable). l However, the regional office should take the applicant's operating history 4 into consideration as an additional piece of information if any of the ! requirements of 10 CFR 55.57(b) are not met. j If the regional office determines that the renewal applicant does not meet the i requirements of 10 CFR 55.57, it will infom the facility licensee of the j deficiencies and request that supplemental information required to make a , relicensing decision be forwarded by the operator to the regional office

within 20 days. If, after evaluating the supplemental information, the i regional office still concludes that the applicant does not meet the

! requirements for license renewal, it will issue a proposed denial letter to

the operator following the procedure in Section F of this standard. i i

F. PROCEDURE FOR DENYING AN APPLICATION FOR LICENSE RENEWAL i If the operator declines to supply additional information or the regional office concludes, after reviewing any additional information supplied by the operator, that the application is still inadequate for license renewal, the ! regional office will deny the renewal application using Attachment 4, " Renewal i Application Denial Letter (Region)." t ~ i Within 20 days of the date of the letter of notification of proposed denial, i the operator may exercise one of the following options: i l 1. Submit a written request for the NRC to review the application. Such a request should be sent to the Director, Division of Reactor Controls and l Human Factors, U.S. Nuclear Regulatory Commission, Washington, D.C.

20555, and should include the reasons for the review request and j supporting documentation as applicable. Note 6 provides additional information on the requirements for supporting documentation.

! 2. Submit a written request for a hearing pursuant to 10 CFR 2.103(b)(2). A hearing request is required to be submitted to the Secretary of the ! Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, , with a copy to the Assistant General Counsel for Hearings and j Enforcement, Office of the General Counsel, at the same address. If the operator exercises option (1) and the staff sustains its denial of the application, the Director, DRCH, will so inform the operator using Attachment 5, " Renewal Application Denial Letter (DRCH)." The appilcant has i the option at this time to request a hearing pursuant to 10 CFR 2.103(b)(2). l NUREG-1021 5 of 14 Interim Rev. 8, January 1997 i 1

                 = . _ ,     _ . - . .

i l l l ES-605 G. PROCEDURE FOR OVERTURNING REQUALIFICATION EXAMINATIONS OR RENEWAL DENIALS , If, upon conducting a hearing or an informal review, the staff reverses its decision regarding the failure of a requalification examination or application denial, DRCH will take one or more of the following actions, as appropriate:

1. reinstate the license l

2. allow the operator to renew the license pursuant to 10 CFR 55.57, if all other requirements are satisfied

3. allow the operator to perform licensed duties when he or she has l successfully completed the facility's requalification program and has  !

met the provisions of 10 CFR 55.53(e) or (f) l If, upon conducting a hearing or an informal review, the staff reverses its l decision regarding the failure of the requalification examination, DRCH will  :' inform the operator that he or she has passed the examination using , " Sample Examination Notification (DRCH)." If, upon conducting a hearing or an informal review, the staff reverses its denial of an operator's renewal application and agrees that the application satisfies the license renewal requirements of 10 CFR 55.57 (assuming all other requirements which were not at issue in the hearing were satisfied), DRCH will inform the operator using Attachment 7, " Sample Renewal Notification (DRCH)." H. NOTES

1. Letters informing an operator of a proposed denial or examination failure must be signed by a branch chief or higher. In the event of an appeal, a copy of DRCH's correspondence will be distributed to the appropriate regional office for tracking purposes.

2. A copy of the correspondence sent to the operator as a result of this process will ba provided the facility licensee's representative authorized to sign the renewal application.

3. The Director, DRCH, will keep regional office management informed of the status of the actions he or she takes during this process and will acknowledge correspondence from the operator.

4. All examination failures or denial correspondence sent to the operator should be sent by certified mail with return receipt requested.

5. It is inappropriate to ask the facility licensee to reassess the need for an operator's license while conducting an informal review or hearing.

NUREG-1021 6 of 14 Interim Rev. 8, January 1997

i ! ES-605

6. Requests for informal reviews by the NRC shall (a) list the items for li which additional review is being requested and (b) include documentation i supporting the contentions made by the operator. The package containing l l

the supporting documentation for the review and the review request must l be mailed or delivered to the Director's office (DRCH) within 20 days of the date of the failure or denial notification. The DRCH staff should 'j complete the review within 45 days of receiving the package. The staff will review requests using the guidance in ES-502. I. ATTACHMENTS / FORMS i Attachment 1, "Requalification Examination Pass Letter" l Attachment 2, "Requalification Examination Failure Letter" , Attachment 3, "Requalification Examination Second Failure Letter" Attachment 4, " Renewal Application Denial Letter (Region)" Attachment 5, " Renewal Application Denial Letter (DRCH)"

Attachment 6, " Sample Examination Notification (DRCH)"

Attachment 7, " Sample Renewal Notification (DRCH)" O i

ES-605 Requalification Examination Attachment 1 Pass Letter NRC Letterhead 10 ate) (Operator's name) (Street address) (City. State Zip code) i

Dear (Name):

I am writing to inform you that you passed the requalification written examination and operating test administered by the U.S. Nuclear Regulatory Commission (NRC) on (date). Enclosed is a copy of your Individual Requalification Examination Report (Form ES-601-5) summarizing the results of your examination. Your facility training department has a copy of the master answer key. If you have a question, please contact (Name) at (number). Sincerely, 9 (Reaional Branch Chief or abo M Docket No. 55-(Number) cc: (Facility-authorized representative who sions NRC Form 398) NUREG-1021 8 of 14 Interim Rev. 8, January 1997

)

ES-605 Requalification Examination Attachment 2 j Failure Letter

, NRC Letterhead 1 (Date) i 1 10oerator's name) (Street address) (City. State Zio code)

Dear (Name):

4 I as writing to inform you that you did not achieve an acceptable score on the requalification (written examination and/or operatina teitl administered by the U.S. Nuclear Regulatory Commission (NRC) on (date). Enclosed is a copy of the results indicating the area (s) in which you exhibited deficiencies. Your i facility training department has a copy of the master answer key. j This failure places you in the same status as if you had failed a facility-administered requalification examination. Therefore, you are subject to the requirements set forth in the NRC-approved requalification program for the facility for which you are licensed and must meet those requirements before resuming licensed duties. The NRC will administer a second requalification examination to you in the areas in which you exhibited deficiencies. If you believe an error was made in grading your examination, you may request that the NRC informally regrade the examination. Requests for informal regrade should be sent within 20 days of the date of this letter to the Director, Division of Reactor Controls and Human Factors, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. In the request, please state  ; the items you wish to have reviewed, and provide supporting documentation as i applicable. l If you have any questions, please contact (name) at (number). Sincerely, (Reaional Branch Chief or above) Docket No. 55-fNumber) , cc: (Facility-authorized representative who sians NRC Form 398) CERTIFIED MAIL, RETURN RECEIPT REQUESTED s NUREG-1021 9 of 14 Interim Rev. 8, January 1997

ES-605 Requalification Examination Attachment 3 Second Failure Letter NRC Letterhead (Date) (Ocerator's name) (Street address) (City. State Zio code)

Dear (Name):

I am writing to inform you that you did not achieve an acceptable score on the requalification (written examination and/or operatino test) administered by the U.S. Nuclear Regulatory Commission (NRC) on (date). Enclosed is a copy of the results indicating the area (s) in which you exhibited deficiencies. Your facility training department has a copy of the master answer key. This failure places you in the same status as if you had failed a facility-administered requalification examination. Therefore, you are subject to the requirements set forth in the NRC-approved requalification program for the facility for which you are licensed and you must meet those requirements before resuming licensed duties. The NRC will administer a third requalification examination to you that will be comprehensive in scope. If you believe an error was made in grading your examination, you may request that the NRC informally regrade the examination. Requests for informal regrade should be sent within 20 days of the date of this letter to the Director, Division of Reactor Controls and Human Factors, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555. In the request, please state the items you wish to have reviewed, and provide supporting documentation as applicable. If you have any questions, please contact (name) at (number). Sincerely, (Reaional Branch Chief or above) Docket No. 55-(Number) cc: (Facility-authorized representative who sions NRC Form 398) CERTIFIED MAIL, RETURN RECEIPT REQUESTED NUREG-1021 10 of 14 Interim Rev. 8, January 1997

i i 1 O ES-605 Renewal Application Denial Attachment 4 Letter (Region) ! NRC Letterhead (Date) ! (Apolicant's name) 1 (Street address) (City. State Zio code)

Dear (Name):

I as writing to inform you that your renewal application for a (senior. reactor) operator license submitted in connection with the (facility name) does not demonstrate your eligibility for license renewal for the following reason (s): (Reaional office to discuss deficiencies and aonlicable cart (s) of the Examination Standards and 10 CFR Part 55 that acoly.) If you do not accept this proposed denial, you may exerc'se one of the following options within 20 days of the date of this letter.

1. Request reconsideration of the application denial. Submit your written '

request to the Director, Division of Reactor Controls and Human Factors, O U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, stating the reasons for the request and providing supporting documentation as applicable. If the proposed denial is upheld, you may then request a hearing pursuant to 10 CFR 2.103(b)(2).

2. Request a hearing pursuant to 10 CFR 2.103(b)(2). To make this request, ,

you must submit a written request for a hearing to the Secretary of the Commission, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General Counsel, at the same address. If you have any questions, please contact (name) at (number). Sincerely, (Reaional Branch Chief or above) Docket No. 55-(Number) cc: (Facility-authorized representative who sians NRC Form 398) CERTIFIED MAIL, RETURN RECEIPT REQUESTED NUREG-1021 11 of 14 Interim Rev. 8, January 1997

ES-605 Renewal Application Denial Attachment 5 Letter (DRCH) NRC Letterhead (Date) IAnolicant's name) (Street address) (City. State Zio code)

Dear (Name):

I am writing to inform you that the U.S. Nuclear Regulatory Commission (NRC) has reviewed the proposed denial of your renewal application for a (senior. r_nctor) operator license. Upon reviewing the information that has been submitted, the NRC has denied your renewal application. (DRCH to discuss deficiencies and aoolicable cart (s) of the Examination Standards. and 10 CFR Part 55 that acoly.) When you have met these requirements, you may submit another application to the appropriate regional office. If you do not accept this denial, you may, within 20 days of the date of this letter, request a hearing pursuant to 10 CFR 2.103(b)(2). To do so, you must submit a written request to the Secretary of the Commission, U.S. Nuclear ' Regulatory Commission, Washington, D.C. 20555, with a copy to the Assistant  ! General Counsel for Hearings and Enforcement, Office of the General Counsel at the same address. If you have any questions, please contact (name) at (number).' Sincerely, (Name.) Director, Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation Docket No. 55-(number) cc: (Facility-authorized representative who sions NRC Form 398) CERTIFIED MAIL, RETURN RECEIPT REQUESTED NUREG-1021 12 of 14 Interim Rev. 8, January 1997

   - . - . . . . -           _ . . . - .        . - - ~ _          . - . - . -           _ -      . - . - - . - . . . - . . - . _ -

ES-605 Sample Lxamination Notification Attachment 6 (DRCH) NRC Letterhead (Date) (Operator's name) (Street address) (City. State. Zio code)

DearfM,

gggi: In response to your letter of (date), we have reviewed the grading of the requalification (written examination. operatina test) administered.to you on (date(s)) and reconsidered the requalification failure issued to you on (date). In light of the additional information you supplied, we have determined that you passed the (written examination. operatina test) and satisfy the requirements of 10 CFR 55.57(b)(2)(iv). [For your information, I am enclosing a copy of the staff's resolution of each of your (written examination. oneratina test) comments.] If you have any questions, please contact (name) at (telephone number). O Sincerely, IName.) Director, Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation Docket No. 55-b umber)

Enclosure:

As stated [] Include only if the regional office does not forward a discussion of the consent resolutions to the operator. NUCEG-1021 13 of 14 Interim Rev. 8, January 1997 o

ES-605 Sample Renewal Notification Attachment 7 (DRCH) NRC Letterhead (Date) ) (Apolicant's name) (Street address) (City. State. Zio code)

Dear (Name):

In response to your letter of (date), we have reviewed your application for  ! license renewal and reconsidered the renewal application denial issued to you on (date). In light of the additional information you supplied, we have determined that your application satisfies the license renewal requirements of 10 CFR 55.57(b). [For your information, I am enclosing a copy of the staff's resolution of each of your comments.] If you have any questions, please contact (name) at (teleohone number). Sincerely, (Name.) Director, Division of Reactor Controls i and Human Factors l Office of Nuclear Reactor Regulation l Docket No. 55-(number) j 1

Enclosure:

l . As stated [] Include only if the regional office does not forward discussion of coment resolutions to the operator. NUREG-1021 14 of 14 Interim Rev. 8, January 1997

l 4 ! l i ! ES-701

ADMINISTRATION OF INITIAL EXAMINATIONS FOR SENIOR OPERATORS LIMITED TO FUEL HANDLING

, A. PURPOSE i j This standard provides specific instructions for the preparation, i administration, grading, and documentation of initial examinations for senior

operators who are limited to fuel handling (LSR0s).

1 B. BACKGROUNO 1 ! Except as noted herein, the guidance in Examination Standards (ES) 201, 202,

204, 301, 302, 303, 401, 402, 403, 501, and 502 for administering unrestricted

! initial licensing examinations at power reactors is also applicable to the l LSR0 examination. The generic fundamentals examination (GFE) program l described in ES-205 does not apply to LSR0 applicants. i l C. RESPONSIBILITIES i 1. Facility Licensee ! The facility licensee is responsible for the same activities specified ! in the unrestricted ES with the following exceptions and modifications:

a. As an exception to ES-202, the facility licensee may request LSR0 4

licenses that are valid for more than one site. The facility i licensee shall provide documentation that describes the differences in the design, procedures, technical data, and

administrative controls of the separate facilities for which the license is being sought.

! b. The scope and content of the_ written examination and operating j test shall be as described in Sections D and E below. ! c. -The written examinations shall be administered in accordance with j the additional instructions in Section D.

2. NRC Reaional Office 4

The NRC regional office is responsible for the same activities specified i in the unrestricted ES with the following exceptions and modifications: i

a. The region should generally schedule the administration of LSR0

examinations to coincide with an initial or requalification examination trip. However, it may be necessary to deviate from

! the national examination schedule and conduct the LSR0

examinations during a time when the fuel handling equipment will j be available for the operating tests.

NUREG-1021 1 of 9 Interim Rev. 8, January 1997 4

  ,                                                                                 - . .       m, , _                     -cr_         ~ . , , - . -

ES-701

b. With the concurrence of the Chief, Operator Licensing Branch, the region may issue LSR0 licenses that are valid for units at more than one site provided the units are manufactured by the same vendor and are of similar design. The applicant must pass an examination that addresses the differences in the design, procedures, technical data, and administrative controls of the separate facilities for which the license is being sought.

c. The scope and content of the written examination and operating test shall be as described in Sections D and E below.

d. The regional office shall ensure that the examinations are administered and documented in accordance with the instructions in Sections D and E.

D. WRITTEN EXAMINATION INSTRUCTIONS

1. Content and Preparation LSR0 written examinations shall consist of one section containing 50 to 70 points and it should be constructed so that a competent applicant can complete the examination in 2.5 hours. The applicants will be allowed a maximum of three hours to complete and review the examination.

Develop an examination model or sample plan using the appropriate knowledge and abilities (K/A) catalog: NUREG-ll22 for pressurized water reactors (PWRs) or NUREG-1123 for boiling water reactors (BWRs). K/As from the following sections of the K/A catalog may be selected: i appropriate generic knowledge and abilities, plant systems associated with fuel handling operations, refueling equipment, and emergency and abnormal plant evolutions related to refueling. If the facility licensee has completed a job task analysis (JTA) for fuel handlers, that analysis would provide an excellent source af information for developing the test outline and test items. The examination points should be distributed among the following subject i areas according to the percentages (%) noted: Samole Sub.iect Area 15% Reactor and fuel characteristics and physical aspects of core construction that are important to fuel handling or shutdown activities. For example, this could include a question on how a task performed during fuel handling activities affects the shutdown margin but should not include a question on power defects. Since LSR0 applicants are not required to take the GFE, questions on basic reactor theory and thermodynamic topics that apply to fuel handling operations are also appropriate and may be selected from prior GFE examinations. NUREG-1021 2 of 9 Interim Rev. 8, January 1997

l  ; L ES-701 30% System equipment and instruments that are important to plant > 1 safety and are located near or used during fuel handling activities or during alternate shutdown procedures. This  :' includes the knowledge and use of radiation monitors, spent fuel pool cooling, and residual heat removal (RHR) systems and the technical specifications associated with those systems. 40% Normal, abnormal, emergency operating and administrative procedures related to fuel handling activities, core safety, and accident mitigation, including general facility events such as a loss of forced circulation or a release of radioactive effluent. The questions should evaluate the - applicant's knowledge of the control room operator's response to events only as it relates to fuel handling 1 activities and the general response expected of employees. 15% Health physics and radiation protection for fuel handling activities and general employee responsibilities. These questions may include administrative procedures associated with radiation protection. - The actual percentages may vary from those listed above by up to five percent (plus or minus). If the examination will be used to license the applicants at more than one facility, ensure that it adequately covers O all of the applicable unit .sAn examination developed for the purpose of cross-qualifying a licensed LSR0 at another similar facility may focus exclusively on the differences between the facilities. The LSR0 written examination should otherwise meet all the guidelines and requirements for test question construction, quality assurance, and facility reviews specified in ES-401 and Appendix B. Form ES-701-1 should be used as an examination cover sheet.

2. Administration The written examination for LSR0s should be administered in accordance with ES-402 with the following modifications:

a. The examination may be administered concurrently in the same room with full-scope, initial license examinations. The proctor should minimize any disturbance to those applicants taking the longer initial licensing examination.

b. The time limit for the applicant to complete the examination is three hours.

O V NUREG-1021 3 of 9 Interim Rev. 8, January 1997

_ .- - - _ _ . - ~ ES-701 E. OPERATING TEST INSTRUCTIONS The LSR0 operating test should generally be prepared, administered, and documented in accordance with ES-301, ES-302, and ES-303. However, the sampling requirements will be less than those required for an SR0 applicant. The operating test should be performance-based to the maximum extent possible. Therefore, the facility licensee should be encouraged to permit the actual use , of equipment to handle dummy fuel elements, assemblies, or modules during the operating test whenever feasible. This may require careful coordination with the facility licensee to establish a schedule and to make sure that a licensed SRO is available, if needed. When actual equipment is not available or inaccessible (e.g., because of high radiation), the examiner should administer the test using walk-through methods near the actual equipment or by using mockup equipment. If the facility licensee has a refueling machine simulator, the examiner should use it to the extent possible during the administration of Categories B and C of the operating test. The operating test shall assess the applicant's ability to execute normal, abnormal, and emergency procedures associated with fuel handling. Each epplicant will be required to simulate or perform tasks related to fuel handling and to answer questions associated with the refueling equipment and associated systems. The operating test will also determine if the applicant has the ability to supervise the operation of equipment and systems to safely conduct fuel haoiling operations. The applicant shall not be held accountable for duties tt at are performed exclusively by the control room staff or shift 4 supervisor. The operating test should normally ?ake between two and three hours, depending < on whether or not refueling equiprat is actually operated. The following additional guidelines clarify the expectations for Categories A, B, and C of the LSR0 operating test.

1. Administrative Tooics.(Cateoorv A1 Topic A.1, " Conduct of Operations" The subjects under this topic should be evaluated as they pertain to refueling activities at that facility. Some subjects (e.g., reactor plant startup requirements) may not be applicable, however, most can be adapted for use during the LSR0 operating test.

Topic A.2, "Equpment Control" These subjects all lend themselves to evaluating the required refueling maintrAance and surveillance actions that the LSR0 should be able to super'ise or perform. NUREG-1021 4 of 9 Interim Rev. 8, January 1997

     . ._._ _ _ _ _ . _ _ . _ _ _ _ _ _ . ~ _ _ . _ _ _ _ _ _ _ _ _ _ _ . . _

!J ! ES-701 ! Topic A.3, " Radiation Control"

These subjects are all' applicable to refueling operations and should be j evaluated on a sampling basis.

Topic A.4, " Emergency Plan"

This topic shall be evaluated to.the extent that the applicant is j required to respond to a declared event and the knowledge required of a j radiation worker.

2. Systems (Cateaory B)

Instead of distributing systems among the safety function groupings in 4 the K/A catalog as discussed in ES-301, this category of the LSR0 operating test shall . evaluate the applicant on 5 systems selected from

Attachment I as follows

. one from decay he t removal (DHR) or auxiliary (AUX), one from instrument and control (IC) or radiation monitoring j (RM), and three from fuel handling equipment (FHE). No distinction i between control room and facility systems is required, because most, if not all, of the systems will be covered outside the control room.

l

In accordance with ES-301, each system should be evaluated by simulating

or actually performing a JPM whenever possible. If a system does not

]- lend itself to the conduct of a JPM, then increase the number of K/As

evaleted with prescripted questions from two to four.

l The uaminer may conduct part of the operating test in the control room i so that those controls, instruments, and other materials or equipment related to fuel handling (e.g., procedures and diagrams) are available i for reference. Although LSR0s will not operate any systems from the i control room, they must be aware of the effects (e.g., alarms) that fuel handling operations will have in the control room. They must also be familiar with the methods and requirements for communicating with the

control room staff and shift supervisor.

i , When documenting this category of the operating test on Fonn ES-303-1, i disregard Category B.2 and enter the appropriate information under l control room systems. Also, in place of the " Safety Function," enter j the applicable system category (e.g., AUX, IC, etc.).

3. Intearated Plant (Refuelina Eauionent) Operation (Cateaory C) i j To adequately evaluate the applicant in this area, prepare two j " discussion" scenarios to evaluate the applicant's competency in much i the same way that it would be done using a simulation facility. Use Forms ES-D-1 and ES-D-2 to develop the scenario events and expected operator actions. Each scenario sSall, at a minimum, simulate an equipment or instrument malfunction requiring operator actions and an event requiring the applicant to follow emergency response procedures NUREG-1011 5 of 9 Interim Rev. 8, January 1997

ES-701 associated with fuel handling activities. The malfunctions and events shall not duplicate activities tested under Category B of the operating test. The scenarios should require the applicant to demonstrate adequate knowledge and ability in areas that are important to fuel handling safety, including an understanding of the associated technical specifications. When documenting the applicant's performance, the examiner will substitute page 3 of Form ES-303-1 with Form ES-701-2, "LSR0 Competency , Grading Sheet," and evaluate the applicant on as many of the rating l factors as possible. The examiner should evaluate the applicant on I rating factr~ 5.b only if actual fuel handling equipment manipulations were perfor . Those rating factors against which the applicant is not evaluated will have their weight equally distributed among those that i were used in evaluating that competency. F. ATTACHMENTS / FORMS Attachment 1, "LSR0 Operating Test Topics and Systems (BWR/PWR)" Form ES-701-1, "LSR0 Written Examination Cover Sheet" Form ES-701-2, "LSR0 Competency Grading Shent' O a

                                                                   . j

d ES-701 LSR0 Operating Test Topics Attachment 1 ] i

   '/                                     and Systems (BWR/PWR) i        1. Decay Heat Removal Systems residual heat removal shutdown cooling service water core cooling normal and emergency AC power 4                reactor coolant system J

2. Auxiliary Systems CVCS-makeup / letdown (in Mode 6)

CVCS-boration/ dilution (in Mode 6) RBCCW/CCW containment HVAC fuel pool / spent fuel pit cooling / cleanup local compressed air system local fire protection system reactor vessel internals temporary and special equipment (seals, dams, etc.)

3. Instrument and Controls Systems source range detection  ;

startup channels ' traveling in-core probe vessel / loop level indicating system

4. Radiation Monitoring containment and fuel handling process monitoring l containment and fuel handling area monitoring  !

temporary and special radiation monitoring (e.g., neutron sources)  ! 1

5. Fuel Handling Equipment new and spent fuel shipping, receiving, and storage fuol element / assembly / module design features to accommodate handling fuel sipping l fuel transfer system video mapping system spent fuel pool and fuel building cranes refueling platform and associated equipment fuel handling tools refueling machine manipulator crane Note: Some items apply to specific vendors k NUREG-1021 7 of 9 Interim Rev. 8, January 1997

ES-701 LSR0 Written Examination Form ES-701-1 Cover Sheet

U. S. NUCLEAR REGULATORY COMMISSION

LSRO WRITTEN EXAMINATION APPLICANT INFORMATION _

Name: Date: Region: I / II / III / IV Facility / Unit: Reactor Type: W / CE / BW / GE Start Time: Stop Time: l 5 INSTRUCTIONS

l Use the answer sheets provided to document your answers. Staple this cover I sheet on top of the answer sheets. The passing grade requires a final '

grade of at least 80.00 percent. Examination papers will be picked up i three hours after the examination starts. J APPLICAWT CERTIFICATION  ! , All work done on this examination is my own. I have neither given nor received aid. . Applicant's Signature 4 RESULTS Test Value Points 2 Applicant's Score Points Applicant's Grade Percent NUREG-1021 8 of 9 Interim Rev, 8, January 1997

(O v / ES-701 LSR0 Competency Grading Sheet Form ES-701-2 Applicant Docket Number 55- Page of C. LSR0 Integrated Plant Operations Grading Stannary Competencies / scenarios connent Rating Factors (1) Weight 3.0 2.0 1.0 Total Observed Page #

1. Alarus/ Annunciators 1 2

s. Prioritize 0.30 0.90 0.60 0.30 _ _ _

b. Interpret 0.35 1.05 0.70 0.35 _ _ ,_

c. Verify 0.35 1.05 0.70 0.35 ._,_ _, _

2. Diagnosis 1 2

a. Recognize 0.33 1.0 0.66 0.33 _ ,_,_ __

b. Accuracy 0.33 1.0 0.66 0.33 _,_, ,,_ _,

c. Diagnose 0.33 1.0 0.66 0.33 ,__ .,_

3. System Response 1 2

a. Interpret 0.35 1.05 0.70 0.35 _ _

b. Attentive 0.20 0.60 0.40 0.20 _ _,,,_

c. Plant Effects 0.45 1.35 0.90 0.45 _,,,_ ,_ .,_,,

4. Procedures 1 2

e. Reference 0.33 1.0 0.66 0.33 ,,,,_, _ _

b. Correct use 0.66 2.0 1.33 0.66 f _,,,_ _ _

              \
                  \                                  5. Control Board operations                                                                         1           2

a. Locate 0.25 0.75 0.50 0.25 __,,, _.,_ _

b. Manipulate (2) 0.25 0.75 0.50 0.25 ,,_ ,_

c. Response 0.25 0.75 0.50 0.25 _,, __ _

d. Manual Control 0.25 0.75 0.50 0.25 _ _

6. Comunications 1 2

a. Clarity 0.70 2.10 1.40 0.70 ._ ,,,, ,,,,,_

l

b. Receive Information 0.30 0.90 0.60 0.30 , _ _

j

7. Directing Operations 1 2 l a. Timely Action 0.25 0.75 0.50 0.25 _ __. ,,

b. Safe ofrections 0.50 1.50 1.00 0.50 ,,,,,_ __ ,,,,_

c. oversight 0.25 0.75 0.50 0.25 _ _

8. Technical specifications . 1 2

a. Recognize 0.40 1.20 0.80 0.40 _ ,,_ _,

b. Locate 0.20 0.60 0.40 0.20 _ _ _

c. Compliance 0.40 1.20 0.80 0.40 __,, __

(1) Evaluate the applicant on as many rating factors as possible. If a rating factor is not evaluated, redistribute its weight among the remaining factors Leder that competency. (2) only evaluate this rating factor if actual fuel handling equipment manipulations were perfrirmed. V NUREG-1021 9 of 9 Interim Rev. 8, January 1997 ' 1

l l O Q ES-702 i ADMINISTRATION OF REQUALIFICATION EXAMINATIONS I FOR SENIOR REACTOR OPERATORS LINITED TO FUEL HANDLING A. PURPOSE The NRC requalification examinations for senior reactor operators limited to fuel handling (LSR0s) are administered under this standard according to the provisions of Title 10, Section 55.59(a)(2)(iii), of the Code of Federal Regulatfons (10 CFR). B. BACKGROUND This standard, in conjunction with ES-601 through ES-603, provides general , guidance to facility licensees and requirements to NRC examiners for l preparing, administering, grading, and documenting NRC requalification examinations for LSR0s. Except as noted herein, the methodology and guidance in ES-601 through ES-603 for administering full-scope requalification examinations at power reactors is also applicable to LSR0 requalification examinations. C. GENERAL DIFFERENCES The LSR0 examinations will be conducted in accordance with the methodology outlined in ES-601, with the following exceptions. Specific exceptions related to each category of the examination are discussed in Section D. O Any questions regarding the program office's expectations regarding the conduct of LSRO requalification examinations shall be referred to the Chief, Operator Licensing Branch (OLB), for resolution.

1. The NRC will coordinate with the facility licensee to schedule the NRC's LSR0 examinations concurrent with the facility licensee's LSR0 requalification examination schedule. If practical, the examination team will conduct the LSR0 examination shortly before or after an outage to facilitate access to refueling equipment because some of the equipment is not accessible during plant operations.

The NRC may administer LSRO requalification examinations concurrent with full-scope initial license or operator requalification examinations.  ;

2. The facility licensee's LSR0 requalification program, LSR0 job task analysis, and associated learning objectives will provide the basis for the examination if they.are of sufficient scope and depth. The items in 10 CFR 55.43 and 55.45 will be sampled as appropriate to the LSR0's limited responsibilities.

3. The LSRO requalification examination will consist of a written examination and a ulk-through operating test, which is administered and >

evaluated individually. References to the crew-based dynamic simulator test and the associated crew evaluation criteria and forms do not apply to LSR0s. NUREG-1021 1 of 7 Interim Rev. 8, January 1997

i l l ES-702

4. Whenever possible, the facility licensee should include an LSR0 on the examination team.

5. The requirement to examine at least 12 operators to arrive at a program evaluation is not applicable. The region and OLB will determine the appropriate sample size based on the number of LSR0s licensed at the facility.

6. The " Corporate Notification Letter" (Attachment 2 of ES-601) shall be revised as necessary to reflect the examination arrangements and to specify a modified list of reference material requirements associated with LSR0 fuel handling activities. The region will review the l

reference material using the applicable portions of Form ES-601-2.

7. The facility licensee is expected to maintain JPM and written examination banks for evaluating LSR0s. These examination banks should be periodically updated to reflect areas of emphasis in training and to ensure that all applicable knowledge and skills are represented. There is no minimum threshold or ceiling for these banks.

8. The region will document the NRC's LSR0 requalification examination results using Forms ES-702-1, "LSR0 Requalification Examination Report,"

and ES-702-2, "LSR0 Requalification Rez ults Summary," instead of Forms ES-601-3, 4, and 5.

9. This standard does NOT provide for a formal LSR0 requalification program evaluation, however if more than one-third of the examined LSR0s at a facility fail, the NRC may need to inspect the LSR0 requalification program. The regional staff is responsible for determining if such an l inspection should be conducted. If an inspection is performed, the l staff should assess at least the following: '

a. the content of the training program, the development of '

examination materials, and quality controls

b. the administrative controls for maintaining training material current with procedural revisions and design changes

c. the training and evaluation techniques of the facility licensee's evaluators

d. the evaluation techniques that the facility licensee uses to determine if it has implemented and assessed its training effectively

e. the frequency, scope, and depth of the training provided to the operators NUREG-1021 2 of 7 Interim Rev. 8, January 1997

j i 1 e i l ES-702 D. EXAMINATION DIFFERENCES 1.. Written Examination i The written examination will be developed, administered, and evaluated i as described in ES-602 with the following exceptions: ! a. The written examination will be open reference and will contain a d minimum of 25 points in a single section; static simulator ! scenarios do not apply to the LSR0 examination. The time limit i for completing the examination shall be two hours, but the ! examination should be constructed so tnat a competent LSR0 'can

complete it in 1.5 hours. The examination should emphasize

refueling procedures, administrative controls, and abnormal and j- emergency procedures. The examination should include questions  !

i associated with industry and licensee event reports (LERs) and l j recent plant modifications affecting refueling operations and l systems that apply to the facility. l 1 ! b. Form ES-702-3 will be used as a cover sheet rather than Form l ES-602-1.

2. Walk-throuah Goeratina Test The walk-through test will be developed, administered, and evaluated as described in ES-603 with the following exceptions:

a. Each LSR0 will be administered an operating test consisting of five tasks /JPMs. Whenever possible, these tasks should include  ;

the use of the refueling equipment to manipulate duary fuel on7y 1 or the use of a refueling machine simulator if one is available at the facility. If dummy fuel manipulation or the use of a simulator are not possible, the refueling tasks should be , simulated. The requirement to conduct a minimum number of JPMs in 1 the control room / simulator is not applicable.

b. Each JPM will consist of a task normally performed by fuel handling personnel and will include tasks performed before and after refueling and for maintenance, surveillance or testing of systems or equipment. The examination team may evaluate the LSRO's ability to perform normal fuel handling administrative tasks including the documentation of clearances, maintenance activities, and surveillances. The operating test should also evaluate the LSR0's response to abnormal or emergency events associated with fuel handling.

NUREG-1021 3 of 7 Interim Rev. 8, January 1997

     ~.         .           _  _.

l ES-702

c. If sufficient facility-developed JPMs are not available, the NRC  !

can conduct a walk-through examination of the type administered to an initial LSR0 applicant, as discussed in ES-701.

3. Dynamic Simulator Operatino Test The dynamic simulator operating test described in ES-604 is not applicable to the LSR0 requalification examination.

E. ATTACHMENTS / FORMS Form ES-702-1 " Individual LSR0 Requalification Examination Report" Form ES-702-2, " Power Plant LSR0 Requalification Results Summary" Form ES-702-3, "LSR0 Written Requalification Examination Cover Sheet" O 9 NUREG-1021 4 of 7 Interim Rev. 8, January 1997

i, r e ! ES-702 Individual LSRO Requalification Form ES-702-1

Examination Report l

i U.S. Nuclear Regulatory Commission , Individual LSRO Requalification Examination Report i I l 4 l Operator's Name: Facility: i 1 j Docket No.: 55- Retake Exam: 1st/2nd/f Date of Last Exam: 3 i License No.: S0P - Expiration Date:  ; i j Written Examination Results 1 l Date of Exam: NRC Examiner: Facility Evaluator: i t l j NRC Facility Overall l Grade (%) -> j 0perating Test Results i ! Date of Test: NRC E-  : Facility Evaluator: i i a No. JPNs Correct of of j . ! Final Grade  %  % 3 NRC Examiner Recommendations Category Results Signature Written Pass / Fail Operating Pass / Fail NRC Supervisor Review Date: Pass / Fail NUREG-1021 5 of 7 Interim Rev. 8, January 1997

l ES-702 Power Plant LSR0 Requalification Form ES-702-2 Results Sumary PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY Power Plant LSRO Requalification Results Summary Facility: Exam Date: Examiners: Total # of Passed Failed Overall Operators (# / %) (# / %) Results Individual Results Operator's Docket Grader JPM % Written Results(P/F) Name 55- Overall (%) Written Operating NRC Fac NRC l Fac NRC Fac NRC Fac NRC Fac NRC Fac NRC Fac PRIVACY ACT INFORMATION - FOR OFFICIAL USE ONLY NUREG-1021 6 of 7 Interim Rev. 8, January 1997

Q ES-702 LSR0 Written Requalification Examination Cover Sheet Form ES-702-3 1 } l ) U. S. Nuclear Regulatory Commission l LSRO Written Requalification Examination l l Operator Information Name: , Date: Region: I / 11 / III / IV Facility / Unit: Reactor Type: W / CE / BW / GE Start Time: Stop Time: Instructions Use the answer sheets provided to document your answers. Staple this cover sheet on top of the answer sheets. Points for each question are indicated O in parentheses after the question. The passing grade requires a final grade of at least 80.00 percent. Exe.h ation papers will be picked up two hours after the examination starts. Operator certification All work done on this examination is my own. I have neither given nor received aid. Operator's Signature Results Test Value Points Operator's Score Points Operator's Grade Percent NUREG-1021 7 of 7 Interim Rev. 8, January 1997

i APPENDIX A OVERVIEW OF GENERIC EXANINATION CONCEPTS ' } A. PURPOSE The purpose of this Appendix is to provide an overview of two fundamental

examination concepts, validity and reliability, as they apply to the 2

development of NRC operator licensing and requalification examinations. The ! following topics are discussed: the rationale for providing guidance for the construction, review,  ! and approval of NRC examinations (Section B) the various aspects of validity and how the NRC establishes the validity of its examinations (Section C) the concept of reliability and how it is maintained on NRC examinations (Section D) B. BACKGROUND The fact that the NRC's operator licensing examinations are prepared and administered by many diffeNnt individuals working in various locations makes it imperative that a defied set of administrative structures and protocols be established and followed to ensure that the examinations are administered O-successfully and consistently. External structures such as the number and kind of items, the length of the examination, security procedures, proctoring instructions, and other administrative details are essential to the orderly conduct of an examination. These factors can have a significant effect on the reliability and validity of an examination, the cornerstones that allow the NRC to make cenfident licensing decisions. 4 In addition to these external structures, the internal attributes of the examination, such as its level of knowledge, level of difficulty, and the use of item banks, also impact the operational and discriminatory validity of the exaaination, which, in turn, can affect its consistency and reliability. When the internal and external structures of examinations are allowed to vary  ; significantly, this challenges the uniform conditions that are required by Section 107 of the Atomic Energy Act of 1954, as amended, and the basis upon which the NRC's licensing decisions rest. The NRC must reasonably control and structure the examination processes to ensure the integrity of the licenses it issues. Acceptable levels of examination consistency, uniformity, and fairness would be impossible to achieve without quantitative and qualitative acceptance criteria. While the Examination Standards identify many of the quantitative criteria necessary for a well-balanced and consistent examination, those criteria can only go so far when it comes to ensuring examination validity and reliability. Although the NRC's Knowledge and Abilities Catalogs (NUREG-1122 and -1123) have brought a degree of consistency to the qualitative issue of O NUREG-1021 1 of 10 Interim Rev. 8, January 1997

i l l Appendix A safety-significance, there is no comparable mechanism to aid in determining an examination's level of knowledge or difficulty before it is given. In the end, the validity and consistency of our examinations depend in large part on the individual and collective judgments of the people who write and review the i examinations. It is hoped that the discussions herein will clarify the intent of the NRC's examination critoria, thereby decreasing the likelihood that inconsistencies among examinations, particularly with regard to the level of knowledge and difficulty, will jeopardize the validity of the NRC's licensing decisions. C. VALIDITY For a test to be considered valid, it must be shown to measure that which it is intended to measure. In the case of the NRC examinations, the intent is to measure the examinee's knowledge and ability such that those who pass will be able to perform the duties of the R0 and SR0 to ensure the safe operation of the plant. The three principal facets of test validity and the techniques l that are used to establish the validity of NRC examinations are outlined below.

1. Content Validity

a. Establish a Link to Job Duties In order to develop valid examinations, the knowledge and abilities (K/As) selected for testing must be linked to and based upon a description of the most important job duties. This is accomplished through the conduct of a job / task analysis (JTA), focusing on the delineation of essential K/As.

This approach to the development of content valid licensing examinations was endorsed by the testing industry in the 1985 revision of the " Standards for Educational and Psychological Testing" published by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. The standards treat licensing examinations in a separate section in recognition of their importance and uniqueness. Accordingly, those seeking additional technical guidance are encouraged to consult Chapter 11 of the document for further clarification. For the purposes of validating the content of the NRC examinations, the JTA performed on the licensed operator position by the Institute of Nuclear Power Operations (INPO) served as the initial source of information. The INP0 JTA identified more than 28,000 K/As and nearly 800 tasks. The extensive number of INP0 tasks and K/A statements is due, in part, to the specific purpose of the INP0 analysis, which was to provide an information base to be used in NUREG-102I 2 of 10 Interim Rev. 8, January 1997

f

( Appendix A developing training programs that would be applicable to all PWR and BWR facilities. Accordingly, many of the individual statements were too specific and/or too elementary for use as the basis for the development of NRC examinations. Perhaps most important, the job 2 content of special interest to the NRC is that subset of K/As that

are required for the safe operation of the nuclear plant. Although safe performance and efficient performance may have considerable 4

overlap, any K/A that contributes to efficiency but not to safety is 1 1 an inappropriate focus for the NRC examination. ' NUREG-1122, " Knowledge and Abilities Catalog for Nuclear Power Plant 1 , Operators: Pressurized Water Reactors," and NUREG-1123, " Knowledge  ! ! and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors," provide the basis for the development of content valid examinations for R0s and SR0s consistent with the testing industry standards described above. The fact that the K/As from l

which test items are developed are drawn or sampled from the same  !

universe regardless of who develops the examination ensures that the examinations are consistently content-valid. Furthermore, i developing the examinations using the appropriate K/A catalog in conjunction with the applicable Examination Standards and related l Appendices will ensure that the examinations cover a representative O i . sample of the topics listed under Title 10, Part 55, of the Code of j () Federal Regulations. '

The NRC's K/A catalogs were developed based upon the INP0 JTA and were reviewed by licensed R0s and SR0s as well as NRC license examiners. These experts reviewed the K/A statements for accuracy and completeness and then rated each statement with respect to its importance to safe plant operation. Further explanation of the content of the K/A catalogs is provided in Section 1 of each catalog.

Beyond the NRC's K/A catalogs, learning objectives from the facility

     ,            licensee's training program often provide a supportive reference for test items to be included in the NRC examination. Since facility learning objectives are specific to the job requirements at that site, they should provide an excellent basis for test item development. However, because they are not always stated at the comprehension or analysis levels of knowledge (the preferred focus for NRC examinations) they should be referenced only to the extent that they support a test item that is being developed.

b. Use a Sample Plan Once the essential K/As have been identified through the conduct of the JTA, test specifications must be developed. The test specifications consist of a content outline or sample plan NUREG-1021 3 of 10 Interim Rev. 8, January 1997

Appendix A indicating what proportion of items or questions shall deal with each K/A. Because a single test cannot measure every knowledge or competency learned during a training program, it must sample the required knowledge or performance in a manner that allows inferences to be made regarding the examinees' performance on the broader population of knowledge even though it was not tested. The sample must be evenly distributed and soundly based so that we can confidently assume that the untested knowledge is proportionately known or not known in relation to the score on the sample. In other words, by testing performance on the sample, it is possible to make inferences upon the broader realm of knowledge not tested. This is referred to as a validity inference. The sample plan is at the heart of making a validity inference. Research indicates that when samples are not chosen systematically and according to the sample plan, the sample is said to be biased which reduces validity. When the sample is biased or skewed in a particular direction, it introduces sampling error that makes it impractical to infer or generalize that the examinees have mastered the larger population of untested knowledge from which the sample was drawn. Test items selected for inclusion in an NRC examination should be based on K/As contained in the appropriate K/A catalog. Testing outside the documented K/As can jeopardize the content validity of the examination. Content validity can also be compromised if important K/As are omitted from the examination. Therefore, the sample of K/As that are tested should cover all the K/A categories in the catalog in a fashion that is consistent with their contribution to the public protection function of the examination. Not all categories are equal in this regard. This conclusion is based on the analysis of ratings on importance and testing emphasis collected from licensed SR0s and NRC license examiners. The specific Examination Standards provide additional guidance on how to develop test outlines that will ensure adequate content coverage. It is important to note that there is a difference in the testing demands for an initial examination versus a requalification examination. The requalification examination is based upon the plant's systems approach to training during the requalification cycle and will more closely parallel the training received in the requalification program. Consequently, the instructional and testing processes are more closely linked. The initial examination, on the other hand, covers all instruction related to safety-significant K/As that either were or should have been taught during the training program. The Examination Standards ensure that the K/As are sampled in a relatively uniform process that would likely include content and instruction that occurred from the beginning to NUREG-1021 4 of 10 Interim Rev. 8, January 1997

t i a i Appendix A i the end of the program and not be focused upon that instruction most i recently presented, studied, and tested.- i i

2. Operational Validity

. The second dimension of validity is operational validity. To the extent

possible, test items should address an actaal or conceivable mental or  :

! psychomotor activity performed on the job. In this regard, the more

operationally oriented a test item is, the more-valid the test item '

, Since operationally valid items involve skills central to job ! performance (i.e., analysis, predictions of-events or system responses, ! or problem-solving) the items should be written et the comprehension or  ; ! analysis levels rather than simple fundamental knowledge. The 1 theoretical level of knowledge classification system upon which the NRC j i bases its operational validity estimates is Bloom's Taxonomy. l l Bloom's taxonomy suggests that testing knowledge at higher cognitive l 2 levels (i.e., comprehension and analysis) is more efficient and i operationally valid because testing at those higher levels subsumes the j fundamental knowledge required in part to answer the higher level. + question. Furthermore, the higher the level tested in the test item, l generally the more operationally valid that test item will be since it  ; is at the higher levels that questions invoke problem-solving, l diagnosis, prediction, and analysis of conditions, events, and responses. . Designing test items that test the app 1fcation of knowledge in different content situations (i.e., process testing) is at the heart of designing good, discriminatory test items. Just as the mathematics teacher would not ask multiplication questions that were identical to practice questions, so too should the examination author minimize asking questions that are identical to those previously rehearsed or tested. Both should attempt to assess similar knowledge applications in different contexts, thereby assessing the problem solving skills of students in new and different applications. These applications should l be item substitutions of comparable difficulty, neither harder nor less difficult than those practiced. This would provide assurance that the examination would be valid and discriminatory, since process rather than content is primarily measured. The NRC cannot make cenfident and consistent validity inferences (i.e., licensing decisions) if one examination assesses knowledge at lower levels and another examination assesses knowledge at higher cognitive levels (greater depth). While each examination may meet sample plan coverage, the examinations are testing different levels of knowledge and consequently, they are different and inconsistent measuring instruments. It is clear to see, therefore, that the validity inferences to be drawn regarding minimally safe operator performance are different in each tO. NUREG-1021 5 of 10 Interim Rev. 8, January 1997 '

Appendix A case. Rcfer to Section D for a more detailed discussion of consistency and reliability and to Appendix B for a more detailed discussion of the various levels of knowledge as they relate to the development of written , test questions.

3. Discrimination Validity The third central validity issue concerns the examination's ability to discriminate, or to make some distinction along a continuum of examinee i

performance. In that regard, the primary objective of the NRC examinations is to determine whether or not the examinees have sufficiently " mastered" the knowledge, skills, abilities, and other attributes to perform the job of rea'ctor operator (RO) or ser.ior reactor operator (SRO) at a specific plant. The NRC examinations are not intended to distinguish among levels of competency or to identify the most qualified individuals, but to make reliable and valid distinctions at the minimum level of competency that the agency has selected in the interests of public protection.

a. Criterion-referenced Testing 4

The NRC's initial and requalification examinations, like most > licensing examinations, are criterion- rather than norm-referenced tests. This means that there is a pass-fail or minimal cut score or grade that the examinee must achieve to demonstrate sufficient knowledge and ability to safely operate the power plant. If the examination does not intend to discriminate at an agreed-upon minimal measure of knowledge or performance, then there is little reason to give the examination. For a criterion-referenced test to be effective, both the individual test items and the examination in total must discriminate between safe and unsafe operator performance,

b. Cut Scores In some commercially-developed examinations, the system for establishing cut scores assumes a fixed test content and variable cut scores. In such a case, the problem is one of finding the optimal value of the cut score variable. However, for NRC examinations the situation ir just the reverse. The cut scores (on the written examination and JPMs) are fixed at 80 percent; it is the content of the examination which varies from occasion to occasion because of the plant-specific character of the test material. As is discussed below, there are several reasons why the cut score must be j fixed, including the uniqueness of each examination, consistency, and public confidence.

NUREG-1021 6 of 10 Interim Rev. 8, January 1997

f l l 1-Appendix A 4 i In the writing, reviewing, setting of scoring standards, and grading ! of any particular NRC examination, both the examination author and the reviewer are well aware of the NRC-established passing score of

80 percent. They may also have knowledge of how prior examinees i have done on questions similar to the ones being used on the j' examination under construction and expectations as to how a

qualified or an unqualified applicant should do on the examination.

They must use this knowledge to control the nature and difficulty of the examination such that an examinee who is deemed to be qualified

scores above the passing grade and an examinee who is deemed to be

! unqualified scores below this grade. The traditional cut score on the examination should not be viewed as ! arbitrary. Rather, it reflects a point on the test at which author I and reviewer judgment separates the qualified from the unqualified.

To be sure, this judgment is implicit rather than explicit.

i Nonetheless, tha judgment is probably similar to other methodologies i for determining passing test scores. For example, rather than

explicitly judging the probability that a minimally qualified applicant will pass an ites, the author is imp?fcitly being asked to write an examination on which, in the author's judgment, the i minimally qualified applicant wi.11 obtain a score of at least 80 l O percent. Achieving this objective requires the author and reviewer to integrate their content and process skills,

c. Cut Scores and the Level of Difficulty For the cut score of 80 percent to be meaningful requires that individual test items be written "near" that level; a target range of 70 to 90 percent level of difficulty is recommended.- Test items '

that are so difficult that few if any of the examinees are expected to answer correctly do not discriminate and should not be used on an NRC examination. Test items that are so easy or fundamental that even those examinees who are known to have performance problems will be able to answer correctly should be used with discretion. Despite the best intentions, it is expected that every examination will contain some test items that all or most of the examinees will answer correctly or incorrectly.. This does not necessarily mean that the test items or the examination are invalid. It should be stressed that the intent is not for everyone to get a score of 80 percent. In fact, historically over 90 percent of examinees score 80 percent or above on the NRC examinatOns. A score of 80 percent is the minimal pass score that the av hor and reviewer must keep in mind as a functional level of discriicination for setting item difficulty. In order to achieve this, the test author must keep in mind and integrate the following concepts: NUREG-1021 7 of 10 Interim Rev. 8, January 1997

Appendix A the level of knowledge required of the examinees taking the examination the operationally validity of the questions (i.e., are they expressed as a conceivable job behavior?) the ability of the distractors to distract the examinees the examinees' past performance on items of similar difficulty By setting the individual item difficulty level in the 70 to 90 percent range, past experience tells us that there will be examinees who score in the upper 90s and others who may score in the upper 70s. It is the latter group that must be identified and can only be identified by developing an examination that intends to discriminate at the 80 percent level.

d. Use of Item Banks Test item banks are a valuable resource for learning and represent one fundamental basis for training and testing. However, it would be inappropriate to copy all or a significant portion of the items for an examination directly from the bank if the same items were previously used for testing or training. Test item banks must be used properly to maintain the validity, reliability, and consistency of the examinations. Previously administered test items reduce examination integrity because examination discrimination is red u d.

Discrimination is reduced because the cognitive level at which the examinees are tested could decrease to the simple recognition level if the item bank is small and available for the examinees to study. The comprehension and analysis levels of knowledge may not be assessable because mental thought has been reduced to a recognition level, and decision-making is absent because test items, JPMs, or scenario events have been rehearsed and are anticipated. In short, challenge and mental analysis are lost and the examinees are tested at a rote-style rehearsal level. An examination cannot assess higher cognitive and analytical abilities if a significant portion of the items within the examination have already been seen. Furthermore, when the bank of items from which the examination i; drawn is known to the examinees prior to the examination, then the examination is said to be highly predictable. Predictable examinations tend not to discriminate because what is being tested is simple recognition of the answer. Although studying past examinations can have a positive learning value, total predictability of examination coverage through over-reliance upon NUREG-1021 8 of 10 Interim Rev. 8, January 1997

4

Appendix A examination banks reduces examination integrity. When the examinees know the precise and limited pool from which test items will be drawn, they will tend only to study from that pool (i.e., studying to the test) and may likely exclude from study the larger domain of job knowledge. When this occurs, it decreases the confidence in the

, validity inferences that are made from performance on the test to that of the larger realm of knowledge or skill to be mastered. Therefore, the NRC has placed limits on the use of facility item banks or other such available banks or resources that have been published, reviewed, or used as the basis for training; the specific limits are discussed in the Examination Standards. The NRC appreciates the amount of resources required to develop new test items that are appropriate for use on an NRC examination, and it realizes that existing test items are a valuable resource that should not be wasted. Therefore, the NRC has elected to strike a balance in setting limits on the mix of previously used bank items, modified bank items, and newly developed (i.e., not previously seen) l i items. Additional limits have been placed on the repetition of test i items from prior quizzes and examinations given at the facility. . D. RELIABILITY 1 () Reliability is the second fundamental testing concept that has played a decisive role in the development of the NRC's initial and requalification examination programs. Whereas the notion of validity emphasizes the appropriateness of the content of the NRC examinations, reliability stresses consistency, repeatability, and the degree of confidence that the examination process will result in valid pass / fail decisions. The reliability of an examination is as important as its validity; if it is not reliable, then it cannot be valid. The importance of examination consistency (reliability) cannot be overstated. Test reliability, in fact, represents the consistency among examinations which, in turn, gives the NRC the confidence that all examinations are valid measures from which to make confident and valid licensing decisions. It is clear to see that the combined effects of item bank use, the level of knowledge tested in the individual test items, and the expected discriminatory (difficulty) level of the items play an important role in determining the reliability of the examination. The higher the reliability of a test, the fewer errors will be made in deter-mining whether the examinees have mastered the job requirements. Examinations should differ only in the specific content covered, not in their developmental processes, manner of sampling, item construction criteria, level of item bank use, or their levels of knowledge and difficulty. It is the standardization of the process that creates consistency of measurement. Ideally, any two examinations that are written in accordance with these procedures and U NUREG-1021 9 of 10 Interim Rev. 8, January 1997

Appendix A guidelines and given to the same group of examinees should produce identical results; likewise, the results of any examination given to different but similarly trained and qualified examinees should also be comparable. The standardized examination development, administration, and grading procedures described in this NUREG have evolved over a period of years in an effort to enhance the reliability and, hence, the validity of the NRC's licensing decisions. The importance of having these procedures and complying with their intent has grown in proportion with the number of individuals and organizations that have become involved with the examination process. Section 107 of the Atomic Energy Act of 1954, as amended, requires the ) Commission to prescribe uniform licensing conditions for operators. ' Therefore, facility licensees are expected to develop and submit their proposed examinations based on the guidelines and instructions contained herein. The NRC discourages facility licensees from using testing methodologies that do not conform to the policies, procedures, and practices defined in this NUREG. Nevertheless, facility licensees may propose deviations from tpecific guidance in NUREG-1021, and the NRC will review and rule on the acceptability of the deviations.  ! l 9' NUREG-1021 10 of 10 Interim Rev. 8, January 1997

O APPENDIX B \'/ WRITTEN EXAMINATION GUIDELINES A. PURPOSE The parpose of this appendix is to provide a background understandi'ng of the principles and practices for developing multiple-choice written test questions for NRC initial and requalification examinations. Examination authors and reviewers should use the guidance herein when selecting, constructing, and reviewing questions for use on NRC written examinations. The following topics are covered: written examination background (Section B) the basic psychometric principles (e.g., low level of knowledge, low operational validity, low discriminatory validity, implausible distractors, confusing language or ambiguous questions, confusing or inappropriate negatives, collection of true/ false statements, backward logic) and other guidelines applicable to the question development process (Section C) a checklist for reviewing multiple-choice questions (Attachment 1) examples of questions that illustrate the psychometric principles /O (Attachment 2) b - a list of references that provide additional information on written examination development (Attachment 3) B. BACKGROUND

1. The importance of the Written Examination Written examinations have been institutionalized into our society as an accepted and important facet of performance testing. A written examination is routinely used as an integral component of measuring

  ,       human performance in nearly every field of study. Educational institutions from elementary through graduate school use a written examination, in part or in whole, to measure intended competencies.

Moreover, many fields of business, including the legal, medical, educational, and accounting professions, use written examinations for licensing and credentialing activities. The importance of knowledge testing should not be underestimated since knowledge is the underpinning of professional performance. The objectives of' knowledge testing are varied; they may include assessment of fundamental understandings as well as testing more advanced levels of expertise. The most effective tests of knowledge include questions and test items that measure applications of knowledge directly related to the job. In the case of the NRC operator licensing examination, the 4

,O  NUREG-1021                             1 of 26       Interim Rev. 8, January 1997 1

l

Appendix B written examination yields a key measure that allows a confident decision to be made on the safety significant performance of the individual seeking a license. When knowledge testing is deemphasized or sidestepped through careless  ; or simplistic testing processes or if it is treated secondarily to other j portions of the examination that are more operationally oriented, then subsequent job performance could be affected. The loss of attention and focus on testing the individual operator's cognitive abilities (i.e., comprehension, problem-solving, and decision-making) or paying insufficient attention to the operator's fundamental understanding of job content (e.g., systems, components, and procedures) ultimately may place job performance at risk of gradual degradation. When the demand for disciplined learning and study declines or the level of knowledge (depth of application) required for the job is reduced, it could lead to less time spent in training preparation, less mental review and practice, more forgetting of factual details, less reinforcement and application of job concepts, and a gradual decline in performance. Moreover, without a solid fundamental knowledge base, we cannot expect operators to perform acceptably in situations that are not specifically addressed in procedures. Since every performance has an underlying knowledge component, that knowledge and its depth needs development and , assessment to ensure the operators' competence on the job. Recent l studies assessing mental performance in cognitively demanding emergencies point out that higher level cognitive thought such as event diagnosis and response planning are important in responding to safety-related events.

2. Objective Versus Subjective Test Items Traditionally, questions that require the examinee to supply an answer (e.g., short answer and essay) have been considered " subjective," while questions requiring the examinee to select an answer (e.g., multiple-choice) have been considered " objective." The names arose from the scoring of the items. If graders require subject matter expertise to interpret the answers the question has been considered subjective. If the examination can be scored by verifying a single letter or number, it has been considered objective.

Multiple-choice items are the most common and most popular of the select-type items. For reasons of consistency and reliability, they are currently the only type of items acceptable for use on NRC initial licensing examinations. Although multiple-choice items are not as easy to construct as other forms, they are very versatile, can be used to test for all levels and types of knowledge, and miaimize the likelihood of the examinee obtaining the correct answer by ghessing. Scoring multiple-choice examinations is also considerably more reliable and less NUREG-1021 2 of 26 Interim Rev. 8, January 1997

I Appendix B I time consuming than scoring open-ended response items. Furthermore,  ! l since each item requires less time to answer, more items can be used to  ! l test a larger sample of K/As. This provides better content coverage, which also increases test reliability. For purposes of NRC requalification examinations and initial operating i tests, the definition of " objective" is different than the traditional  ! one described above. An objective test item is one for which (1) there i l is only one correct answer, and (2) all qualified graders would agree on l l the amount of credit allowed for any answer. Therefore, all questions  :' l on NRC examinations shall be objectively gradable regardless of the item ! format. Questions with no single correct answer or for which the credit l given can vary, depending on who graded it or when it was graded, have , no place on an NRC examination. l l C. QUESTION DEVELOPMENT j Examination authors and reviewers should observe the following generic l principles and question construction guidelines when preparing NRC written j i examinations. The guidance is based upon psychometrics, the process of l l applying sound qualitative processes to mental measurements. The generic  ; principles apply to all question formats, including multiple-choice, while the i guidelines 'e Section C.2 apply strictly to the multiple-choice format. It is  ; i important u minimize the number of psychometric errors in NRC examinations l i because test items that are free of psychometric errors yield greater  ; l measurement validity. .l l The following principles and guidelines are summarized on Attachment 1, which l can be used as a desk-reference during the question development and review ! processes. The list appears to be long, but with practice, the concepts

become internalized, and the process becomes less difficult. Many of the i

principles are accompanied by examples that illustrate the psychometric errors that should be avoided. Additional examples are provided in Attachment 2.

1. Generic Princioles l

l a. Ensure that the concept being measured has a direct, important j relationship to the ability to perform the job. Although the importance of relevant knowledge and abilities (K/As) and testing objectives was stressed in Appendix A, it is equally important that construction of the question itself clearly reflects the importance of the topic. Word the question so that is has " face validity" as well as underlying content validity. That is, make sure that the question would be considered reasonable to other ! subject matter experts utilizing the same reference materials. NUREG-1021 3 of 26 Interim Rev. 8, January 1997 i l

Appendix B

b. Make sure that the question matches the intent of the K/A.

l It is very easy to end up with a question that tests a relatively  ! trivial aspect of an important K/A topic. When reviewing your draft j question, ask yourself whether it is likely that someone could j answer the question correctly and still not meet the objective or l intent of the K/A or perform the responsibilities or tasks for which  ! the K/A is needed. If you are having difficulty translating a K/A into a test question, ask yourself the following questions to help you generate ideas for  ! potential test questions:  ! l (1) What are the common misconceptions about this topic? l (2) Why is this topic important to satisfactory job performance? (3) In what sort of circumstances might it be important to l understand this topic?  ! (4) What might the individual do who does not understand this topic? (5) What might be the consequences of a lack of knowledge about this topic? (6) How can the individual demonstrate the knowledge?

c. State the question unambiguously and precisely. State the question as concisely as possible, but provide all necessary information. j Often the individuals who develop a question assume that certain stipulations or conditions are inherent in the question when, in fact, they are not. It is very diffictit for the person who wrote

 ,            the question to review it impartially or through the eyes of a new       )

reader. Therefore, it is very important to have others review your ' questions to ensure that all necessary information is included, and that all extraneous or superfluous information is deleted. Refer to Section C.3 for additional guidance regarding examiration reviews. I

d. Write the question at the highest level of knowledge reflected in the testing objective.

One of the most challenging aspects of question development is l attaining the appropriate level of knowledge. The reference i benchmark that the NRC uses to classify the levels of knowledge of test items is Bloom's Taxonomy, a classification scheme that permits the grouping of items by the level (depth) of mental thought and l NUREG-1021 4 of 26 Interim Rev. 8, January 1997 l

f , i Appendix B - l performance required to answer the items. (Refer to Attachment 3  ! for additional references for further reading on Bloom's Taxonomy). Although Bloom's Taxonomy is most pertinent to written examination ' questions it can also be applied to simulator scenarios and JPM ' items. In ascending order, the three levels are defined below; refer , to Section A of.' Attachment 2 for examples of each level: Level 1 (i.e., fundamental knowledge.or simple memory) tests the recall or recognition of discrete bits of information;  ; examples include knowledge of terminology, definitions, set , points, patterns, structures, or other specific facts. l Level 2 (i.e., comprehension) involves the mental process of r understanding the material through relating it to its own l parts or to some other material. Examples can include l rephrasing information in different words, describing or i recognizing relationships, showing similarities and differences among parts or wholes, and recognizing how systems interact, including consequences or implications. l Level 3 (i.e., analysis, synthesis, or application) testing is a more active and product-oriented testing which involves O the multi-part mental process of assembling, sorting, or ' integrating the parts (information bits and their relationships) to predict an event or outcome, solve a problem, or create something new. This level requires mentally using the knowledge and its meaning to solve , problems. Although test questions should be written to reflect the level of k knowledge that is most appropriate for a specific K/A, it is best to  ; avoid high percentages of knowledge-level questions on the ~ examination. (Refer to ES-401-for specific limits.) When there is a choice between two levels of knowledge, try to write the question to reflect the higher level. In general, test items at the comprehension and analysis levels are the most operationally or;iented and, therefore, tend to be the most valid and discriminatory measure of operator knowledge and safe performance. Questions that require only memorization or recall are not acceptable for use on open-reference (i.e., requalification) examinations.

e. Avoid questions that are unnecessarily difficult or irrelevant.

i As discussed conceptually in Appendix A, both the level of knowledge

and item difficulty are at the heart of examination discrimination.

Examination authors should develop examinations that are estimated j to center around the 80 percent cut score level, with individual i e*

NUREG 1921 5 of 26 Interim Rev. 8, January 1997

Appendix B ites difficulty estimated to fall in the 70 to 90 percent difficulty range. These parameters should not be viewed as precise benchmarks, but rather as approximate end points. Examination authors should consider the results of past examinations when preparing a new one. Past performance on individual test questions may provide a basis for generating new questions and for estimating the level of difficulty of the examination. For example, questions that everyone got wrong may be an indicator that the topic was not given sufficient emphasis in training or that the item was poorly worded. Conversely, questions that everyone got correct may indicate that the item was written at too low a level or that the distractors were not very plausible. Since item difficulty can usually be decreased or increased by revision, the examination author need not be overly preoccupied with difficulty when writing the items. The author should focus on achieving a valid measure of the concept he is attempting to evaluate. When attempting to determine the appropriate level of difficulty, it may be helpful to think of two groups of individuals, one composed of experienced operators and the other of typical applicants, and evaluate the likelihood that each group of individuals will be able to answer the question. If at least 80 percent of the job incumbents or license applicants should be able to answer the question as written based on the expected knowledge levels for the position (operator or senior operator), then the item is likely written at an appropriate discriminatory level. Examination authors and reviewers may also ask themselves the following questions in an effort to identify questions that are unnecessarily difficult or irrelevant: Could someone do the job safely and effectively without being able to answer the question? If so, is it because the content is inappropriate, because the wording is unclear, or l because the level of understanding is too great? What espects of the item or option might cause the most difficulty? Has the item been made artificially difficult? Can a person understand the principle being tested and still miss the item? Estimates of difficulty made by the examination author and reviewers may vary somewhat but should not vary widely. Unless there is some reason to doubt the estimates of some reviewers, the average estimate may be taken as a basis for judging the suitability of item difficulty for the examination. Items should be revised if estimates fall well below or above the 70 to 90 percent target range. NUREG-1021 6 of 26 Interim Rev. 8, January 1997

s Appendix B Research has shown that when authors write test items in their own area of specialization, there is a tendency to underestimate the difficulty of a concept or principle being tested. This tendency can manifest itself in two ways: (1) the author will view items of average difficulty as easy, or 2 in an effort to get plausible misleads among distractors in a(mu)ltiple-choice test item, the author may make the ites even more difficult. For this reason, an estimate of item difficulty made by the reviewers will probably be more accurate than one made by the author of the item. Examination authors should take care not to develop an examination that averager an 80 percent by writing items with wide swings of individual difficulty. For example, writing half the items at a 60 percent difficulty level with the other half at a 100 percent level would yield an average of 80 percent, however, the flaws in this approarn are numerous. The items at the 100 percent level, by design, would be meaningless since they would fail to discriminate at any level because the expectation is that nearly everyone would get the answer correct. On the other hand, those written at the 60 percent difficulty level, by design, would also not discriminate and likewise be unfair because those items would be expected not to be answerable by 40 percent of the examinees.

f. Limit the question to one concept or topic, unless a synthesis of concepts is being tested.

l There is a common misconception that testing for multiple K/A topics i in one question is a time-efficient way to examine. Questions l containing a variety of topics and issues only serve to confuse the i examinee about the purpose of the question and what is expected in I terms of a correct response. Each individual question should test l one K/A topic, and that topic, as well as the intent of the ,

question, should be clear to both the, reviewer and the examinee. l

g. Avoid copying text directly from training or other reference i material.

Another common tendency among examination developers is to copy sentences directly from reference material and turn them into test questions. Unfortunately, questions written in this way generally encourage rote memorization. Further, copying from reference material can cause ambiguity or deficiency in questions because the material lifted often draws its meaning and importance from its surrounding context. Therefore, important assumptions or stipulations stated elsewhere in the material are often omitted from the test question. Finally, such questions can frequently be answered correctly by examinees who do not really understand the i concept, but do remember the specific wording on a page of reference i NUREG-1021 7 of 26 Interim Rev. 8, January 1997

i l l l l i Appendix B material. Conversely, examinees who understand the topic, but not in the exact way it was written in the material, may miss the l question because of unstated assumptions or other missing l information. I

h. Avoid " backward logic" questions that ask for what should be provided in the question, and provide what should be required in the examinee's response.

In addition to testing on valid topics, it is important to examine on those topics in a way that is consistent with how the K/A should be remembered and used. Do not test on the topic in a backwards way. Section G of Attachment 1 provides examples of backward logic questions.

2. Other Question Construction Guidelines J

The following principles and guidelines apply specifically to multiple-choice questions.

a. Use four answer options.

The four-distractor multiple-choice item with only one correct answer is the only style acceptable for NRC examinations. The five answer option contributes nothing to the question but confusion, and any format with fewer than four distractors makes guessing correctly more probable. The following four basic models are acceptable and may be used in combination with one another. Model A: a. correct answer

b. incorrect answer

c. incorrect answer

d. incorrect answer This model depicts the traditional multiple-choice design format with one correct single word / phrase answer followed by three incorrect single word / phrases options. Notice that the lengths of all the options are similar.

Model B: a. correct answer

b. plausible misconception

c. incorrect answer

d. incorrect answer i

l This model, in which a plausible misconception is used as an I incorrect answer, is a variation of Model A. Notice again that the lengths of all options are similar. NUREG-1021 8 of 26 Interim Rev. 8, January 1997 1

                                                                                                                                          \

l l 1 Appendix B 1 Model C: a, correct answer with correct condition (e.g., l because, since, when, if, etc.) l b. correct answer with incorrect condition I c. incorrect answer with incorrect condition ! d. incorrect answer with incorrect condition Model C depicts an acceptable design that uses answers with conditions (i.e., a setting, event, cause/effect) that may make the answer correct or incorrect. Notice that Model C shows only one

correct answer with its correct condition and that all options are j uniform in length.

Model D a. correct answer

b. incorrect answer l c. correct answer with incorrect condition l

d. incorrect answer with incorrect condition Model D is useful when it is not possible to create all options in uniform length. This model shows paired lengths (two long and two short options) which avoids any single option from standing apart (either too long or too short) from the remaining options.

b. Do not use "none of the above" or "all of the above."

                                            "All of the above" questions rovide inadvertent clues to the examinee. When the "all of t e above" option is the correct

, response, the examinee need only recognize that two of the options ! are correct to answer the question correctly. When the "all of the above" option is used as a distractor, the examinee needs only to be able to determine that one option is incorrect in order to eliminate this option. "None of the above" responses should not be used with "best answer" multiple-choice questions, since it may always be defensible as a response.

c. Do not present a collection of true-false (T/F) statements as a multiple-choice item.

As discussed earlier, each item should focus on one K/A topic determined by the stem of the question. A question containing answer options related to many se)arate issues does not increase the efficiency of the question. On tie contrary, questions with multiple topics only confuse the examinee about the meaning and purpose of the question. A way of determining if you have a test item that is a collection of ! T-F statements is to check whether the answer can be determined or l the distractors can be rejected without the information contained in l the stem. If they can, then you likely have a question that is a l T/F collection. l f NUREG-1021 9 of 26 Interim Rev. 8, January 1997 l l 1

I Appendix B Refer to Section F of Attachment 2 for sample questions that illustrate this psychometric deficiency.

d. Define the question, task, or problem in the stem of the question.

In designing multiple-choice questions that are operationally based l and require an application /use scenario, one suggestion is to provide the conditions in the first part of the question separated l by a double space from the body of the question, and blocked to the left column with each condition bulleted, as in the following example:

                   "Given the following conditions:

Both main feed pumps tripped 1 AFW automatically started j AFW valves reset to control steam generator water level AFW suction pressure decreases to seven (7) psig ' Which ONE of the following describes the AFW pump response for the given conditions?

a. suction will automatically shift to the nuclear service water system b, suction will automatically shift to the upper surge tank

c. trip when suction pressure decreases to five (5) psig

d. trip after a six (6) second time delay" Include as much necessary information as possible about the problem or situation in the stem, leaving only the solution, action or effect for the answer options. Consider the following " poor" and "better" examples:

(Poor) "At 50% power:

a. the equilibrium xenon reactivity worth is approximately equal to the equilibrium xenon worth at 100% power

b. th equilibrium xenon reactivity worth is approximately one-half the equilibrium xenon worth as 100% power

c. the equilibrium xenon reactivity worth is approximately two-thirds the equilibrium xenon worth 1 at 100% power

d. the equilibrium xenon reactivity worth is I approximately three-fourths the equilibrium xenon .

worth at 100% power" l NUREG-1021 10 of 26 Interim Rev. 8, January 1997

1 Appendix B-i L (Better) "How does the 50%. power equilibrium xenon { reactivity worth compare to the 100% power equilibrium xenon l l reactivity worth?  ! . a. equal to  ! j b. one-half

c. two-thirds j

d. three-fourths" i

l e. When possible, avoid using negatively stated stems. If a negative i stem is necessary, highlight the negative word (e.g., not, never, i l- least).  ; 1  ! It is very tempting to write negatively stated questions, since they i can be constructed by picking three true statements out of the , i reference material and changing a fourth statement to make it false.  ! ! However, studies have shown that examinees do not do as well on ! negatively stated questions, either because they overlook the  ; i negative word and/or because negatively stated questions require  ! i examinees to pick an answer that is not true or characteristic, ' i which.can be somewhat confusing. In addition, these questions tend j to emphasize negative learning. For example, consider the following i

stem of a multiple-chsice question:

j "During 100% power operation, the feedwater heater 2A high level dump valve opens inadvertently. The condensate pumps 1 will not do which of the following:" j This stem can be made to read positively:

                                                       "During 100% power operation, the feedwater heater 2A high                                    )

level dump valve opens inadvertently. The condensate pumps will:

a. increase flow to maintain feedwater flow rate

b. trip due to a runout condition

c. have no response

d. trip due to low suction pressure" There are times when a negatively stated question is unavoidable.

However, never use a negatively stated stem with a negatively stated answer option, as illustrated by example E.3 in Attachment 2.

f. Provide sufficient counterbalance in questions with multi-part answers.

NUREG-1021 11 of 26 Interim Rev. 8, January 1997 l

l Appendix B Multiple-choice questions can legitimately contain multi-part answer options. However, if the answers contain too many parts and/or too many options for each part, cues indicating the correct answer may be unavoidable. Consider the following example:

                   "The RCS is in hot standby with no reactor coolant pumas running. If OTSG pressure is decreased, according to tie plant verification procedure, which of the following temperature responses indicates the presence of natural circulation?

a. T-H increases, T-C remains the same

b. T-H increases, T-C decreases

c. T-H decreases, T-C decreases

d. T-H remains the same, T-C decreases" The examinee could choose the correct answer (c) without knowing about the T-C temperature response in this situation, since "T-H <

decreases" only occurs in option "c".  ! Notice that two-part answers, with each part containing a two-option response, provides complete counterbalance, since all contingencies can be covered in four responses, as in the following example:

                   "Which of the following is a definition of quadrant power tilt ratio (QPTR)?

a. minimum upper detector output divided by avera3e upper detector output

b. maximum upper detector output divided by average upper detector output

c. minimum upper detector output divided by average lower detector output

d. maximum upper detector output divided by average lower detector output" A multi-part question format which is highly recommended is one in which the two-part answer options consist of a two-level response (e.g., yes/no; off/on) and a reason, as in the following example:

                   "Which of the following best describes the behavior of equilibrium xenon reactivity over core life?

a. it decreases because of the increased fuel burnup

b. it decreases because of the decrease in plutonium-xenon yield

c. it increases because of the increase in thermal flux

d. it increases because of the decrease in boron concentration" NUREG-1021 12 of 26 Interim Rev. 8, January 1997

     - - _          . .       - . - -           -       .~.    .-            _    - . - - - - - . - - . -

1 4 Appendix B f i g. When possible, include common misconceptions as distractors. Since i the purpose of the examination is to differentiate between competent . and less-than-competent examinees, a good source of questions 4 involves topics in which there are common misconceptions about i important K/A topics. For example, the following question was based upon a common misconception about loss of subcooling margin:

                                      "During a small break LOCA with a resultant loss of subcooling margin, why are the reactor coolant pumps (RCPs) i secured?

i a. to prevent pump damage resulting from operation under i two-phase conditions i-

b. to prevent core damage resulting from rapid phase separation upon subsequent loss of RCS flow

c. to reduce RCS pressure by removing the pressure head i developed by the RCPs

d. to remove the heat being added to the RCS by the

operating RCPs" i

h. Make all answer options homogeneous and highly plausible.

Consider the following " poor" and "better" examples:

                                      "On a loss of condenser circulating water intake canal, the upper surge tank, hotwell, and condensate storage tank will i                                      supply sufficient feedwater to allow decay heat removal for

approximately

i i Poor: a. 15 minutes Better: a. 8 hours l b. 8 hours b. 24 hours j c. 48 hours c. 48 hours

d. d. 72 hours" 3 months 4

i

       .                Notice how one method of changing the difficulty level of a question is to vary the similarity of the answer options. The distractors should be similar enough to be chosen by those who do not meet the testing objective, yet different enough so they do not test trivial

issues or distinctions. Also note how the answer options in each case have been listed in order of magnitude.

i. If the answer options have a logical sequence, put them in order (as in "h" above).

i a 4 i l NUREG-1021 13 of 26 Interim Rev. 8, January 1997 l

l l I Appendix B

j. Avoid overlapping answer options, as in the following example:

                   "The SPND uses rhodium i.hich decays with a half-life of 42                                       I seconds. How long will it take for a detector to indicate approximately 95% of an instantaneous power level change?

Poor: a. 2 to 4 minutes Better: a. I to 2 minutes l

b. 4 to 6 minutes b. 3 to 4 minutes '

c. 6 to 8 minutes c. 5 to 6 minutes

d. 8 to 10 minutes d. 7 to 8 minutes" l

k. Do not include trivial distractors with more important distractors.

In the search for distractors, it is very tempting to include relatively trivial facts along with options focused on more , important issues or concepts, as in the following example: l "Which of the following is true concerning the turbine? '

a. The turbine is rotated at low speed when shut down in order to prevent distortion of the turbine casing.

b. Turbine eccentricity is the measure of turbine speed. i

c. The turbine blades are cooled by hydrogen gas,

d. Technical specifications require at least one turbine overspeed protection system be operable in Mode 2."

Relative to the other options, "c" could be considered a trivial l distractor. Even if included as a wrong answer, the inclusion of relatively unimportant information jeopardizes the content validity of the question. Also note that this question consists of a collection of true/ false statements as described in Section C.2.c. I

1. Vary the location of the correct answer; avoid a pattern.

Make sure the position of the correct answer is randomized throughout the examination. This means that the "a", "b", "c" and  ; 1 "d" options should be correct about an equal number of times, but in no specific order.

m. Avoid " specific determiners" that give clues as to the correct answer. Specific determiners include the following:

(1) distractors that do not follow grammatically from the stem, as in the following example: NUREG-1021 14 of 26 Interim Rev. 8, January 1997

i () Appendix B "During 100% normal power operation a single steam flow element in the steam generator feedwater control system fails high. This will cause:

a. the feedwater valves to increase steam generator level slightly before returning the level to normal

b. before returning the level to slightly above normal, the feedwater valves to increase the steam generator level significantly

c. the feedwater valves to increase the steam generator level to the level of a reactor trip

d. the feedwater valves to increase the steam generator level slightly and maintain the increased level" Note the improvement when distractor "b" is reworded as follows:

                          "b.        the feedwater valves to increase the steam generator level significantly before returning the level to slightly above normal" (2) options that can be judged correct or incorrect without reading the stem (3) equivalent and/or synonymous options, which rule out both                       ,

options for an examinee who recognizes the equivalence (4) an option which includes another option (for example: a) less than 5; b) less than 3...) (5) implausible distractors (6) a correct answer which is longer than the distractors (7) qualifiers in the correct answer (e.g., probably and ordinarily) unless they are also used in the distractors (8) words such as "never" or "always" which suggest a wrong option (9) a correct option that differs from the distractors in favorableness, style, or terminology, as in the following example: s NUREG-1021 15 of 26 Interim Rev. 8, January 1997

i Appendix B l

                            "Which action or occurrence is likely to cause water       '

hammer?

                                                                                      )

a. maintaining the discharge line from an auto )

starting pump filled with fluid  ;

b. water collecting in a steamline  :

c. pre-warming of steam lines

d. slowly closing the discharge valve of an operating pump" In the above question, all options except for "b" (the l correct answer) describe preventive actions, while option '

                     "b" describes a condition that occurs as a result of negligence or oversight. A test-wise examinee need only know that water hammer is not a desired occurrence to determine that "b" is the least favorable and therefore the correct answer.

3. Reviewina Test Items Examination reviewers can assist examination authors by performing technical content, level of difficulty, psychometric, and editorial  :

checks. There is an advantage in considering each of these four areas separately and in this order. If there is a need to revise an item on ' the basis of one stage of the review, the changes should be made before going further because the changes at each stage could well affect the reviews which follow.' For example, a criticism which appears to affect only one distractor may ultimately lead to changes in other parts of the , item, so time spent reviewing the item for grammar and punctuation may l be wasted. ' There are some advantages in having someone not familiar with the area being tested review the questions for clarity, grammar, expression, spelling and punctuation. Such a reviewer can determine whether the , item is answerable by a person without knowledge of the field. The examination author and reviewers should ask themselves the following 1 types of questions: Will the examinees clearly know what they are expected to do? Do they have all the information they need to work with? Does answering the question depend on certain assumptions that must be stated? A more thorough list of suggestions for examination authors and reviewers to check is included in Attachment 1. D. ATTACHMENTS / FORMS l Attachment 1, " Question Development Checklist" Attachment 2, "Exampl es" l Attachment 3, " References" NUREG-1021 16 of 26 Interim Rev. 8, January 1997 j

 /

Appendix B Question Development Checklist Attachment 1 1

1. Does the concept being measured have a direct, important relationship to the ability to perform the job?

2. Does the question match the testing objective and intent of the K/A? l

3. Is the question clear, concise, and easy to re*J7 Could it be stated more simply and still provide the necessary 4 formation? Should it be reworded or split up into more than one que', tion?

4. Is each question stated positively, unless the intent is to test '

knowledge of what not to do?

5. Does the question provide all necessary information, stipulations, and assumptions needed for a correct response? Is as much information as possible included in the stem?

6. Is the question written at the highest appropriate level of knowledge or ability for the job position of the person being tested?

7. Is the question free of unnecessary difficulty, trickiness, or irrelevancy?

O' 8. Is the question limited to one concept or topic, making it something other than a collection of true-false items?

9. Does the question have face validity?

10. Are key points underlined or highlight::d?

11. Is each question separate and independent of all other questions?

12. Are the answer options homogeneous and highly plausible? Are common misconceptions used as distractors? Is the question free of trivial distractors?

13. Are "none of the above" and "all of the above" avoided?

14. Are there four answer options for each question?

15. Are the answer options of the questions ordered sequentially?

16. Is the question free of " specific determiners" (e.g., logical or grammatical inconsistencies, incorrect answers which are consistently different, verbal associations between the stem and the answer options)?

NUREG-1021 17 of 26 Interim Rev. 8, January 1997

1 l l Appendix B Examples Attachment 2 I A. LEVELS OF KNOWLEDGE The first three examples illustrate how the level of knowledge tested can vary among a series of questions that focus on the same pair of K/As. Even though the K/A statements use verbs (identify, define) that elicit a fundamental or simple memory level of knowledge, the item writer can increase the question's operational validity by testing at a higher cognitive level. 191004K101 (PWR) or 291004K101 (BWR): Identification, symptoms, and  ; consequences of cavitation  ; 193006K111 (PWR) or 293006K109 (BWR): Define or explain cavitation

1. Fundamental Knowledoe/Simole Memorv "Which one of the following describes pump cavitation? l

a. Vapor bubbles are formed when the enthalpy difference between pump  ;

discharge and pump suction exceeds the latent heat of vaporization. l

h. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.  ;

c. Vapor bubbles are produced when the localized pressure exceeds the  :

vapor pressure at the existing temperature. l

d. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer."

i This question simply asks for a description of cavitation and as such, l is a low cognitive order question. There is no understanding, analysis, j or problem-solving involved. The examinee only needs to recognize the  ; correct description (b); the other options appear plausible but are, nonetheless, incorrect.

2. Comprehension

        " Cavitation in an operating p wp may be caused by:

a. lowering the pump suction temperature

h. throttling the pump suction valve

c. increasing the pump backpressure

d. increasing the pump suction pressure" This example requires the examinee to determine causation, which requires an understanding of the correct answer and that the incorrect  !

ansvers are indeed, incorrect. The quality of the item, as with any item, is determined by the distractibility of the incorrect options. NUREG-1021 18 of 26 Interim Rev. 8, January 1997 9ii

1 Appendix B 2 Attachment 2 f i.

3. Analysis j "While on surveillance rounds, an operator notices that a centrifugal g i pump is making a great deal of noise (like marbles rattling inside the '

i pump casing) and the discharge pressure is fluctuating. This set of , l conditions indicates pump:

a. runout

h. cavitation

c. bearing deterioration j d. packing deterioration" l

This example requires the candidate to analyze multiple abnormal i indications (multiple effects) for an operating centrifugal pump and

determine the cause (complex cause-effect). All the distractors are initially plausible in that they have face validity (i.e., they have

. reasonable connections to centrifugal pump operation). l 4. The following four examples illustrate "Jow level of knowledge"

;              questions that should be used judiciously on NRC examinations.

4 l "Which one of the following is powered from 4160 VAC bus IA? 1 RHR pump A 3'

b. RHR pump B

c. RHR pump C

d. RHR pump D" j " Select the full core display indication of a drifting control rod.

1 1 red light i b. white light j c. blue light

d. amber light"

) Although the above items have a high K/A value, they are written at a i low level of knowledge and also have low operational validity and low

,              discriminatory value. The foll.owing question tests at a low level of

! knowledge because it doesn't ta-t the examinee's ability to recognize i what class a fire is or selecf .4e correct extinguisher. All the ! examinee has to know is that water is for class A fires, i s j NUREG-1021 19 of 26 Interim Rev. 8, January 1997 i I

Appendix B 3 Attachment 2 "Concerning use of water as a fire extinguishing agent, select the corrcct statement from the following: 1 It is the primary agent for extinguishing Class A fires and also  ; effective on Class B and C fires.  :

b. It is the primary agent for extinguishing Class B fires and also 1 effective on Class A and C fires.  !

c. It is the primary agent for extinguishing Class A and B fires but ,

not effective on Class C fires. '

d. It is the primary agent for extinguishing Class B and C fires but not effective on Class A fires."

The next question might be considered a fundamental knowledge level question that errs in the opposite direction: It could be too difficult unless the operators are expected to memorize the correct time requirement in order to preclude damage to equipment. Moreover, this item may also have low discriminatory validity unless at least 80 percent of the examinees are expected to know the answer from memory. '

     "RCP 2A tripped after running for 50 minutes. The RCP was restarted, but tripped again within 15 seconds. Which ONE of the following is the minimum required interval before the next attempt to start RCP 2A?

a. 15 minutes

h. 30 minutes

c. 45 minutes I

d. 60 minutes" l

B. LOW OPERATIONAL VALIDITY The next three questions illustrate another common psychometric deficiency, I low operational validity, that should be avoided on NRC examinations. ) i

1. "Under which one of the following conditions should the Shift Supervisor l inform the shop steward? I

a. initiation of a directed overtime request

b. disciplinary action against a supervisory employee

c. medical injury of a contractor employee

d. personnel error by a bargaining unit member"  :

While this question may be related to a shift supervisor's job, it has nothing to do with nuclear safety and should not be included on an NRC examination. 4 I NUREG-1021 20 of 26 Interim Rev. 8,.bnuary 1997

t i i e 4 i

           . Appendix B                                     4                                             Attachment 2 I            2.     "Which one of the following main steam line components is designed to                                            !

limit the differential pressure across the steam dryer assembly? I !' a. main steam line flow elbows

b. main steam isolation valves

c. main steam shutoff valves  !

i

d. main steam line flow restrictors" Knowing the purpose of a flow restrictor is not a good indicator of the

( operator's ability to operate the plant. Knowing the answer to this j question is not clearly job related. i , i 3. "Given that all components controlled by the ' Locked Valve, Breaker, and

.                  Component Control' administrative procedure must be properly sealed and i                   tagged, which one of the following is the correct location for the 4
                   "XXXX-XXXX" tag for an electrical breaker?

a. wired to the breaker handle l h. glued to the breaker cubicle

c. attached to the breaker cubicle with a magnetic clip

! d. wired to the breaker cabinet door"  ! This question is likely unrelated to the reactor operator's job function 1 and would therefore be unacceptable. 4 ! C. LOW DISCRININATORY VALIDITY j The next three questions illustrate another common psychometric deficiency, ! low discriminatory validity, that should be avoided on NRC examinations.

1. "Which one of the following reactor water levels will initiate the RHR j pumps?

! 1 level 1 only I b. level 1 and 2 only j c. level 1 and 2 and 3 only j d. level 6 only" l This information in this question should be known by all operators at i all times. The question has low discriminatory value and also tests at i a low level of knowledge. 1 i i i i 4 l NUREG-1021 -21 of 26 Interim Rev. 8, January 1997 I i

Appendix B 5 Attachment 2

2. "The plant is recovering from a scram due to a spurious Group I isolation. The cause of the isolation has been repaired and preparations are being made to reopen the MSIVs. Reactor pressure is currently 825 psig and the main steam lines are being pressurized.

WHICH ONE (1) of the following is the LOWEST main steam line pressure that will allow the MSIVs to be opened per the procedure?

a. 625 psig

h. 675 psig

c. 725 psig

d. 775 psig" This question does not discriminate and has low operational validity because in real life the applicant may not be expected to have memorized the procedure.

3. "S.G. (corrected) = S.G. (uncorrected) + (T - 77 dearees F)(.001) +

3 (Level Mark)(.003) Based on the above information, the specific gravity (SG) is  ? which  ? meet the Technical Specification Category A limit. Note: This question requires the use of TS 3.8.2.3.

a. 1.198, does NOT l

b. 1.195, does NOT

c. 1.207, does

d. 1.201, does" j This question c.3ht appears to test the examinees' ability to understand and apply batter) parameters to the determination of TS operability.

However, the question really only tests their ability to substitute certain parameters into a given equation and perform an arithmetic calculation. Reference to the TS noted in the question is not required based on the three different values of SG (corrected) supplied as distractors. Therefore, the question has a low discriminatory value since any individual possessing adequate arithmetic knowledge will arrive at the correct answer. D. IMPLAUSIBLE DISTRACTORS The next two questions illustrate the concept of implausible distractors, another common psychometric deficiency that should be avoided on NRC examinations. NUREG-1021 22 of 26 Interim Rev. 8, January 1997

i l 1 l

                                                                                                       )

Appendix B 6 Attachment 2

1. "Which.of the following will cause the RHR pumps to start during a '

design basis LOCA?

                                                                                                       ]

1

a. low drywell pressure

b. high reactor water level

s. high drywell pressure

d. MSIVs in the NOT OPEN position" Distractors "a," "b," and "d" are implausible, considering minimal knowledge of the plant response to a loss of coolant accident.

2. "Which ONE of the following conditions will NOT result in a shutdown of the SBGT System?

a. manual shutdown

b. high temperature 225 degree F charcoal bed

s. high temperature 180 degree F heater inlet

d. overloads in the local control panel" Distractor "a" is very implausible, and distractor "d" is subjective.

The question is also written from a negative perspective. E. CONFUSING LANGUAGE The following questions illustrate how confusing language and inappropriate negatives in the stem of the question can mislead examinees. Such questions should be avoided on NRC examinations.

1. "Which one of the following parameters will start HPCI, RCIC and SBGTS?

a. low reactor water level

h. high primary containment pressure

c. high reactor building exhaust radiation

d. low reactor building differential pressure" This question could result in 4 correct answers since the question could be interpreted individually or collectively.

1 NUREG-1021 23 of 26 Interim Rev 8, January 1997  :

Appendix B 7 Attachment 2

2. "Which ONE of the following most accurately describes the response to a static inverter failing.

a. The power supply will automatically transfer to the alternate 600 V Bus 2C / Vital AC Transformer 2A.

b. The 125 VDC hattery will maintain power to the Vital AC Cabinet for up to 5 hours.

c. The power supply can be manually transferred to the alternate 600 V Bus 2C / Alternate Static Inverter by depressing a transfer pushbutton.

d. The power supply can be manually transferred to the alternate 600 V Bus 2C / Vital AC Transformer 2A by positioning the transfer switch to ALTERNATE."

This question implies an automatic response, but the listed correct answer and one distractor are operator actions, not responses to the loss of the static inverter.

3. "Regarding temporary plant alterations (TPA), technical reviews are NOT required for -

a. a TPA NOT installed using an approved procedure

b. TPAs installed on B0P systems but are required for safety related systems

c. a TPA that has NOT been directed by the shift supervisor to be an emergency TPA

d. all TPAs directed by the shift supervisor.

This question contains multiple problems: (1) While negative questions can be used, they should be used for good reason; there appears to be no good basis for asking this question negatively. (2) Two of the distractors ("a" and "c") also contain a negative, creating a double negative and readability confusion, a violation of good item writing practice. The question should more appropriately ask the conditions under which technical reviews are required, thereby eliminating the negative in the stem. F. COLLECTIONS OF TRUE/ FALSE STATEMENTS Collections of true/ false statements typically only test simple rote memory; the examinee needs only to recall a definition or condition. The questions elicit nr, comprehension or problem-solving; hence, they lack operational validity. This type of question allows an examinee to answer the question without referring to the stem of the question and should be avoided on NRC examinations. NUREG-1021 24 of 26 Interim Rev. 8, January 1997

l l Appendix B 8 Attachment 2

1. "Which ONE of the following is true?

A. High drywell pressure will auto start the emergency diesel generators.

b. Low reactor water level will trip the main turbine.

c. High reactor pressure will initiate RCIC.

d. High reactor power with the mode switch in startup will NOT close '

the MSIVs."

2. "Which one of the following describes pump cavitation? i

a. Vapor bubbles are formed when the enthalpy differene.e between pump discharge and a pump suction exceeds the latent heat of vaporization.

h. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.

c. Vapor bubbles are produced when the localized pressure exceeds the vapor pressure at the existing temperature.

d. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer."

G. BACKWARD LOGIC Backward logic questions ask the examinee for information normally received, and provide the examinee with information he/she normally has to supply. In an operational setting, operators are faced with conditions and required to know what procedure to use. These questions ask them to do just the opposite and should be avoided on NRC examinations.

1. "Which of the following parameters will simultaneous 1v start HPCI, RCIC and SBGTS?

a. high RPV water level

h. high drywell pressure

c. low RPV water level

d. low drywell pressure" It would be better to select a parameter and then request the expected system response because that is more operationally relevant.

2. "If it takes 12.5 cubic feet of concrete to build a square loading pad 6 inches thick, what is the length of one side of the pad?"

This question gives the examinees information they should be asked to calculate, while it requires them to provide information they would be supplied in an actual work situation. NUREG-1021 25 of 26 Interim Rev. 8, January 1997

Appendix B References Attachment 3

1. B1oan, Benjamin (1956) Taxonomy of Instructional Objectives: The Classification of Educational Goals, Handbook I: Cognitive Domain, New York, McCay.

2. Isaac, Stephen (1990) Handbook in Research and Evaluation, 2nd ed., San Diego, EDITS.

3. Novac, J and Gowin, D. (1993) Learning How to Learn, Cambridge University Press.

O NUREG-1021 26 of 26 Interim Rev. 8, January 1997

n (' APPENDIX C JOB PERFORMANCE MEASURE GUIDELINES A. PURPOSE This Appendix provides a f:amework for preparing and evaluating job performance measures (JPMs) to ensure they are of appropriate substance and format for initial operator licensing and requalification examinations. The following elements are discussed in detail or attached for information: a basic procedure for developing new JPMs (Section B), including forms to document the JPM and to assess the quality of the product (Form ES-C-1 and ES-C-2) guidelines for the development and use of alternate-path JPMs (Section C) a discussion of walk-through evaluation techniques (Section D) selected examples of acceptable prescripted JPM follow-up questions (Attachment 1) Adhering to the concepts and guidelines discussed herein, in association with the specific operating test criteria cited in ES-301 or ES-603, as applicable, will enhance the consistency and validity of the walk-through tests. B. DEVELOPING AND REVIEWING JPMs The major JPM components and instructions for their development are summarized below. The instructions apply to both the initial and the requalification l examination programs, except as noted. Although they are written from the  ; perspective of new JPM development, the instructions should also be I referenced, as necessary, when modifying existing JPMs for reuse and reviewing proposed JPMs for quality. Select the systems and tasks to be evaluated during the walk-through portion of the operating test in accordance with the specific initial and requalification examination criteria in ES-301 and ES-603, respectively. If a JPM already exists for the selected task, it should be reviewed against the guidelines and criteria discussed below to ensure that it is acceptable for use. If a new JPM is required to evaluate the selected system or task, prepare the JPM in accordance with the following basic steps and document the JPM using Form ES-C-1, " Job Performance Measure Worksheet," or equivalent. Form ES-C-2, " Job Performance Measure Quality Checklist," can be used to verify that the relevant criteria are satisfied.

1. Specify Initial Conditions Determine those system and plant conditions that would permit the task to be performed realistically. They should provide sufficient information regarding the status of the plant and system to facilitate NUREG-1021 1 of 15 Interim Rev. 8, January 1997  !

Appendix C task performance, without coaching the examinee. If the task is intended to be performed on the simulator, it is worthwhile to ,

differentiate those specific initial conditions and system realignments i that are necessary for the task to be performed as planned from those i other general conditions that add realism and set the stage for performing the task but have no real bearing on the successful execution of the task. Breaking down the initial conditions in such a manner will  ;

, simplify the simultaneous administration of different tasks by two or l more examinees. ' If the JPN is intended to evaluate the examinee's ability to implement I an alternate path (refer to Section C) within the facility licensee's procedural guidance, the initiating equipment or instrument failure should be reflected in the simulator initial condition specifications. The JPM shall also include an ! nit!ating cue that provides the stimulus for the examinee to begin the task performance. When appropriate, the I cue should clearly specify the desired endpoint for the task. For ' example, if it is desired for the examinee to start and load the emergency diesel generator, the cue should state the load at which the , task will be considered complete. Alternate path tasks, as described in

Section C, may have an actual endpoint different from that stated in the initiating cue.

The initial conditions and initiating cue may be duplicated on a separate sheet of paper so that they can be handed to the examinee. l This is particularly helpful for tasks with detailed initial conditions i or those that will be performed in high-noise areas. Take care to l ensure that the initial conditions and initiating cue do not reveal the I nature of any alternate path JPMs that are planned. l

2. Identifv References and Tools The JPM shall identify those plant procedures that require task l performance and the procedures that provide guidance, directions, or standards for performing the task. When reviewing JPMs selected from the facility licensee's bank, it is important to ensure that the procedures identified in the JPM are still current.

The JPM shall also identify any special tools or equipment (e.g., a stop watch, wrench, fuse puller, or spool piece) that the examinee will need to perform the task. It is helpful to the examiner who will be giving the test if the location in which these items may be found is stated in the JPM. It is expected that any required tools will be readily available to the plant operators; they should not be staged specifically for the examination. NUREG-1021 2 of 15 Interim Rev. 8, January 1997

                                                                                                                                                                 )

l l Appendix C 4 3.- Develon Performance Criteria The JPN should have meaningful performance requirements that will provide a legitimate basis for evaluating the examinee's ability to safely operate the system or the plant. Artificially subdividing  : existing tasks to generate new ones may dilute the value of the JPMs to  ! a point where they become meaningless. ' l i The JPN shall identify specific performance standanfs, or check points,  ! L that will permit the examiner to evaluate successful progress toward completing the task in accordance with the procedural references. Detailed control and indication nomenclature and criteria (e.g., switch ) positions and meter readings) .should be identified whenever possible, l even if these criteria are not specified in the procedural step. The JPM should also note any faportant observations that should be made by - the examinee while performing the task. The JPN must clearly identify the task standani; i.e., the predetermined ' outcome (qualitative and/or quantitative) against which task performance will be measured. Every procedural step that the examinee must perform < correctly (i.e., accura.tely, in the proper sequence, and at the proper I time) in order to accomplish the task standard shall be identified as a critical step and shall have an associated performance standard. If there are any specific procedural restrictions on the sequence in which the steps are performed they shall be clearly noted in the JPM.

4. Develoo Examiner Cues The JPM shall identify appropriate system response cues so that the examiner can provide the examinee with specific feedback regarding the component and system reactions to the examinee's manipulations, especially those procedural steps identified as critical to task completion. The response cues are particularly important in the following situations:

for in-plant tasks that will be simulated because the examinee will not have available the normal indications (e.g., alarms, flow rates, temperatures, and pressures) that would be observed during actual task performance for alternate path JPMs that require the examinee to perform auxiliary procedures when equipment or instrumentation fails during use

System response cues may not be necessary for those tasks that will be l performed on the simulator, b

} NUREG-1021 3 of 15 Interim Rev. 8, January 1997 i

1 Appendix C i To the extent that it is possible to anticipate incorrect actions that ' the examinees might take, it is beneficial to note the expected system response cues in the JPM as an aid to the examiner who will be administering and evaluating the task. The JPM shall also identify any additional cues or instructions that the examiner might need to provide to the examinee in response to procedural steps for which the examinee will not be held accountable; i.e., those steps that have either already been performed or will be performed by other personnel in remote locations.

5. Develoo a Time Standard Every JPM shall identify an estimated average time for completing the task. The time should be measured from the moment that the examinee is read the initiating cue at the plant location in which an operator would normally be given the order to perform the specified task.

JPMs that are considered time-critical (i.e., those having a task standard that must be completed within a time period specified in a regulation or a facility commitment to the NRC) shall be uniquely identified and specifically validated. The facility licensee must agree that a failure to complete the task within the time specified will justify a failure of the JPM.

6. Develop Pre-scrioted Follow-up Ouestions If the JPM will be used during an initial licensing examination, it shall include a sufficient number of prescripted questions (with answers and references) to evaluate the examinee's understanding of two knowledge and/or ability (K/A) statements related to the system or task.

More than one question may be required to effectively evaluate some K/A statements. The most appropriate format for this application is the short-answer i question, which requires the examinee to compose a response rather than l select from among a set of alternative responses, as is the case with multiple-choice, matching, and true/ false questions. Keep the following guidelines, in addition to the generic item construction principles in Appendix B, in mind when preparing the questions: 1

a. Provide clear, explicit directions / guidelines for answering the question so that the examinee understands what constitutes a fully ccrrect response.

Choose words carefully to ensure that the stipulations and requirements of the question are appropriately conveyed. Words such NUREG-1021 4 of 15 Interim Rev. 8, January 1997 i

J 1' 3 I l- Appendix C j i as+" evaluate," " outline," and " explain," can invite a lot of detail ! that is not necessarily relevant.

b. Make sure that the answer key response matches (and is limited to) l' the requirements posed in the question. When appropriate, indicate

. the amount of partial credit to be granted for an incomplete answer.

! For questions requiring computation, specify the degree of precision j expected. Try to make the answer turn out to be whole numbers. ! c. Avoid g\ving away part or all of the answer by the way the question j is worded. For example: l

"If the letdown line became obstructed, could boration of the plant

be accomplished shortly after a reactor trip to put the plant in j cold shutdown? If so, how?"

j A test-wise examinee c.an realize that the answer has to be yes, or else the second part of the question would have read something like l "If not, why not?"

d. Avoid what could be considered " trick" questions in which the answer

key does not precisely match the question. For example, asking "How l [do] the SI termination criteria change following a SI i reinitiation?" implies that the termination criteria will change, j when in actuality they do not.

1 e. Avoid direct look-up questions that only require the examinee to recall where to find the answer.to the question.

The operational orientation required of questions on the walk-

through test and the examinee's access to reference documents, argue

against the use of questions that test for recall and memorization.

{ Any questions that do not require any analysis, synthesis, or 4 i application of information by the examinee should be answerable i ! without the aid of reference materials. Refer to ES-602, l i- Attachment 1, for a more detailed discussion of direct look-up l j questions.  !

Form ES-602-1, "NRC Checklist for Open-Reference Test Items," contains a j list of questions that can be used to evaluate the suitability of the  !

! follow-up questions planned for the walk-through portion of the l operating test. Although the checklist was developed for use in evaluating requalification written examinations, most of the criteria j (i.e., all except 9, 10, 11, and the K/A rating on item 7) are l j generically applicable and could be used as a basis for revising or  ; j rejecting proposed walk-through questions.  ! i i I l NUREG-1021 5 of 15 Interim Rev. 8, January 1997  ; i i  ! e w---,2-w --y- rt- -twe- ----w- ,-%3 w a -,w a k--ge -,--.p- , , , , . - f- ....m -,. _- -...-,,--;,.y..,.--,-. _,--_.---,,-. ,

l 1 l l l Appendix C C. DEVELOPMENT AND USE OF ALTERNATE PATH JPMs JPMs are intended to be tasks that an operator must be able to perform that are related to their particular job task analysis (JTA). Often times, operators are challenged to perform auxiliary procedures when equipment or instrumentation fails during use. Therefore, examinees are expected to be able to use alternative methods to perform tasks. Alternative paths are evaluated during an examination by incorporating malfunctions to instrumentation or components that require the examinee to perform actions other than those performed when a system responds normally. JPMs where malfunctions occur are used to provide a methodology to evaluate whether an examinee has the skills and knowledge at the level needed to safely operate the system. This type of JPM, called " alternate-path," provides an excellent opportunity to observe the examinees execute alternative paths , within the wide spectrum of procedures under their cognizance that would not i otherwise be examined. All alternate-path JPMs should include the following  ! five characteristics:

1. Success Path - Each JPM should have a valid, facility-endorsed success path. This path may require analyzing initial conditions to determine an alternative method for completing the task, mitigating a system-related problem that occurs during the task, or realigning the system.  ;

2. Procedurally Driven - For each JPM, a procedure should address the actions that are required (i.e., if the JPM requires an alternative method to complete the task, the procedure would have an exit step that directs the use of this alternative method). The examinee may be required to use some common practices endorsed by the facility that are addressed through generic administrative procedures or policies (e.g.,

shifting controls to manual).

3. Loaical Seauence - The sequence of procedurally driven actions should be logical. For example, an examinee performing a normal evaluation when a malfunction occurs should not be expected to enter emergency operating procedures (EOPs). More realistically, the examinee would attempt to correct the problem by referring to an annunciator response procedure (ARP) or abnormal operating procedure (A0P). However, an examinee performing a normal evolution may encounter a situation requiring a reactor trip. The JPM should not contain a cascading sequence of malfunctions, for which several procedures must be used simultaneously, that occur while performing a ta;k. This type of activity is better tested in the dynamic simulator portion of the examination.

4. Independent of Crew Dynamics - Each JPM should allow the examinee to complete the task or mitigate a problem that occurs during a task without having to rely on the actions of other control room operators.

, This provision does not prohibit simulator operators from acknowledging NUREG-1021 6 of 15 Interim Rev. 8, January 1997

l l l 1 i Appendix C non-pertinent alarms or unexpected reactions of other systems not associated with the task. Also, the JPMs may still require the examinee I to use the simulator operator to perform needed manipulations in the  ; j plant.  : i

5. Validated in Advance - Each JPN should be validated before the examination begins and not changed after it begins. The JPM should not ,

j be a surprise to the examiners or the simulator operators. Each JPM i should be validated as early as possible before the examination is to be l administered to allow time for changes to be made. l D. WALK-THROUGH EVALUATION TECHNIQUES ) This guidance is intended to assist NRC examiners and facility evaluators in  !

administering JPMs by illustrating good and bad examples of walk-through

examination techniques.  ;

1. Providina Cues l l

l Cuing refers to the information provided to an examinee by an examiner ' when conducting a JPM. When conducting JPMs on the simulator, the i simulator provides most of the required cues. However, when conducting JPMs outside of the simulator, the examiner must provide realistic and timely information to the examinee.

a. Verbal Cues i

Many times verbal cues are required to provide relevant system information, such as valve position, meter deflection, or indicating light status. The examiner has to be careful to provide the examinee with the indications that should be readily observed (e.g.,

                      "the red light just illuminated," "the valve position indicator does not move"). An examiner can give too much information or inappropriate information (e.g., providing indications that are not visible or audible to the examinee) that could invalidate the JPM.

The examiner must keep in mind what the examinee would see and hear while performing the JPM, and provide consistent cues.

b. Non-Verbal Cues

Maintaining a " poker face" when an examinee provides an incorrect i response or performs the wrong procedural step is important. Voice inflections indicating something has been performed incorrectly,-or changing the manner in which cues are given (e.g., talking more  ;

methodically, or rapidly) are examples of non-verbal communications i that should be avoided. l i l NUREG-1021 7 of 15 Interim Rev. 8, January 1997 i l

Appendix C Thorough preparation and familiarity with the JPM is vital to providing proper cuing. Knowledge of what indications will be available and how they will respond to the examinee's actions allow an examiner to give accurate and timely cues when an examinee is incorrectly performing the task.

2. Evaluation Skills When evaluating an examinee, an examiner must have the ability to differentiate between what he knows or believes to be true about an examinee's ability and how the examinee actually performs on the JPM.

As previously discussed, an exami.1er must be familiar with the JPM to be able to accurately evaluate performance. Errors made by the examinee performing the JPM may not be seen or pertinent questions may not be asked if the examiner has not prepared for the examination. An examiner must remain attentive to the examinee's actions at all times. This will ensure the examiner provides timely cues and detects errors in performance.

3. Exam Administration While conducting the walk-through examination, the examiner must be aware of conduct that is appropriate for a trainer, but is inappropriate for an examiner. As a trainer, interacting with the examinee during the performance of the JPM to gain insight into what the examinee is thinking is a good practice. However as an examiner, this is distracting to the examinee and may inadvertently result in prompting or leading the examinee.

When conducting JPMs in the simulator, examiners should not manipulate any controls or silence / acknowledge alarms. The examiner must take a

       " hands off" approach to maintain the proper testing environment.

The examiner must be careful to shield any notes or grading from the examinee to prevent giving an indication of performance, which may either provide a false sense of security or increase stress levels. If an examinee's actions are not clear, the examiner must be prepared to ask appropriate follow-up or clarifying questions. Documenting these questions and the subsequent answers is important as they may have a bearing on an examinee's overall grade. E. ATTACHMENTS / FORMS Attachment 1, " Example Job Performance Measures" Form ES-C-1, " Job Performance Measure Worksheet" Form ES-C-2, " Job Performance Measure Quality Checklist" NUREG-1021 8 of 15 Interim Rev. 8, January 1997

l Appendix C Example Follow-up Questions Attachment 1 l { Question 1: 1 1 l K/A G2.1.32 (3.4/3.8); ability to explain and apply system limits and  !

precautions '

i Precaution E.5 of 2300-6 states, "HPCI flow indication on 1(2)-2340-1, . HPCI FLOW CONTROLLER, is indicated flow only, not system total flow OR , reactor injection flow." Explain what that means, and show me where to j read system total flow, and where to read reactor injection flow. Expected Anseer: This means that the controller indicates pump i discharge flow rate and provides input to turbine speed control [ref: i ILT 2300/High Pressure Coolant Injection p. 48]. There is no place i where one can read system flow nor reactor injection flow [ref: Facility

prep week]. On print M-46, the examinee should show that the flow .

instrument is after MIN FLOWS and LUBE OIL COOLING and GLAND SEAL

COOLING taps and before TEST RETURN, so one cannot be sure how much is i going back though the test return nor though the min flows and lube-oil

coolers.

l Thought process: This question is at the comprehension level' because l l the examinee will discuss the implications of the precaution by

combining HPCI system knowledge with the ability to read the prints.

i Question 2: K/A 272000A405 (2.3*/3.7*), ability to convert process radiation monitor l readings to off-site release rates; G2.1.33 (3.4/4.0), ability to

recognize entry-level conditions for technical specifications; G2.4.41 j (2.3/4.1), knowledge of the EAL thresholds and classifications Step B.I.b says to calculate the release rate per QEP 155-2 and verify l that the release rate is less than the Technical Specification LC0 and

the Unusual Event EAL. Is it? Assume the current conditions have l , existed for two hours.

l l Response: The operators must select either the GE, SPing, or the SA-9, j then use the appropriate section of QEP 155-S8 (they should use page 3

for the given conditions) to make the calculations.

1 i The UE EAL is >2 x OCDM for >= 60 minutes (ref QEP 200-T1 RU2) or 2.8 E+04 microCi/sec. The examinee may also note that since no yellow-boxed ! alarms on the 912-1 are illuminated, the TS release is not exceeded. Thought Process: This question is at the application level because the i examinee will have to execute QEP 155-2 and compare the results with TS t and emergency guidelines to determine an answer. E ls NUREG-1021 9 of 15 Interim Rev. 8, January 1997 i 1

1 Appendix C 2 Attachment 1 l Question 3: K/A 223001A209 (3.4/3.6), ability to predict impact of vacuum breaker malfunction and use procedures to correct, control, or mitigate the consequences; G2.1.33 (3.4/4.0), ability to recognize entry-level conditions for technical specifications The reactor is at power. You've just been notified that during vacuum breaker testing, one torus-to-drywell vacuum breaker is mechanically bound full-open, and the torus and drywell are equalizing in pressure. What must be done and why? Expected Response: Under the given conditions, the torus and drywell pressures will equalize and the drywell-suppression chamber differential pressure will not be maintainable at or above 1.20 psid as required by TS 3.7/4.7 A.6.a. TS 3.7/4.7 A.6.b requires the dp to be restored within 24 hours or be in STARTUP/ HOT STANDBY within the next 6 hours and COLD SHUTDOWN within the next 24 hours. However, the operability of the vacuum breakers TS 3.7/4.7 A.4 is MORE restrictive, requiring entry into TS 3.0/4.0, which requires HOT SHUTDOWN within 12 hours and COLD  ! SHUTDOWN within 24 hours. But since they failed during testing, TS I 3.7/4.7 A.6.d also applies; it requires the valves to be restored to  ! operable within 6 hours, otherwise 24 hours to COLD SHUTDOWN. So 6 hours to fix and 24 hours to COLD SHUTDOWN is correct. Thought Process: This question is at the application level because the examinee must determine the correct course of action. The examinee will have to recognize various conditions covered by the TS and apply the most limiting one. Note that this question is not a direct look-up because it does not direct the examinee to any particular reference or TS; even if the question were to say "per the TS," there are multiple attractive incorrect answers. Question 4: K/A 290003K401 (3.1/3.2), knowledge of control room HVAC design features and interlocks that provide for system initiation / reconfiguration The control room HVAC is in normal alignment on a spring day (Train A AHU running, outside air supply damper passing M. flow outside air). The toxic gas analyzer detects an ammonia spill and the control room HVAC responds as-expected. After the operator shifts the toxic gas sample point, a small but very smokey fire starts in the control room. How does the CRHVAC system respond? Specify which train is running and in what mode it's running and why. NUREG-1021 10 of 15 Interim Rev. 8, January 1997

i 1 Appendix C 3 Attachment 1 Response: Toxic gas causes trains A and B to go into recirculation mode ! and causes train 8 AFU to trip and isolate, although B normally is not i 1 running. The toxic gas signal is sealed in. The operator is then to i shift the toxic gas analyzer to C (located upstream of the control room l l train A return air fan and damper). The seal-in will cause the system , j NOT to shift when the smoke is detected, so the system will continue in J 100% RECIRC MODE.  ; Thought Process: This question is at the analysis level because it l requires the examinees to apply their CRHVAC system knowledge to I determine the correct plant response. It also requires the examinees to o analyze, possibly with the aid of prints, the system response and status . rather than simply recall memorized facts. Question 5: K/A 264000A209/A210 (3.7/4.1; 3.9/4.2*), ability to predict the impacts of a loss of AC power /LOCA on the emergency diesel generators The reactors were both at 100%. No surveillances were in progress. A tornado caused a loss of off-site power. A minute later, Unit I experienced high-drywell pressure (+3 psig). A minute after that, Unit [ 2 experienced low reactor water level (-60 inches). What is the 1/2 EDG response during each event? Resp,onse: First, the 1/2 EDG will close to whichever bus lost power first or to the one selected by keylock switches (these are normally off). When Unit I receives the LOCA signal, the 1/2 EDG will supply bus 13-1, switching from bus 23-1 if necessary. When the LOCA comes in on Unit 2, the 1/2 EDG trips off both buses, leaving them unloaded. Note that the operators are to place the keylock switch to ON for the unit that has a LOCA. If the operator does that, then Unit I will keep the EDG. l Thought Process: This question is at the analysis level because the l examinees must analyze the impact of each event and demonstrate that 1

           'they understand the implications of each. This question requires the examinees to apply their EDG system knowledge to determine the correct plant response. It requires the examinees to analyze, possibly with the aid of prints, the system response and status rather than simply recall memorized facts.                                                                               1 NUREG-1021                                  11 of 15           Interim Rev. 8, January 1997 4

Appendix C Job Performance Measure Form ES-C-1 Worksheet Facility: Task No: Task

Title:

Job Performance Measure No: K/A

Reference:

Examinee: NRC Examiner: Facility Evaluator: Date: Method of testina: Simulated Performance Actual Performance Classroom Simulator Plant READ TO THE EXAMINEE I will explain the initial conditions, which steps to simulate or discuss, and provide initiating cues. When you complete the task successfully, the objective for this job performance measure will be satisfied. Initial Conditions: Task Standard: Required Materials: General

References:

Initiating Cue: Time Critical Task: YES/NO Validation Time: NUREG-1021 12 of 15 Interim Rev. 8, January 1997

1 i 1 1

                                                                                         ' assuBP g     Appendix C                                                             2                                     Form ES-C-1

! PERFORMANCE INFORMATION (Denote critical steps with a check mark)  ! 4 I Performance step: h i Standard: i i Connent: ? j Performance step: 4 , 4 i Standard: Comment: { , Performance step: 1 > \ ! Standard: 4 i ! Comment: l Terminating cue: i 4 i 3 d i 1 i 1 1 T I NUREG-1021 13 of 15 Interim Rev. 8, January 1997 1 4

Appendix C 3 Form ES-C-1 VERIFICATION OF COMPLETION Job Performance Measure No. Examinee's Name: Examiner's Name: Date performed: Facility Evaluator: Number of attempts: 4 . Time to complete: Question Documentation: Question:

Response

Result: SAT or UNSAT Ecaminer's signature and date: _ NUREG-1021 14 of 15 Interim Rev. 8, January 1997

t l f Appendix C Job Performance Measure Quality Checklist Form ES-C-2  : l Every JPM should: l 4 .

1. be supported by facility licensee's job task analysis. l 2

2. be operationally important (meets NRC K/A Catalog threshold criterion of 2 (3 for requalification exams) or as  ;

j determined by the facility and agreed to by the NRC). l l 3. be designed as either SRO only, R0/SRO or A0/R0/SR0. l

4. include the following, as applicable: l l

                   .a.                  initial conditions

b. initiating cues i

c. references and tools, including associated procedures l d. validated time limits (average time allowed for completion) and specific designation of those JPMs that are deemed to be j time-critical by the facility operations department

e. specific perfomance criteria that include:

(1) expected actions with exact control and indication

! nomenclature and criteria (switch position, meter

reading), even if these criteria are not specified in j the procedural step

' (2) system response and other cues that are complete and correct so that the examiner can properly cue the

examinee, if asked

(3) statements describing important observations that

should be made by the examinee (4) criteria for successful completion of the task (5) identification of those steps that are considered

, critical l (6) restrictions on the sequence of steps l f. prescripted follow-up questions and answers to evaluate two ] K/A statements related to the system or task (initial examination only) i NUREG-1021 15 of 15 Interim Rev. 8, January 1997 5 a

l 1 [ ' APPENDIX D , SIMULATOR TESTING GUIDELINES - i i A. PURPOSE - j This Appendix provides a framework for preparing and evaluating simulator j scenarios to ensure they are of appropriate scope, depth, and complexity for i initial operator licensing and requalification examinations. The following elements are discussed in detail or attached for information: 1 - a basic procedure for developing new simulator scenarios

(Section B), including a description of the associated qualitative

! and quantitative attributes (Section C) and the critical task ,

methodology (Section D) the competencies in which reactor operators (R0s) and senior reactor '

j operators (SR0s) are expected to be proficient (Section E) the simulator security considerations that should be kept in mind j during scenario validation and administration (Section F) l l' selected examples of initial and requalification scenarios (Attachments 1 and 2)  ! Adhering to the concepts and guidelines discussed herein, in association with l , Cs 1 the specific criteria c.ited in ES-301 or ES-604, as applicable, will enhance ~ 4 V the consistency and validity of the dynamic simulator operating tests.  : j B. INTEGRATED SCENARIO DEVELOPMENT ) The major activities applicable to the development of dynamic simulator scenarios are summarized below. The instructions apply to both the initial l and the requalification examination programs, except as noted. Although they 1 are written from the perspective of new scenario development, the instructions ! should also be referenced, as necessary, when modifying existing scenarios for

reuse and while reviewing proposed scenarios for quality.

Identify Scenario Obiectiy.g1 1. l A scenario should explicitly identify its objectives. For a 2 requalification examination, these should come, in part, from the j facility's requalification training program objectives. However, 4 10 CFR Part 55 requires that the initial licensing and the annual l requalification operating tests be a comprehensive sampling of items (2) i through (13) listed in 10 CFR 55.45. Therefore, both tests should sample from all the operating skills and abilities required of an ! operator and the operating crew. Limiting the requalification , examination to topics covered in the requalification cycle is not

sufficient.

i I J NUREG-1021 1 of 42 Interim Rev. 8, January 1997 4 l 1

Appendix D The basic objective of a scenario should be to evaluate the operators' ability to respond to events that are most appropriately tested in a dynamic simulator environment. These are events that require the operators to demonstrate their knowledge of integrated plant operations, diagnose abnormal plant conditions, and demonstrate their ability to work together and to mitigate plant transients that exercise their knowledge and use of AOPs and E0Ps. Additionally, the scenario should require the operators (usually the SR0s) to utilize technical specifications (TS) and, for requalification examinations, to implement the emergency plan. The full range of competencies in which the operators must demonstrate proficiency during the simulator test are described in Section E of this Appendix. Briefly describe the objectives in the space provided at the top of Form ES-D-1, " Simulator Outline," or equivalent. Also enter an identifying number for the scenario and the name of the plant after which the simulator is modeled.

2. Select Initial Conditions Initial conditions must be established that will allow the scenario to commence realistically. The initial conditions should be representative of a typical plant status, with various components, instruments and annunciators out of service. Io have maintenance or surveillance activities in progress is realistic. All, some, or even none of these initial conditions may have a bearing on subsequent scenario events.

Initial conditions should be frequently changed, to prevent predictability of future events. The initial conditions should be varied among the scenarios and should include startup, low-power, and full-power situations. Briefly describe the initial conditions, including any items that should be addressed during the shift turnover, in the space provided at the top of Form ES-D-1, or equivalent.

3. Select and Document Events Once the initial conditions are established, select a sequence of events designed to attain the stated objectives. Section C discusses a number of qualitative and quantitative criteria that should be considered when selecting events. The specific requirements for each quantitative criterion are enumerated in ES-301 and ES-604, as applicable.

Each event should have or contribute to an objective, whether it is to evaluate the operators' knowledge of a recent system modification, evaluate their ability to respond to a safety-significant event, or assess their use of TS for a particular safety-related component. Uncomplicated events that require no operator action beyond the NUREG-1021 2 of 42 Interim Rev. 8, January 1997

i i , Appendix D acknowledgment of alarms and verification of automatic actions provide '. little basis for evaluating the operators' competence and should not be l included on the operating test unless they are necessary to set the

stage.for subsequent events.

The scenarios should be developed so that a variety of systems is

affected within each type of event (i.e., normal evolutions, instrument failures, component failures, and major plant transients). Having one j equipment failure cause or exacerbate another can be used to evaluate

the operators' understanding of system and component interactions.

Balancing the severity of events and the demands they place on each

operating position (e.g., RO and BOP) will allow each operator to demonstrate his or her competence across a range of conditions.

All events do not have to be linked, that is, one event need not occur ! for the next event to logically occur, although in many instances, such i a relationship adds to the credibility of the scenario. However, the i d scenario should not consist of a series of totally unrelated events. A well-crafted scenario should flow from event to event, giving the i- operators sufficient time in each event to analyze what has happened, l evaluate the consequences of their action (or inaction), assign a l priority to the event given the existing plant conditions, and determine a course of action. Exercise care that one event does not fully mask l the symptoms of another because the operators could overlook the

malfunction and cause the event or competency coverage for the scenario set to be deficient.

. Record each planned operation, malfunction, and transient on Form ES-D-1 ! and number them sequentially. Cross-reference each event to a simulator i malfunction number, if applicable, or briefly describe the simulator j instructions that must be entered. i For each event listed on Form ES-D-1, prepare a Form ES-D-2, " Operator i Actions," (or equivalent) by entering the scenario, event, and page numbers and a brief description of the event at the top of the form.

Each event description should include when it is to be initiated,

! whether by signal of the lead examiner / evaluator, time line, or plant j parameter. The form shall also identify the symptoms or cues that the operators will be provided, the expected actions to be taken, communications to be made, the references to be used by each operating position (e.g., the SRO, RO, and B0P operators) on the crew, and the event terminus (i.e., the anticipated point at which the examiners or evaluators will have enough information on operator performance to move on to the next event). Every expected operator action shauld be included on the form, particularly the critical tasks (refer to Section D) and other verifiable actions and behaviors that will provide a useful basis for NUREG-1021 3 of 42 Interim Rev. 8, January 1997

l I i l Appendix D evaluating the operators' competence. Critical tasks shall be flagged in a manner that makes them apparent to the individuals who will be administering the operating test (e.g., by using underlines, asterisks, or bold type). When possible, set points and other parameters should be  ; included to provide an objective method for evaluating the operators' performance. Statements such as " Performs actions in accordance with Procedure XXXXX" generally do not provide sufficient guidance and are inadequate. However, the statement " Performs actions of steps XXX of Procedure XXX (attached)" is acceptable. Requalification scenarios must indicate the crew critical tasks (cts) for each event and the expected operator actions required that meet the  ; criteria of Section D, " Critical Task Methodology." Although cts are  ! not specifically identified on simulator scenarios used for initial  ; operator licensing, the concept is still valid and should be used to assess the significance of the expected operator actions. Although the expected actions should, to.the extent possible, be listed , in chronological order, certain actions may be required throughout the  ! event (for instance, if a safety or relief valve fails open, the operators should continually monitor pressure and water level). Flag these actions to show that they are continuous. The expected actions on Form ES-D-2 should be widely spaced to leave room for notes to document the operator's performance during the , simulator test. The far-left column of the form should also be left blank so that it may be used to record the actual time at which key i actions occurred while giving the test.  !

4. Determine the Scenario Endooint The last operator action sheet (Form ES-D-2) in the scenario should specify the endpoint of the scenario by identifying a particular plant condition, procedural step, or other point that is clearly recognizable.

The scenario should not be terminated until the stated objectives have been achieved.

5. Validate the Scenario Every scenario should be validated to ensure that it will run as intended. If a previously validated scenario is being modified slightly, real-time validation may not be necessary. However, if there are major changes or if someone questions the validity, revalidation in real-time is recommended.

l NUREG-1021 4 of 42 Interim Rev. 8, January 1997 9I

 -- . - . -          - - - - - - . - -               -. -         ~ .-.. - - -              . - . - - - -

i i Appendix D C. SCENARIO ATTRIBUTES All valid scenarios contain common elements that make them useful as l 4 evaluation tools. A properly constructed scenario provides for an accurate l 1 test of each individual operator's skills and abilities as well as an opportunity to evaluate the crew members' team-dependent skills and abilities. , The scenario should be of sufficient scope and complexity to demonstrate the i difference between competent operators and crews and.those that are not performing at an acceptable level. It also should require that the crew i demonstrate its ability as a team to adequately protect the public health and i safety in emergency cos.'" ions, using the facility's emergency operating

procedures (EOPs).

I Scenario attributes can be characterized as both qualitative and quantitative.

No single qualitative or quantitative attribute or group of attributes can be used to determine the acceptability of a scenario. However, a trained examiner should be able to assess of the adequacy of a scenario or develop a new scenario, using both sets of attributes. This assessment, combined with validation of the scenario on a real-time basis, should be sufficient to determine if a scenario provides an acceptable tool to measure the competency of a crew and/or its individual members.

1. Qualitative Attributes

a. Realism / Credibility Introducing unrealistic or incredible events into a scenario can affect the validity of the scenario and provide negative training.

Piping, component, and instrument failures often occur in such a way that deterioration can be tracked over a discrete time period (e.g., , a small leak that propagates over time or a pump failure preceded by l a high vibration condition). Including such precursors into scenarios is important, where appropriate. A great deal of evaluative feedback can be obtained by observing how an operator and crew responds to a gradually worsening condition. A good technique inserts an event precursor (e.g., small steam generator tube leak) and maintains the plant at a slightly degraded condition to observe how the crew incorporates that condition into its conduct of subsequent plant operations. Although faults that occur with little or no warning (e.g., valve operators fail, fires occur in breakers or transformers, undetected pipe erosion results in piping failures) may be included in scenarios, they often provide minimal evaluative benefits because  ; they happen so suddenly that operators have little to do but watch  ! the event unfold. These events are most useful when trying to  ! establish a plant condition for subsequent evaluation goals or to NUREG-1021 5 of 42 Interim Rev. 8, January 1997

Appendix 0 assess an operator's or a crew's ability to use procedures in a symptom-based rather than an event-based mode. Mechanistic component failures are well documented events that occur each year, many times in multiple numbers. However, non-mechanistic failures (e.g., pipe breaks) generally occur singularly; therefore, unless there is a connective precursor, such as a seismic event, it would not be realistic or credible to have several piping systems fail during any one scenario. Simulated events that appear to violate the laws of physics and thermodynamics contribute to negative training and are to be avoided. Time compression techniques, which are discussed later, Aay contribute to negative training. However, if the intent of a scenario is to evaluate a crew's ability to execute procedural steps that may take a long time to reach during an event (e.g., hydrogen generation during a core uncover event), such a technique may be useful. However, the scenario must contain a cue that, when the indications for such events are detected by the crew, the crew is informed that the parameters are not responding as expected for the actual plant and that time is being compressed. This cue should be presented at the first opportunity that does not distract the crew from responding to available indications and before the crew challenges the validity of the indications. For example, in the first PWR scenario (Attachment 1), the cue should be given following the crew's determination that a reactor coolant system (RCS) feed and bleed may be necessary (per FR-H.1) but prior to steam generator levels requiring initiation.

b. Event Sequencing The sequence of events has a major effect in establishing the complexity of a simulator scenario. The pace at which malfunctions are entered can adversely affect the way an operator or a crew responds.

Malfunctions may be entered simultaneously at separate control panel locations if each event can be handled by an individual applicant and does not require extensive assistance. Too short a time between malfunctions may mask the effects of a particular malfunction and divert the operators' attention. This cuts short the observers' ability to evaluate the operators' response to the prior malfunction and may be prejudicial to a fair evaluation. Conversely, extending the time between malfunctions so that no operator activity is in progress may cause undue stress. During an examination, the operators expect something to occur; too much time between events should be avoided. NUREG-1021 6 of 42 Interim Rev. 8, January 1997

Appendix D i i , Therefore, insertion of malfunctions in the scenario should be carefully timed. Rigorously following a planned time sequence of  ; } events is often less valid than initiating malfunctions on the basis ' of plant parameters or operator actions. The appropriate sequencing  : l of events relates directly to the objectives of the scenario. Event sequencing may involve' time compression to speed up the i response of key parameters so that the scenario can proceed to the next event within a reasonable time. Time compression may be accomplished by adjusting parameter indications or accelerating i L plant behavior characteristics so that an event is triggered by  ; l plant indications more quickly than would typically occur in reality

(e.g., opening a drain path from a steam generator that is not noticeable to the operator so that the entry conditions for a loss of heat sink are reached.) This method is acceptable as long as the

, time compression allows the operators time to perform tasks that ! they would typically perform during the period in which time 1.s compressed. To avoid wasting the operators' time determining the ! validity of their indications, the crew should be informed before l the scenario begins that time compression may be used during an i event and debriefed after the scenario to minimize the potential for

negative training.

i Frequently, important evaluative benefit in terms of safety j significance is gained by having key components or instruments fail t after entering the E0Ps. This process compels the operators to j respond immediately to a safety-related situation by taking i alternate actions to mitigate the event. This process also allows i for a better evaluation of the operators' overall knowledge of plant procedures and systems because the event must be incorporated into the mitigation strategy for the remainder of the scenario. i c. Simulator Modeling i Despite the certification of simulators to a set standard i (ANSI /ANS-3.5), not all simulators are equally capable of performing major transients. The scenario should not exceed the limits of the facility licensee's configuration management system by altering a simulator model to obtain a desired effect. For example, increasing the post-trip decay heat input in order to maximize internal core  ! temperatures during a loss of cooling event is not appropriate; the simulator model should be allowed to perform as designed. The scenario may simulate events for which a simulator malfunction does not exist by using overrides or remote functions for local operator actions. An example would be failing indicators to simulate an inoperable component. NMEG-1021 7 of 42 Interim Rev. 8, January 1997 y-,- .-

Appendix D

d. Evaluating Competencies Each scenario set shall ensure that all of the rating factors within each competency can be evaluated and that, if the crew performs poorly, their performance could cause plant degradation or threaten the public health and safety. Therefore, events must be incorporated into the scenario that will allow an unsatisfactory evaluation of an operator or crew in a particular rating factor if they perform poorly. Scenarios that require little analysis or problem-solving and few operator actions may not provide an adequate basis to evaluate the required rating factors.

The individual competencies applicable to the R0 and SR0 license levels during initial and requalification examinations are described . in Section E; the rating factors within each competency are I identified in ES-303 (specifically, on Forms ES-303-3 and ES-303-4 l for R0 and SR0 applicants, respectively). The crew competencies l applicable to requalification examinations only are identified in l ES-604. ) i

e. Level of Difficulty l

The dynamic simulator operating test must discriminate between those examinees who have and those who have not adequately mastered the knowledge, skills, and abilities required to be licensed operators. I Simulator scenarios that are either too easy or difficult are not effective discriminators. In general, as the quantitative attributes, such as the number of malfunctions or critical tasks (discussed below), of a simulator scenario increase, the level of difficulty of the scenario will  ; increase as well. However, counting the number of scenario i quantitative attributes is not always indicative of the scenario's level of difficulty; two scenarios having the same quantitative attributes can vary significantly in level of difficulty. There are no definitive minimum or maximum attribute values that can be used to identify inappropriate scenarios that will not discriminate i because they are too easy or difficult. The two most important determinants of the level of difficulty of a simulator scenario are the amount of analysis and problem-solving and the number of operator actions required to mitigate the events in the scenario. Malfunctions that require analysis or problem-solving increase the level of difficulty, because they require the examinees to integrate a number of system conditions, evaluate their interrelationships, and take actions that demonstrate an understanding of the underlying knowledge. Scenarios that consist NUREG-1021 8 of 42 Interim Rev. 8, January 1997

  - ~ _ _ _ _ _ _ . _ _ _ _ _ . _ _ _ . _ _ . _ . _ _ _ _ _ _ . _ . _ _ _ _ _ _ _ _ _ _

1

l. Appendix D i 4 of a number of unrelated malfunctions that require little or no operator analysis or response are generally less challenging. ,

I

2. Quantitative Attributes Those traits discussed in the previous section provide for a qualitative assessment of the complexity of a simulator scenario. However, there are some characteristics of a scenario that can be quantified and that generally have a bearing on the complexity and level of difficulty of the scenario. These characteristics are described below, and a target range for each trait that is applicable to the initial and requalification examination is enumerated in ES-301 and ES-604, respectively. The ranges are not absolute limitations; some scenarios may be an excellent evaluation tool but may not fit within the ranges.

A scenario that does not fit into these ranges should be evaluated to ensure the scenario is appropriate.

a. Normal Evolutions Normal evolutions include activities such as a feed pump startup, turbine loading, generator synchronization, and reactivity manipulations, which include evolutions such as- a reactor startup or changing power with boron concentration, control rods, or core flow.

Reactivity manipulations are considered significant if they produce a clearly observable plant response, such as bringing the reactor critical from a substantially subcritical state, raising power to the point that reactivity feedback from nuclear heat addition is noticeable and a heatup rate is established, or changing reactor power manually with control rods or recirculation flow. Normal evolutions can be used as a backdrop on which to stage the emergency or abnormal situations. For example, a main feedwater control valve may fail passively (i.e., as is) before the operators conduct- a normal power change. Time consuming normal evolutions such as a power escalation from low power can provide an opportunity to evaluate the SR0's supervisory 4 or resource management skills. Events such as component or l instrument failures may be added to challenge the operators while ' continuing the power escalation. j Short surveillances (e.g., exercising safety rods or paralleling the i emergency diesel generator with the grid) may be used to examine the  ; operators' dexterity on the control panels or to involve operators , who are not engaged in other activities. i NUREG-1021 9 of 42 Interim Rev. 8, January 1997

l 1 Appendix 0

b. Total Malfunctions Total malfunctions are the number of instrument (e.g., nuclear, control, or process) and component failures (e.g., pump, motor, i valve, or pipe) used to initiate the events that constitute a  !

scenario, including those initiated after E0P entry (see Item C.2.c ' below) . To count as a separate malfunction, they must involve a ) significant system response and require operator action to correct.  ; For example, an anticipated transient without scram or trip  ! (ATWS/ATWT) is a single malfunction, regardless of how many instructions a simulator operator must program to produce it. Components that are placed out of service at the beginning of a scenario as part of the shift turnover conditions, and which the crew is made aware of, are not considered malfunctions. Component or instrument failures that require no operator actions or response do not count toward the recommended total number of malfunctions.

c. Malfunctions After E0P Entry Some malfunctions should result in vital instruments or components failing after the E0Ps have been entered (these may have been inoperable at the beginning of the scenario or before E0P entry) and i influence the operators' choice of mitigation strategy. For 1 example, failing a high head safety injection (SI) pump to start on a large-break loss-of-coolant accident (LOCA) does not affect the mitigation strategy; however, this would have an effect if it were the only available high head SI pump on a small-break LOCA.

d. Abnormal Events Each scenario should evaluate the operators' ability to implement abnormal operating procedures (A0Ps). An abnormal event may or may not be a precursor to the major transient (see Item C.2.e below),

although it can add to the credibility of a scenario, such as a preceding a total loss of feed water with a single feed pump trip. However, certain events may cue the operators about subsequent events. Therefore, if a scenario is derived from the facility licensee'r. bank, it is wise to vary or modify the precursor events that lead to the major transient. It is also good to insert abnormal events that are not always predictive of the same major transient (e.g., a steam generator tube leak does not always lead to a subsequent tube rupture), i Some abnormal events for each scenario should require that the operators recognize and interpret technical specifications. This . recognition and interpretation can also be incorporated into the NUREG-1021 10 of 42 Interim Rev. 8, January 1997

i' i i Appendix D i i scenario by designating TS-related equipment that is out of service  ! at the start of the scenario.  ! Components or instrument failures that occur following E0P entry do l not count toward the recommended total number of abnormal events. -

e. MajorTransients  !

A major transient is one that has a significant effect on plant i safety and that leads to an automatic (or manual, if initiated by an ' operator) protective system actuation, such as a reactor trip or an engineered safety system actuation. A single major transient that actuates more than one automatic protective system actuation will be  ; counted as a single major transient. Examples include loss of offsite power, LOCA, steam or feed line break, steam generator tube > rupture, and loss of feed water. A major transient should normally involve activation of the facility's emergency plan.

f. E0Ps Used A scenario that requires the operators to refer to many different E0Ps may not be as complex as a scenario for which only one E0P is O used, but which requires use of alternative decision paths and prioritization of actions within the E0P to deal with the situation.

Therefore, this attribute should reflect the E0Ps that have measurable actions that the crew must take. Moreover, the primary scram response procedure that serves as the entry point for the E0Ps is not counted. For BWRs, the number of "EOPs Used" should be counted consistent with the following four top level guidelines of the Emergency Procedures Guidelines: (1) RPV Control, (2) Primary Containment Control, (3) Secondary Containment Control, and Radioactivity Release Control. Use of multiple control sections of the above listed guidelines do not count separately as "EOPs Used." For example, use of RPV level control and RPV pressure control should be counted as one E0P Used - RPV Control.

g. E0P Contingency Procedures Used Contingency procedures are used when there is a challenge to a critical safety function or if plant conditions have become severely degraded. Therefore, using them in a scenario provides an opportunity to observe the operators attempt to execute a mitigation strategy that clearly has safety significance to the plant and the.

public health and safety. Each scenario set should require the operators to enter and perform safety-related tasks within an E0P contiagency procedure at least once. O NUREG-1021 11 of 42 Interim Rev. 8, January 1997 } i l _ ._ . _ _ _ _ _ . . _ . __ _ . _ - _ . _ - - _ .

l l l l l Appendix D ~ 1 The following list of contingency procedures is not unique or all-inclusive. Scenario developers and reviewers should consider it as

                                                                                   )

a set of general guides that may not fully apply to all scenarios.  ; (1) Westinahouse Optimal Recovery Procedures designated as Emergency Contingency Action (ECAs) procedures: Loss of All AC Power With or Without SI Required Loss of Emergency Coolant Recirculation LOCA Outside Containment Uncontrolled Depressurization of All Steam Generators Steam Generator Tube Rupture (SGTR) With Loss of Reactor Coolant-Subcooled Recovery SGTR With Loss of Reactor Coolant-Saturated Recovery SGTR Without Pressurizer Pressure Control Functional Recovery Procedures entered as a result of RED or ORANGE conditions on the Critical Safety Function Status Trees: Response to Nuclear Power Generation /ATWS  ! Response to Inadequate Core Cooling Response to Degraded Core Cooling Response to Loss of Secondary Heat Sink Response to Imminent Pressurized Thermal Shock Conditions Response to High Containment Pressure Response to Containment Flooding (2) Combustion Enoineerina Entry into Functional Recovery Procedures (FRPs)

                       . Transition among Functional Recovery Safety Function success paths Transition from one safety function to another within the FRPs (3)     Babcock and Wilcox (B&V)

The B&W E0P structure does not identify procedures that can be easily recognized as contingency procedures. However, use of the descriptions given above for Westinghouse contingency procedures provides guidance on the type of events to be considered. NUREG-1021 12 of 42 Interim Rev. 8, January 1997

i l Appendix D t

(4) General Electric

? Alternate Level Control . Emergency Reactor Pressure Vessel (RPV) l Depressurization Primary Containment Flooding l - Level / Power Control i - RPV Flooding j - Steam Cooling

h. Simulator Run Time l

i A scenario should be designed to run approximately 60 to 90 minutes. However, this does not preclude scenarios taking more or less time. l The nominal run time of 60 minutes may not provide sufficient time to conduct a scenario that progresses through several E0Ps or that i requires performance 'of fairly involved procedural steps. It is t possible to conduct very meaningful and involved scenarios in less ! time, but care should be taken not to place an undue burden on the

operators by initiating malfunctions at too rapid a pace. This i parameter is one of many that should be considered in assessing the y overall quality of a scenario, and as long as the scenario meets the other criteria stated in this guidance, the scenario run time is a j secondary concern.

i. E0P Run Time

, 1 i The time during which the operators are involved in E0Ps has a l stror.g relationship to the complexity of the scenario because most l critical tasks occur in the E0Ps and the actions the operators take i' have the most potential for affecting the health and safety of the public. Therefore, a significant percentage of the time a scenario is progressing should be spent in the E0Ps. Usually, more time is

required when contingency procedures are in effect, because it j ,

generally takes some time for the plant to degrade to a point where ! critical safety functions are jeopardized. However, operators

should be evaluated in E0P activities beyond the point at which an

event is diagnosed and initial mitigation actions are taken. Many l i of the actions taken to stabilize the plant and recover from a j transient are safety significant. Therefore, scenarios should be i

! allowed to progress so that these operations can be observed.  : i i Scenarios should not be just E0P oriented. Valuable assessments can

be made within AOPs with the plant at power because of the level of  !

safety significance associated with transients in these conditions. ' 1

NUREG-1021 13 of 42 Interim Rev. 8, January 1997 i

t

Appendix D

j. Crew Critical Tasks Critical tasks range between fairly simplistic but safety-significant tasks (starting the standby liquid control system on an anticipated transient without scram condition or tripping a reactor coolant pump during a small-break LOCA) and other tasks that require a much higher level of skill involving several crew members  ;

(executing a rapid cooldown within predefined limits using steam i generator power-operated relief valves or using low pressure - injection systems to maintain the vessel level while cooling the suppression pool). Therefore, the difficulty level must be considered to judge the appropriateness of the number of cts in a i scenario or scenario set. I 1 Refer to Section D for a detailed explanation of the critical task methodology. D. CRITICAL TASK METHODOLOGY The requalification examination uses critical tasks (cts) for evaluating crew l performance on tasks that have safety significance to the plant or the public. l The cts are objective measures for determining whether an individual's or a l crew's performance is satisfactory or unsatisfactory. Although cts are not I directly used to evaluate operator performance on the initial licensing examinations, they are used to provide a basis for the individual operator competency evaluations because they help the examiner focus on those tasks that have a significant impact on the safety of the plant or the public.  ! Refer to ES-303 and ES-604 for specific instructions on the use of cts in grading initial and requalification examinations.

1. Identification of Critical Tasks A critical task must include the following elements:

a. Safety Significance In reviewing each proposed CT, assess the task to ensure that it is essential to safety. A task is essential to safety if the improper performance or omission of this task by an operator will result in direct adverse consequences or in significant degradation in the mitigative capability of the plant.

If an automatically actuated plant system would have been required to mitigate the consequences of an individual's incorrect performance or the performance necessitates the crew taking r.ompensatory action that would complicate the evant mitigation strategy, the task is safety significant. NUREG-1021 14 of 42 Interim Rev. 8, January 1997

i i i Appendix D I l Examples of cts involving essential safety actions include those for which operation or correct performance prevents the following: j degradation of any barrier to fission product release degraded emergency core cooling system (ECCS) or emergency power capacity l. !~ - a violation of a safety limit a violation of the facility license condition  ! l incorrect reactivity control (such as failure to initiate , emergency boration or standby liquid control, or manually )' insert control rods) a significant reduction of safety margin beyond that , irreparably introduced by the scenario  ! Examples of cts involving essential safety actions include those for I which a crew demonstrates the following abilities:- O - effectively direct or manipulate engineered safety feature V (ESF) controls that would prevent any condition described in the previous paragraph recognize a failure or an incorrect automatic actuation of an ESF system or component take ..e or more actions that would prevent a challenge to  ! plai.s safety prevent inappropriate actions that create a challenge to  ! plant safety (such as an unintentional reactor protection system (RPS) or ESF actuation)  !

b. Cueing For a CT to be valid, an external stimulus prompts at least one operator to perform the task. A cue prompts the operators to respond by taking certain actions and provides the initial conditions.. The cue need not indicate the task as " critical."

Appropriate cues include the following examples: verbal direction by or reports from other crew members NUREG-1021 15 of 42 Interim Rev. 8, January 1997 N l

l i l Appendix D procedural steps, such as satisfying entry conditions, flow chart decision points, and " response not obtained" columns indication of a system or a component malfunction (including passive failures) by meters or alarming devices

c. Measurable Performance Indicators A measurable performance indicator consists of positive actions that an observer can objectively identify taken by at least one member of the crew.

The NRC and facility licensee should review each critical task to ensure it is objective. For example, "If pressure falls below 1400 psi, start pump xyz," is a performance measure that is not 1 objective. The operator performing this task could conceivably start the pump when pressure reaches zero psi and still not violate the performance measure stated in the procedure, even though the facility licensee expects the operator to start the pump sooner. The NRC and facility licensee should agree in writing that the l limits for each CT are acceptable before the requalification  ! examination begins. For the example given above, adding an  ! acceptable pressure tolerance (e.g., within 200 psi) would clarify i the standard of performance that is expected. Measurable performance indicators include the following examples: actions taken as the result of transitioning in emergency operating procedures (EOPs), for example transitioning to and performing the actions required in FR-S.1 if the reactor does not trip (Westinghouse), or performing an automatic depressurization after confirming indications of high suppression pool temperature (General Electric) control manipulations such as a manual reactor irip or the start of an ECCS pump verbal reports or notifications of abnormal parameters or conditions such as "all control rods are not inserted" or

                      " containment pressure is greater than 2 psi" The following are examples of performance indicators that cannot be measured objectively during a simulator scenario:

understanding, such as of the significance of a certain plant response verification that an expected response has occurred NUREG-1021 16 of 42 Interim Rev. 8, January 1997

( Appendix 0 passive observations such as monitoring the performance of a system

d. Performance Feedback Each CT must provide at least one member of the crew with performance feedback. The feedback provides the crew member with information about the effect of the crew's actions or inaction on the CT. This requirement must be met for all cts.

4

2. Critical Tasks as " Generic" Safety Tasks Avoid assigning the "CT" designation to generic tasks that have safety significance but that do not meet all of the criteria required to identify a critical task.

Although a crew is not performing optimally if it fails to anticipate an automatic action given sufficient time to assess plant behavior, crew members are not required to anticipate an automatic action. A crew member may, at any time, take manual action in advance of an automatic action if, in the crew member's judgement, manual action is needed to place the reactor in a safe condition. If an operator takes an action Og that the examiners did not expect, the exam,iners must further evaluate Q the individual's rationale for taking those actions. This preemptive action may indicate a misunderstanding of plant conditions or a weakness in integrated plant knowledge that should be clarified with follow-up questions. Taking manual control of an automatic safety system qualifies as a CT only if the auto-initiation feature fails to work. It is then safety significant for the crew to take manual actions, as plant conditions l clearly indicate that an automatic action should have occurred and did ' not. Moreover, during scenario development and validation, i identification of cts is based on those actions which, if performed ' incorrectly or omitted, degrade the mitigation strategy needed in the scenario. If the manual system has also failed and no action will be , effective, this should not be identified as a CT. However, if an l operator or the crew significantly deviate from or fail to follow l procedures affecting assurance of basic safety functions, those actions may fom the ba' sis of a CT identified in the post-scenario review. Experience has shown emergency event classification to be an important evaluation area, but generally not a critical task. The argument is made that an incorrect classification could adversely affect the public health and safety if the appropriate instructions are not given to public service agencies in a timely manner. If a misclassification NUREG-1021 17 of 42 Interim Rev. 8, January 1997

l 1 Appendix D i occurs, the emphasis for corrective action is piaced on the facility licensee and an apprupriate period allotted for implementation of the corrective actions. , Therefore, although emergency classificat ion is still an area that is to , be evaluated, it should not receive the weight of a critical task. If a misclassification occurs, the examiners should determine the rationale used to establish the classification in order to determine if the crew l understood the status of the plant and incorporate into the program ' evaluation those pertinent corrective actions deemed appropriate. If a widespread problem during a program evaluation is observed, the examiner should share this information with other inspection program managers. E. COMPETENCY DESCRIPTIONS

1. Reactor Operator

3. Understand and Interpret Annunciators and Alarm Signals This competency involves the ability to notice and acknowledge  :

alarms. It includes the abilities to prioritize one's attention in  ! keeping with the severity and importance of annunciators and alarm signals and to correctly interpret and verify that signals are consistent with plant and systen conditions (with the use of a1 arm response procedures, as appropriate). This competency deals strictly with the understanding and interpretation of annunciators and alarm signals and, therefore, does not include knowledge of, or the ability to diagnose, overall plant and system status on the basis of other indications.

b. Diagnose Events and Conditions Based on Signals and Readings This competency involves the ability to accurately and promptly ,

recognize and analyze off-norma 1 trends and diagnose piant ' conditions to guard against and mitigate conditions that are out of specification. It includes the use of control room reference - materials, such as prints, books, and charts, to aid in the diagnosis and classification of events and conditions. It does not include knowledge of system operation, such as set points, interlocks, or automatic actions, or the understanding of how one's actions affect the plant and system conditions.

c. Understand Plant and System Response This competency involves knowledge of system operation, including set points, interlocks, and automatic actions. It includes the ab111 ties to locate and interpret piant and system instruments and to understand how one's actions affect plant and system conditions.

NUREG-1021 18 of 42 Interim Rev. 8, January 1997 i i

 -   _.-_ _ _._._ _                            _ _ ._. - ~ _ _ _ _ _ _ . _ _ _ _ _.                                  _ _ _ _

Appendix D It does not include the ability to notice or attend to annunciators and alarm signals or the ability to diagnose or classify events and conditions on the basis of control room indications.

d. Comply With and Use Procedures and Technical Specifications This competency involves the ability to refer to and comply wf th normal, abnormal, emergency, and administrative procedures in a timely manner (i.e., in sufficient time to avoid adverse impacts on plant status). It includes the ability to recognize emergency operating procedure entry condftions, carry out ianediate actfons without assistance, and recognize and comply with required 7/miting condf tfons for operatfon and action statements.

e. Operate the Control Boards This competency involves the ability to locate and manipulate controls to attain a desired plant and system response or condition.

It includes the ability to take manual control of automatic functions, when appropriate.

f. Communicate and Interact With Other Crew Members This competency involves the ability to provide and recef ve pertinent information, both oral and written (e.g., log entries).  !

It includes the ability to carry out supervisory instructions and to  ; fateract with other crew members with respect to conditions affecting safe plant operation regardless of which applicant's control board is directly affected.

2. Senior Reactor Ooerator

a. Understand and Interpret Annunciators and Alarm Signals This competency involves the ability to notice and attend to alarms.

It includes the ability to prioritize one's attention in keeping with the severity and importance of the annunciators and alarms and the ability to correctly interpret the significance of each alarm and verffy that it is consistent with plant and system conditions (with the use of alarm response procedures, as appropriate). This competency deals strictly with the understanding and interpretation of annunciators and alarm signals and, therefore, does not include knowledge of, or the ability to diagnose, overall plant and system status on the basis of other indications. ! O NUREG-1021 19 of 42 Interim Rev. 8, January 1997

1 1 Appendix D l

b. Diagnose Events and Conditions Based on Signals and Readings l This competency involves the ability to diagnose plant conditions to  :

guard against and mitigate conditions that do not meet i specifications. It includes the Ibility of both the supervisor and l the crew to recognize and analyze off-normal trends in an accurate , and timely manner and to use control room reference materf als, such ' as prints, books, and charts, to aid in diagnosing and classifying , events and conditions. It does not include knowledge of system  ! operation, such as set points, interlocks, or automatic actions, or ' the understanding of how one's actions affect the plant and system conditions,

c. Understand Plant and System Response This competency involves knowledge of system operation, including I set points, interlocks, and automatic actions. It includes the ability to locate and remain attentive to control room indicators, interpret those indicators to verify the status and operation of systems, and understand how one's actions and directives affect )

plant and system conditions. It does not include the ability to notice or attend to annunciator and alarm signals or to diagnose or classify events and conditions on the basis of control room i indications. i

d. Comply With and Use Procedures l This competency involves the ability to refer to and comply with normal, abnormal, emergency, and administrative procedures in a j timely manner (i.e., in sufficient time to avoid adverse impacts on plant status). It includes the ability to use procedures correctly and ensure correct implementatfon by the crew,

e. Operate the Control Boards This competency involves the ability to locate and manipulate controls to attain a desired plant and system response or condition.

It includes the ability to take manual control of automatic functions, when appropriate.

f. Communicate and Interact With the Crew and Other Personnel This competency involves the ability to provide and receive pertinent information in a clear, easily understood manner. It inc1udes the ability to keep crew members and personnel outside the control room informed of plant status.

NUREG-1021 20 of 42 Interim Rev. 8, January 1997

\ . l l l l Appendix D l ' i g. Direct Shift Operations I i This competency involves the ability to take timely and decisive  ! actions in response to problems during both normal and off-normal i situations. It includes the ability to provide timely and well  ! thought out directions that indicate concern for safety; to i encourage a team approach to problem solving and decision making by 1 soliciting and incorporating feedback from members of the crew; and i to remain in a position of oversight to maintain the " big picture." i

h. Comply With and Use Technical Specifications This competency involves the ability to recognize when conditions are covered by technical specifications. It includes the ability to locate the appropriate technical spacification and ensure correct compliance with any limiting conditions for operation and action l statements.

F. SECURITY CONSIDERATIONS FOR SIMULATOR OPERATING TESTS Simulators present a unique set of integrity concerns during the development and administration of operating tests. NRC examiners and facility licensees O should be aware of the simulator's vulnerabilities and take appropriate measures to ensure that operating test security is maintained in three areas: y/ the instructor station, the programmers' tools, and the external interconnections. Because facility licensees are more familiar w h their simulator's unique capabilities, limitations, and vulnerabilities than the NRC examiners are, it is expected that the licensees will take responsibility for determining and implementing whatever measures might be necessary to ensure the integrity of the operating tests. Most of the instructor station features can be checked through the tableau or l graphic interface provided at the instructor's console. The programmers' l tools and the external interconnections are not generally apparent to the instructor or the examiner. The simulator staff should be consulted to determine the status of those items. i

1. Instructor Station Features Snapshots - All simulators have snapshot capability. Initial 4 t

conditions (ICs) are recorded for future recall. 1 Backtrack - Backtrack files are snapshots that are automatically  ; recorded at pre-determined intervals, usually up to I hour of ' operation at intervals as frequent as 1 minute. Backtrack files are usually only accessible through the BACKTRACK feature. The files , typically cannot be erased, only overwritten by real-time operation. O- NUREG-1021 21 of 42 Interim Rev. 8, January 1997 i j i

i l Appendix 0 l Replay / Playback - The replay / playback feature steps through a series of snapshots and displays the I/O status (lights, meters, etc.) for i each sequentially. Often, the replay feature uses the backtrack ' files, although sep rate replay file storage may be provided.  ; Scripts / Computer Assisted Exercises - Many simulators have a feature that allows pre-programmed implementation of malfunctions and remote l functions based on time and/or logical conditions. Scripts may be used by the simulator staff to facilitate scenario administration, Scripts can typically be stored for future use. Stored scripts can also be selected for review and editing from the instructor station. Initial Conditions Summary - Snapshots are usually labeled on the instructor station IC menu with date/ time recorded, pertinent plant parameter status, and instructor comments. Even if the comment  ! field has been changed to indicate that a snapshot is available for l re-use, the data (scenario initialization) may still be i representative of test conditions until the snapshot is actually overwritten or updated. ) i 1

          #alfunctfon Suasary - Malfunction summary menus display the status                                          j of selected malfunctions, both active and inactive. The malfunction                                         i summary is usually IC dependent and therefore depicts the                                                   i malfunctions that were active or staged when an IC, such as a                                               !

scenario validation, was stored. l Monitored Parameters - Instructors are afforded the capability to define individual or groups of parameters for display or printout. The monitored parameter group assignments can be recalled for review and editing. If used to facilitate scenario validation or examination administration, the monitored parameters can provide 1 insight into the focus of the examination. l Trend Recording - Groups of parameters can be defined and assigned to trend recorders. The recorders may be, but is not necessarily, located at the instructor station. The recording may also be in file format for presentation on instructor station screens. Recording sessions are typically activated or de-activated at the instructor station. Student Performance Konitoring - Specia1 groups of parameters and simulated plant operating conditions can often be assigned to a tracking and recording function that plots an individual student's performance during training exercises. Recording sessions are typically activated or de-activated at the instructor station. NUREG-1021 22 of 42 Interim Rev. 8, January 1997

m __ _ ._. . _ _. _ . _ . _ _ _._. _ __. . .. _ .. _ . _ _ _ _ _ _ ._. _. _ _ _ _ _ _ _ l Appendix D l Video and Audio Recording - Many simulators are equipped with video  : and audio recording capability in the control room. Video and audio controls are typically located at the instru: tor station. j 2.. Proar rs' Tools Software TersInals - Simulator engineers have access to real-time monitoring and control of simulator and model conditions through software support terminals. These terminals may be located in the computer facility or at the engineer's desk. Independent Executives - The conditions for scenarios can sometimes be replicated off-line using independent executive programs. These programs should not be in communication with the I/0. Independent executives and their associated initialization files may provide an indication of planned exercises if they have been used to resolve problems during scenario validation. Graphical User Interfaces - Instructor station graphical user interfaces often display simulated plant conditions and performance in real-time. At remote locations, such as a programmer's desk, the i GUI could display the full scenario.

3. External Interconnections ESF Feeds - Many simulators have data links to the ESF and the operations management offices for emergency' planning drills. These links can display simulated plant condition to observers outside the simulated control room during scenario validation or examinations.

Remote Plant Process Computer and Instructor Statfon Screens - l Repeater screens in the training area can display scenarios in real l time to observers outside the simulated control room.

                                                                                                                                    ~

Modens and Remote Sinulator Support Systens - Many simulators are equipped with modems from the instructor station or simulation computers for outside monitoring and control of simulator status and activities by parties off site. E. ATTACHMENTS / FORMS Attachment 1, " Example Initial Dynamic Simulator Scenarios" Attachment 2, " Example Requalification Dynamic Simulator Scenarios" Form ES-D-1, " Scenario Outline" Form ES-D-2, " Operator Actions" i 4

NUREG-1021 23 of 42 Interim Rev. 8, January 1997

Appendix D Example initial Dynamic Simulator Scenarios Attachment 1 Facility: PWR Scenario No.: 1___ Op-Test No.: 1 Examiners: Operators: Obiectives: To evaluate the applicants' ability to implement the AOPs for a SG tube leak and a loss of charging, to reduce power per GOP-3, and to execute the EOPs for a loss of all AC power (ECA O.0), loss of coolant (E-1), and a SG tube rupture (E-3), with a success path via ES-3.1, post-SGTR cooldown using backfill. Initial Conditions: IC-38; 100% power, middle of life; CCP "B" is running: Unit 2 is in Mode 5. Tumover The following equipment is out of service: DG "A" (6 hrs); CCW pump "A" (2 days); VCilevel transmitter LT-185; the block valve for PORV 456 is inoperable with power removed; MFP "A;" and AFW pump "A" (30 hrs). All required surveillances have been done. A severe thunderstorm warning is in effect. Event Malf. Eunt Event No. No. Type' Description 1 XXXX, C(RO), 70 opm tube leak on "A" SG (ramped over 5 min) with running XXXX N(BOP), CCP trip and failure of standby pump to start; requires power R(RO) reduction 2 XXXX l(RO) pressurizerlevelinstrument L-459 fails low 3 XXXX C(ALL) instrument bus 112 inverter failure 4 XXXX, M(ALL), 450 gpm tube rupture on "A" SG (ramped over 3 min) with an XXXX f(BOP) "A' SG pressure transmitter failure causing the PORV to open 5 XXXX, M(ALL), concurrent failures of the station auxiliary transformer and the XXXX, "B" DG result in a loss of all AC power; power remains XXXX available through Unit 2 C(BOP) TDAFW pump trips on overspeed (can be reset) (N)ormal, (R)eactivity, (1)nstrument, (C)omponent, (M)ajor , l Note: The scenarios in this attachment are individual examples; they are not intended to represent j complete scenario sets / operating tests. ' For each of the planned events, enter on a Form ES-D-2 (or equivslent) a description of the event and detailed actions required by the applicable plant procedures (e.g., normal, i abnormal, emergency, and administrative, including the TS and emergency plan) for each I operating position (i.e., SRO, RO, BOP) in a manner similar to the first event on the next page. NUREG-1021 24 of 42 Interim Rev. 8, January 1997 l l

_ m. - _ . _ . . _ _ _ . _ . _ _ _ - . _ _ _ _ . _ . _ . _ . . _ _ . . ._._. . _ -_.___ - _ _._. _ _ l O Appendix D 2 Attachment 1

                                                                                                                                                   ~

1 Op-Test No.: _1_ Scenano No.: _1_ Event No.: _1_ Page _1., of _5_ Event Descripte: A 70 opm tube leak on the "A" SG (ramped over 5 minutes), combined with i a trip of the running CCP and a failure of the backup CCP to start, forces a reduction in power l because RCS leakage exceeds TS limits. Time Posite Applicant's Actions or Behavior RO/SRO/ BOP Recognize indications of the tube leak on the "A" SG -

                                                   - air ejector off gas radiation monitor l                                                   - steam line radiation monitor
                                                   - charging / letdown mismatch
                                                   - SG blowdown radiation monitor SRO                 Direct RO/ BOP actes per AOP-1.2 -
                                                   - monitor and control pzr level & pressure
                                                   - monitor and control VCT level
                                                   - verify leakage greater than TS limit announce possible high radiation in turbine bldg
                                                   - venfy tube leak with SG samples

! - have health physics verify release calculation l - commence unit shutdown

                                                   - notdy NRC
                                                   - minimize secondary contamination
                                                   - classify the event per the EPIPs (unusual event)

RO/ BOP Execute AOP actions per SRO directions SRO/RO Recognize running CCP tripped -

                                                   - no charging flow
                                                   - pump tripped light
                                                   - various charging / letdown annunciators l

l SRO May direct RO/ BOP per AOP-1.3 -

                                                   - isolate letdown
                                                   - monitor pressurizer level and pressure
                                                   - start the standby CCP l                                                   - reestablish letdown l                                                   - refer to TS 3.8.1
                                                   - initiate repairs SRO                 Supervise / coordinate power reduction -
                                                   - review precautions in GOP-3
                                                   - ensure delta-1 maintained within limits
                                                   - verify load reduction rate RO                  Coordinate with BOP to initiate power reduction -
                                                   - review GOP-3 precautions l                                                   - calculate / estimate boration required for shutdown i
                                                   - contact load dispatcher
                                                   - borates and/or inserts rods to maintain T-ave within 5F of T-ref and maintains delta-1 within limits BOP                 Coordinate with RO to initiate power reduction -
!                                                  - review GOP-3 precautions j                                                   - operate turbine controls to maintain unloading rate i

l NUREG-1021 25 of 42 Interim Rev. 8, January 1997

Appendix D 3 Attachment 1 Facility: PWR Scenario No.: 2__ Op-Test No.: 2 Examiners: Operators: Obiectives: To evaluate the applicants' ability to implement the AOPs for a loss of CCW to the RCPs, loss of reactor makeup control, and the loss of an emergency bus; to initiate a normal plant shutdown; and to executa the EOPs for a steam line break in containment with an ATWS (FR-S.1) and a subsequent loss of all feed flow (FR-H.1) requiring bleed and feed operations. Initial Conditions: IC-20; approximately 100% power,218 ppm boron (EOL), equilibrium xenon; bank "D" rods are at step 216 Turnover: The operations department is making preparations to shut down the plant due to equipment problems. Train "B" CSS logic failed an actuation test last shift; the LCO for TS 3.3.2 was entered 2 hrs ago; I&C is working on the problem. MDAFW pump "B" is out of service to repair an oil leak and should be back in about 45 min. The block valve for PORV 445A is closed and deencrgized for leakage control. Event Malf. Event Event No. No. Type

• Description 1 XXXX, l(BOP) spurious containment spray actuation, phase "B" isolation, and XXXX CSS pump "A" failure to auto start (reset malf to allow i equipment restoration and before required stop of RCPs) I 2 N/A N(BOP), begin normal shutdown due to CS problems i R(RO) ,

1 3 XXXX C(RO) boric acid filter plugged (100% in 1 min) at start of boration; when asked, filter d/p is 80# (remove when backflushed) 4 XXXX l(RO) narrow range RCS temperature detector fails high 5 XXXX, C(BOP) emergency bus 1 A-SA normal feeder breaker trips and DG "A" XXXX breaker trips 2 min later 6 XXXX, M(ALL), "A" SG line break in containment with auto Si on high XXXX, containment pressure but failure of reactor and turbine trip; XXXX, C(BOP), the local manual breaker is operable and the turbine will XXXX C(RO) follow; TDAFW pump overspeed on Sl; PORV "B" failure to open in auto or manual (N)ormal, (R)eactivity, (1)nstrument, (C)omponent, (M)ajor For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event and detailed actions required by the applicable plant procedures (e.g., normal, abnormal, emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event for the first PWR scenario (page 2 of this Attachment). NUREG-1021 26 of 42 Interim Rev. 8, January 1997

0 Appendix D 4 Attachment 1 Facility: BWR Scenario No.: 1__ Op-Test No.: 1 Examiners: Operators: i l Otnectives: To evaluate the applicants' ability to raise and subsequently lower reactor power in response to a TS 3.0.5 assessment; to implement the AOP for a loss of UPS; to execute the , EOPs for a turbine trip on high vibration with a failure of all control rods to insert and a loss of j RPIS; to determine power by alternate means due to APRM failure; and to control pressure with SRVs due to the inoperability of the bypass valves. l . Initial Conditions: IC-11; approximately 90% reactor power at dispatcher request; at power for 28 days, beginning of cycle; core spray pump 2A is out of service to replace a breaker closing l coil; APRM F failed downscale last shift and is bypassed Turnover: Raise power to 100% when contacted by dispatcher; test core spray pump 2A when the clearence is lifted (imminent) Event Malf. Event Event No. No. Type' Description

     ,           1           N/A        R(RO)         raise reactor power to 100% upon load dispatcher's request 2           XXXX       N(BOP),       test core spray pump 2A starting at step 7.9.2 of PT-07.2.4a i                                        C(BOP)        and respond to the motor overload 3           XXXX       C(SRO)        individual bus breaker failure (MCC DGD), requiring DG #4 to i                                                      be declared inoperable and a plant shutdown per TS 3.0.5 4           XXXX       l(RO),        UPS inverter 2A malfunction and loss of UPS (no APRMs, rod C(BOP)        positions, or rod control) 5           XXXX       C(BOP)        turbine bearing #3 vibration alarm I

6 XXXX, M(ALL), turbine trip and reactor scram with very few rods inserted 4 XXXX, (SLC pump 2A will trip after initiation and the scram discharge  ! XXXX, volume vents and drains fail to reopen when RPS is reset) ' i XXXX j bypass valves fail closed after turbine coasts down (no UPS)

C(ALL) 4 (N)ormal, (R)eactivity, (1)nstrument, (C)omponent, (Miajor 1 For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event i and detailed actions required by the applicable plant procedures (e.g., normal, abnormal,

,             emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event for the first PWR scenario (page 2 of this
   ;          Attachment).                                                                                                      l s      NUREG-1021                                            27 of 42           Interim Rev. 8, January 1997 1

I 1 Appendix D 5 Attachment 1 Facility: BWR Scenario No.: 2,_ Op-Test No.: 2 Examiners: Operators: I l Obiectives: To evaluate the applicants' ability to lower reactor power in accordance with plant procedures; to respond to an SSW pump failure with TS implications (EDG operability); diagnose and respond to a feedwater master controller failure with rising RPV level; respond to I a loss of an ESF bus and the running CRD pump; and implement the EOPs and an emergency depressurization in response to a loss of service transformers, failure of the second EDG to start, and a recirculation loop break.  ! l Initial Conditions: IC-17; 100% reactor power; B CRD pump is in service l Turnover: The load dispatcher has asked that power be lowered to 70%, and chemistry requests an SSW surveillance to be run at the beginning of the shift. Event Malf. Event Event 1 No. No. Type

• Description I 1 N/A R(RO) decrease power to 70%

2 XXXX N(BOP), perform SSW surveillance per chemistry request; SSW pump B l C(BOP) will trip shortly after start  ; 3 XXXX l(RO) feedwater master controller fails as is 4 XXXX C(BOP) loss of power to Division 2 ESF bus 5 XXXX, M(ALL), 1.5 minutes after event 4, the service transformers lock out, XXXX, C(BOP), the Division 1 EDC fails to start, and a 5% recirculation loop XXXX M(ALL), break develops in the drywell C(BOP) 30 seconds after initiating, the high pressure core spray pump trips . I (N)ormal, (R)eactivity, (l)nstrument, (C)omponent, (M)ajor i l For each of the planned events, enter on a Form ES-D-2 (or equivalent) a description of the event and detailed actions required by the applicable plant procedures (e.g., normal, abnormal, I emergency, and administrative, including the TS and emergency plan) for each operating position (i.e., SRO, RO, BOP) in a manner similar to the first event for the first PWR scenario (page 2 of this Attachment). NUREG-1021 28 of 42 Interim Rev. 8, January 1997

  ~

Appendix D Exanple Requalification Attachment 2 . Dynamic Simulator Scenarios The following are two PWR and two BWR simulator scenario outlines that can be used for reference when developing or reviewing requalification examinations. ! P R SCENARIO ONE - LOSS OF HEAT SINK Scenario Ob.iectives Evaluate the operators in their use of the " Loss of Heat Sink" procedure, FR-H.l. Evaluate the crew in performing a "bir:ed-and-feed" sequence, using

reactor head vents and pressurizer vents.

Scenario Summary i j Initial Conditions: 75 percent power "B" auxiliary feedwater pump inoperable One PORY (A) leaking and isolated 1 Events: Feed pump control problem that will eventually trip causing a partial l

loss of feed Total loss of main feedwater Loss of all feedwater Scenario Seouence "A" feedwater pump hydraulic control unit problems prompt the crew to 4 reduce power.

During power reduction, the "A" feedwater pump trips, causing a plant runback. Teedline break occurs causing a reactor trip. Auxiliary feedwater pumps fail over several minutes, causing a loss of all feedwater, and prompting the crew to initiate a bleed-and-feed procedure.

Event one - malfunction / loss of feed oumo

Crew responds to a problem with the "A" feed pump, which eventually trips causing a runback.

Malfunctions required: 2 (RFP "A" HCU failure and RFP "A" Trip) h ., . 'l-1021 29 of 42 Interim Rev. 8, January 1997

Appendix D 2 Attachment 2 Objectives: Evaluate the crew in using normal operating procedures to reduce power when the feed pump starts to fail. Evaluate the crew in using abnormal operating procedures (A0Ps) to respond to a partial loss of feed. 1 Success Path: Use the normal operating procedures to reduce power when initial i problems occur with the feedwater pump. Use the A0Ps to respond to the partial loss of feed water and stabilize the plant to avoid a reactor trip. Event two - feedline rupture / reactor trio Crew responds to a total loss of feed flow with only the remaining motor-driven AFW pump available. Malfunctions required: 1 (feedline rupture) l l Objective: l Evaluate the crew's response to a loss of feed transient requiring a reactor trip by using the reactor trip response and reactor trip recovery E0Ps. Success Path: Recognize the impending reactor trip, trip the reactor if time permits, i and implement the appropriate immediate actions. Make the correct transition to the reactor trip recovery E0P upon completing the immediate and applicable subsequent actions of the reactor trip E0P. Event three - loss of all AFW/PORV failure Crew responds to a total loss of feed flow, eventually implementing a bleed-and-feed procedure with a failed PORV. Evaluators inform the crew that time compression is being used to accelerate the decrease in steam generator level. Malfunctions required: 2.(failure of all AFW and "B" PORV fails to open) Objective: Evaluate the crew's ability to recognize that there is no longer a heat sink, and correctly implement the applicable contingency procedure (loss of heat sink), including performing the bleed-and-feed procedure. NUREG-1021 30 of 42 Interim Rev. 8, January 1997

1 Appendix D 3 Attachment 2 Success Path: Implement the E0P for loss of heat sink. Attempt to reestablish auxiliary feed flow; when SG 1evels become too low, initiate the bleed-and-feed procedure Recognize the failure of the available PORV and reenergize, unblock, and open the leaking PORV; open both pressurizer and reactor head vents to ensure adequate bleed flow. Scenario Recapitulation Total Malfunctions: 5 Abnormal Events: 1 Major Transients: 2 (loss of main feed) (total loss of feed) E0Ps Entered: 1 E0P Contingencies: 1 (loss of heat sink) O l I l NUREG-1021 31 of 42 Interim Rev. 8, January 1997

Appendix D 4 Attachment 2 PWR SCENARIO TWO - LOCA AND COLD LEG RECIRCULATION Scenario Obiectives Evaluate the crew's response to unidentified primary leakage. Evaluate the crew's response to a circulating water pump trip and a condenser tube leak. Evaluate the crew in using the E0Ps during a LOCA with adverse 4 containment conditions. Evaluate the crew's sensitivity to key parameters and ability to implement cold leg recirculation. Scenario Summary Initial Conditions: 100 percent power Inoperable "A" diesel generator and "A" instrument air compressor Seismic event occurred during last shift Events: Primary leak increases to a point requiring a reactor trip. AFW pumps fail to automatically start on reactor trip. Leak leads to a safety injection (SI) and high pressure SI pumps fail to start automatically; LOCA occurs, RWST leak occurs, and crew must initiate cold leg recirculation. Scenario Seauence A small pressurizer steam space leak increases to a point requiring a reactor trip and eventually to the point of SI initiation. The high pressure SI pumps fall to start automatically. A LOCA occurs as a result of the seismic event. When the SI pumps start, the thermal shock causes a LOCA in the RCS. The high pressure of the LOCA causes adverse containment conditions. An RWST leak will also occur concurrent with the SI that will eventually prompt the crew to initiate cold leg recirculation. RWST level will eventually drop to the point where the crew must initiate cold leg recirculation. , Event one - Unidentified leakaoe due to oressurizer steam space leak The crew reacts to unidentified primary leakage, eventually requiring a reactor trip.

                                                                                       ]

Malfunctions required: 1 (pressurizer steam space leak) NUREG-1021 32 of 42 Interim Rev. 8, January 1997 9 l i

n Appendix D 5 Attachment 2 l Objectives: I Evaluate the crew's use of A0Ps and TS to respond to unidentified  ! i primary leakage.  ; i - Evaluate the crew's knowledge of parameters in the AOP that require a  : trip because of primary leakage. Success Path: ' 1 Use the AOPs, increase reactor make-up and calculate a leak rate. i Use the NOPs to commence a reactor shutdown in accordance with TS. ' j - When leakage exceeds the A0P parameters, trip the reactor. l 1 Event two - reactor trio /AFW oumo fails to start automatically l l The crew trips the reactor on excessive leakage per the A0P. The AFW pumps l

fail to start automatically, requiring manual initiation. l

l

Malfunctions required

1 (AFW failure to auto start) 3 Objective:

l Evaluate the crew's use of the E0Ps following a reactor trip, with the complication of the AFW pumps failing to start automatically.  ! Success Path: Recognize that the AFW pumps failed to start automatically and manually start the pumps. Correctly perform the reactor trip EOP and make the transition to the reactor trip recovery E0P once the immediate actions and applicable subsequent actions are completed. Event three - increasina pressurizer leak /SI oumos fail to start The pressurizer leak increases causing a loss of pressurizer level / pressure requiring an SI. The charging pumps fail to automatically start requiring manual start. Malfunctions required: 2 (pressurizer leak increases and charging pumps fail 1 to auto start) Objectives: Evaluate the crew's ability to monitor important parameters in the E0Ps  ; and initiate SI when required. , Evaluate the crew's ability to manually start the charging pumps l following a SI signal.  ; NUREG-1021 33 of 42 Interim Rev. 8, January 1997 i

Appendix 0 6 Attachment 2 Success Path: Initiate SI when pressurizer level and pressure decrease to the values stated in the E0Ps. Recognize the failure of charging pumps to automatically start and manually start the required charging pumps to complete the SI initiation sequence. Event four - LOCA/ adverse containment A LOCA occurs as a result of the seismic event, which leads to adverse containment conditions. RWST level decreases to the point where the crew must enter the E0P for initiating cold leg recirculation. Evaluators in.' arm the crew that time compression is being used to accelerate the decrease in RWST level. Malfunctions required: 2 (LOCA and RWST leak) Objectives: Evaluate the crew's use of the E0Ps with adverse containment. Evaluate the crew's ability to recognize the need for and use the cold leg recirculation procedure. Success Path: Correctly enter and use the LOCA E0P and the containment functional recovery E0P using adverse containment criteria. When RWST levels reach the low-low alarm and the reactor sump level is high enough, enter and implement the cold leg recirculation E0P. I Scenario Recapitulation Total Malfunctions: 6 Abnormal Events: 2 Major Transients: 2 (leak requiring SI) (LOCA with high containment pressure) E0Ps Entered: A enter LOCA E0P twice) E0P Contingencies: , (containment safety) NUREG-1021 34 of 42 Interim Rev. 8, January 1997 '

i i I( Appendix D l 7 Attachment 2 a j BWR SCENARIO ONE - LOSS-0F OFFSITE POWER WITH A LOCA , Scenario Ob_iective { Evaluate the operators in using the " Emergency Depressurization" and "RPV

Flooding" E0P contingency procedures.

9 Scenario S'- ry Initial Conditions: 98 percent power j

               "A" average power range monitor (APRM) failed and bypassed

Events

j - Reactor core isolation cooling (RCIC) becomes isolated during a RCIC

flow surveillance.

Loss of offsite power / division III diesel generator fails to start,

disabling the high pressure (HP) core spray.

l - Small break LOCA occurs. i - Adverse containment conditions make the reactor level instrumentation j unusable. ! Scenario Seouence l The RCIC becomes isolated during surveillanca testing, rendering the ! RCIC system inoperable. j - Faults in the 345 KV switchyard and the reserve auxiliary transformer

result in a loss of offsite power and a reactor scram.

The Division III diesel generator fails to start and will not start i manually, disabling the HP core spray system. j - The plant transient causes a recirculation line break resulting in a i small break LOCA that develops over several minutes. Reactor level instrumentation becomes erratic and unusable because of ] the rapid decrease in pressure and the elevated drywell temperature. l Eyent one - RCIC isolation i l Tne crew responds to an isolation of the reactor core molation cooling (RCIC)

system during a full flow test surveillance.

i Malfunctions required: 1 (RCIC isolation) Objective: Evaluate the crew in using technical specifications to determine that RCIC is inoperable. NUREG-1021 35 of 42 Interim Rev. 8, January 1997 l l l i

Appendix D 8 Attachment 2 Success Path: Use technical specifications to recognize that the RCIC system should be declared inoperable until the problem can be investigated and corrected. Event two - loss of offsite power with concurrent division III diesel aenerator failure (HP core sorav) The crew responds to the loss of offsite power, reactor scram and loss of high pressure injection sources. Malfunctions required: 2 (loss of offsite power and HPCS failure) Objective: Evaluate the crew's response to a plant transient that causes a reactor scram and a loss of high pressure injection sources by using the reactor pressure vessel (RPV) and primary containment control E0Ps. Success Path: Maintain RPV pressure at less than 1065 psig using the main turbine bypass valves. Manually control prescure with safety relief valves (SRVs) upon a loss of electro-hydraulic control (EHC) hydraulic pressure because of the loss of power to the EHC pumps. Initiate suppression pool cooling and pump down in accordance with E0Ps if the temperature in the suppression pool exceeds 90 degrees or the level exceeds 18.5 feet. Event three - small break LOCA The crew responds to a loss of vessel inventory and an inability to maintain a level greater than top of active fuel, eventually implementing emergency depressurization. Malfunctions required: 1 (LOCA) Objective: Evaluate the crew's ability to recognize an inability to maintain reactor water level and correctly implement the applicable contingency procedures including emergency depressurization. Success Path: Execute RPV emergency depressurization so reactor pressure can be decreased to allow injection by the low pressure ECCS systems. NUREG-1021 36 of 42 Interim Rev. 8, January 1997

l i Appendix 0 9 Attachment 2 Event four - reactor level instrumentation failure The crew recognizes a loss of reactor level instrumentation and responds in accordance with RPV flooding E0P. Malfunctions required: 1 (reactor level instrumentation failure) Objective: Evaluate the crew's ability to recognize failed reactor level instrumentation ) and correctly implement the applicable actions of the RPV flooding E0P to ensure adequate core cooling. Success Path: l Reflood the RPV in accordance with the E0Ps and establish adequate core cooling. Adequate core cooling will be ensured when reactor pressure can be maintained greater than 120 psig with at least 3 SRVs opened by manually controlling low pressure ECCS injection flow. Scenario Recacitulation Total Malfunctions: 5 C\ Abnormal Events: 3 V Major Transients: 2 (emergency depressurization) (RPV flooding) E0Ps Entered: 2 E0P Contingencies: 3 (alternate level control) (emergency depressurization) (RPV flooding) NUREG-1021 37 of 42 Interim Rev. 8, January 1997

Appendix D 10 , , , , , Attachment 2 BWR SCENARIO TWO - POWER OSCILLATIONS WITH AN ATWS $cenario Obiective Evaluate the operators in using the " Level / Power Control" and " Emergency Depressurization" E0P contingency procedures. Scenario Summary Initial Conditions: 75 percent recctor power , High Pressure Core Spray pump out of service 1 "B" recirculation pump flow control valve is locked Events: The "A" reactor recirculation pump trips, causing power oscillations, i and an SRV fails open during the power oscillations. ' Anticipated transient without scram (ATWS) requiring lowering of level to control power. Feed system pumps will fail to restart and standby liquid control (SLC) pumps and RCIC pump trip during the transient, complicating recovery from the event. Scenario Seauence i The "A" recirculation pump trips resulting in power oscillations within l 5 minutes. The reactor fails to manually scram.  ; The safety relief valve (SRV) sticks open during power oscillations. Condensate booster and feedwater pumps fail to restart, and the SLC pumps trip after power is reduced less than 3 percent. RCIC pump trips after it is restarted by an operator. Event one "A" recirc oumo trio resultina in oower oscillations The crew responds to a recirculation pump trip and a failure of the reactor scram system. l l Malfunctions required: 2 (recirculation pump trip and ATWS) l Objectives: 1 Evaluate the crew's use of A0Ps and E0Ps ta respond to an ATWS and to restore the power and flow parameters to acceptable values. Evaluate the crew's use of TS that apply to single recirculation loop operation. , NUREG-1021 38 of 42 Interim Rev. 8, January 1997

l m Appendix D 11 Attachment 2 Success Path: Recognize power to be in region B or C of the power and flow map and , initiate control rod insertion to reduce thermal power.  ! Recognize symptoms of thermal hydraulic instability and attempt to manually scram. Use the E0P flow charts for RPV level, power, and pressure control,  ; Trip the "B" recirculation pump and initiate actions to achieve control i rod insertion and to actuate the standby liquid control system in accordance with the E0Ps. Event two - SRV sticks open durina power oscillations The crew recognizes and responds to the stuck open SRV, eventually implementing the actions of the primary containment control E0P. Malfunctions required: 1 (SRV sticks open) Objective: Evaluate the crew's ability to recognize the failed open SRV and implement the l , applicable abnormal and emergency procedure actions. I Success Path: Initiate actions to close the SRV. Use E0Ps to initiate suppression pool cooling and reduce the level. Terminate all injection into the RPV except for the control rod drive

and SLC systems when suppression pool temperature exceeds 110 degrees with reactor power less than 3 percent.

Event three - failure of in.iections sources after control rod insertion i The crew responds to a loss of vessel inventory and the inability to maintain level greater than top of active fuel by eventually implementing emergency . depressurization. Malfunctions required: 3 (feedwater system failure, SLC pump trip, and i RCIC fails to start)

Objective

Evaluate the crew's use of E0Ps to respond to an inability to maintain reactor water level and to initiate an emergency depressurization. Success Path: 4 Execute RPV emergency depressurization to allow for injection by the low pressure ECCS systems. J NUREG-1021 39 of 42 Interim Rev. 8, January 1997

Appendix 0 12 Attachment 2 Scenario Recacitulation Total Malfunctions: 6 Abnormal Events: 2 Major Transients: 2 (ATWS) (emergency depressurization) E0Ps Entered: 2 E0P Contingencies: 3 (level and power control) (alternate level control) temergency depressurization) l O i NUREG-1021 40 of 42 Interim Rev. 8, January 1997

l l 6 Appendix 0 Scenario Outline Form ES-D-1 %J Facility: Scenario No.: Op-Test No.: Examiners: Operators: Objectives: Initial Conditions: i Turnover: 1 i Event Malf. Event Event No. No. Type

• Description i O

l l I

   *    (N)omal,     (R)eactivity,    (I)nstrument,     (C)omponent,      (M)ajor NUREG-1021                          41 of 42        Interim Rev. 8, January 1997

Appendix D Operator Actions Form ES-D-2 Op-Test No.: Scenario No.: Event No.: Page of Event

Description:

Time Position Applicant's Actions or Behavior O I . 1 NUREG-1021 42 of 42 Interim Rev. 8, January 1997

    - - - - - - .. - - . - -                            .   .. - - . - . - . - - - - -                        ~ - - - - -

APPENDIX E

POLICIES AND GUIDELINES FOR TAKING NRC EXAMINATIONS

, Each examinee shall be briefed on the policies and guidelines applicable to i the examination category (written and/or operating test) being administered. i The applicants may be briefed individually or as a group. Facility licensees

are encouraged to distribute a copy of this appendix to every examinee before 4

the examinations begin. All items apply to both initial and requalification examinations, except as noted. PART A - GENERAL GUIDELINES { ! 1. {Aead Fer6atis] Cheating on any part of the examination will result in j a denial of your application and/or action against your license.

2. If you have any questions concerning the administration of any part of l the examination, do not hesitate asking them before starting that part j- of the test.

t

4. SRO applicants will be tested at the level of responsibility of the senior licensed shift position (i.e., shift supervisor, senior shift supervisor, or whatever the title of the position may be).

5. You must pass every part of the examination to receive a license or to continue performing license duties. Applicants for an SRO-upgrade O license may require remedial training in order to continue their R0 duties if the examination reveals deficiencies in the required knowledge and abilities.

6. The NRC examiner is not allowed to reveal the results of any part of the examination until they have been reviewed and approved by NRC management. Grades provided by the facility licensee are preliminary until approved by the NRC. You will be-informed of the official examination results about 30 days after all the examinations are complete.

PART B - WRITTEN EXAMINATION GUIDELINES

1. [ Read Fer6atis] After you complete the examination, sign the statement on the cover sheet indicating that the work is your own and you have not received or given assistance in completing the examination.

2. To pass the examination, you must achieve a grade of 80.00 percent or greater. -Every question is worth one point.

3. For an initial examination, the time limit for completing the examination is four hours.

For a requalification examination, the time limit for completing both sections of the examination is three hours. If both sections are NUREG-1021 1 of 5 Interim Rev. 8, January 1997

Appendix E , l administered in the simulator during a single three-hour period, you may

return to a section of the examination that was already completed or l retain both sections of the examination until the allotted time has expired.

4. You may bring pens and calculators into the examination room. Use only black ink to ensure legible copies.

l 5. Print your name in the blank provided on the examination cover sheet and the answer sheet. You may be. asked to provide the examiner with some form of positive identification.

6. Mark your answers on the answer sheet provided and do not leave any question blank. Use only the paper provided and do not write on the back side of the pages. If you decide to change your original answer, draw a single line through the error, enter the desired answer, and i initial the change.

7. If the intent of a gestion is unclear, ask questions of the NRC examiner or the de .v n :ted facility instructor only.

8. Restroom trips an p, rmitted, but only one applicant at a time will be allowed to leave. % id all contact with anyone outside the examination '

room to eliminate even the appearance or possibility of cheating.

9. When you complete the examination, assemble a package including the examination questions, examination aids, answer sheets, and scrap paper and give it to the NRC examiner or proctor. Remember to sign the statement on the examination cover sheet indicating that the work is l your own and that you have neither given nor received assistance in completing the examination. The scrap paper will be disposed of .

immediately after the examination.  !

10. After you have turned in your examination, leave the examination area as defined by the proctor or NRC examiner. If you are found in this area while the examination is still in progress, your license may be denied or revoked.

11. Do you have any questions?

PART C - GENERIC OPERATING TEST GUIDELINES (CATEGORIES A. B. AND C)  !

1. If you are asked a question or direc^.ed to perform a task that is unclear, you should not hesitate to ask for clarification.

2. The examiner will take notes throughout the test to document your performance, and sometimes the examiner may take a short break for this reason. The amount of note-taking does not reflect your level of NUREG-1021 2 of 5 Interim Rev. b, January 1997 l

1

Appendix E performance. The examiner is required to document satisfactory as well as less than satisfactory performance.

3. The operating test is considered "open reference." The reference material that is normally available to operators in the facility and control. room (including calibration curves, previous log entries, piping and instrumentation diagrams, calculation sheets, and procedures) is also available to you during the operating test. However, you should know from memory certain automatic actions,. set points, interlocks, operating characteristics, and the immediate actions of emergency and

, other procedures, as appropriate to the facility. If you desire to use l a reference, you should ask the examiner if it is acceptable to do so for the task or question under consideration. t L You may not solicit technical information from other operators, , engineers, or technical advisors.  ! , 4. You must not discuss any aspect of your operating test with any other j l examinee until after all the examinations are complete. l PART D - WALK-THROUGH TEST GUIDELINES (CATEGORIES A AND B) 1.- The walk-through test covers control room systems, local system ,O l operations, and administrative requirements. The examiner will evaluate each of these areas using a combination of job performance measures (JPMs) and specific questions. i

                                                                                                                           ~

The initial walk-through consists of ten JPMs for R0 and SR0(I) i applicants and five for SR0(U) applicants. Seven of the JPMs (two or will be conducted in the control room or three for and simulator upgrade applicants)ill the remainder w be conducted in the plant. The requalification walk-through consists of five JPMs total, with at least two in the control room / simulator and at least two in the plant. l l 2. The examiner is a visitor at this facility. When you enter the plant, i you may be expected to escort the examiner and ensure that he or she complies with safety, security, and radiation protection procedures. , i

3. You should not operate plant equipment without appropriate permission j from the operating crew. Nothing the examiner says or asks will be intended to violate this principle.

4. Before beginning each JPM, the examiner will describe the initial i

                                   - conditions, explain the task that is to be completed, indicate whether                l the task is time-critical, and explain which steps are to be simulated or discussed. You should perform or simulate the required actions as if

directed by plant procedures or shift supervision. Do not assume that i NUREG-1021 3 of 5 Interim Rev. 8, January 1997 t . - .

Appendix E the examiner will accept an oral description of the required action unless the examiner indicates otherwise.

5. Time-critical JPMs have been validated by your facility and must be completed within the predetermined time interval in order to obtain a satisfactory grade for that JPM. You will be permitted to take whatever time is necessary to complete those JPMs that are not time-critical, provided you are making reasonable progress toward achieving the task standard.

6. When performing JPMs, you are expected to make decisions and take actions based on the facility's procedural guidance and the indications available. Some of the tasks that the examiner asks you to perform will require the implementation of an alternative method directed by plant procedures. >

7. As part of the initial examination, the examiner will ask follow-up questions at the end of each JPM to investigate your knowledge of the applicable system or task. Many of the questions will require you to use plant reference material, while others should be answered without the use of references. If you need to consult a reference to answer a question, ask the examiner if it is acceptable to do so. There is no specific time limit for any question, however, you may be evaluated as unsatisfactory on a question if you are unfamiliar with the subject or reference material and are unable to answer the question in a reasonable period of time. You will not be permitted to conduct unlimited searches of the plant reference material during the examination.

Although the requalification examination does not include prescripted follow-up questions, the examiner may ask questions as necessary at the end of any JPM to clarify your performance.

8. To facilitate the examination, please inform the examiner when you consider your performance of each JPM and your answer to each question to be complete.

9. If you need a break during the test, you should ask the examiner.

10. Do you have any questions before we begin the walk-through test?

PART E - SIMULATOR TEST GUIDELINES (CATEGORY C)

1. Your primary responsibility is to o)erate the simulator as if it were the actual plant. If you believe t1at the simulator is not responding properly, you should make decisions and recommendations on the basis of the indications available, unless directed otherwise by the examiner.

2. If the examiner asks you a question, you should answer it only if doing so will not interfere with simulation facility operations.

NUREG-1021 4 of 5 Interim Rev. 8, January 1997 !

l

 /~N Appendix E i

3. Teamwork and communications are evaluated. You can enhance the j evaluation process by vocalizing your observations, analyses, and the '

bases for your actions. I Requalification examinations evaluate the crew's ability to safely operate the plant and the performance of both the individuals and the crew.

4. If you recognize but fail to correct an erroneous decision, response, answer, analysis, action, or interpretation made by the operating team or crew, the examiner may conclude that you agree with the kicorrect item.

5. You should keep a rough log during each scenario that would be sufficient to complete necessary formal log entries.

l 6. A designated facility instructor (or an examiner) will act as the l auxiliary operators, radiation health and chemistry technicians, I maintenance supervisors, plant management, and anyone else needed outside the control room. ! 7. The facility instructor (or examiner) will provide a shift turnover ! briefing before the scenario begins. The briefing will cover present plant conditions, power history, equipment out of service, abnormal conditions, surveillances due, and instructions for the shift. l 8. Control board switches may be purposely misaligned to enhance a scenario or transient where appropriate. You will not be required to locate misaligned switches as part of the evaluation. If a switch is misaligned, it will be tagged or otherwise highlighted as appropriate to the facility and will be noted during the turnover briefing. The examiners will not misalign switches during the scenario.

9. Time compression may be used to expedite the sequence of events in some scenarios, but it will not preclude you from performing the actions that you would typically be required to perform in response to the events.

If time compression is used, you will be so informed during and after the scenario. i

10. You will normally be given about five minutes to familiarize yourselves with plant conditions before starting each simulator scenario.

11. The initial test vill normally consist of two or three scenarios lasting a total of three to four hours. The requalification test will normally consist of two scenarios lasting about one hour each. You will be given

, a short break between scenarios.

12. Do you have any questions before we begin the simulator test?

O , V NUREG-1021 5 of 5 Interim Rev. 8, January 1997 i

I APPENDIX F l GLOSSARY Achievement test: An instrument designed to measure a trainee's grasp of some ' body of knowledge or skill proficiency. Annual: In most instances, a period of time equal- to 365 days reckoned from i any point in a calendar year to the same point in time in the following  : calendar year. However, annual requirements in successive years can reach a  ; period of nearly two years. Annual could encompass a range extending to 729 1 days depending on when an event occurred in the first calendar year and viewing December 31 of the following calendar year as meeting the annual l requirement. l  ; f l Anolicant: Any individual who has submitted an NRC Form 398 in pursuit of an ] l RO or SRO license. For purposes of this and the other Examination Standards, l it is synonymous with " candidate." i l l Anolicant license level: The level of operator license (i.e., R0 or SRO) for j l which the applicant has applied. , Antitude test: An instrument designed to assess an individual's potential for  ; l performing some task or skill area. 5 Average: A score that provides an indication of the typical performance of a i group of scores. The mean, median, and mode of a distribution of scores are all commonly used as averages. i Biennial: In most instances, a period of time equal to 730 days and  : synonymous with the ters "two years." Biennial requirements can extend beyond 730 days if the requirement is met during the anniversary month of the second l year. For example, a biennial medical examination last performed on January !- 10, 1995, would be due again by January 31, 1997. January is seen as the anniversary month, the period of time between the two examinations is longer than 730 days, but the biennial requirement is satisfied. l Bloom's Taxonouv: A classification system that depicts knowledge and information processing of knowledge in a hierarchy from lowest to highest as

follows

fundamental knowledge, comprehension, analysis, synthesis, and evaluation.

l Calendar auarter: One of four parts of a calendar year, each consisting of a 3-month segment. In any calendar year, the first quarter is from the first l day of January to the last day of March, the second quarter is from the first day of April to the last day of June, the third quarter is from the first day of July to the last day of September, and the fourth quarter is from the first day of October to the last day of December.

Cateaory: One of 3 major subdivisions of related subjects on the operating l

test. Refer to Section D of ES-301 for a description of and detailed

instructions for developing each operating test category, NUREG-1021 1 of 6 Interim Rev. 8, January 1997 f

1 _, _ - - _ _ _~ _ - . _ . - - .

i Appendix F Central tendency: A term referring to the most typical performance of a group of individuals; generally the mean, median, or mode Coanitive: Aspects of a person or test level that refer to knowledge or understanding. Content validitv: The degree to which a test measures the specific objectives or content of that test. Correlation coefficient: A numerical value ranging from -1 to +1 that indicates the relationship between two sets of scores or other measures of  ; each individual in a group. A value of 0 indicates no relationship; +1 or -1  ; indicates a perfect relationship, either positive or negative. l Criterion: A characteristic or combination of characteristics used as the l basis for judging a performance. Criterion-referenced test: An examination based upon mastery of objectives of content that was or should have been taught and mastered and one that uses an established standard or cutoff score as a measure of acceptable performance. Cut score: The score at which a trainee is deemed to have met the criteria on an exam. < Diaanostic test: An instrument that is designed to identify the strengths and weaknesses of an individual for a given content area. Difficulty index: A numerical index ranging from 0.00 to 1.00 that indicates l the percentage of trainees who answer a test item correctly. An index of 0.00 l' indicates that no one answered the test item correctly while an index of 1.00 indicates that all individuals answered the item correctly. Discrimination index: A measure of a test item's ability to differentiate between good and poor trainees. A high discrimination index indicates that more high performers than low performers answered the item correctly (high and low are typically determined by overall test scores but may also be established by external criteria). Discrimination validity: Setting the item difficulty at an estimated level i around the cut score. ' Distractor: An incorrect alternative among the choices of a test item. Error of measurement: Any difference between an obtained score and a true score on a test is referred to as error of measurement. The actual error of measurement can only be estimated since it is impossible to know what the true score is. NUREG-1021 2 of 6 Interim Rev. 8, January 1997

Appendix F v Eauivalent forms: Two or more exams that test the same objectives using i different test items or the same test items in a different sequence. Freauency distribution: A graphic display listing scores, or score intervals on one axis of a graph, and the number of trainees at that score or in that , interval on the other. 1 Item analysis: A set of procedures performed on examination items to determine their difficulty and discriminating power. L'2.:; oank: A group of test items covering a defined area. Items for a test can be chosen from this source. Item stem: The part of a test item that presents the problem or situation to be solved. The stem may be a question requiring a response or a statement that is followed by the alternatives from which the trainee must choose the best answer. ' Job oerformance measure (JPM): An evaluation tool that is based on tasks contained in the facility's job task analysis (JTA) or the applicable NRC Knowledge and Abilities Catalog (NUREG-1122 or 1123) and requires the applicant to perform (or simulate) a task applicable to the license level of the examination. 2 Job task analysis (JTA): A systematic analysis of the knowledge, skills, and abilities required to perform a particular occupation. Learnin: ob.iective: A statement of the behavior a trainee is expected to exhibit following instruction. Masterv test: A term synonymous with criterion-referenced test, i.e., evaluating the expected behavior following instruction. 1 Mgan: An indication of central tendency; it usually refers to the arithmetic mean, which is computed by summing all the scores of a group and dividing that sua by the number of scores in the group. Median: A measure of central tendency; the point on a scale of scores that splits the scores in half; 50 percent of the scores are below this point, and 50 percent of the scores are above this point. ~ M94g: The least reliable of the common measure of central tendency; the made is the most frequently occurring score in a distribution of scores.

 !     Multiple choice item: A test item composed of a stem and several alternatives from which the trainee must select the best answer.

NUREG-1021 3 of 6 Interim Rev. 8, January 1997

Appendix F Normal distribution: A theoretical frequency distribution represented by a ' symmetrical bell-shaped curve; sometimes referred to as the bell curve. Norm-referenced: A score interpretation based on the comparison of an individual's score with a comparable reference group. l l Nuclear oower olant experience: As defined in Section 2 of ANSI /ANS-3.1-1981,  ; means experience acquired in preoperational and startup testing activities or ' operation of nuclear power plants. Experience in design, construction, and ) operational training may be considered applicable nuclear power plant i experience and should be evaluated on a case-by-case basis. Refer to ANSI /ANS-3.1-1981 for additional information regarding the equivalence of simulator and on-the-job training, and military and other experience. l Nuclear power olant fundamentals: Refer to Section 5.2.1.1 of ANSI /ANS-3.1- ) 1981 for a list of topics to be covered in a fundamentals course. j Obiective test: A test that can be scored without subjective judgment in the scoring. On-the .iob trainino: Participation in nuclear power plant startup, operation, maintenance, or technical services as a trainee under the direction of experienced personnel. Ooeratina test: That portion of the operator licensing examination based on direct interaction between an examiner and an applicant. It tests the applicants' knowledge of the design and operation of the reactor and its associated plant systems, both inside and outside the control room. It is administered in a plant walk-through and in a simulation facility. Operational validity: A test item that is 1) related to the operations of the job and appears reasonable to ask and 2) expressed in an operational context that requires the candidate to mentally or physically perform through understanding or analysis. Performance test: Any test that requires the trainee to demonstrate either mental performance through knowledge testing or skill by actual operation or manipulation of tools and equipment. Typically, performance tests connote the meaning of skill testing. Plant-refer:nced simulator: As defined in 10 CFR 55.4, means a simulator modeling the systems of the reference plant with which the operator interfaces in the control room, including operating consoles, and which permits use of thn reference plant's procedures. A plant-referenced simulator demonstrates expected plant response to operator input, and to normal, transient, and accident conditions to which the simulator has been designed to respond. NUREG-1021 4 of 6 Interim Rev. 8, January 1997

i l i Appendix F 1 Power plant experience: As defined in Section 2 of ANSI /ANS 3.1-1981, means experience acquired in the testing, operation, and maintenance of power generating facilities. Experience in design and construction may be considered applicable power plant experience and should be evaluated on a

case-by-case basis, i

Predictiye validity evidence: The ability of a test to forecast future performance on a subsequent measure. Psychomotor: The domain of human performance that relates to physical ! performance based on mental activity. Rahgi: The smallest interval on a scale of scores that will include all scores, mathematically defined as the largest score minus the smallest score J plus one.

   . Raw score: The numerical score first assigned when scoring a test before conversion to a derived score.

Reactor operator aoolicant: An unlicensed individual who is applying for an l R0 license. l O Reference olant: As defined in 10 CFR 55.4, means the specific nuclear power , plant from which a simulation facility's control room configuration, system I control arrangements, and design data are derived. Related technical trainina: As defined in Section 2 of ANSI /ANS-3.1-1981, means formal training beyond the high school level in technical subjects associated with the position in question, such as acquired in training schools or programs conducted by the military, industry, utilities, universities, i vocational schools, or others. Such training programs shall be of a scheduled  ! and planned length and include text material and lectures. Reliability: The consistency or repeatability of any measure as an indicator of confidence of that measure. ICADAtin: An integrated group of events that simulate a set of plant malfunctions and evolutions at a simulation facility. Scenario set: A group of scenarios that constitutes a complete simulator test (i.e., Category C, " Integrated Plant Operations," of the operating test). l Score: A numerical indication of the performance an individual displays on a ! test. Senior reactor operator unarade (SRO-U) aoolicant: A licensed R0 who is l applying for an SR0 license on the same unit (s). ! O fV NUREG-1021 5 of 6 Interim Rev. 8, January 1997 l l

Appendix F Senior reactor operator instant (SRO-I) aonlicant: An unlicensed individual who is appiying for an SRO license. Simulation facility: As defined in 10 CFR 55.4, means one or more of the following components, alone or in combination, used for the partial conduct of operating tests for operators, senior operators, and applicants:

1. the plant

2. a plant-referenced simulator

3. another simulation device This definition provides flexibility in the conduct of the " Integrated Plant Operations" category of the operating test, as permitted in 10 CFR 55.45(b).

It allows examiners to administer the operating test on the plant itself, a plant-referenced simulator, or some other type of NRC-approved simulation device, such as a part-task or basic-principles simulator. Standard deviation: A measure of variability of a set of scores around the group mean. The SD is mathematically defined as the square root of the mean of the squared deviations of the scores from the mean of the distribution. Standard error of measurement: An estimate of the standard deviation of the errors of measurement associated with the test scores in a given test. Standardized test: A test that has the directions, time limits, and conditions of administration made consistent for all offerings of the test; this test is usually norm-referenced. Statistic: A numerical value computed on a sample of data. Igit: A measurement instrument; examination. True score: The ideal or correct score for an individual. Its value cannot l be known, but it can be estimated when assumptions regarding error of measurement are made. Validitv: The degree to which a test measures what it purports to ineasure. l l 1 NUREG-1021 6 of 6 Interim Rev. 8, January 1997 l

     ~_                      .      ...             .                                               .. - -     -                   -                 - .       . . - .

NRCFortM ses u.s. NUCLEAR RiiGuLATORY """*" 1. REPORT NUMBE2 DM (Assigned by VC, Add vol., supp., R:v., EE BIBUOGRAPHIC DATA SHEET ""*""""^""'"'"*1 (See msMstune en memvwee)

TITLE AND SUBTTTLE NUREG-1021, interim Revision 8 Operator Ucensing Examination Standards 6r Pow x Reactors

3. DATE REPORT PUBLISHED MONTH YEAR l

January 1997

4. FIN OR GRANT NUMBER

6. AUTHOR (5) 6. TYPE OF REPORT 4

Sta# Technical

7. PERIOD COVERED (incluswe Deres)

I

s. PERFORMNG ORGANEZATION - NME AND ADDRESS grme, povme Dvam. omoe or Asym U.S. NucherRegumy commusen, and messag addren. #contecer.

Fonde aanw and memng emeses) 1 Dmeson of Reactor Controls and Human Factors l Office of Nuclear Reactor Regulation l U.S. Nuclear Regulatory Commission l Washington, DC 20555-0001

9. SPONSORING ORGANIZATION . NM AND ADDRESS (rme, 6pe seme as e6 ave; #conesetr. pavide NRc oween, orrze or Rege, u s. Nuceer Reguietary comm,ssa, and madng edeemsJ Same as above i

10. SUPPLEMENTARY NOTES l l__ Replace : Revision 7 of NUREG-1021, entitled " Operator Ucensing Examiner Standards." and Revision 5 of NUREG/BR-0122 l

11. ABSTRACT (200 eards or Ases)

NUREG-1021," Operator Ucensing Examination Standards for Power Reactors," establishes the policies, procedures, and practices for examining licensees and applicants for reactor operator and senior reactor operator Ecenses at power reactor facilities pursuant to Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55). The examination standards are intended to assist NRC examiners and facility licensees to better understand the procseses associated with initial and requalificaton examinations. The standards also ensure the equitable and consistent administration of examinations for all applicants. The standards are for guidance purposes and are not a substitute for the operator Ecensing regulations (i.e.,10 CFR Part 55), and they are subject to revision or other changes in internal operator Scensing poicy. This interim revision permits facility Econsees to prepare their initial operator Ecensing examinations on a voluntary basis pending an amendment to 10 CFR Part 55 that wil require facility participation. The NRC intends to solicit comments on this revision during the rulemaking process and to issue a final Revision 8 in conjunction with the final rule. l 1 3 AvAiLAsitrry STATEMENT

12. PIY WORDSOESCRIPTORS (Let ease er pereses met we es.smf resse eners m 4xserg me repart) unlimited Examination Standards 14 sEcuRrryctAssiricAT)oN Operator Ucensing (The Page) unclassified (Tha Repert) unclassified

15. NUMBER OF PAGES l f

16. PRICE NRC FORM 335 Q49) The % was emity produced by Ehte Federal Forms. Inc

i i

9 J

1 I t 1 I O l l P Printed on recycled paper Federal Recycling Program a}}