ML21211A578

NUREG 1021 Revision 12 Comment Resolution

ML21211A578
Person / Time
Issue date:	07/30/2021
From:	Maurin Scheetz NRC/NRR/DRO/IOLB
To:
Scheetz M
References
Download: ML21211A578 (50)

Text

NUREG-1021 Revision 12 Public Comments Number Section Comment Description Recommendation NRC Disposition Section 1, General Added clarification: Procedures and guidance How are procedures and guidance Provide clarification, if necessary, ES1.1 page 1 are normally identified in the introductory 1 differentiated within the standard? How is on the difference between (line 18) paragraph or identified in the section or each identified? procedures and guidance.

subsection titles. This was added to ES 1.1, C.

Missing statement "and the licensee shall first Confirm if NRC contact is required No change required. The NRC point of contact is ES1.2 step 3, notify the NRCs regional office to ensure that during written exam and ensure it still necessary if the NRC is not on site during 2 page 1 of 6 a point of contact remains available to respond is added back in the section if so the written examination. See ES 4.3 for Written (line 4146) to questions." Is contact no longer required or intended. Examination Administration.

not allowed?

Add similar wording to the Indicates that applicants tablets, cell phones operating test section if they are or other communication devices are not not allowed or add a statement Added an item to the Operating Test Guidelines ES1.2 step 4, allowed into the examination room for the that they cannot be used if that is in ES 1.2: Applicants shall not use personal cell 3 page 2 of 6 written exam. There is no corresponding the intent. Another option would phones or electronic devices during the (line 1012) statement about them not being allowed be to place the statement in the operating test.

during the operating test. overall section covering all aspects of the exam.

States; Note that answers to questions you asked during the examination are documented ES1.2 step 7, and taken into consideration during the Consider modifying the statement Added to the same sentence that questions will 4 page 2 of 6 grading process. to include questions and answers also be documented.

(line 30) To improve clarity and intend, ensure that will be documented.

questions asked and answers provided will be documented.

The following statement was removed: Many of the questions will require you to use plant No change made because this statement reference material, while others should be Add statement back in from Rev.

ES1.2, step appears in the ES 1.2 Section C, "General answered without the use of references. If you 11 if the intent is still for applicants 5 10, page 4 of 6 Operating Test Guidelines," as No. 3. This type need to consult a reference to answer a to request permission to use a (lines 3435) of information was relocated to the subsection question, ask the examiner if it is acceptable to reference.

as part of the streamline effort.

do so. Should that guidance be added back for clarity?

This statement was deleted from R12 ES1.2 B.2 (R11 Appendix E B.2): SROU applicants No change made. In ES 2.1, A, "General who do take the RO portion of the exam and Guidelines, " the term "additional training" was score below 80 percent on that part of the used to replace "remediation training" and this exam can still pass overall but may require now applies to ANY applicant that ES1.2 B.2, remediation. Confirm if statement should be demonstrates deficiencies on the written 6 Page 1 of 6 Does this change imply that an SROU who added to R12 ES1.2. examination, not just SROU applicants. In (line 3439) takes the RO portion of the examination with other words, the training organization should overall grade of 80% and greater than 70% on make sure that the applicants understand what SROonly items no longer require analysis for questions they got wrong and why. This is an RO questions to determine if remediation is expectation of a SAT based training program.

required?

ES1.2 B.3 pages 12 outlines times for completing initial and requal examinations.

The first paragraph states: "The times allotted for taking each examination noted above shall not be extended except for unavoidable The statement of not extending situations (e.g., loss of power, building time should either be repeated or evacuation, emergency" response). modified to state that there are no The time allotments are for NRC conducted The following paragraph states: "For a time extensions for either exams.

ES1.2 B.3, requalification examinations. Revised the 7 requalification examination, the time limit for Pages 1 to 2) statement to allow for extensions resulting completing both sections of the examination is Provide clarification if it is from unavoidable circumstances.

3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />." Is the intent that the time for requal acceptable if a utility procedure exams not be extended as well? states that an extension may be granted Additionally, does the time limit for requal exams only apply for NRC developed requal exams? If a utility procedure states that an extension may be granted, is that acceptable?

From R11 Appendix E D.3, statement was added If the JPM task requires the opening of If consideration for how to panel doors, check with your examiner before No change made. This statement was added coordinate opening doors with opening the door; the examiner will let you based on operating experience from past JPMs ES1.2 D.5, control room staff should be know if the operating crew has given in the plant. Regarding the second part of the 8 Page 4 of 6 included as an examiner cue within permission. comment, anything the applicant does during (line 68) JPMs, consider adding appropriate Does this change indicate that requirements to the JPM can be considered in the evaluation of guidance to ES3.2, Developing Job open cabinet doors be discretely evaluated the applicant's performance.

Performance Measures.

during JPM development and results included in the JPM?

R12 ES1.2 E.5 (from R11 Appendix E E.5) added the statement; the examiner may The NRC added, "the examiner may consider consider these notes when evaluating your these notes when evaluating your performance.

performance," because this type of information ES1.2 E.5, What is intent of this statement? If rough logs Consider removing statement or is something the applicant is expected to know 9 Page 5 of 6 are not maintained, how will it impact provide clarification on how these before taking the operating test. Rough logs (line 2324) evaluation? Statement implies additional notes will be evaluated.

can provide an examiner information about importance has been placed on maintaining applicant knowledge. Simulator logs do not rough logs, which could be reconstructed post provide this type of performance information.

event using event recorders, chart recorders, computer logging, etc.

No change made. The change is not needed R11 Appendix E E.11 stated You may be given because applicant breaks are in accordance a short break between scenarios, which was with facility licensee procedures and the ES1.2 E. 11, Confirm intent of change in R12 deleted in R12. examination schedule; this level of detail was 10 Page 6 of 6 change summary or include Does this change imply that breaks between removed from the ES 1.2 Guidelines. The (line 7) allowance to provide break in R12.

scenarios are no longer permitted or is level of facility licensee and the NRC chief examiner detail being reduced? coordinate the examination schedule in accordance with ES 3.5.

Section 2, PreExamination Activities Consider adding more detail to the ES2.1 page 8 Statement is incomplete; should include Change made consistent with the statement to include review 11 of 20 (lines 30 review findings to the written exam and op recommendation, added written examination findings to the written exam and

31) test. and operating test to the scope of this step.

optest.

Is there any time limit on previous No change made because there is no time limit employment? Example, if an examiner worked ES2.1 page 10 for this situation. The restriction is for an NRC at a utility 10 years ago when the candidates 12 of 20 (line 9 Add a time limit to this restriction. examiner that was previously employed at the were in initial non licensed operator training

15) facility AND involved in training any of the are they allowed to be part of the license current license applicants.

exam?

This item requires the facility to submit their entire JPM bank as part of the reference material to provide for each operator licensing initial examination. Significant resources can be expended in order to maintain an entire JPM bank ready for use at any time.

Reword this requirement for the Additionally, the wording of this requirement No change made because the beginning of ES ES2.1, 4.a.(4) facility to submit a list of available infers that the JPM bank will reach a finite size, 2.1 F says that this list can be tailored for the Page 12 of 20 JPMs, along with a status of each 13 however exam development rules require new specific examination. This should be Lines 10 thru one to indicate if it is, or was, or modified JPMs to be developed with each coordinated with the NRC chief examiner during 15 recently validated and considered exam. In practice, a new or modified JPM could the initial meeting.

ready to use.

be developed to test much of the facilitys job task analysis.

For cold licensing plants, the number of available JPMs may be small, and the inplant JPMs will be constructed to use the alternative means described in ES3.7.

No change made. The recommendation may not provide enough information about the This item requires the facility to submit a list of available malfunctions for NRC developed all malfunctions that the simulator can ES2.1, 4.o.(2) examinations or for NRC review of facility perform, with causeandeffect information and licensee examinations. Additionally, ES 2.1, F and concise descriptions of each. Many 4.o.(4) Recommend rewording this states: The NRC expects facility licensees to simulators available for use at newer plants, 14 Page 13 of 20 requirement to provide a list of the provide reference materials for each NRC and at existing fleet plant, have vast Lines 2 thru 5, available malfunctions operator licensing initial examination. The NRC capabilities to make variable malfunctions. As and regional office will customize the list of this capability has expanded, the severity of 8 thru 10 reference materials to support the specific malfunctions is more and more dependent on examination assignment. The regional office the starting conditions when they occur.

shall consider the administrative burden it places on the facility licensee and will request

Previous version contained references to the ES2.1 page 17 applicable Section of the standard. This was 15 of 20, Form 2.1 Recommend adding them back in. References added to Form 2.11 useful for reviewing the actual 1

requirement/guideline.

Figure 1 identifies operator licensing milestones for 10 CFR Part 52 plants under Recommend moving the construction. Consider making the following milestone for start of monitoring changes, based on experiences gained during initial accreditation activities closer Additional information has been added on how current new plant deployment activities: to the start of the first class, and to use Figure 1 in ES 2.2. This figure contains a The initial accreditation of operator add an additional milestone for suggested sequence of milestones and is not ES2.1, Figure licensing training programs did not occur until the completion of full expected to encompass every new reactor case.

16 1 Page 15 of close to, or following the start of the first accreditation before 103(g) finding Use of a draft K/A catalog is an option that was 20 Line 1 training programs Consider changing K/A catalog used for developing some AP1000 examinations The K/A catalogs for new plants were used bullet to identify a draft catalog but that may or may not be necessary in all as drafts, rather than as approved catalogs will be used future new reactors cases.

Additional effort may be required very Add a new bullet early in the early in the process to identify potential process to consider exam changes needed for licensing exams on plants methodology changes using new technology The 75day submittal to the facility for an NRC developed exam does not provide adequate time for technical review and validation. When Extend timeline for NRC developed the facility develops the exam and submits it to exams from 75 days to at least 90 Changed to 100 days based on this ES2.1, Form the NRC at 75 days (i.e., item #9) the exam has 17 days to account for the additional recommendation and lessons learned report 2.11, item #10 already been technically reviewed and time needed for facility technical from the 2018 Ginna examination.

validated. The timeline for an NRC developed review and validation.

exam that is submitted to the facility PRIOR to facility technical review and validation needs to be extended.

Section references ACAD 10001, Revision 1.

Revised eligibility requirements are contained in ACAD 10001, Revision 2, which will be Revise to reference ACAD 10001, released soon. Revision 2 or make a general Changes were made to ES 2.2 to recognize Additionally, the ACAD may be revised more statement to reference the latest ES2.2, page 3 ACAD10001 as part of the NANT guidelines for frequently than NUREG 1021 to update revision of the ACAD 10001 or 18 of 22 (starting NNABaccreditated training programs and to programmatic requirements for initial license latest revision of NANT academy on line 10) make this a more general recognition of the program content. A revision stating to guideline since the NRC process and not a specific revision of the ACAD.

reference the latest revision of ACAD 10001 or participates in the revision process latest revision of the NANT academy guideline per INPO guidance.

for operator eligibility and selection would be more accurate.

A regulatory basis for the number of hours considered extensive actual operating experience in ES 2.2 Section F.2.b, Multiunit Added basis to ES 2.2 for 520 hours0.00602 days <br />0.144 hours <br />8.597884e-4 weeks <br />1.9786e-4 months <br /> which is Examination Waiver, is not provided in the equivalent to the unit familiarity time period draft NUREG. Additionally, the proposed used in initial licensing training programs. The wording appears inadequate to address SNC recommends flexibility in the NRC plans to amend § 55.47 as part of the multiunit stations that are built simultaneously NUREG to provide NRC staff Alignment of Licensing Processes and Lessons to a Part 52 certified standard design.

guidance to address the varying Learned from New Reactor Licensing Though the additional operating experience degrees of unit similarity and time rulemaking (Docket ID NRC20090196).

ES2.2 Section may be necessary for operating a subsequent 19 lapses between construction for Specifically, the NRC would add a new set of F.2.b unit with significant differences from existing which licensees may request a criteria that justify a waiver of any or all of the unit(s), it is unnecessary when operating waiver under § 55.47 (e.g., Part 50 written examination and operating test for identical units. Because design certification plants decades apart vs. Part 52 applicants to be licensed on subsequent units at and ITAAC ensures subsequent units are plants months apart). a multiunit site that is under construction.

essentially the same, operators licensed on the Thus, the new criteria would address multiunit first unit built have already taken an stations that are built simultaneously to a Part examination applicable to subsequent units, 52 certified standard design.

provided the licensee can demonstrate that there are no examinable differences between the units.

ES2.3 Form Item a of Simulator references Form 2.31. Verify the correct form to Change made consistent with the 20 2.32, page 9 Should it be Form 3.41? reference, and consider if items a recommendation.

of 19 Consider switching items a and b under JPMs. and b should be switched.

Change made to clarify that this is simply a label A bullet has been added under Walkthrough Form 2.32 and not a new requirement for alternate path Criteria, that states specific designation if it Provide clarification if the added (revised op JPM. The added bullet is a check that the meets alternate path criteria. JPM cover bullet results in new/additional 21 test QA form), alternate path JPMS are labeled or marked as sheets typically designate alternate path. This requirements for alternate path ES2.3, page "alternate path" to distinguish them the same information is also typically in the body of the JPMs.

10 of 19 way that a time critical JPM is marked to JPM.

distinguish it for notification purposes.

R11 ES202 C.1.c description for control No change made. The statement from Revision manipulations states Every effort should be 11 to perform some of the manipulations in the made to perform at least some of the plant was removed because 10 CFR 55.31(a)(5)

ES2.2 C.3, manipulations on the actual plant and to allows these to be performed on the facility for Confirm that this clarification is not 22 page 5 of 22 diversify the reactivity and power changes for which the license is sought or on a plant required.

(line 7) each applicant. R12 does not include this referenced simulator. The information about clarification. This change appears to reduce diversifying them still exists in Revision 12:

importance of performing control "Applicants should perform diverse significant manipulations on the actual plant. control manipulations."

R11 ES202 C.1.c. description for control manipulations states while those on the plant Added a pointer in ES 2.2 to the definition of may be smaller but of sufficient magnitude for significant control manipulation in the glossary the operator to experience appropriate (ES8). The definition in ES 8 of Significant feedback (i.e., clearly observable effects on the Control Manipulations states: An operation ES2.2 C.3, plant, which could include maintaining power Confirm that this clarification is not (excluding those required for fuel handling) of 23 page 5 of 22 constant while performing a dilution/boration required. an apparatus or mechanism that directly affects (line 1920) evolution) as a result of the control the reactivity or power level of a critical reactor manipulation would be considered. R12 does by an amount of sufficient magnitude to allow not include this clarification. Without this for the observation of clear effects on the plant clarification, boration or dilution of any by the operator.

magnitude would seem to satisfy performance of a control manipulation.

This section discusses cold licensing of operators, and endorses Nuclear Energy Institute (NEI) 0613A, Template for an Industry Training Program Description as an acceptable method to acquire the knowledge and experience required.

Specifically it endorses Revision 2, and the attached safety evaluation. NEI 0613A Revision 2, in turn, references Regulatory ES2.2, A, 7th Guide 1.8, Qualification and Training of Consider revising this section to paragraph Personnel for Nuclear Power Plants, revision 3 Added statement: Facility licensees may seek state that facilities may seek other 24 Page 2 of 22, and ANSI/ANS3.11993, Selection, alternative methods for the cold licensing of alternative methods to meet these Lines 11 thru Qualification and Training of Personnel for operators.

requirements.

20 Nuclear Power Plants. Regulatory Guide 1.8 has been revised to revision 4 since the SER for NEI 0613A was issued. The ANSI standard has also be revised to the 3.12014 revision, and is no longer endorsed by Regulatory Guide 1.8, revision 4.

Additionally, NEI 0613A does not specifically address newer advanced designed, passively cooled reactors that have alternative staffing requirements.

The NANT guideline ACAD 10001, revision 1, is ES2.2, B, 3rd identified as an acceptable method for Update this reference to the paragraph meeting the eligibility requirements for newest revision if it is published 25 Page 3 of 22, Same disposition as comment 18 education and experience. This document is prior to the approval of this Lines 12 and currently being NUREG.

13 revised to the next revision.

This section states that for control manipulations performed on a simulator to count towards the five required control manipulation, they are required here to be Recommend revising section C, 2, performed on a plantreferenced simulator 3 to state clearly how, during cold that meets the requirements of 10 CFR licensing of operators, control 55.46(c). This would be a plantreferenced manipulations can only be simulator (PRS), not a commissionapproved performed on a plant referenced simulator (CAS). simulator, not on a commission ES2.2, C, 3, Also, section G.2.a, on page 15 of this section, approved simulator. Also consider The NRC staff is considering the issues 3rd lines 24 thru 26, states that cold plant removing the allowance for cold highlighted in this comment in the Parts 50 and paragraph 26 applicants may defer these manipulations until plants to defer the control 52 alignment rulemaking. The proposed Page 5 of 22, they can be completed on a plantreferenced manipulations until a PRS is changes are described in Regulatory Basis for Lines 20 thru simulator. available, in section G.2.a. Public Comment (ML20149K680), Appendix E.

22 The current definition of a PRS in 10 CFR 55.46, Also revise the definition of plant (c)(2)(i) states that the simulator models referenced simulator (in the relating to nuclear and thermal hydraulic glossary) to include how significant characteristics that replicate the most recent control manipulations can be core load in the referenced plant. performed during the cold license It is not clear how the first operators at a cold phase.

plant will be able to do this if plant performance data is required before a simulator can meet this PRS requirement.

ES2.3, page 4, 27 Actions is used twice in this sentence. Delete the 2nd actions. Deleted the second "actions."

line 15 ES2.3, page 5, Delete one instance of this Deleted the first occurrence of this statement in 28 This is redundant to page 4, lines 2528.

lines 3841 sentence. lines 2528 on page 4.

This step states that the written examinations Recommend adding example of and operating test should not overlap in their controller failing upscale on the content. However, the example provided is too operating exam and the same The example was revised to clarify: For ES2.3, Page 6 limiting as it defines overlap exists if the same controller failing downscale on the example, overlap exists if the same pressurizer of 19, pressurizer level controller concept is tested 29 written exam. This illustrates that level controller failure (i.e., fails high) is line 3436, during a simulator event and in a written this is different enough to both be tested during a simulator event and in a written After C. 4. c. examination question.

allowed on the same examination examination question.

The term concept should be refined to allow and does not violate the overlap subsets of a concept to be tested if they have restrictions.

different recovery actions.

1. note at bottom of page 11 and page 16 ES2.3, page should state exam or test vice outline.

30 11 and 16, Update form as stated. Change made in accordance with the comment.

This looks like a copy and paste error from Form 2.32 Form 2.31.

Form 2.35 Instructions for Written Clarification was added to what the term Form 2.35 and ES4.2 have conflicting Examination Recommend aligning the two 'difficult level' means in ES 4.2 by replacing that information covering the written examination 31 Review sections to reflect the LOD 15 term with a better description. Correlating level of difficulty. The form uses LOD rating 1 Worksheet requirement. difficulty level with LOD is outside the scope of 5 and ES4.2 discusses a percentage.

ES4.2, Page 9 this revision.

of 29 Lines 412

During a public meeting the NRC stated that on Recommend adding a box on the No change required. Form 2.34, "Written this form they, added check for technical actual form to review the provided Examination Quality Checklist" includes a check Form 2.35 reference and direct lookup questions.

reference to make sure it is not a for each question to have a technical reference.

Written There is not a box for references on the form.

direct lookup and that the Direct lookup questions are to be marked as 32 Examination In the instructions section for the worksheet, reference does not provide LOD 1 on Form 2.35 in accordance with the Review Item #2 Level of Difficulty, it discusses that a additional information that would instructions on this form. Step 8 on Form 2.34 Worksheet direct lookup should be marked LOD 1. The help answer another question on includes a check that reference material does intent should be more clearly stated in the the test. not give away answers.

form.

Each JPM must contain at least two critical ES2.3, Page steps. An exception to this requirement 13 of 19, should be made for admin JPMs. The Admin Form 2.33 JPM task standard should be to match the Recommend that the Admin JPM No change made. Regarding the example of EAL Instructions answer key. The Admin JPMs should not be task standard should state match determinations, two typical critical steps are 33 for Completing held to a minimum number of critical steps. the answer key vice require a the EAL classification within the time period (15 the JPM Table (Example: How many steps do you have to minimum of two critical steps. minutes).

ES3.2, Page 7 have for an EAL call? They should make the of 18, proper declaration in the required amount of Line 10 time.)

Section 3, Operating Tests ES3.1, page 4 Steps 13.b and 13.c cover the same topic, using Consider combining the two steps Deleted the sentence in 13.b "Consider doing 34 of 5 (lines 35 JPMs to test knowledge of the differences into one step. JPMS on the other unit."

and 38) between plants on multiunit sites.

ES3.2, B.3.a. lists examples for Conduct of Ops This K/A was deleted in the latest revision for Topics, including access controls for Access controls for the PWR, BWR, and AP1000 K/A catalogs, but vital/controlled plant areas. However, Rev 3 vital/controlled plant areas ES3.2, page 2 still remains in the ABWR catalog (2.1.11).

35 of NUREG1123 deleted KA 2.1.13, Knowledge should no longer be used as an of 18 (line 29) Deleted the example per the recommendations, of facility requirements for controlling example. Recommend and replacing with a new example is not vital/controlled access. Thats the only KA replacement with a new example.

needed, as there are several examples listed.

statement that was applicable to that example.

ES3.2, page 3 Consider closing bulleted lines (i.e.,

Spacing between bullets adjusted to match the 36 of 18 (lines 26 Bullet formatting is different than 3.a and 3. b. no space in between each bulleted rest of ES3.2.

32) line).

This K/A was deleted in the latest revision for ES3.2, B.3.c. lists examples for Radiation Radiation work permits should the PWR, BWR, and AP1000 K/A catalogs, but Control Topics, including radiation work ES3.2, page 3 no longer be used as an example. still remains in the ABWR catalog (2.3.4).

37 permits. However, Rev 3 of NUREG1123 of 18 (line 31) Recommend replacement with a Deleted the example per the recommendations, deleted KA 2.3.7, the only KA that was new example. and replacing with a new example is not associated with radiation work permits.

needed, as there are several examples listed.

The clarification that a critical step might not involve a verifiable action and two examples are ES3.2, page 7 Provide clarification whether or new for Rev 12. The specification to include 38 of 18 (lines 5 Is this new text?

not this is a new comment. JPM termination criteria was added because in 6, and 2730) practice examination authors included criteria for when a JPM should be terminated.

There is not a definition of alternate path JPM for the glossary. The following clarification was added on page 9: An alternate path JPM allows the NRC to evaluate whether the applicant has the skills and knowledge at the level needed to safely operate the system, and the ability to recognize and diagnose an unexpected system Definition/standard of alternate path JPM response and then execute one or more ES3.2, page 9 39 should be clearly stated in this section since Add clarification to this section. alternative paths within the wide spectrum of of 18 (line 20) the term is introduced. procedures under the applicants cognizance; these are typically procedures that the applicant would not otherwise be examined on.

Additionally, added to 1. Success Path the following: The success path must be an alternate path that differs from the normal success path in order to test the applicants ability to use an alternate operation.

Sentence is incomplete; should be preceded by "Alternate path JPMs are used...". Revised the sentence in accordance with the ES3.2, page 9 Additionally, it appears that there are two recommendation and deleted the term 40 of 18 (lines 26 Revise the sentence.

separate subjects: 1) what the intent of "observe" since this implied that it was done 28) alternate path JPMs is; and 2) how the during JPM implementation.

alternated path is implemented.

ES3.2, page The phrase senior reactor operator is spelled Consider using SRO vice senior 41 12 of 18 Form out, unlike RO, even though it is a standard Change made in accordance with the comment.

reactor operator for consistency.

3.21, step 3 abbreviation (on list of abbreviations).

ES3.2 D.1.c requires JPMs to contain at least two critical steps.

Some safety significant tasks may require completion of a single critical step to No change made. This requirement is aligned demonstrate competent performance. For with Rev 11 guidance for limiting the use of example, once RCP trip criteria are met simple one step JPMs. Each JPM must contain ES3.2 D.1.c, following a small break LOCA, the verifiable Consider removing requirement at least two critical steps because this is what is 42 Page 7 of 18 critical step is simply stopping the RCPs. for two critical steps for a JPM. practiced. The example provided in the (line 10)

Requiring two critical steps for a JPM may comment for tripping RCPs could be split into eliminate testing of some safety significant more than one critical step, for example, each tasks from the examination bank or require RCP trip is one critical step.

extensive revision of JPMs to include two critical steps without a corresponding increase in safety significance.

R12 ES3.2 D.1.d requires including a statement that describes the conditions to be met for the NRC examiner to terminate the JPM, considering both applicant success and failure to meet the task standard.

Although applicant success with completing the task may be predictable given confines of task performance outlined in the JPM, Added clarification that this is not a Consider removing requirement potential failure modes may be less so. For requirement by adding the words "such as."

ES3.2 D.1.c, that implies JPM termination example, the applicant may not be able to This was added in Rev. 12 because it is what is 43 Page 7 of 18 requirements require applicant locate the appropriate equipment in the field, being practiced and is consistent with the steps (line 2830) success and failure conditions to perform the task on the wrong Unit, fail to of developing a JPM. The intent is not to make be included in the JPM.

correctly engage a manual override, etc. the examiner list ALL possible failure modes.

Including all these potential failure paths into the evaluation tool may make it unwieldly and confusing. ES3.2 D.1.d contains a requirement to Include a statement that describes the conditions to be met for the NRC examiner to terminate the JPM. This guidance would seem adequate.

Primarily from Appendix D of R11. A better title for this standard might be "Generic...."

Consider making this an Attachment to ES 3.4 Consider making this an rather than a stand alone standard. Attachment to ES 3.4 rather than a ES3.3, page 1 Added "General" to the title of this section. The For line 26, The following paragraph simply stand alone standard.

44 of 17 (line 1 material in this section does not need to be describes what the objective of using a Sim Consider placing the paragraph and line 26) moved to another section.

Scenario is and does not provide any direction after line 26 in the preceding for determining objectives of any given introduction.

scenario. This information is better suited for the preceding introduction.

Text states the guidelines also apply to Clarify how simulator testing requalification examinations. It is unclear what guidelines specifically apply to ES3.3, page 1 specifically applies to requalification programs requalification programs, if at all, Added clarification that this would apply to NRC 45 of 17 (lines 5, 6 and what is inspectable per IP 71111.11, if and if IP 71111.11 will change to conducted requalification examinations.

and 22, 23) anything. Clarify if the guidelines only apply incorporate those specific when the NRC writes a requalification exam requirements.

per ES6.

Step 2 states the IC should be Revise to state: the IC should be representative of a typical plant status with representative of a typical plant ES3.3, page 1 various components, instruments, and status, which may include various Added clarification for ICs in accordance with 46 of 17 (lines 42 annunciators out of service. Although this is components, instruments, and the recommendation.

and 43) not a change from Rev. 11, it seems to denote annunciators out of an unnecessary requirement by not allowing service.

clean ICs.

The last sentence of the paragraph states: As such, the operating test should not include such events they are necessary to set the stage Add unless between events ES3.3, page 2 Change made in accordance with the 47 for subsequent events or to test the SRO and they in the sentence to of 17 (line 18) recommendation.

applicants knowledge of TS actions. It correct the statement.

appears the word unless should be included before they.

The word with does not seem to belong in the following sentence.

ES3.3, page 5 If this jump is used, the crew must receive Remove with or revise to clarify Change made in accordance with the 48 of 17 (line 10) with a turnover or cue addressing any relevant intent. recommendation.

plant conditions that changed due to the time compression.

This states a component/instrument failure that occurs before the major event could be credited for actions before AND after the major event provided the actions to deal with the failure are different when comparing the ES3.3, page 8 response before and after. The provided Provide a positive example of using Change made in accordance with the 49 of 17 (starting example of excess letdown demonstrates this allowance to add clarity of recommendation. Added a positive example of at line 8) when this could NOT be used since excess acceptance. how to use this allowance in ES 3.3.

letdown actions are the same both before and after the major event. Is it acceptable to count the same malfunction twice, once before and once after the major, provided the actions to address the failure are different?

ES3.3, page 10 of 17 item Formatting is different for the Combustion Close bulleted lines (i.e., no space Removed space between bullet items for list of 50 3, (lines 42, 43) Engineering PWR as compared to the others in between each bulleted line). CE contingency procedures.

and page 11 of (double spacing between bulleted lines).

17 (line 1)

A scenario should be designed to run ES3.3, Page approximately 60 to 90 minutes.

11 of 17 11. The operating test for initial licensing will Line 18 normally consist of two or three scenarios each Recommend changing ES1.2 to Changed ES 1.2 page 6 of 6 to 1 1.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> per 51 ES1.2, Page 6 lasting approximately 1.5 to 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. align with ES3.3. scenario for initial licensing examinations.

of 6, The scenarios should run 4560 minutes with a lines 56 requal crew during validation and should run 4590 minutes for the ILO students.

Recent changes to the BWR Owners Group guidelines have changed the setup on some Consider adding statement that contingency procedures. Specifically, the identified contingency Added the following Note: Because of the ES3.3, step 5, Alternate Level Control is no longer a 52 procedures need not be General Electric BWR EOP structure, these may page 10 of 17 separate contingency procedure and has been standalone EOPs and may be not be standalone procedures.

added to the RPV Control EOP. It is still an included in the base EOPs.

EOP contingency path/procedure and should be treated as such.

Raising the grading threshold (i.e., making it harder for an applicant to pass) should be limited to closing gaps with licensing applicants that were determined to not display the minimum requirements necessary to be licensed as competent licensed operators.

There is no evidence that the current grading ES3.3, step k, criteria is inadequate to license competent The NRC reconsidered the grading of CPDs Page 11 of 17 applicants and the change was made as one of based on industry feedback and a CPD has been (line 45) the actions to improve grading clarity and Recommend maintaining a missed returned to a 3point deduction; two or more 53 ES 3.6, page 4 consistency. Many of the other changes, CT or CPD as a 3point deduction. CPDs will result in a failure of the simulator of 27 (line 43) including the new SPD category will help operating test regardless of the calculated Table 3.61, improve grading clarity and consistency competency area scores.

page 6 of 27 without the need for increasing the significance to automatic failure for a missed CT or CPD. The industry also performed a study of a smaller population and determined that there will be increased failures of licensed operator candidates during the simulator operating examination.

The NRC reconsidered the grading of CPDs Additionally, the elimination of a broad based on industry feedback and a CPD has been category of items contained in Rev. 11 that can returned to a 3point deduction; two or more constitute a critical task (e.g., scrams) while CPDs will result in a failure of the simulator maintaining the requirement of at least 2 operating test regardless of the calculated preidentified critical tasks will potentially make competency area scores. The CT criteria in Rev scenarios longer, more complex and increase 12 will not make the scenarios longer. The difficulty due to the need to make CTs EOP concern with having one CT per scenario means based similar to Rev. 10. Because the studies Recommend maintaining a missed 53 continued that some applicants might not receive a CT.

conducted by the LOFG and NRC staff used CT or CPD as a 3point deduction.

The NRC chief examiner has the ability to previously graded exams as the subject matter, control the scenarios if they are getting too long the potential consequences of this change or complex. The majority of Revision 11 CTs could not be assessed. These losses of should still be usable. The major change to the competent operators from talent pipelines will CT methodology was removing unintentional negatively impact organizational staffing and reactor protection system (RPS) or ESF no clear safety issue exists to warrant the actuations from the list of what qualifies as a NRCs imposition of this burden.

postscenario CT.

Proposed Replacement Language for Page 13 of ES3.3: The developer should apply the Critical Task Methodology Step C.1 is following guidance to identify and Identifying ScenarioSpecific Critical Tasks designate CTs in conjunction with directs scenario developers to apply guidance facility CT lists or in the absence of to IDENTIFY and DESIGNATE CTs. The list of such a list:

items provided mainly describe applicant Do conditions exist which The NRC agrees with the commenter that the actionswhat applicants must do or not do represent significant safety instructions for identifying critical tasks were when responding to plant conditions to challenges? Examples include the written in terms of applicant actions and not satisfactorily address the CT. When developing following: useful for selecting critical tasks during scenario CTs, the author can only know what actions Conditions that warrant development. The NRC made changes to ES 3.3 ES3.3, page the applicants SHOULD take, based on the initiation of emergency Critical Task Methodology instructions for 54 12/13 of 17 procedural guidance and projected plant depressurizations (BWR) identifying critical tasks that are "task (line 38) response. The author CANNOT know, at this Conditions requiring orange or orientated," for example: "actions that directly point in the process, what actions the red path CSF response (W and lead to the restoration of one or more safety applicants WILL take when they perform the AP1000) functions" and "EOPdirected actions that are scenario. The list of bullets on page 13 appear Conditions that warrant essential to an events overall mitigative to be a description of how to determine if a performance of FRG transition (CE) strategy."

post scenario CT has been created. Rev. 11 CT Conditions that warrant methodology describes how to determine declaration of SAE or GE whether a proposed malfunction is a safety Conditions which are beyond the significant CT. control of the crew or which are irreparably introduced by the scenario should not be designated as CTs.

This requires simulator guides to include both the procedure steps and the verifiable actions that the applicants are expected to take. For example, instead of only stating, The BOP starts emergency diesel generator A in Change was made for clarity. The intent is that accordance with Procedure XYZ, Steps Consider adding clarification to the Scenario Guide or Form 3.32 contains the 1.10-1.25, reference the procedure steps and paragraph that states, If a verifiable actions and not only procedure step ES3.3 A.4, include verifiable actions for that set of procedure step describes the numbers because this provides clear objective 55 Page 3 of 17 procedure steps that the BOP is expected to verifiable actions, including the grading criteria for grading purposes. There was (line 1216) perform. This requirement could be step description alone would no change to the Rev 11 requirement in interpreted to require separate clarification of satisfy this requirement. Appendix D to "include every required operator verifiable steps when step performance action."

inherently contains verifiable actions. Adding a separate clarification to identify and specify which activities are verifiable actions may make the evaluation tool unwieldly and confusing.

This requires simulator guides to identify and document events that impact TS functions (such as for inoperable instruments). This Consider clarifying requirement to requirement is reasonable prior to a major state, At a minimum, identify and ES3.3 A.4, transient. For example, a design basis accident document those events that 56 Page 3 of 17 Added "prior to major transient."

and resulting emergency procedure directed impact TS functions (such as for (line 3132) actions may result in impact to TS functions inoperable instruments) prior to but evaluating this impact to TS is less major transients.

important to safety than appropriate accident response procedure implementation.

Deleted the sentence in question because it was ES3.3 page 12 This does not seem to fit the concept of "risk 57 redundant to the first sentence in the of 17 (line 2) informed."

paragraph.

R12 ES3.4 A. contains guidance for developing simulator scenarios while limiting overlap. R11 No change was made because ES 3.4 A states:

ES3.4 A, Page ES301 D.5.b recognized that because of a Because of a limited number of methods for 58 1 of 9 (line 22 limited number of methods for adding Consider adding this clarification.

adding reactivity, reactivity manipulation events

30) reactivity, reactivity manipulation events can be repeated from past examinations.

would be exempt from prescribed overlap limits. This allowance was not included in R12.

Segregating the steps for completing the outline (Form ES 3.31) from the steps of completing the scenario guides (From ES 3.3 Consider separating the steps for ES3.4 page 2 2), into separate sections would better completing the outline from the Change made in accordance with the 59 of 9 (line 35) describe the process of developing a Simulator steps of completing scenario recommendation.

Scenario and clearly identify which elements guides.

are part of the outline development and which are part of the scenario guide development.

The additional limitation that the scenario should not duplicate operator tasks that appear on the JPM portion of the operating test or the written examination unless the operator actions for the same task are different for the related simulator event. Since No change was made. This instruction existed JPMs and simulator scenarios evaluate skill in Rev 11, on Form ES3013, to check for performance, this restriction is warranted to ES3.4 A, Page Consider revising limitation to overlap with the entire exam, therefore this is prevent duplication. However, written 60 2 of 9 (line 18 exclude consideration of the not a new requirement. One concern is that examinations evaluate knowledge that may

20) written examination. applicants could be penalized more than once apply to numerous normal and abnormal for knowledge or performance deficiencies for tasks. Evaluating all written examination the same topic/task.

questions to eliminate potential overlap for the skill portion of examinations will be subjective and time consuming. In addition, knowing how a task should be performed and performing actions are inherently two different evaluations.

Information about the use of surrogates should be in the standard on administration (ES 3.5 Section D). Scenarios are not designed around the use of surrogates, unless the scenario will be used only with a set of crews The steps for surrogate operators have been Determine if this section should be ES3.4, Page 3 that will use a surrogate in the same position moved to a separate Subsection B: Determining 61 kept in this location or a subtopic of 9, Section 2 for all iterations of the scenario administration. the Number of Scenarios to Develop and Use of to item 1.

If kept here, this should not be listed as a Surrogate Operators.

separate step in the "development process." It seems to be more appropriate to list this as a subtopic to item 1 (Determining the number of scenarios).

Recommend adding the following example: If a watch team has two The example on line 5 could more clearly show SROIs with one RO, both SROIs ES3.4, page 3 the benefit of the application of the would not need to be ATC The examples in this section have been revised 62 of 9 requirement on determining the number of requiring 3 scenarios. You could based on this comment.

(line 5) scenarios required based on crew composition. complete all requirements with two scenarios, eliminating requirement for 3rd scenario Added that manual control of automatic function event must include verifiable actions.

R12 Table 3.42 Events and Evolutions for Also added: d. Manual control of an automatic License Level includes minimum number of function can be for a safety or a nonsafety events or evolutions for each type of license related system that has both an automatic and and now requires Manual Control of Automatic a manual control feature; the control feature is Action for RO and SROI positions during the Consider clarifying any restrictions not limited to component controllers. For ES3.4, Page 4 simulator examination. This requirement does for crediting manual control of example, the event involves the failure of the 63 of 9 (line 28 not contain any clarification on what automatic function or provide automaticstart feature of a pump and the 30) constitutes manual control of an automatic examples. applicant has to manually start that pump. The function. For example, does taking manual event must involve the applicant manually, with control of an automatic secondary side verifiable actions, taking control of the feedwater heater drain valve satisfy this automatic function. For example, verification requirement? that a digitalcontroller auctioneered out a bad input would not be considered a verifiable action for this event.

This paragraph uses the term examination ES3.4, page 1 developers. Other text uses examination ES 3.2, 3.3 and ES 3.4 revised to use Recommend using examination 64 of 9 authors and examination writers. For "examination author" consistently instead of authors throughout.

(line 9) consistency and clarity, using a common term scenario/exam developer or scenario author.

is suggested.

Change made to clarify that this is required (as ES3.4, page 1 Bullet 3 regarding Scenarios extracted Consider deleting bullet 3 or it was in Rev 11) and therefore was added to 65 of 9 should be deleted since this requirement is combine with bullet 2. the instructions for significantly modifying bank (line 39) encompassed in bullet 2.

scenarios in Rev 12.

The bullet states; The scenario should not duplicate operator tasks that appear on the JPM portion of the operating test or on the written examination unless the operator Recommend removing written actions for the same task are different for the examination check or add related simulator event. Also see sections; ES flexibility similar to Rev. 11 that 3.1 page 2 & ES2, page 15, step 3 on form 3.2 This is not a new requirement. Clarification was had within acceptable limits.

2. Scenarios and written exams are performed added by revising the bullet in ES 3.4 to state Consider wording such as; Efforts ES3.4, page 2 in different contexts and are separated in time. the following:

should be taken to minimize tasks 66 of 9 As currently written, this could cause a *The operator actions for the scenario that are the same on the both the (line 1820) significant level of increased effort to cross events/evolutions should differ from operator operating test and written check thirty to fifty (or more) scenario tasks that appear on the JPM portion of the examination.

elements against 100 written exam questions operating test or on the written examination.

Also, consider specifically calling with very little benefit, and risk to examination out that this is intended for tasks quality. As discussed at the recent public associated with malfunctions.

meeting held on Oct 21, 2021, it appears that the intent of this requirement is for tasks that result in actions taken when due to malfunctions.

ES3.4, B.1, third bullet conflicts somewhat Example changed to say: For example, a crew with the example immediately after. Third consisting of two ROs and one SROI will bullet says SROI needs to be evaluated in The example after the third bullet normally require three scenarios to evaluate ES3.4, page 3 either the BOP OR ATC position. There are no should say while the SROI each applicants performance on the primary 67 of 9 conditions similar to the 3015 in Rev. 11 (Form applicant is in a reactor operator plant component controls; however, a (line 2) 3.41 in Rev. 12). The example immediately position. surrogate SRO may have to fill the supervisory after seems to specify ATC position since lead role while the SROI applicant is in the ATC/BOP operator was defined as the ATC in the bullet position.

before.

ES3.4, page 3 This is essentially that same as, and redundant Change made in accordance with the 68 of 9 Eliminate redundancy.

to, what is on page 2, lines 22 26. recommendation.

(lines 2529)

Regarding use of surrogatesregional Eliminate need to consult with management should have the authority to NRR on use of surrogates to No change. NRR operator licensing program ES3.4, page 3 permit the use of surrogates in order to streamline performance of the tracks this type of information. It should not 69 of 9 streamline performance of the operating test operating test. Recommend take long to get concurrence nor impact the (lines 3133) without NRR involvement. This would increase replacement with regional branch exam administration schedule.

efficiency. chief concurrence.

Rev 12 leaves only EOPBased CTs available to meet the at least 2 criteria. Maintaining the requirement to have 2 CTs per scenario while removing a large batch of what constitutes a CT will make scenarios potentially longer and Recommend changing CT criteria No change was made. The CT criteria in Rev 12 more complicated. In addition, considering a to at least 1 (versus 2) per will not make the scenarios longer. The CT failure will result in a critical performance scenario since the population of concern with having one CT per scenario means deficiency (CPD) and an automatic failure of events in a typical scenario that that some applicants might not receive a CT.

the operating exam, existing CTs that are not can result in a critical task have The NRC chief examiner has the ability to ES3.4, page 4 commensurate with a penalty of this severity been reduced. control the scenarios if they are getting too long 70 of 9, Table 3.4 will likely not be applied in future initial Additionally, limiting the maximum or complex. The majority of Revision 11 CTs 1

licensing examinations. For example, at some number of CTs to 2 would reduce should still be usable. The major change to the BWR stations, inserting a manual scram on a the likelihood of having scenarios CT methodology was removing unintentional 2nd control rod drift is categorized as a critical that are too long or complex and reactor protection system (RPS) or ESF task. The safety significance is avoiding may help in consistency during actuations from the list of what qualifies as a potential fuel damage due to an unanalyzed scenario development. postscenario CT.

control rod pattern. This seems likely to be omitted as a CT on Rev. 12 based exams due to the severe penalty not aligning with the safety significance (i.e., potential fuel damage).

No change was made. Scenario set is defined in There is no definition as to what constitutes a ES8. The Rev 12 requirement is: One EOP Recommend adding a statement scenario set as it applies to contingency contingency required per scenario set, not one identifying a scenario set means ES3.4, page 4 EOPs. The requirement is for one contingency EOP contingency per scenario nor per applicant.

the scenarios the individual 71 of 9, Table 3.4 EOP per scenario set, but theres nothing that These are targets but a scenario may also be operator will see and not the set of 1 (line 3) clarifies if that means each operator must be acceptable without meeting all of the target scenarios selected for the overall evaluated with a scenario that contains a numbers. This is also for examination security class.

contingency EOP. to prevent applicants from "expecting" an EOP contingency procedure.

This is not a new requirement for Rev 12, Having at least 1 Manual Control of Automatic Add additional guidance on what instead an existing Rev 11 element of scenarios Function event for RO and SROI applicants qualifies as manual control of an ES3.4, page 4 (from the RO competency checklist) that has represents a new requirement as compared to automatic function. For example, 72 of 9, Table 3.4 been relocated to the list of events and Rev. 11. There would be benefit to including does placing the backup EHC 2 evolutions for each applicant (see ES 301 page additional guidance on what falls into this pressure regulator in service 16 of 33). Additional guidance has been added category. qualify?

for this item (see Disposition for comment #63).

The last sentence in this paragraph states the ES3.4, page 5 ODCM cannot be used to meet the minimum Clarify whether the TRM can be No change was made. Clarification is already 73 of 9 (lines 38 TS evaluation requirement. Can the TRM be used to meet minimum TS provided in the glossary definition of TS which

40) used to meet the evaluation requirements. includes TRM.

minimum TS evaluation requirement?

ES3.5, Page 1 of 13 (line 40)

Page 11 of 13 The term error is used in several cases in ES (lines 21 and Revised ES3.5 to replace the term 'error" with 74 3.5. Should performance deficiency be used Clarify/modify as necessary.

33) "performance deficiency."

instead?

Page 12 of 13 (lines 43 and 46)

The last sentence in section A.2 is missing some words. It should probably state Obtain Modify as necessary by adding ES3.5, page 1 concurrence from the NRR operator licensing Changed made in accordance with the 75 obtain and if to the sentence of 13 (line 29) program office if more than 30 days will elapse recommendation.

as highlighted.

between the completion of one and the start of the other.

This statement seems to be unnecessary since it is okay to do it either way. Additionally, the ES3.5, page 2 exception to the examiner of record 76 Consider removing this statement. Deleted step 4 on page 1 of 13.

of 13 (line 1) administering each JPM to their assigned applicant is addressed in Part H (Specific instructions for administering JPMs).

Moved step 7 to check simulator ES3.5, page 2 This should be included in the Consider placing in the differences/deficiencies from ES 3.5 to ES 2.3 77 of 13 (lines 13 development/review process and not here. development/review process. for the NRC chief examiner to check during op 15) test review.

R12 ES3.5 A.9 prevents the facility licensee No change was made. The NRC does not have a from audio or video recording the operating clear process for how to use videotaped ES3.5 A.9, test. Video recordings do not necessarily scenarios after examination administration.

78 Page 2 of 13 Consider removing this restriction.

require additional observers to obtain the Additionally, states have different laws for (line 2729) recording and recordings can be controlled to allowing videotaping so this may not be satisfy examination security requirements. implemented consistently across the regions.

This wording would imply that only the Chief Examiner administers the optest. This should probably read: "Ensure that each applicant ES3.5, page 2 Merged this statement number 11 with 79 identified on Form 2.21, "List of Applicants," is Consider combining with item 13.

of 13 (line 35) statement number 13.

administered the applicable portions operating test specified on the form." Consider combining with item 13 listed below.

No change was made. This is material that ES3.5, page 3 This needs to be more specific as to what Consider making this statement 80 supports the licensing decision. MD 3.53 has of 13 (line 2) materials this sentence refers to. more specific.

more information on recording keeping.

ES3.5, page 3 This should be moved to the next section as it Consider moving to the next Change made in accordance with the 81 of 13 (lines 13 applies to all examiners. section recommendation.

16)

Grammar/typoremove that ES3.5, step 7, An applicant may request that the Change made in accordance with the 82 page 4 of 13 Remove that from item #7.

administration of his or her operating test recommendation.

(line 24) without 24 extraneous observers.

No change was made. This information is ES3.5, page 4 Move the content of ES 3.4 Section B.2 to this Consider moving content of ES 3.4 83 important to remain in ES 3.4 when planning of 13 (line 29) standard. Section B.2.

how many scenarios to develop.

Grammar/typo in section 16.a - perform should be performed Verify that each examiner observed that his ES3.5, page 9 Change made in accordance with the 84 or her applicant perform the required 20 Change perform to performed.

of 13 (line 20) recommendation.

number of transients and events to allow adequate evaluation of all required 21 competencies.

This states an SPD exists if an avoidable ES3.5, page 9 emergency action level entry or escalation is The NRC agrees with the comment and has of 13 (line 42 reached. Consider a threshold of ALERT or changed the threshold for SPDs to 44) 85 An error resulting in EAL entry or escalation at higher for meeting the criteria of entry/escalation to an Alert or higher but less ES3.6, page 5 the Unusual Event level does not require an SPD. than the CPD criteria. Change made in ES 3.5 of 28 (lines 12 staffing the emergency response centers or and ES 3.6 for SPD.

13) have increased safety consequences.

The note states that subsequent RPS/ESF actuations that do not alter equipment ES3.5, page 9 alignments are not treated as additional Consider revising current note or of 13 (line 32 significant performance deficiencies. Other add an additional note that single

40) Added note for this part of ES 3.5 for Single 86 examples that shouldnt be considered an SPD channel actuations or half ES3.6, page 5 channel RPS/ESFAS actuation.

would be single channel actuations or half scrams should not be considered of 38 (lines 5 scrams. These would not alter equipment an SPD.

10) alignments or only open reactor trip breakers that would not result in an automatic scram.

This states an SPD exists if performance deficiencies result in an unplanned power ES3.5, page 9 change of more than 10 percent rated thermal Consider adding a statement that of 13 (line 46 power. The intent is that the SPD is due to placing the plant at a lower power

47) inadequate power control. At times unit Added note for this part of ES 3.5/ES 3.6 for 87 level as a result of conservative ES3.6, page 5 supervisors could direct reducing or controlling unplanned power changes.

decision making would not apply of 38 (lines 15 power at a lower power level due to to this criterion.

16) conservative decision making. Conservatism is an operator fundamental that is strongly reinforced by the utility training programs.

To add clarity, consider modifying As written, step 17 may cause confusion. with the following wording; If a simulator scenario includes emergency Since the simulator operating plan event classification, because the simulator tests for the initial licensing operating tests for the initial licensing examination are conducted with ES3.5, step examination are conducted with only one only one applicant in the SRO 17, page 10 of Change made in accordance with the 88 applicant in the SRO position, the NRC does position, the NRC does not require 13 (lines 14 recommendation.

not require the SRO applicant to complete an the SRO applicant to complete an 18) emergency classification within the normal emergency classification within the event classification period of time. The normal event classification period scenario does not need to include event of time. The scenario does not classification. need to include event classification.

This is something that should be part of the ES Step has been revised as follows: deleted the ES3.5, page 1.2 briefing not here. This is something the Consider removing from this statement about the examiner encouraging the 89 11 of 13 (line applicant should do without be prompted or section. applicant to use reference material since this is 15) encouraged during the exam administration. contained in ES 1.2.

The discussion on Critical Performance Deficiencies, states that a significant performance deficiency (SPD) includes performance deficiencies (PDs) resulting in In ES 3.5 and ES 3.6, a note was added that unplanned power changes of more than 10 ES3.6, Section Provide clarification on the intent placing the plant at a lower power level as a 90 percent rated thermal power. This seems to B.4.a of this statement. result of conservative decision making would indicate that acting in a conservative manner not be a SPD (see comment # 87).

to reduce the energy state of the reactor could result in a SPD. Is this the intent of the statement? The statement seems to be open for a wide variation in interpretation.

PD and PDs are used throughout this ES3.6, section, as is performance deficiency. CPD Add PD to Abbreviations and Change made in accordance with the 91 throughout and SPD are included in the Abbreviations and Acronyms for consistency. recommendation.

Acronyms section, but PD is not.

This paragraph introduces grading subjectivity, especially if the critical task was met/completed. PDs such as procedure usage or place keeping errors or intervention by other crew members may be graded as a CPD. Consider the following In many cases, there may be no way to recommendations to this area to determine if the applicant would have caught reduce subjectivity to benefit and corrected the error in a reasonable examiners and to account for the amount of time during the scenario. While we increased No change was made as the recommendation recognize that there will be a level of safety significance. creates a more complicated grading rubric. The judgement by the examiner when evaluating a A PD associated with performance NRC evaluates the individual performance of ES3.6, page 4 performance deficiency in this area, this will of actions in support of completing applicants. Regarding the concern that 92 of 27 (lines 35 likely lead to additional candidate appeals if a a CT requiring intervention by applicants may be less willing to coach or 40)

CPD is assigned, resulting in automatic other crew members to complete assistant one another has implications for the simulator examination failure even if the the CT would be an SPD if the performance evaluation of the other crew associated CT itself was completed. There may applicant would not have been members.

have been an opportunity for the candidate to able to identify and correct the selfidentify and correct the error without the error in a timely manner (i.e.,

intervention but control room teams are before the CT would be trained and expected to immediately coach unrecoverable).

and correct behaviors when standards are not being met or errors are identified. Additionally, this change may result in less challenge between control room team members.

This should be clarified to indicate applicability ES3.6, page 5 to entry/escalation not involving loss of Added statement for SPD criterion to 93 Consider adding clarification.

of 27 (line 12) multiple fission product barriers (which would distinguish from CPD criteria.

be a CPD)

ES3.6 Page 6 of 27 (line 16) The term error is used in several cases in ES Clarify/modify the use of error as Page 8 of 27 3.6. Should PD or Performance deficiency Change made in accordance with the 94 desired and remove the word for (line 6) be used instead? Or in some cases, error can recommendation.

on page 6, line 16.

Page 9 of 27 be eliminated from the sentence.

(lines 6 and 9)

No change was made. Followup questions must be tied to an observed PD so this is not a standalone situation. The CPD/SPD criteria is for observed performance not followup questions. See ES 3.5 E Instructions for Use of Followup questions. A CPD would not be assigned based on applicant response to a CPDs should not be assigned to followup question only. That would not meet understanding RFs. The applicant needs to Recommend not allowing CPDs to the CT methodology in ES 3.3 ES3.6, page 6 demonstrate the inability to take CTlevel 95 be assigned to understanding The following are examples of performance of 27 (line 17) safetysignificant actions to result in a CPD.

RFs. standards that cannot be measured objectively Should not be based on failing to provide a during a simulator scenario and, therefore, are correct answer to a follow up question.

not suitable for CT performance standards:

• understanding (such as an applicant must understand the significance of a certain plant response)

• observing that an expected response has occurred

• observing the performance of a system

The departure from nucleate boiling TS Consider using a TS example that is ES3.6, page 8 example provided is PWR specific. May be 96 applicable to all reactor Example was added for BWR plants in ES 3.6.

(line 26) beneficial to use an example that applies more technologies.

generically to other reactor types.

No change was made. The instructions state, "Every missed TS entry represents a PD, except for missed TS entries that fall under RF 6.a, which are limited to one PD per TS event." PDs in TS space can also be assigned to other Competency areas. TS events are treated like all With only three RF points to work with, the Consider simplifying and adjusting other events in that there may be more than ES3.6, page 8, allowance to assign multiple PDs for each TS in 97 grading criteria described in this one opportunity for a PDs in each event. If TS 9 a single event is not proportional. Each TS section. were graded with one PD per TS event, then event should be limited to one PD normally.

there would be a situation where some applicants may never get more than two opportunities to make a PD and therefore could not be graded lower than 2 on TS competency.

Another control in place is that no error is carried forward for individual TS events.

The assignments of CPDs (or even SPDs) in Communications seems excessive. All The change was made so all communication PDs ES3.6, page Consider simplifying 98 communications errors should be assessed as a will be graded using the Rev 11 criteria for 10 (line 35) communications competency RFs.

PD after the first one. This may make grading communication PDs.

simpler and more consistent.

The current exemptions for the AP1000 provides a condition when the alternative Added additional information about using compliance measures end when the site 10 ES3.7 Recommend adding both of these alternative JPMS. The allowance to use actual 99 CFR 52.103(g) finding is issued.

Page 1 of 1 items to this section. plant equipment already exists through the It also allows for using actual plant equipment "normal" JPM instructions in ES 3.2.

for inplant JPMs as it becomes available during the construction phase.

ES3.7, page 1 Step A.4 - a period is missing from the end of Change made in accordance with the 100 Add period.

(line 21) the sentence. recommendation.

Section 4, Written Examinations The following change was made to ES 2.1 to clarify that NRC chief examiner permission is still required to deviate from the approved outline: Any time it becomes necessary to deviate from the previously approved written In R11 ES401 D.2.a, deviations from the examination outline, contact the NRC chief previously approved examination outline examiner for concurrence on the need to required a discussion of the proposed Confirm that discussion with the deviate and request a replacement K/A. The ES4.2 B.3, deviation with the NRC chief examiner to NRC chief examiner is no longer NRC chief examiner will randomly select and 101 Page 3 of 29 obtain replacement K/As (if needed). R12 ES required to deviate from the provide a replacement K/A. The examination (line 1922) 4.2 B.3 provides examples of when deviations examination outline. author must document the eliminated K/A may be required and seems to require statements on Form 4.11, or equivalent, with documentation only. an explanation of why the K/A was rejected.

Also added to kickoff call list of topics (ES 2.1) for the NRC chief examiner and the licensee point of contact to discuss expectations for deviating from the approved written examination outline and selecting replace K/As.

R11 ES205 D clarifies that questions used on the GFE shall conform to the applicable construction and style guidelines in Appendix B. The examination shall include 40 questions taken directly from the NRCs GFE question Consider clarifying ES4.2 B.4 by The following change was made: Added bank for the applicable vendor type, 5 modifying the following statement, clarification for GFE exam bank in ES 4.2. This ES4.2 B.4, questions that are derived from existing bank This definition includes NRC definition includes NRC Generic Fundamentals 102 Page 3 of 29 questions by making one or more significant examination questions used at Examination (GFE) bank questions and NRC (line 2425) modifications, and 5 questions that are newly other facility licensee sites and the examination questions used at other facility developed. ES4.2 B.4 contains generic NRCs GFE question bank . licensee sites.

requirements for examination question selection from examination banks. It is not clear what rules will apply to Theory question development under R12.

ES4.3, Page 3 What is the purpose or value of developing of 6 (line 1),

and providing a seating chart. I have never page 5 of 6 Consider evaluating the need for No change was made. This change is outside the 103 had a need or reason to look at the seating (line 111), and providing a seating chart. scope of this revision.

chart following the exam nor is it required to ES4.4, page 3 be retained.

of 6 (line 22)

No change was made. For Rev 12, wording changed from "grading" to "review of grading" ES4.4, C, 3, g There appears to be a missing section header Recommend adding Grade the because usually the facility licensee grades the 104 Page 5 of 6, between item f. and item g. In revision 11 the Examination as item 4, and written examination in accordance with Line 31 header is Grade the Examinations. renumber subsequent steps. 55.40(b) Power reactor facility licensees may prepare, proctor, and grade the written examinations required by §§ 55.41 and 55.43.

Section 5, PostExamination Activities and Other Licensing Actions The proposed changes to ES3.6 now defines Recommend deleting a simulator ES5.1, Page 1 critical performance deficiencies and failure based on a single error with No change made. The grading criteria for CPDs 105 of 20, Line 16 significant performance deficiencies, so the serious safety consequences or was changed following public comment period.

and 17 previous examples of deviations from nominal from this parenthetical.

grading practices may no longer be accurate.

ES5.1, D, 3 Recommend adding closing Missing close of quotation at the end of 106 page 5 of 20, quotation mark following the word Change made in accordance with the comment.

Sample Pass Letter.

Line 2 Letter.

No change was made. The NRC has proposed to add a requirement for applicants to participate Recommend adding a discussion in a SATbased continuing training program in During the cold license phase the time about how a Pass Letter could be situations described by the commenter. The between receiving a Pass Letter and used during cold license phase. proposed changes are described in Regulatory ES5.1, D, 3 completing all the elements of 10 CFR 55.31 Additionally, add a requirement Basis for Public Comment (ML20149K680),

107 Page 5 of 20, was prolonged, so candidates were enrolled in for those candidates to enter a SAT Appendix E. Proposed changes to NUREG1021 Line 11 a SAT based continuing training program while based continuing training program that would be made if the rule is implemented, completing the items. if including changes to the pass letter template if that is appropriate. determined to be necessary, will be available to the public for comment when the proposed rule is issued for public comment

If cold license phase candidates with Pass Letters are enrolled in a SAT based continuing training program that is based on a typical No change was made. Making this license operator requalification training Recommend clearly stating that recommended change in Rev 12 may contradict ES5.3, A, 1, b program, the first class are likely to be in this the requirement to begin future rule language for this issue (See 108 Page 2 of 12, program for two years. Clearly state when the requalification exams begins from comment #107). The NRC staff is considering Line 14 requirement to begin requalification exams the issuance of the first licenses. updating guidance on this issue in NUREG1021 begins, from the entry of the first class into this as part of the Parts 50 and 52 rulemaking.

program or from the issuance of the first licenses.

Section 6, Requalification Examinations ES6.1, step 5, Change postexamination to Change made in accordance with the 109 page 8 of 33 postexamination should be hyphenated.

post examination recommendation.

(line 21)

Regarding first and second retakes, the document does not specify whether a second The following change was made: Added retake is required following passing the first Consider adding a statement that a clarification If the second (first retake)

ES6.1, H.2.c retake. second retake does not apply examination was failed, the NRC will normally 110 and d, Page 14 Note that this is not a change from ES605 following successful completion of administer a third (second retake) examination of 33 (page 13), and it implies a second retake isnt the first retake. approximately 6 months after issuing the necessary following passing of the first, but second failure.

may be added in the interest of clarity.

ES6.1, Page Change made in accordance with the 111 15 of 33 (line Form 6.11 Has no title description Include form title.

recommendation.

21)

ES6.1, Page Change made in accordance with the 112 15 of 33 (line Form 6.16 Has no title description Include form title.

recommendation.

46)

Change Preexamination and Form 6.13, ES Preexamination and Postexamination Postexamination to Pre Change made in accordance with the 113 6.1 Page 20 of should be hyphenated. examination and Post recommendation.

33 examination.

Under III. Quality, Exam Section goes from; A.

Move the guidance for written ES6.1, Page Sample Plan to C. Walkthrough 114 exam quality from revision 11 to Section B has been added from ES601.

23 & 24 of 33 It appears that there should be another section revision 12.

for; B. Written Exam ES6.1, Page The page number shows 2 of 33 versus 26 of Correct the page number to 26 of Change made in accordance with the 115 26 of 33 33. 33. recommendation.

This section is for NRC conducted The required minimum number of open requalification examinations only. This change reference written exam items is 700 questions, for new reactors is outside the scope of Rev 12 350 per section. During cold licensing and objectives and will be considered in future ES6.1, Form initial operating cycle for new construction Recommend adding an allowance revisions. As stated in ES 6.1, "The NRC will 116 ES 6.14, Page plants a 700 question exam bank requires a for new plants to submit the consider preferentially using the facility 21 of 33 significant amount of time and resources to available questions in their bank. licensees requalification examination structure accumulate. This is more difficult considering or methodology if it is different from that the higher likelihood of design changes during described herein provided that it complies with construction completion. 10 CFR 55.59, Requalification, and is free of significant flaws."

The required minimum number of job performance measures is 95, plus 10 per year following the initial requalification exam until the job task analysis is fully covered. The This section is for NRC conducted written exam bank and scenario exam bank requalification examinations only. This change have upper size limits. ES6.3, Recommend changing the required for new reactors is outside the scope of Rev 12 B.1.d (on page 3 of 7 of ES6.3) only requires a JPM bank size to match ES6.3, of objectives and will be considered in future representative sample of JPM, and states that ES6.1, Form approximately 125 to 150 JPMs. revisions. As stated in ES 6.1, "The NRC will the NRC anticipates that bank will be 117 ES 6.14, Page Also recommend adding an consider preferentially using the facility approximately 125 to 150 JPMs. During cold 21 of 33 allowance for new plants to submit licensees requalification examination structure licensing and initial operating cycle for new their available JPM bank at or methodology if it is different from that construction plants a 95 JPM bank requires a submittal time. described herein provided that it complies with significant amount of time and resources to 10 CFR 55.59, Requalification, and is free of accumulate. This is more difficult considering significant flaws."

the higher likelihood of design changes during construction completion, and further complicated by the use of alternative means described in Section 3.7 for cold license JPMs.

This section discusses that the Plant and Incorporate the clarification Control Systems section of the exam (section provided in the Operator Licensing A) is administered using a static simulator. Program Feedback into ES6.2 to There is an entry about this in the NRC improve the clarity and intent of No change made because this is outside the Operator Licensing Program Feedback page. It this section: As discussed in scope of this revision. Additionally, this acknowledges that this exam style was Section C of [ES6.1], if a facility statement exists in ES 6.1 and applies to ES 6.2 developed by a working group in 1987, but licensees requalification program as well: The NRC conducted requalification that some licensees stopped using that format uses an examination structure or examination normally consists of three parts, since the NRC shifted to an inspectionbased methodology different from that including a two section open reference written oversight program in 1994. The entry allows described in the [ES6] series and examination, a walkthrough evaluation, and a that the NRC will consider using the facility the NRC decides to conduct an dynamic simulator evaluation. ES6.2, ES 6.3, ES6.2, A, Page licensees requalification examination examination, the NRC will consider and ES 6.4 further describe the three 118 1 of 24 Lines structure or methodology if it is different, preferentially using the facility examination parts. The NRC will consider 13 thru 21 provided it complies with 10 CFR 55.59 and is licensees requalification preferentially using the facility licensees free of significant flaws. Section 6.1, B, 2nd examination structure of requalification examination structure or paragraph says that [T]the NRC will consider methodology if it is different from methodology if it differs from that described preferentially using the facility licensees that described in the ES, provided here, and if it complies with 10 CFR 55.59, requalification examination structure of it complies with 10 CFR 55.59 and Requalification, and is free of significant methodology if it differs from what is is free of significant flaws. State flaws. The regional office shall consult with the described here if it complies with 10 CFR 55.59 that the plant and control system NRR operator licensing program office to and is free of significant flaws, but does not section of the written exam is determine the appropriate examination specifically discuss the use of static exam allowed to be tested in the same procedure.

questions. It also requires the regional office to manner as the Administrative consult with NRR operator licensing program Controls/ Procedural Limits office prior to making this decision. without prior approval.

No change made because this is outside the scope of this revision. Additionally, this statement exists in ES 6.1 and applies to ES 6.2 as well: The NRC conducted requalification examination normally consists of three parts, including a two section open reference written There is no clear direction for how examination, a walkthrough evaluation, and a digital/embedded electronic formatted Recommend clearly stating that dynamic simulator evaluation. ES6.2, ES 6.3, ES6.2, D, 1, d. procedures are addressed. During normal the normal access to the and ES 6.4 further describe the three 119 Page 15 of 24 operation crew members routinely check digital/embedded electronic examination parts. The NRC will consider Line 39 thru 47 Alarm Response Procedures or Critical Safety format procedures should be preferentially using the facility licensees Function Status Tree indication, in addition to provided. requalification examination structure or the other reference material available. methodology if it differs from that described here, and if it complies with 10 CFR 55.59, Requalification, and is free of significant flaws. The regional office shall consult with the NRR operator licensing program office to determine the appropriate examination procedure.

No change made because this is outside the scope of this revision. Additionally, this statement exists in ES 6.1 and applies to ES 6.2 as well: The NRC conducted requalification examination normally consists of three parts, including a two section open reference written examination, a walkthrough evaluation, and a The list of plant procedures available as open dynamic simulator evaluation. ES6.2, ES 6.3, reference material includes emergency ES6.2, D, 1, e. and ES 6.4 further describe the three operating procedures, abnormal operating Recommend adding alarm 120 Page 16 of 24 examination parts. The NRC will consider procedures, and normal operating procedures, response procedures to this list.

Line 6 preferentially using the facility licensees but does not include alarm response requalification examination structure or procedures.

methodology if it differs from that described here, and if it complies with 10 CFR 55.59, Requalification, and is free of significant flaws. The regional office shall consult with the NRR operator licensing program office to determine the appropriate examination procedure.

The sentence systems that are the subject of ES6.3, page 2 NRC information notices is a separate thought Change made in accordance with the 121 Make sentence its own bullet.

of 7 (line 3) from the one above and should be a separate recommendation.

bullet.

Section 8, Glossary There is a definition for simulation facility, Recommend adding a separate ES8, Glossary which includes bullets for a plantreferenced definition for a commission Added a pointer to 10 CFR 55.4, "Definitions" Page 6 of 7 simulator and a commission approved 122 approved simulator, which for PRS and simulation facility so that there is Lines 20 thru simulator. There is also a separate definition includes how it is used in operator only one place for these definitions.

27 for a plantreferenced simulator, which licensing.

includes how it is used in operator licensing.

Appendix B, Examples of Written Examination Questions Add examples of plantspecific, operationally valid theory Tier 4 Theory is new to the written questions to Appendix B to aid the examination and previously used generic facility Examination Authors in No change was made. The NRC added examples of reactor and thermodynamic developing satisfactory some examples of Tier 4 questions to a 123 Appendix B theory questions may or may not be operationally valid theory acceptable on final licensing examinations. questions (refer to NEI letter on general Operator Licensing Program There would be a benefit to developing Generic Fundamentals Feedback item.

examples to include. Reintegration, Appendix 1, Recommendation 1, dated March 4, 2020 (ML20083F400).

Other / Miscellaneous General Recommend removing blank pages No change was made. Blank pages are required 124 Revision 12, Several blank pages Comment to reduce document size. by NRC NUREG publishing procedures.