ML23101A094

Operator Licensing Program Feedback

ML23101A094
Person / Time
Issue date:	03/21/2023
From:	Cushing S NRC/NRR/DRO/IOLB
To:
References
Download: ML23101A094 (122)

Text

OPERATOR LICENSING PROGRAM FEEDBACK Record of Changes Date Section(s) Affected Summary 03/21/2023 2.2, 3.4, 3.6, 4.1, Deleted 2.2.11 (202.19) due to no longer 4.2, 5.1, 6.1, 6.3, 8 necessary to clarify the NRC Form 398 instructions.

Question 3.4.4 added to address that an unused spare scenario may be used on a future examination.

Question 3.6.7 added to discuss the information that may be requested when evaluating to implement the allowance to fail an individual whose grade per ES-3.6 would indicate a pass.

Question 4.1.19 added to list generic KAs that may be prescreened from the Tier 3 portion of examination outlines.

Question 4.2.18 added to provide examples of Tier 3 examination questions for selected generic K/As.

Question 5.1.2 added to clarify that the spare scenario is used in the evaluation of submittal quality.

Question 8.1 added to clarify the definition of low-power for JPMs.

Revised Gen.55 question and response to be concise, generic, and improve clarity.

10/21/2022 3.3 Questions 3.3.1 through 3.3.5 were added to clarify the instructions in ES-3.3 regarding critical tasks. These are questions and answers generated during the first several months of implementing the revised critical task methodology in NUREG-1021 Revision 12.

06/02/2022 2.2, General Question 2.2.15 was added to clarify expectations for applicants time as extra person on shift.

Gen.45 and Gen.56 were updated to align with 1

Date Section(s) Affected Summary Paperwork Reduction Act requirements.

12/10/2021 All The OLPF was updated to be consistent with current practices and NUREG-1021, Revision 12, and questions were edited to improve clarity and to be inclusive. Questions were deleted or archived as appropriate. Please see the associated Change Catalog for these detailed changes, the historical (before July 2021) record of changes (ML22004A210), and the Archived OLPF questions (ML22004A210). The old question identifiers are in parenthesis next to the current identifier (e.g., 3.5.2 (302.2)) These updates were made to OLPF ML21167A354.

2

ES-1.3 Examination Security 1.3.1 (201.11)

Why does the NRC not have to sign a security agreement?

The primary purpose of the security agreement is to prevent inadvertent compromises by ensuring that the people having knowledge of the examination content are aware of their responsibilities. NRC examiners are aware of their responsibilities with regard to examination security and rarely find themselves in a position where they could inadvertently compromise the examination. They are only on-site to validate and administer the examinations and they do not routinely interact with the license applicants.

1.3.2 (201.12)

NUREG-1021, ES-1.3 Section C.3, prohibits someone on the exam security agreement from doing on-the-job training (OJT), practice, coaching, and signoffs. Does this prohibit an operator (on exam security) who is standing a regularly scheduled shift from signing off a trainee scheduled to stand that shift under instruction in the position? This is not referring to signing of individual OJT tasks, just the shift itself.

When the operator comes out to validate the written, can they have OJT contact with an applicant after the operator is on the security agreement?

Section C.3 of ES-1.3 prohibits all OJT activities. A license applicant should not be standing watches under instruction with, or receive OJT signoffs from, a licensed operator who has knowledge of the examination content.

1.3.3 (201.14)

Why do the standards not allow the utility to give the same JPMs and scenarios the following day if the applicants sign a confidentiality agreement? If an individual examinee is on security agreement, can you then reuse a JPM set?

No, the NRC takes examination security very seriously and prohibiting the reuse of test materials, including on subsequent days, is the most effective way to minimize the risk of compromising an examination.

1.3.4 (201.15)

The requirement preventing an instructor, who is on examination security and has knowledge of the examination, from direct training activities results in excessive staffing.

May we use the instructor in technical training and rely on the integrity from examination security or under the penalty of law, etc.?

3

What are any compensatory measures that we can take to use the instructor in both the examination development process and in candidate instruction/supervision?

While developing the current examination process, the NRC identified several vulnerabilities (including independence and public perception, examination security and integrity) associated with allowing facility licensees to prepare the initial licensing examinations, which had, theretofore, been prepared exclusively by NRC examiners or contractors. To the extent possible, the NRC established guidelines and criteria in ES-1.3 of NUREG-1021, including the personnel and security restrictions, to mitigate the vulnerabilities. Please refer to SECY-96-206 (the rulemaking plan) and SECY-98-266 (the final rule) for a discussion of the NRC's rationale.

It should be noted that the current restrictions are consistent with the change recommended by the Nuclear Energy Institute (NEI) during the rulemaking process.

Although ES-1.3 clarifies that supervisors can counsel applicants regarding non-technical issues, direct training activities are still prohibited. There is some flexibility to address unique situations on a case-by-case basis; however, a generic change in policy is unlikely unless the industry can adequately address the NRC's concerns regarding public perception and confidence.

1.3.5 (201.16)

Providing individual applicant feedback is a prohibited activity for individuals on the security agreement. How does this apply to Manager/Supervisor situations such as sitting on a performance review committee or coaching/counseling associated with a non-technical situation (e.g., classroom behavior)?

Managers/supervisors on the security agreement may continue to counsel the applicants concerning non-technical issues. However, as stated in Section C.3 of ES-1.3 of NUREG-1021, they are not allowed to provide any technical guidance, training, or any other direct feedback that may compromise examination integrity as defined in 10 CFR 55.49.

1.3.6 (201.17)

Is a facility required to check with a contractor to determine if they are concurrently developing a similar exam for another utility? If so, do these exams need to be given on the same day? Also, what other security requirements need to be met?

If you have a common group develop examinations for two different plants, do you have to worry about overlap between these exams? What are the criteria?

Pursuant to 10 CFR 55.40(b)(2), facility licensees that prepare their own examinations are expected to take reasonable measures to control examination security and integrity. As noted in Section C.1 of ES-1.3, facility licensees may use contractors or other outside assistance to develop the examinations, but the licensees bear full responsibility for the product, including conformance with the examination criteria and maintenance of examination security and integrity. Additionally, Section B.6 of ES-4.2 (in NUREG-1021) discusses the requirements for 4

controlling and documenting the source of test items and the predictability of the examination content. Licensees should obtain this information from their examination contractor if one is used. If there is a basis for the applicants to predict the content of the examination and the overlap with the other utility's examination is significant, then the utility must evaluate the issue, determine if compensatory measures are appropriate, and discuss the issue with the NRC as early as possible. Factors to consider would include the timing between the exams and the physical and corporate distance between the facilities. For example, this evaluation could reasonably differ if, in one case, the sites are owned by the same utility, located 20 miles apart, and the exams are separated by a month, versus another case in which the exams are 8 months and 2000 miles apart.

1.3.7 (201.18)

As part of normal instructor duty, 10 questions were submitted to an examination team.

Does the instructor have any examination information?

If the instructor is not aware whether any of the questions meet the sample plan and the questions are placed in the exam bank, then the instructor would not be considered to have exam information. However, if the questions are given to the examination team with the expectation that they will be used as new questions, then the instructor should be on the security agreement. Specific questions regarding this issue should be discussed with the NRC.

1.3.8 (201.19)

If involved in an initial examination, is there a restriction from teaching requalification?

An initial licensed operator upgrade candidate attends licensed operator requalification training with their crew. The instructor is on the initial NRC exam team and has signed the exam security documents. Is the initial NRC exam candidate allowed to remain in the class/simulator or must they leave?

Use of instructors is still an issue. The use of an instructor, who is on the exam security agreement, can't teach candidates attending the requalification program. This is an unnecessary burden on resource restrictions.

SRO upgrade applicants who are removed from the watch rotation do not have to attend RO requalification training while they are training for the SRO license. If there are no upgrade applicants in the requalification class, there would be no restriction on the instructors. However, as stated in Section C.3 of ES-1.3 (NUREG-1021), if SRO upgrade applicants are present in the class, instructors would not be permitted to teach in areas in which they have examination knowledge, and their activities would have to be documented on Form 1.3-1. They can teach subjects about which they have no examination knowledge, which is a good reason to limit everyone's access to only those portions of the exam for which they have responsibility.

Instructors with examination knowledge should not be used in training environments that require one-on-one contact with trainees. There is no problem with them teaching a requalification lecture or simulator session, but the trainer with examination knowledge must avoid direct individual interaction with the applicants.

5

1.3.9 (201.20)

Is it acceptable to password protect exam files and leave them on a local area network (LAN) or password protect them on a hard drive?

Yes. As stated in section D.3 of ES-1.3 (in NUREG-1021), the use of passwords should provide adequate security if normal computer security practices (e.g., selecting and changing passwords) are observed. Special cases may need additional consideration. For example, if a trainee has extended access to the LAN in their normal position, additional security measures might be appropriate.

1.3.10 (201.21)

Will you allow transfer of electronic files of exam materials over the Internet via e-mail if the file is "password protected?"

The NRC has a checklist that NRC staff and facility licensees can use to assist in secure transmittals, Checklist for Transmitting and Receiving NRC Exam Material over the Internet. As stated in Section D of ES-1.3 (in NUREG-1021), examinations shall not be transmitted via non-secure electronic means. Licensees may make arrangements with the NRC resident inspector and transmit the exams via the NRC's internal LAN. Licensees may also transmit password-protected electronic files over the Internet if the licensee's word processing software provides adequate security and is compatible with the NRC's and the password is separately provided to the NRC chief examiner by mail or phone. The files do not need to be encrypted. Additionally, the NRC has authorized the use of Box file sharing service to transmit exam materials between the licensee and the NRC. Box temporarily saves large documents for 90 days. The same password protection requirements apply.

1.3.11 (201.22)

If the examination is password protected, how much hacking do we have to protect against?

Pursuant to 10 CFR 55.49, the NRC expects facility licensees to take reasonable measures to prevent inadvertent examination compromises. Section D of ES-1.3 describes several examination security guidelines that facility licensees may consider. The NRC does expect reasonable computer security measures to be in place, but it does not require facility licensees to defend their examinations against willful acts, such as computer hacking.

1.3.12 (201.23)

The person who issues the password and knows what it is for a computer system - are they in possession of examination material?

The facility licensee needs to evaluate whether their circumstances (procedures, computer security, etc.) require them to take action for exam security in accordance with 10 CFR 55.40(b)(2).

6

1.3.13 (201.25)

When does someone have to go on examination security?

Per Section C.2 of ES-1.3 (in NUREG-1021), they must acknowledge their security responsibilities by reading and signing the security agreement (Form ES-1.3-1) before they obtain detailed knowledge of any part of the examination.

1.3.14 (201.27)

Is there a "hard-limit" to the number of people that can sign in on a security agreement?

No. Section C.1 of ES-1.3 of NUREG-1021 outlines the expectations in this regard.

1.3.15 (201.33)

Can the initial license exam author or an exam team member provide difficulty ratings for weekly written quizzes given to an initial license class? There is no contact with the class and no direct feedback. Operations and Training Management use the difficulty ratings to gauge student progress.

The NRC takes examination security and integrity very seriously. However, based on your assertion that the raters would have no contact with the class and no direct feedback and that the difficulty ratings would only be used to gauge student progress, there should be no problem with your proposal.

1.3.16 (201.34)

Is it acceptable for a dedicated, locked examination security room to have a ceiling with removable tiles or is a hardened ceiling required for exam security purposes?

The NRC expects facility licensees to take reasonable measures to prevent inadvertent examination compromises. Section D of ES-1.3 of NUREG-1021 describes several examination security guidelines that facility licensees may consider, but it does not address the need for hardened examination development facilities. If the examinations are prepared in a hardened room with no drop ceiling and a decent lock on the door, then the authors could probably leave the exams lying about the room without much worry. However, if the exam room has a drop ceiling that someone could easily crawl over, then the authors should probably consider locking the exam materials in a file cabinet when the room is going to be unoccupied for a considerable period of time (e.g., nights and weekends) and there is a possibility that someone could crawl over the wall undetected (e.g., the exam room is in an isolated part of the building). A room with a locked door would likely provide sufficient protection for an exam left on the desk while the author goes to the rest room, even if the ceiling contains removable tiles. Licensees need to exercise common sense and decide for themselves how much they want to spend to maintain examination security and how much risk and expense they can tolerate if an exam is compromised.

7

1.3.17 (401.33)

Does the licensee need to supply names, positions, etc. of validation team prior to using them to review the exam? From ES-1.3, Section C.5, regarding certain individuals for exam validation: What is a "supervisor or coworker?" This could be any licensed operator.

Section C.5 of ES-1.3 discourages facility licensees from using certain individuals to validate the written examination. The applicants' supervisors and coworkers may not be the most appropriate to use for exam validation because it would raise concerns regarding the potential for examination compromise. Moreover, SectionC.3 states that individuals having knowledge of the examination contents are prohibited from performing several activities, including all on-the-job training, practice, coaching, and signoffs. Although licensees are not required to obtain NRC concurrence before placing personnel on the security agreement, it would be prudent to assess the security risk and discuss any questions with the NRC chief examiner. The supervisor/coworker connection would be of most concern for ROs seeking to upgrade their licenses.

1.3.18 (401.41)

Do practice exams late in the program have to be accounted for in the exam overlap restrictions?

That depends on whether they are developed before or after the facility licensee begins working on the licensing examination. Although NUREG-1021 has eliminated the restrictions on repeating questions from training quizzes and the past two licensing examinations, the facility licensee must still take measures to ensure that the audit exam and any other quizzes developed after starting work on the licensing exam do not compromise the integrity of the exam. Section D of ES-1.3 provides examples of acceptable control measures.

8

ES-2.1 Preparing for Operator Licensing Initial Examinations 2.1.1 (201.1)

What is the time expectation for turnaround of an examination submitted for review?

Per Section C.4.h and C.4.j of ES-2.1 (NUREG-1021), chief examiners and the facility are to discuss the timeline for the NRC to provide the written examination outline as early in the process as possible. Per Form 2.1-1, Examination Preparation Checklist target due dates are provided and can be adjusted as necessary to accommodate a given situation. After complete submission of draft examination materials, the review should be completed within 25 days.

Facility licensees are encouraged to discuss their specific schedule requirements and expectations with their chief examiner.

2.1.2 (201.2)

Is the request for NRC to write the examination required in writing?

Yes. Section 10 CFR 55.40(c) of the amended rule states that the Commission shall prepare the examination upon written request from the power reactor facility licensee pursuant to

§55.31(a)(3). It must be a corporate decision with a formal request in writing signed by an authorized facility representative.

2.1.3 (201.3)

Can the utility write part of the examination and the NRC write the other part of the examination? How do you work the "split exam" concept? How can you maintain NRC examiner proficiency if developing "split exams?"

Yes. Allowing the facility licensee and its NRC Regional Office to split responsibility for exam development provides both parties with greater flexibility in scheduling their resources. For example, the Regional Office might be able to support an examination on a specific date if it only has to prepare the written exam or the operating test, but not both.

The desire to split an exam should be reflected in the facility licensee's response to the NRC's annual letter soliciting examination schedule information (e.g., RIS 2003-14) and coordinated with the appropriate NRC Regional Office.

Keep in mind that each Regional Office is still required to prepare one complete examination per year to maintain examiner proficiency, but it can do the written portion of one examination and the operating test on another.

9

2.1.4 (201.5)

Can we have an exam development team from the utility come to the region and work directly with the chief examiner to develop the written exam?

The NRC currently does not believe that this is a viable option because it raises concerns regarding independence, accountability for the quality of the final product, and possible adverse public perception.

2.1.5 (201.6)

What are the requirements regarding examination security and examination overlap when the same utility examiners write the initial licensing exam and the audit exam? If you use independent groups to develop an audit examination and an NRC examination, do you have to worry about overlap?

As stated in Section C of ES-1.3 (NUREG-1021) provides examples of acceptable control measures to develop a comprehensive audit or screening examination that does not compromise the integrity of the operator licensing initial licensing examination. Individuals who are on the security agreement may prepare the audit examination, but the examination would be subject to review by the NRC for test item duplication (none is allowed unless the examinations are independently developed).

2.1.6 (201.7)

Should the utility NRC exam writer be "certified" by the NRC?

No. Although the NRC has considered that and other ways to improve the training and qualifications of utility examination authors, there are no current plans to implement such a program.

2.1.7 (201.9)

Does "independent review" by a supervisor include question-by-question approval/comment?

Yes. The independent managerial or supervisory reviewer is confirming and signing that the written examinations and operating tests meet the requirements of NUREG-1021. The extent of the review will typically be a function of the experience of the examination author and the quality of facility's examination bank.

2.1.8 (201.28)

If an exam compromise is suspected, are the examiners expected to leave the site?

No. In accordance with Section D.3 of ES-2.1 (in NUREG-1021), examiners must immediately report any perceived compromise to the responsible regional supervisor so that the necessary actions can be taken to restore the integrity of the examination. Per section B of ES-1.3, those actions might include not giving the exam, making additional changes to the exam, voiding the results if the exam has already been given, reevaluating the licensing decisions pursuant to 10 10

CFR 55.61(b), and possibly the imposition of enforcement action. It is much easier to determine the most appropriate action if the examiners remain on-site to assess the situation. The final course of action would be determined in collaboration with regional management and the NRR operator licensing program office.

2.1.9 (201.29)

Why doesn't the NRC have additional staff to support emergent utility exam needs?

Writing of exams is not voluntary because of resource constraints. What is the NRC doing about it?

The NRC staff does budget some additional resources for retake examinations, but the NRC's Congressional budget allocation does not permit us to maintain a dedicated corps of examiners capable of handling every conceivable peak workload. That is why it may be necessary for licensees to shift their examinations (usually no more than a few weeks) to a time when NRC resources are available. The NRC does have a few staff who are trained to administer the operating test; however, those individuals' primary responsibilities in their current positions generally have priority, so they are not always available on short notice. The operator licensing program office has established a national exam schedule framework to ensure collaboration between NRC regional offices so that resources are efficiently used.

2.1.10 (201.30)

If a utility does not have enough staff to write an initial licensed operator exam, is it better to have a vendor or the NRC write the exam?

This is a decision that facility management will have to make based upon cost, resource availability, scheduling flexibility, and other factors. The chances of getting an exam at a specific time are best if the licensee (or its vendor) prepares it.

2.1.11 (401.5)

If the utility is producing the written exam, when (how many days/weeks) is your expectation for the chief examiner to get the sample plan to the utility? The point is -

getting the sample plan in accordance with NUREG-1021 will not work.

As stated on Form ES-2.1-1, the examination outline should normally be completed at least 195 days before the scheduled examination date. These are target dates and are based on licensee prepared examinations and the examination date identified in the corporate notification letter. These dates are for planning purposes and may be adjusted in coordination with the facility licensee (refer to Section B.2 of ES-2.1). If the facility licensee needs more than 195 days to prepare an examination based on an NRC-developed outline, it needs to work out the schedule with the Regional Office.

11

2.1.12 (201.35)

Are there any requirements set by the NRC as to when Operation's training freezes procedure changes prior to an NRC initial licensing examination? Can a plant freeze multiple procedures at different times based on the scope of the procedure change and how its implementation date affects examination development and administration?

Guidelines regarding the freezing of plant procedures in advance of an initial operator licensing examination were added to NUREG-1021 as part of Supplement 1 to Revision 9, which was published in October 2007. Section C.4.e of ES-2.1 now specifically requires the topic to be discussed when confirming the examination arrangements, and section G of the same ES provides some general guidance and cautions.

12

ES-2.2 Applications, Medical Requirements, and Waiver and Excusal of Examination and Test Requirements 2.2.1 (201.10)

If a reactor operator is testing for an upgrade and their physical is current, do they have to have another physical?

No. In accordance with Section E of ES-2.2, the medical examination documented on NRC Form 396 is good for two years from the date of the medical examination. Per 10 CFR 55.25, facility licensees are required to notify the NRC within 30 days of learning that a licensed operator has developed a permanent physical or mental condition that causes the operator to fail to meet the eligibility requirements.

2.2.2 (201.26)

If an applicant fails a section of a licensing examination that was developed using one revision of NUREG-1021 and applies for a partial retake examination after the next revision of the NUREG has been issued, what version of the NUREG will be used to prepare the retake examination?

The decision would be based on maintaining continuity in examination content and format. If there is essentially no change in the content and format of the exam between the two revisions of NUREG-1021, it makes no difference which version is used, and it generally makes more sense to use the current version, especially if other applicants will be taking the entire examination. However, if the format or content of the exam has changed substantially (as it did when the pre-scripted JPM questions were deleted in Revision 8) it might make sense to administer the exam using the older format (e.g., if missed pre-scripted questions contributed to the failure). In summary, the NRC would default to the new standard, unless there is a logical basis to stick with the previous version and an exemption is submitted and granted.

2.2.3 (202.1)

Significant reactivity manipulations were defined in the Q&A portion of NUREG-1262.

Information Notice 97-67, Failure to Satisfy Requirements for Significant Manipulations of the Controls for Power Reactor Operator Licensing seems to conflict with NUREG-1262. An answer to what is a significant manipulation should support NUREG-1262.

Reactivity manipulations for initial licensed operator training: What is the status of allowing simulator manipulations (when unable to perform in-plant)? Also, define what constitutes a control manipulation. Why is a rod operability surveillance acceptable at one plant but not another? What constitutes a large change?

What is acceptable for reactivity manipulations? (any real-life examples of problems or rejected applications)

What are the criteria for doing reactivity manipulations on the simulator?

13

Information Notice 97-67restated and clarified the NRC's position on this issue. The staff does not believe that the IN contradicts the guidance in NUREG-1262.

Effective on November 16, 2001, 10 CFR 55.31(a)(5) was revised to allow the use of plant-referenced simulators to conduct the required control manipulations. Facility licensees that propose to use a plant-referenced simulator to perform the control manipulations must ensure that simulator fidelity has been demonstrated pursuant to 10 CFR 55.46(c).

The same test (e.g., started at a comparable power level, including a comparable number of rods, and a comparable reactivity change) should be acceptable on either plant. Without specifics, it is not possible to speculate why one was acceptable and the other was not.

10 CFR 55.31(a)(5) requires five "significant" control manipulations, and 10 CFR 55.59(c)(3)(i) provides a number of examples (which are not requirements). Per Example F, and as noted in IN 97-67, a power change of at least 10% is an example of a significant (or large) control manipulation. It would also be acceptable, when defining allowed reactivity manipulations, to evaluate the knowledge and abilities exercised in a controlled large evolution and then accept all smaller tasks that comparably exercise the same knowledge and abilities. The NRC expects such evaluations to be formally documented as part of the licensee's SAT-based (systematic approach to training) program.

The criteria for doing the 10 CFR 55.31(a)(5) reactivity manipulations on the simulator are discussed in SECY-99-225, the staff paper that forwarded the associated rulemaking plan to the Commission for approval and SECY-00-0083, the proposed rulemaking paper, which was issued on April 12, 2000. Facility licensees that propose to use a plant-referenced simulator to perform the control manipulations required by 10 CFR 55.31(a)(5) must ensure that simulator fidelity has been demonstrated pursuant to 10 CFR 55.46(c). Control manipulations performed on the plant-referenced simulator may be chosen from a representative sampling of the control manipulations and plant evolutions described in 10 CFR 55.59(c)(3)(i)(A-F), (R), (T), (W), and (X), as applicable to the design of the plant for which the license application is submitted.

As discussed in Section C.3 of ES-2.2 (in NUREG-1021) power changes (10 CFR 55.59(c)(3)(i)(E) and (F) only) that are performed on the simulator must be 10% or greater in magnitude, while those on the plant may be smaller (to limit unnecessary transients on the facility) but of sufficient magnitude for the operator to experience appropriate feedback (i.e.,

clearly observable effects on the plant) as a result of the control manipulation.

2.2.4 (202.2)

Can a candidate enrolled in a reactor operator initial license training program receive credit for significant control (reactivity) manipulations performed in the control room as the Balance of Plant (BOP) operator? For example, can the following manipulation, 10 CFR 55.59(c)(i)(C), be performed as BOP? Manual control of steam generators or feedwater or both during startup and shutdown.

A related question is: Do direct SRO candidates (i.e., instant SROs) have to perform the manipulations as ROs to get credit, or can they supervise them as SROs (i.e., procedure readers) to get credit?

14

10 CFR 55.31(a)(5) requires that an applicant provide evidence that the applicant, as a trainee, has successfully manipulated the controls of either the facility for which a license is sought or a plant-referenced simulator that meets the requirements of 10 CFR 55.46(c). At a minimum, five significant control manipulations must be performed that affect reactivity or power level. Control manipulations performed on the plant-referenced simulator may be chosen from a representative sampling of the control manipulations and plant evolutions described in 10 CFR 55.59(c)(3)(i)(A-F),(R),(T),(W), and (X) of this part, as applicable to the design of the plant for which the license application is submitted.

Therefore, two criteria drive the requirements for the five control manipulations, they must be significant and must affect reactivity or power level. "Manual control of steam generators or feedwater or both during startup and shutdown" is only sufficient to meet those two criteria if the licensee can clearly show that the manual control was significant and noticeably affected reactivity or power level. There is no requirement for the control manipulations to be completed in the RO watch position, so any manipulation done in the BOP watch station would qualify if it meets the requirements discussed above.

With regard to direct, or instant, SRO applicants, the control manipulations must be done in either the RO or BOP positions (i.e., hands-on); supervising another operator performing the manipulations would not be acceptable.

Keep in mind, as noted in Revision 2 of Regulatory Guide 1.8, "Qualification and Training of Personnel for Nuclear Power Plants," that every effort should be made to have a diversity of reactivity changes for each applicant. Moreover, in keeping with the definition of "Controls" in 10 CFR 55.4, it is preferable that the required manipulations focus on those apparatus and mechanisms that directly affect the reactivity or power level of the reactor (e.g., control rods, boration/dilution, and turbine load for a PWR; control rods and recirculation flow for a BWR).

After all, in accordance with 10 CFR 50.54(i), those are the only apparatus and mechanisms (i.e., controls) that can be manipulated exclusively by operators and senior operators licensed (or in training for a license) pursuant to 10 CFR 55.

2.2.5 (202.4)

Can a reactor startup below the point of adding heat constitute a manipulation?

What constitutes "significant?"

What is the current position on diversity, e.g., can 5 power changes using boration be used?

Yes.

As indicated in Information Notice 97-67, "Failure to Satisfy Requirements for Significant Manipulations of the Controls for Power Reactor Operator Licensing," and defined in 10 CFR 55.59(c)(3)(i)(E), a 10 percent or greater power change is an example of a significant control manipulation.

As stated in the IN and Regulatory Guide 1.8, Revision 4, diversity of control manipulations is expected but not required. Similarly, if the training program is developed using a systematic approach, it would seem inappropriate to conduct the same control manipulation five times.

15

Some diversity is better than none, i.e., the 5 boration power changes should be as diverse as possible. See Questions 2.2.4 (202.2 and Error! Reference source not found.) for more information.

2.2.6 (202.12)

Can a 1-hour reactivity change be counted towards the needed on-shift time? Can a 4-hour evolution be counted if the applicant attends all prerequisites and post-activities?

Per 10 CFR 55.31(a)(4), license applicants must provide evidence that they have successfully completed the facility licensee's requirements to be licensed as an operator or senior operator.

The NRC's regulations and guidance documents do not specify how to count the 3 months of on-shift time. However, if the facility licensee's accredited training program or other commitments (e.g., its final safety analysis report or technical specifications) provide such guidance, then the NRC would expect the facility and applicant to comply. Since the intent of this training is for the applicant to experience the full range of routine, day-to-day shift activities, the NRC would expect, in the absence of a contradictory facility requirement, that the training would be accomplished in full-shift increments.

2.2.7 (202.13)

Can the 6 months on-site power plant experience occur prior to a break in service (e.g.,

the individual works on-site for over 6 months in a responsible position, then they leave the site and return sometime later)? Is the 6 months satisfied already?

Per 10 CFR 55.31(a)(4), license applicants must provide evidence that they have successfully completed the facility licensee's requirements to be licensed as an operator or senior operator.

The NRC's regulations and guidance documents do not specify when the 6 months of on-site experience needs to take place. However, if the facility licensee's accredited training program or other commitments (e.g., its final safety analysis report or technical specifications) prohibit a break in service, then the NRC would expect the facility and applicant to comply.

2.2.8 (202.14)

Can a facility be committed to ANSI N18.1-1971 for candidate eligibility, yet incorporate guidance of ES-2.2/RG-1.8 or other document(s) without changing the committed document?

In 1987, Generic Letter 87-07 (which was issued in connection with a revision to 10 CFR 55) gave facility licensees the option of substituting an accredited training program for their initial and requalification training programs previously approved by the NRC. Most facility licensees elected this option in writing, but some of them neglected to revise the training program descriptions in their technical specifications, final safety analysis reports, and other documents.

Facility licensees need to resolve conflicting and contradictory training program commitments and requirements.

16

Refer to NUREG-1021 for a detailed discussion of the current guidelines for the qualification and training of licensed operators.

2.2.9 (202.16)

Can we eliminate the hours of operation on NRC Form 398 for license renewal applications?

The requirement to supply that information is contained in 10 CFR 55.57(a)(3). The only way it could be eliminated from the form is by amending the regulation or requesting an exemption.

This issue was also raised in connection with a recent extension request for the Office of Management and Budget (OMB) Clearance covering 10 CFR Part 55. The NRC staff reassessed the need to collect this information and decided that a revision NRC Form 398 sufficiently minimized the record-keeping burden by establishing three broad ranges (i.e., less than 100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br />, between 100 and 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br />, and more than 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br />) from which renewal applicants can select.

2.2.10 (202.17)

In NUREG-1021, expand the detail requirements for people who had a license at the unit and dropped it longer than 2 years ago. NUREG-1021 covers initial, upgrade and less than 2 years, but not in between.

The regulations (specifically 10 CFR 55.47) allow a waiver of the operating and written test if the applicant had extensive actual operating experience at the facility or a comparable facility within the last two years. After two years the applicant must take the full license examination or request and justify an exemption. The NRC currently has no plans to change this aspect of the regulation.

2.2.11 (202.19)

Deleted.

17

2.2.12 (202.20)

The National Academy for Nuclear Trainings (NANT) Guidelines for Initial Training and Qualification of Licensed Operators -- ACAD 10-001 -- were revised on May 20, 2021, i.e., NANT, ACAD 10-001, Revision 2. The revision updated and clarified the experience and education eligibility guidance for the selection of reactor operators (ROs) and senior reactor operators (SROs) at existing nuclear power plants.

However, Revision 11 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, in ES-202 references the NANT guidelines issued in February 2010 (NANT 2010) and states: Unless otherwise informed by a facility licensee, the NRC believes that the education and experience guidelines described in NANT 2010 constitute the facility licensees education and experience requirements to be licensed as an RO or SRO. Given this NUREG-1021 description, does the NRC also consider the ACAD 10-001, Revision 2, guidelines an acceptable methodology for eligibility determinations at existing nuclear power plants?

NUREG-1021, Revision 11, states that the guidelines for education and experience issued by the National Academy for Nuclear Training (NANT) outline acceptable methods for implementing the Commissions regulations in this area. NRC Form 398, Personal Qualifications StatementLicensee, revised October, 2019, states that certifying completion of the Operator Training Program accredited by NANT indicates that you have completed a SAT based training program that is accredited by the National Nuclear Accrediting Board and meets the education and experience requirements outlined in the National Academy for Nuclear Training in its current [emphasis added] guidelines for initial training and qualification of licensed operators. Accredited facility licensees have been using the ACAD 10-001, Revision 1 guidelines since 2018. On May 20, 2021, NANT published ACAD 10-001, Revision 2.

Revision 1 of the guidelines will be retired on May 31, 2022. After publication of Revision 2, and before retirement of Revision 1, the NRC will consider either revision to be current guidelines in order to allow a transition period. After the retirement date of Revision 1 on May 31, 2022, Revision 2 will be the current guidelines for initial training and qualification of licensed operators.

18

2.2.13 (202.21) 10 CFR 55.31, How to apply, states that the applicant shall complete NRC Form 398, Personal Qualification Statement - Licensee, and provide certification by the facility licensee of the applicants medical condition on NRC Form 396, Certification of Medical Examination by Facility Licensee. 10 CFR 55.31 and §55.23 also state that these forms can be obtained by writing the NRC or by accessing the NRCs web site.

Given that the forms are only available by writing the NRC or accessing the NRCs website, how long do licensees have to use the most recent version of NRC Form 396 and 398 from the revision date?

There are no specified implementation dates for using the newly revised NRC Forms 398 and 396. However, following the direction outlined by 10 CFR 55.23 and 10 CFR 55.31 an applicant is expected to complete the forms that can be obtained by contacting the NRC via phone, mail or use of the NRC public website. Therefore, assuming that the NRC would provide the most recent revision of the forms, applicants should use the most recent revision of the form that is available at the time they initiate the application process (start to complete the forms). For license renewals, 10 CFR 55.57 provides no additional clarification as to which version of the forms should be used, therefore the same expectation applies regarding the use of the forms which can be obtained by contacting the NRC at the time the renewal process is initiated.

Ultimately, the program office can foresee no situation for which an applicant or licensed operator would initiate filling out a Form 396 or 398 greater than three months prior to submitting them to the NRC in support of an initial license application, medical update, or license renewal.

This general assumption may be adjusted on a case-by-case basis through consultation with the appropriate NRC regional office. The regional licensing officials are ultimately responsible for ensuring that the forms submitted provide the necessary information to support accurate and timely licensing decisions.

2.2.14 (202.23)

Can facility licensees use contractors as licensed reactor operators and senior reactor operators?

There is no NRC regulatory prohibition per se regarding the use of contractors as licensed operators. Instead, the regulations in 10 CFR Part 55 apply to any individual who manipulates or directs the manipulation of the controls of any utilization facility. Additionally, although the 10 CFR Part 55 regulations regarding applications and re-applications for, and the expiration and renewal of, operator licenses refer to the operator applicant/licensee being employed by the facility licensee, these regulations do not define what being employed means. Therefore, licensed operators may be considered employed by the facility for which they are licensed for purposes of NRC licensing and regulation regardless of whether they are contractors or the facility licensees employees. Accordingly, facility licensees can use contractors as licensed reactor operators. Facility licensees may also use contractors as licensed senior reactor operators provided that under other applicable laws, such an employer-contractor relationship does not preclude the licensed senior reactor operator from directing the licensed activities of the facility licensees licensed reactor operators.

19

All contractor operator applicants/licensees must meet the same licensing requirements in 10 CFR Part 55 and be subject to the same penalties as employee operator applicants/licensees.

They must also meet the 10 CFR Part 26 Fitness for Duty Program requirements and be liable for deliberate misconduct under 10 CFR 50.5.

2.2.15 NUREG-1021 Rev. 11 contained guidance that applicants should spend 3 months as an extra person on shift in training to apply for an RO or SRO license. However, NUREG-1021, Rev. 12 does not contain this guidance. How much time as an extra person on shift is required?

NUREG-1021, Rev. 11, ES-202, Section D, NRC License Eligibility Guidelines, summarized the guidelines in Regulatory Guide 1.8, Qualification and Training of Personnel For Nuclear Power Plants, Revision 3 and American National Standards Institute/American Nuclear Society (ANSI/ANS)-3.1-1993, Selection, Qualification, and Training of Personnel for Nuclear Power Plants, including the 3 months as an extra person on shift. While NUREG-1021, Rev. 12 does not summarize the guidelines, if a facility licensee has committed to ANSI/ANS 3.1-1993 or RG 1.8, Rev. 3, then the facility needs to meet the commitment. If a facility has some other time requirement as an extra person on shift, then the facility should either follow or modify the commitment. See 2.2.8 (202.14) regarding conflicting and contradictory training program commitments and requirements, if applicable.

For NRC operator licensing, 10 CFR 55.31(a)(4) states, in part, that the Commission may accept certification that the applicant has successfully completed a Commission-approved training program that is based on a systems approach to training. NUREG-1021, Rev. 12 says that NNAB-accredited training programs are considered Commission-approved training programs based on a systems approach to training. Therefore, absent a commitment to ensure applicants spend at least 3 months as an extra person on shift, stations with an accredited training program would need to determine the amount of time to schedule applicants as the extra person on shift using the systems approach to training process (e.g., by considering the time needed for applicants to successfully complete on-the-job training tasks and under instruction watches).

20

ES-3.1 Overview of the Operating Test for Operating Licensing Initial Examinations 3.1.1 (301.2)

Our experience has been that we are told ALL items of 10 CFR 55.45 and §55.43(b) must be sampled.

If 100% of sampling for topics in §55.45(a) is not required, is there a definition of representative sample?

What is meant by a "representative sample" of the 13 items identified in 10 CFR 55.45(a)?

ES-3.1 section A states that all 13 items in 10 CFR 55.45 do not need to be sampled on every operating test. Although NUREG-1021 does not include a similar statement with regard to the written examination, the same policy still applies. In accordance with ES-4.1 section B, the topics for the written examination are to be systematically selected from the appropriate Knowledge and Abilities Catalog (NUREG-1122 or 1123). Although the NRC has not developed a definition of a "representative sample," logic dictates that it should include a reasonably complete, thorough, balanced, and varied cross-section of the items in the population to be sampled. All of the items should be sampled from time to time, and, absent a basis for emphasizing certain items, it is expected that every item would be sampled at about the same frequency. An examination constructed in accordance with NUREG-1021 will normally contain a "representative sample" of the required items.

3.1.2 (301.3)

Do the audit exam and the NRC exam have to be 100% different?

ES-3.1, B.4 - No reuse of audit material for subsequent exams?

To what extent do "similar events" between the audit and NRC exam need to be identified? For example, if the audit examination contained a faulted SG [steam generator] in one scenario (safety valve stuck open) and the NRC examination contained a faulted SG (pipe rupture in containment), would these situations be considered "similar?"

No. As noted in ES-3.1 Section B.4 (NUREG-1021), simulator events and JPMs that are similar to those that were used on the audit test (or audit tests in the case of retake applicants) are permitted provided the actions required to mitigate the transient or complete the task (e.g., using an alternate path as discussed in Section E of ES-3.2) are significantly different from those required during the audit examination. The facility licensee shall identify for the NRC chief examiner those simulator events and JPMs that are similar to those that were tested on the audit examination.

21

The two events cited in the example are "similar" (in that they both involve a faulted SG) and should be discussed with the NRC chief examiner. In this case, the mitigation strategy for the two events - one being inside and the other outside containment - are sufficiently different that their use would probably be acceptable (unless there were other predictable patterns between the two scenarios).

3.1.3 (301.5)

How is the JPM system selection supposed to occur? Shouldn't there be a systematic (e.g., random) selection of systems within each of the safety functions? Otherwise, won't the operating exam be somewhat predictable? Same concern with event selection for simulator exams (scenarios).

ES-3.1 Section B.4 discusses a number of general guidelines applicable to the entire operating test, and sections B and C of ES-3.2 provide specific guidance applicable to the walk-through, including the requirements to distribute the JPMs among the applicable safety functions and administrative topics, to limit the repetition of tasks from the previous licensing exam, and to include new and modified tasks on each test. Although ES-3.1 and 3.2 do not specify the use of systematic or random sampling for the operating test as ES-4.1 does for the written exam, it would certainly be an acceptable method for determining the test content.

3.1.4 (301.14)

Would it be appropriate to do an administrative job performance measure during the systems or dynamic portion of the operating test?

Yes. ES-3.1 Section B.4 encourages examiners to integrate the evaluation of the administrative topics into the systems and simulator evaluations because it improves the flow of the operating test. For example, as noted in Section B.3.d of ES-3.2, the "Emergency Plan" can be evaluated by integrating it into a simulator transient that requires implementation of the emergency plan.

Similarly, an alternate path job performance measure in which a component fails could set the stage for an equipment clearance job performance measure for "Equipment Control." As noted in Section B.3, the applicants' proficiency in the administrative topics should be deliberately evaluated and not inferred from observations made during the simulator operating test.

Moreover, in accordance with Section G.15of ES-3.5, examiners will limit their discussions with the applicants while the scenarios are running so as not to create a distraction.

3.1.5 (301.16)

Is it NRC policy for every JPM [job performance measure] to have adverse safety consequences if the operator makes an error?

No. As stated in Section B.6.a of ES-3.1, the K/As covered during the operating test should have importance factors of at least 2.5. Moreover, as stated in Section A.4 of ES-3.2, the JPMs should, individually and as a group, have meaningful performance criteria that will provide a legitimate basis for evaluating the applicant's understanding of and ability to safely operate the associated systems and the plant. Although Section C.1 of ES-3.6 requires examiners to explain the safety consequences (as applicable) of the applicant's errors, this should not be misconstrued as a requirement for every JPM to have adverse safety consequences if the applicant makes an error.

22

Refer to Question 301.6 in the archived OLPF document for a related discussion regarding discriminatory JPMs.

3.1.6 (301.17)

Does the exam have to cover radiological protection and emergency planning (10 CFR 55.43)?

Why does there have to be an administrative JPM on radiological items/E-plan for RO's?

This is general employee training (GET) material.

Why are GET-type radiation area, contaminated area, radiological work permit (RWP)

JPMs involved in a license exam? These are not discriminatory to a safe licensed operator. GET should be left to GET and eliminated as a part of the licensing exam.

The regulations currently require the written examination and the operating test to cover a representative sample of the items listed in 10 CFR 55.41 and §55.43 (depending on the license level) and 55.45, respectively, to the extent that they are applicable to the facility. With regard to testing GET-type topics, exam developers should strive to write questions or JPMs that test the applicants at a licensed level, such as their response to a problem that would be part of their licensed duties. Refer to Question 3.1.1 (301.2) for a discussion of "representative sampling."

23

ES-3.2 Developing Job Performance Measures 3.2.1 (301.1)

One of the recognized factors for test item validity is differentiation of job position, however, the walk-through examination has a significant portion done in the plant, outside the control room. These tasks are non-licensed operator level, thus, fail to differentiate for the job positions of reactor operator or senior operator.

10 CFR 55.45(b)(1) requires the operating test to be administered in a plant walk-through and a simulation facility. Therefore, it would not be possible to eliminate the in-plant portion without first amending the regulation. Reactor operators and senior operators need to be familiar with in-plant operations that they oversee and could conceivably be called upon to perform during emergency situations. Per ES-3.2 of NUREG-1021, tasks selected for the walk-through should have meaningful performance requirements and their K/A (knowledge and ability) importance factors, which were derived by a panel of subject matter experts from the industry and NRC, should be at least 2.5.

3.2.2 (301.8)

When determining allowable JPM overlap for a retake applicant, do you use the exact 10 JPMs the applicant saw on the original exam or the entire JPM set used for the exam?

(These numbers could be different.)

In accordance with ES-3.2 of NUREG-1021 (refer to Form ES-3.2-2), the current systems walk-through may repeat up to 3 JPMs randomly selected from the last two licensing examinations (including all the operating test sets) at the facility. However, the 30% is an upper limit and may not be appropriate in the case of retake applicants. Section A.4 also prohibits the repetition of any exact-same items from the applicant's audit test or tests, in the case of retake applicants.

Similar items (with different success paths) may be acceptable and shall be identified to the NRC chief examiner for approval.

3.2.3 (301.9)

What is the difference between a faulted JPM and an alternate path JPM?

The concept of alternate path JPMs is discussed in some detail in ES-3.2 of NUREG-1021.

Although most alternate path JPMs do involve some sort of system fault, the goal is to assess the applicant's response to a situation that is not as it should be or is somehow different from what the applicant might have expected based on the initiating cue for the task.

Faulted JPMs are a subset of alternate path JPMs.

24

3.2.4 (301.10)

Use of 4 of 10 alternate path JPMs I believe is "negative" training and evaluation. I expect our plant to operate every time. Maybe for 2 of 10 faulted is fine. 4 of 10 will train the operators to expect the plant controls not to function. Should the initial license examination be PRA based?

The NRC is sensitive to the issue of negative training but is also obligated to ensure that the licensing examinations do not become predictable and effectively differentiate between safe and unsafe applicants. Experience shows that some JPMs may not provide an adequate basis for evaluating the applicants' understanding of the system unless they require the applicant to exercise an alternate success path. Therefore, the number of alternate path JPMs was increased to compensate for the elimination of prescripted questions with every JPM. As discussed in the previous question, system faults provide only one source of alternate path JPMs. It would certainly be appropriate to use risk insights when selecting operator actions to be tested using alternate path JPMs.

3.2.5 (301.13)

What is counted in the simulator?

As stated in ES-3.2 section D.1.c, an applicant should only be given credit for those events that require the applicant to perform verifiable actions that provide insight to the applicant's competence. The required instrument and component failures should normally be completed before starting the major transient; those that are initiated after the major transient should be carefully reviewed because they may require little applicant action and provide little insight regarding competence. Each event should only be counted once per applicant; for example, a power change can be counted as a normal evolution OR as a reactivity manipulation, and, similarly, a component failure that immediately results in a major transient counts as one or the other, but not both.

25

ES-3.3 General Testing Guidelines for Dynamic Simulator Scenarios 3.3.1 Regarding the following criterion in ES-3.3, C. Critical Task Methodology, Step 1 for identifying critical tasks: Tasks that directly lead to the restoration of one or more safety functions. Is this referring to vendor-specific safety functions or the safety functions listed in the applicable K/A catalog?

Safety functions in the context of the NUREG-1021 critical task criteria mean plant functions that are monitored during an event at that plant. Consider the following examples by plant type:

• NUREG-0696, section 5.5, defines that the important plant functions that must be included on a plants safety parameter display system must include, at a minimum, the following:

o Reactivity control o Reactor core cooling and heat removal from primary system o Reactor coolant system integrity o Radioactivity control o Containment integrity

• For Westinghouse, the critical safety functions are:

o Reactivity o Core Cooling o Heat Sink o RCS Integrity o Containment o Inventory

• For CE plants, the critical safety functions are:

o Reactivity Control o Maintenance of Vital Auxiliaries o RCS Inventory Control o RCS Pressure Control o RCS and Core Heat Removal o Containment Isolation o Containment Temperature & Pressure Control o Containment Combustible Gas Control

• For B&W plants, critical safety functions are not defined, but specific control functions are used for event mitigation:

o Reactivity Control o Reactor Coolant Inventory Control o Reactor Coolant Pressure Control o Steam Generator Pressure Control o Steam Generator Inventory Control 26

• For BWRs, critical safety functions arent defined either, but event mitigation is based on control of the following parameters:

o Reactor Power o Reactor Pressure Vessel Pressure o Reactor Pressure Vessel Level o Containment Pressures o Containment Temperatures o Suppression Pool Water Level o Radiation Release Since not all plant designs use the terminology of safety functions, other criteria for identifying CTs was included in NUREG-1021. The task must meet at least one of the items listed in ES-3.3, Section C.1.

3.3.2 Can an automatic reactor trip/setpoint or automatic Engineered Safeguard Feature actuation/setpoint be used as the boundary condition for a CT? If yes, and if the boundary condition is exceeded, then would the associated PD be a CPD or an SPD?

Whether a task is a CT is determined by whether the task meets the criteria for CTs listed in ES-3.3, Section C.1 (i.e., the task meets at least one of the items in the bulleted list of criteria). Also, CTs must possess all of the elements listed in ES 3.3, Section C.2, including objective boundary conditions, which are established to help the examiner assess whether performance of the task was successful or not. Boundary conditions associated with exceeding a plant parameter that are RPS or ESF trip setpoints are objective and are one of the alternative boundary conditions discussed in ES-3.3; specifically, exceeding a parameter value (e.g., limits from the facilitys final safety analysis report or design documentation), as agreed upon by the NRC chief examiner and the facility licensee. Because an SPD is, by definition, a category of a PD that does not meet the criteria for a critical performance deficiency, a PD associated with failure to complete a CT must be graded as a CPD and not as an SPD. The fact that the boundary condition for a CT is a plant parameter that is also an RPS or ESF trip setpoint does not change the way that a CPD is graded, nor does it change the fact that a task meets the criteria to be considered a CT. Similarly, if an applicant fails to perform a task that is not a critical task, and an unintended RPS or ESF actuation occurs, then that PD would be graded as an SPD. The key distinction is whether the task meets the criteria to be considered a CT; the boundary condition does not make the task critical or not critical.

Therefore, NUREG-1021, Revision 12, supports the use of plant parameters that are RPS and ESF trip setpoints as alternative boundary conditions for critical tasks when preferred boundary conditions are not practical or available and when agreed upon by the NRC chief examiner and the facility licensee.

3.3.3 Can manually tripping the reactor be used as a Critical Task?

A manual reactor trip can be used as a CT if the task to manually trip the reactor meets the CT criteria. For example, if the action to manually trip the reactor is a task that directly leads to the restoration of the reactivity safety function (e.g., during an event where the reactor fails to trip automatically after reaching a reactor trip setpoint).

27

3.3.4 Does the requirement for a CT to have a performance feedback element apply to the boundary condition element of a CT?

ES-3.3, C.2 states, in part (underline added for emphasis), During the time span of a CT, performance feedback must be available to at least one member of the crew. This feedback provides the crew members with information about the effect of the crews actions or inaction related to or because of the CT. The crew must be able to determine that its action had an impact or that its inaction is causing plant conditions to degrade. There is no requirement that the performance feedback apply to the boundary condition. Specifically, the performance feedback element must exist regardless of whether the boundary condition has been exceeded.

3.3.5 When is it appropriate to use an alternative boundary condition (ABC) as described in step 2 of the ES-3.3 CT methodology?

ES-3.3, C.2 distinguishes between preferred and alternative boundary conditions (ABCs).

Some boundary conditions are preferred, meaning that they are used or wanted in preference to others (i.e., the ABCs). Given this distinction in NUREG-1021, NRC examiners and industry exam writers are expected to first consider boundary conditions that are preferred.

ES-3.3, C.2 also states, in part, with respect to the performance feedback element of a CT that, The crew must be able to determine that its action had an impact or that its inaction is causing plant conditions to degrade. Definitions for degrade per Merriam-Webster include to lower to an inferior or less effective level and to impair in respect to some physical property. CTs with preferred boundary conditions will in most, if not all cases, provide a more significant or observable degree of performance feedback - specifically, indications that inaction is causing plant conditions to degrade - when the applicant has not taken the needed action as compared to a CT with ABCs. However, the phrase inaction is causing plant conditions to degrade does not mean that a preferred boundary condition has been or has to have been exceeded.

If it is not possible to establish a preferred boundary condition (for example, the condition cannot be reached during the length/duration of the exam scenario), then an ABC should be considered.

Boundary conditions, including the ABCs, must not be arbitrary (defined per Merriam-Webster as existing or coming about seemingly at random or by chance). However, ABCs are not arbitrary solely because they are not preferred boundary conditions. ES-3.3, C.2 states that boundary conditions are agreed [upon] limits for what is acceptable for task completion and what constitutes task failure. During examination development, the NRC chief examiner and the facility licensee determine and validate the boundary conditions (whether preferred or alternative) for each CT and ensure that these agreed upon boundary conditions are documented on the associated operating test forms.

28

ES-3.4 Developing Scenarios 3.4.1 (301.4)

Can there be scenario repetition with similar transients?

Although the same scenarios and job performance measures may not be repeated on subsequent days during the examination week(s), events and tasks that are similar to those that were tested on previous days during that examination are permitted provided the actions required to mitigate the transient or complete the task are significantly different from those required on the previous examination. This is consistent with the policy for repeating events and tasks from the applicants' audit examination as stated in ES-3.1 Section B.4.

3.4.2 (301.20)

Is it possible to develop a new scenario by utilizing events from various scenarios from the previous two NRC examinations combined into a unique scenario? Will retake examinations be considered one of the previous two NRC examinations when considering the requirement for two new events?

NUREG-1021, Rev. 12, ES-3.4, Section A establishes requirements to ensure operating test integrity and prevent simulator scenario predictability. The intent of this requirement is to ensure that an applicant cannot predict the subset of events that will potentially be used for their initial examination simulator scenarios. Therefore, creating a new scenario which is composed entirely of events from the previous two NRC examinations (i.e., 100% overlap) would not be acceptable. A new scenario requires that none of the scenarios non-reactivity events that occur prior to or after a major event have been used on the previous two NRC examinations (i.e., no overlap). Additionally, scenarios should not be re-used in any distinguishable pattern such that an applicant could reasonably predict which scenario(s) may be re-used. A simple example is always reusing a scenario from 3 years ago.

Put simply, it requires that all simulator scenarios on NRC examinations contain at least two events which have not been utilized on the previous two NRC examinations. As is stated in Section A, reactivity events are exempt from the requirements of this section. An additional recommendation is that if any major event is repeated from either of the previous two NRC initial licensing operating tests, the examination author should change the major event, the ICs, or subsequent malfunctions (or a combination) to alter the course of action (within the emergency procedures) for the given scenario(s).

When considering which examinations apply towards the previous two NRC initial licensing operating exams, a retake examination is considered applicable to this provision assuming the retake examination consisted of at least one operating test scenario.

29

3.4.3 (302.4)

Can we use more than 2 ROs if Technical Specifications (TS) require it? Does this apply to administrative requirements (e.g., however ops may use more than 2 ROs)?

Can we increase the number of candidates/scenarios?

If the facility's TS (not administrative procedures) require more than 2 ROs in the control room, the NRC will allow additional surrogates during the simulator operating test to fill the normal crew complement. There will never be more than two RO applicants on any simulator operating crew. Refer to ES-3.4 Section B.

3.4.4 Can a spare scenario that was unused on an initial license examination be used on a subsequent examination?

Yes, within the limitations of NUREG-1021.

OLMC-520, Operator Licensing Examination Records and Documentation, that was issued in February 2023 allows for delayed public release of examination material, including the spare scenario, for two years or up to 2.5 years upon request by the facility licensee. NUREG-1021, ES 3.4, Developing Scenarios, section A, General Instructions, states in part, Events found in spare scenarios will count as previously used events if they were made publicly available in the NRCs Agencywide Documents Access and Management System [ADAMS]. While a spare scenario and its associated Form 2.3-2 are in a delayed public release status, they are not available to applicants. Therefore, the spare scenario events are not considered to be made publicly available and they would not count as previously used events until they are publicly available.

ES 3.4 states, If the scenario is extracted from the facility licensees bank, it must be altered to the degree necessary to prevent the applicants from immediately recognizing the scenarios based on the ICs, the sequence and repetition of events used, or other cues. Therefore, bank scenarios must be modified whether or not the scenario is publicly available in ADAMS. Also, if an individual could recognize a scenario because they previously validated that same scenario, then the scenario needs to be significantly modified before using it on the license examination.

30

ES-3.5 Administering Operating Tests 3.5.1 (302.1)

If the shift technical advisor is licensed, are they at risk if they are a surrogate? Can anyone do it?

Can a formerly licensed or certified person be used as a surrogate on an initial examination?

If a licensed operator is filling the role of a surrogate operator, and they perform errors, is their license in jeopardy (by the NRC)?

Section B of ES-3.4 (in NUREG-1021) addresses the use of surrogates and shift technical advisors.

Although licensed operators are generally preferred, NUREG-1021 does not require the surrogate operators during the dynamic simulator operating test to be licensed. Anyone who does play a surrogate role must be knowledgeable and competent because, per ES-3.5 section D, they will be expected to assume the full responsibilities of the roles they take during the test.

Using unqualified surrogates may place the license applicants at greater risk of failure if the surrogate makes an error.

Surrogates who are licensed operators are at risk because the NRC expects facility licensees to take remedial action (including removal from licensed duty, retraining, and testing, as appropriate) if a licensed operator makes significant performance errors during the operating test or while on shift in the control room.

The NRC could take licensing action against the individual pursuant to Subpart G of 10 CFR 55, but it has never done so in the case of an operator filling a surrogate role during a simulator operating test. The NRC would only take such an action as required to protect the public.

3.5.2 (302.2)

Can an applicant fill the STA role during a scenario? If yes, can they actively fill the role or will "normal" surrogate activity be expected?

No. Section G.1 of ES-3.5 clearly states that the only senior operator position that can be filled by an SRO applicant during the simulator operator test is that of the senior licensed operator responsible for control of the unit.

31

3.5.3 (302.3)

What role can the STA play when they are the extra person?

What determines if an STA is necessary?

Although the rules now allow the use of surrogates as STAs, we severely limit the surrogates role as part of the team. This results in training the candidates under conditions, roles and responsibilities that are different than real operating practice and standards. Why do we limit the STAs role resulting in a "train for the exams" culture?

As stated in Section G item 4 ) of ES-3.5 (in NUREG-1021), consultations with an STA shall be conducted in accordance with the facility licensee's normal control room practice; e.g., an STA shall not be stationed in the simulator if they are on-call at the site. The STA should not take a proactive role in assisting or coaching the applicants because it would hinder the examiners' ability to evaluate the applicants' competence. ES-3.5 requires examiners to brief STAs on the content of the scenarios and their expected actions in response to every event. If necessary, examiners will run additional scenarios to make a licensing decision.

3.5.4 (302.5)

Why is videotaping the initial operating test prohibited?

At the time the no-taping policy was set, experience indicated that videotaping would not provide sufficient detail to support individual licensing decisions for every member of the operating crew. Moreover, the practice was considered intrusive to the applicants and examiners, and several facility licensees expressed concern over how the video tapes would be used. This issue was addressed in response to Question Nos. 403 and 404 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses."

In accordance with Section G.8 of ES-3.5, the licensee should, in coordination with the NRC chief examiner, record as many key parameters as possible and provide a copy of the recordings to the chief examiner for use in the grading process. This is particularly important if the applicants failed to accomplish the expected actions and there is a possibility of a test failure. The examiners will collect and retain other forms of documentation (e.g., logs, notes, and checklists) generated by the applicants.

32

3.5.5 (302.6)

Do SRO-upgrade applicants acting as RO panel operators to complete a crew have to have a specific evaluator observe them (B.3)?

No. As noted in Section A.3 of ES-3.5 (in NUREG-1021), if a three-person operating crew consists entirely of senior reactor operator (SRO) upgrade applicants (who do not have to be evaluated on the control boards), the chief examiner may assign only two examiners to observe the crew. Although the applicants in the reactor operator and balance of plant positions may not be individually evaluated, they will be held accountable for any deficiencies, and they will be graded on their ability to operate the controls. SRO-instant applicants will always be individually evaluated by an NRC examiner regardless what operating position they are filling during a given scenario.

3.5.6 (302.7)

Why can't we add a Shift Manager to the NRC-examined crew to handle communications, etc.?

As explained in Attachment I (Section II) of SECY-98-266, the staff does not permit more than one person to fill a senior operator position during the simulator test because the principal duties of the shift manager position (i.e., assuming the role of the emergency director, performing emergency classifications, and making protective action recommendations) are normally a part of the operating test for senior operator applicants.

3.5.7 (302.8)

When evaluating SRO success in "Classifying the [radiological emergency plan] REP" during the operating exam, what criteria do the examiners use for when to start the 15-minute clock (expectation)? (15 minutes from event to classification)

Since the simulator operating tests for the initial licensing examination are conducted with only one applicant in the SRO position, the NRC does not require the SRO to complete the emergency classification within the normal period of time. In most cases, the applicant is asked to classify the event after the scenario is complete and the simulator is in freeze. Another option is to do a separate emergency plan classification as a JPM, which is only considered time-critical if the facility licensee has a validated time standard.

3.5.8 (302.9)

Do you tell a person that it is a time-critical task?

Yes. Section D.6 of ES-1.2 requires examiners to describe the initial conditions, explain the task to be completed, explain which steps to simulate and which ones to discuss, and indicate whether the task is time critical.

33

3.5.9 (302.10)

If during a JPM, the applicant misses or skips a procedure step or steps and later on recognizes that they missed the steps - can they request to start the JPM over?

No. The applicant cannot start the JPM over but can perform the missed step(s) after complying with the facility's policy for reporting procedural errors and receiving permission. This is consistent with the grading policy in Section B.2 of ES-3.6 (in NUREG-1021), which states that if an applicant initially misses a critical step, but later performs it correctly and accomplishes the task standard without degrading the condition of the system or the plant, the applicant's performance on that JPM would generally still be graded as satisfactory. The examiner would be expected to ask follow-up questions based on the applicant's error, document those questions and answers, and determine a grade based on the applicant's overall performance.

Once the applicant has completed the JPM, they cannot go back and start over, but the examiner will consider any corrected information provided when grading the operating test (refer to Section H of ES-3.5. Note that if an applicant exceeds twice the validated time estimate for any JPM (including time-critical) because they have selected an incorrect procedure or operated the wrong equipment (despite being presented with sufficient plant feedback to correct the error), the examiner should stop the JPM, document the circumstances, and proceed with the next JPM. However, if the applicant is on the correct path but has simply stopped making progress toward completing a non-time critical JPM, the examiner should ask the applicant to describe the work to be done and how long it should take to complete the JPM. If the applicant does not then make timely progress toward completing the described actions, the examiner should inform the applicant that the allowed time for the JPM has elapsed and the applicant will be evaluated on the work completed. The examiner should then proceed with the next JPM.

3.5.10 (302.12)

During scenario follow-up questions, is there a "standard" method for applicants to answer open reference walk-through questions (i.e., if fairly certain of answer give it or always look it up)?

There is no standard method for applicants to answer follow-up questions during the operating test. If they are confident that they know the answer, there is no need to look it up. Examiners are not required to confirm the source and looking up every answer can significantly extend the length of the test. Section E of ES-3.5 states that the applicant may use reference information such as diagrams and procedures. Any follow-up questions that do not require any analysis, synthesis, or application of information by the applicant should be answerable without the aid of reference materials. Furthermore, as stated in Section C.3 of ES-1.2, if the applicant needs to consult a reference to answer a follow-up question, the applicant should ask the examiner if it is acceptable to do so. Although there is no specific time limit for any question, an applicant may be evaluated as unsatisfactory on a question if they are unfamiliar with the subject or reference material and is unable to answer the question in a reasonable period of time. Applicants will not be permitted to conduct unlimited searches of the plant reference material during the examination.

34

3.5.11 (601.3)

What is the basis for the statement in Section C.2 of ES-3.51, "Under no circumstances will another operator be allowed to witness an operating test?" There are instances where the crew being examined may want another operator to observe. (e.g., We had an initial license exam during the annual operating test. When the initial license candidate completed their exam and was assigned to a crew, the crew's shift manager requested that the new crew member be able to observe their operating test from the simulator instructor's booth.)

The bases for this policy include the desire to minimize undue stress on the operators (or applicants) that are being evaluated and the need to minimize crowding in the simulator (for the examinees, NRC examiners, facility evaluators, operations and training representatives, and simulator operators that have to be there). Moreover, the NRC believes it is inappropriate to use NRC-conducted licensing and requalification examinations as training tools for other applicants and operators. Facility licensees are free to establish their own examination policies for requalification examinations in which the NRC is not involved.

35

ES-3.6 Grading and Documenting Operating Tests 3.6.1 (302.11)

If an applicant shows system knowledge weaknesses during administration of a JPM, how far can the examiner go with the unscripted questions? Can the examiner ask questions about another system or another function of the same system covered in the JPM?

As stated in Section B.2 of ES-3.6, the examiner should ask question as necessary to confirm the applicant's understanding of the system as it relates to the task that was performed. The examiner should not ask questions about another system or another function of the same system unless it relates to the task that was performed.

3.6.2 (303.1)

There are no longer going to be prescripted follow-up questions for job performance measures, but job performance measure questions will be evaluated - please explain.

Revision 7 of NUREG-1021 required every system selected for evaluation in the walk-through operating test to be examined with a job performance measure, at least two prescripted questions, and additional follow-up questions as deemed necessary by the examiner to investigate the applicant's performance deficiencies. Although Revision 8 of NUREG-1021 eliminated the prescripted questions, examiners are still required to ask for-cause follow-up questions, if necessary, based on the applicant's performance and to consider the applicant's answers to those questions in the grade for the applicable system. (Refer to Section B.2 of ES-3.6.)

3.6.3 (303.2)

ES-3.6 needs more specific documentation for final results (i.e., some way for very specific feedback to candidate).

Section B.1 of ES-3.6 requires examiners to document every deficiency noted during the operating test. However, only those deficiencies that contribute to a test failure need to be justified in detail. The test report is not intended to be a retraining vehicle; the facility licensee should be able to take the information provided and develop more specific feedback and training for the applicants.

3.6.4 (303.3)

Will operating test follow-up questions be documented?

Can they fail an applicant even though the applicant accomplished the critical step (task)?

Yes. Section C of ES-3.6 for a performance deficiency related to a follow-up question, document the follow-up question and the applicants response.

36

Yes. Per Section B.4of ES-3.6, an applicant could fail even though all the critical steps were accomplished. The examiner must justify the basis for the unsatisfactory grade in accordance with Section B.4.

3.6.5 (303.5)

Is there written guidance on pass/fail for follow-up questions?

Yes. Section B.2 of ES-3.6 (in NUREG-1021) describes how examiners will grade the job performance measure follow-up questions. NRC examiners bear the burden of justifying an unsatisfactory grade for the system if the applicant was able to accomplish the task standard.

Both the chief examiner and the regional operator licensing branch chief must also concur in the failure recommendation.

37

3.6.6 (303.6)

If a candidate is performing a JPM, and during the performance of the task performs an unsafe action with respect to personnel safety, does this constitute a failure of the JPM?

It may, depending on the safety significance of the applicant's action. Section B.8 of ES-3.6 allows the NRC examiner to recommend a failure if an applicant made an error with serious safety consequences even if the grading instructions in Section D would normally result in a passing grade. Normally, this would require adverse consequences related to reactor safety, however, it could also apply to personnel safety issues with potentially serious consequences.

Under such circumstances, the examiner shall thoroughly justify and document the basis for the failure on Form 3.6-4. Moreover, the NRC regional office shall obtain written concurrence from the NRR operator licensing program office before completing the licensing action.

3.6.7 What kind of additional information may the NRC request from the facility if the NRC is evaluating whether to implement the allowance in NUREG-1021, Revision 12, ES-3.6 Section B.8 to fail an individual who demonstrated a performance deficiency with serious safety consequences, even though the competency grading per the ES-3.6 instructions would indicate a pass? What impact will that have on licensing actions?

10 CFR Part 55.31(b) states, in part, that the NRC can require further information to enable it to determine whether to grant or deny an application. If the NRC is considering exercising the provision in the NUREG to recommend an operating test failure due to the applicant demonstrating a performance deficiency with serious safety consequences, then the NRC may request additional information from the facility licensee to help determine the safety consequences of that performance deficiency. Examples of the types of information that may be requested include, but are not limited to:

• additional data, possibly from a targeted simulator scenario beyond that performed during the examination week,

• procedures such as the Severe Accident Mitigation Guidelines,

• accident analyses, including newly developed analyses based upon the specifics of the performance deficiency identified, and

• radioactive release analyses based upon the specifics of the performance deficiency identified.

The staff will consider the time and effort required from the facility to perform any additional analyses or data collection as well as the other information currently available to determine if the additional analysis is necessary. Although the staff will try to maintain the timeframe discussed within ES-5.1, section D to perform licensing actions within 30 days of receipt of the postexamination package, if a request for additional information under 10 CFR 55.31(b) is issued, a reasonable timeframe will be provided to the facility to submit the requested information. This may mean that the licensing action associated with the individual in question may be delayed beyond the goals stated in the NUREG.

38

ES-4.1 Developing Written Examinations Outlines 4.1.1 (401.1)

I do not feel that the written exam is a discriminatory tool. How many people fail the written exam but pass the operating test? Let us use our process to take care of the written with our audit exam.

Recommendation noted. As is evident from the transition program that was completed in 1999, the NRC is generally in favor of increasing power reactor facility licensees involvement in the examination process. Additional changes are possible if the NRC concludes that they will reduce unnecessary regulatory burden, increase public confidence, improve efficiency and effectiveness, and maintain reactor safety.

The NRC has not analyzed applicants' grades on the written exam and operating test to see how well they correlate. However, it is true that some applicants who fail the written examination do quite well on the operating test, while others who fail the operating test perform well on the written exam. The NRC believes that both parts of the licensing examination are important. As discussed in Section D of Appendix A of NUREG-1021, the importance of knowledge testing (i.e., the written exam) should not be underestimated since knowledge is the underpinning of professional performance. The objectives of knowledge testing are varied; they may include assessment of fundamental understandings as well as testing more advanced levels of expertise. The most effective tests of knowledge include questions and test items that measure applications of knowledge directly related to the job. In the case of the NRC operator licensing examination, the written examination provides a key measure that allows a confident decision to be made on the safety significant performance of the individual seeking a license.

39

4.1.2 (401.2)

There are still occasions in NUREG-1021 for examination requirements that are subjective and, therefore, can (and will) vary from region to region and examiner to examiner.

What are the objective criteria for determining that an exam question is satisfactory or unsatisfactory?

The criteria for determining whether a written examination question is satisfactory are summarized on Form ES-2.3.5 and discussed in Appendix A and some examples are provided in Appendix B of NUREG-1021.

The NRC acknowledges that some of the guidance in NUREG-1021 still requires examination authors, NRC examiners, and their supervisors to judge the level of knowledge, level of difficulty, quality of distracters, and other psychometric aspects of the examination.

Nevertheless, the NRC believes that writers of examinations and NRC examiners who are trained in the subject matter, measurement principles, and psychometrics, and who have general knowledge of operator and trainee performance on similar test items, can make informed judgments in these areas based on the guidance in NUREG-1021.Section II of of SECY-98-266, the paper that forwarded the final operator licensing examination rule change to the Commission for approval, responded to a similar comment.

4.1.3 (401.3)

How do we determine "level of difficulty" for written exam questions?

What is the process for determining the level of difficulty for a question?

Where can I find the criteria for the 1-5 difficulty rating on exam questions?

A level of difficulty should be established that differentiates between applicants who have and have not mastered the required knowledge, skills, and abilities. Section B.3.c and B.3.d of Appendix A discuss the concepts of discrimination validity and level of difficulty.

NRC examiners are required to rate the level of difficulty of every written examination question that has not been previously validated by the NRC at that facility. This is done using a 1-5 (easy

- hard) difficulty rating scale as specified on Form ES-2.3.5; questions in the 2-4 range of difficulty are acceptable.

4.1.4 (401.7)

What do you do if your randomly selected questions identify a K/A that you know was not trained on or has been deselected for training? Do you ask it anyway or do you select another system, or does it go deeper?

Can you change a K/A if no one can write a question for it?

What if a random K/A knowledge or ability cannot be used to prepare a discriminating question? Is it fair to replace the K/A with one that is more difficult? (Can we throw out a K/A simply because it is too hard to write a discriminatory question?)

40

Section B of ES-4.1 (in NUREG-1021), allows the examination author to systematically and randomly select another K/A category and/or statement, as applicable, if the systematic selection process identifies a K/A statement having an importance rating that is below 2.5, a K/A statement that clearly does not apply to the subject facility, a generic K/A statement for which it would not be possible to develop a Tier 1 or Tier 2 question, or a K/A category that contains no K/A statements. Failure to train on a selected K/A is not an acceptable basis for selecting another one. The author should use Form ES-4.1-1, "Record of Rejected K/As," or an equivalent, to document the basis for excluding from the examination outline any K/A statements that were randomly selected and submit the form to the NRC with the completed outline.

As stated in Section B.3 of ES-4.2, if it becomes necessary to deviate from the previously approved examination outline, the facility contact is expected to discuss the proposed deviations with the NRC chief examiner and obtain concurrence. The facility should be prepared to explain why the original proposal could not be implemented and why the proposed replacement is considered an acceptable substitute.

4.1.5 (401.9)

How close must the provided outline model be to the actual outline for a specific facility?

As stated in Note 2 on the bottom of Form ES-4.1-BWR and Form ES-4.1-PWRof NUREG-1021, the actual point totals for each group and tier on the proposed examination outline must match those specified in the applicable table. However, the final point total for each group and tier, based on revisions required by the NRC reviewers, may deviate by 1 from that specified in the table. The final RO exam must total 75 points and the SRO-only exam must total 25 points.

4.1.6 (401.11)

Technical specifications (TS) are too complicated to memorize. They should be open reference or better yet covered by the operating exams (JPM). We do not want our operators to spend valuable time memorizing TS, nor do we want them to operate from memory.

The NRC does not expect operators to memorize the TS, nor does it endorse operating the plant from memory. However, the NRC does expect operators to recognize TS entry conditions, immediate actions, and (in the case of senior operators) bases when presented in a multiple-choice format on the written examination. If they do not compromise the integrity of other questions on the exam, it is acceptable to provide extracts from the TS to the license applicants for use in answering application-level questions.

4.1.7 (401.12)

Based on the SAT-based training program, you test on objectives. The current NUREG-1021 allows asking questions not covered by the utility's training program (objectives).

This is contrary to the SAT-based training system. Should there be a way to ensure the students are examined on the training program content? (If it is determined that the program is SAT.)

41

Learning objectives are not required for the NRC examination, but our SAT-based program still requires them. Do we no longer follow our SAT-based program? (Section II) to SECY-98-266, the Commission paper associated with the April 1999 final rule, responded to a similar public comment on Interim Revision 8 of NUREG-1021.

It notes that Sections 55.41(a), 55.43(a), and 55.45(a) of the rule states that the knowledge, skills, and abilities selected for evaluation on a written examination and an operating test will be identified, in part, from learning objectives derived from a systematic analysis of licensed RO and SRO duties performed by each facility licensee. While the answers to Questions 129 - 130 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses," confirmed the NRC's intent that the training program's learning objectives would become the major source of the licensing examination, it also cautioned that the NRC would not be limited to those learning objectives.

The NRC licensing examination is not a part of the facility licensee's SAT-based training process. The systematic sampling procedures for preparing the written and walk-through examination outlines per NUREG-1021 are designed around the structure of the NRC's K/A Catalogs (NUREG-1122 and -1123) and may not be compatible with the facility-specific task lists. NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific knowledge and ability requirements on a case-by-case basis. Allowing facility licensees to substitute their entire site-specific task lists for the NRC's K/A Catalogs could decrease the level of examination consistency. The current approach of requiring facility licensees to explain deviations from the NRC's K/A Catalogs is conservative, consistent, and effective.

Facility licensees should continue to follow their SAT-based training programs, with the understanding that the content of the NRC licensing examination is not necessarily restricted by the SAT-based training process. Licensees should consider developing learning objectives covering all the topics required by 10 CFR 55 and all the NRC K/As having importance ratings of 2.5 or higher, unless it can demonstrate that the K/A is not applicable at their facility.

4.1.8 (401.15)

Once we use a comprehensive level question, does it become a knowledge-based question the next time we use it?

No. The cognitive level of any question taken from the bank will be counted at its face value, even though it may function at a lower level because it is available for study. Section B.3.e of Appendix A of NUREG-1021 discusses cognitive level of questions.

42

4.1.9 (401.19)

Administrative-type items are best suited to open-referenced method because of the expectation for these items in the actual job position. However, the written examination, a closed-reference format, has a significant percentage of administrative questions. This appears contradictory.

10 CFR 55.41(a) and §55.43(a) require the written examinations for operators and senior operators to sample a number of administrative topics. The administrative questions that are used on the written exam should generally be answerable based on recall and/or recognition.

As discussed in Section B.6 of ES-4.2, under certain conditions, selected reference materials may be provided to the applicants as attachments to the written examination.

4.1.10 (401.29)

Regarding ES-4.1: How do you assure that the extra 10 CFR 55.43 topics are covered in a "representative sample" in the test outline?

The SRO-only examination outlines sample only those K/A categories that are linked to 10 CFR 55.43(b), including a number of the generic K/As in Section 2 of the catalogs (NUREG-1122 and

-1123) and all of the Category A2, AA2, and EA2 K/A statements. All the K/A categories related to the fuel handling facilities are also subject to sampling because that system is specifically identified in 55.43(b)(7). As stated in Section B of ES-4.1, the specific topics to be sampled on the examination shall be systematically selected.

4.1.11 (401.36)

According to ES-4.1, the 25 "SRO-level" questions on the written examination shall be derived from the seven areas in 10 CFR 55.43. However, this guidance is sometimes being misinterpreted such that questions testing 10 CFR 55.43 topics are being rejected as "SRO-level" if the facility licensee also expects ROs to possess the same 10 CFR 55.43 knowledge. Is it correct to say that an "SRO-level" question is simply different from the questions on the RO examination and related to one of the seven items listed in 10 CFR 55.43 (b)?

The fact that a facility licensee expects its ROs to master certain 10 CFR 55.43 knowledge, skills, and abilities does not mean that they can no longer be used as the basis for "SRO-level" questions. However, ES-4.2 also requires questions to be written to be appropriate for the job level being examined. Therefore, "SRO-level" questions need to be carefully constructed to ensure that they accurately test the additional knowledge and abilities required for the higher license level according to 10 CFR 55.43(b). For example, both 10 CFR 55.41(b)(10) and 55.43(b)(5) require emergency operating procedure (EOP) knowledge, but the latter requires the "SRO-level" questions to evaluate the additional knowledge and abilities necessary for "assessment of facility conditions and selection of appropriate procedures during ... emergency situations." Questions that evaluate the knowledge of specific bases for EOPs (K/A 2.4.18) and/or the operational implications of EOP cautions (K/A 2.4.20), but not the higher level "assessment and selection" knowledge, would generally not be valid "SRO-level" questions.

However, questions that evaluate K/A number 2.4.21 (knowledge of the parameters and logic used to assess the status of EOP safety functions) would generally be considered valid "SRO-level" questions even if the facility licensees SAT-based program has identified this additional 10 CFR 55.43(b)(5) knowledge as an RO job requirement. Consequently, questions that test 43

knowledge and abilities per 10 CFR 55.43(b) can be considered "SRO-level" per Section E.1 of ES-4.2 even though the facility licensees training program requires the same level of knowledge for its ROs.

4.1.12 (401.40)

Why does a group with only 1 or 2 safety-significant knowledge and abilities (K/A's) have as much weight as one with 200? Can NUREG-1021 be changed to remove this artificiality?

The NUREG-1021 superstructure forces you to sample the systems K/A of about the same rate (1 or 2) per system. However, some systems have 5 K/As that are above 2.5 and some have 200. This forces you to over-sample some systems and under-sample others. Can the superstructure be realigned to eliminate this problem by lumping all the system K/As together and selecting the number needed from the total?

The relative safety-significance of the plant systems and emergency/abnormal plant evolutions (E/APEs) was considered by the team of industry and NRC subject matter experts that originally designed the 3-tiered written examination sample plan (as part of NUREG/BR-0122 "Examiners' Handbook for Developing Operator Licensing Written Examinations") that has since been revised to 4 tiers in ES-4.1 of NUREG-1021. The more important items that are included in Group 1 are weighted much more heavily than the items of lesser safety significance that are included in Group 2.

4.1.13 (401.42)

Why is it valid to use a closed reference exam for initial license exams when it is really important that the operator use all of the tools available to them on shift? Where is the NRC headed on the use of open-reference requalification questions on initial exams?

Open-reference items on the initial license examination should be used judiciously and sparingly because the examination should focus on the broader content areas that rely primarily upon learned information, committed to memory.

In nearly every field of study (e.g., medicine, law, and education), the testing required for initial licensing or certification is more demanding than that required to maintain certification. The rationale is that newly licensed personnel should possess a broad body of knowledge and ability to perform their job independently and without the aid of supplemental knowledge contained in procedures. This by no means suggests that procedures should not be used, but rather that initial license testing should emphasize those areas where procedures need not be used.

Through their training, operators must learn setpoints, immediate actions, system designs and interrelationships, administrative procedures, and applications of knowledge to the job. The knowledge that is learned is expected to be demonstrated through the NRC examination format that measures recognition and recall of safety-significant knowledge without relying on references. This approach is consistent with the timely retrieval of information that may be required during the licensed operators' job and that might otherwise not be possible if the applicants prepared only for open-reference examinations. If too many open-reference questions are allowed on the initial licensing examination, the need and ability to learn and retrieve a broad body of knowledge would be lessened. Similarly, the confidence that the baseline body of knowledge had been truly established could be questioned.

44

Once initial competency is assured, then ongoing training and testing, which is more review-like, focused and specialized in nature, can make more appropriate use of the open-reference format, as is done on requalification examinations. However, for the reasons stated above, the NRC does not plan to increase the limited and judicious use of open-reference questions on the initial license examination.

4.1.14 (401.45)

Is the following scenario acceptable for purpose of controlling any overlap from the audit to the NRC exam? 1. Audit exam is last years NRC exam. It was developed using randomly generated sample plan 1 year ago. 2. NRC exam is developed using randomly generated sample plan. Some overlap occurs in K/As tested on the audit and the NRC exam.

Yes. Since both examinations were randomly generated and presumably the questions match the selected K/As, it is acceptable. Some overlap may occur.

4.1.15 (401.46)

For a written retake exam, the subsequent audit exam focuses somewhat on identified weaknesses from the previous NRC exam. Therefore, the audit exam is not totally random. Is this acceptable?

How do we apply the audit/screening exam criteria for written re-exam efforts? Does an upgrade remedial program for the applicant exam count as an audit? Since 60 days have elapsed, does the initial audit exam fall into the bank question category?

ES-1.3 of NUREG-1021Section D discusses acceptable methods for ensuring that the audit exam does not compromise the licensing exam. The example given would be acceptable if the audit exam is finalized before the NRC exam development is started or if there is no duplication between the audit and the NRC exam. As long as the NRC licensing examination is developed using the random and systematic process described in ES-4.1, there are no restrictions on repeating questions from any prior examinations and quizzes, including old audit and licensing exams. Once an audit or any other exam is given, all the questions on that exam would be considered bank questions that could be used to evaluate the associated K/A if it is randomly selected for a subsequent examination. However, the content of any practice or audit exam or quiz that the facility licensee develops after it starts working on an NRC licensing examination would have to be controlled to protect the integrity of the licensing exam.

45

4.1.16 (401.49)

K/A Categories A-3 Monitor Auto operation of ... and A-4 Manually operate ... dont seem to be well tailored to a written exam. These topics for the written exam are almost always covered in K1-6, A1, or A2. Why not eliminate these categories from the NUREG-1021 superstructure since they are more properly tested by the operating test and the knowledge is already sampled by K/As in other categories?

Recommendation noted. However, questions can be written to test the applicants' ability/knowledge of proper automatic operation and how to manually operate a component or system.

4.1.17 (401.55)

Some Tier 1, Emergency and Abnormal Plant Evolutions, written examination questions have been categorized as deficient, and in some instances, Unsatisfactory as a result of the NRC Form ES-2.3-5 Written Examination Review Worksheet because they do not test knowledge of, or information contained in, the sites abnormal operating procedures (AOPs) and emergency operating procedures (EOPs).

Is a proposed Tier 1 written examination question deficient or unacceptable if it does not do that?

NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Revision 12 (NUREG-1021), is used in conjunction with the applicable knowledge and abilities (K/A) catalog to develop content-valid examinations. Section 4.0 of each K/A catalog contains the K/A statements for emergency plant evolutions (EPEs) and abnormal plant evolutions (APEs). As defined in the K/A catalogs, an emergency plant evolution is any condition, event or symptom which leads to entry into emergency operating procedures (EOPs), and an abnormal plant evolution is any degraded condition, event, or symptom not directly leading to an EOP entry condition, but nonetheless, adversely affecting a safety function."

In accordance with NUREG-1021, Sections ES-4.1, K/A statements for Tier 1 questions are selected from Section 4.0 of the applicable K/A catalog. The objective of Tier 1 questions is to test an applicants knowledge of how to safely operate the plant during emergency and abnormal conditions. These facility evolutions are a significant aspect of operating a nuclear power plant safely, and knowledge of these evolutions, including the use of abnormal and emergency procedures as appropriate, must be robustly understood by each licensed operator.

All Tier 1 questions must meet this objective. Satisfactory ways of meeting this objective include, but are not limited to, the following: (1) information contained in the sites procedures, including alarm response procedures, AOPs, EOPs, and their associated bases documents; (2) diagnosis that leads to selection of the procedures that should be used to respond to the evolution, (3) the progression of an event, and (4) assessment of the integrated plant response to emergency or abnormal situations crossing several plant systems and/or safety functions.

The selected K/A statement from the approved sample plan will influence how the Tier 1 question is written to meet this safety objective. If it is not possible to write a Tier 1 question that meets this objective, then the K/A may be replaced with a more suitable K/A.

46

Accordingly, if a Tier 1 question does not meet the safety objective of Tier 1 questions, then it may be rated as either unsatisfactory (i.e., in need of repair or replacement) or in need of editorial enhancement on Form ES-2.3-5 depending on the extent to which the question must be changed to meet the safety objective of Tier 1 written examination questions. For example, a question may be rated as in need of editorial enhancement if the addition or deletion of a sentence or phrase would be sufficient to meet the objective of Tier 1 questions. A question may be rated as unsatisfactory if only simple editorial changes would not be sufficient to ensure the question meets the safety objective of Tier 1 question (e.g., a proposed Tier 1 question that asks the applicant to identify the power supply to a pump that is designed to provide emergency cooling to the core during a loss of coolant accident would likely be rated as unsatisfactory because it does not test the applicants knowledge of how to operate the plant).

4.1.18 (401.56)

A new revision to the PWR and BWR Knowledge and Ability Catalogs will be published soon. Which revision of the catalog should be used to prepare an upcoming examination outline?

The operating reactor knowledge and ability (K/A) catalogs, NUREG-1122, Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors, and NUREG-1123, Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors, are both currently in Revision 2, Supplement 1. Revision 3 for both catalogs is scheduled to be officially issued on September 25, 2020. This revision represents the culmination of almost eight years of effort from the NRC and the industry.

NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Revision 11, says, Use the latest revision of the K/A catalog (NUREG-1122 or NUREG-1123) available at the time the facility licensee requests the written examination outline. So, a licensee may request a written examination outline verbally or in writing prior to September 25, 2020, and the associated examination will be prepared using Revision 2, Supplement 1 of the respective K/A catalog. On or after September 25, 2020, any request for a written examination outline will result in the preparation of a written examination outline using Revision 3 of the respective K/A catalog.

The staff recognizes that facilities need time to update training programs to Revision 3 of the K/A catalogs. For the purpose of allowing time for change management, facilities may request examination outlines prior to September 25, 2020, with a longer lead time than usual. This will allow facilities with upcoming examinations to use Revision 2, Supplement 1 of the K/A catalogs, particularly for license classes that are beginning close to the revision date or are ongoing. Due to outline development workload and examination security concerns, the staff has asked that facility requests to generate outlines using Revision 2, Supplement 1 of the K/A catalogs be limited to those written exams scheduled to begin prior to August 31, 2022.

47

4.1.19 Some generic K/As are unsuitable for Tier 3 questions on any written examination because they dont list a plant-wide generic topic. Is there a list of K/As that can be prescreened from the Tier 3 portion of the outlines for all PWR, BWR, and AP1000 written examinations?

Yes. The staff reviewed the list of generic K/As for the PWR, BWR, and AP1000 catalogs for topics that are not plant-wide generic topics and instead are about system-specific or emergency-abnormal plant evolution specific knowledge. Note that these generic K/As are still available for selection on the Tier 1 and Tier 2 portion of the written examination outlines. Also note that this list may be updated as more experience is gained.

The staff determined that the following K/As could be pre-screened or deselected from the Tier 3 portion of written examination outlines:

• 2.1.27 Knowledge of system purpose and/or function

• 2.1.46 Ability to use integrated control systems to operate plant systems or components

• 2.2.42 Ability to recognize system parameters that are entry-level conditions for TS

• 2.4.2 Knowledge of system setpoints, interlocks and automatic actions associated with emergency and abnormal operating procedure entry conditions

• 2.4.4 Ability to recognize abnormal indications for system operating parameters that are entry-level conditions for emergency and abnormal operating procedures

• 2.4.18 Knowledge of the specific bases for emergency and abnormal operating procedures

• 2.4.50 Ability to verify system alarm setpoints and operate controls identified in the alarm response procedure 48

ES-4.2 Developing Written Examinations 4.2.1 (205.3)

How is the requirement for questions to have operational validity met for theory (Tier 4) test questions?

In the development process for the initial license written examination, the NRC strives to create questions that are technically, operationally, and psychometrically valid. For example, to achieve operational validity -- a hallmark of good test item writing that seeks to ask questions within the context of the actual job -- we strive to develop questions that assess applicant understanding, use, and application of the safety-significant knowledge that is required for licensing. These types of items assess whether applicants can use and apply the knowledge they learned vice merely recalling the facts.

An operationally valid fundamentals question does assess understanding and application of components, reactor theory, and thermodynamics within a realistic, job-related context of the applicants plant they are to be licensed at.

The NRC endeavors to administer licensing examinations that are valid and reliable indicators of the applicants knowledge and abilities. The most valid operator licensing written examinations use questions that have valid content, operational relevance, and the ability to differentiate between different levels of applicant knowledge. Therefore, the fundamental knowledge addressed by a K/A will often be tested by requiring the applicant to apply the knowledge in the context of a realistic, or operational, setting.

The fact that a specific word or term is absent from a fundamentals K/A statement does not disqualify a related knowledge from being tested on the initial written examination. K/A statements are often written as general statements of required knowledge. Therefore, fundamental questions are not required to contain specific words found in fundamentals K/A statements. However, they are required to preserve the intent of the valid K/A. (Please see Questions 4.2.3 (401.14), 4.1.8 (401.15), 4.2.4 (401.16), and 4.2.11 (401.38) for related discussions).

4.2.2 (401.13)

If learning objectives say that, ". . . given a copy of procedure," can we use as closed reference question?

In accordance with Section B.3 of ES-4.2, a facility learning objective is not necessarily required for every question. However, if one is referenced it should be adhered to unless the licensee makes a conscious decision to deviate from it. In those cases, the licensee should consider revising the learning objective to match the question.

The NRC does not review every learning objective during the approval process. When a question appears on the examination, the NRC will conclude that the facility licensee expects its operators to be able to answer the question without a reference regardless what the learning objective says. If such a question is challenged during a license appeal, the NRC may ask the facility licensee to support the question in writing as discussed in Section C ES-5.2.

49

As noted in Section B.6.d of ES-4.2, reference materials may be used on a selective basis as attachments to or embedded in the written examination, provided they do not give away the answers to any of the questions or improve the applicants chances of guessing the correct answer by eliminating incorrect distracters.

4.2.3 (401.14)

Please clarify the difference between knowledge-based and higher order.

Section C of ES-4.2 of NUREG-1021 discusses Bloom's Taxonomy and briefly explain the three levels of knowledge (i.e., fundamental knowledge or simple memory; comprehension; analysis, synthesis, or application).Section I of Appendix B cites Benjamin Bloom's book on the subject as a reference tool that explains the concept in greater detail.

4.2.4 (401.16)

Regarding the ES-4 series. Discrimination validity should not be evaluated separate from operational validity and content validity. If operational validity and content validity are present, then discrimination will be present if good test item writing principles (e.g.,

plausible distracters, absence of clues) are applied.

Please remove level of difficulty evaluation from Form ES-2.3-5 and all other requirements. There is no need to assess difficulty if content validity, operational validity, and 50-60 higher cognitive level requirements are met.

Why is it unacceptable to have a question with a difficulty rating of "1," if that is what the randomly generated sample plan called for?

Comments noted. However, to determine whether an item has discrimination validity you must ask yourself whether an applicant who has not achieved the minimum level of competence is likely to miss the answer and be drawn to a distractor. Questions can be psychometrically sound, content valid, and operationally valid, but still not differentiate well. Refer to Section B.3 of Appendix A of NUREG-1021 for a discussion of discrimination validity.

The sample plan does not prescribe the difficulty level for questions; rather, the sample plan determines the topical content areas from which test items will be developed. Moreover, K/A importance values should not be confused with item difficulty measures. Easy questions can be created from high importance K/As and difficult questions can be created from low importance K/As.

4.2.5 (401.17)

Why would a validated question not be a good question?

Although a question that was previously used on an NRC examination at the facility since 10/1/95 (i.e., a validated question) may be acceptable in its own right, it may have to be edited or replaced if it conflicts with another question on the examination or if necessary to meet the criteria on the Written Examination Quality Checklist (Form ES-2.3-4 in NUREG-1021).

Technical and psychometric flaws that cause the question to have no or multiple correct answers would have to be corrected regardless when they are identified.

50

4.2.6 (401.23)

If a question is used at a different facility (IP2/IP3) what or where does this fall into the 75/15/10 percent?

For questions taken from a non-facility specific exam bank (e.g., the national exam bank) the questions must be changed as appropriate to make them correct for the facility. In this situation, the question may be different than the original bank question but may not meet the criteria to be a "modified question" and are also not "new". What should these questions be called and how should they be categorized on the ES-2.3-4 form?

At what point does a "modified" question become a "new" question?

In accordance with the glossary of NUREG-1021, a bank question is a written examination question taken from any facility licensee collection of questions that have previously appeared on any operator training-related examination at the facility. This definition includes NRC examination questions used at other facility licensee sites. Additionally, in accordance with Section B.4 of ES-4.2, a significantly modified bank question meets the following: (1) change at least one pertinent condition in the stem and at least one distractor or, (2) change the conditions in the stem such that one of the three distractors in the original question becomes the correct answer or, (3) the NRC chief examiner agrees that the bank question is significantly modified, and an applicant would not be able to arrive at the correct answer because they recognize the question from the bank. . "New" questions, on the other hand, do not have their basis from an existing bank question. Rather, they have been developed from the author's "fresh start" and, as such, are categorized as "new."

4.2.7 (401.24)

If a bank is 100% pre-approved NRC exam questions and the utility modified these to make them site-specific by changing the stem or distracters, can the utility mark them as 100% modified?

The NRC considers all banks to be open and available for study by the license applicants. See the definition 4.2.6 (401.23) and Section B.4 of ES-4.2 that an applicant would not be able to arrive at the correct answer because they recognize the question from the bank. Therefore, the questions can only be classified as modified for purposes of an NRC licensing examination if the modified versions are kept out of the bank until after they are used on an examination. They will show up on an examination only if they match a knowledge or ability that is part of the systematically developed sample plan.

4.2.8 (401.26)

Can the NRC provide examples of questions which contain a "psychometric flaw," in an attachment to NUREG-1021?

Appendix B of NUREG-1021 already contains a number of example questions that illustrate psychometric flaws commonly seen on NRC examinations. The NRC encourages the use of 51

industry-sponsored item-writing workshops as a venue for obtaining and sharing this type of information.

4.2.9 (401.35)

Certain "newer" K/As have a 10 CFR 55 reference given in parenthesis to show a tie between the CFR and NUREG-1122(1123). We were told that questions did not meet the criteria of SRO only (those 25 questions only on the SRO written) if the K/A reference included both 10 CFR 55.41 and §55.43. It is our understanding that questions need be written at SRO knowledge level in these situations. We do not think that this dual CFR reference should be interpreted to eliminate the K/A from being selected for an SRO question.

The policy regarding the 25 SRO-only questions on the written examination is stated in Section E of ES-4.2. The fact that a K/A is linked to both 10 CFR 55.41 and §55.43 does not mean that the K/A cannot be used to develop an SRO-only question. Questions related to §55.41 topics may be appropriate SRO-level questions if they evaluate knowledge and abilities at a level that is unique to the SRO job position as determined by the facility licensees learning objectives.

Although your observation is valid, please note that NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific K/As on a case-by-case basis and to use K/As having importance ratings below 2.5 if it is justified based on plant-specific learning objectives.

When the NRC revised NUREG-1122 and -1123 to incorporate cross-references to specific items in 10 CFR 55, the primary purpose was to establish at least one regulatory connection for every K/A. However, this does not mean that the CFR references are complete and accurate.

The fact that a particular K/A does not reference 55.41 or 55.43 does not, in and of itself, disqualify the K/A from testing on the RO or SRO written examination.

4.2.10 (401.37)

ES-4.2 does not address using a K/A that references 10 CFR 55.43 for testing on the RO written examination; is that acceptable?

Yes, it is. 10 CFR 55.41(a) states that "the knowledge, skills, and abilities [to be tested on the RO written examination] will be identified, in part, from learning objectives derived from a systematic analysis of licensed operator duties performed by each facility licensee and contained in its training program." Although ES-4.2 does not specifically address using a K/A linked to 10 CFR 55.43 to develop an RO written examination question, it does allow the facility licensee to use plant-specific priorities (and a site-specific task list) to justify using an otherwise unimportant K/A for questioning. Therefore, questions associated with topics in 10 CFR 55.43(b) should be acceptable for the RO examination if they are supported by documented RO learning objectives derived from the RO job task analysis at the site.

52

4.2.11 (401.38)

Why are we testing abilities and/or skills on the written exam vs. on the simulator exam (via K/A [knowledge and abilities] catalog)? Shouldn't we test knowledge on the written exam and abilities on the operational portion of the exam?

This question suggests that there is a dichotomy between knowledge and skill testing, when, in fact, knowledge and skill are interrelated, and testing in one format does not preclude assessing understanding in the other format. Although skills and abilities testing are more commonly associated with JPMs and simulator scenarios, it is incorrect to assume that they cannot be tested on the written examination.

Good test items, whether part of a written examination, walk-through, or simulator scenario, should be operationally valid. You should not assume that written questions are passive items where only facts, principles, or concepts are recognized. Ideally, they should assess the applicants' ability to integrate and use information on plant conditions. For example, such questions could require the applicant to use information in the stem of the question to determine appropriate actions or predict system responses. These "scenario style" questions are dynamic in nature, requiring the applicant to sort, merge and integrate contrived, but possible conditions.

They assess at the application level of operator action -- a quality consistent with Bloom's Taxonomy (see Section C of ES-4.2 and Appendix B of NUREG-1021) and the goal of attaining high operational validity. To this extent, operator knowledge and skill are simultaneously embedded within the written test questions.

When it is not possible to test a randomly selected skill or ability on the written examination, then another K/A should be randomly selected. However, as stated in Section B of ES-4.1, the facility licensee shall provide written justification for replacing any randomly selected K/A.

4.2.12 (401.47)

Has the K/A catalog been reviewed, and each K/A evaluated for cognitive level? (Some appear to support only basic Level 1 questioning.)

Are fundamental K/As being eliminated from the K/A catalog? For example: the purpose of charcoal filters in iodine removal systems.

The knowledge and abilities in the NRCs K/A Catalogs (NUREG-1122 and NUREG-1123) have not been reviewed for cognitive level. The K/A catalogs were developed by a group of utility personnel and the NRC and only list knowledge and abilities with importance values related to performing licensed duties. K/As are topical content areas and should not be confused with the cognitive levels of test items; K/A importance values and cognitive level are separate and distinct exam development parameters. The fact that some of the K/As do not support the development of higher cognitive level questions does not make them unusable on the NRC licensing examination because ES-4.2 specifies that 40 to 50 percent of the RO questions will be written at the fundamental level of knowledge.

53

4.2.13 (401.48)

Is there any movement towards going to 3-part multiple choice questions vs. 4? The 4th distracter is very expensive, most times demanding more time than the others combined.

No, that is not being considered. The four-distractor format is the only one acceptable to the NRC. Refer to NUREG-1021, Appendix B.

4.2.14 (401.53)

Guidance provided from examiners to facility authors indicates a limit on the quantity of questions that may have reference material provided to the candidate for initial licensing written examinations.

Specifically, no more than 50% on the SRO portion of the examination has been offered as guidance. However, NUREG-1021 does not provide specific guidance on the percentage of use for these type of questions, and many of the SRO K/As and station training objectives require analysis of conditions along with use of reference materials to determine the correct action. These questions demonstrate operational validity and discrimination for the types of knowledge and abilities an SRO is expected to possess.

ES-4.2 B.6.d describes the criteria for reference material use on the written examination.

Is there a limit on the percentage of written exam questions that utilize references for the initial licensing exam? If so, what is the basis?

Question 4.1.13 (401.42) addresses the differences between initial license examinations and requalification examinations regarding the approach of "closed-book" versus the use of references during examinations. That question properly states that references in closed-book, initial license examinations should be used judiciously and sparingly.

With regard to the initial license examinations, the Agency goal is to ensure that applicants prepare and study a broad, yet defined, body of knowledge. Applicant mastery of such a body of knowledge, as required in 10 CFR 55.41 and §55.43 better ensures that operators will be equipped to address public health and safety needs that may arise in the conduct of their reactor operator (RO) and senior reactor operator (SRO) duties. When an examination relies too heavily on the use of references to answer questions, then the applicants preparation for such an examination is altered; the applicant likely will primarily focus their preparation on the use of references to answer questions and devote less attention on mastering the body of knowledge. In sum, the mental demands, requirements, and format of the examination will determine the applicants method of preparation; moreover, the level of preparation will likely be deeper and more thorough given the expectation of a closed-book, limited reference examination.

The RO and SRO initial license examinations largely and properly remain "closed-book;" current policy regarding the judicious and sparing use of references is appropriate. However, because of the supervisory nature of the SRO position that relies on a greater use of references and because of its separate 25-question examination that addresses content in 10 CFR 55.43, there is a justifiable basis to allow a greater number of references to be used on the SRO exam. Yet, in the spirit of Question 4.1.13 (401.42), both RO and SRO exams, as initial license exams, should similarly rely more heavily on knowledge memory and the application of this knowledge, which the NRC staff believes does not diminish operational validity.

54

In this regard, the following ranges are provided regarding the allowable use of references on initial license examinations consistent with the principles discussed in Question 4.1.13 (401.42)

(note that the ranges below are for references that are provided in addition to the steam tables and the generic fundamentals equation sheet). Note that these quantitative ranges are not absolute limitations, nor should they be construed as goals or requirements. You should also note that NUREG-1021 does not permit any "direct lookup" questions or questions with references that provide an advantage in answering other "closed-reference" questions on the initial licensing examination.

RO (75 items) = up to ~5% or 4 questions SRO (25 items) = up to ~20% - 25% or 5 - 6 questions 4.2.15 (401.54)

According to ES-4.2, Section E, for writing SRO level Tech Spec questions, one screening criteria is to determine if the question can be answered solely by knowing the

'above the line' information. RO candidates are required to know the LCO statement and the Modes of applicability. Is RO knowledge limited specifically to these words in the LCO statement, or is knowing the subparts of the system which makes it Operable also considered required knowledge? The T/S basis contains a statement for the LCO which usually lists the necessary components and lineups which would make a system or component Operable.

Knowledge of the bases information/discussion in Technical Specifications (TS) for limiting conditions for operation (LCOs) is not considered RO required knowledge with the following one exception. Knowledge of and ability to apply less than or equal to one hour TS action statements is considered RO knowledge. In this instance, RO knowledge is NOT limited specifically to the words in the LCO statement and the TS bases knowledge indicating the necessary components and lineups which would make a system or component Operable is appropriate for testing on the RO written examination or operating test. Additionally, information in the TS bases may be RO required knowledge if it is located elsewhere where the RO would be expected to have the knowledge (as an example, if the information is also located in the system description).

In summary, application of knowledge contained within the TS bases (unless also located elsewhere the RO would be expected to know) and NOT associated with an immediate or less than or equal to 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> TS Action Statement should not be tested on the RO examination when testing RO above this line TS knowledge as discussed in ES-4.2, Section E and depicted below:

55

For the TS 3.5.1 example above, a RO applicant would be expected to possess knowledge and understanding of the Above this line information but would generally not be expected to have knowledge of the TS LCO B 3.5.1 system/component parametric values and/or conditions necessary to determine Accumulator System OPERABILITY.

However, ACTION D for Two or more accumulators inoperable as shown below requires that LCO 3.0.3 be entered Immediately. Therefore, in this instance and notwithstanding that the information is provided in the LCO B 3.5.1 Bases, a RO would be expected to understand that for an accumulator to be considered OPERABLE, the isolation valve must be fully open, power removed above 1000 psig, and the TS Surveillance limits for accumulator volume, boron concentration, and nitrogen pressure must be met.

56

4.2.16 (401.57)

Diverse and Flexible Mitigation Capability (FLEX) strategies are not specifically discussed in the knowledge and ability (K/A) catalogs or NUREG-1021. Can FLEX tasks be included on NRC exams?

FLEX tasks can be included on operator licensing examinations, and the information provided below should be considered when selecting test items related to FLEX tasks for NRC examinations.

• Initial Exams: Although FLEX is not specifically mentioned in the K/A catalogs or NUREG-1021, there are K/A statements that may lend themselves to developing a test item on FLEX topics that is of appropriate importance for an initial operator licensing examination. For example, K/As 2.4.16, 2.4.51, and 2.4.52, which address the relationship between the emergency operating procedures and other plant procedures and guidelines, and certain K/A statements associated with the station blackout evolution. Test items on FLEX topics that meet one or more of the following criteria will provide evaluative benefit for making a licensing decision:

o The task is expected to be performed by licensed operators during an extended loss of all AC power scenario; or o The test item covers a FLEX strategy that is credited in the emergency operating procedures or abnormal operating procedures; or 57

o The task is expected to be directed or coordinated by a licensed operator, and the test item allows the examiner to evaluate the applicants familiarity with the design and operation of systems that are located outside the main control room and that might be used to restore a safety function.

• Requalification Exams: FLEX tasks are typically included in the operator requalification program (task training frequency may be anywhere from 2-6 years). Because the content of requalification exams is determined by the topics in the requalification cycle (current and past), FLEX tasks, including those tasks performed by licensed as well as non-licensed plant personnel, may appear on requalification exams (e.g., as a JPM on the walkthrough portion of the operating test). Facility exam writers are responsible for ensuring that test items on FLEX tasks are appropriate to the applicable license level and meet criteria for importance ratings (e.g., at least 3.0 for the walkthrough examination).

4.2.17 (201.8)

Does the facility licensee need to track the question history if the facility licensee writes the examination?

NUREG-1021 eliminated the limits on written question repetition from quizzes given during the training program, thereby eliminating the need to track question histories. However, as stated in Section B.6.c of ES-4.2, facility licensees are required to identify those questions that were used on an NRC license examination at the facility.

4.2.18 Some generic K/A topics do not lend themselves well to Tier 3 plant-wide generic questions. Specifically, some generic K/A topics appear to be covering system-specific or emergency-abnormal plant evolution specific knowledge. Can you please provide examples of Tier 3 questions for these generic K/As?

Please refer to question 4.1.19 for a list of generic K/As that may be pre-screened or deselected from the Tier 3 portion of written examination outlines. The staff has reviewed the remaining generic K/As and determined that it is possible to write Tier 3 questions for those topics. The following list provides ideas or starting points for possible questions for these Tier 3 topics. The list is not intended to limit other appropriate Tier 3 questions for these topics. The staff recognizes that some of these Tier 3 topics will not apply to some sites. Those topics may be preselected or deselected for those sites.

Table 1. Tier 3 K/As Not Applicable to Some Sites 2.1.7 Ability to evaluate plant performance and make operational judgments based on operating characteristics, reactor behavior, and instrument interpretation.

Note: This could test various conduct of operations topics, such as appropriate actions to take if the core thermal power average exceeds limits.

Question idea: According to OPDP-1, Conduct of Operations, the crew should take additional action to lower thermal power only if the core thermal power (10 minute / 1 58

hour) average is found to exceed the licensed thermal power limit. This transient (is/is not) a violation of the license thermal power limit.

2.1.19 Ability to use available indications to evaluate system or component status Note: This could test some plant-specific way to monitor parameters that applies to all systems.

Question idea: An instrument qualified to operate in an accident environment is designated by a (border, color, shape, etc.).

2.1.20 Ability to interpret and execute procedure steps.

Note: This could test rules of usage (see example below) or authority to change an in-progress procedure (see example in 2.1.23).

Question idea: In emergency operating procedures, a continuous action step applies (all times/only when) and a fold-out page item applies (only in this procedure/in this and subsequent procedures).

2.1.23 Ability to perform general or normal operating procedures during any plant condition.

Note: This could test procedure use rules.

Question idea: According to Procedure Use and Adherence, which of these typographical errors must be corrected prior to performance? (List of typographic errors to choose from) 2.1.30 Ability to locate and operate components, including local controls Note: This could test a plant-specific way to determine the location of a component.

Typically, a licensee has a computer program or other document that will identify what room a component is in.

Question idea: One way to determine the location of a valve is: (list of computer programs) 2.1.31 Ability to locate control room switches, controls, and indications, and to determine that they correctly reflect the desired plant lineup Note: This could test the meaning of lights used to reflect plant status.

Question idea: A lit amber light associated with the control switch for a pump indicates that the pump (list of pump and breaker states) 59

2.1.36 Knowledge of procedures and limitations involved in core alterations Note: This could test various administrative controls for fuel handling, such as authority for actions, which actions require a license, or notification requirements.

Question idea: The Fuel Handling Supervisor shall notify (various positions) prior to overriding or defeating any fuel handling system safety interlock.

2.2.1 Ability to perform pre-startup procedures for the facility, including operating those controls associated with plant equipment that could affect reactivity Note: It is possible to write a question about reactivity control SRO responsibilities, or a review associated with a mode change checklist.

Question idea: The Reactivity Control SRO is ONLY required in Modes: (list of modes) 2.2.3 (Multi-unit license) Knowledge of the design, procedural, and/or operational differences between units Note: It is possible to test administrative differences between the units, such as labelling, or operational responsibility differences such as switchyard control.

Question idea: For component identification, which color and unit combination is correct? (list of colors and units/common equipment) 2.2.4 (Multi-unit license) Ability to explain the variations in control room layouts, systems, instrumentation, and/or procedural actions between units at a facility Note: It is possible to test administrative differences between the units for this KA, such as physical control differences, or operational responsibility differences such as switchyard control.

Question idea: The Unit (1/2) control room contains the switchyard controls, and the Unit (1/2) control room has responsibility for controlling personnel access to the switchyard?

2.2.36 Ability to analyze the effect of maintenance activities, such as degraded power sources, on the status of limiting conditions for operation Note: This could test TS rules of usage.

Question idea: Technical Specifications require that an evaluation be performed in accordance with the Safety Function Determination Program (SFDP) when (list of TS Section 3 LCOs) is applied.

60

2.2.44 Ability to interpret control room indications to verify the status and operation of a system and understand how operator actions and directives affect plant and system conditions Note: See Number 2.1.31.

2.4.45 Ability to prioritize and interpret the significance of each annunciator or alarm Note: This could test alarm color coding or some plant-specific way to identify alarm priority.

Question idea: Per the Conduct of Operations Procedure, which of the following alarms is the highest priority? (list of annunciator types, such as borders or ESFAS group) 2.4.46 Ability to verify that the alarms are consistent with the plant conditions Note: This question could test the timing of responding to annunciators.

Question idea: Following a plant transient and EOP entry, the SRO has suspended the announcement of transient related annunciators per Procedure X, STRATEGIES FOR SUCCESSFUL TRANSIENT MITIGATION. Per Procedure X, when will the SRO re-instate the annunciator response requirements of Procedure Y, WATCH-STANDING PRACTICES, including verifying ALL annunciators are consistent with plant conditions and the appropriate plant response has occurred? (list of plant states and procedure transitions) 2.4.49 Ability to perform without reference to procedures those actions that require immediate operation of system components and controls Note: A question could be written to test whether the applicant knows the proper way to perform immediate actions (e.g., follow up by reading the step, get SRO approval prior to taking action, pause before taking announced action to give crew members a chance to intervene, etc.).

Question idea: Unit 1 is operating at 100% power when 1RC-431, Pressurizer PORV, opens. What direction do you provide to the CO? (list of immediate actions and procedure entries with choices for the timing of the action) 61

ES-4.3 Administering Written Examinations 4.3.1 (402.2)

Must the facility proctor read the entire ES-1.2 verbatim or just the first part regarding cheating?

Only those items specifically identified in ES-1.2 (i.e., Items A.1 and B.1) need to be read verbatim by the proctor; the others may be paraphrased. Per Section B.2.e of ES-4.3, every applicant shall also be given a copy of ES-1.2 to review before starting the examination.

4.3.2 (402.3)

What is the guidance on providing additional information or clarifying statements to the candidates during the written exam? Specifically, for facility written exams.

The requested guidance is located in Section B.3 of ES-4.3 (in NUREG-1021); it is the same regardless who prepared the examination. Anyone providing additional information during the examination must be extremely careful not to lead the applicants or give away answers when clarifying questions. If the proctor has any doubt about how to respond to an applicant's question, it is best to withhold additional guidance and instruct the applicant to do their best with the information that is provided. Per Section A.2.c of ES-4.3, an NRC examiner will always be available to respond to questions while the examinations are in progress.

62

ES-4.4 Grading and Documenting Written Examinations 4.4.1 (501.3)

If the chief examiner conducts a re-grade (78-82%), what is the focus of the re-grade?

(Re-grade per the key?) (Validity of the questions?)

Multiple grading changes and reviews often result in answer sheets that are difficult to read and could result in licensing errors. Therefore, Section C.4 of ES-4.4 requires the chief examiners to re-grade borderline exams using the clean answer sheets if necessary. The re-grade would be done after all the facility's comments have been resolved and the answer key has been finalized. It would normally not involve a revalidation of the exam questions.

63

ES-5.1 Issuing Operator Licenses and Post-Examination Activities 5.1.1 (501.5)

Is there a format for the utility to provide the NRC with feedback on how the exam went?

Sort of a reverse exam report? I would think the NRC would be open to feedback so you can also improve the exam process from your end. (I mean a formal feedback process -

not casual.)

NUREG-1021 requires the regional operator licensing branch chiefs to solicit feedback from the licensee before the examinations are given (Section C.12 of ES-2.1) and encourages the discussion of lessons learned after the examinations are complete (Section E of ES-5.1). As discussed in Section C.12 of ES-2.1, facility licensees are encouraged to call the NRC chief examiner, regional branch chief, or program office any time they have concerns regarding an examination.

5.1.2 Are the spare scenario events graded for submittal quality along with the rest of the licensee-developed operating test?

Yes. Revision 12 of NUREG-1021, ES-5.1, section G, Determine Quality of Submitted Examination, states in part [emphasis added]:

After examination administration and once all postexamination comments have been resolved, the NRC regional office will determine the quality of the submitted written examination and operating test material based on the following for documentation in the examination report (refer to the most current revision of OLMC 510, Operator Licensing Examination Reports (Revision 0, issued September 2021 ADAMS Accession No. ML21109A143).

Since issuance of Revision 12 of NUREG-1021, the NRC staff updated OLMC-510 (Issued February 2023 Accession No. ML23052A114) and OLMC-520, Operator Licensing Examination Records and Documentation, (Issued February 2023 Accession No. ML23052A118) to clarify that the spare scenario is used in the submittal quality determination.

64

ES-5.2 Application Denials 5.2.1 (502.1)

How will the facility representatives get a copy of the NRC appeal correspondence?

It is normal practice for the NRC to send a copy of its appeal correspondence to the individual who signed the applicant's license application (NRC Form 398). However, applicants who file an appeal are not required to send a copy of their request to the facility licensee.

5.2.2 (502.2)

Who is responsible for defending a question during the appeal process?

Once the NRC approves an examination it essentially takes ownership of the document.

Therefore, if a question is challenged during an appeal, the NRC will evaluate the question.

However, as stated in Section C of ES-5.2 (in NUREG-1021), facility licensees are expected to provide reference material and technical support (and possibly confirmation of the test items validity if the facility wrote the examination) as necessary for the NRC to evaluate and resolve any concerns raised by a license applicant.

5.2.3 (502.3)

What would the NRC do if a question from the national exam bank was found unacceptable after it was used? How far back would the NRC search for previous use of the question, which could affect already issued licenses?

Any question (not just those from the national bank) determined to be invalid during the grading process (i.e., after the exam was given but before the licenses are issued) would be deleted from the exam and the applicants grades would be adjusted accordingly. However, this would not affect applicants who had already been granted a license.

5.2.4 (502.4)

If an applicant's license examination failure is overturned due to appeal and the question that was reviewed affects the licenses of other applicants, will licenses be granted to all applicants that would have received a passing grade due to the review, even if those applicants chose not to appeal?

Yes. The NRC regional office will determine if any of the test item changes (i.e., question deletions or answer key changes) made as a result of the NRR operator licensing program office review for the appealing applicant(s) alter the outcome for any applicant who failed the examination but chose not to request an administrative review or hearing. If the test item changes cause any of the non-appealing applicant(s) to achieve a passing score, the regional office will issue licenses, as appropriate.

65

ES-5.3 Maintaining, Changing, and Renewing Operator Licenses 5.3.1 (605.1)

NUREG-1021 allows postponement of requalification requirements for up to 2 years for off-site development assignments, such as INPO. We also have on-site development assignments, such as Work Control or Site Engineering, which are intensive from a workload standpoint. Why can't the requirements of requalification be suspended for an on-site/off-shift developmental assignment?

The Operator Licensing Program Office has a number of concerns regarding such a policy change (e.g., the quality of the make-up training and testing, limits on the number and duration of the assignments, public perception, NRC involvement and resource implications). The issue has been discussed during public meetings with the Nuclear Energy Institute's operator licensing focus group members, and everyone appeared to understand the basis for limiting the requalification suspension option to off-site assignments. Operators who wish to pursue on-site developmental opportunities can terminate their licenses, pursue other activities for up to two years without having to worry about attending requalification training, and then reapply for a license. In accordance with 10 CFR 55.47, the NRC can waive the requirement for an examination if the specified conditions are met. Refer to Section E.2.d of ES-2.2 for more information regarding such waivers.

5.3.2 (605.4)

Can someone stand 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> of a normal 12-hour watch?

As discussed in Section A.2 of ES-5.3, the 10 CFR 55.53(e) requirement for licensed operators to maintain their proficiency may be satisfied with a combination of complete 8- and 12-hour shifts (in a position required by the plant's technical specifications) at sites having a mixed shift schedule. Watches shall not be truncated when the minimum quarterly requirement (56 hours6.481481e-4 days <br />0.0156 hours <br />9.259259e-5 weeks <br />2.1308e-5 months <br />) is satisfied. Overtime may be credited if the overtime work is in a position required by the plant's technical specifications. Overtime as an extra "helper" after the official watch has been turned over to another watch-stander does not count toward proficiency time.

5.3.3 (605.5)

Are there any unwritten restrictions for "no solo" license conditions?

No. The nature of the restriction, which is determined case-by-case based on the individual's medical status and the recommendation of the facility licensees physician, is clearly stated on the license. Section A.3.d of ES-5.3 (in NUREG-1021) describes some typical medical restrictions.

66

5.3.4 (605.6)

The regulations (specifically 10 CFR 55.55(b)) require license renewal applications to be filed at least 30 days before the expiration date of the existing license to ensure that the license does not expire while the Commission reviews the application? However, the regulation does not specify a "no earlier than" date for filing renewal applications. How early is too early?

In order for the NRC to have current information on which to base a renewal decision pursuant to 10 CFR 55.57(b), it is recommended that renewal applications be filed no more than 60 days before the existing license expires. If a facility licensee submits its operator license renewal applications more than 60 days in advance, the NRC regional office may contact the facility to determine whether it would prefer to have the licenses renewed immediately with a new effective date (the licenses will not be predated, nor will they exceed a six-year term) or to resubmit the applications within the 60-30 day window preceding the expiration date.

5.3.5 (605.7) 10 CFR 55.53(f)(2) requires that part of the 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> include a plant tour. Can the plant tour be performed alone or does it have to be with an active license holder?

The NRC staff's position, based on the wording of the regulation, is that the plant tour, being part of the 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> to be completed under the direction of an operator or senior operator (as appropriate), must be done in the company of an active watch stander. That way the active watch stander can ensure that the reactivating watch stander is made aware of on-going activities and abnormal situations in the plant.

5.3.6 (605.9)

Is it acceptable for an operator with a no-solo license to stand watch with another no-solo operator, i.e., can two no-solo operators back each other up, or does the backup have to have an unrestricted license? If a no-solo operators backup has to leave the site unexpectedly to take care of a personal emergency, would the remaining operator be considered in noncompliance with their license condition or would this be covered by the temporary staffing deviation provision in the facilitys technical specifications (TS)?

The possibility that both no-solo operators standing watch together would become incapacitated at the same time is pretty remote. Therefore, yes, it would be acceptable for two no-solo operators to back each other up. If a backup operator has to leave unexpectedly (or is incapacitated while on watch), and prompt action is taken (per the facilitys TS) to restore compliance with control room staffing requirements, the remaining no-solo operator would not be subject to individual enforcement action.

67

5.3.7 (605.10)

In NUREG-1021, ES-5.3 A.3.d, there is a description of the "no-solo" restriction for SRO licenses. What are the actual restraints on the SRO that could have to leave the control room to perform an alternate shutdown due to fire in the control room and as part of those duties will have to manipulate controls locally in the plant? Also, if the STA on shift is licensed, could they be the extra licensed operator with the "no-solo" SRO when they perform the manipulations locally in the plant?

When the operator licensing program office last revised the wording of the no-solo license restriction, we tried to minimize the impact that it would have on the facilities and the individuals involved by distinguishing between those activities that require a license per 10 CFR 50.54 (i.e.,

manipulation of the controls that directly affect reactivity or power) and those that, with the knowledge and consent of a licensed operator or senior operator, can be performed by non-licensed personnel (i.e., apparatus and mechanisms other than controls that may affect reactivity or power). We also gave due consideration to the fact that most control manipulations are planned in advance and conducted in accordance with facility peer-check requirements and guidelines.

In the event that a control room evacuation becomes necessary, we expect that the operators would, under most circumstances, have sufficient time to trip/scram the reactor before relocating to the alternate/remote shutdown panel(s). This would generally mitigate the need to perform additional control manipulations locally in the plant and any concern regarding solo operations by a restricted operator. However, facility licensees should take reasonable measures to ensure that additional operators are available on-site to respond to an emergency scenario when no-solo operators are assigned the watch. For example, when a no-solo SRO is on watch, the other SRO on-site could be instructed to respond to the alternate shutdown panel if that is the location the no-solo SRO would respond to during such an emergency. Similarly, a no-solo RO could have another licensed individual on-site respond to the control room or "catch up" to the RO during their performance of the in-plant portion of the alternate shutdown procedure.

As written, the current no-solo restriction requires another licensed operator to be in view only when the restricted operator actually manipulates a control (i.e., that small subset of apparatus and mechanisms that directly affect reactivity or power) while in the control room or out in the plant; at all other times while performing SRO licensed duties, another SRO would have to be present on site. An STA would be acceptable to fill the role of the second operator if they have an active SRO license and is up-to-date in the licensed operator requalification program. If, despite the compensatory measures discussed above, a second licensed operator is not immediately available to oversee an emergency control manipulation in the control room or in the plant, we would expect the restricted operator to perform the necessary control manipulations to protect the plant in a timely manner even if it results in a failure to comply with their license condition.

68

5.3.8 (605.11)

What are the requirements with respect to a retired licensed operator (RO, SRO, or LSRO) returning to work as a licensed operator at the same facility after retirement?

• In accordance with 10 CFR 55.55, Expiration, an operators license expires upon termination of employment with the facility licensee. Therefore, if the facility wishes to retain the individuals license, it will have to execute a re-employment agreement with the individual before the retirement takes effect. If the operator actually terminates employment with the facility licensee without executing a re-employment agreement, the license would be considered expired and the individual would have to reapply for a new license in accordance with 10 CFR 55.31. However, the applicant may request and be able to justify a waiver of the examination and test requirements pursuant to 10 CFR 55.47 and Section E.2.d of ES-2.2 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors. In either case, the individual would be required to make up any requalification training and testing that might have been missed during any break in service and to reactivate the license, as necessary, per 10 CFR 55.53(f) before resuming licensed duties.

• The terms licensee, operator, and senior operator are defined in 10 CFR 55.4 and all refer to any individual licensed under 10 CFR Part 55 to manipulate the controls of the facility and, in the case of a senior operator, to additionally direct the licensed activities of licensed operators. Moreover, 10 CFR 55.31(a)(3), which addresses how to apply for a license, states that a license applicant shall submit a written request from an authorized representative of the facility licensee by which the applicant will be employed Although Part 55 does not define the term employed, a retired and subsequently rehired RO, SRO, or LSRO would be considered an employee of the facility regardless how the facility classifies the individual.

• Pursuant to 10 CFR 55.2(a), the regulations in Part 55 apply to any individual who manipulates the controls of any utilization facility licensed pursuant to 10 CFR Part 50, without regard to employment status. Consequently, a rehired operator would be subject to all the same license conditions specified in 10 CFR 55.53 (e.g., observe all applicable rules, regulations, and orders of the Commission; maintain or re-establish proficiency; complete the requalification program; have a biennial medical examination; and comply with fitness for duty requirements) as a regular employee/operator. All other regulatory requirements and potential enforcement sanctions in Part 55 would also apply regardless of the individuals employment classification.

• Given that the responsibilities of senior operators, as defined in 10 CFR 55.4, include directing the licensed activities of licensed operators, the extent and nature of such direction may result in creating, under applicable state law, an employment relationship with those licensed operators (ROs). Therefore, facility licensees are advised to consult with their attorneys to ensure compliance with employment law requirements in the state in which their facility is located.

69

5.3.9 (605.12)

Can a shift (watch) that begins for an operator or senior operator at 1800 of the final day of the calendar quarter be counted for meeting the minimum shifts required by 10 CFR 55.53(e) for maintaining active license status even though the shift is completed at 0600 on the first day of the succeeding quarter?

10 CFR 55.53 (e) requires to maintain active status that every licensee actively perform the functions of an operator or senior operator on a minimum [emphasis added] of seven 8-hour or five 12-hour shifts per calendar quarter. There is no allowance provided in 10 CFR 55 or NUREG-1021 that permits counting a shift that will not be completed in the current calendar quarter for meeting 10 CFR 55.53(e) as long as it is started in the current calendar quarter. Thus, given the example provided in the question, the operators license would not be considered active as of 2400 on the last day of the calendar quarter since the operator/senior operator had not performed the functions of an operator or senior operator on a minimum

[emphasis added] five 12-hour shifts in the current calendar quarter.

5.3.10 (605.13)

The no-solo restriction described in ES-5.3, A.3.d is not explicit for an SRO supervising core alterations. Is there a difference in requirements for no-solo SROs and LSROs when supervising core alterations?

10 CFR 50.54m (2)(iv) requires an SRO or LSRO to be present to directly supervise any alterations of the core (including fuel loading or transfer) and to be assigned no other duties while supervising the alterations. ES-5.3, A.3.d describes an LSRO with a no-solo license restriction as requiring another individual capable of summoning assistance in view while the restricted LSRO is performing licensed duties (e.g., directly supervising core alterations). A no-solo SRO is required to have a licensed operator in view when the restricted SRO is performing control manipulations, however there is no explicit guidance on what type of restriction is required for a no-solo SRO supervising core alterations.

The requirement for a no-solo SRO performing control manipulations is intended to be more restrictive than for a no-solo LSRO or SRO who is directly supervising core alterations. It is expected that if an LSRO or SRO becomes incapacitated while supervising core alterations, the individuals performing the evolution will be able to safely stop the fuel movement and call for assistance. A requirement to have an additional licensed operator in view of a no-solo SRO when supervising core alterations would likely negate the reason to have the restricted SRO present in the first place. While supervising core alterations, a no-solo SRO is expected to adhere to the same restriction imposed on a no-solo LSRO and thereby ensure that another individual is in view who is capable of summoning assistance if needed.

It is important to realize that this clarification is intended to provide guidance that is consistent with current industry practices. No additional restrictions or changes to the methods currently being implemented are being established. However, for the purposes of clarification, the typical wording on all future SRO and LSRO licenses will be changed to read as the following:

70

SRO No-Solo:

Another licensed operator must be in view when you are performing control manipulations, and another senior operator must be present on-site at all other times while you are performing SRO licensed duties or someone capable of summoning assistance must be present in the control room at all other times while you are performing RO licensed duties. Another individual capable of summoning assistance must be in view when you are directly supervising core alterations.

LSRO No-Solo:

Another individual capable of summoning assistance must be in view when you are directly supervising core alterations.

No revisions or corrections will be made to current SRO and LSRO licenses. The new wording will be applied to new licenses as they are issued and to current licenses as they are renewed during the normal 6-year cycle. Currently licensed SROs with no-solo restrictions should understand that when directly supervising core alterations, their licenses require them to have an individual capable of summoning assistance in view at all times.

71

ES-6.1 Conducting NRC Requalification Examinations 6.1.1 (601.1) 2.5 versus 3.0. What is the minimum task importance threshold for initial exams versus requalification? There should be higher standard for requalification than initial.

As noted in Form 6.2-1 (in NUREG-1021), all test items used on an NRC requalification examination should normally have a K/A importance rating of 3 or greater. The minimum K/A importance rating for initial exams is 2.5. In either case, test items with lower NRC K/A values may be used with appropriate justification.

The NRC expects facility licensees to comply with their own requalification program requirements regarding test item importance.

Initial license applicants are held to a higher standard (i.e., more K/As eligible for testing) because the NRC has no prior basis for judging their competence. Once an operator has a license, their competence is continually evaluated on the job and in requalification training, thereby justifying a lower threshold for the NRC requalification examination.

6.1.2 (601.2)

Is there a policy for use of computers and maintaining exam security?

Does there need to be a specific procedure for requalification examination security?

The requirements of 10 CFR 55.49 apply to all examinations required by the regulation, including requalification exams, while the requirement to establish, implement, and maintain examination integrity and security procedures in accordance with 10 CFR 55.40(b)(2) only applies to power reactor licensees that elect to prepare their own initial operator licensing examinations. However, it would be appropriate for those licensees that do establish procedures to address all exams required by Part 55. Refer to the section on ES-1.3 for related security questions.

6.1.3 (604.1)

For requalification examinations, do you test how you normally staff?

Yes. As stated in Section C.2.a of ES-6.1 (in NUREG-1021), the NRC expects facility licensees to train and examine their operators in the same crew configurations with which they normally operate the plant.

72

ES-6.3 NRC requalification walk-through tests 6.3.1 (603.1)

Section A of ES-3.1 states that initial license exams should sample the items listed in 10 CFR 55.43 but need not cover all 13 items. Is this also true of a requalification annual operating examination?

Is there an expectation that every SRO do an Emergency Plan classification in either a scenario or a JPM?

Yes. As specified in 10 CFR 55.59(a)(2)(ii), the operating test shall cover a comprehensive (i.e.,

thorough or broad, but not necessarily complete) sample of the items specified in 10 CFR 55.45(a)(2) through (13) as applicable to the facility. Also refer to Question IP.13.

No. Every operating test is a sample and does not have to, and should not always, include an Emergency Plan classification.

6.3.2 (603.2)

Is changing a JPM to an alternate path JPM considered a different test item?

Yes. This is consistent with the initial examination policy regarding the repetition of test items from the individual's audit examination (refer to Section B.4of ES-3.1 of NUREG-1021).

6.3.3 (603.3)

Are simultaneous JPMs allowed?

The NRC would allow the simultaneous administration of JPMs in the simulator or control room during NRC-conducted tests provided there is no interference between the operating stations.

When licensees are conducting the tests, they should follow their approved requalification program.

6.3.4 (603.4)

To what extent is it acceptable to just mark up a procedure versus following the ES format for JPMs?

In accordance with Section B.1.d of ES-6.3, Form 3.2-3, "Job Performance Measure Template" or an equivalent facility form should be used to construct and format the JPMs. However, as long as the JPMs include the elements identified in and using Form 3.2-4 Job Performance Measure Development Job Aid" (e.g., initiating and terminating cues, critical steps, and performance criteria), it should be possible to adapt facility procedures for use as JPMs by identifying critical steps and entering comments on how to execute particular steps. Section D of ES-3.2 authorizes that practice for initial operating tests.

73

6.3.5 (603.5)

Is the initial licensing walk-through alternate path JPM requirement a required item for annual requalification exams?

No. However, per ES-6.1 of NUREG-1021 (Section III.C of Form ES-6.1-4), facility licensees are expected to include some alternate path JPMs in their test item banks for use during NRC-conducted requalification examinations.

6.3.6 (603.6)

ES-6.3 guidance for generating an annual operating evaluation states the sample plan is to be based on the "current" cycle. Suppose a facility licensee is in the first six months of the "current" cycle and we want to generate an annual operating exam. Since there is insufficient material for an exam would it be acceptable to generate the exam based on a sample plan developed covering the "current" cycle and include that part of the previous cycle up to the last exam (i.e., the last six months of the previous cycle)?

Keep in mind that the ES-6 series in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, provide guidance for the preparation and administration of licensed operator requalification examinations in which the NRC is an active participant. When facility licensees prepare and administer their own requalification examinations, the NRC does not expect or require them to comply with the guidance in the ES-6 series unless the facility licensee has formally incorporated that guidance as part of its accredited (by the National Academy for Nuclear Training) training program.

Although requalification programs that are based on a systematic approach to training (SAT) should evaluate the trainees mastery of the objectives during training, Attachment 3 of ES-601 encourages reserving a portion of the examination to test high importance topics that were not necessarily covered during the requalification cycle. This is consistent with 10 CFR 55.59(c)(4)(i) which (in lieu of a SAT-based program) requires the comprehensive written exams and annual operating tests to determine areas in which retraining is needed. Moreover, 10 CFR 55.59(a)(2)(ii) requires the operating test to evaluate the operators understanding of and ability to perform the actions necessary to accomplish a comprehensive sample of the items specified in 55.45(a)(2) through (13) inclusive to the extent applicable to the facility.

Notwithstanding the definition of annual in ES-8 of NUREG-1021, we encourage facility licensees to conduct their annual operating tests at approximate 12-month intervals (i.e., at the midpoint and end of their 24 month requalification training cycles). Facility licensees need to exercise caution when they reschedule examinations around the plants operating schedule to ensure they comply with the regulation by doing an operating test every calendar year.

The NRC expects facility licensees to comply with the requirements in 10 CFR 55.59 and their accredited training programs. The regulations do not appear to prohibit the use of test items covering topics outside the scope of the current requalification training cycle. Therefore, whether test items covering topics outside the scope of the current requalification training cycle are acceptable is determined by the requirements of the accredited facility licensees program.

74

ES-6.4 Dynamic Simulator Requalification Examinations 6.4.1 (604.2)

Can an individual who fails in the simulator for a specific task be retested with a JPM, or must it be a scenario?

If an operator fails an annual operating exam scenario due to an independently performed competency, can a JPM be used as a retake exam?

If an operator fails any portion of an NRC-conducted operating test (initial or requalification), the retest will be in the same format as the part that was failed. If an operator fails a facility-conducted requalification examination, the facility licensee would be expected to administer the retest in accordance with its approved requalification program.

6.4.2 (604.3)

Can an individual failure on the simulator operating test be retested with surrogates, or must it be with a shift?

Surrogates would be acceptable for an NRC-conducted test, but the facility licensee would have to follow its program requirements if it conducts the test.

75

ES-8 Glossary 8.1 What is meant by low power / shutdown conditions for scenarios and for JPMs? Are low power JPMs allowed to test post-trip actions, or should they solely be for other shutdown/low power conditions (such as a planned outage for refueling)?

For scenarios and JPMs, low power / shutdown conditions at commercial nuclear power plants were evaluated by the staff in NUREG-1449 and included operations with the reactor in a subcritical (shutdown) state and in transition between subcriticality and 5% power (low power).

NUREG-1021, Revision 7, issued January 1993, added requirements that at least one JPM be a low power JPM and that scenario initial conditions should be varied and include low power conditions. This revision was made to place more emphasis on those operating conditions evaluated in NUREG-1449 and therefore the same definition was used. This definition has been incorporated into ES-8 of NUREG-1021, Revision 12, and applies to both initial and NRC-developed requalification examinations.

The NRC intends for operating tests to sample the full range of operating conditions and power levels so that the tests do not become predictable. There are no explicit requirements on how the low power or shutdown condition is achieved. Therefore, if a JPM has initial conditions that include the fact that the plant has recently tripped, or that post-trip actions are being performed, that would meet the requirements. But, if the facility is consistently using a recent plant trip as the initial condition for a low power JPM over multiple exams, this could impact exam predictability. Although a single exam may use post-trip actions for the low power JPM, the facility should not use this for every exam and should instead vary the initial conditions so that other shutdown and low power conditions are tested across multiple exams.

76

IP-71111.11 Requalification inspections IP.1 10 CFR 55.59 - the use of SAT-based program vice regulatory based programs. Why do you have to track individual control manipulations if you have a SAT-based program?

10 CFR 55.59(c) allows licensees to substitute the appropriate SAT-based program elements (as defined in 10 CFR 55.4) for the requirements in sections (c)(2), (3), and (4) (i.e., lectures, on-the-job training, and evaluation). Record-keeping is not a SAT-based program element, and the NRC needs to know that each individual actually performed the requisite control manipulations.

While a SAT-based process can replace the requirements of 10 CFR 55.59(c)(3), it is still the NRC's expectation and requirement per 10 CFR 55.59(c)(5) that individual participation in the requalification program be recorded. How each utility chooses to do this should be clearly defined in its accredited SAT-based program.

Pursuant to 10 CFR 55.57(a)(4), an authorized representative of the facility licensee must provide a statement that each operator license renewal applicant at the facility has satisfactorily completed the requalification program. Making such a statement would be difficult if the facility licensee does not individually track and document each operator's participation in the program (e.g., classroom lecture attendance, completion of on-the-job training including control manipulations, and performance on examinations).

IP.2 "Control Manipulations" in requalification - a prior guidance from previous NRC meeting clearly indicated that counting control manipulations from the Denton letter was a thing of the past - SAT based requalification training would naturally contain a large portion of the annual/biennial tasks and evolutions, therefore, program participants would be involved during simulator training/evaluation, and/or annual Op. Eval. JPMs; "individuals simulator critical tasks" went away and "crew critical tasks" were required. Teamwork/

communications, command & control/by the team was the most important. Bottom line -

the implied expectation expressed on 8/12/99 is not congruent with guidance provided in 1989. It appears that we are returning to the middle to early 80's again.

Reactivity Manipulations for licensed operator continuing training? LOCT: The Institute of Nuclear Power Operation's (INPO's) policy for tracking manipulations seems to be in conflict with NRC requirement (INPO doesn't require tracking on an individual basis).

The control manipulations conducted per 10 CFR 55.59(c)(3) or your SAT-based requalification program are individual, on-the-job training requirements, which are not to be confused with individual or crew critical tasks on the annual simulator operating test.

77

Pursuant to 10 CFR 55.57(a)(4), an authorized representative of the facility licensee must provide a statement that each operator license renewal applicant at the facility has satisfactorily completed the requalification program. Making such a statement would be difficult if the facility licensee does not individually track and document each operator's participation in the program (e.g., classroom lecture attendance, completion of on-the-job training including control manipulations, and performance on examinations).

IP.3 Is it required that each SRO be evaluated during the Emergency Operating Procedures (EOPs)? Does their documentation for the evaluation need to be done in accordance with the requirements of conducting annual exams? If so, what is the basis for this requirement?

Although each SRO does not have to be evaluated during the EOPs on every annual operating test, every SRO should be at risk of being evaluated on all of the items in 10 CFR 55.45(a) during any test. The NRC does not differentiate between different levels of SROs, so the test-item sampling should be the same regardless whether or not the operator normally stands watch in an EOP-reader position. SROs would be considered "at risk" if the facility licensee holds them responsible for the actions of the EOP readers. However, they do not necessarily have to approve each and every action required by the EOPs.

Note that ES-6.4 does not require crew position rotation and states that an individual would pass the dynamic simulator test if the operating crew performs satisfactorily. The NUREG-1021 requalification examination crew-based grading methodology presumes that all individual crew members, including senior crew managers, are held accountable for all of the crews actions, and therefore are evaluated. Crew position rotation, if not required by the facility licensees requalification program, would only be considered if it was determined to be the only way to evaluate the scope and depth of a demonstrated individual performance deficiency. The facility licensees dynamic simulator requalification examination process is not required to be the same as that discussed in ES-6.4. However, if the facility licensee evaluates individual and crew performance consistent with the guidance of ES-6.4, then the test requirements of 10 CFR 55.59(a) would be met.

IP.4 Are requalification inspections conducted using NUREG-1021 as the standard (i.e., ES-6 series) for the inspection? Are facilities subject to violations because an aspect of NUREG-1021 is not utilized during a requalification exam or is it just the inspection plan (i.e., IP 71111-11 vs. ES-6)?

Requalification inspections are conducted using IP-71111.11. Facility licensees are not required to use the ES-6 series of NUREG-1021 to conduct their requalification examinations.

However, if a licensee's requalification program endorses or incorporates the NUREG-1021 examination process, the NRC will expect the facility to comply with its established program.

78

IP.5 Can I take credit for questions other than multiple choice questions in the licensed operator requalification (LOR) exam bank, including maintenance of the bank?

Yes. However, licensees are encouraged not to abandon their multiple-choice question banks in case the NRC determines that a for-cause requalification examination is necessary. Facility licensees are expected to follow their own program guidelines for bank maintenance; the guidelines in ES-6.1 would only apply if the licensee has endorsed NUREG-1021 as part of its LOR program.

IP.6 How is the cognitive level determined if essay and short answer are used? (applies to operator requalification exams)

As discussed in Section C of ES-4.2 of NUREG-1021, the NRC uses Bloom's Taxonomy to classify the cognitive level of test questions. That classification approach would apply regardless of the question format. Facility licensees are not obligated to use the same approach.

IP.7 What are the criteria (guidance) for test item reuse throughout a biennial requalification cycle? That is, 1) items used on more than 1 weekly quiz; 2) item used on weekly quizzes to be used on biennial exam. Need a number (upper limit) on requalification test question reuse. Subjective limits lead to variability in standards and enforcement.

Suggest 20-25% limit.

What is the expectation or threshold on reuse of exam materials? During the Region I Conference the NRC stated that internal policy is <50% duplication of items between exams. We all agree we want to protect the validity of the exams. However, without clear expectations from the NRC, and subjective application by an evaluator, it will be difficult to predict acceptability.

Does ES-6.1 D.3 allow for subjective interpretation from examination to examination based on what the specific examiner "feels" is appropriate; can we not identify this internally and have the examiner base their decision on plant specific requirements?

Biennial requalification exam -- What is the standard for reusing exam questions from weekly exams from the last 2-year biennial training program?

The NRC does not have definitive criteria (i.e., regulations) regarding the number of test items that can be reused on weekly quizzes or biennial examinations. However, as stated in Section D.3) of ES-6.1, the amount of item duplication will be taken into consideration during the program evaluation because it could affect the discrimination validity and integrity of the examinations. Whenever test items are repeated, they should be selected in a distributed manner and approximately equally over all previous examinations to reduce predictability (if a large number of items were taken from the most recent examination). As always, facility licensees are expected to comply with their approved training program requirements, which 79

would be expected to vary based on the licensee's specific circumstances. For example, the same level of question repetition would have less impact if the licensee does not distribute or post its examinations until after they are all complete. The NRC will evaluate every situation on its own merits; the same upper limit may not always be appropriate, nor would it be enforceable unless it was adopted as a regulatory requirement or licensee commitment.

NRC examiners and inspectors that document test item repetition as a weakness must demonstrate that the integrity of the examination was compromised or the discrimination validity of the examination was affected by inappropriate reuse of test items. In December 2003, the NRC revised IP-71111.11, the requalification program inspection procedure, to trigger a performance-based review if and when a facilitys comprehensive requalification examination repeats more than 50 percent of its test items from previously administered comprehensive requalification examinations between and among crews undergoing the same requalification training program. The inspectors would apply the guidance in Appendix D of the IP to examine the crews average scores to determine whether they show any pattern of rise over successive crew examination administrations or any unexplained higher-than-expected crew mean scores.

Although the IP focuses specifically on the written examinations, the same 50 percent repetition philosophy would apply equally to the operating test.

IP.8 If a JPM exam is failed, can one of the failed JPM's be used in the retake examination?

It would certainly be appropriate to test the operator to determine if the remedial training was successful, and to include the failed material in that sample. However, the annual operating test given pursuant to 10 CFR 55.59 should consist of a new sample of test material to confirm the operator's overall competence.

In accordance with Appendix F of IP-71111.11, the requalification program inspection procedure, NRC inspectors will ensure that any test items that appeared on the original failed examination are not included as a part of the retake examination. Reusing the same items (missed or correct) from the original failed test on the retake examination is a flawed practice that would falsely bias the test results upward, inflating and distorting true retake performance.

Moreover, including any of the same items on the retake test amounts to little more than a review - not a test as it is operationally defined.

IP.9 During a recent inspection, the validation of a scenario did not match crew response.

The utility's examiner response was to remove the scenario from the exam. What and where are the standards for this?

If the NRC were administering the test, it would not replace the scenario because a crew did not perform as expected unless the scenario was found to contain a serious flaw. Rather, the examiners would document actions taken by each of the crews and later determine if they responded correctly under the given conditions. The examiners would also expect the facility licensee to determine whether the deviation could have resulted from a simulator fidelity problem.

80

In accordance with 10 CFR 55.4, a training program based on a systematic approach must be evaluated and revised based on the performance of the trained personnel in the job setting.

The fact that a crew deviates from a validated scenario suggests a problem in the training program that may not be fully understood if the scenario is replaced.

IP.10 Is it an issue if an instructor, who doesnt know in advance, sees a scenario, trains the next crew, and then administers the same scenario to that crew?

Yes. This clearly raises a question regarding the validity of the second crew's operating test.

The facility licensee should probably administer an additional scenario to remove any question regarding the operators' competence The facility licensee should also evaluate its testing program to determine if corrective measures are necessary to preclude similar situations from recurring. If the facility licensee's program includes exam security restrictions similar to those endorsed by the NRC in Section C.6 of ES-6.1 (in NUREG-1021), then the instructor should not have been involved in training activities after gaining knowledge of the exam contents.

IP.11 Can the annual operating exam (simulator & JPMs) be split between two consecutive cycles (i.e., successive retraining weeks which is approximately every 5 weeks for a crew)? The licensed operators received annual JPMs in Nov./Dec. 1999 then received the annual simulator exams in Jan./Feb. 2000. The two together comprise the annual operating exam.

The answer to Question #354 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators Licenses," states that the annual operating test needs to be done at one time and provides an unacceptable example in which the parts of the test are separated by six months. However, your proposal to administer the dynamic simulator and walk-through portions of the operating test during consecutive requalification training weeks (nominally 5 weeks apart) is acceptable (and we understand from our Regional Offices is already being done at some facilities) subject to the following conditions:

• The regulation (10 CFR 55.59(a)(2)) requires each operator to pass an annual operating test. Splitting the test such that the walk-through is given in one calendar year and the simulator test in the next (as in your example) may create a problem with regard to regulatory compliance.

• The operating test (scenarios and JPMs) must be comprehensive and conducted in accordance with the facility licensees approved, SAT (systems approach to training) based training program.

81

• Any significant remedial training that is determined to be necessary should be completed in a timely manner and not deferred until the entire operating test has been administered. If an operator fails either portion of the operating test, this would include removal from licensed duties pending satisfactory completion of the required remedial training and retesting.

IP.12 Archived IP.13 What are the requirements for sampling all items in 10 CFR 55.41 and §55.43 on the requalification exam?

As noted in response to a similar question related to the operating test (refer to Question 6.3.1 (603.1)), the sample should be thorough or broad, but not every item listed in the regulation has to be covered on every examination. Moreover, the response to Question IP.3 indicates that operators should be at risk of being evaluated on all of the applicable items during any examination. Since the requalification examinations are part of a systems approach to training (SAT), they should emphasize the topics covered during the training cycle; however, the NRC expects that they would also cover topics from outside the requalification cycle in order to determine areas in which retraining is needed (refer to 10 CFR 55.59(c)(4)(i)).

IP.14 What happens if an individual is unable to successfully complete the requalification exam prior to the end of the 2-year program cycle? They are already administratively restricted from standing watch.

As noted in response to Question #328 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses," it is only under extenuating circumstances (e.g., a special temporary assignment to a remote location, an extended illness, or enrollment in a degree program) that the NRC condones removing licensed operators from the requalification program. In such cases, the NRC generally invokes the provisions of 10 CFR 55.59(b), "Additional Training," to ensure that the affected operator is qualified prior to returning to licensed duties. Planned absences are processed as described in Section A.1.c of ES-5.3 of NUREG-1021. Unplanned incompletions and restorations should be documented and handled on a case-by-case basis in consultation with the NRC regional office.

82

IP.15 It is not uncommon to have on-shift crews staffed to beyond the minimum complement required by technical specifications. For this type of situation, is it acceptable to have a licensed operator participate in one scenario and still fulfill the requirement of completing an annual operating test (provided the facilities training program allowed this)? NUREG-1021, ES-6.4, is quite clear on crew dynamic simulator tests needing to be two scenarios but does not specify whether or not every crew member needs to be in an evaluated position for both scenarios.

As noted in the response to Question IP.4 above, facility licensees are not obligated to follow NUREG-1021 unless it is incorporated as part of their approved requalification program. The fact that you have more than the minimum required number of operators on shift, does not mean that you should leave some of them on the bench during a simulator scenario or a real event in the control room. The NRC would expect you to construct your operating tests with a sufficient number of events and scenarios to ensure that every operator on the crew gets a meaningful evaluation in accordance with the facility licensees approved requalification program.

83

Simulation Facilities Continued Assurance of Simulator Scope, Fidelity, and Testing Sim.1 Why is scenario-based-testing the simulators performance a challenge?

NRC operating test scenarios and scenarios used for performing control manipulations that affect reactivity to establish eligibility for an operators license may be used as simulator performance tests. Hence the term scenario-based test (SBT). The overriding challenge to licensees is to conduct the SBT in a manner sufficient to ensure that simulator fidelity has been demonstrated (and met) so that significant control manipulations are completed without exceptions, simulator performance exceptions, or deviation from the approved training scenario.

Simulation facility licensees should consult RG 1.149, Revision 4, Regulatory Position No.3 which describes the staffs acceptance and endorsement of the SBT implementation guidance described in NEI-09-09, Revision 1. NEI-09-09, Revision 1, is an acceptable method for demonstrating compliance with the requirements of Section 3.4.3.2 and 4.4.3.2 of ANSI/ANS-3.5-2009 regarding simulator SBT.

Sim.2 What impact do computer upgrades and re-hosting have on performance tests?

Upgrades to licensee simulation facility plant-referenced simulator computer systems and re-hosting onto new computer platforms should not alter model performance characteristics. It is expected that similar results will be achieved when comparing performance test runs after an upgrade or re-host to the same test runs before the upgrade or re-host. Verification and validation testing shall be conducted, as required by Section 4.4.1 and 4.4.2 of the standards (2009, and 1998), following a system upgrade or re-host to confirm that model characteristics have not changed. Although not a requirement of the ANSI/ANS-3.5 standard or the 10 CFR 55.46 regulations, it is prudent to run the simulator operability tests (i.e., steady-state, and transient tests) following a computer upgrade or re-host to ensure or demonstrate no unintended consequences to models.

84

Sim.3 Are simulator design specifications required to be updated?

Plant-referenced simulators model systems of a reference plant. Reference plant is defined in 10 CFR 55.4 as the specific nuclear power plant from which a simulation facilitys control room configuration, system control arrangement, and design data are derived.

ANSI/ANS-3.5-2009 (1998), Section 5.1.2 Simulator Design Data Base Update, requires that the simulator design data base (i.e., design specifications) shall be periodically updated (i.e.,

within 18 months of the reference units commercial operation date or the simulators operational date, whichever is later; or following the initial update, new data shall be reviewed, and revised, once per calendar year). Maintaining the fidelity of the plant-referenced simulator includes updating the design specifications. The particular methodology for updating design specifications is determined, for the most part, by the facility licensees simulator configuration management control (i.e., ANSI/ANS-3.5 standard requires, among other criteria, that a means for establishing and maintaining a simulator design baseline shall be included in the configuration management).

Sim.4 What is actually required when documenting scenario-based test (SBT)?

Please refer to RG 1.149, Revision 4.

Sim.5 What is the periodicity for SBT?

Simulator scenario-based tests (SBT) periodicity is not specifically addressed by the regulations or the industrys adopted standard ANSI/ANS-3.5-2009. SBT are uniquely developed as NRC operating tests (or in the case for which a scenario is developed for performing control manipulations that affect reactivity to establish eligibility for an operators license). That said, periodicity for a specific SBT per se is not appropriate since that SBT may or may not be used again in the future. However, should the specific SBT be used again (without alteration or modification), the expectation is that the specific SBT undergo performance testing again before it is used again to ensure that fidelity has not changed since the last time it was performed.

85

Sim.6 What is the staffs position with regard to installing modifications on the simulator before being installed on the referenced plant?

In general, the staff accepts and endorses industrys consensus standards (ANSI/ANS-3.5-2009

(-1998, -1993, and -1985)) through incorporation by reference in RG 1.149 (Revisions 4, 3, 2, and 1 respectively). Each revision of the ANSI/ANS-3.5 allows for simulator modifications to be completed either before or after the modifications in the reference plant. Decisions as to timing of the simulator modifications should be based on an analysis of training needs and must also take into consideration proposed uses of the simulator and the effect on operator actions.

When a plant-referenced simulator is used in NRC initial and or licensed operator requalification examinations (and, in some cases, for meeting eligibility requirements of 10 CFR 55.31),it must accurately reflect current design of the referenced plant and not produce negative training. In cases where a plant-referenced simulator differs from its reference plant as a result of plant modifications, the NRC expects differences training to compensate for deviations from the reference plant to preclude or compensate for any negative training. For example, if a reference plant modification is planned for completion in the last few weeks leading up to an initial license examination, it might be desirable to delay installation of the modification on the simulator until after the examination to avoid disrupting the orderly planning and administration of the exam.

However, this choice could call any licensing decision made using that simulator into question because the potential exists that skills demonstrated on the simulator would be different from what would be required in the plant for which a license is to be issued. In this case, a facility licensee could request in writing Commission approval to use the simulator while it differs from the reference plant. The request should address steps to be taken to prevent or compensate for negative training. The NRC has the option of granting such a request.

Several facility licenses have successfully implemented reference plant modifications (design changes), such as feed-water controls and digital EHC main turbine-generator controls, on the plant-referenced simulator without any regulatory approval or change in simulator status as a plant-reference simulator.

Sim.7 Will the staff determine whether or not a particular model is correct?

No, it is the responsibility of each facility licensee that maintains a simulation facility. Each facility licensee is expected to ensure their simulator adequately demonstrates expected plant response through appropriate testing. NRC staff evaluates and assesses whether or not the simulation facility complies with the scope and fidelity requirements describe in 10 CFR 55.46 during biennial baseline licensed operator requalification program inspections (IP-71111.11).

86

Sim.8 We have replaced some models with new models. What if the new model shows a different response than the old model? (With regard to malfunctions such as LOCAs and transients with no plant data).

Facility licensees that maintain a simulation facility must demonstrate continued assurance of simulator fidelity by conducting performance testing in a manner sufficient to ensure that simulator fidelity has been demonstrated and met (10 CFR 55.46). If the results of performance test are significantly different (e.g., does not meet the same acceptance criteria as before) a re-evaluation should be conducted to determine the extent of condition and whether or not a detailed engineering analysis is needed to resolve modeling discrepancies identified.

If the re-evaluation reveals significant modeling problems with the previous model and the model had been used to negatively train operators, then reactor safety may have been impacted. The facility licensees corrective action program would need to determine the extent to which the operators had been negatively trained. Retraining, if indicated, would follow.

Generally, NRCs licensed operator requalification program baseline inspections monitor performance in this area.

Sim.9 What constitutes an adequate degree of replication and, if not adequate, what is the safety significance?

The degree of replication depends on the type of evolution (steady state, transient/malfunction, normal evolution) and the applicable operability test acceptance criteria assuming adequate acceptance criteria have been established. For example, the ANSI/ANS-3.5 standard requires that certain steady state parameters meet a 2 percent tolerance. If there has been an identification of a fidelity issue in which the applicable parameter is beyond 2 percent, then the degree of replication is unacceptable since it would fail the steady state acceptance criterion.

For alarms and automatic action (or interlocks), the plants calibration and surveillance testing acceptance criterion (instrument tolerances) should be an adequate method for determining the degree of replication.

An ancillary question to the above is: What are the first order principles for NRC staff analysis in order to determine if a simulator fidelity performance deficiency is minor or not with respect to 10 CFR 55.46(c)(1) and what safety significance level could result? The issue is related to the human performance attribute in the three reactor safety cornerstones of initiating events, mitigation, and barrier controls per MC 0612, Appendix B (manual chapter links found here).

Performance deficiencies are more than minor and are of very low safety significance if they involve actual or potential impact on operator actions per MC 0609, Appendix I, [Blocks 13, 14, and 15, along with the basis statements for the questions in the blocks] (Note: This is a broader definition of negative training from that defined in ANSI/ANS 3.5 definitions section). These issues are not of greater significance because they did not have an adverse impact on operator actions such that safety related equipment was made or would have been made inoperable during normal operations or in response to a plant transient. If there was an effect to this degree, the performance deficiency would be analyzed per MC 0609, Appendix A (PRA basis).

87

Minor performance deficiencies that have no effect or impact on operator actions are generally not documented in the inspection report.

Sim.10 Record retention: ... retained for four years after the completion of each performance test or until superseded by updated test results. How long can the or in this statement be - the life of the plant, for example?

Four Year Record Retention: do records older than four years have to be retained, such as acceptance tests from original certification, etc.?

Per 10 CFR 55.46(d)(1), the performance test (as defined in 10 CFR 55.4) results are expected to be retained for four years after the completion of each performance test. Generally, simulator performance tests are conducted on a periodic basis in accordance with ANSI/ANS 3.5 and the facility licensees simulator testing schedule. The test results are subject to review by the NRC and a retention period of four years is prescribed so that an evaluation and comparison can be made for a given performance test over a period of time (up to four years) to ensure that simulator fidelity is being maintained. However, if a performance test is not repeated until a period of more than four years has passed, then the record of the performance test should be retained until superseded by the subsequent test. When a performance test is superseded before four years, then the four-year period resets for the updated test. The rule still requires that the facility licensee conduct performance testing throughout the life of the simulation facility.

Keep in mind that the standard requires that: (A) in Section 4.4.1, that verification tests (i.e.,

software design documentation) be generated and be updated. (B) in Section 4.4.2, that validation test documentation is generated and that a record of the conduct of this test, the tests results, and the tests evaluation be maintained. It further requires that these tests be conducted prior to the simulators use in training and examination for the following situations: (1) completion of simulator initial construction; (2) whenever models are changed or modified in a way that potentially affects fidelity relative to the reference unit; and (3) whenever there are changes which have the potential to affect simulator capabilities or repeatability. (C) in Section 4.4.3.1, that operability tests be conducted on a periodic basis and that a record of the conduct of this test and its evaluation be maintained. (D) in Section 4.4.3.2, that SBTs be tested before use for operator training or examination and that a record of the conduct of these tests, and the evaluation of the tests results be maintained. Implementing these standard requirements are measures acceptable to the staff for implementing the demonstration requirements of 10 CFR 55.46(c)(1).

Updating and maintaining tests documentation is ongoing. No relief is provided in the standard that allows cessation of maintaining the test records. Simulator test records provide evidence of simulator fidelity. If for no other reason, it would be prudent for licensees to retain all such records as a means of providing assurance of fidelity should it be brought into question by a future plant or industry event.

88

Sim.11 Core performance: what standards are being used to ensure the simulator performance replicates reference plant nuclear and thermal hydraulic operating characteristics, since there is a broad range of core models out there?

ANSI/ANS-3.5-2018 (-2009, -1998, -1993, -1985) establishes the functional requirements for the plant-referenced simulator. It also establishes the criteria for the degree of simulation, performance, and functional capability. With regard to ensuring that the nuclear and thermal hydraulic characteristics are replicated appropriately, the standard, in Section 3.1, Simulator Capabilities, requires that the response of the simulator resulting from operator action, no operation action, improper operation action, automatic reference unit controls, and inherent operating characteristics shall be realistic and shall not violate the physical laws of nature.

Nuclear and thermal hydraulic characteristics are fundamental and must be consistent with the laws of nature. The standard (2009), in Section 4.1.3.2, requires that performance of procedures on the simulator, including core performance type procedures, shall be compared and demonstrated to correctly represent the response of the reference unit at the same power level consistent with the reference unit procedures and data availability. The standard establishes six acceptance criteria with regard to simulator response during the conduct of the performance tests: (1) be the same as the reference unit startup test procedure acceptance criteria; (2) be the same as the reference unit surveillance procedure acceptance criteria; (3) be the same as the reference unit normal operating procedure acceptance criteria; (4) require that the observable change in the parameters correspond in direction to those expected for a best estimate of normal unit operation; (5) require that the simulator shall not fail to cause an alarm or automatic action if the reference unit would have cause an alarm or automatic action under identical circumstances; and (6) require that the simulator shall not cause an alarm or automatic action if the reference unit would not cause an alarm or automatic action under identical circumstances. These standards are quite high when applying them to the nuclear and thermal hydraulic characteristics.

Sim.12 10 CFR 55.31 versus §55.46: If a candidate got some of their reactivity manipulations on a core in the plant that was then refueled and then they got additional manipulations, the earlier manipulations would still count and yet this is not the case with the simulator core load. Why?

Reactivity manipulations performed on the plant-reference simulator for an applicant to meet the experience eligibility requirements may be credited when the simulator, at the time of performance, meets the requirements of 55.46(c)(2)(i) and (ii). The rule requires that the plant-referenced simulator utilizes models relating to nuclear and thermal-hydraulic characteristics that replicate the most recent core load in the nuclear plant for which a license is sought; ... The Commission, in its response to public comments during the rule making process, interpreted most recent as the current core, or if in a refueling outage, the previous core. The intent is to ensure that the applicant has a like-kind experience as they would have in the reference plant.

As is the case with reactivity manipulations conducted on the plant, any appropriate reactivity manipulation performed on the simulator may be credited provided the simulator replicates the most recent core at the time of the manipulation.

89

Sim.13 Please define the term replicate as found in 10 CFR 55.31 and §55.46.

SECY-01-0125, dated July 10, 2001, Analysis of Public Comments, Comment 3-3 Response addressed this question. The Commission believes that the terminology (in the proposed rule and subsequently in the final rule) is appropriate and consistent with ANSI/ANS-3.5-2009 (1998). It means that the plant-referenced simulators nuclear and thermal-hydraulic models operate within the tolerances specified in Section 4.1.3, Steady-State and Normal Evolutions, of the industry standard.

See also Sim.9.

Sim.14 Is core performance testing the same thing an operator would do in the course of their job?

No. The regulations, in 10 CFR 55.4, define performance testing as testing conducted to verify a simulation facilitys performance as compared to actual or predicted reference plant performance. The term Core refers to the nuclear reactor core, including but not limited to the design, configuration, and nuclear and thermal hydraulic characteristics of the core as well as the associated nuclear instrumentation that monitors or measures the various parameters which provide insight to the behavior and operating characteristics of the core.

Core performance testing means testing conducted to verify a simulation facilitys core performance replicates actual or predicted reference plant core performance. Core performance testing is not the same thing an operator may or may not do in the performance of their job. Absent conduct of the same core performance tests on the simulator as are performed on the plant and demonstration through such testing that the simulator meets actual or predicted plant performance within the acceptance criteria of the ANSI/ANS 3.5 standard, the NRC may not be able to confirm core replication in the simulator. This could adversely impact crediting of experienced gained on the simulator.

See also Sim.11.

Sim.15 Core vs. Thermal-hydraulics replication: weve talked a lot about core performance testing: how does the NRC propose how to test thermal-hydraulic performance?

Generally, the NRC does not prescribe how to conduct a performance test, but instead challenges a licensee to demonstrate that certain regulatory requirements are being met.

Thermal-hydraulic performance could be demonstrated by comparing simulator performance to actual plant performance during startup, power ascension, normal operation, and transient response. Startup test procedures and licensee event reports are good data sources.

90

Sim.16 Is it acceptable to do off-line testing of core performance (i.e., not use the actual simulator but instead a stand-alone system) to satisfy 10 CFR 55.46(d)(1)?

There is nothing to preclude core performance testing off-line for the sake of designing, debugging, and testing without other system interfaces to assure that the model is ready to be integrated into the simulated plant. However, fully integrated core performance testing on the plant-referenced simulator is expected and necessary to ensure that the appropriate input and output from and to other models are sufficient in scope and fidelity to ensure that the simulator responds as the reference plant would under the same operating conditions.

Sim.17 Updating models: is it encouraged to update our reactor vessel/core models to comply with 10 CFR 55.46?

The Commission in its statements of consideration during the rule making, emphasized that facility licensees would not be required to update their core models in order to comply with the requirements of 55.46. Refer to Regulatory Guide 1.149, Revision 4. This assumes that the simulator core model has been performance tested and the test results meet the appropriate acceptance criteria when compared to the reference plant performance or best estimate performance where actual performance data is not available.

Sim.18 If the reference plant undergoes a significant design change, such as a steam generator replacement or a power uprate, that leads or lags the installation of the same change on its plant-referenced simulator, will it be necessary to obtain Commission approval, pursuant to 10 CFR 55.46(b)(1), to use other than a plant-referenced simulator to administer the operating tests required by the regulation?

The fact that a simulator may lead or lag design changes made to the reference plant is accommodated by ANSI/ANS-3.5 (2018, 2009, 1998, 1993, and 1985). Given that the purpose of making the design changes is, ultimately, to ensure that the simulator demonstrates reference unit response, any reasonable lead or lag in the process will not alter the simulators plant-referenced classification.

The Operator Licensing Program Office expects that (1) any simulator design changes would be tested in accordance with ANSI/ANS-3.5, (2) any simulator-to-plant differences resulting from a lead or lag situation would be appropriately addressed in the training program (as would any unit differences at a multi-unit facility), and (3) that such differences will be resolved within the time frame specified in the ANSI standard. The NRC may refuse to administer operating tests on a simulator that has not been appropriately tested or if it is unable to meet the requirements of 10 CFR 55.46(c)((1)(i), i.e., if the simulator has not demonstrated the expected plant response for conditions to which it was designed to respond, the differences in response must be evaluated to confirm they do not interfere with the conduct of the operating tests. See Sim.6.

91

Sim.19 If a comparison is made between actual core plant data (Dynamic Rod Worth for Control Bank D is measured at 910 pcm) verses engineering predicative core data (calculated at 1000 pcm) verses simulator core performance data (measured at 1090 pcm) and the results show that the delta between the plant data verses predictive data is in an acceptable range, is there a performance issue with the plant-referenced simulator?

The situation described shows a deviation of more than 18% between the simulators rod worth and the reference plants rod worth for Control Bank D. Although the deviation between the simulator and predictive is less than 10%, the simulators performance is judged against the actual plant since actual data is available and measured. Additionally, the ANSI/ANS-3.5 tolerances apply in this case. The simulators performance deviation would be considered a simulator performance exception as well as a modeling discrepancy identified from performance testing. As a result, this type of modeling discrepancy could increase the potential for negative training and operator error.

92

10 CFR 55 Questions related to the operator licensing regulations CFR.1 How long does it take for an exemption request to be received and to be answered?

The time required will depend on the nature of the request and the quality of the licensees submittal. The licensee should discuss the nature of the requested exemption and the review schedule with the NRC project manager. Facility licensees are encouraged to have pre-submittal meetings with the NRC project manager and operator licensing staff in NRR. Certain COVID-related requests may be approved using an expedited process (see https://www.nrc.gov/about-nrc/covid-19/reactors/part-55-operators-licenses-form.html).

93

General Questions that do not fit within another category Gen.1 Is there some way to do a better distribution of clarifications/rulings from one site in the region to another? This would help all of us meet your expectations.

One of the NRC's goals in establishing this web site is to improve communications with facility licensees and to enhance consistency.

Gen.2 Archived.

Gen.3 What is the relationship between product owners intellectual property of the examination and the requirement for administered licensing examinations to be made publicly available? Has the question been asked about the "intellectual rights" of the examination work product owner versus publish of examinations?

Examination authors are not prohibited from copyrighting their work. However, the NRC cannot accept copyrighted materials unless the holder of the copyright signs a release form to allow its publication. When those materials are placed in the public document room, users are permitted to make one copy for personal use. If additional copies are required, the user will have to obtain permission from the copyright holder.

Gen.4 Archived.

Gen.5 What is/where do I find my "Commission Approved" training program?

As noted in the Statements of Consideration for the 1987 amendment to 10 CFR 55, a facility licensee's training program is considered Commission-approved when it becomes accredited by the National Nuclear Accrediting Board.

Gen.6 How familiar are, and what kind of training have the examiners received on the SAT process? How familiar (knowledgeable) are the headquarters management on the SAT process? What kind of training have they received?

The staff of the NRC Operator Licensing Program Office includes training and assessment specialists who are well-versed on SAT-based training processes and have many years of combined training experience. Issues and questions that come up regarding SAT-based 94

training requirements and expectations are referred to one or more of those specialists for resolution. NRC examiners and managers having responsibilities in this area have received instruction on the SAT process during periodic operator licensing examiner training and conferences.

Gen.7 I would like to see the NRC go more toward an inspection process for plants that volunteer to write the exams. Have only one NRC examiner involved, allow the utility to administer all parts of the exam and use the resident if more oversight is needed during the exam administration. The NRC should continue to make the final licensing decision.

Update: The Part 53 rulemaking effort is evaluating various examination administration methods.

Comment noted. Although the NRC favors reducing unnecessary regulatory burden, the examination policies will only be changed if the NRC concludes that the changes will not have a negative impact on reactor safety, public confidence, efficiency, and effectiveness. At the present time, the NRC sees significant benefit in continuing its current level of involvement in the operator licensing process.

Gen.8 NRC needs to understand that increased difficulty of the exam process is a negative motivator and could be a distraction to competent board operators. Recommend survey to understand scope and potential impact on safe plant operations.

Exam difficulty has gone beyond reason and is impacting the requalification program.

People are not willing to put up with the hassle and it does not result in better operators.

It is impossible to meet question standards and avoid "tricky" questions, very knowledgeable operators can appear less that competent based on complexity of question rather than a test of knowledge.

As reported in Attachment 1 (Section 1) of SECY-98-266, the NRC has also noted a slight decrease in the average passing rates on both the written and operating portions of the facility-prepared examinations when compared with the passing rates on NRC-prepared examinations.

However, the decrease could be caused by a number of factors including variations in the average level of experience of the license applicants, changes in the quality of the training or the facility licensee's threshold for screening its applicants before they take the licensing examination, or variations in the average level of difficulty of the examinations. Although the staff did not intend for the level of difficulty or the failure rate on the examinations to increase, the examiners' efforts to achieve NRC standards regarding the cognitive level of questions and to improve the plausibility of the distracters may have improved the discrimination validity of the examinations. Consequently, those applicants who may have passed an examination containing lower cognitive level questions on which some of the distracters could be eliminated as implausible are now having more difficulty selecting the correct answers; in essence, their chances of passing the examination by guessing some of the correct answers have diminished.

Considering the historical fluctuation in the average examination passing rates and the other 95

factors that could be responsible for some or all of the observed decline, the NRC has concluded that any increase in the level of difficulty is not significant.

The Operator Licensing Program Office will continue to monitor the applicants' performance for indications that the examinations are becoming too difficult. The initial operator licensing examination performance trends since 1991 are available for review on the Operator Licensing Process page.

Gen.9 The most common issue raised by hot license candidates and requalification license holders are "trick questions" on the operator written exam and that the exam is not a fair test of operator knowledge.

The NRC exam has become an exercise in exam taking skills instead of a knowledge assessment.

The NRC goes to considerable lengths to ensure that its examinations measure what they are intended to measure, thereby enabling the NRC to distinguish between applicants who have and have not mastered the knowledge and abilities required to be safe nuclear power plant operators. The principles of fairness, validity, and safety have guided the NRC throughout the process of developing and implementing NUREG-1021. As stated in Form 4.2-2 of NUREG-1021, the NRC strives to minimize unnecessary difficulty, trickiness, and irrelevancy in its written examination questions. Authors and (multiple) reviewers are expected to identify and correct these psychometric deficiencies. Moreover, Section E.4 of ES-401 encourages facility licensees to peer-validate the written examination in a final effort to identify and correct deficiencies that might affect the validity of the examination.

Although the NRC has increased its emphasis on higher cognitive level questions and the plausibility of distracters in an effort to enhance the discrimination validity of the examinations, some may have misinterpreted these actions as an effort to trick or fool otherwise knowledgeable applicants. Truly knowledgeable applicants should be able to pass the examination regardless of their test-taking skills. Applicants who rely too much on their test-taking skills or their ability to guess the right answer after eliminating the implausible distracters should not be able to pass the licensing examination.

96

Gen.10 Guidelines shouldn't be open for individual examiner interpretation if it could show up as a weakness in the exam report. Example: Amount of question/operating test overlap on the requalification exam from week to week.

There are still regional "requirements" (not NUREG interpretations) outside of NUREG-1021 such as regional office interactions (ROIs) [etc. for example: "one scenario must have a computer failure." Why are these things still out there? Shouldn't they be in NUREG-1021 if they are required?

What is the NRC doing to ensure that the examiners are working to the same standards?

Comments noted.

The NRC's existing measures to maintain consistency in the examination process were summarized in Attachment 1 to SECY-98-266, "Final Rule - Requirements for Initial Operator Licensing Examinations." NRC examiners are expected to comply with the guidelines in NUREG-1021 and to exercise good judgment in those areas requiring a subjective evaluation.

The reviews and audits conducted by NRC regional management and the operator licensing program office and the continuing training program for examiners help minimize individual examiner interpretations and ensure consistency.

Section C of ES-1.1 requires the NRC Regional Offices to consult the operator licensing program office if the instructions of NUREG-1021 cannot be met. Furthermore, the NRC Regional Offices must obtain written approval prior to implementing any initiative that has the potential to undermine examination consistency.

Gen.11 We are interested in attending a regional workshop to discuss exam development, criteria, and receive NRC input. Is the something the NRC can coordinate?

The NRC has sponsored and participated in numerous examination workshops and, to the extent possible, will continue to work with facility licensees and industry training groups in this area. Please contact the applicable NRC regional office or the NRC Operator Licensing Program Office.

Gen.12 Archived.

Gen.13 Deleted.

97

Gen.14 Where do I get a copy of the 2-year NRC examination schedule?

The examination and inspection schedule (covering at least the next year) is posted on this web site. We expect to update the schedule at least quarterly.

Gen.15 Archived.

Gen.16 How will probabilistic risk assessment (PRA) need to be identified in future exams?

Section B of ES-3.1 (in NUREG-1021) requires examination authors to consider PRA insights (e.g., dominant accident sequences and risk-important operator actions) when preparing the operating tests. The Examination Outline Quality Checklist (Form ES-2.3-1) requires NRC examiners to assess whether plant specific priorities (including PRA and IPE insights) are covered in the appropriate exam section. Although there is currently no requirement to identify which test items address the PRA insights, the examination author should be able to explain to the chief examiner how those insights were covered. The NRC has no immediate plans to change this requirement.

Gen.17 How do we stabilize the examination process so that it won't have a detrimental effect on industry staffing needs?

Many of the changes that have recently been made in the examination process can be directly attributed to industry requests. The NRC will continue to be responsive to its industry stakeholders as long the agency's goals related to safety, public confidence, efficiency, and effectiveness are not compromised. In that regard, the operator licensing program office will continue to work with the NEI operator licensing focus group and other industry stakeholders in an effort to identify those changes that are in the best interest of the industry and the public.

Gen.18 Can facility licensees electronically submit NRC Form 398, "Personal Qualification Statement -- Licensee," and NRC Form 396, "Certification of Medical Examination by Facility Licensee?"

The NRC has permitted the electronic submittal of documents by outside participants since January 1, 2004, when the NRC rule governing electronic submittals ("E-Rule") took effect. This rulemaking expanded participation in electronic communication by giving all licensees, vendors, applicants, and members of the public the option of submitting documents to the NRC in various electronic formats, including CD-ROM, e-mail, and a special Web-based interface, the Electronic Information Exchange ("EIE"). EIE has digital signature capabilities, and its use is 98

explained at length in the guidance document accompanying the E-Rule Appendix A, United States Nuclear Regulatory Commission Guidance for Electronic Submissions to the Commission. The E-Rule and accompanying guidance can be found at http://www.nrc.gov/site-help/e-submittals.html.

See Gen.57 for information on the use of electronic signatures for submittals.

Gen.19 During a recent inspection, it was noted that the facility licensee's UFSAR requires the control room operators to take Potassium Iodide (KI) pills under certain post-accident conditions to minimize long term consequences from potential exposure to radionuclides. Section 5.2.2 of ANSI/ANS-3.4-1983, Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants, requires operators to be free of any conditions that are considered by the designated medical examiner as significantly predisposing to incapacity for duty, including any treatment involving drugs, chemicals, diets, or other agents. As a result, is the licensee required to test its control room operators for sensitivity to KI as part of its periodic physical examinations?

The NRC staff has researched this issue and determined that allergic or allergic-like reactions are rare in people who take KI and that testing for this is not practical. The current literature also suggests that allergy to seafood or radio-contrast material does not necessarily confer an elevated risk for allergy to KI, so the usual history of "iodine allergy" would not be helpful.

Moreover, the experience in Poland (after Chernobyl) has shown that serious reactions to KI itself were extremely rare. Therefore, it appears that the administration of KI pills would not significantly predispose the operators to incapacity for duty. However, facility medical examiners should evaluate each operators specific circumstances in light of their facility licensees KI administration practices to determine if the operator is medically qualified or if some type of license restriction is necessary.

99

Gen.20 What is a permanent disability? The rule (10 CFR 55.25), Regulatory Guide 1.134, and ANSI/ANS-3.4 do not define it; they use terms like minimum conditions and disqualifying conditions.

You are correct; there is no formal definition of a permanent disability.

Section A.3 of ES-5.3 of NUREG-1021 and Information Notices 04-20, 94-14, and 91-08 (all of which are available via links from the operator licensing web page) provide additional information regarding the staffs expectations with respect to medical standards for licensed operators. With regard to permanence, Section A.3.b of ES-5.3 indicates that if an operator does not meet the specific minimum standards/requirements in the applicable version of ANSI/ANS-3.4 but is expected to meet those standards (without exception) again in the future, then the operators condition/disability is considered temporary and does not need to be reported to the NRC; however, the facility licensee is expected to administratively restrict the operators activities, as appropriate, during the term of the condition/disability. While most of the medical conditions/disabilities, including those that result in failure to meet the minimum requirements for medical qualification, identified in ANSI/ANS-3.4 are probably permanent, it is up to the examining physician to evaluate each operators situation on a case-by-case basis and assess whether the operator will be capable of meeting the standards in the foreseeable future.

For example, the facility should consider reporting a condition for an operator who requires medication to meet the minimum standard for blood pressure (i.e., less than or equal to 160/100 mmHg), unless the physician can reasonably determine that the condition will be controllable without medication in the foreseeable future. The NRC will review the facility licensees administrative controls and its physicians explanation for why the condition was considered temporary during the licensed operator requalification program inspections.

If an operator develops any permanent physical or mental condition that could adversely affect their performance of assigned operator job duties or cause operational errors endangering public health and safety, it must be reported to the NRC within 30 days of learning of the diagnosis (per 10 CFR 55.25 and §55.33(a)(1)). It does not matter whether the operator has tripped the specific minimum requirement or the related disqualifying condition threshold in ANSI/ANS-3.4 - all conditions, disabilities, and incapacities should be reported to the NRC for evaluation, regardless whether or not the facility has implemented compensatory measures. If an operator develops a condition that is not identified in the industry- and NRC-approved ANSI standard, but the examining physician believes that it could affect the operators performance or cause errors, then it would be prudent to report it anyway (or at least enquire whether it should be reported).

If the examining physician concludes that the operators condition, disability, or incapacity does not affect performance or safety, they can request and justify a waiver of the medical requirement; for example, a color-blind operator might be granted a waiver based on a satisfactory practical test. If the operators condition, disability, or incapacity can be safely accommodated by a restriction on the license (e.g., no-solo, more frequent monitoring, or requiring medication), then the physician should make an appropriate recommendation to the NRC on Form 396. However, if the operators condition, disability, or incapacity is such that it cannot be reasonably waived or accommodated, then the facility licensee should request the NRC to terminate the operators license.

100

Gen.21 In the past, if we had an individual on daily medications for hypertension and the condition was stable, we would send an information letter to the NRC. If the condition was not stable, or there were multiple medications for treatment, or the condition was outside the regulation we would request a no-solo license. How do we interpret this now that NRC Form 396 has been revised? Many physicians treat hypertension well below the 160/100 limit allowed by the ANSI standard. If an individual is treating with medications because their blood pressure is 148/88 and their physician is more aggressive, do we check the information only box or do we check the medication box on NRC Form 396?

The purpose for placing a take your medicine condition on operators licenses is to impress upon them the importance of maintaining their medical qualifications and to ensure that their medical condition and general health will not adversely affect their performance of assigned duties or cause operational errors endangering public health and safety (as required by 10 CFR 55.33). Presumably, if the examining physician directs an operator to take a prescription medication for whatever reason, it is to protect their general health and to prevent them from exceeding a threshold that would disqualify them from performing licensed duties - in this case the 160/100 mmHg blood pressure limit - or affect their job performance. If we fail to put a condition on an operators license at the time of initial diagnosis and treatment, we may likely not get another chance to do so, assuming the treatment is successful. Therefore, you should check the medication box on NRC Form 396 even if the threshold for disqualification has not yet been exceeded.

Gen.22 When we select the "Solo operation is not authorized" box on the revised NRC Form 396 there is no corresponding box to enter the wording of the restriction - will the wording be provided by the NRC based on the license holder's status (RO vs. SRO)?

Yes, the NRC will enter the standard wording based on the individuals license level. The instructions on the back of Form 396 refer to Section A.3.d of ES-5.3 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, which includes the standard wording for a number of operator license medical conditions.

Gen.23 If we previously submitted an "Information Only" letter for a medical condition that did not result in a license restriction (e.g., a well-controlled asthmatic or hypertensive on medication), do we need to submit a request for the new "Must take medication as prescribed to maintain medical qualifications" restriction based on the revised NRC Form 396?

Typically, there is no need to submit Information Only NRC Form 396s as the NRC will review all NRC Form 396s received regardless of the purpose of the submittal. Information only submittals should be in accordance with NRC Form 396 instructions for box 11. Per 10 CFR 55.25, you do not need to submit a revised NRC Form 396 during the term of an operators license unless there is a permanent change in the operators medical condition that would 101

cause them to fail to meet the requirements of 10 CFR 55.21. The next time the operators license is due for renewal, you would need to submit a new NRC Form 396 in accordance with 10 CFR 55.57(a)(6) and check Box 4 if the examining physician has determined that the operator must take a prescription medication to maintain their medical qualifications.

Gen.24 What are the qualifications/parameters for using the "must take medication as prescribed to maintain medical qualifications?" (Diabetics, previous heart attack, organ transplant patients, asthmatics requiring DAILY use medication?)

The instructions on the back of NRC Form 396 indicate that Box 4 should be checked if, in the opinion of the examining physician, the applicant's medical qualification per the applicable ANSI standard is contingent on taking a prescription medication. It does not matter if the medication is administered on a daily, weekly, monthly, or as-needed basis; the license condition would simply require the operator to take the medication as prescribed.

Gen.25 If a licensed operator is already taking medication for hypertension and the physician prescribes either an increased dosage or a change in medication, would this have to be reported to the NRC?

As discussed in the response to Question Gen.23, 10 CFR 55.25 only requires facility licensees to submit a revised NRC Form 396 during the term of an operators license if there is a permanent change in the operators medical condition that would cause them to fail to meet the requirements of 10 CFR 55.21. The examining physician would have to make that determination based on the guidance in whichever revision of ANSI/ANS-3.4 the facility is committed to and then recommend a conditional license if they deem it necessary to accommodate any disability that the operator has developed. The next time the operators license is due for renewal, the facility licensee would need to submit a new NRC Form 396 in accordance with 10 CFR 55.57(a)(6) and check Box 4 if the examining physician has determined that the operator must take a prescription medication to maintain their medical qualifications. Simply increasing or changing a hypertensive operator's medication would not normally need to be reported unless the examining physician believes the operator's blood pressure is out of control to a point that it requires more frequent monitoring or a no-solo license (i.e., the addition of license condition # 4 or 6 on NRC Form 396). Refer to Question Gen.21 for additional guidance.

Gen.26 We recently received an amended license for one of our SROs as a result of reporting, for information only, a new diagnosis and medication for borderline type 2 diabetes. What is the purpose of amending an operator's license for that condition? What happens if their physician ceases that treatment? In the time required to issue another amended license, they are technically in violation of their license for not taking the medication.

If a previously healthy and unrestricted operator develops type 2 diabetes, which could conservatively be classified as "a permanent physical or mental condition that causes the licensee to fail to meet the requirements of 10 CFR 55.21," the facility licensee would be 102

required (by 10 CFR 55.25) to notify the Commission within 30 days of learning of the diagnosis.

Since the physician has determined that the operator requires medication to control their diabetes (and remain medically qualified per ANSI/ANS-3.4), the facility licensee should check Box 4 on NRC Form 396 and provide appropriate medical evidence with the form (as required by 10 CFR 55.23(b)) for evaluation by the NRC. If the physician later determines that the operator no longer needs to take medication, the operator would not be in violation of their license given that the condition specifically states to "take medication as prescribed;" if it's no longer prescribed by the physician, the condition becomes irrelevant.

Gen.27 What are the reporting requirements if a licensed operator is newly diagnosed with and medicated for hypercholesterolemia (i.e., high cholesterol)?

High cholesterol is not, in and of itself, a condition that is addressed in ANSI/ANS-3.4; therefore, it would generally not disqualify an individual from having an unrestricted license, would not have to be reported to the NRC, and would not require a license condition. However, Section 5.4.9 of ANSI/ANS-3.4 (1996) indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would disqualify an operator.

Therefore, when the physician makes a new diagnosis and prescribes medication for hypercholesterolemia or any other disorder (whether or not it is addressed in the ANSI standard), the physician needs to consider the possible side effects to ensure (as required by 10 CFR 55.33(a)(1)) that they will not cause operational errors or affect the operator's capacity to safely perform licensed duties.

Gen.28 Deleted.

Gen.29 How far back does a medical history have to go?

The medical history includes any information related to a potentially disqualifying condition that may currently exist no matter when the condition occurred. Even if the applicant had a potentially disqualifying condition as an infant, the physician would still have to make a determination that the applicant does not currently suffer from any disqualifying condition in accordance with the ANSI-3.4 (15.4) that the facility is committed to. Therefore, the condition should be documented in the medical history but would not need to be reported to the NRC unless a restricted license is requested due to a permanent medical condition.

Gen.30 Can the NRC clarify in writing whether it is acceptable for an operator to satisfy the near visual acuity requirement in one eye and the distant acuity requirement in the other (as might be the case if someone had Lasik surgery)?

103

Such conditions have been reviewed on an individual basis and found to be acceptable.

Applicants who have uncorrected near visual acuity of at least 20/40 in one eye and uncorrected distant visual acuity of at least 20/40 in the other eye do not require a conditional license.

Gen.31 Can the NRC provide examples of actual operators who have been permanently disqualified?

Decisions to permanently disqualify an operator are generally, if not always, made by the facility licensee. If a facility licensee determines that a new license applicant is medically disqualified, the NRC would never see the individuals application or medical history. If an operators license is suspended by the facility licensee, the NRC might ask to be notified of and in agreement with the operators acceptable medical status before they return to licensed duty. However, we often do not get any further medical input and have to assume that the operator has been permanently disqualified.

Gen.32 Deleted.

Gen.33 Archived.

Gen.34 Does the NRC expect us to report solely the for the medications side effects when an operator is taking medication that is not to treat a potentially disqualifying condition?

The fact that an operator is taking medication is not, in and of itself, reportable to the NRC. The facilitys physician would need to evaluate the effects of any medications (prescription and over-the-counter) that an operator is taking to determine if they are at risk of incapacitation.

Gen.35 We fill out NRC Form 396 after every biennial examination so compliance has the form on file, which causes about half of our submittals to be on the old version of the form. Is this a problem, or should we be filling out the forms as needed for a submittal?

It is only necessary to fill out and submit NRC Form 396 when applying for a license or reporting a change in medical status. Section B, Certification, should be signed and dated at the time of submittal; it should not be back-dated to coincide with the date of the last medical examination, which is entered in Section A of the form. Our preference would be for you to use the latest version of the form within 60 days of release, which is available on the NRCs web page at https://www.nrc.gov/reading-rm/doc-collections/forms/nrc396info.html. Because the revision date is month/year, the NRC assumes that it was the end of the month and would give until the 104

end of the month that is 2 months that follows the revision date. For example, if it were revised in September, the facility licensee would have until the end of November to use the new form.

Gen.36 If a licensed operators doctor restricts them with no overtime should we report this to the NRC?

Personal physicians are primarily concerned with their patients well-being; they are unlikely to be familiar with the requirements in ANSI/ANS-3.4 or to have an overriding interest in reactor safety. However, if they restrict one of your licensed operators to no overtime, they presumably have a medical basis for imposing such a restriction. Consequently, it would be prudent for the facilitys physician to evaluate the operators status to determine if any disqualifying conditions exist and whether a reportable change warranting a license restriction has occurred.

Gen.37 If an operator has one blood pressure reading over 160/100 followed by two readings that are lower, should we report the high reading to the NRC? Should we restrict the operator immediately or just refer them for treatment?

If this is a new condition and cannot be attributed to a measurement error or anomaly, then the conservative response would be to immediately restrict the individual from duties requiring a license until an evaluation can be performed to determine if the operator has developed a permanent physical condition that causes them to fail to meet the requirements of 10 CFR 55.33(a)(1). Under todays treatment guidelines, most physicians will begin medicating their patients for hypertension before they reach the 160/100 mmHg threshold stated in ANSI/ANS-3.4, so, if that is the treatment regimen that is followed in this case, it would have to be reported to the NRC within 30 days after learning of the diagnosis using NRC Form 396. The form should include a recommendation to apply a take your medicine condition to the individuals license plus any other condition(s) that the examining physician might determine to be necessary based on their evaluation of the operators condition vis--vis the criteria in the ANSI standard. Refer also to Question Gen.21.

Gen.38 What fasting blood sugar level is deemed uncontrolled and in need of further evaluation? Is there a cutoff? Is there an A1C level cutoff?

Uncontrolled diabetes is a non-specific term, so it is up to the examining physician to use their judgment. For coding purposes, it has been described as elevated blood sugar with symptoms, and/or a blood sugar level of more than 300, or an A1C twice normal, or blood sugars that vacillate up and down considerably. Although these criteria are helpful, the NRC does NOT endorse or require that they be used.

105

Gen.39 Archived.

Gen.40 If an operator is medically disqualified for licensed duties and awaiting a final determination from the doctor, is it acceptable to use the individual as a procedure reviewer if the facility requires that position to have an active license?

Although the NRCs regulations do not require procedure reviewers to have an active operators license, the NRC staff recognizes the benefits of using licensed operators in that capacity.

Nevertheless, the NRC does expect facility licensees to implement their procedures as written, so using medically disqualified operators to perform such duties could be a regulatory concern depending on how the facilitys administrative procedure is written. In the absence of a regulatory requirement, the facility would be free to change its administrative procedure to allow medically disqualified operators to perform that function.

Gen.41 If an operator takes a sleep aide (e.g., zolpidem or eszopiclone) how long do they need to wait before returning to licensed duties? There are reports that the effects can linger for up to 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />; do they need to wait that long?

Section 5.3.9 [5.4.9] of ANSI/ANS-3.4-1983 [1996] indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would be considered disqualifying. Most medications have multiple side effects that may vary considerably depending upon the dosage and the takers individual body chemistry. Therefore, the facilitys physician would need to determine, based on the operators history and physical exam, whether the effects of any medication (prescription and over-the-counter) that the operator is taking might disqualify them from performing licensed duty and for how long.

Gen.42 What are the reporting requirements if a licensed operator takes a prescribed medication (e.g., modafinil) as needed to improve wakefulness while on shift work?

Section 5.3.9 [5.4.9] of ANSI/ANS-3.4-1983 [1996] indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would be considered disqualifying. Most medications have multiple side effects that may vary considerably depending upon the dosage and the takers individual body chemistry. Given that dizziness, which is a possible disqualifying condition, is sometimes observed with the use of modafinil, the facilitys physician would need to evaluate the effects that the operator is experiencing to determine whether they might be disqualified when taking (or neglecting to take) the medication.

Assuming that the physician has concluded that no other mental, psychological, or physical condition exists that might be impairing the operators alertness (which could be disqualifying per Section 5.3.8 [5.4.8] of ANSI/ANS-3.4-1983 [1996] and a reportable condition), prescribing 106

modafinil (or a similar drug) as needed to promote wakefulness while on shift work would not be reportable to the NRC. Although mental alertness is identified as a general health requirement in Section 5.2 of the standard, the NRC staff understands that rotating shift work can affect sleep patterns, thereby leading to fatigue and diminished mental alertness. These effects would generally be considered transient in nature and not permanent physical or mental conditions that would need to be reported pursuant to 10 CFR 55.25.

Gen.43 Section B, Certification, of NRC Form 396 requires the name, title, and signature of the senior management representative on site. Who is that?

As stated in Section C.4 of ES-2.2 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, the term senior management representative on site is synonymous with authorized representative of the facility licensee, which includes examples such as the plant manager or site vice-president. In accordance with 10 CFR 55.31, How to Apply [for a license], that individual must certify when an applicant has completed all of the facility licensees requirements and commitments for the desired license level (e.g., experience, control manipulations, training, and medical fitness). That certification involves signing Block 27a of NRC Form 398 and Section C of NRC Form 396.

Gen.44 If we submitted a license renewal application in 2002 for an operator taking medication, but the NRC did not condition the license, do we need to provide supporting medical information when submitting the NRC Form 396 for a 2008 renewal if the operator is still taking the same medication but our physician currently does not believe a take your medicine condition is necessary?

If we reported a change in medical status with a letter in 2004 indicating that an operator was taking medication, but the NRC did not condition the license for this condition, do we have to provide additional supporting details with the Form 396 for a current renewal if the operators condition and treatment have not changed (e.g., same treating physician, same symptoms, same medicine, same dosage)?

If the NRC previously said that a medication was not a license condition, and the operator is still taking the same medication, do we have to report this medication every time the renewal occurs?

The license renewal process provides an opportunity, once every six years, for the NRC to review every licensed operators medical condition and general health (including any medications that the operator is taking) to ensure they will not adversely affect the performance of assigned operator duties or cause operational errors endangering public health and safety.

Because NRC Form 396 did not contain a must take medication restriction prior to 2006, a medical condition reported before the form was revised could, today, result in a determination that a must take medication restriction is warranted. Every time you submit NRC Form 396 for an operator, regardless whether you are reporting a change in medical status or renewing their license, you should check all the condition/restriction boxes that apply to that operator on the date that the certification is signed and provide supporting documentation, as necessary.

107

For medical conditions that existed before the must take medication restriction was implemented, you should submit sufficient supporting documentation with NRC Form 396 to enable our physician to determine whether or not a restriction is warranted, even if a must take medication restriction was not imposed for the same condition in the past. If the supporting documentation was submitted with the previous application or status report and has not changed, an entry to that effect in the explanation field on the form would be sufficient. Also refer to Questions Gen.21, Gen.23, Gen.24, and Gen.25 above for additional discussion of information-only reports and medication restrictions.

Gen.45 I received an RO and/or an SRO license from the NRC for a 10 CFR 50 facility licensee, but I am no longer licensed. Does the NRC keep records of those licenses issued and, if so, how can I obtain a physical copy? How do I obtain my just my license number?

Operator license records are protected by the Privacy Act (PA). The PA works as a companion with the Freedom of Information Act (FOIA). Information pertaining to the Freedom of Information Act/Privacy Act can be found at: https://www.nrc.gov/reading-rm/foia/foia-privacy.html.

Due to record retention requirements, the NRC does not maintain records for licensed operators if the operator license has been terminated for greater than 10 years.

To request operator license records, follow the instructions provided in the Privacy Act Request Guide at https://www.nrc.gov/reading-rm/foia/privacy-request.html#access to submit a Privacy Act Request with NRCs Freedom of Information/Privacy Team at FOIA.resource@nrc.gov.

In order to request an operator license number, the appropriate operator license record, such as the initial licensing letter, would need to be requested as a Privacy Act Request.

Gen.46 What are the reporting requirements for an individual who has sleep apnea? What if they are using a Continuous Positive Airway Pressure (CPAP) machine for treatment?

The licensee needs to report the condition of sleep apnea since it is generally a permanent medical change that can affect the individual licensee's capacity to perform their required duties.

Depending on the severity of the condition and the requirement to use the CPAP treatment, the NRC medical authority may require a license restriction for taking the prescribed treatment, similar to the required taking of prescribed medication. In addition, the condition can cause additional medical problems including hypertension, which may be a disqualifying medical condition depending upon its severity. Refer to Section 5.2, Health Requirements, of ANSI/ANS-3.4-1996, Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants; Section 5.2(1) specifically requires operators to have acuity of senses, and Section 5.2(3) requires metal alertness and emotional stability. Section 5.2, General Requirements, of ANSI/ANS-3.4-1983 contains the same requirements.

108

Gen.47 What is the NRCs policy regarding restricted operators (i.e., those who must take medications as prescribed to comply with a license condition) keeping medications on site in the event of an extended work stay that might be required during a hurricane, pandemic, or other emergency situation?

Although the NRC has no formal policy to address this specific situation, individual operators and facility licensees should consider the following regulatory requirements guidance in planning their response to emergency situations that may require operators to remain on-site for extended periods of time.

• Section 5.4 [5.3] of ANSI/ANS-3.4-1996 [1983], American National Standard for Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants, identifies a number of conditions that, unless adequately compensated for, shall disqualify an individual from licensed duty. Section 5.4[3].9 specifically states that "any medication taken in such a dosage that the taking or [temporary] delay of taking might be expected to result in incapacity [] would be disqualifying.

• 10 CFR Part 26, Fitness for Duty [FFD] Programs, includes a number of requirements that appear pertinent under such situations. For example, FFD programs must: (1) provide reasonable assurance that individuals are not mentally or physically impaired from any cause; (2) describe the individual's responsibility to report FFD concerns; (3) describe the process that the licensee will use to ensure that individuals who are called in to perform an unscheduled working tour are fit for duty; (4) establish controls and conditions under which an individual can perform work, if called in, after reporting that they considers themself to be unfit for duty for reasons including illness, fatigue, or other potentially impairing conditions; and (5) provide training to ensure that the individuals who are subject to the rule have knowledge of their responsibilities under the FFD program.

• Licensed operators are ultimately responsible for compliance with any conditions stated on their license. If possible, they should consider storing a one-week supply of necessary and critical medications at their work site, having such medications readily available at their homes to take to their work sites on short notice, and/or making contingency plans to have their medications brought to the site, if needed.

• If a licensed operator is required to remain at the facility for a period of time that exceeds the prescribed medication frequency with no medication available, they should inform the facility licensee and, if possible (*), discontinue licensed duties.

• Facility licensees are encouraged to accommodate and facilitate their operators compliance with medical requirements. In addition to training their operators on their FFD responsibilities, they should, if possible, provide any assistance they might need to store their required medications on-site safely and securely.

• If a licensed operator is required to remain at the facility for a period of time that exceeds the prescribed medication frequency with no medication available, the operator should be removed from licensed duties, if possible (*), and assigned to other work consistent with the operators diminished capacity. If possible, the facility should permit the operator to return home to retrieve the required medications or provide other assistance, as necessary, in procuring a supply.

109

(*) Note that performing operator duties in violation of a license condition or FFD requirement could result in enforcement action against the individual and/or the facility licensee.

However, the NRC may exercise discretion, in accordance with its Enforcement Policy, and mitigate or refrain from enforcement action based on the relevant circumstances of the particular case.

Gen.48 How does the NRC balance the medical reporting requirements for an operator's medical certification with the individual's rights for privacy of information under HIPAA laws?

Should there be a privacy agreement between the license holder and the NRC (also with all other groups within each utility that handle this confidential information)?

It is our understanding that the Health Insurance Portability and Accountability Act of 1996 (HIPAA) applies only to three types of entities, a covered health care provider, a health care clearinghouse, or a health plan. The NRC is none of those entities, so HIPAA requirements do not apply to the operator medical records that the NRC maintains. However, the Privacy Act does apply because they are records of individuals as defined by the Act. The limited medical information that the NRC obtains from individuals and facility licensees as part of the operator licensing and license renewal process is maintained in a Privacy Act system of records and afforded all of its protections. Please refer to http://www.nrc.gov/reading-rm/foia/privacy-systems.html for more information regarding the NRC's Privacy Act System of Records, including NRC-16, which covers all of the operator licensing records.

The NRC does not disseminate any of that medical information outside of the NRC and maintains it only for official use in making licensing and re-licensing decisions pursuant to 10 CFR 55.33 and §55.57. Those regulations permit the NRC to approve an initial/renewal application only if it finds that the applicant's medical condition and general health will not adversely affect the performance of assigned operator job duties or cause operational errors endangering public health and safety. In most cases, that finding is based on the facility licensee's certification with little or no transmission of personal medical information. However, in some instances, when an applicant's general medical condition does not meet the minimum standards, the NRC may still approve the application per 55.33(b) based on the facility's recommendation and supporting medical evidence provided by the licensee and the examining physician.

When it comes to the disclosure, protection, and exchange of private medical information, 45 CFR 164.512 identifies a number of situations when "covered entities" may disclose protected health information without the individual's written consent, and subsection (d) specifically allows disclosure for health oversight activities such as government regulatory programs for which health information is necessary to determine compliance with program standards. Although a separate privacy agreement between the license holder and the NRC should not be necessary, the facility should take measures to ensure that personnel within their organization handle the information appropriately.

110

Gen.49 When a facility licensee submits a NRC Form 396 involving a license condition or restriction, the medical information is reviewed by the NRCs physician. What is the purpose of this review?

When making operator license decisions, the NRC considers all information certified by the facility senior management representative on site using NRC Form 398, Personal Qualification Statement - Licensee. Part of the personal qualifications being certified relates to medical qualification and is certified via Form 396. Because evaluation of an individuals medical qualification often requires medical expertise not found within the staff, the NRC retains the services of contract physicians to perform a review to support making the licensing decision.

As stated on NRC Form 396, the overriding purpose of licensed operator medical qualification is that the individual would not be expected to cause operational errors endangering public health and safety. The guidance contained in industry consensus standards, specifically versions of ANS/ANSI-3.4 (power reactors) and 15.4 (non-power reactors), forms the basis in reaching this determination. In some cases, conditions or restrictions must be placed on an individuals license to compensate for a medical shortcoming relative to these standards to ensure safety.

Such conditions or restrictions are recommended by the facility licensees examining physician on Form 396 and must be supported by medical evidence. The purpose of the NRC physician review is to evaluate the Form 396 and supporting medical evidence to determine if the physical condition and general health of the applicant/operator are such that they would not be expected to cause operational errors which might endanger public health and safety. The NRC physician review is a confirmation that the facility physicians request regarding license conditions or restrictions is appropriate and that the applicant/operator will satisfy ANSI/ANS-3.4[15.4]

requirements, or that a requested waiver (exception) is appropriate. The NRC physicians review is not for the purpose of re-diagnosing the individual.

Gen.50 In the NRC Form 396 block where the facility requests license conditions or restrictions on the basis of physician recommendations, the form states Provide explanation and attach supporting medical evidence for NRC review. What constitutes supporting medical evidence for the purposes of the NRC review?

Supporting medical evidence consists of the findings, laboratory data, examination results, diagnoses, and treatment plans (such as prescribed medications, use of therapeutic devices and planned monitoring) that support a determination of whether or not an individual meets the physical condition and general health requirements to be licensed as an operator. The evidence must address the general health and disqualifying conditions contained in ANS/ANS-3.4[15.4]. Insights into the general prognosis as it relates to the need for more frequent monitoring (such as 3/6/12 month status reporting and no solo restrictions) are beneficial for the reviewing physician. The following specific examples are provided as illustrations:

• If the must take medication condition is recommended for hypertension, the name of the prescribed medications and dosages must be stated. Additionally, blood pressure readings from the most recent examination need to be reported so that the reviewing physician can confirm compliance with the ANSI/ANS-3.4[15.4] limits. Additionally, if information on the effectiveness of medications (how well are they controlling blood pressure) and any side 111

effects (presence or absence) is available, it needs to be included to help the reviewing physician determine the individuals medical qualification status.

• Commonly reported conditions involving medications include diabetes and thyroid disease.

The supporting medical evidence provided for these conditions should follow the same general form as the hypertension example. In the case of diabetes, the reviewing physicians would rely on fasting blood sugar and/or hemoglobin A1C laboratory data to determine if the disease was being controlled as required by the ANSI/ANS standard.

Similarly, thyroid function study data is useful in confirming that the disease is controlled [ref.

ANSI/ANS-3.4-1996, Section 5.4.3.(2)]. As is always the case when medications are involved (refer to Questions Gen.27, Gen.34, Gen.41, and Gen.42), an evaluation of side effects and the potential for incapacitation must be made. The results of any such assessment should be shared with the NRC reviewing physicians to facilitate their determination of medical qualification.

• Certain cardiovascular conditions can be disqualifying. When reporting instances of coronary heart disease, available data (e.g., EKG or other test procedure or examination results) that indicate satisfactory cardiac function to consider an individual as medically qualified must be submitted. Information on medications, therapeutic devices, any co-morbidities (obesity, diabetes, hypertension, etc.) and the need for follow-up monitoring are useful to the reviewing physician in making a general assessment of the applicants/operators health and its potential effect on safe plant operation. A statement regarding the individuals physical capability to satisfactorily perform all assigned duties, including a brief description of any accommodations in place to assure capability to perform these duties, would facilitate the NRC physicians review to determine medical fitness for licensing.

In summary, for the NRC reviewing physicians to perform a meaningful review, some basic medical evidence/information relative to the following questions must be included:

• What is the medical problem/issue? (link to ANSI/ANS-3.4[15.4] disqualifying condition)

• What are the related medical examination results? (readings, laboratory data, physician observations)

• What is the diagnosis? (i.e., is the condition stable? being controlled? likely to result in incapacitation or eventual disqualification?)

• What is the treatment plan (medications, therapeutic devices, accommodations, monitoring) and proposed license restriction(s) to ensure ANSI/ANS-3.4[15.4] and not endanger public health and safety requirements are met?

Gen.51 There have been many cases where NRC Form 396s have been returned from the NRC physicians requesting additional information before a determination can be made. What suggestions can the NRC offer to help ensure the reviews can be completed more efficiently?

112

Some specific suggestions to facilitate a smooth review process based on recent experiences are:

• Make a clear link to ANSI/ANS-3.4[15.4] conditions/requirements in the submittal. Use the provided block on Form 396 to clearly link the proposed restriction to the ANSI/ANS-3.4[15.4] disqualifying condition.

• When proposing an other restriction or exception (Box 9 on Form 396), use the Proposed Wording of Restriction block to clearly state what the license condition should say in order to assure there is no misunderstanding.

• If medical information is being submitted as information only, indicate by checking Box 11 on Form 396. Clearly state the impact (or absence thereof) of the information on the individuals qualification relative to that condition. If medication is involved, a statement regarding possible side effects and the potential for incapacitation needs to be included when possible. (Refer to Questions Gen.27, Gen.34, Gen.41, and Gen.42)

• When making a submittal that involves a change in medication (although not necessarily required - see question Gen.25), it needs to contain a brief statement of the reason for the medication change, a confirmation that ANSI/ANS-3.4[15.4] requirements continue to be met, and that the existing license conditions remain adequate (e.g., the medical situation is stable such that more frequent monitoring or no solo changes are not warranted). This information will allow the NRC medical reviewer to have a more complete picture of the basis for the reported change and allow for an evaluation of the impact on overall medical qualification in accordance with the standard.

• If a no solo restriction is proposed, a simple statement identifying a specific ANSI/ANS-3.4[15.4] condition and why the no solo restriction will compensate is helpful.

• The NRC physicians do not maintain medical files on applicants/operators. Therefore, sufficient medical history/background must be contained in the Form 396 and supporting medical evidence such that any proposed license restrictions and overall conclusions relative to medical qualification for licensing are clearly supported. A brief history of medical status and changes since the last submittal (in the case of renewals) will enable the reviewing physician to make a more meaningful review of suitability for licensing within the context of the individuals overall health.

Gen.52 Archived Gen.53 Archived Gen.54 If a medical condition was previously reported for hypertension to the NRC, but the operator had a dose increase but still meets the B/P standard, do you have to report the medication dose increase to the NRC?

113

No, as stated in Question Gen.25, simply increasing or changing a hypertensive operator's medication would not normally need to be reported unless the examining physician believes the operator's blood pressure is out of control to a point that it requires more frequent monitoring or a no-solo license (i.e., the addition of license condition #6 on NRC Form 396). This answer presumes that the operators license is already conditioned to take medication as prescribed to maintain medical qualifications (NRC Form 396, Box #4), as discussed in Question Gen.21.

Gen.55 Is it required to submit every operators biennial medical examination to the NRC?

No. All licensed operators are required to have a biennial medical exam, facility licensees are required to maintain a copy of that exam and certify on the NRC Form 396 that it was performed (NUREG-1021, ES-5.3, Section C, License Renewal), but medical evidence submitted would still be in accordance with existing guidance to report relevant medical evidence needed to support a conditional license. See the NRC Form 396 instructions and Gen.50 and Gen.51 for additional information.

Gen.56 As a third-party organization, and not a government agency, are we eligible to submit verification requests for a 10 CFR 50 facility licensee?

Operator license records are protected by the Privacy Act (PA). The PA works as a companion with the Freedom of Information Act (FOIA). Information pertaining to the Freedom of Information Act/Privacy Act can be found at: https://www.nrc.gov/reading-rm/foia/foia-privacy.html.

The regulations in 10 CFR 9.80(a) state, NRC Commissioners and NRC personnel shall not disclose any record which is contained in a system of records maintained by NRC by any means of communication to any person, or to another Government agency, except pursuant to a written request by, or with the prior written consent of, the individual to whom the record pertains, unless disclosure of the record is for any of the items listed in 10 CFR 9.80(a)(1) - (12). Therefore, unless the disclosure of the record meets one of these 12 items, the third party must provide a verifiable written authorization from that person designating the third part as a representative acting on his or her behalf. Because the Privacy Act covers disclosure of records, the third party should request the applicable operator license records, such as an initial license letter and termination letter, as applicable.

Due to record retention requirements, the NRC does not maintain records for licensed operators if the operator license has been terminated for greater than 10 years.

To request operator license records, follow the instructions provided in the Privacy Act Request Guide at https://www.nrc.gov/reading-rm/foia/privacy-request.html#access to submit a Privacy Act Request with NRCs Freedom of Information/Privacy Team at FOIA.resource@nrc.gov.

114

Gen.57 Due to technological advances and the recent COVID-19 pandemic, can facilities use electronic signatures to sign the Form 398 and Form 396?

The NRC has determined that the following options meet the intent of the NUREG-1021 statement regarding electronic signature of Forms 398 and 396 by facility licensee staff:

1. Any handwritten, optically scanned signature will continue to be accepted, regardless of transmission process used (hardcopy, Part 55 EIE, or e-mail)

2. Facility licensees may use any generally recognized form of electronic signature for forms being submitted through Part 55 EIE. The digital trail created by the EIE credential registration/issuance process, EIE system login, and other system database timestamps, offers sufficient documentation of authenticity.

3. Any utility desiring to use a digital certificate to sign the documents needs to obtain specific approval from the NRC by sending in a request, detailing how the facility meets the attributes in Table 1 below. The NRC may approve use of this certificate for signing and submitting Forms 398 and 396 through either e-mail or Part 55 EIE.

4. Facility licensees may NOT use electronic signature for forms submitted through email. The only possible exception is if the facility uses a digital certificate that the NRC has reviewed and determined to be acceptable per item 3 above. The document must contain the digital certificate information (it cannot digitally sign the document and then convert the file in such a way that the digital certificate information is lost). An EIE certificate is not sufficient to send documents using e-mail.

The NRC policy related to use of electronic signatures in place of a handwritten signature is to use the electronic signatures whenever practical and logical. Electronic signature is a broad term covering a multitude of technologies and methods, from digitizing a handwritten signature to be placed in document signature blocks, to competitive corporate solutions, all the way to signing with a Personal Identity Verification (PIV) card. Digital signature is a specific type of electronic signature, created using Public Key Infrastructure (PKI) certificates, such as those encoded on government-issued PIV cards. Digital signature, especially when created by certificates residing on government-issued PIV or CAC cards is the most secure of electronic signature and is vastly superior to a handwritten signature in almost every way. It offers an incredibly high degree of non-repudiation/non-refutability.

That said, PKI certificates, especially those generated as the result of a vigorous, frequently audited, and policy-constrained identity proofing, issuance, and activation process are not popular outside of government. This is primarily due to cost and complexity, as well as the availability and general suitability of other technologies for common use cases (e.g., use of One-time Passwords for logging in, instead of a PIV card). The NRC providing suitable certificates for use cases such as this would be technically complex and frustrating for users. This means that the best type of electronic signature isnt plausible for this case, so the staff evaluated other, less ideal means to determine their suitability for this application. There are differing factors in play - the need to support efficiency and effectiveness (supports use of electronic signature) and the need for signatures that are legally-binding when necessary (does not support use of electronic signature, especially the forms that offer lower levels of non-repudiation).

115

Even low assurance signatures can have process augmentations or other controls that can increase their suitability. Much like the Defense in Depth concept, this layering effect can be applicable here as well. Anything that can help offer a digital bread crumb trail or otherwise increase non-repudiation can be considered. A specific and highly relevant example is the use of the Part 55 EIE system to submit the relevant forms. EIE users are sent through an NRC credentialing process to be issued the credential used for system log in. The system also logs user activity and has other means (such as database timestamps) that help document that a given action was initiated by a user. Even in cases where the EIE submitter is submitting documents on behalf of their company (e.g., they are not directly the signer), this still helps to create that digital chain in a significant way.

The required forms are completed by the facility licensee staff and then reviewed and signed by a senior member of the staff. They are submitted via paper (in which case consideration of electronic signature is not applicable) or electronically via email or Part 55 EIE. They are then reviewed and electronically signed by NRC staff. Some of these forms are audited for accuracy. These forms are used to form the basis for making licensing decisions, meaning that there is some importance of non-repudiation and some risk significance to the use of the forms and the electronic signatures.

Table 1. Specific Case - Required Attributes for Facility Use of Root Certificate Authority and Subservient Issuing Certificate Authority Attribute Contain unique identifying attributes about the person it was issued to (name, email address, etc.)

Have a reasonable validity period (such as one year)

Be issued by a government entity, which is subject to a certain level of cybersecurity and process scrutiny by default Have the proper key usage defined 116

Gen. 58 What kinds of measures can be established to help minimize personnel risk while conducting exams during the Coronavirus Disease 2019 (COVID 19) Public Health Emergency?

The NRCs Office of Nuclear Reactor Regulation Division of Reactor Oversight established a multidisciplined team to evaluate lessons learned and best practices during the initial phases of the COVID-19 PHE. The team prepared a report titled Initial Report on Challenges, Lessons Learned and Best Practices from the 2020 COVID-19 Public Health Emergency - Focus on Regulatory Oversight of Operating Nuclear Reactors (Agencywide Documents Access and Management System (ADAMS) Accession No. ML20308A389). As discussed in this report, considerations for conducting operator licensing exams, while accounting for the COVID-19 prevention guidance provided by the CDC and compensatory measures, were provided to all NRC examiners and are publicly available (ADAMS Accession No. ML20323A243). The compensatory measures include the following:

1. Regional management should ensure that acceptable travel options, lodging and food are expected to be available to examiners while on travel for examination activities.

2. Examination teams should comply with national, state, and local restrictions for mission critical work.

3. Minimize activities conducted onsite, in person (conduct meetings, briefings remotely as possible).

4. Consider daily health screening of applicants, licensee personnel and examiners (temperature check, symptom questionnaire), in coordination with licensee practices for minimizing COVID 19 infections.

5. Identify locations within the simulator for examiners to observe effectively with minimal interaction with applicants and training staff.

6. Provisions for public health PPE for examiners, licensee staff, and applicants; along with an assessment of use of masks/gloves on applicant performance (barrier to communications, fogged eyeglasses, etc.).

7. Coordinate with licensee for simulator, high traffic areas, and NRC examination team rooms to be cleaned/disinfected regularly.

8. Take precautions to minimize the potential spread of virus due to exchanges of paperwork during the exam. Considerations should include ways to minimize potential spread of virus from exam paperwork, while maintaining exam security requirements.

Paperwork could be isolated for a period of time, cleaned, or digitized before changing possession, for example.

9. Consider site specific contingencies where available - use of electronic tablets, binoculars, lapel microphones, headsets, etc.

10. Consider use of separate examiner crews, in separate rooms, especially if two simulators are available.

11. If possible, limit crew size for validation and limit examiners conducting in-plant JPM validation.

117

12. Regions should act in the best health interest of licensee and NRC staff in the event an applicant, licensee employee or examiner exhibits COVID19 symptoms during exam validation or administration. NRC will consider licensee practices along with NRC policies and guidance for determining response and exam path forward in these cases.

Gen. 59 Medical personnel may not perform pulmonary function tests (e.g., spirometry) during operator medical examinations due to concerns about spreading the virus that causes COVID-19. If a pulmonary function test (PFT) is not performed due to the COVID-19 public health emergency, how should the NRC Form 398 and NRC Form 396 be completed?

The answer depends on the version of ANSI/ANS 3.4, Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants, in use at the facility. Initial license applicants and operator license renewal applicants at facilities that are committed to the 1983 version of ANSI/ANS 3.4 do not need to request permission or a medical waiver when a PFT will not be performed because the PFT is not a requirement IAW the 1983 version of ANSI/ANS 3.4. Initial license applicants and operator license renewal applicants at facilities that are committed to the 1996 and 2013 versions of ANSI/ANS 3.4 should request a medical waiver on NRC Form 398 (Block 12.c.3) and follow the guidance below (and also available at ADAMS Accession No. ML20091M853) for completing the NRC Form 396.

118

Gen. 60 How can facility licensees plan for the rollout of NUREG-1021, Revision 12, specifically in regard to the integration of generic fundamentals topics in the written examination and any changes to the operating test?

The NRC staff expects to publish Revision 12 of NUREG-1021 before the end of calendar year 2021. The public comment period on draft Revision 12 closed on February 16, 2021 (for more information on the draft NUREG for public comment, refer to NRC Docket 2020-0227).

One of the proposed changes in draft Revision 12 is to discontinue the Generic Fundamentals Examination (GFE) that is described in NUREG-1021, Revision 11, Section ES-205, Procedure for Administering the Generic Fundamentals Examination Program, and reintegrate generic fundamentals topics into the site-specific written examination. To account for this reintegration of generic fundamentals topics into the site-specific examination, the Revision 12 written examination outline differs from the Revision 11 written examination outline as follows:

• There is one less K/A in Tier 1/Group 2 (RO K/A category only);

• There is one less K/A in Tier 2/Group 2 (RO K/A category only);

• There are four less K/As in Tier 3 (RO K/A category only), and the distribution of the six K/As in Tier 3 is as follows: 2 in Conduct of Operations, 2 in Equipment Control, 1 in Radiation Control and 1 in Emergency Procedures/Plan;

• Tier 2 G (generic) K/As will also include topics selected from Section 5, Components, of the applicable K/A catalog (RO K/A category only); and

• There is a new Tier 4 section on the RO portion of the examination, which consists of six K/As from Section 6, Theory, of the applicable K/A catalog; the K/A categories Reactor Theory and Thermodynamics each have three items.

(Note: Refer to Gen. 61 for example Tier 4 questions.) Under Title 10 of the Code of Federal Regulations (10 CFR) Section 55.40, Implementation, the Commission shall use the criteria in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, in effect six months before the examination date to prepare the written examinations required by 10 CFR 55.41 and 10 CFR 55.43 and the operating tests required by 10 CFR 55.45. The Commission shall also use the criteria in NUREG-1021 to evaluate the written examinations and operating tests prepared by power reactor facility licensees in accordance with § 55.40(b). The NRC staff 119

prepares the written examination outline for both NRC and licensee-developed examinations (see NUREG-1021, Revision 11, Section ES-201, C.1.f, Page 4/32). Most facility licensees request a written examination outline approximately one year before the examination administration date. Therefore, for facility licensees that have an examination date scheduled for six or more months after the publication of NUREG-1021, Revision 12, and an outline prepared using Revision 11 of NUREG-1021, the outline will need to be revised to conform to NUREG-1021, Revision 12. The NRC staff recognizes that issuance of Revision 12 could result in an increase in workload for chief examiners and facilities licensees in cases where the written examination outline was generated under Revision 11 and the examination date is six or more months after the publication of Revision 12. If the sample plan needs to be revised to conform to Revision 12, then questions may need to be replaced if they have already been developed. As a result, when the facility licensee point of contact requests the written examination outline from the NRC, NRC chief examiners have been instructed to help facility licensees that have examinations scheduled in the first half of 2022 prepare for this transition by providing the facility licensee point of contact for the examination two written examination outlines: one that conforms to the Revision 11 criteria and a second version of the same outline that has been altered to comply with the written examination criteria in draft Revision 12, Examination Standard (ES)- 4.1. If new K/As need to be selected as part of revising a written examination outline that was initially developed using Revision 11 of NUREG 1021 to conform to Revision 12 of NUREG-1021, then these K/As should be selected from the same version of the K/A catalog that was used to develop the outline.

Regarding the proposed changes to the operating test in draft Revision 12, the NRC staff anticipates that only minor adjustments to the developed operating test material, if any, will be necessary and that it is possible to develop an operating test that conforms to both the Revision 11 and Revision 12 criteria. For example, to avoid any potential need for rework, an examination author could develop critical tasks that do not credit the avoidance of RPS trips or ESFAS actuations (simulator scenarios) and ensure that each JPM has at least two critical steps. The facility licensee contact should work with the NRC chief examiner to make sure that the operating test material complies with the applicable revision of NUREG-1021 that will be in effect for the examination date. Facility licensees should know whether their NRC examination is impacted by revision change approximately six months before their scheduled examination date. This can be determined on the date that NUREG-1021 Revision 12 is published in the Federal Register, or through communications with the NRC regional office. The examination date is determined by the start date of the operating test or written examination, whichever is first.

Alternatively, facility licensees also have the option to request an exemption from § 55.40(a) in accordance with § 55.11, Specific Exemptions. Facility licensees interested in seeking an exemption should work with the assigned NRC project manager for the facility to submit the request. The facility licensee may also request the NRC project manager arrange a presubmittal meeting with the NRC staff in the Operator Licensing and Human Factors Branch (i.e., the Operator Licensing Program Office) to discuss the request before submitting the request for review.

120

Gen 61.

One of the proposed changes in draft NUREG-1021, Revision 12, is to discontinue the Generic Fundamentals Examination (GFE) that is described in NUREG-1021, Revision 11, Section ES-205, Procedure for Administering the Generic Fundamentals Examination Program, and reintegrate generic fundamentals topics into the site-specific written examination as discussed in Gen. 60. The Nuclear Energy Institute provided the results of a pilot effort to evaluate the reintegration of the GFE into the site-specific licensing examination on March 4, 2020. The following questions were developed by the facilities involved in the pilot, which consisted of two boiling water and two pressurized water facilities. These questions are provided as examples only and may or may not meet all the criteria for a written examination question at a specific facility. Facility examination authors and NRC examiners will need to continue to develop all NRC initial written examination questions, including any GFE bank questions that may be selected for use on an NRC initial written examination, that meet the criteria for written examination question development in NUREG-1021.

Example Tier 2 Question, K/A 291001, Valves, K1.11, Operation of manual valves and verification of position with indicator lights Example Tier 4 Question, K/A 193005, Thermodynamic Cycles, K1.03, Describe how changes in secondary system parameter affect thermodynamic efficiency 121

Example Tier 4 Question, K/A 192008, Reactor Operational Physics, K1.18, Describe the monitoring and control of T-ave, T-ref, and power during operation 122