ML20164A011

Operator Licensing Program Feedback

ML20164A011
Person / Time
Issue date:	06/11/2020
From:	Eric Cushing NRC/NRR/DRO/IOLB
To:
Cushing E
References
Download: ML20164A011 (157)

Text

1 Record of Changes Date Section(s) Affected Summary 06/11/2020 General New Question Gen. 57 created to address electronic signature usage on the NRC Form 396 and NRC Form 398.

04/06/2020 ES-401 Question 401.55 and its response were revised to clarify the intent and rating of Tier 1 written exam questions.

03/30/2020 General Question Gen.45 updated to reflect current process of obtaining physical copy of an RO and/or SRO license.

New Question Gen.56 created to address process of third-party obtaining license verification information.

06/13/2019 ES-401 Question 401.55 was revised to be clearer on the intent and rating of Tier 1 written exam questions that do not reference specific procedures or require procedural knowledge to answer.

06/27/2018 ES-401 Question 401.55 was revised to provide further clarification on evaluating Tier 1 questions as unacceptable or deficient based on the question failing to reference a procedure.

11/28/2017 ES-301 New Question 301.20 was added to provide clarification to Section D.5.b and what is meant by a new scenario.

11/14/2017 ES-202 and Record New Question 202.22 was added to highlight and of Changes summarize the noteworthy changes made to NRC Form 396 and NRC Form 398 as a result of the October 2017 revisions.

Organized Record of Changes table to show most-recent change at the top.

10/26/2017 ES-202 New Question 202.21 was added to provide clarification and guidance when revisions to NRC Form 396 and 398 are issued and as a result of the October 2017 revisions to NRC Form 396 and 398.

1

2 Date Section(s) Affected Summary 08/31/2017 All and General Incorporated formatting changes and hyperlinks throughout to improve ease of use. Removed FAQ terminology.

New Questions Gen. 54 and Gen. 55 were added to provide clarification on medical reporting requirements and Question Gen. 25.

Questions Gen. 24 and Gen. 25 were revised to reference the correct NRC Form 396 medical condition, i.e., NRC Form 396 Box 5 condition changed to Box 4.

08/25/2017 ES-401 New Question 401.55 was added to provide clarification on evaluating Tier 1 questions as unacceptable or deficient based on the question failing to reference an Emergency or Abnormal Operating Procedure even though the question met the K/A statement.

08/16/2017 ES-605 New Question 605.14 was added to provide clarification and interim guidance as a result of NUREG-1021, Revision 11, requiring the operators signature when a license amendment is requested.

07/07/2017 ES-205 New Question 205.6 was added to provide clarification on how the revision to ES-205, Attachment 4, GFE Test Item Distribution, as a result of NUREG-1021, Revision 11, will effect content in September 2017 GFE.

03/13/2017 ES-202 Question 202.20 revised to include discussion of latest revision to ACAD 10-001 (revised in November 2016).

Question 205.1 revised to include Revision 11 of NUREG-1021 changes (i.e., the allowance for facility licensee (other authors) to develop GFEs) and to further clarify response.

03/18/2016 ES-202 Slight clarification added to Question 202.1 to align with what is stated in 10 CFR 55.59 and NUREG-1021 for what constitutes a power change.

09/22/2015 ES-401 New Question 401.54 was added to clarify the required Technical Specification knowledge for a RO applicant 2

3 Date Section(s) Affected Summary 3/18/2015 ES-301 New Question 301.19 was added to clarify Form ES-301-5 new instruction 4 which addresses position rotation requirements for SRO-I evaluations.

9/21/2012 General New Question Gen 53 was added to clarify documentation requirements associated with minor medical conditions.

2/1/2012 Simulators This section was updated to address current practices concerning simulator scope, fidelity and performance testing.

7/12/2011 ES-401 Question 401.30 was updated to reflect additional guidance on SRO questions.

6/28/2011 ES-605 New Question 605.13 was added to clarify the no solo restriction for SROs supervising core alterations.

6/17/2011 General New Question Gen 52 was added to clarify reporting requirement for temporary medical conditions/restrictions.

4/21/2011 ES-605 New Question 605.12 was added to clarify requirements for maintaining active license status.

3/11/2011 ES-605 Question 605.3 was revised to clarify notification requirements concerning temporary suspension of licenses.

6/4/2010 ES-202 New Question 202.20 was added to address ACAD 10-001 instructions on eligibility determinations.

5/12/2010 General New Questions Gen 49 - 51 were added to provide additional guidance on NRC Form 396, Certification of Medical Examination by Facility Licensee.

1/7/2010 ES-401 Updated Questions 401.35 and 401.52 to reflect clarifications and enhancements incorporated by NUGEG-1022/1123 Rev2, Supp. 1 5/21/2009 General New Question 202.19 was added to provide guidance on repeating control manipulations and other training for license reapplications; Question 204.1 was edited to clarify the selection of GFE waiver examinations; and Gen.48 was added to address the privacy of medical information.

3/5/2009 ES-202 New Question 202.18 was added to clarify the instructions for NRC Form 398, Personal Qualification Statement - Licensee item 12a associated with eligibility waivers.

3

4 Date Section(s) Affected Summary 12/11/2008 ES-401 and General New Question 401.53 was added to address the use of open-reference questions on the initial exam and Gen.47 to address keeping required medications on-site for emergencies.

9/10/2008 ES-301, ES-605, New Questions 301.18, 605.11, and Gen.46 and General were added to address the Emergency Procedures/Plan Administrative Topic of the walk-through operating test, rehiring retired operators, and sleep apnea, respectively.

5/24/2008 General New Question Gen.45 was added to address Privacy Act record requests.

11/29/2007 ES-201 and ES-401 Question 201.35 was revised to reference new procedure freeze guidance that was added by Supplement 1 to Revision 9 of NUREG-1021.

Questions 401.53 and 54 were deleted based on Supplement 1 changes to NUREGs-1021, 1122, and 1123.

9/25/2007 General New Questions Gen.29 - Gen.44 were added to clarify medical requirements and guidance. Most of the questions were submitted during the August 2007 medical issues seminar.

2/8/2007 ES-605 Delete Question 605.10 which is no longer necessary given the 2006 revision of NRC Form 398, and replace with a new question regarding the no-solo license restriction.

8/23/2006 General New Question Gen.28 was added to further clarify medical reporting requirements.

8/3/2006 General New Questions Gen.25 - Gen.27 were added to clarify medical reporting requirements.

7/21/2006 ES-201 and ES-302 New Questions 201.35 and 302.13 were added to clarify policy regarding procedure freezes prior to initial licensing examinations and the use of surrogates during simulator operating tests.

3/10/2006 ES-401; General New Questions 401.53 - 54 were added to clarify the K/A suppression criteria in Attachment 2 of ES-401; new Questions Gen.19 - 24 were added to address a number of medical issues, including permanent disabilities and recent changes to NRC Form 396.

7/14/2005 ES-605; Simulators; New Question 605.10 was added to clarify the General need to include requalification training time on renewal applications; Sim.37 was added to clarify policy regarding design changes; Gen.18 was added to address electronic submittal of applications.

4

5 Date Section(s) Affected Summary 3/30/2005 ES-605 New Question 605.9 was added regarding no-solo license conditions.

2/29/2005 All; ES-204 All question numbers were revised to include a section designator (e.g., 401.7, Sim.2); new Question 204.1 was added.

12/9/2004 Various Revision 9 changes were marked with searchable (R9) flags.

5

6 ES-201 Initial licensing examination process; examination security 201.1 What is the time expectation for turnaround of an examination submitted for review?

Per Section C.3.e of ES-201 (NUREG-1021), chief examiners are expected to complete their review of the examination outlines within 5 working days. Section C.3.f goes on to say that the sampling review of the written exam (which is discussed in Section E of ES-401) should be completed within one week after receiving the exam and the entire review should be done within two weeks. Facility licensees are encouraged to discuss their specific schedule requirements and expectations with their chief examiner.

201.2 Is the request for NRC to write the examination required in writing?

Yes. Section 10 CFR 55.40(c) of the amended rule states that the Commission shall prepare the examination upon written request from the power reactor facility licensee pursuant to

§55.31(a)(3). It has to be a corporate decision with a formal request in writing signed by an authorized facility representative. As stated in Section C.1.a of ES-201, a written response to the NRC's annual letter soliciting examination schedule information (e.g., RIS 2003-14) will satisfy this requirement.

201.3 Can the utility write part of the examination and the NRC write the other part of the examination? How do you work the "split exam" concept? How can you maintain NRC examiner proficiency if developing "split exams?"

Yes. Allowing the facility licensee and its NRC Regional Office to split responsibility for exam development provides both parties with greater flexibility in scheduling their resources. For example, the Regional Office might be able to support an examination on a specific date if it only has to prepare the written exam or the operating test, but not both.

The desire to split an exam should be reflected in the facility licensee's response to the NRC's annual letter soliciting examination schedule information (e.g., RIS 2003-14) and coordinated with the appropriate NRC Regional Office.

Keep in mind that each Regional Office is still required to prepare one complete examination per year to maintain examiner proficiency, but it can do the written portion of one examination and the operating test on another.

6

7 201.4 The utilities should NOT be the ones to develop the sample plan. This should be developed by the NRC for all examinations administered in the region.

Comment noted. Some facility licensees may prefer to develop their own sample plan. Facility licensees can make arrangements to split responsibility for developing various parts of the examination with the NRC Regional Office. This approach should be reflected in the facility licensee's response to the NRC's annual letter soliciting examination schedule information (e.g.,

RIS 2003-14) and coordinated with the appropriate NRC Regional Office.

201.5 Would you comment on the following proposal? Have a "team" from the utility come to the region and work directly with the chief examiner to develop the written exam. I would propose that a team of experienced utility instructors could bring the exam bank and associated reference material and they, with the chief, could produce the written exam in less than 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br />. Benefits - lower man hours cost, reduced security concerns (less time on site), fewer negative exam report comments.

The NRC currently does not believe that this is a viable option because it raises concerns regarding independence, accountability for the quality of the final product, and possible adverse public perception.

201.6 A question has come up on the issue of using the same utility examiners to write the initial exam and the audit exam. What are the requirements for this? If you use independent groups to develop an audit examination and an NRC examination, do you have to worry about overlap? Why?

As stated in Section D.2.b of ES-201 (NUREG-1021), individuals who are on the security agreement may prepare the audit examination, but the examination would be subject to review by the NRC for test item duplication (none is allowed unless the examinations are independently developed).

Note that ES-401 has eliminated the limits on written examination overlap based on the random selection of specific K/A statements and strict adherence to the intent of the selected statements. However, the facility licensee still has to take measures to ensure that the final audit or screening examination and any quizzes that are given after beginning work on the licensing exam do not compromise the integrity of the licensing exam. Section C.1.f of ES-401 provides examples of acceptable control measures, which include the use of independent teams to develop the examinations.

201.7 Should the utility NRC exam writer be "certified" by the NRC?

No. Although the NRC has considered that and other ways to improve the training and qualifications of utility examination authors, there are no current plans to implement such a program.

7

8 201.8 If the NRC writes the outline, does the facility licensee have to track the question history if the facility licensee writes the examination?

NUREG-1021 eliminated the limits on written question repetition from quizzes given during the training program, thereby eliminating the need to track question histories. However, as stated in Section C.1.h of ES-201, facility licensees are encouraged to identify those questions that were used on an NRC license examination at the facility since October 1995 because they will generally undergo less rigorous review by the NRC.

201.9 Does "independent review" by a supervisor include question-by-question approval/comment?

Yes. The independent managerial or supervisory reviewer is confirming and signing that the written examinations and operating tests meet the requirements of NUREG-1021. The extent of the review will typically be a function of the experience of the examination author and the quality of facility's examination bank.

201.10 If a reactor operator is testing for an upgrade and his/her physical is current, does he/she have to have another physical?

No. In accordance with Section D.1.c of ES-204, the medical examination documented on NRC Form 396 is good for two years from the date of the medical examination. Per 10 CFR 55.25, facility licensees are required to notify the NRC within 30 days of learning that a licensed operator has developed a permanent physical or mental condition that causes the operator to fail to meet the eligibility requirements.

201.11 Why does the NRC not have to sign a security agreement?

The primary purpose of the security agreement is to prevent inadvertent compromises by ensuring that the people having knowledge of the examination content are aware of their responsibilities. NRC examiners are aware of their responsibilities with regard to examination security and rarely find themselves in a position where they could inadvertently compromise the examination. They are only on-site to validate and administer the examinations and they do not routinely interact with the license applicants.

8

9 201.12 ES-201 (of NUREG-1021), Section D.2.b, Bullet #2, prohibits someone on the exam security agreement from doing on-the-job training (OJT), practice, coaching, and sign-offs. Does this prohibit an operator (on exam security) who is standing a regularly scheduled shift from signing off a trainee scheduled to stand that shift under instruction in the position? This is not referring to signing of individual OJT tasks, just the shift itself. (We currently do not permit this, I just want to be clear on the requirements of the examination standard).

When the operator comes out to validate the written, can they have OJT contact with an applicant after the operator is on the security agreement?

Section D.2.b of ES-201 prohibits all OJT activities. A license applicant should not be standing watches under instruction with, or receive OJT sign-offs from, a licensed operator who has knowledge of the examination content.

201.13 Why does ES-201, Section D.2.b, Bullet #1, permit a person signed onto the initial exam security agreement to operate the simulator from the booth when this is not permitted in ES-601 for requal? Why the inconsistency?

This inconsistency, which resulted from an oversight during the development of Revision 8 of NUREG-1021, has been corrected. The security restrictions on Form ES-601-1 are now the same as in ES-201.

201.14 Why do the standards not allow the utility to give the same JPMs and scenarios the following day if the applicants sign a confidentiality agreement? If an individual examinee is on security agreement, can you then reuse a JPM set?

The NRC takes examination security very seriously, and prohibiting the reuse of test materials is the most effective way to minimize the risk of compromising an examination. No.

9

10 201.15 Although some relaxation was included in final Revision 8 of NUREG-1021, it is still much too restrictive (in my opinion). Why is it that an instructor cannot teach once he has knowledge of the exam? This requirement causes me to need additional staffing because once he has knowledge of sample plan, he is not available. Why can't we use the instructor, and rely on his integrity (via signature, under penalty of law, etc.)?

What is it going to take to use the instructor in both the exam development process and in candidate instruction/supervision?

While developing the current examination process, the NRC identified a number of vulnerabilities (including independence and public perception, examination security and integrity) associated with allowing facility licensees to prepare the initial licensing examinations, which had, theretofore, been prepared exclusively by NRC examiners or contractors. To the extent possible, the NRC established guidelines and criteria in ES-201 of NUREG-1021, including the personnel and security restrictions, to mitigate the vulnerabilities. Please refer to SECY-96-206 (the rulemaking plan) and SECY-98-266 (the final rule) for a discussion of the NRC's rationale. It should be noted that the current restrictions are consistent with the change recommended by the Nuclear Energy Institute (NEI) during the rulemaking process.

Although ES-201 clarifies that supervisors can counsel applicants regarding non-technical issues, direct training activities are still prohibited. There is some flexibility to address unique situations on a case-by-case basis; however, a generic change in policy is unlikely unless the industry can adequately address the NRC's concerns regarding public perception and confidence.

201.16 Providing individual applicant feedback is a prohibited activity for individuals on the security agreement. How does this apply to Manager/Supervisor situations such as sitting on a performance review committee or coaching/counseling associated with a non-technical situation (e.g. classroom behavior)?

Managers/supervisors on the security agreement may continue to counsel the applicants concerning non-technical issues. However, as stated in Section D.2.b of ES-201 of NUREG-1021, they are not allowed to provide any technical guidance, training, or any other direct feedback that may compromise examination integrity as defined in 10 CFR 55.49.

10

11 201.17 Is a facility required to check with a contractor to determine if they are concurrently developing a similar exam for another utility? If so, do these exams need to be given on the same day? Also, what other security requirements need to be met?

If you have a common group develop examinations for two different plants, do you have to worry about overlap between these exams? What are the criteria?

Pursuant to 10 CFR 55.40(b)(2), facility licensees that prepare their own examinations are expected to take reasonable measures to control examination security and integrity. As noted in Section C.1.d of ES-201, facility licensees may use contractors or other outside assistance to develop the examinations, but the licensees bear full responsibility for the product, including conformance with the examination criteria and maintenance of examination security and integrity. Additionally, Section C.1.h of ES-201 (in NUREG-1021) discusses the requirements for controlling and documenting the source of test items and the predictability of the examination content. Licensees should obtain this information from their examination contractor if one is used. If there is a basis for the applicants to predict the content of the examination and the overlap with the other utility's examination is significant, then the utility must evaluate the issue, determine if compensatory measures are appropriate, and discuss the issue with the NRC as early as possible. Factors to consider would include the timing between the exams and the physical and corporate distance between the facilities. For example, this evaluation could reasonably differ if, in one case, the sites are owned by the same utility, located 20 miles apart, and the exams are separated by a month, versus another case in which the exams are 8 months and 2000 miles apart.

201.18 As part of normal instructor duty, 10 questions were submitted to an examination team.

Does the instructor have any examination information?

As long as the instructor is not aware if any of the questions meet the sample plan and the questions are placed in the exam bank, then the instructor would not be considered to have exam information. However, if the questions are given to the examination team with the expectation that they will be used as new questions, then the instructor should be on the security agreement. Specific questions regarding this issue should be discussed with the NRC.

11

12 201.19 If involved in an initial examination, is there a restriction from teaching requal?

An initial licensed operator upgrade candidate attends licensed operator requalification training with his crew. The instructor is on the initial NRC exam team and has signed the exam security documents. Is the initial NRC exam candidate allowed to remain in the class/simulator or must he/she leave?

Use of instructors is still an issue. The use of an instructor, who is on the exam security agreement, can't teach candidates attending the requalification program. This is an unnecessary burden on resource restrictions.

SRO upgrade applicants who are removed from the watch rotation do not have to attend RO requalification training while they are training for the SRO license. If there are no upgrade applicants in the requalification class, there would be no restriction on the instructors. However, as stated in Section D.2.b of ES-201 (NUREG-1021), if SRO upgrade applicants are present in the class, instructors would not be permitted to teach in areas in which they have examination knowledge, and their activities would have to be documented on Form ES-201-3. They can teach subjects about which they have no examination knowledge, which is a good reason to limit everyone's access to only those portions of the exam for which they have responsibility.

Instructors with examination knowledge should not be used in training environments that require one-on-one contact with trainees. There is no problem with them teaching a requalification lecture or simulator session, but the trainer with examination knowledge must avoid direct individual interaction with the applicants.

201.20 Is it acceptable to password protect exam files and leave them on a local area network (LAN) or password protect them on a hard drive? (The concern is that floppy disks are more susceptible to damage).

Yes. As stated in Attachment 1 of ES-201 (in NUREG-1021), the use of passwords should provide adequate security if normal computer security practices (e.g., selecting and changing passwords) are observed. Special cases may need additional consideration. For example, if a trainee has extended access to the LAN in his normal position, additional security measures might be appropriate.

201.21 Will you allow transfer of electronic files of exam materials over the Internet via e-mail if the file is "password protected?"

As stated in Attachment 1 of ES-201 (in NUREG-1021), examinations shall not be transmitted via non-secure electronic means. Licensees may transmit the exams via the NRC's "AUTOS" local area network by making arrangements with the NRC resident inspector at the facility.

Licensees may also transmit password-protected electronic files over the Internet if the licensee's word processing software provides adequate security and is compatible with the NRC's and the password is separately provided to the NRC chief examiner by mail or phone.

The files do not need to be encrypted.

12

13 201.22 If the examination is password protected, how much hacking do we have to protect against?

Pursuant to 10 CFR 55.49, the NRC expects facility licensees to take reasonable measures to prevent inadvertent examination compromises. Attachment 1 of ES-201 describes a number of examination security guidelines that facility licensees may consider. The NRC does expect reasonable computer security measures to be in place, but it does not expect facility licensees to defend their examinations against willful acts, such as computer hacking.

201.23 The person who issues the password and knows what it is for a computer system - is he in possession of examination material?

Although the people who issue computer passwords may not have possession of examination material, they probably have access to that material and any other sensitive or classified information stored on that computer system. These individuals should be aware of their authority and responsibility with regard to accessing and safeguarding sensitive information.

There would certainly be no harm in having them sign the examination security form.

201.24 What are the time frames when security restrictions begin?

The security restrictions begin whenever someone makes the first decision regarding the topics to be tested on any part of the licensing examination.

201.25 When does someone have to go on examination security?

Per Section D.2.b of ES-201 (in NUREG-1021), they must acknowledge their security responsibilities by reading and signing the security agreement (Form ES-201-3) before they obtain detailed knowledge of any part of the examination.

201.26 If an applicant fails a section of a licensing examination that was developed using one revision of NUREG-1021 and applies for a partial retake examination after the next revision of the NUREG has been issued, what version of the NUREG will be used to prepare the retake examination?

The decision would be based on maintaining continuity in examination content and format. If there is essentially no change in the content and format of the exam between the two revisions of NUREG-1021, it makes no difference which version is used, and it generally makes more sense to use the current version, especially if other applicants will be taking the entire examination. However, if the format or content of the exam has changed substantially (as it did when the prescripted JPM questions were deleted in Revision 8) it might make sense to administer the exam using the older format (e.g., if missed prescripted questions contributed to the failure). In summary, the NRC would default to the new standard, unless there is a logical basis to stick with the previous version.

13

14 201.27 Is there a "hard-limit" to the number of people that can sign in on a security agreement?

No. Section D.2.a of ES-201 of NUREG-1021 outlines the expectations in this regard.

201.28 If an exam compromise is suspected, are the examiners expected to leave the site?

No. In accordance with Section C.3.b of ES-201 (in NUREG-1021), examiners must immediately report any perceived compromise to the responsible regional supervisor so that the necessary actions can be taken to restore the integrity of the examination. Per Section C.2.k, those actions might include not giving the exam, making additional changes to the exam, voiding the results if the exam has already been given, reevaluating the licensing decisions pursuant to 10 CFR 55.61(b), and possibly the imposition of enforcement action. It is much easier to determine the most appropriate action if the examiners remain on-site to assess the situation. The final course of action would be determined in collaboration with regional management and the NRR operator licensing program office.

201.29 Why doesn't the NRC have additional staff to support emergent utility exam needs?

Writing of exams is not voluntary because of resource restraints. What is the NRC doing about it?

The NRC staff does budget some additional resources for retake examinations, but the NRC's Congressional budget allocation does not permit us to maintain a dedicated corps of examiners capable of handling every conceivable peak work load. That is why it is sometimes necessary for licensees to develop their own examinations (which require fewer NRC resources) or to shift their examinations (usually no more than a few weeks) to a time when NRC resources are available. The NRC does have a limited pool of former examiners to draw upon in response to utility examination needs. However, those individuals' primary responsibilities in their current positions generally have priority so they are not always available on short notice. The operator licensing program office is currently working with the regions in an effort to share resources, when possible, to satisfy regional peaks in examination demand.

201.30 If a utility does not have enough staff to write an ILO [initial licensed operator] exam, is it better to have a vendor or the NRC write the exam?

This is a decision that facility management will have to make based upon cost, resource availability, scheduling flexibility, and other factors. The chances of getting an exam at a specific time are best if the licensee (or its vendor) prepares it.

201.31 What are the final Rev. 8 [of NUREG-1021] and supplemental security relaxation benefits?

Final Revision 8 of NUREG-1021 removed all restrictions on who can write the initial operator licensing examinations. However, the NUREG still requires anyone who has knowledge of the 14

15 examination contents to sign a security agreement and refrain from most training-related activities involving the license applicants. Refer to Section D.2 of ES-201 for the details.

Additional changes in Supplement 1 further clarified the types of training-related activities that managers and supervisors can perform once they have knowledge of the examination contents.

201.32 Why does anyone feel that we got what we asked for when Virginia Power requested that utilities be able to write and administer the exams?

Because the August 30, 1994, letter from Virginia Power made five process recommendations, including NRC administration of the operating tests and written examinations, all of which have been adopted in the revised examination program.

201.33 Can the initial license exam author or an exam team member provide difficulty ratings for weekly written quizzes given to an initial license class? There is no contact with the class and no direct feedback. Operations and Training Management use the difficulty ratings to gauge student progress.

The NRC takes examination security and integrity very seriously. However, based on your assertion that the raters would have no contact with the class and no direct feedback and that the difficulty ratings would only be used to gauge student progress, there should be no problem with your proposal.

201.34 Is it acceptable for a dedicated, locked examination security room to have a ceiling with removable tiles or does it need to have a hardened ceiling to be considered sufficient for exam security purposes?

The NRC expects facility licensees to take reasonable measures to prevent inadvertent examination compromises. Attachment 1 of ES-201 of NUREG-1021 describes a number of examination security guidelines that facility licensees may consider, but it does not address the need for hardened examination development facilities. If the examinations are prepared in a hardened room with no drop ceiling and a decent lock on the door, then the authors could probably leave the exams lying about the room without much worry. However, if the exam room has a drop ceiling that someone could easily crawl over, then the authors should probably consider locking the exam materials in a file cabinet when the room is going to be unoccupied for a considerable period of time (e.g., nights and weekends) and there is a possibility that someone could crawl over the wall undetected (e.g., the exam room is in an isolated part of the building). A room with a locked door would likely provide sufficient protection for an exam left on the desk while the author goes to the rest room, even if the ceiling contains removable tiles.

Licensees need to exercise common sense and decide for themselves how much they want to spend to maintain examination security and how much risk and expense they can tolerate if an exam is compromised.

15

16 201.35 Are there any requirements set by the NRC as to when Operation's training freezes procedure changes prior to an NRC initial licensing examination? Can a plant freeze multiple procedures at different times based on the scope of the procedure change and how its implementation date affects examination development and administration?

Guidelines regarding the freezing of plant procedures in advance of an initial operator licensing examination have been added to NUREG-1021 as part of Supplement 1 to Revision 9, which was published in October 2007. Section C.2.c of ES-201 now specifically requires the topic to be discussed when confirming the examination arrangements, and Attachment 2 of the same ES provides some general guidance and cautions.

16

17 ES-202 How to apply for a new license; eligibility; training; experience; reactivity manipulations; medicals 202.1 Significant reactivity manipulations were defined in the Q&A portion of NUREG-1262.

The information notice issued a couple/three years ago seems to conflict with NUREG-1262. An answer to what is a significant manipulation should support NUREG-1262.

Reactivity manipulations for [initial licensed operator] ILO training: What is the status of allowing simulator manipulations (when unable to perform in-plant)? Also, define what constitutes a control manipulation. Why is a rod operability surveillance ok at one plant but not another? What constitutes a large change?

What is acceptable for reactivity manipulations? (any real-life examples of problems or rejected applications)

What are the criteria for doing reactivity manipulations on the simulator?

Information Notice 97-67, "Failure to Satisfy Requirements for Significant Manipulations of the Controls for Power Reactor Operator Licensing," restated and clarified the NRC's position on this issue. The staff does not believe that the IN contradicts the guidance in NUREG-1262.

Effective on November 16, 2001, 10 CFR 55.31(a)(5) was revised to allow the use of plant-referenced simulators to conduct the required control manipulations. Facility licensees that propose to use a plant-referenced simulator to perform the control manipulations must ensure that simulator fidelity has been demonstrated pursuant to 10 CFR 55.46(c).

The same test (e.g., started at a comparable power level, including a comparable number of rods, and a comparable reactivity change) should be acceptable on either plant. Without specifics, it is not possible to speculate why one was acceptable and the other was not.

10 CFR 55.31(a)(5) requires five "significant" control manipulations, and 10 CFR 55.59(c)(3)(i) provides a number of examples (which are not requirements). Per Example F, and as noted in IN 97-67, a power change of at least 10% is an example of a significant (or large) control manipulation. It would also be acceptable, when defining allowed reactivity manipulations, to evaluate the knowledge and abilities exercised in a controlled large evolution and then accept all smaller tasks that comparably exercise the same knowledge and abilities. The NRC expects such evaluations to be formally documented as part of the licensee's SAT-based (systematic approach to training) program.

The criteria for doing the 10 CFR 55.31(a)(5) reactivity manipulations on the simulator are discussed in SECY-99-225, the staff paper that forwarded the associated rulemaking plan to the Commission for approval and SECY-00-0083, the proposed rulemaking paper, which was issued on April 12, 2000. Facility licensees that propose to use a plant-referenced simulator to perform the control manipulations required by 10 CFR 55.31(a)(5) must ensure that simulator fidelity has been demonstrated pursuant to 10 CFR 55.46(c). Control manipulations performed on the plant-referenced simulator may be chosen from a representative sampling of the control 17

18 manipulations and plant evolutions described in 10 CFR 55.59(c)(3)(i)(A-F), (R), (T), (W), and (X), as applicable to the design of the plant for which the license application is submitted.

As discussed in Section C.1.c of ES-202 (in NUREG-1021) power changes (10 CFR 55.59(c)(3)(i)(E) and (F) only) that are performed on the simulator must be 10% or greater in magnitude, while those on the plant may be smaller (to limit unnecessary transients on the facility) but of sufficient magnitude for the operator to experience appropriate feedback (i.e.,

clearly observable effects on the plant) as a result of the control manipulation.

202.2 Can a candidate enrolled in a reactor operator initial license training program receive credit for significant control (reactivity) manipulations performed in the control room as the Balance of Plant (BOP) operator? For example, can the following manipulation, 10 CFR 55.59(c)(i)(C), be performed as BOP? Manual control of steam generators or feedwater or both during startup and shutdown.

A related question is: Do Direct SRO candidates (i.e., instant SROs) have to perform the manipulations as ROs to get credit, or can they supervise them as SROs (i.e., procedure readers) to get credit?

10 CFR 55.31(a)(5) requires that an applicant provide evidence that the applicant, as a trainee, has successfully manipulated the controls of either the facility for which a license is sought or a plant-referenced simulator that meets the requirements of 10 CFR 55.46(c). At a minimum, five significant control manipulations must be performed that affect reactivity or power level. Control manipulations performed on the plant-referenced simulator may be chosen from a representative sampling of the control manipulations and plant evolutions described in 10 CFR 55.59(c)(3)(i)(A-F),(R),(T),(W), and (X) of this part, as applicable to the design of the plant for which the license application is submitted.

Therefore, two criteria drive the requirements for the five control manipulations, they must be significant and must affect reactivity or power level. "Manual control of steam generators or feedwater or both during startup and shutdown" is only sufficient to meet those two criteria if the licensee can clearly show that the manual control was significant and noticeably affected reactivity or power level. There is no requirement for the control manipulations to be completed in the RO watch position, so any manipulation done in the BOP watch station would qualify as long as it meets the requirements discussed above.

With regard to direct, or instant, SRO applicants, the control manipulations must be done in either the RO or BOP positions (i.e., hands-on); supervising another operator performing the manipulations would not be acceptable.

Keep in mind, as noted in Revision 2 of Regulatory Guide 1.8, "Qualification and Training of Personnel for Nuclear Power Plants," that every effort should be made to have a diversity of reactivity changes for each applicant. Moreover, in keeping with the definition of "Controls" in 10 CFR 55.4, it is preferable that the required manipulations focus on those apparatus and mechanisms that directly affect the reactivity or power level of the reactor (e.g., control rods, boration/dilution, and turbine load for a PWR; control rods and recirculation flow for a BWR).

After all, in accordance with 10 CFR 50.54(i), those are the only apparatus and mechanisms 18

19 (i.e., controls) that can be manipulated exclusively by operators and senior operators licensed (or in training for a license) pursuant to 10 CFR 55.

202.3 Does maintaining power constant at 1-2% and diluting 1000 pcm due to xenon over a shift count as a reactivity manipulation?

Yes. Although this example does not precisely fit any of the items in 10 CFR 55.59(c)(3)(i), it would be acceptable to count as one of the five required reactivity manipulations. As noted in Regulatory Guide 1.8, Revision 2, every effort should be made to have a diversity of reactivity changes for each applicant. See Question 202.2 for more information.

202.4 Can a reactor startup below the point of adding heat constitute a manipulation?

What constitutes "significant?"

What is the current position on diversity; e.g., can 5 power changes using boration be used?

Yes.

As indicated in Information Notice 97-67, "Failure to Satisfy Requirements for Significant Manipulations of the Controls for Power Reactor Operator Licensing," and defined in 10 CFR 55.59(c)(3)(i)(E), a 10 percent or greater power change is an example of a significant control manipulation.

As stated in the IN and Regulatory Guide 1.8, Revision 2, diversity of control manipulations is expected but not required. Similarly, if the training program is developed using a systematic approach, it would seem inappropriate to conduct the same control manipulation five times.

Some diversity is better than none; i.e., the 5 boration power changes should be as diverse as possible. See Questions 202.2 and 202.3 for more information.

202.5 Does the 1-year waiver clock start at the time the denial is received from the NRC following the exam or does it start after all appeals have been resolved?

As stated in Section D.1.a of ES-204 (NUREG-1021), the 1-year waiver clock starts on the date when the original examination was completed.

202.6 We believe an applicant meets the eligibility requirements, but ask the NRC to evaluate this to make sure - is this a waiver request?

No. It would not constitute a waiver request until you submit a license application (NRC Form-398) that specifically requests a waiver of the eligibility guideline or requirement.

19

20 202.7 If a utility is preparing an examination per NUREG-1021, is it required to comply with ES-202, Section D (license eligibility requirements), which is based on Regulatory Guide 1.8?

When verifying entry level prerequisites for a candidate, do I have to validate them to the requirements stated in ES-202? If not, to which standard must the candidate be validated against? If I have a SAT [systematic approach to training] based program, why is the NRC concerned about entry level verification? This renewed interest appears to contradict the information in NUREG-1262.

No. Participation in the examination development does not affect the facility licensee's prior commitments regarding license eligibility (i.e., experience, education, and training). As always, the NRC expects facility licensees to comply with their commitments; if a licensee has made conflicting or contradictory commitments, it would generally be held to the more conservative or restrictive obligation.

Refer to Regulatory Issue Summary (RIS) 2001-01, "Eligibility of Operator License Applicants,"

for a discussion of this issue. Also note that, in May 2000, the NRC issued Revision 3 of Regulatory Guide 1.8, which endorses ANSI/ANS-3.1-1993, and that ES-202 has been updated to reflect this change.

202.8 Can self-study hours be counted on the application as part of the required 500 training hours?

As a general rule, self-study time should NOT be used as a substitute for classroom instruction time that is specified in a facility licensee's approved (i.e., accredited) training program and licensing basis. However, if the licensee's program includes provisions for waivers and equivalence determinations, it may be appropriate to customize an individual's training based on prior instruction and experience. Such a program might include independent study with specific learning objectives and follow-up testing to ensure that the learning objectives have been mastered.

20

21 202.9 What are experience requirements for SRO/RO?

In accordance with 10 CFR 55.31(a)(4), an applicant must provide evidence that he or she has successfully completed the facility licensee's requirements to be licensed as an operator or senior operator. The facility licensee's requirements, as embodied in its licensing basis (e.g., its technical specifications, quality assurance plan, and final safety analysis report) and approved training program, should be clearly defined and consistent. Pursuant to SAT-based (systematic approach to training) principles, the NRC expects the facility licensee to formally evaluate and document the applicants' training and experience vis-a-vis its requirements and commitments.

Refer to Regulatory Issue Summary (RIS) 2001-01, "Eligibility of Operator License Applicants,"

for a detailed discussion of this issue. Also note that, in May 2000, the NRC issued Revision 3 of Regulatory Guide 1.8, which endorses ANSI/ANS-3.1-1993, and that NUREG-1021, Section ES-202, has been updated to reflect this change.

202.10 For a [systematic approach to training] SAT-based program, what and where are the requirements for "responsible power plant" experience?

What are the real requirements if you have SAT- based program?

Refer to Regulatory Issue Summary (RIS) 2001-01, "Eligibility of Operator License Applicants,"

for a detailed discussion of the NRC's current guidelines regarding the training and qualification of licensed operators. Also note that, in May 2000, the NRC issued Revision 3 of Regulatory Guide 1.8, which endorses ANSI/ANS-3.1-1993, and that NUREG-1021, Section ES-202, has been updated to reflect this change.

21

22 202.11 Regarding the 6-months on-site experience requirement:

- ANSI allows 13 weeks on-shift training to count toward the 6 months

- ANSI allows simulator training to count (simulator training is usually 3 or more months)

Can training program provide the 6-months of on-site experience?

What is "responsible power plant experience?" Need a definition that is broader than staff engineer and operator? For example, operations instructor, ex-NRC examiner, and maintenance supervisor.

"Responsible" power plant experience - This issue needs to be resolved; INPO, NRC, NEI need to determine the specifics and let us know. We need to know without reservation that SRO-instant candidates meet this ambiguous "experience" requirement prior to them entering a license class.

Responsible Power Plant experience acceptance needs to be explicit. For example, why does an NRC Resident or Water Treatment power plant engineer receive one for one credit while a licensed simulator instructor or plant equipment operator receives no credit?

As noted in Section D of ES-202, the NRC considers training and experience to be separate aspects of license eligibility. Per NUREG-1262 (Question No. 113), a person should meet the experience guidelines before entering the license training program. Time spent in training before entering the license training program may qualify as experience, but time spent in a training program leading up to license application (including the time spent on-shift and in simulator training) should normally not be double-counted as experience.

Refer to Regulatory Issue Summary (RIS) 2001-01, "Eligibility of Operator License Applicants,"

for a detailed discussion of the NRC's current guidelines for the qualification and training of licensed operators. Also note that, in May 2000, the NRC issued Revision 3 of Regulatory Guide 1.8, which endorses ANSI/ANS-3.1-1993, and that ES-202 has been updated to reflect this change.

As stated in the Executive Summary of NUREG-1021, facility licensees are encouraged to resolve any applicant eligibility questions with their NRC Regional Office before commencing a license training class. Pursuant to SAT-based (systematic approach to training) principles, the NRC expects facility licensees to formally evaluate and document their applicants' training and experience vis-a-vis the facility's requirements and commitments. As discussed in Section D.2.a(4) of ES-202, the NRR operator licensing program office will assess the eligibility of equipment operators, plant technicians, and non-degreed licensed operator instructors, who do not satisfy the strict definition of RNPPE and might otherwise be disqualified, on a case-by-case basis to determine the amount of credit to be granted.

202.12 Can a 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> reactivity change be counted towards the needed on-shift time? Can a four hour evolution be counted if the applicant attends all prerequisites and post-activities?

22

23 Per 10 CFR 55.31(a)(4), license applicants must provide evidence that they have successfully completed the facility licensee's requirements to be licensed as an operator or senior operator.

The NRC's regulations and guidance documents do not specify how to count the 3 months of on-shift time. However, if the facility licensee's accredited training program or other commitments (e.g., its final safety analysis report or technical specifications) provide such guidance, then the NRC would expect the facility and applicant to comply. Since the intent of this training is for the applicant to experience the full range of routine, day-to-day shift activities, the NRC would expect, in the absence of a contradictory facility requirement, that the training would be accomplished in full-shift increments.

202.13 Can the 6-months on-site power plant experience occur prior to a break in service (e.g.,

the individual works on-site for over 6 months in a responsible position; he/she then leaves the site and returns some time later. Is the 6 months satisfied already?)

Per 10 CFR 55.31(a)(4), license applicants must provide evidence that they have successfully completed the facility licensee's requirements to be licensed as an operator or senior operator.

The NRC's regulations and guidance documents do not specify when the 6 months of on-site experience needs to take place. However, if the facility licensee's accredited training program or other commitments (e.g., its final safety analysis report or technical specifications) prohibit a break in service, then the NRC would expect the facility and applicant to comply.

202.14 Can a facility be committed to ANSI N18.1-1971 for candidate eligibility, yet incorporate guidance of ES-202/RG-1.8 or other document(s) without changing the committed document?

In 1987, Generic Letter 87-07 (which was issued in connection with a revision to 10 CFR 55) gave facility licensees the option of substituting an accredited training program for their initial and requalification training programs previously approved by the NRC. As discussed in response to Question 202.7, most facility licensees elected this option in writing, but some of them neglected to revise the training program descriptions in their technical specifications, final safety analysis reports, and other documents. As a result, some facility licensees have conflicting and contradictory training program commitments and requirements.

Refer to Regulatory Issue Summary (RIS) 2001-01, "Eligibility of Operator License Applicants,"

for a detailed discussion of the NRC's current guidelines for the qualification and training of licensed operators. Also note that, in May 2000, the NRC issued Revision 3 of Regulatory Guide 1.8, which endorses ANSI/ANS-3.1-1993, and that ES-202 (in NUREG-1021) has been updated to reflect this change.

202.15 Can a "program" be split as follows: Complete phase 1 which concludes with a GFE; then suspend the program so that the trainees can get 6-months onsite experience; then restart and complete the program and get a license.

Possibly. The NRC does not require the site-specific training to begin immediately after taking the generic fundamentals examination. However, the NRC does expect facility licensees to 23

24 comply with their licensing basis requirements and commitments regarding licensed operator experience and training. Also, note that, beginning with Revision 9 of NUREG-1021, applicants must satisfactorily complete the GFE within 24 months before the date of license application.

202.16 Can we eliminate [the] hours of operation on [NRC Form] 398 [for license renewal applications]?

The requirement to supply that information is contained in 10 CFR 55.57(a)(3). The only way it could be eliminated from the form is by amending the regulation or requesting an exemption.

This issue was also raised in connection with a recent extension request for the Office of Management and Budget (OMB) Clearance covering 10 CFR Part 55. The NRC staff is reassessing the need to collect this information and will consider eliminating the requirement the next time it undertakes an administrative revision to 10 CFR 55. In the interim, the staff has revised NRC Form 398 to minimize the record-keeping burden by establishing three broad ranges (i.e., less than 100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br />, between 100 and 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br />, and more than 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br />) from which renewal applicants can select.

202.17 For Revision 9 of NUREG-1021, expand the detail requirements for people who had a license at the unit and dropped it longer than 2 years ago. NUREG-1021 covers initial, upgrade and less than 2 years, but not in between.

The regulations (specifically 10 CFR 55.47) allow a waiver of the operating and written test if the applicant had extensive actual operating experience at the facility or a comparable facility within the last two years. After two years the applicant must take the full license examination or request and justify an exemption. The NRC currently has no plans to change this aspect of the regulation.

24

25 202.18 The instructions for NRC Form 398, Personal Qualification Statement - Licensee, state that [c]hecking YES in item 12.a indicates that you have completed a SAT-based training program that is accredited by the National Nuclear Accrediting Board and meets the education and experience requirements outlined by the National Academy for Nuclear Training in its current guidelines for initial training and qualification of licensed operators. How should item 12.a be checked if an applicant is requesting an eligibility waiver in item 4.f because he or she has not completed all the training, education, or experience requirements?

With regard to training requirements, facility licensees have the mechanisms and authority within their accredited training programs to evaluate, grant, and document waivers on a case-by-case basis. Granting such a training waiver would not preclude the applicant/facility from checking YES in item 12.a because the applicant has still completed the facilitys SAT-based training program. This long-standing policy is explained in response to Question #103 in NUREG-1262, Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators Licenses.

If an applicant requests an eligibility waiver by checking item 4.f on NRC Form 398, the instructions clearly direct the applicant/facility to enter a detailed explanation/justification in item 17, the Comments section of the license application. Moreover, the instructions for item 12 indicate that checking YES in both items 12.a and 12.b eliminates the requirement to complete items 13 and 15 unless the applicant is taking an exception or waiver from the education and experience requirements outlined by the National Academy for Nuclear Training, which must be explained in item 17. Given that the NRC reviews the information on the application in its entirety when making eligibility determinations, the most important consideration is to ensure that the information in items 4.f, 12.a, 15, and 17 is consistent. If item 4.f is checked, then there needs to be an explanation in item 17, regardless whether item 12.a is checked YES or NO.

Checking NO in item 12.a would not disqualify a license applicant as long as there is a satisfactory explanation/justification in item 17.

The NRC will attempt to clarify the instructions when NRC Form 398 is updated and the associated OMB clearance (No. 3150-0090) is renewed in the fall of 2009. Facility licensees are encouraged to provide comments in response to the Federal Register notice (FRN) that will solicit feedback on the OMB clearance.

25

26 202.19 If an operator fails a license exam, can the reactivity manipulations completed during the first attempt be used to satisfy the license application in any future attempts? Would they have to be repeated after a certain length of time? If so, how long?

10 CFR 55.31(a)(5) requires that an applicant complete five significant control manipulations that affect reactivity or power level at the facility for which a license is sought or at a plant-referenced simulator that meets the requirements of 10 CFR 55.46. There is no expiration date or time limit specified in 10 CFR 55 for these control manipulations. Therefore, assuming that the facility licensee's Commission-approved training program has no expiration period for crediting the required control manipulations and the facility licensee has maintained documentation and/or evidence for completion of these control manipulations, a reapplication for an applicant who previously performed these control manipulations will generally be accepted with no questions regarding the requisite manipulations. However, if there have been any significant changes to reactivity manipulation procedures, core response to rod movements, or a power up-rating of the station then the facility licensee may be asked to explain how these changes do not result in any different and/or observable core response to operator control manipulations. Similarly, if the plant-referenced simulator was used for any of the control manipulations, the facility licensee may be asked to explain how the plant-referenced simulator meets the requirements specified in 10 CFR 55.46(c)(2) including the requirement that the simulator utilizes models that replicate the most recent core load.

Would the same operator have to repeat the 3 months of observation? Would job observation have to be repeated after a certain length of time? If so, how long?

Similar to the above discussion, there is no expiration date or time limit specified for the completion of these requirements unless specified in the facility licensee's Commission-approved training program. That said, the facility licensee may be asked for further explanation regarding the efficacy of these training/experience requirements if there have been any significant changes to the facility or the operating and/or administrative procedures. Finally, if the individual is issued a RO or SRO license but a considerable period of time has elapsed since the individual performed his/her 3 months as an extra person on-shift then it may be appropriate for the facility licensee to consider having the individual perform 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> of shift functions according to 10 CFR 55.53 (f)(2) in addition to the facility's administrative requirements for license activation at the facility.

26

27 202.20 The National Academy for Nuclear Trainings (NANT) Guidelines for Initial Training and Qualification of Licensed Operators -- ACAD 10-001 -- were revised in November 2016, i.e., NANT, ACAD 10-001, Revision 1. The revision updated and clarified the experience and education eligibility guidance for the selection of reactor operators (ROs) and senior reactor operators (SROs) at existing nuclear power plants in Section 2.0, Figures 2-1 through 2-4.

However, Revision 10 and 11 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, in ES-202 reference the NANT guidelines issued in February 2010 (NANT 2010) and states unless otherwise informed by a facility licensee, the NRC believes that the education and experience guidelines described in NANT 2010 constitute the facility licensees education and experience requirements to be licensed as an RO or SRO. Given this NUREG-1021 description, does the NRC also consider the ACAD 10-001, Revision 1, guidelines an acceptable methodology for eligibility determinations at existing nuclear power plants?

The NRC considers the eligibility guidelines for education and experience at existing nuclear power plants promulgated by the NANT including those that were issued in November 2016 --

ACAD 10-001, Revision 1 -- as acceptable methods for meeting 10 CFR 55.31(a)(4).

Notwithstanding the fact that NUREG-1021, Revisions 10 and 11, reference the ACAD 10-001 guidelines issued in February 2010, NUREG-1021 also states that the guidelines for education and experience issued by the National Academy for Nuclear Training (NANT) [i.e., ACAD 10-001 Revisions 0 and 1] outline acceptable methods for implementing the Commissions regulations in this area. NUREG-1021 also states that, when a facility licensee certifies [on NRC Form 398] pursuant to 10 CFR 55.31(a)(4), that an applicant has successfully completed a Commission-approved, SAT-based training program, it means that the applicant meets or exceeds [emphasis added] the minimum education and experience [eligibility] guidelines currently [emphasis added] outlined in NANT 2010. Since ACAD 10-001, Revision 0 will not be retired until May 29, 2018, the NRC Form 398 directions for Block 12.a (Power Reactor Operator Training Program) allow for the applicant to check YES indicating that he/she has completed a SAT-based training program accredited by the NNAB based on meeting the guidelines in ACAD 10-001, Revision 0 (through May 29, 2018) or Revision 1.

27

28 202.21 10 CFR 55.31, How to apply, states that the applicant shall complete NRC Form 398, Personal Qualification Statement - Licensee, and provide certification by the facility licensee of the applicants medical condition on NRC Form 396, Certification of Medical Examination by Facility Licensee. 10 CFR 55.31 and §55.23 also state that these forms can be obtained by writing the NRC or by accessing the NRCs web site.

Given that NRC Forms 396 and 398 were revised in October 2017 with only the revised forms available by writing the NRC or accessing the NRCs web site as of October 20, 2017, what is the implementation date for their use by applicants on impending examinations, license renewals, and medical condition updates?

There are no specified implementation dates for using the newly revised NRC Forms 398 and 396. However, following the direction outlined by 10 CFR 55.23 and 10 CFR 55.31 an applicant is expected to complete the forms that can be obtained by contacting the NRC via phone, mail or use of the NRC public website. Therefore, assuming that the NRC would provide the most recent revision of the forms, applicants should use the most recent revision of the form that is available at the time they initiate the application process (start to complete the forms). For license renewals, 10 CFR 55.57 provides no additional clarification as to which version of the forms should be used, therefore the same expectation applies regarding the use of the forms which can be obtained by contacting the NRC at the time the renewal process is initiated.

In the specific case of initial license applications for which a facility chooses to utilize the preliminary application process, NUREG-1021 Examination Preparation Checklist, Form ES-201-1, specifies a target date of -30 days prior to the examination date (which may be adjusted on a case-by-case basis by the NRC in coordination with the facility licensee) for submittal of preliminary license applications. Therefore, the NRC will continue to accept the previous revision to NRC Forms 398 and 396 dated March 2016 if they were in development consistent with the Form ES-201-1 examination timeline. In other words, the NRC will accept the March 2016 revision to these forms if they were in development or submitted to the NRC as part of the Form ES-201 preliminary license application process as of October 20, 2017.

Ultimately, the program office can foresee no situation for which an applicant or licensed operator would initiate filling out a Form 396 or 398 greater than three months prior to submitting them to the NRC in support of an initial license application, medical update, or license renewal.

This general assumption may be adjusted on a case-by-case basis through consultation with the appropriate NRC regional office. The regional licensing officials are ultimately responsible for ensuring that the forms submitted provide the necessary information to support accurate and timely licensing decisions.

28

29 202.22 As noted in Question 202.21 above, NRC Form 396, Certification of Medical Examination by Facility Licensee and NRC Form 398, Personal Qualification Statement - Licensee, were revised in October 2017. Please provide a summary of the changes implemented by the October 2017 revisions.

The following summarizes the noteworthy changes to NRC Forms 396 and 398:

NRC Form 396 noteworthy changes:

• Added an applicant/operator mailing address block.

• Added a 10 CFR Part 52 Facility Docket check box to support operator license applications for those facilities licensed under 10 CFR 52.

• Revised the Section A, MEDICAL EXAM INFORMATION, to include, as part of the physicians certification statement, that the applicant/operator meets THE MEDICAL REQUIREMENTS FOR LICESED OPERATORS at the facility.

• Updated the GUIDANCE USED check blocks to allow applicants to certify their medical status using the most recent revisions of American National Standards Institute/American Nuclear Society (ANSI/ANS) 3.4-2013, Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants, and ANSI/ANS 15.4-2016, Selection and Training of Personnel for Research Reactors.

• Added Physicians Certification Date to document the date on which the facility licensees Physician/Medical Review Officer (MRO) makes the final certification of the applicant/operators medical suitability.

• Added Name of Applicant/Operator and Docket Number to Page 2 of the form.

• Added an applicant/operator acknowledgement statement and an applicant/operators signature and date to ensure that the individual applicant/operator understands and acknowledges the details of the medical certification being submitted in support of his or her license application and authorizes the release of the related medical information to the NRC.

NRC Form 398 noteworthy changes:

• Added an E-mail Address box (Block 5) to allow for additional communication options for correspondence with licensed operators and license applicants.

• Added Deferral/Excusal/Waiver check boxes (Block 12) to better align with the 10 CFR 55.35 (excusals) and 10 CFR 55.47 (waivers) processes and the terminology used in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Rev.

11.

• Added 10 CFR Part 52 Facility Docket check boxes (Block 14) to support license applications for those facilities licensed under 10 CFR 52.

• Added Trainee position to Block 18, Current Position at Facility, and removed Non-Licensed Operator position details.

29

30

• Added an Electronic Correspondence Option certification check box (Block 27a) to support current effort to develop an electronic correspondence option for initial operator license applications.

30

31 ES-204 Examination and eligibility waivers 204.1 We are planning to randomly select an exam from the NRC GFE bank to satisfy the waiver criteria in ES-204 of NUREG-1021. What do we do if the exam we select has had a question deleted or has two answers based on post-exam comments from the industry?

All exams on the GFE web site are posted, as administered. If a question was deleted or had two correct answers due to post-examination comments, the changes would be reflected in the answer key. By NRC policy, such flawed questions are never entered into the GFE test bank, at large; this ensures that only technically valid test items are entered into the bank for future applicant study and examination development.

For waiver purposes, you should randomly select from among all the 100- and 50-question GFEs (i.e., pre- and post-2004 examinations, respectively) posted at http://www.nrc.gov/reactors/operator-licensing/generic-fundamentals-examinations/past-exams.html regardless when the individual originally took the examination; your selection method should be simple, clear, and explainable (if asked during a future inspection). It does not matter if your selected exam had answer key changes or question deletions resulting in something less than 100 or 50 questions. If your random exam selection does contain a flawed question(s), simply remove it (them) from the exam before you administer it, and score the exam on the basis of its revised denominator. For example, if you randomly choose the October 2002 BWR exam, you will note that Question # 21 had two correct answers. Therefore, remove that question before administering the exam and determine each examinees grade based on a 99-point exam.

31

32 ES-205 Generic fundamentals examination (GFE) 205.1 At what point will the GFE be a computer-based exam including immediate grading?

Proctor would be onsite. Could anything be done such that the GFES [Generic Fundamentals Examination Section] is generated (and thus administered) just-in-time?

In light of the increasing size of the GFEs question banks, the NRC changed the bank/modified/new question distribution from 25/20/5 to 40/5/5 (based on a 50-question examination). Also, due to the reduction in the frequency of GFES starting in CY 2017, Revision 11 of NUREG-1021 permits facility licensees (other authors) to develop up to 2 additional GFES per year (for review and approval by the NRC).

The NRC is uncertain, at this point, when and if GFEs will be based entirely on bank questions or be a computer-based exam. The fundamental question to be addressed before either of these potential changes could be implemented is whether these changes will be able to ensure a discriminating, valid, and reliable examination (10 CFR 55.49).

32

33 205.2 What are the opportunities for industry comment on the Generic Fundamentals Examination (GFE)?

In response to an industry request, ES-205 of NUREG-1021 has been revised (refer to Section C.4) to include provisions for one BWR and one PWR instructor to review the GFE before it is administered. The reviewers must be drawn from facilities that will not participate in the scheduled GFE and must sign security agreements. If they do not provide feedback to the NRC staff within the time allotted, the examinations will proceed on schedule. The NRC will evaluate the reviewers comments and make changes as deemed appropriate.

According to the examination proctor instructions and procedures, each GFE administration is followed by a five-day review and comment period for industry. This period of time allows all utilities participating in any GFE to comment on the examination as a whole and on any of its questions. Additionally, for those utilities that did not participate in a particular GFE, a copy of the exam will be available on the NRCs GFE web site and utility comments are welcome. The NRC reviews and analyzes all utility comments and, based upon their merit, makes adjustments to the answer key before the final grade reports are issued. We believe this process -- that allows industry input -- is a positive one that has worked well over the years. In fact, generally one or two questions per examination do end up with answer key changes (e.g., 2 correct answers or deletion) attributed to utility comments.

In order to minimize the differences of opinion that can occur, the NRC encourages utilities to provide solid technical information and documentation to support their position for any answer key changes. Otherwise, the NRC staff may have insufficient justification to make the desired adjustments. When comments are supported with documentation, we carefully review each comment at two levels: the examination developer level and the NRC staff level. Both levels involve subject-matter-expert analysis of the question and the reference information before any final decision is made.

33

34 205.3 What is meant by operational validity and have GFE test items become more difficult by testing plant-specific system knowledge?

In the development process for the GFE, the NRC strives to create questions that are technically, operationally, and psychometrically valid. For example, to achieve operational validity -- a hallmark of good test item writing that seeks to ask questions within the context of the actual job -- we strive to develop questions that assess applicant understanding, use, and application of the safety-significant knowledge that is required for licensing. These types of items assess whether applicants can use and apply the knowledge they learned vice merely recalling the facts. To improve operational validity, GFE questions will often use basic plant terms and situational contexts.

(PLEASE NOTE: The operational validity of a GFE question does not require that the applicant be able to operate the plant. The GFE does not test knowledge of plant-specific system design, general or emergency operating procedures. However, an operationally valid GFE question does assess understanding and application of components, reactor theory, and thermodynamics within a realistic, job-related context. Therefore, applicants are expected to possess some basic understanding of plant systems and plant response.)

The NRC has received occasional comments that selected GFE questions require an inappropriate level of plant systems knowledge. There are many GFE knowledge and abilities (K/As) that directly or indirectly require some basic knowledge of power plant systems. For example, knowledge of the basic function of some plant systems (such as the reactor, reactor coolant system, control rod drive system, main turbine and main generator) is required. Without some assumed basic system knowledge, we would have to limit fundamentals knowledge testing to theoretical facts alone. By assuming some basic plant systems knowledge, we are able to move from theoretical fact testing (i.e., fundamental knowledge) into the real, or physical, domain where our examinations are more operationally valid.

During the GFE review process, the examination author and NRC staff evaluate each question to determine whether inappropriate plant systems knowledge is required. In striving to achieve high operational validity, there is some risk that we will occasionally cross the fine line that separates appropriate (basic) plant systems knowledge from inappropriate (more advanced) plant systems knowledge. On a few occasions, utility post-examination comments have expressed this concern, and the NRC has made changes to the examination answer key prior to issuing final grades. The NRC endeavors to administer licensing examinations that are valid and reliable indicators of the applicants knowledge and abilities. The most valid operator licensing written examinations (including the GFE) use questions that have valid content, operational relevance, and the ability to discriminate between different levels of applicant knowledge. Therefore, the fundamental knowledge addressed by a K/A will often be tested by requiring the applicant to apply the knowledge in the context of a realistic, or operational, setting.

The fact that a specific word or term is absent from a generic fundamentals K/A statement does not disqualify a related knowledge from being tested on the GFE. K/A statements are often written as general statements of required knowledge. Therefore, GFE questions are not required to contain specific words found in generic fundamentals K/A statements. However, 34

35 they are required to preserve the intent of the valid K/A. In summary, the NRC staff endeavors to exercise good judgment and not to go beyond normal GFE training bounds. We welcome feedback from utilities that believe we may have transcended those boundaries and will seek to correct those instances. (Please see Questions 401.14, 401.15, 401.16, and 401.38 for related discussions).

205.4 Are there any other statistical factors involved in evaluating GFE questions?

The GFE is a nationally-administered, standardized examination. Since a large number of individuals are evaluated (in comparison to the site-specific examinations), the NRC is able to calculate statistics that provide insight into how the examination performed. After every GFE, we evaluate the overall examination and individual question performance statistics to determine, among other things, if there is a basis to make any changes in the answer key or the questions before they are reused. One statistical indicator of the overall exam is the mean score of the applicants taking the GFE. Typically, the mean scores have been relatively high, hovering in the 88 to 91 percent range. This is indicative of a moderately easy examination for well-trained applicants.

Another statistic evaluated during the post-examination review is the item discrimination ratio (IDR). The IDR is calculated and expressed as a correlation coefficient for each test question.

The IDR indicates whether the question discriminated between masters and non-masters, i.e.,

between high scorers and low scorers. We would expect higher performers overall to answer any given question correctly more often than lower performers overall. Therefore, when the IDR is a positive number, it confirms that the question discriminated as intended.

205.5 Are there differences in viewpoint on how the validity of GFE questions is determined?

The NRC staff believes that the overall validity of the GFE can be viewed from different perspectives. A utilitys view will most certainly be influenced mainly by the learning objectives and content of its fundamentals training program; a GFE that examines only those topics that the utility trained on would be the most valid. The fact that all utility fundamentals training programs are probably not exactly the same suggests that there may be a variety of different viewpoints regarding the validity of GFE questions.

However, the GFE is a nationally-administered, standardized examination whose content validity is determined mainly by the K/A statements listed in the NRCs K/A catalogs, NUREG-1122 and -1123. From the NRCs perspective, a GFE that maintains clear links to those fundamentals K/As would be most valid. Therefore, the extent to which a utility that has adjusted its fundamentals training program to include learning objectives and content that encompasses all the fundamentals K/As (with 2.5 or greater importance rating) would likely determine the extent to which the utility shares the NRCs perspective on GFE validity. (Please see Question 401.12 for a related discussion).

35

36 205.6 NUREG-1021, Rev. 11 becomes effective August 15, 2017. How will the changes implemented by Rev. 11 of NUREG-1021 effect the content of the September 2017 GFE?

NUREG-1021, ES-205, Attachment 4, GFE Test Item Distribution, was revised in Rev. 11 to include a new topic, Basic Energy Concepts. This new topic will be tested with one question on the BWR GFE and one question on the PWR GFE, using a selected K/A for Basic Energy Concepts, as listed in NUREG-1123, Rev. 2, Supplement 1, or in either NUREG-1122 or -1123, Rev. 3 Draft Reports for Comment, published in April 2017 (82 FRN 18018).

In order to account for the addition of one question associated with Basic Energy Concepts, both the BWR and PWR GFEs will test one less question associated with the GFE topic, Thermal Hydraulics.

36

37 ES-301 Preparing operating tests (JPMs and scenarios) for initial licensing examinations 301.1 One of the recognized factors for test item validity is discrimination of job position, however, the walk-through examination has a significant portion done in the plant, outside the control room. These tasks are non-licensed operator level, thus, fail to discriminate for the job positions of reactor operator or senior operator.

10 CFR 55.45(b)(1) requires the operating test to be administered in a plant walk-through and a simulation facility. Therefore, it would not be possible to eliminate the in-plant portion without first amending the regulation. Reactor operators and senior operators need to be familiar with in-plant operations that they oversee and could conceivably be called upon to perform during emergency situations. Per ES-301 of NUREG-1021, tasks selected for the walk-through should have meaningful performance requirements and their K/A (knowledge and ability) importance factors, which were derived by a panel of subject matter experts from the industry and NRC, should be at least 2.5.

301.2 Our experience has been that we are told ALL items of 10 CFR 55.45 and §55.43(b) must be sampled.

If 100% of sampling for topics in §55.45(a) is not required, is there a definition of representative sample?

What is meant by a "representative sample" of the 13 items identified in 10 CFR 55.45(a)?

Section B of ES-301 states that all 13 items in 10 CFR 55.45 do not need to be sampled on every operating test. Although NUREG-1021 does not include a similar statement with regard to the written examination, the same policy still applies. In accordance with Section D.1.b of ES-401, the topics for the written examination are to be systematically selected from the appropriate Knowledge and Abilities Catalog (NUREG-1122 or 1123). Although the NRC has not developed a definition of a "representative sample," logic dictates that it should include a reasonably complete, thorough, balanced, and varied cross-section of the items in the population to be sampled. All of the items should be sampled from time to time, and, absent a basis for emphasizing certain items, it is expected that every item would be sampled at about the same frequency. An examination constructed in accordance with NUREG-1021 will normally contain a "representative sample" of the required items.

37

38 301.3 Do the audit exam and the NRC exam have to be 100% different (D.1.a)?

ES-301, D.1.a - No reuse of audit material for subsequent exams?

To what extent do "similar events" between the audit and NRC exam need to be identified? For example, if the audit examination contained a faulted SG [steam generator] in one scenario (safety valve stuck open) and the NRC examination contained a faulted SG (pipe rupture in containment), would these situations be considered "similar?"

No. As noted in Section D.1.a of ES-301 (NUREG-1021), simulator events and JPMs that are similar to those that were used on the audit test (or audit tests in the case of retake applicants) are permitted provided the actions required to mitigate the transient or complete the task (e.g.,

using an alternate path as discussed in Appendix C) are significantly different from those required during the audit examination. The facility licensee shall identify for the NRC chief examiner those simulator events and JPMs that are similar to those that were tested on the audit examination.

The two events cited in the example are "similar" (in that they both involve a faulted SG) and should be discussed with the NRC chief examiner. In this case, the mitigation strategy for the two events - one being inside and the other outside containment - are sufficiently different that their use would probably be acceptable (unless there were other predictable patterns between the two scenarios).

301.4 Can there be scenario repetition with similar transients?

Although the same scenarios and job performance measures may not be repeated on subsequent days during the examination week(s), events and tasks that are similar to those that were tested on previous days during that examination are permitted provided the actions required to mitigate the transient or complete the task are significantly different from those required on the previous examination. This is consistent with the policy for repeating events and tasks from the applicants' audit examination as stated in Section D.1.a of ES-301.

38

39 301.5 How is the JPM system selection supposed to occur? Shouldn't there be a systematic (e.g., random) selection of systems within each of the safety functions? Otherwise, won't the operating exam be somewhat subject to predictability? Same concern with event selection for simulator exams (scenarios).

Section D.1 of ES-301 discusses a number of general guidelines applicable to the entire operating test, and Sections D.2 through 4 provide specific guidance applicable to the walk-through, including the requirements to distribute the JPMs among the applicable safety functions and administrative topics, to limit the repetition of tasks from the previous licensing exam, and to include new and modified tasks on each test. Although ES-301 does not specify the use of systematic or random sampling for the operating test as ES-401 does for the written exam, it would certainly be an acceptable method for determining the test content.

39

40 301.6 The continuous ratcheting of expectations is bypassing the [systematic approach to training] SAT process. Example - Cannot use a high importance JPM because it is perceived to be too easy, and operators are trained and tested on it.

Current subjectivity on what is a discriminatory JPM with the removal of the questions.

Why can't the selection of JPM's for the license exam be driven by the SAT process and K/A value? "Low discriminatory value" is a euphemism for "too easy" and as a result, the difficulty of the exam is ratcheting up to an unreasonable level. This is contrary to the NRC stated goals.

The NRC does not agree that the difficulty of the walk-through portion of the operating test is being ratcheted up to an unreasonable level. On a nationwide basis, the RO and SRO operating test passing rates have generally ranged between 94 and 98 percent since the early 1990s. Refer to the examination performance trend graphs posted on the Licensing Process page.

Keep in mind that the NRC licensing examination is not a part of the facility licensee's SAT-based training process. As stated in 10 CFR 55.45(a), the content of the operating test will be identified, in part, from the learning objectives derived from a systematic analysis of operator duties performed by the facility licensee.

As stated in Section D.4.b of ES-301(NUREG-1021), the JPMs should, individually and as a group, have meaningful performance requirements that will provide a legitimate basis for evaluating the applicant's understanding of and ability to safely operate the associated systems and the plant (as required by 10 CFR 55.45). Previously, when each system evaluation consisted of a JPM plus at least two prescripted follow-up questions, the questions would sometimes compensate for the minimal discriminatory potential of the JPM. Now that the prescripted questions have been eliminated, examiners have been instructed to place increased emphasis on the discriminatory value of the JPMs. However, that does not mean that high importance JPMs will be excluded from the sample. High-importance JPMs will always be acceptable if they discriminate and provide a legitimate basis for evaluating the applicants' understanding of and ability to safely operate the associated system. A walk-through test that is heavily weighted with simplistic, one- or two-step tasks during which everything works as designed will not provide the NRC with an adequate basis to make a licensing decision.

40

41 301.7 My 1998 exam was comprised of 20 JPMs. The 1999 exam is comprised of 30 JPMs (3 sets of 10). If I repeat 30% of the 1998 JPMs, I can use a total of 6 JPMs on the 1999 exam or 30% of each of the 3 sets of 10 JPMs is 9 JPMs. Is it 30% of the JPMs of the previous exam or is it 30% of the current exam that can be repeated?

The repetition limits specified in ES-301 (refer to Forms ES-301-1 and 2) apply to the current operating test, and, beginning with Revision 9 of NUREG-1021, will limit the use of JPMs to be randomly selected from the last two licensing examinations at the facility. Therefore, each of the three 10-JPM sets for 2004 can include no more than three JPMs from among the 20 that were used on your 2002 and 2003 operating tests. You cannot use all nine of the repeated JPMs on one test set and none on the other two, and the same JPMs cannot be repeated on subsequent days. Ideally, the test sample should be developed systematically from the total population of operator tasks and then checked to confirm that the repetition from the previous exam is within limits.

301.8 When determining allowable JPM overlap for a retake applicant, do you use the exact 10 JPMs the applicant saw on the original exam or the entire JPM set used for the exam?

(These numbers could be different.)

In accordance with ES-301 of NUREG-1021 (refer to Form ES-301-2), the current systems walk-through may repeat up to 3 JPMs randomly selected from the last two licensing examinations (including all the operating test sets) at the facility. However, the 30% is an upper limit and may not be appropriate in the case of retake applicants. Section D.1.a also prohibits the repetition of any exact-same items from the applicant's audit test or tests, in the case of retake applicants. Similar items (with different success paths) may be acceptable and shall be identified to the NRC chief examiner for approval.

301.9 Please define "alternate path" JPMs and give one or more examples. Does a fault have to occur to qualify as an "alternate path" JPM?

What is the difference between a faulted JPM and an Alternate Path JPM?

The concept of alternate path JPMs is discussed in some detail in Section C of Appendix C of NUREG-1021. Although most alternate path JPMs do involve some sort of system fault, the goal is to assess the applicant's response to a situation that is not as it should be or is somehow different from what the applicant might have expected based on the initiating cue for the task.

Alternate path and faulted JPMs are effectively synonymous.

41

42 301.10 Use of 4 of 10 faulted JPMs I believe is "negative" training and evaluation. I expect our plant to operate every time. Maybe for 2 of 10 faulted is fine. 4 of 10 will train the operators to expect the plant controls not to function. Should maybe be PRA based?

We acknowledge your concern. The NRC is sensitive to the issue of negative training but is also obligated to ensure that the licensing examinations do not become predictable and effectively discriminate between safe and unsafe applicants. Experience showed that some JPMs may not provide an adequate basis for evaluating the applicants' understanding of the system unless they require the applicant to exercise an alternate success path. Therefore, the number of alternate path JPMs was increased to compensate for the elimination of prescripted questions with every JPM. As discussed in the previous question, system faults provide only one source of alternate path JPMs. It would certainly be appropriate to use risk insights when selecting operator actions to be tested using alternate path JPMs.

301.11 For examinations spread over two weeks, are different administrative job performance measures required?

Yes. As stated in Section D.1.a of ES-301, the same job performance measures and simulator scenarios shall not be repeated on subsequent days (i.e., they shall not be used for more than one day during an examination).

301.12 Why are we using more JPM's [job performance measures] for the administrative section?

Since Revision 8 of NUREG-1021, the NRC has preferred to test the five administrative topics using JPMs rather than questions because JPMs are generally a better, more performance-based measurement tool. When Revision 9 combined the administrative and systems walk-through portions of the operating test, good testing and measurement practice prompted the NRC to shift entirely to a JPM format rather than retain the option for mixed testing media in the combined walk-through.

301.13 What is counted in the simulator?

As stated in Section D.5.d of ES-301, an applicant should only be given credit for those events that require the applicant to perform verifiable actions that provide insight to the applicant's competence. The required instrument and component failures should normally be completed before starting the major transient; those that are initiated after the major transient should be carefully reviewed because they may require little applicant action and provide little insight regarding competence. Each event should only be counted once per applicant; for example, a power change can be counted as a normal evolution OR as a reactivity manipulation, and, similarly, a component failure that immediately results in a major transient counts as one or the other, but not both.

42

43 301.14 Would it be appropriate to do an administrative job performance measure during the systems or dynamic portion of the operating test?

Yes. Section D.3 of ES-301 encourages examiners to integrate the evaluation of the administrative topics into the systems and simulator evaluations because it improves the flow of the operating test. For example, as noted in Section D.3.d of ES-301, the "Emergency Plan" can be evaluated by integrating it into a simulator transient that requires implementation of the emergency plan. Similarly, an alternate path job performance measure in which a component fails could set the stage for an equipment clearance job performance measure for "Equipment Control." As noted in Section D.3, the applicants' proficiency in the administrative topics should be deliberately evaluated and not inferred from observations made during the simulator operating test. Moreover, in accordance with Section D.3.n of ES-302, examiners will limit their discussions with the applicants while the scenarios are running so as not to create a distraction.

301.15 Operating Exam - Admin.: This part of the exam process needs to be integrated into the written and JPM (walk-through) segments, and eliminated as a separate entity - only a couple of areas are examined, with no margin for error! An individual can score high on the written exam, do excellent on the simulator, and pass all of the systems JPMs yet fail to get licensed due to not passing a couple of admin "questions" - the knowledge and/or abilities could easily be included with other exam segments.

Why is the admin[istrative] area a stand-alone area on the exam? Why is it even there at all?

JPM [job performance measure]/Admin sample rate is small. Therefore more than 1 failure results in an overall failure. Is it possible to get something with more balance?

As discussed in Section B.1 of ES-301(NUREG-1021), the "Administrative Topics" of the operating test implement Items 9 through 12 of 10 CFR 55.45(a). Prior to Revision 4 of NUREG-1021, which was issued in May 1987, examiners often made too many inferences regarding the applicants' understanding of the administrative topics based upon their actions in the simulator. Therefore, the NUREG was revised to require examiners to discuss and evaluate a selection of administrative topics in a separate operating test category.

However, based on stakeholder feedback during a number of public meetings in 2001 and 2002 (refer to the Operator Licensing Public Involvement page), the NRC concluded that the scope and format of the operating test had placed too much emphasis on the administrative topics.

Consequently, with Revision 9 of NUREG-1021, the NRC has consolidated the administrative and systems topics into a single walk-through operating test, consisting entirely of JPMs. The revised test structure replaces one of the RO administrative tasks with an extra task in the systems area and generally de-emphasizes the administrative topics (refer to Section D.3 of ES-301).

43

44 301.16 Is it NRC policy for every JPM [job performance measure] to have adverse safety consequences if the operator makes an error?

No. As stated in Section D.1.c of ES-301, the K/As covered during the operating test should have importance factors of at least 2.5. Moreover, as stated in Section D.4.b, the JPMs should, individually and as a group, have meaningful performance criteria that will provide a legitimate basis for evaluating the applicant's understanding of and ability to safely operate the associated systems and the plant. Although Section D.3.b of ES-303 requires examiners to explain the safety consequences (as applicable) of the applicant's errors, this should not be misconstrued as a requirement for every JPM to have adverse safety consequences if the applicant makes an error.

Refer to Question 301.6 for a related discussion regarding discriminatory JPMs.

301.17 Does the exam have to cover RP [radiological protection] and EP [emergency planning]

(10 CFR 55.43)?

Why does there have to be an administrative JPM [job performance measure] on radiological items/E-plan for RO's? This is GET (general employee training) material!

Why are GET-type radiation area, contaminated area, radiological work permit (RWP)

JPMs involved in a license exam? These are not discriminatory to a SAFE LICENSED operator. GET should be left to GET and eliminated as a part of the licensing exam.

The regulations currently require the written examination and the operating test to cover a representative sample of the items listed in 10 CFR 55.41 and §55.43 (depending on the license level) and 55.45, respectively, to the extent that they are applicable to the facility. With regard to testing GET-type topics, exam developers should strive to write questions or JPMs that test the applicants at a licensed level, such as their response to a problem that would be part of their licensed duties. Refer to Question 301.2 for a discussion of "representative sampling."

As discussed in response to Question 301.15 above, Revision 9 of NUREG-1021 has restructured the walk-through operating test to de-emphasize the administrative topics, particularly for RO applicants. As outlined in Section D.3.a of ES-301, RO applicants will be tested on four, rather than five, administrative tasks, and they generally need not be evaluated on each of the four administrative topics (Equipment Control, Radiation Control, or Emergency Plan can be omitted by performing two tasks related to Conduct of Operations).

This affords the test developer greater flexibility in tailoring the content of the test to ROs job requirements at the facility.

44

45 301.18 In Revision 9 of NUREG-1021, Section B.1 of ES-301, which describes the Administrative Topics portion of the walk-through operating test, did NOT include subjects related to emergency operating procedures (EOPs) - E-Plan JPMs were allowed, but not EOPs.

Supplement 1 to Revision 9 changed that. Why?

Supplement 1 to Revision 9 of NUREG-1021 erroneously listed "emergency operating procedures" as an example of the type of information that could be evaluated under the "Emergency Procedures/Plan" Administrative Topic of the operating test. The topic heading and list of examples were mistakenly revised to conform with Section 2.4 of the NRCs Knowledge and Abilities (K/A) Catalogs (NUREG-1122 and -1123) that underwent significant revisions. However, Supplement 1 to Revision 9 of NUREG-1021 did NOT change the selection process for, or the content of, the Emergency Procedures/Plan Administrative Topic, as described in Section D.3.a of ES-301. Both Revision 9 and the Supplement state that "only those K/As related to the emergency plan and implementing procedures [not those associated with the Emergency Operating Procedures (EOPs)] (emphasis added) are applicable to this category of the operating test" because it is intended to implement Section 55.45(a)(11) of 10 CFR Part 55.

The NRC regrets any confusion that this error may have caused.

45

46 301.19 Revision 10 of NUREG-1021, Form ES-301-5, Transient and Event Checklist, added new Instruction 4 to allow placement of SRO-I applicants in either RO position to provide the best evaluation of these applicants in the manipulation of controls.

However, Section D.5.a of ES-301 and Instruction 1 on Form ES-301-5 were not changed.

Both of these guidelines indicate that Instant SRO (SRO-I) applicants must serve in both the SRO and the at-the-controls (ATC) positions.

Please clarify this seeming inconsistency. In other words, are SRO-I applicants required to serve in the SRO and ATC positions or is it now allowable to evaluate SRO-I applicants in the SRO and balance-of-plant (BOP) positions?

The intent of new Instruction 4 was to address new reactor facility licensees only and allow the NRC chief examiner to place SRO-I applicants in either the ATC or BOP position taking into consideration which position - ATC or BOP - that provides the best evaluation of SRO-I applicants in manipulating plant controls. There was no intent to change the requirement per Instruction 1 for existing reactor facility licensees that SRO-I applicants must be evaluated in both the SRO and ATC positions.

Therefore, given that new Instruction 4 (Form ES-301-5) was intended for only new reactor facility licensees, the following clarifications/changes will be implemented immediately and incorporated into the next revision of NUREG-1021:

ES-301 D.5.a Based on the anticipated crew compositions, determine the number of scenarios and scenario sets necessary to rotate each RO and SRO-I applicant into the lead reactor operator (i.e., the at-the-controls) position. For example, . and every RO applicant rotates through the balance-of-plant (BOP) position for at least one scenario.

However, for new reactor facility licensees that use the ATC operator primarily for monitoring plant parameters, the chief examiner may place the SRO-I applicants in either the ATC or BOP position to best evaluate the SRO-I in manipulating plant component controls per Competency 3.

Form 301-5 Instruction 4 For new reactor facility licensees that use the ATC operator primarily for monitoring plant parameters, the chief examiner may place SRO-I applicants in either the ATC or BOP position to best evaluate the SRO-I in manipulating plant controls.

46

47 301.20 NUREG-1021, Rev. 11, ES-301, Preparing Initial Operating Tests, Section D.5.b states in part, To maintain test integrity, every scenario shall be new or significantly modified to ensure that the applicant has not had the opportunity to rehearse or practice the scenario. A significant modification means that, for each scenario, at least two events have not been used on the previous two NRC initial licensing operating exams. This requirement is repeated in Appendix D, Simulator Testing Guidelines, Section C.1.f.

These provisions define significantly modified scenarios, however they do not define what requirements must be met for a scenario to be considered new. Is it possible to develop a new scenario by utilizing events from various scenarios from the previous two NRC examinations combined into a unique scenario? Will retake examinations be considered one of the previous two NRC examinations when considering the requirement for two new events?

NUREG-1021, Rev. 11, ES-301, Section D.5.b establishes requirements to ensure operating test integrity and prevent simulator scenario predictability. The intent of Section D.5.b is to ensure that an applicant cannot predict the subset of events that will potentially be used for his or her initial examination simulator scenarios. Therefore, creating a new scenario which is composed entirely of events from the previous two NRC examinations (i.e., 100% overlap) would not be acceptable. A new scenario is a scenario in which none of the scenarios non-reactivity events that occur prior to or after a major event have been used on the previous two NRC examinations (i.e. no overlap).

To state the overall requirement more simply, Section D.5.b requires that all simulator scenarios on NRC examinations contain at least two events which have not been utilized on the previous two NRC examinations. As is stated in Section D.5.b, reactivity events are exempt from the requirements of this section. Appendix D provides further clarification on major events by stating, Additionally, if any major event is repeated from either of the previous two NRC initial licensing operating tests, the examination author should change the major event, the ICs, or subsequent malfunctions (or a combination) to alter the course of action (within the emergency procedures) for the given scenario(s).

When considering which examinations apply towards the previous two NRC initial licensing operating exams, a retake examination is considered applicable to this provision assuming the retake examination consisted of at least one operating test scenario.

47

48 ES-302 Administering operating tests for initial licensing examinations 302.1 If the shift technical advisor is licensed, is he at risk if he is a surrogate? Can anyone do it?

Can a formerly licensed or certified person be used as a surrogate on an initial examination?

If a licensed operator is filling the role of a surrogate operator, and he/she performs errors, is his/her license in jeopardy (by the NRC)?

Is the NRC going to provide specific guidance for the use of surrogates in the exam process?

Section D.1.j of ES-302 (in NUREG-1021) addresses the use of surrogates and shift technical advisors.

Although licensed operators are generally preferred, NUREG-1021 does not require the surrogate operators during the dynamic simulator operating test to be licensed. Anyone who does play a surrogate role must be knowledgeable and competent because, per Section D.1.j of ES-302, they will be expected to assume the full responsibilities of the roles they take during the test. Using unqualified surrogates may place the license applicants at greater risk of failure if the surrogate makes an error.

Surrogates who are licensed operators are at risk because the NRC expects facility licensees to take remedial action (including removal from licensed duty, retraining, and testing, as appropriate) if a licensed operator makes significant performance errors during the operating test or while on shift in the control room.

The NRC could take licensing action against the individual pursuant to Subpart G of 10 CFR 55, but it has never done so in the case of an operator filling a surrogate role during a simulator operating test. The NRC would only take such an action as required to protect the public.

302.2 Can an applicant fill the STA role during a scenario? If yes, can he/she actively fill the role or will "normal" surrogate activity be expected?

No. Section D.1.j (second bullet) of ES-302 clearly states that another applicant will, under no circumstances, be allowed to witness an operating test.

48

49 302.3 What role can the STA play when they are the extra person?

ES-302 - General (D.1.j) - What determines if an STA is "necessary"?

Although the rules now allow the use of surrogates as STAs, we severely limit the surrogates role as part of the team. This results in training the candidates under conditions, roles and responsibilities that are different than real operating practice and standards. Why do we limit the STAs role resulting in a "train for the exams" culture?

As stated in Section D.1.j (first bullet) of ES-302 (in NUREG-1021), consultations with an STA shall be conducted in accordance with the facility licensee's normal control room practice; e.g.,

an STA shall not be stationed in the simulator if they are on-call at the site. The STA should not take a proactive role in assisting or coaching the applicants because it would hinder the examiners' ability to evaluate the applicants' competence. ES-302 requires examiners to brief STAs on the content of the scenarios and their expected actions in response to every event.

Examiners will run additional scenarios if necessary to make a licensing decision.

302.4 Can we use more than 2 ROs if Technical Specifications (TS) require it? Does this apply to administrative requirements (e.g., however ops may use more than 2 ROs)?

Can we increase the number of candidates/scenario?

If the facility's TS (not administrative procedures) require more than 2 ROs in the control room, the NRC will allow additional surrogates during the simulator operating test to fill the normal crew complement. There will never be more than two RO applicants on any simulator operating crew. Refer to Section D.1.j of ES-302.

49

50 302.5 For purposes of appeal - why is video taping of scenarios NOT allowed? I'm not looking for rule change; more what forms of documentation should be used and kept for appeal purposes.

Why discriminate against taping initial operating tests when there is no similar requirement in ES-600 series?

Why is video taping the operating test prohibited?

At the time the no-taping policy was set, experience indicated that video taping would not provide sufficient detail to support individual licensing decisions for every member of the operating crew. Moreover, the practice was considered intrusive to the applicants and examiners, and several facility licensees expressed concern over how the video tapes would be used. This issue was addressed in response to Question Nos. 403 and 404 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses."

In accordance with Section D.3.f of ES-302, the licensee should, in coordination with the NRC chief examiner, record as many key parameters as possible and provide a copy of the recordings to the chief examiner for use in the grading process. This is particularly important if the applicants failed to accomplish the expected actions and there is a possibility of a test failure. The examiners will collect and retain other forms of documentation (e.g., logs, notes, and checklists) generated by the applicants.

302.6 Do SRO-upgrade applicants acting as RO panel operators to complete a crew have to have a specific evaluator observe them (B.3)?

No. As noted in Section D.1.d of ES-302 (in NUREG-1021), if a three-person operating crew consists entirely of senior reactor operator (SRO) upgrade applicants (who do not have to be evaluated on the control boards), the chief examiner may assign only two examiners to observe the crew. Although the applicants in the reactor operator and balance of plant positions may not be individually evaluated, they will be held accountable for any errors that occur as a result of their action(s) or inaction(s) and graded on their ability to "Operate the Control Boards" (i.e.,

SRO Competency 3). SRO-instant applicants will always be individually evaluated by an NRC examiner regardless what operating position they are filling during a given scenario.

302.7 Why can't we add a Shift Manager to the NRC-examined crew to handle communications, etc?

As explained in Attachment I (Section II) of SECY-98-266, the staff does not permit more than one person to fill a senior operator position during the simulator test because the principal duties of the shift manager position (i.e., assuming the role of the emergency director, performing emergency classifications, and making protective action recommendations) are normally a part of the operating test for senior operator applicants.

50

51 302.8 When evaluating SRO success in "Classifying the [radiological emergency plan] REP" during the operating exam, what criteria do the examiners use for when to start the 15 minute clock (expectation)? (15 minutes from event to classification)

Since the simulator operating tests for the initial licensing examination are conducted with only one applicant in the SRO position, the NRC does not require the SRO to complete the emergency classification within the normal period of time. In most cases, the applicant is asked to classify the event after the scenario is complete and the simulator is in freeze. Another option is to do a separate emergency plan classification as a JPM, which is only considered time-critical if the facility licensee has a validated time standard.

302.9 Do you tell a person that it is a time-critical task?

Yes. Part D, Item 4 of Appendix E requires examiners to describe the initial conditions, explain the task to be completed, explain which steps to simulate and which ones to discuss, and indicate whether the task is time critical.

302.10 If during a JPM, the applicant misses or skips a procedure step or steps and later on recognizes that he/she has missed the steps - can he/she request to start the JPM over?

No. The applicant cannot start the JPM over, but can perform the missed step(s) after complying with the facility's policy for reporting procedural errors and receiving permission. This is consistent with the grading policy in Section D.2.a of ES-303 (in NUREG-1021), which states that if an applicant initially misses a critical step, but later performs it correctly and accomplishes the task standard without degrading the condition of the system or the plant, the applicant's performance on that JPM would generally still be graded as satisfactory. The examiner would be expected to ask follow-up questions based on the applicant's error, document those questions and answers, and determine a system grade based on the applicant's overall performance.

Once the applicant has completed the JPM, he or she cannot go back and start over, but the examiner will consider any corrected information provided when grading the operating test (refer to Section D.2.f of ES-302. Note that if an applicant exceeds twice the validated time estimate for any JPM (including time-critical) because he or she has selected an incorrect procedure or operated the wrong equipment (despite being presented with sufficient plant feedback to correct the error), the examiner should stop the JPM, document the circumstances, and proceed with the next JPM. However, if the applicant is on the correct path but has simply stopped making progress toward completing a non-time-critical JPM, the examiner should ask the applicant to describe the work to be done and how long it should take to complete the JPM. If the applicant does not then make timely progress toward completing the described actions, the examiner should inform the applicant that the allowed time for the JPM has elapsed and the applicant will be evaluated on the work completed. The examiner should then proceed with the next JPM.

51

52 302.11 If an applicant shows system knowledge weaknesses during administration of a JPM, how far can the examiner go with the non-prescripted questions? Can the examiner ask questions about another system or another function of the same system covered in the JPM?

As stated in Section D.2.f of ES-302, the examiner should ask question as necessary to confirm the applicant's understanding of the system as it relates to the task that was performed. The examiner should not ask questions about another system or another function of the same system unless it relates to the task that was performed.

302.12 Is there a "standard" method for applicants to answer open reference walk-through questions (i.e., if fairly certain of answer give it or always look it up)?

There is no standard method for applicants to answer follow-up questions during the operating test. If they are confident that they know the answer, there is no need to look it up. Examiners are not required to confirm the source, and looking up every answer can significantly extend the length of the test. As discussed in Attachment 1 of ES-301, the operational orientation required of follow-up questions on the walk-through test and the applicant's access to reference documents, argue against the use of questions that test for recall and memorization. Any follow-up questions that do not require any analysis, synthesis, or application of information by the applicant should be answerable without the aid of reference materials. Furthermore, as stated in Part D, Item 8 of Appendix E, if the applicant needs to consult a reference to answer a follow-up question, the applicant should ask the examiner if it is acceptable to do so. Although there is no specific time limit for any question, an applicant may be evaluated as unsatisfactory on a question if he or she is unfamiliar with the subject or reference material and is unable to answer the question in a reasonable period of time. Applicants will not be permitted to conduct unlimited searches of the plant reference material during the examination.

52

53 302.13 Can a surrogate operator be used to replace an applicant who has already completed the minimum event/scenario requirements in ES-301 (of NUREG-1021), thereby limiting the applicants risk of making additional errors?

The NRC staff generally will not accept this practice for the following reasons:

• Section D.1.j of ES-302 (of NUREG-1021), which addresses the use of surrogate operators, clearly states that surrogate operators would be used when they are required to complete the operating crew (e.g., during retake tests or for a class consisting entirely of ROs).

• Section D.3.o of ES-302 directs examiners to run additional scenarios, if necessary, to ensure that all required evolutions and competencies are covered. For example, if an applicant has only one opportunity to demonstrate competence on a particular rating factor, but makes an error that does not affect his or her performance of a critical task, the examiners shall give the applicant another opportunity to demonstrate competence or to make a second error that would justify an unsatisfactory score for the subject rating factor (refer Section D.2.b of ES-303 for detailed simulator grading instructions).

• Part E, Item 11, of Appendix E (of NUREG-1021), which is used to brief license applicants in preparation for the simulator operating test, clearly states that the initial test will normally consist of two or three scenarios.

• Although the staff routinely emphasizes the maintenance of consistency and fairness in the examination process, those objectives are generally subordinate to the overriding goal of maintaining reactor safety by ensuring that only qualified and proficient applicants are licensed to manipulate the controls. The minimum event/scenario requirements specified in ES-301 are meant to ensure that examiners will have sufficient performance data to evaluate the applicant on every competency applicable to their license level. Exceeding the minimum requirement by one scenario is not a significant consistency issue; moreover, a reasonable amount of variation in the number and level of difficulty of events and scenarios between one applicant and another is to be expected and does not invalidate the examination process. A properly-trained and competent license applicant should easily be able to pass the operating test regardless whether it contains two or three scenarios. Additionally, the staff generally prefers to use surrogates only when necessary because they are specifically briefed on the scenario contents and could inadvertently affect the operating test outcome and complicate the licensing decisions.

Therefore, surrogate operators will not be permitted to replace applicants solely because these applicants have performed the minimum event/scenario requirements.

53

54 ES-303 Grading operating tests for initial licensing examinations 303.1 There are no longer going to be prescripted follow-up questions for job performance measures, but job performance measure questions will be evaluated - please explain.

Revision 7 of NUREG-1021 required every system selected for evaluation in the walk-through operating test to be examined with a job performance measure, at least two prescripted questions, and additional follow-up questions as deemed necessary by the examiner to investigate the applicant's performance deficiencies. Although Revision 8 of NUREG-1021 eliminated the prescripted questions, examiners are still required to ask for-cause follow-up questions, if necessary, based on the applicant's performance and to consider the applicant's answers to those questions in the grade for the applicable system. (Refer to Section D.2.a of ES-303.)

303.2 ES-303 needs more specific documentation for final results (i.e., some way for very specific feedback to candidate).

Comment noted. Section D.3.c of ES-303 requires examiners to document every deficiency noted during the operating test. However, only those deficiencies that contribute to a test failure need to be justified in detail. The test report is not intended to be a retraining vehicle; the facility licensee should be able to take the information provided and develop more specific feedback and training for the applicants.

303.3 Will operating test follow-up questions be documented?

Can they fail an applicant even though he accomplished the critical step (task)?

Yes. Section D.2.f of ES-302 requires examiners to document all performance-based questions and answers for later evaluation.

Yes. Per Section D.2.a of ES-303, an applicant could fail even though all the critical steps were accomplished. The examiner must justify the basis for the unsatisfactory grade in accordance with Section D.3 of ES-303.

54

55 303.4 What is meant by "critical task errors are not essential?"

With regard to the dynamic simulator operating test, it means that an applicant does not have to miss a critical task to justify a low grade on a rating factor or an overall failure of that test (as explained in Section D.2.b of ES-303).

With regard to the systems walk-through, it means that an examiner can ask performance-based follow-up questions even if the applicant was able to perform every critical step and accomplish the task standard (as explained in Section D.2.f of ES-302). Moreover, per Section D.2.a of ES-303, an examiner can recommend an unsatisfactory grade for a system based on the follow-up questions even if the applicant completed all the critical steps. The examiner must justify the basis for the unsatisfactory grade in accordance with Section D.3 of ES-303.

303.5 Is there written guidance on pass/fail for non-prescripted questions?

Yes. Section D.2.a of ES-303 (in NUREG-1021) describes how examiners will grade the job performance measure follow-up questions. NRC examiners bear the burden of justifying an unsatisfactory grade for the system if the applicant was able to accomplish the task standard.

Both the chief examiner and the regional operator licensing branch chief must also concur in the failure recommendation.

303.6 If a candidate is performing a job performance measure (JPM), and during the performance of the task performs an unsafe action with respect to personnel safety, does this constitute a failure of the JPM?

It may, depending on the safety significance of the applicant's action. Section C.2 of ES-303 allows the NRC examiner to recommend a failure if an applicant made an error with serious safety consequences even if the grading instructions in Section D would normally result in a passing grade. Normally, this would require adverse consequences related to reactor safety, however, it could also apply to personnel safety issues with potentially serious consequences.

Under such circumstances, the examiner shall thoroughly justify and document the basis for the failure in accordance with Section D.3.b. Moreover, the NRC regional office shall obtain written concurrence from the NRR operator licensing program office before completing the licensing action.

55

56 ES-401 Preparing initial written examinations 401.1 I do not feel that the written exam is a discriminatory tool. How many people do poorly on the written exam but are not weak on the operating test? Let us use our process to take care of the written with our audit exam.

Recommendation noted. As is evident from the transition program that was completed in 1999, the NRC is generally in favor of increasing power reactor facility licensees involvement in the examination process. Additional changes are possible if the NRC concludes that they will reduce unnecessary regulatory burden, increase public confidence, improve efficiency and effectiveness, and maintain reactor safety.

The NRC has not analyzed applicants' grades on the written exam and operating test to see how well they correlate. However, it is true that some applicants who fail the written examination do quite well on the operating test, while others who fail the operating test perform well on the written exam. The NRC believes that both parts of the licensing examination are important. As discussed in Section B.1 of Appendix B of NUREG-1021, the importance of knowledge testing (i.e., the written exam) should not be underestimated since knowledge is the underpinning of professional performance. The objectives of knowledge testing are varied; they may include assessment of fundamental understandings as well as testing more advanced levels of expertise. The most effective tests of knowledge include questions and test items that measure applications of knowledge directly related to the job. In the case of the NRC operator licensing examination, the written examination provides a key measure that allows a confident decision to be made on the safety significant performance of the individual seeking a license.

56

57 401.2 There are still occasions in NUREG-1021 for examination requirements that are subjective and, therefore, can (and will) vary from Region to Region and examiner to examiner.

What are the objective criteria for determining that an exam question is SAT[isfactory] or UNSAT[isfactory]?

The criteria for determining whether a written examination question is satisfactory are summarized on Form ES-401-9 and discussed in Appendix B of NUREG-1021.

The NRC acknowledge that some of the guidance in NUREG-1021 still requires examination authors, NRC examiners, and their supervisors to judge the level of knowledge, level of difficulty, quality of distracters, and other psychometric aspects of the examination.

Nevertheless, the NRC believes that writers of examinations and NRC examiners who are trained in the subject matter, measurement principles, and psychometrics, and who have general knowledge of operator and trainee performance on similar test items, can make informed judgments in these areas based on the guidance in NUREG-1021. Section II of of SECY-98-266, the paper that forwarded the final operator licensing examination rule change to the Commission for approval, responded to a similar comment.

401.3 How do we determine "level of difficulty" for written exam questions?

What is the process for determining the level of difficulty for a question?

Where can I find the criteria for the 1-5 difficulty rating on exam questions? Has any utility perfected the application of this?

A level of difficulty should be established that discriminates between applicants who have and have not mastered the required knowledge, skills, and abilities. Section C.3 of Appendix A and Section C.1.e of Appendix B discuss the concepts of discrimination validity and level of difficulty.

NRC examiners are required to rate the level of difficulty of every written examination question that has not been previously validated by the NRC at that facility. This is done using a 1-5 (easy

- hard) difficulty rating scale as specified on Form ES-401-9; questions in the 2-4 range of difficulty are acceptable.

57

58 401.4 Evaluate changing initial exam grading to a curve for pass/fail.

As noted in Section C.3.a of Appendix A of NUREG-1021, the NRC's initial and requalification examinations, like most licensing examinations, are criterion-referenced rather than norm-referenced tests. This means that there is a pass-fail or minimal cut score or grade that every examinee must achieve to demonstrate sufficient knowledge and ability to safely operate the power plant.

If the passing grade is determined by comparing each applicant's score with that of the group taking the examination at that time, an applicant who scores in the low 80s could fail if all the other applicants score above 90%.

401.5 If the utility is producing the written exam, when (how many days/weeks) is your expectation for the chief [examiner] to get the sample plan to the utility? The point is -

getting the sample plan in accordance with NUREG-1021 will not work.

As stated in ES-201, the examination outline should normally be completed at least 75 days before the scheduled examination date. The actual due dates must be negotiated and confirmed with the NRC Regional Office (chief examiner and/or branch chief) at the time the examination arrangements are confirmed (refer to Section C.2.c of ES-201). If the facility licensee needs more than 75 days to prepare an examination based on an NRC-developed outline, it needs to work out the schedule with the Regional Office.

58

59 401.6 Clarify what you mean by "random selection." Does the random selection have to go all the way down to the specific K/A number?

For purposes of the NRCs licensing examination, random means without bias or predisposition.

Yes. Section D.1.b of ES-401 in NUREG-1021 requires the K/As to be systematically and randomly selected from the applicable NRC K/A catalog (NUREG-1122 or -1123. Attachment 1 of ES-401 describes a sample method for selecting K/As, with Step 4 specifically instructing that the K/A statements within each randomly selected K/A Category will also be randomly selected.

If you select a K/A that is not applicable to your plant or that has an importance value less than 2.5, you may have to randomly select another K/A statement. Failure to train on a selected K/A is not an acceptable basis for selecting another one. If you determine, when reviewing the completed outline in accordance with Section D.1.d, that one of the K/A Categories is over- or under-sampled, you should randomly select another K/A. In accordance with Section D.2.f, if your question bank contains more than one question applicable to the selected K/A and there is no appropriate basis for selecting a specific question (e.g., cognitive level, discrimination validity, operational orientation) it would be best to randomly select from among the questions rather than chose the same question every time.

In accordance with Section D.1.b, facility licensees shall describe for the NRC the process that was used to generate the examination outline and the reasons for rejecting any randomly selected K/A statements.

59

60 401.7 What do you do if your randomly selected questions identify a K/A that you know was not trained on or has been deselected for training? Do you ask it anyway or do you select another system or does it go deeper?

Can you change a K/A if no one can write a question for it?

What if a random K/A [knowledge or ability] cannot be used to prepare a discriminating question? Is it fair to replace the K/A with one that is more difficult? (Can we throw out a K/A simply because it is too hard to write a discriminatory question?)

Section D.1.b of ES-401 (in NUREG-1021), allows the examination author to systematically and randomly select another K/A category and/or statement, as applicable, if the systematic selection process identifies a K/A statement having an importance rating that is below 2.5, a K/A statement that clearly does not apply to the subject facility, a generic K/A statement for which it would not be possible to develop a Tier 1 or Tier 2 question, or a K/A category that contains no K/A statements. Failure to train on a selected K/A is not an acceptable basis for selecting another one. The author should use Form ES-401-4, "Record of Rejected K/As," or an equivalent, to document the basis for excluding from the examination outline any K/A statements that were randomly selected, and submit the form to the NRC with the completed outline.

As stated in Section D.2.a of ES-401, if it becomes necessary to deviate from the previously approved examination outline, the facility contact is expected to discuss the proposed deviations with the NRC chief examiner and obtain concurrence. The facility should be prepared to explain why the original proposal could not be implemented and why the proposed replacement is considered an acceptable substitute.

401.8 Regarding ES-401 and the random selection of K/A's: How do you document obvious non-applicable K/A's to the chief examiner? Can we remove them prior to the random selection or do we select and then drop (with documentation) from the sample plan?

As stated in Section D.1.b of ES-401, facility licensees can reject and explain K/As as they prepare the exam outline, pre-screen the entire K/A catalog (NUREG-1122 or -1123) to eliminate inapplicable K/A statements before beginning the random selection process, or take a combination of these approaches when preparing the examination outline. If the facility licensee decides to pre-screen the K/As, it should make arrangements for the NRC regional office to review the associated documentation and justification prior to submitting the examination outline. Form ES-401-4,"Record of Rejected K/As," or an equivalent should be used to document the basis for rejecting/deselecting K/As.

60

61 401.9 How close does model have to be to actual?

As stated in Note 2 on the bottom of Forms ES-401-1 and 2 of NUREG-1021, the actual point totals for each group and tier on the proposed examination outline must match those specified in the applicable table. However, the final point total for each group and tier, based on revisions required by the NRC reviewers, may deviate by 1 from that specified in the table. The final RO exam must total 75 points and the SRO-only exam must total 25 points.

401.10 After systematically/randomly generating a sample plan you discover it is lopsided in one area, how do you "balance" the exam? Where do the questions come from?

If, for example, the systematic/random outline for Tier 2 ends up with 7 items under Category K1 and only 1 item under Category K4, you can balance the coverage by randomly deselecting one of the items in Category K1 and then randomly selecting a replacement item for the same system from Category K4. If Category K4 for that system does not include a K/A with an importance rating of 2.5 or higher, you can randomly select another system within the same group. Always remember to document and justify any changes in accordance with Section D.1.b of ES-401. The questions used to implement the outline once it is approved by the NRC shall be taken from the bank, modified from bank questions, or newly developed in accordance with Section D.2 of ES-401.

401.11 Tech[nical] spec[ifications] (TS) are too complicated to memorize. They should be open reference or better yet covered by the operating exams (JPM). We do not want our operators to spend valuable time memorizing TS, nor do we want them to operate from memory.

The NRC does not expect operators to memorize the TS, nor does it endorse operating the plant from memory. However, the NRC does expect operators to recognize TS entry conditions, immediate actions, and (in the case of senior operators) bases when presented in a multiple choice format on the written examination. If they do not compromise the integrity of other questions on the exam, it is acceptable to provide extracts from the TS to the license applicants for use in answering application-level questions.

61

62 401.12 Based on the SAT-based training program, you test on objectives. The current NUREG-1021 allows asking questions not covered by the utility's training program (objectives).

This is contrary to the SAT-based training system. Should there be a way to ensure the students are examined on the training program content? (If it is determined that the program is SAT.)

Learning objectives are not required for the NRC examination, but our SAT-based program still requires them. Do we no longer follow our SAT-based program? (Section II) to SECY-98-266, the Commission paper associated with the April 1999 final rule, responded to a similar public comment on Interim Revision 8 of NUREG-1021.

It notes that Sections 55.41(a), 55.43(a), and 55.45(a) of the rule state that the knowledge, skills, and abilities selected for evaluation on a written examination and an operating test will be identified, in part, from learning objectives derived from a systematic analysis of licensed RO and SRO duties performed by each facility licensee. While the answers to Questions 129 - 130 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses," confirmed the NRC's intent that the training program's learning objectives would become the major source of the licensing examination, it also cautioned that the NRC would not be limited to those learning objectives.

The NRC licensing examination is not a part of the facility licensee's SAT-based training process. The systematic sampling procedures for preparing the written and walk-through examination outlines per NUREG-1021 are designed around the structure of the NRC's K/A Catalogs (NUREG-1122 and -1123) and may not be compatible with the facility-specific task lists. NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific knowledge and ability requirements on a case-by-case basis. Allowing facility licensees to substitute their entire site-specific task lists for the NRC's K/A Catalogs could decrease the level of examination consistency. The current approach of requiring facility licensees to explain deviations from the NRC's K/A Catalogs is conservative, consistent, and effective.

Facility licensees should continue to follow their SAT-based training programs, with the understanding that the content of the NRC licensing examination is not necessarily restricted by the SAT-based training process. Licensees should consider developing learning objectives covering all the topics required by 10 CFR 55 and all the NRC K/As having importance ratings of 2.5 or higher, unless it can demonstrate that the K/A is not applicable at their facility.

62

63 401.13 If learning objectives say that, ". . . given a copy of procedure," can we use as closed reference [question]?

In accordance with Section D.2.g of ES-401, a facility learning objective is not necessarily required for every question. However, if one is referenced it should be adhered to unless the licensee makes a conscious decision to deviate from it. In those cases, the licensee should consider revising the learning objective to match the question.

The NRC does not review every learning objective during the approval process. When a question appears on the examination, the NRC will conclude that the facility licensee expects its operators to be able to answer the question without a reference regardless what the learning objective says. If such a question is challenged during a license appeal, the NRC may ask the facility licensee to support the question in writing as discussed in Section C.2.a of ES-502.

As noted in Section D.2.g of ES-401, reference materials may be used on a selective basis as attachments to the written examination, provided they do not give away the answers to any of the questions or improve the applicants chances of guessing the correct answer by eliminating incorrect distracters.

401.14 The definition of knowledge based versus higher order is not clear. Explain.

Sections C.3.c of Appendix A and C.1.d of Appendix B of NUREG-1021 discuss Bloom's Taxonomy and briefly explain the three levels of knowledge (i.e., fundamental knowledge or simple memory; comprehension; analysis, synthesis, or application). Attachment 3 of Appendix B cites Benjamin Bloom's book on the subject as a reference tool that explains the concept in greater detail.

401.15 Once we use a comprehensive level question, does it become a knowledge based question the next time we use it?

No. As stated in Section D.2.c of ES-401, the cognitive level of any question taken from the bank will be counted at its face value, even though it may function at a lower level because it is available for study (refer to Section C.3.e of Appendix A of NUREG-1021).

63

64 401.16 Regarding the ES-400 series. Discrimination validity should not be evaluated separate from operational validity and content valid. If operational validity and content validity are present, then discrimination will be present if good test item writing principles (e.g.,

plausible distracters, absence of clues) are applied.

Remove level of difficulty evaluation from Form ES-401-9 and all other requirements.

There is no need to assess difficulty if content validity, operational validity, and 50-60 higher cognitive level requirements are met.

Why is it unacceptable to have a question with a difficulty rating of "1," if that is what the randomly generated sample plan called for?

Comments noted. However, to determine whether an item has discrimination validity you must ask yourself whether an applicant who has not achieved the minimum level of competence is likely to miss the answer and be drawn to a distracter. Questions can be psychometrically sound, content valid, and operationally valid, but still not discriminate well. Refer to Section C.3 of Appendix A of NUREG-1021 for a discussion of discrimination validity.

The sample plan does not prescribe the difficulty level for questions; rather, the sample plan determines the topical content areas from which test items will be developed. Moreover, K/A importance values should not be confused with item difficulty measures. Easy questions can be created from high importance K/As and difficult questions can be created from low importance K/As.

401.17 Why is a validated question not a good question?

Although a question that was previously used on an NRC examination at the facility since 10/1/95 (i.e., a validated question) may be acceptable in its own right, it may have to be edited or replaced if it conflicts with another question on the examination or if necessary to meet the criteria on the Written Examination Quality Checklist (Form ES-401-6 in NUREG-1021).

Technical and psychometric flaws that cause the question to have no or multiple correct answers would have to be corrected regardless when they are identified.

401.18 NRC validated questions used on previous license examinations at the facility will get limited review. What about questions on similar units?

If it was deemed a satisfactory question by NRC is it "automatically" satisfactory for any facility? (Assuming the question is valid)

The current policy (per Section E of ES-401) is that examiners will review in detail all questions that have not been validated at that facility. Questions previously used on exams at similar units will be reviewed in detail.

64

65 401.19 Administrative-type items are best suited to open-referenced method because of the expectation for these items in the actual job position. However, the written examination, a closed-reference format, has a significant percentage of administrative questions. This appears contradictory.

10 CFR 55.41(a) and §55.43(a) require the written examinations for operators and senior operators to sample a number of administrative topics. Per ES-401 of NUREG-1021, such questions make up approximately 13 percent of the RO examination and 28 percent of the SRO examination. The administrative questions that are used on the written exam should generally be answerable based on recall and/or recognition. As discussed in Section D.2.g of ES-401, under certain conditions, selected reference materials may be provided to the applicants as attachments to the written examination.

401.20 How large must the exam bank be before you can select 50 questions from it for use on an exam?

Is there a bank size limitation for use of 50 questions?

How can facilities maximize use of bank question (up to 50) if they don't fit the sample plan? Recommend systematically selecting the first 50 questions from bank, then systematically selecting remaining K/As to complete outlines. Could also select 40 questions from bank systematically for modification.

The NRC is not controlling the size of examination banks. The limits on bank use in Section D.2.f of ES-401 apply to every facility licensee, regardless of its bank size. However, from a practical standpoint, the larger the licensee's bank is, the more questions will match the systematically and randomly selected sample plan, and the fewer questions the licensee will have to modify or develop. The national examination question bank being maintained by the Institute of Nuclear Power Operations should greatly enhance licensees' ability to find bank questions that fit their systematically developed sample plans. Recommendations noted.

65

66 401.21 We are allowed to use 50 questions from the exam bank (including 25% exact repeats from the last two exams and quizzes), 40 modified questions, and 10 new questions.

In theory we would only need to write 10 new questions. This reduces burden for the exam writer, and reduces difficulty on the student. In reality, students generally are exposed to the entire exam bank during the program so the "50" becomes 25. Also, with the lottery (systematic-random) method of choosing K/As, the likelihood of having more than a handful of repeat or modified questions.

Recommend allowing exam writers to randomly select the 25 repeats and 40+ for modification by pulling questions randomly from all questions asked of the students during the program.

Comment and recommendation noted.

The NRC has made no effort to control the size of licensees' examination banks, nor does it control the number of quizzes or questions asked of the students during their training program.

The proposed solution would certainly make it easier to prepare an examination, but it would also be a disincentive for licensees to ask any more than 65 questions during the training program.

The changes implemented with Supplement 1 to NUREG-1021, Revision 8 (refer to Section D of ES-401) raised the upper limit on the number of questions on an exam that can be taken directly from an examination bank from 50 to 75 percent. However, because only those questions that fit the systematic and randomly generated sample plan can be used on the examination, the practical limit on bank use is, for the time being, determined by the size of the bank from which the questions are drawn. Although facility licensees may have to develop more new and modified questions in the short term, the burden should decrease as the local and national examination banks grow is size.

401.22 Regarding ES-401, Section D.2.f, does a bank question that the students saw during their training program but is then modified (as defined in the standard) count against the 25 questions that can be reused from the last two NRC exams and training quizzes?

NUREG-1021, Revision 8 (Section D) eliminated the limits on repeating questions from previous quizzes and NRC examinations based on the random selection of specific K/A statements and strict adherence to the intent of the selected statements.

66

67 401.23 If a question is used at a different facility (IP2/IP3) what or where does this fall into the 50/40/10?

For questions taken from a non-facility specific exam bank (e.g., the national exam bank) the questions must be changed as appropriate to make them correct for the facility. In this situation, the question may be different than the original bank question, but may not meet the criteria to be a "modified question" and are also not "new". What should these questions be called and how should they be categorized on the ES-401-6 form?

In accordance with Section C.1.h of ES-201 (in NUREG-1021), a question that was obtained from another facility or the national exam bank and simply tailored or adapted to meet the specifications and terminology of your facility would be treated as a "bank" question. However, in accordance with Section D.2.f of ES-401, if you take a bank question and modify it (beyond terminology changes) by (1) changing one or more of the conditions in the stem and (2) changing at least one distracter such that you have created a similar, but like kind item, then you can properly categorize it as a "modified" question. "New" questions, on the other hand, do not have their basis from an existing bank question. Rather, they have been developed from the author's "fresh start" and, as such, are categorized as "new." Note that the nominal question distribution criteria in ES-401 have changed from 50/40/10 to 75/15/10 percent.

401.24 If a bank is 100% pre-approved NRC exam questions and the utility modified these to make them site-specific by changing the stem or distracters, can the utility mark them as 100% modified?

The NRC considers all banks to be open and available for study by the license applicants.

Therefore, the questions can only be classified as modified for purposes of an NRC licensing examination if the modified versions are kept out of the bank until after they are used on an examination. They would only show up on an examination if they match a knowledge or ability that is part of the systematically developed sample plan.

67

68 401.25 At what point does a "modified" question become a "new" question?

When has a written question been changed enough to be qualified as a NEW question on the written initial exam?

Can we clarify the definition of a "significantly modified" question?

A modified question tests the same content topic as the original question but significantly alters the technical elements in the question (as discussed in Section D.2.f (last bullet) of ES-401) and gives it a different appearance. The intent of the modification is to preserve the focus and topic (i.e., the K/A reference) of the original question. If the question is created without reference to a bank question and has not been previously exposed at the facility, then it can be considered a "new" question.

Note that changing the conditions in the stem such that one of the three distracters in the original question becomes the correct answer would also be considered a significant modification.

401.26 Can the NRC provide examples of "significantly modified," and "psychometric flaw,"

questions in an attachment to NUREG-1021?

Appendix B of NUREG-1021 already contains a number of example questions that illustrate psychometric flaws commonly seen on NRC examinations. The NRC encourages the use of industry-sponsored item-writing workshops as a venue for obtaining and sharing this type of information.

68

69 401.27 With a National Exam Bank, how should utilities address number of questions from bank, modified, or new?

If [the Institute of Nuclear Power Operations] INPO creates a national initial licensed operator exam bank, will the NRC consider the INPO bank to be current questions that cannot be used as new questions on the exam to be developed?

If INPO develops/maintains a national exam bank, what will be the limitations associated with this bank? i.e., will exams still be subject to the 50/40/10 criteria? If so, can 50% of the questions come from the bank? Current NUREG-1021 guidance allows NRC review for "obvious flaws" for exam questions used on NRC exams since October 1995, "at that facility." How will this affect NRC review of exam questions that are part of the national exam bank used at other facilities? What type of security restrictions will be placed on the bank?

Is there a current effort to share "opened and published" exam banks between utilities?

If not, who would be interested in this?

On Form ES-401-6, it is required to categorize questions as to the number questions from the bank, modified, or new. For questions taken from a non-facility specific exam bank (the INPO bank for example) the questions must be changed as appropriate to make them correct for the facility. In this situation, the question may be different than the original bank question, but may not meet the criteria to be a "modified question" and are also not "new." What should these questions be called and how should they be categorized on the ES-401-6 form?

The NRC reassessed its policies regarding bank use based on the results of the Revision 8, Supplement 1 trial examinations. In accordance with Section C.1.h of ES-201, questions obtained from any bank will now be treated as "bank" questions. However, only those bank questions that are previously validated at that facility will be eligible for reduced review by the NRC. In accordance with Section D of ES-401 the upper limit on the number of questions on an exam that can be taken directly from an examination bank has been raised from 50 to 75 percent.

Other than the National Examination Bank being developed by INPO, the NRC is not aware of any utility initiatives to share banks. The regional training organizations, owners' groups, Nuclear Energy Institute, and INPO might be able to provide more information in this area.

The use of an INPO bank item - by tailoring or adapting the item to meet the technical specifications of your utility for examination use - is an acceptable and appropriate step toward meeting both technical and psychometric validity. As such, this kind of bank item adaptation results in an item that remains a BANK item and should be categorized as a BANK item. In this instance, you have not MODIFIED the item, as per the definition (ES-401, Section D.2.f), nor can you consider it to be NEW since it has been drawn from the INPO bank. The difference lies in the degree of change you make to the bank item. We expect utilities to make some adaptations to BANK questions so as to fit the logical terminology (stem and distracters) for its own utility. In such cases, you still have a BANK item. However, if you use a bank item and 69

70 modify it (beyond nomenclature changes) by (1) changing one or more of the conditions in the stem and (2) changing at least one distracter such that you have created a similar, but like kind item, then you can properly categorize it as a MODIFIED item. NEW items, on the other hand, do not have their basis from a drawn bank test item. Rather, they have been developed from the author's "fresh start" and, as such, are categorized as NEW.

401.28 In light of the NRC's new goals of reducing unnecessary regulatory burden and increasing efficiency and effectiveness, would it be possible to allow a licensee to build an initial license exam entirely from the bank (rather than 50% new questions), assuming the bank was an appropriate size and security concerns could be solved?

The NRC continues to believe that every examination should have some new and/or modified questions. Based on the results of the Revision 8, Supplement 1 (of NUREG-1021) trial examinations, the NRC staff raised the upper limit on bank questions to 75 percent, with the remaining questions being either new (at least 10) or modified bank questions (refer to Section D.2.f of ES-401). However, because only those questions that fit the systematic and randomly generated sample plan can be used on the examination, the practical limit on bank use is, for the time being, determined by the size of the bank from which the questions are drawn.

Although facility licensees may have to develop more new and modified questions in the short term, the burden should decrease as the local and national examination banks grow is size.

401.29 Regarding ES-401: How do you assure that the extra 10 CFR 55.43 topics are covered in a "representative sample" in the test outline?

The SRO-only examination outlines sample only those K/A categories that are linked to 10 CFR 55.43(b), including a number of the generic K/As in Section 2 of the catalogs (NUREG-1122 and

-1123) and all of the Category A2, AA2, and EA2 K/A statements. All the K/A categories related to the fuel handling facilities are also subject to sampling because that system is specifically identified in 55.43(b)(7). As stated in Section D of ES-401, the specific topics to be sampled on the examination shall be systematically selected.

401.30 Regarding ES-401, Section D.2.d: Cannot write SRO only questions for all seven items listed under §55.43(b). Only three items lend themselves to SRO only type questions.

Need multiple examples and training for writing SRO only questions for all seven items.

The NRCs initial response to this question indicated a commitment to look into the quality and consistency of SRO-only questions and [the NRC] may develop additional guidance in this area. The review was completed in March 2010. The clarification guidance for SRO written exam items is located at http://www.nrc.gov/reactors/operator-licensing/op-licensing-files/sro-only-ml1007100031.pdf and provides examples of SRO exam items for each of the seven 10 CFR 55.43(b) topics.

401.31 If an instructor has used bank questions, is there a restriction from using them on an examination?

70

71 Yes. Although Supplement 1 to Revision 8 of ES-401 (in NUREG-1021) eliminated the limits on repeating questions from previous training quizzes and NRC examinations, the facility licensee still has to take measures to ensure that the final audit or screening examination and any quizzes that are given after beginning work on the licensing exam do not compromise the integrity of the licensing exam. Refer to Section C.1.f of ES-401 for examples of acceptable control measures.

401.32 When an instructor writes questions, are they no longer allowed to use them?

If an instructor writes a question with the intent of using it as a new question on the next NRC examination, then it cannot be used. If an instructor simply writes questions for the bank, they would be treated as any other bank item and can be used on other examinations. Theoretically, all the questions in the bank should have an equal probability of being selected for the NRC exam. They would be counted as bank items and would be subject to the other criteria in NUREG-1021 (e.g., repetition from the audit exam).

401.33 Does the licensee need to supply names, positions, etc. of validation team prior to using them to review the exam? From ES-401, Section E.4, regarding certain individuals for exam validation: What is a "supervisor or co-worker?" This could be any licensed operator.

Section E.4 of ES-401 discourages facility licensees from using certain individuals to validate the written examination. The applicants' supervisors and coworkers may not be the most appropriate to use for exam validation because it would raise concerns regarding the potential for examination compromise. Moreover, in accordance with Section D.2.b of ES-201, individuals having knowledge of the examination contents are prohibited from performing a number of activities, including all on-the-job training, practice, coaching, and sign-offs. Although licensees are not required to obtain NRC concurrence before placing personnel on the security agreement, it would be prudent to assess the security risk and discuss any questions with the NRC chief examiner. The supervisor/coworker connection would be of most concern for ROs seeking to upgrade their licenses.

401.34 For 5 hour5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> exams, do the exams need to be time validated for 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> (i.e., does the exam have to be made more difficult because the time has been extended?)

No. The exams do not have to be made more difficult. Section D.2.c of ES-401 (in NUREG-1021) indicates that the examination should be designed so that competent applicants can take and review it within four hours, the same as before. Moreover, Section D.4.d of ES-402 has been revised to increase the nominal time limit for the RO exam to 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> in order to reduce the need for interaction with the NRC regarding minor time extensions and to ensure that the applicants are not time-limited when taking the exam.

71

72 401.35 Certain "newer" K/As have a 10 CFR 55 reference given in parenthesis to show a tie between the CFR and NUREG-1122(1123). We were told that questions did not meet the criteria of SRO only (those 25 questions only on the SRO written) if the K/A reference included both 10 CFR 55.41 and §55.43. It is our understanding that questions need be written at SRO knowledge level in these situations. We do not think that this dual CFR reference should be interpreted to eliminate the K/A from being selected for an SRO question.

The policy regarding the 25 SRO-only questions on the written examination is stated in Section D.2.d of ES-401. The fact that a K/A is linked to both 10 CFR 55.41 and §55.43 does not mean that the K/A cannot be used to develop an SRO-only question. Questions related to §55.41 topics may be appropriate SRO-level questions if they evaluate knowledge and abilities at a level that is unique to the SRO job position as determined by the facility licensees learning objectives. Although your observation is valid, please note that NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific K/As on a case-by-case basis and to use K/As having importance ratings below 2.5 if it is justified based on plant-specific learning objectives.

When the NRC revised NUREG-1122 and -1123 to incorporate cross-references to specific items in 10 CFR 55, the primary purpose was to establish at least one regulatory connection for every K/A. The fact that a particular K/A does not reference 55.41 or 55.43 does not, in and of itself, disqualify the K/A from testing on the RO or SRO written examination.

72

73 401.36 According to ES-401, the 25 "SRO-level" questions on the written examination shall be derived from the seven areas in 10 CFR 55.43. However, this guidance is sometimes being misinterpreted such that questions testing 10 CFR 55.43 topics are being rejected as "SRO-level" if the facility licensee also expects ROs to possess the same 10 CFR 55.43 knowledge. Is it correct to say that an "SRO-level" question is simply different from the questions on the RO examination and related to one of the seven items listed in 10 CFR 55.43 (b)?

The fact that a facility licensee expects its ROs to master certain 10 CFR 55.43 knowledge, skills, and abilities does not mean that they can no longer be used as the basis for "SRO-level" questions. However, ES-401 also requires questions to be "appropriate for the job level being examined." Therefore, "SRO-level" questions need to be carefully constructed to ensure that they accurately test the additional knowledge and abilities required for the higher license level according to 10 CFR 55.43(b). For example, both 10 CFR 55.41(b)(10) and 55.43(b)(5) require emergency operating procedure (EOP) knowledge, but the latter requires the "SRO-level" questions to evaluate the additional knowledge and abilities necessary for "assessment of facility conditions and selection of appropriate procedures during ... emergency situations."

Questions that evaluate the knowledge of specific bases for EOPs (K/A 2.4.18) and/or the operational implications of EOP cautions (K/A 2.4.20), but not the higher level "assessment and selection" knowledge, would generally not be valid "SRO-level" questions. However, questions that evaluate K/A number 2.4.21 (knowledge of the parameters and logic used to assess the status of EOP safety functions) would generally be considered valid "SRO-level" questions even if the facility licensees SAT-based program has identified this additional 10 CFR 55.43(b)(5) knowledge as an RO job requirement. Consequently, questions that test knowledge and abilities per 10 CFR 55.43(b) can be considered "SRO-level" per Section D.2.d of ES-401 even though the facility licensees training program requires the same level of knowledge for its ROs.

401.37 ES-401 does not address using a K/A that references 10 CFR 55.43 for testing on the RO written examination; is that acceptable?

Yes, it is. 10 CFR 55.41(a) states that "the knowledge, skills, and abilities [to be tested on the RO written examination] will be identified, in part, from learning objectives derived from a systematic analysis of licensed operator duties performed by each facility licensee and contained in its training program." Although ES-401 does not specifically address using a K/A linked to 10 CFR 55.43 to develop an RO written examination question, it does allow the facility licensee to use plant-specific priorities (and a site-specific task list) to justify using an otherwise unimportant K/A for questioning. Therefore, questions associated with topics in 10 CFR 55.43(b) should be acceptable for the RO examination if they are supported by documented RO learning objectives derived from the RO job task analysis at the site.

73

74 401.38 Why are we testing abilities and/or skills on the written exam vs. on the simulator exam (via K/A [knowledge and abilities] catalog)? Shouldn't we test knowledge on the written exam and abilities on the operational portion of the exam?

This question suggests that there is a dichotomy between knowledge and skill testing, when, in fact, knowledge and skill are interrelated, and testing in one format does not preclude assessing understanding in the other format. Although skills and abilities testing is more commonly associated with JPMs and simulator scenarios, it is incorrect to assume that they cannot be tested on the written examination.

Good test items, whether part of a written examination, walk-through, or simulator scenario, should be operationally valid. You should not assume that written questions are passive items where only facts, principles, or concepts are recognized. Ideally, they should assess the applicants' ability to integrate and use information on plant conditions. For example, such questions could require the applicant to use information in the stem of the question to determine appropriate actions or predict system responses. These "scenario style" questions are dynamic in nature, requiring the applicant to sort, merge and integrate contrived, but possible conditions.

They assess at the application level of operator action -- a quality consistent with Bloom's Taxonomy (see Appendix A and B of NUREG-1021) and the goal of attaining high operational validity. To this extent, operator knowledge and skill are simultaneously embedded within the written test questions.

When it is not possible to test a randomly-selected skill or ability on the written examination, then another K/A should be randomly selected. However, as stated in Section D.1.b of ES-401, the facility licensee shall provide written justification for replacing any randomly selected K/A.

401.39 What would it take to go back to (pre-revision 8) a site-specific K/A [knowledge and abilities] catalog in line with the SAT-based [systems approach to training] process?

Before Revision 8 of NUREG-1021, when the NRC and its contractors prepared all of the licensing examinations, the NRC determined what K/As would be tested. The NRC generally used NUREG-1122 or -1123 (which are based on a generic job task analysis performed by the Institute of Nuclear Power Operations with importance ratings established by a panel of industry and NRC subject matter experts) to ensure that the examinations were content-valid, but site-specific catalogs were permitted on a case-by-case basis. Now that facility licensees are preparing most of the examinations and determining what K/As will be tested, the NRC believes that certain measures are necessary to ensure that consistency and public confidence are maintained. The NRC staff believes that it would be inappropriate to give licensees complete control over the content of the training program as well as the licensing examinations. As explained in response to Question 401.12, NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific K/A requirements on a case-by-case basis if they are justified and agreed to by the NRC chief examiner.

74

75 401.40 Why does a group with only 1 or 2 safety-significant K/A's [knowledge and abilities] have as much weight as one with 200? Can [NUREG-]1021 be changed to remove this artificiality?

The NUREG-1021 superstructure forces you to sample the systems K/A of about the same rate (1 or 2) per system. However, some systems have 5 K/As that are above 2.5 and some have 200. This forces you to over-sample some systems and under-sample others. Can the superstructure be realigned to eliminate this problem by lumping all the system K/As together and selecting the number needed from the total?

The relative safety-significance of the plant systems and emergency/abnormal plant evolutions (E/APEs) was considered by the team of industry and NRC subject matter experts that originally designed the 3-tiered written examination sample plan (as part of NUREG/BR-0122 "Examiners' Handbook for Developing Operator Licensing Written Examinations") that is still in use in ES-401 of NUREG-1021. For example, Tier 1 of the PWR RO sample plan is broken down into two groups of E/APEs that make up 24 and 12 percent of the exam, but include 22 and 34 items, respectively. The more important items that are included in Group 1 are weighted much more heavily than the items of lesser safety significance that are included in Group 2.

If all the K/As were lumped together, some of the stratified system categories that presently exist could go unsampled. This would bias the exam and reduce the number of areas tested and reduce exam validity. Moreover, the strict guidance of ES-401 helps to make exams more uniform between the different groups that develop them. Examinations should differ only in the specific content covered, not in their development process, manner of sampling, item construction criteria, level of item bank use, or their levels of knowledge and difficulty.

401.41 Do practice exams late in the program have to be accounted for in the exam overlap restrictions?

That depends on whether they are developed before or after the facility licensee begins working on the licensing examination. Although NUREG-1021 has eliminated the restrictions on repeating questions from training quizzes and the past two licensing examinations, the facility licensee must still take measures to ensure that the audit exam and any other quizzes developed after starting work on the licensing exam do not compromise the integrity of the exam. Section C.1.f of ES-401 provides examples of acceptable control measures.

75

76 401.42 Why is it valid to use a closed reference exam for initial license exams when it is really important that the operator use all of the tools available to him on shift? Where is the NRC headed on the use of open-reference requalification questions on initial exams?

Open-reference items on the initial license examination should be used judiciously and sparingly because the examination should focus on the broader content areas that rely primarily upon learned information, committed to memory.

In nearly every field of study (e.g., medicine, law, and education), the testing required for initial licensing or certification is more demanding than that required to maintain certification. The rationale is that newly licensed personnel should possess a broad body of knowledge and ability to perform their job independently and without the aid of supplemental knowledge contained in procedures. This by no means suggests that procedures should not be used, but rather that initial license testing should emphasize those areas where procedures need not be used.

Through their training, operators must learn set points, immediate actions, system designs and interrelationships, administrative procedures, and applications of knowledge to the job. The knowledge that is learned is expected to be demonstrated through the NRC examination format that measures recognition and recall of safety-significant knowledge without relying on references. This approach is consistent with the timely retrieval of information that may be required during the licensed operators' job and that might otherwise not be possible if the applicants prepared only for open-reference examinations. If too many open-reference questions are allowed on the initial licensing examination, the need and ability to learn and retrieve a broad body of knowledge would be lessened. Similarly, the confidence that the baseline body of knowledge had been truly established could be questioned.

Once initial competency is assured, then ongoing training and testing, which is more review-like, focused and specialized in nature, can make more appropriate use of the open-reference format, as is done on requalification examinations. However, for the reasons stated above, the NRC does not plan to increase the limited and judicious use of open-reference questions on the initial license examination.

76

77 401.43 Can 25 questions from the previous 2 NRC exams be used if randomly generated without modification?

Yes. However, in accordance with Section D.1.b of ES-401 (in NUREG-1021), the specific K/A statements (e.g., K1.03 or A2.11) for the examination outline must be selected in a truly random fashion (as verified by the NRC chief examiner) and the questions selected to implement the outline must clearly match the intent of the selected K/A statements (which will be verified on a sampling basis by the NRC chief examiner).

Given the number of K/A statements in the testable population, the NRC staff believes that it is extremely unlikely that a random selection process would result in that many duplicate questions. Per Item 4 on Form ES-401-6, the NRC will review the facility licensees sampling process to ensure that it was random and systematic if more than 4 RO and 2 SRO-only questions are repeated from the last two NRC licensing exams.

401.44 With a completely random process, the generic K/As tend to get over-sampled (about 30%) on the written exam. Since the administrative section of the operating test is all generics, they tend to get way over-sampled. Can the generics be eliminated from the plant systems and emergency/abnormal plant evolutions (E/APE) tiers?

Revision 0 of the NRCs K/A Catalogs (NUREGs-1122 and -1123) included a list of system-generic K/As at the end of every system and E/APE. Those K/As were sampled as part of Tiers 1 (E/APEs) and 2 (plant systems) of the examination. When the NRC revised the K/A Catalogs, the system-generic K/As were subsumed in Section 2, Generic Knowledge and Abilities, but there was no intent to change the distribution of questions among the three tiers of the exam.

Consequently, the guidance in Section D.1.b of ES-401 indicates that only those generic topics that are relevant to the selected evolution or system will be included in the sample for Tiers 1 and 2. Section D.2.a of ES-401 further clarifies that the questions selected for Tier 3 shall maintain their focus on plant-wide generic knowledge and abilities and not become an extension of Tier 2, Plant Systems. If none of the generic K/As were testable in Tiers 1 and 2, it would not be possible to ask a system-specific technical specification question.

401.45 Is the following scenario acceptable for purpose of controlling any overlap from the audit to the NRC exam? 1. Audit exam is last years NRC exam. It was developed using randomly generated sample plan 1 year ago. 2. NRC exam is developed using randomly generated sample plan. Some overlap occurs in K/As tested on the audit and the NRC exam.

Yes. Since both examinations were randomly generated and presumably the questions match the selected K/As, it is acceptable. Some overlap may occur.

77

78 401.46 For a written retake exam, the subsequent audit exam focuses somewhat on identified weaknesses from the previous NRC exam. Therefore, the audit exam is not totally random. Is this acceptable?

How do we apply the audit/screening exam criteria for written re-exam efforts? Does an upgrade [remedial] program [for the applicant] exam count as an audit? Since 60 days have elapsed, does the initial audit exam fall into the bank question category?

ES-401 of NUREG-1021, Section C.1.f, discusses acceptable methods for ensuring that the audit exam does not compromise the licensing exam. The example given would be acceptable if the audit exam is finalized before the NRC exam development is started or if there is no duplication between the audit and the NRC exam. As long as the NRC licensing examination is developed using the random and systematic process described in ES-401, there are no restrictions on repeating questions from any prior examinations and quizzes, including old audit and licensing exams. Once an audit or any other exam is given, all the questions on that exam would be considered bank questions that could be used to evaluate the associated K/A if it is randomly selected for a subsequent examination. However, the content of any practice or audit exam or quiz that the facility licensee develops after it starts working on an NRC licensing examination would have to be controlled to protect the integrity of the licensing exam.

401.47 Has the K/A catalog been reviewed and each K/A evaluated for cognitive level? (Some appear to support only basic Level 1 questioning.)

Are fundamental K/As being eliminated from the K/A catalog? For example: the purpose of charcoal filters in iodine removal systems.

The knowledge and abilities in the NRCs K/A Catalogs (NUREG-1122 and NUREG-1123) have not been reviewed for cognitive level. The K/A catalogs were developed by a group of utility personnel and the NRC and only list knowledge and abilities with importance values related to performing licensed duties. K/As are topical content areas and should not be confused with the cognitive levels of test items; K/A importance values and cognitive level are separate and distinct exam development parameters. The fact that some of the K/As do not support the development of higher cognitive level questions does not make them unusable on the NRC licensing examination because ES-401 specifies that 40 to 50 percent of the RO questions will be written at the fundamental level of knowledge.

401.48 Is there any movement towards going to 3-part multiple choice questions vs. 4? The 4th distracter is very expensive, most times demanding more time than the others combined.

No, that is not being considered. The four-distracter format is the only one acceptable to the NRC. Refer to NUREG-1021, Appendix B, Section C.2.a.

401.49 K/A Categories A-3 Monitor Auto operation of ... and A-4 Manually operate ... dont seem to be well tailored to a written exam. These topics for the written exam are almost always 78

79 covered in K1-6, A1, or A2. Why not eliminate these categories from the NUREG-1021 superstructure since they are more properly tested by the operating test and the knowledge is already sampled by K/As in other categories?

Recommendation noted. However, questions can be written to test the applicants' ability/knowledge of proper automatic operation and how to manually operate a component or system.

401.50 Is NRC considering allowing the Institute of Nuclear Power Operations (INPO) to oversee the development and administration of the written test?

The Nuclear Energy Institute (NEI) at one time proposed that option for consideration by the NRC staff, but it was determined to be unworkable.

401.51 Why isn't the Control Room Ventilation System listed as one of the systems in the PWR KA catalog? Was this a deliberate omission? I noticed that Control Room Ventilation is included in the BWR catalog.

The NRCs K/A catalogs, NUREG-1122 [and 1123], Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized [Boiling] Water Reactors, are based on the job/task analysis (JTA) performed on the licensed operator position by the Institute of Nuclear Power Operations (INPO). The INPO JTA identified more than 28,000 knowledge, skills, and abilities (K/As) and nearly 800 tasks to be used as a basis for developing training programs applicable to all PWR and BWR facilities. The K/A catalogs were reviewed by licensed SROs as well as license examiners from the NRC. These experts reviewed each statement for accuracy and completeness and then rated each statement with respect to its importance to safe operation. Many of the INPO K/A statements were omitted from the NRCs K/A catalogs because they were too specific and/or too elementary for use in developing license examinations, or, more importantly, because they had little bearing on the safe operation of the nuclear plant - the job content that is of primary interest to the NRC.

The two K/A catalogs were developed independently and, consequently, had a number of significant differences. The PWR catalog was issued in July 1985, and the BWR catalog was issued in September 1986. Both catalogs were revised in 1995 and again in 1998 to incorporate links to the applicable 10 CFR 55.41-45 item numbers, and to reorganize and/or expand the generic K/A statements, the safety functions and plant systems, and the emergency and abnormal plant evolutions. Revision 1 added the component cooling water and instrument air systems (which were already covered in the PWR catalog) to the BWR catalog; however, no new systems were added to the PWR catalog. Without doing a significant amount of research into the archives, it would be difficult to say for sure whether the inconsistency you have raised was deliberate or coincidental.

As noted above, the K/As in NUREG-1122 and 1123 are but a subset of the total population of K/As that a license applicant needs to master to become a competent operator or senior operator. The fact that a particular K/A or system did not make it from the original INPO JTA into the NRCs K/A catalog does not justify its omission from a facility licensees systematically-79

80 developed operator training program, nor does it mean that the K/A or system is inappropriate for testing on the licensing examination. As indicated in 10 CFR 55.41-45, the K/As covered on the RO and SRO license examinations will be drawn, in part, from learning objectives derived from a systematic analysis of the operators duties performed by each facility licensee and contained in its training program. Although the control room ventilation system is not included among the 45 systems in the PWR catalog, K/As related to that system may still be selected for testing in connection with other systems (e.g., area radiation monitoring (ARM) system - K1.04),

abnormal plant evolutions (e.g., accidental gaseous rad waste release - AA1.02), and the generic K/As (e.g., 2.1.26 - knowledge of industrial safety procedures such as chlorine).

Moreover, note that Revision 9 of ES-401 (in NUREG-1021) requires test developers to add any operationally-important systems or E/APEs that pertain to the facility but are not included in the generic lists on Form ES-401-1 to the examination outline before selecting examination topics.

401.52 Are technical specification (TS) action statements that require action "within one hour" addressed by NUREG-1123, K/A 2.2.39? We have received different interpretations from different examiners. We believe that they are NOT since action could be taken at the end of sixty minutes and still be within compliance.

Although the "within one hour" TS action statements and K/A 2.2.39 were not identically worded, the TS action statements and the K/A 2.2.39 wording was equivalent in their intent and meaning. It is agreed that action might not be taken or initiated until 60 minutes have elapsed.

However, should that be the case, the requisite action or actions must also be completed by the end of 60 minutes. In other words, the knowledge required for the operators to properly complete the required system action statements is the same no matter if completed in 59 or 60 minutes. Therefore, the wording difference has been rectified by revising K/A 2.2.39 in NUREG

-1122 and NUREG-1123 to read less than or equal to one hour. Related clarification for valid testing of TS K/As and K/As without a related facility learning objective can be found in the responses to Questions 401.11 and 401.12.

80

81 401.53 Recently, guidance provided from examiners to facility authors indicates a limit on the quantity of questions that may have reference material(s) provided to the candidate for initial licensing written examinations.

Specifically no more than 50% on the SRO portion of the examination has been offered as guidance. However, NUREG-1021 does not provide specific guidance on the percentage of use for these type of questions, and many of the SRO K/As and station training objectives require analysis of conditions along with use of reference materials to determine the correct action(s). These questions demonstrate operational validity and discrimination for the types of knowledge and abilities an SRO is expected to possess.

ES-401, D.2.g states that [R]eference materials (such as diagrams, sketches, and portions of facility procedures) may be used on a selective basis as attachments to the written examination. Ensure that any reference material used in the examination is easy to read and clearly marked, provides an effective and objective way for the applicant to demonstrate knowledge of the topic or concept, and does not give away the answers to other questions on the examination or improve the applicants chances of guessing the correct answer by eliminating incorrect distracters.

Is there a limit on the percentage of written exam questions that utilize references for the initial licensing exam? If so, what is the basis?

Question 401.42 addresses the differences between initial license examinations and requalification examinations regarding the approach of "closed-book" versus the use of references during examinations. That question properly states that references in closed-book, initial license examinations should be used judiciously and sparingly.

With regard to the initial license examinations, the Agency goal is to ensure that applicants prepare and study a broad, yet defined, body of knowledge. Applicant mastery of such a body of knowledge, as required in 10 CFR 55.41 and §55.43 better ensures that operators will be equipped to address public health and safety needs that may arise in the conduct of their reactor operator (RO) and senior reactor operator (SRO) duties. When an examination relies too heavily on the use of references to answer questions, then the applicants preparation for such an examination is altered; the applicant likely will primarily focus his or her preparation on the use of references to answer questions and devote less attention on mastering the body of knowledge. In sum, the mental demands, requirements, and format of the examination will determine the applicants method of preparation; moreover, the level of preparation will likely be deeper and more thorough given the expectation of a closed-book, limited reference examination.

The RO and SRO initial license examinations largely and properly remain "closed-book;" current policy regarding the judicious and sparing use of references is appropriate. However, because of the supervisory nature of the SRO position that relies on a greater use of references and because of its separate 25-question examination that addresses content in 10 CFR 55.43, there is a justifiable basis to allow a greater number of references to be used on the SRO exam. Yet, in the spirit of Question 401.42, both RO and SRO exams, as initial license exams, should 81

82 similarly rely more heavily on knowledge memory and the application of this knowledge, which the NRC staff believes does not diminish operational validity.

In this regard, the following ranges are provided regarding the allowable use of references on initial license examinations consistent with the principles discussed in Question 401.42. Note that these quantitative ranges are not absolute limitations, nor should they be construed as goals or requirements. You should also note that NUREG-1021 does not permit any "direct lookup" questions or questions with references that provide an advantage in answering other "closed-reference" questions on the initial licensing examination.

RO (75 items) = up to ~5% or 4 questions SRO (25 items) = up to ~20% - 25% or 5 - 6 questions 401.54 According to ES-401, Attachment 2[1], for writing SRO level Tech Spec questions, one screening criteria is to determine if the question can be answered solely by knowing the

'above the line' information. RO candidates are required to know the LCO statement and the Modes of applicability. Is RO knowledge limited specifically to these words in the LCO statement, or is knowing the subparts of the system which makes it Operable also considered required knowledge? The T/S basis contains a statement for the LCO which usually lists the necessary components and lineups which would make a system Operable.

Knowledge of the bases information/discussion in Technical Specifications (TS) for limiting conditions for operation (LCOs) is not considered RO required knowledge with the following one exception. Knowledge of and ability to apply less than or equal to one hour TS action statements is considered RO knowledge. In this instance, RO knowledge is NOT limited specifically to the words in the LCO statement and the TS bases knowledge indicating the necessary components and lineups which would make a system [or component] Operable is appropriate for testing on the RO written examination or operating test.

In summary, application of knowledge contained within the TS bases and NOT associated with an immediate or less than or equal to 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> TS Action Statement should not be tested on the RO examination when testing RO above this line TS knowledge as discussed in ES-401, and depicted below:

[1]

ES-401 Attachment 2, Clarification Guidance for SRO-only Questions, provides clarification and guidance for fulfilling the intent of 10 CFR 55.43 The use of this document is not a regulatory requirement .

82

83 For the TS 5.3.1 example above, a RO applicant would be expected to possess knowledge and understanding of the Above this line information but would generally not be expected to have knowledge of the TS LCO B 3.5.1 system/component parametric values and/or conditions necessary to determine Accumulator System OPERABILITY.

However, ACTION D for Two or more accumulators inoperable as shown below requires that LCO 3.0.3 be entered Immediately. Therefore, in this instance and notwithstanding that the information is provided in the LCO B 3.5.1 Bases, a RO would be expected to understand that for an accumulator to be considered OPERABLE, the isolation valve must be fully open, power removed above 1000 psig, and the TS Surveillance limits for accumulator volume, boron concentration, and nitrogen pressure must be met.

83

84 84

85 401.55 Some Tier 1, Emergency and Abnormal Plant Evolutions, written examination questions have been categorized as deficient, and in some instances, Unsatisfactory as a result of the NRC Form ES-401-9 Written Examination Review process because they do not test knowledge of, or information contained in, the sites abnormal operating procedures (AOPs) and emergency operating procedures (EOPs).

Is a proposed Tier 1 written examination question deficient or unacceptable if it does not do that?

NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Revision 11 (NUREG-1021), is used in conjunction with the applicable knowledge and abilities (K/A) catalog to develop content-valid examinations. Section 4.0 of each K/A catalog contains the K/A statements for emergency plant evolutions (EPEs) and abnormal plant evolutions (APEs). As defined in the K/A catalogs, an emergency plant evolution is any condition, event or symptom which leads to entry into emergency operating procedures (EOPs), and an abnormal plant evolution is any degraded condition, event, or symptom not directly leading to an EOP entry condition, but nonetheless, adversely affecting a safety function."

In accordance with NUREG-1021, Sections ES-401 and 401N, K/A statements for Tier 1 questions are selected from Section 4.0 of the applicable K/A catalog. The objective of Tier 1 questions is to test an applicants knowledge of how to safely operate the plant during emergency and abnormal conditions. These facility evolutions are a significant aspect of operating a nuclear power plant safely, and knowledge of these evolutions, including the use of abnormal and emergency procedures as appropriate, must be robustly understood by each licensed operator. All Tier 1 questions must meet this objective. Satisfactory ways of meeting this objective include, but are not limited to, the following: (1) information contained in the sites procedures, including alarm response procedures, AOPs, EOPs, and their associated bases documents; (2) diagnosis that leads to selection of the procedures that should be used to respond to the evolution, (3) the progression of an event, and (4) assessment of the integrated plant response to emergency or abnormal situations crossing several plant systems and/or safety functions. The selected K/A statement from the approved sample plan will influence how the Tier 1 question is written to meet this safety objective. If it is not possible to write a Tier 1 question that meets this objective, then the K/A may be replaced with a more suitable K/A.

Accordingly, if a Tier 1 question does not meet the safety objective of Tier 1 questions, then it may be rated as either unsatisfactory (i.e., in need of repair or replacement) or in need of editorial enhancement on Form ES-401-9 depending on the extent to which the question must be changed to meet the safety objective of Tier 1 written examination questions. For example, a question may be rated as in need of editorial enhancement if the addition or deletion of a sentence or phrase would be sufficient to meet the objective of Tier 1 questions. A question may be rated as unsatisfactory if only simple editorial changes would not be sufficient to ensure the question meets the safety objective of Tier 1 question (e.g., a proposed Tier 1 question that asks the applicant to identify the power supply to a pump that is designed to provide emergency cooling to the core during a loss of coolant accident would likely be rated as unsatisfactory because it does not test the applicants knowledge of how to operate the plant).

85

86 ES-402 Administering initial written examinations 402.1 Regarding the written exam duration: The exam duration should be presented to candidates as: "The exam duration is scheduled (targeted) for 5 hour5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />s: but extensions can be granted," i.e. don't rush through exam to meet the 5 hour5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> time limit.

What is the interpretation of "prior approval" for extensions of 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> for the initial written examination? Why is there a time limit for written exams?

Why not just an upper limit with no extensions? Maybe 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />?

Comments noted. The time limit is largely an examination design and resource planning tool, and is not intended to rush the applicants. Some applicants will take whatever time is allowed, which would place an additional burden on facility proctors and NRC examiners who are required to be available by telephone while examinations are being administered.

As noted in Attachment 1 (Section II) of SECY-98-266, the nature of the NRC licensing examination is such that allowing sufficient time to demonstrate knowledge is of primary concern. Section E.4 of ES-401 (in NUREG-1021) encourages facility licensees to conduct a peer review of the examination, which should confirm that the level of difficulty is appropriate and that the applicants will have sufficient time to complete the exam.

As discussed in Section C of ES-402, it is important that the licensee coordinate the administration of the written examination so there will always be an NRC contact available to respond to questions or problems that might arise. Therefore, if the facility licensee determines, while proctoring the exam, that any of the applicants will not be able to complete the examination within the time allotted, the licensee shall contact the NRC Regional Office as discussed in Section D.4.d of ES-402, before granting the extension and, again, after all the applicants have completed the examination. The NRC does not want to discover after the fact that the licensee has given the applicants more than the allotted time to complete the examination. Per Section E.3.a of ES-501, the NRC will document the time extension in the examination report and expect the facility licensee to evaluate whether a problem with the examination validation or the training of the applicants is indicated.

The fact that Supplement 1 to Revision 8 (specifically Section D.4.d of ES-402) extended the nominal time limit for completing the RO exam to six hours, that the examination is designed for four hours (refer to Section D.2.c of ES-401), and that Revision 9 shortened the examination to 75 questions should eliminate the need for time extensions under normal circumstances.

86

87 402.2 Must the facility proctor read the entire Appendix E verbatim or just the first part regarding cheating?

Only those items specifically identified in Appendix E (i.e., Items A.1 and B.1) need to be read verbatim by the proctor; the others may be paraphrased. Per Section D.2.c of ES-402, every applicant shall also be given a copy of the Appendix to review before starting the examination.

402.3 What is the guidance on providing additional information or clarifying statements to the candidates during the written exam? Specifically, for facility written exams.

The requested guidance is located in Section D.3.b of ES-402 (in NUREG-1021); it is the same regardless who prepared the examination. Anyone providing additional information during the examination must be extremely careful not to lead the applicants or give away answers when clarifying questions. If the proctor has any doubt about how to respond to an applicant's question, it is best to withhold additional guidance and instruct the applicant to do his or her best with the information that is provided. Per Section C.2.b of ES-402, an NRC examiner will always be available in the NRC Regional Office to respond to questions while the examinations are in progress.

87

88 ES-403 Grading initial written examinations 403.1 Is there a checklist that states make copy prior to grading?

Please add note to Form ES-403-1 for the grader to copy the answer sheets. I would also suggest making two copies, NRC and facility to have. (ES-403, Section D.2.a)

Yes; it is included on Form ES-403-1 (in NUREG-1021). Moreover, Section D.2.a of ES-403 instructs the grader to make a copy before marking the original, and Section C.1.a of ES-501 instructs the facility licensee to submit the clean copy with the examination package. There is no restriction on the licensee keeping copies of the answer sheets.

88

89 ES-501 Initial post-examination activities (documentation and reporting) 501.1 Does the time-line (5 days) for completing the requirements of ES-501, Section C.1.a, begin after completing the written or the entire exam including the operating test?

Assuming the time begins after completing the entire exam, how does this factor into the 30-day allowance between the administration of the written and operating tests as described in ES-402, Section C.2.b?

Can the NRC expectation for exam comments be delayed until exam completion for utility-administered examinations?

The purpose of the 5-day time-line is to enable the NRC to achieve its goal of completing the licensing actions within 30 days after the examinations are given. With the exception of the Security Agreements (Form ES-201-3 in NUREG-1021), all of the items listed in Section C.1.a of ES-501 are associated with the written examination. Consequently, those items should be forwarded to the chief examiner as soon as practical (but not necessarily within 5 days) after the written exams were given, even if the operating tests are given at a later date. This will allow the NRC to resolve any comments and review the written examination grading, thereby expediting the completion of the licensing actions after the operating tests are administered.

As always, facility licensees should confirm their specific schedule with the chief examiner. If the personnel who will compile the post examination comments are busy with other exam activities, talk to the chief examiner and arrange an alternate date for submitting the comments.

Supplement 1 to Revision 8 of ES-501 clarified the guidance regarding submittal of post-examination comments.

89

90 501.2 ES-501, Section C.1.a (Bullet 4) states that any comments made by the applicant(s) after the written exam with explanations of why the comment was accepted or rejected must*

be submitted to the NRC. (* To be consistent with ES-402, Section E.4, this submission should be "optional.")

Do all comments made regarding the written exam by the applicant and a reason for accepting/rejecting the comment need to be submitted (ES-402, Sections E.4 and 5). I was told not to submit student's rejected comments, only those that cause an exam change. This is a "should," can it be changed to only sending in comments requiring an exam change?

ES-402 (Section E) and ES-403 (Section D) encourage facility licensees to collect examination comments from the license applicants and consider them during the initial grading process because this will enhance examination validity. Although licensees are only required to submit comments and documentation to the NRC to justify question deletions and changes in the answer key, it is useful for the NRC to know, if an applicant submits an appeal, that the facility licensee had previously reviewed and rejected the applicant's concern(s). If the facility licensee wrote the examination, the NRC may request the licensee to state its position regarding the applicant's contentions.

Supplement 1 to Revision 8 of NUREG-1021 changed Section C.1 of ES-501 to make it consistent with ES-402.

501.3 If the chief examiner conducts a re-grade (78-82%), what is the focus of the re-grade?

(Re-grade per the key?) (Validity of the questions?)

Multiple grading changes and reviews often result in answer sheets that are difficult to read and could result in licensing errors. Therefore, Section D.2.c of ES-501 requires the chief examiners to re-grade borderline exams using the clean answer sheets copied per Section D.2.a of ES-403. The re-grade would be done after all the facility's comments have been resolved and the answer key has been finalized. It would normally not involve a revalidation of the exam questions.

90

91 501.4 Since senior site management tends to "expect perfection," maybe the NRC could communicate that a number of comments are expected (in the final examination report).

Comments contained in reports should remain specific to deviations from 10CFR or NUREG. (State the facts, refrain from the use of "several" or "many.")

When does the clock start for the 20% untestable questions?

Comment noted. The NRC has tried to communicate exactly that message during the operator licensing workshops conducted by each of the NRC Regional Offices.

Supplement 1 to Revision 8 of NUREG-1021 clarified the guidance in Section E.3 of ES-501 regarding the portrayal of examination quality in the final report. It established a 20%

unacceptable test item threshold below which the report will simply indicate that the proposed examination was within the expected range of acceptability. This policy has been in effect since the spring of 2000.

501.5 Is there a format for the utility to provide the NRC with feedback on how the exam went?

Sort of a reverse exam report? I would think the NRC would be open to feedback so you can also improve the exam process from your end. (I mean a formal feedback process -

not casual.)

NUREG-1021 requires the regional operator licensing branch chiefs to solicit feedback from the licensee before the examinations are given (Section C.2.j of ES-201) and encourages the discussion of lessons learned after the examinations are complete (Section E.1.d of ES-501).

As discussed in Section C.1.j of ES-201, facility licensees are encouraged to call the NRC chief examiner, regional branch chief, or program office any time they have concerns regarding an examination.

501.6 If candidates score in the 80-81% range, are licenses held? If so, how long? (No failures)

If there are no written examination failures, there is no reason for the NRC to withhold a license so they would all be issued simultaneously. As discussed in Section D.3.c of ES-501 (in NUREG-1021), the NRC would only hold the license for an applicant that scored between 80%

and 82% (70 - 74% on the SRO-only questions) if another applicant failed the examination and there is a possibility that enough of the questions that the passing applicant got correct could be deleted from the examination on appeal or have their answers changed, thereby causing the applicant's score to fall below 80% (70% on the SRO-only questions).

91

92 501.7 Has the NRC considered changes resulting from deregulation with regard to making examinations public?

In accordance with 10 CFR 2.390, all final NRC records and documents will be made available in the NRC's Public Electronic Reading Room unless there is a compelling reason for non-disclosure or the document qualifies for one of the exceptions specified in the regulation. It is the intent of the NRC to automatically make publicly available information that is anticipated to be of interest to the public without anyone having to file a request under the Freedom of Information Act. Without more specific information, it is unclear how the deregulation of the electric power industry would or should affect the NRC's responsibility to keep the public informed regarding its health and safety mission.

92

93 ES-502 Initial examination appeals and hearings 502.1 How will the facility representatives get a copy of the NRC appeal correspondence?

It is normal practice for the NRC to send a copy of its appeal correspondence to the individual who signed the applicant's license application (NRC Form 398). However, applicants who file an appeal are not required to send a copy of their request to the facility licensee.

502.2 Who is responsible for defending a question during the appeal process?

Once the NRC approves an examination it essentially takes ownership of the document.

Therefore, if a question is challenged during an appeal, the NRC will take the lead in defending the question. However, as stated in Section C.2 of ES-502 (in NUREG-1021), facility licensees are expected to provide reference material and technical support (and possibly confirmation of the test items validity if the facility wrote the examination) as necessary for the NRC to evaluate and resolve any concerns raised by a license applicant.

502.3 What would the NRC do if a question from the national exam bank was found unacceptable after it was used? How far back would the NRC search for previous use of the question, which could affect already issued licenses?

Any question (not just those from the national bank) determined to be invalid during the grading process (i.e., after the exam was given but before the licenses are issued) would be deleted from the exam and the applicants grades would be adjusted accordingly. However, this would not affect applicants who had already been granted a license.

502.4 In accordance with ES-502, Section D.2.C, if an applicant's license examination failure is overturned due to appeal and the question that was reviewed affects the licenses of other applicants, will licenses be granted to all applicants that would have received a passing grade due to the review, even if those applicants chose not to appeal?

Yes. The NRC regional office will determine if any of the test item changes (i.e., question deletions or answer key changes) made as a result of the NRR operator licensing program office review for the appealing applicant(s) alter the outcome for any applicant who failed the examination but chose not to request an administrative review or hearing. If the test item changes cause any of the non-appealing applicant(s) to achieve a passing score, the regional office will issue licenses, as appropriate.

93

94 ES-601 NRC requalification examination process 601.1 2.5 versus 3.0. What is the minimum task [importance] threshold for initial exams versus requalification? Should be higher standard for requal than initial.

As noted in Attachment 3 of ES-601 (in NUREG-1021), all test items used on an NRC requalification examination should normally have a K/A importance rating of 3 or greater. The minimum K/A importance rating for initial exams is 2.5. In either case, test items with lower NRC K/A values may be used with appropriate justification.

The NRC expects facility licensees to comply with their own requalification program requirements regarding test item importance.

Initial license applicants are held to a higher standard (i.e., more K/As eligible for testing) because the NRC has no prior basis for judging their competence. Once an operator has a license, his/her competence is continually evaluated on the job and in requalification training, thereby justifying a lower threshold for the NRC requalification examination.

601.2 Is there a policy for use of computers and maintaining exam security?

Does there need to be a specific procedure for requalification examination security?

The requirements of 10 CFR 55.49 apply to all examinations required by the regulation, including requalification exams, while the requirement to establish, implement, and maintain examination integrity and security procedures in accordance with 10 CFR 55.40(b)(2) only applies to power reactor licensees that elect to prepare their own initial operator licensing examinations. However, it would be appropriate for those licensees that do establish procedures to address all exams required by Part 55. Refer to the section on ES-201 for related security questions.

94

95 601.3 What is the basis for the statement [in Section E.1.b of ES-601], "Under NO circumstances will another operator be allowed to witness an operating test?" There are instances where the crew being examined may want another operator to observe. (e.g.,

We had an initial license exam during the annual operating test. When the initial license candidate completed his exam and was assigned to a crew, the crew's shift manager requested that the new crew member be able to observe their operating test from the simulator instructor's booth.)

The bases for this policy include the desire to minimize undue stress on the operators (or applicants) that are being evaluated and the need to minimize crowding in the simulator (for the examinees, NRC examiners, facility evaluators, operations and training representatives, and simulator operators that have to be there). Moreover, the NRC believes it is inappropriate to use NRC-conducted licensing and requalification examinations as training tools for other applicants and operators. Facility licensees are free to establish their own examination policies for requalification examinations in which the NRC is not involved.

95

96 ES-602 NRC requalification written examinations 602.1 Why [is there a] static [written exam] if [the] NRC administers requalification? What value [is] added?

Static Exams - If [the] NRC administers [a] requal exam, a static is required. If we administer our own, a static is not required. Some utilities have stopped maintaining a static exam bank and use of it, while others (such as us) are continuing to use them. The reason we do is, if NRC comes into a program that hasn't done statics for a long time, and the crews are subjected to statics, and they aren't used to them, a high failure is likely. So, why does this difference exist?

Why is there a difference between what the NRC would do for a "for cause" requalification [exam] versus facility requalification [exam]? [This is] unfair [to the operators and may lead to a] high failure rate.

The requalification examination format, including the static written examination, was developed by an NRC/industry working group in 1987. The NRC understands that most facility licensees have stopped using the static written format since the NRC shifted to an inspection-based oversight program in 1994, and the fact that it is still included in the ES-600 series of NUREG-1021 has prompted some facility licensees to continue using it as well or at least to maintain their static scenario banks. As discussed in Section C of ES-601, if a facility licensee's requalification program uses an examination structure or methodology different from that described in the ES-600 series and the NRC decides to conduct an examination, the NRC will consider preferentially using the facility licensee's requalification examination structure or methodology if it is different from that described in the ES, provided it complies with 10 CFR 55.59 and is free of significant flaws; the regional office shall consult with the NRR operator licensing program office to determine the appropriate examination procedure.

602.2 What is the policy/requirement regarding extension of time limit for the requalification written exam? ES-401 allows time extensions. Does the ES-600 series? Are time extensions for requalification exams similar to [the initial] written?

Although the examination should be time-validated to preclude the need for extensions, the NRC would consider extending the time limit for NRC-conducted requalification examinations, as it does for initial licensing examinations. When facility licensees conduct their own requalification examinations, the NRC expects them to comply with their program requirements (including the ES-600 series, as written, if the licensee has endorsed the ES as part of its program).

96

97 602.3 What is an effective sample plan generation?

The concept of examination sample plans is discussed in Attachment 3 of ES-601. If that does not provide the information you need, please submit a more specific question.

602.4 If the yearly requal exam is randomly and systematically developed, can we eliminate the 50% overlap restriction that currently exists?

Although there is no official 50% overlap restriction (refer to Section E.3.b(6) of ES-601 of NUREG-1021), the random and systematic development of requalification exams would eliminate the NRCs concerns regarding exam integrity and validity. Moreover, assuming that a facility has a reasonably sized examination question bank, it would be highly improbable that a 50% overlap would occur under a random and systematic selection process.

97

98 ES-603 NRC requalification walk-through tests 603.1 Section B of ES-301 states that initial license exams should sample the items listed in 10 CFR 55.43 but need not cover all 13 items. Is this also true of a requalification annual operating examinations?

Is there an expectation that every SRO do an Emergency Plan classification in either a scenario or a JPM?

Yes. As specified in 10 CFR 55.59(a)(2)(ii), the operating test shall cover a comprehensive (i.e.,

thorough or broad, but not necessarily complete) sample of the items specified in 10 CFR 55.45(a)(2) through (13) as applicable to the facility. Also refer to Question IP.13.

No. Every operating test is a sample and does not have to include an Emergency Plan classification.

603.2 Is changing a JPM to an alternate path JPM considered a different test item (for the 50%

[repetition] requirement)?

Yes. This is consistent with the initial examination policy regarding the repetition of test items from the individual's audit examination (refer to Section D.1.a of ES-301 of NUREG-1021).

603.3 Are simultaneous JPMs allowed?

The NRC would allow the simultaneous administration of JPMs in the simulator or control room during NRC-conducted tests provided there is no interference between the operating stations.

When licensees are conducting the tests, they should follow their approved requalification program.

603.4 To what extent is it acceptable to just mark up a procedure versus [following] the ES format [for JPMs]?

In accordance with Section C.1.d of ES-603, Form ES-C-1, "Job Performance Measure Worksheet," or an equivalent facility form should be used to construct and format the JPMs.

However, as long as the JPMs include the elements identified in Appendix C (e.g., initiating and terminating cues, critical steps, and performance criteria), it should be possible to adapt facility procedures for use as JPMs by identifying critical steps and entering comments on how to execute particular steps. Section D.2.b of ES-301 authorizes that practice for initial operating tests.

98

99 603.5 Is the initial licensing walk-through alternate path JPM requirement, a required item for annual requalification exams?

No. However, per ES-601 of NUREG-1021 (Section III.C of Form ES-601-2), facility licensees are expected to include some alternate path JPMs in their test item banks for use during NRC-conducted requalification examinations.

603.6 ES-603 guidance for generating an annual operating evaluation states the sample plan is to be based on the "current" cycle. My question is this; suppose we are in the first six months of the "current" cycle and we want to generate an annual operating exam, since there is insufficient material for an exam would it be acceptable to generate the exam based on a sample plan developed covering the "current" cycle and include that part of the previous cycle up to the last exam (i.e. the last six months of the previous cycle)?

Keep in mind that the ES-600 series in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, provide guidance for the preparation and administration of licensed operator requalification examinations in which the NRC is an active participant. When facility licensees prepare and administer their own requalification examinations, the NRC does not expect or require them to comply with the guidance in the ES-600 series unless the facility licensee has formally incorporated that guidance as part of its accredited (by the National Academy for Nuclear Training) training program.

Although requalification programs that are based on a systematic approach to training (SAT) should evaluate the trainees mastery of the objectives during training, Attachment 3 of ES-601 encourages reserving a portion of the examination to test high importance topics that were not necessarily covered during the requalification cycle. This is consistent with 10 CFR 55.59(c)(4)(i) which (in lieu of a SAT-based program) requires the comprehensive written exams and annual operating tests to determine areas in which retraining is needed. Moreover, 10 CFR 55.59(a)(2)(ii) requires the operating test to evaluate the operators understanding of and ability to perform the actions necessary to accomplish a comprehensive sample of the items specified in 55.45(a)(2) through (13) inclusive to the extent applicable to the facility.

Notwithstanding the liberal definition of annual in Appendix F of NUREG-1021, we encourage facility licensees to conduct their annual operating tests at approximate 12-month intervals (i.e.,

at the midpoint and end of their 24 month requalification training cycles). Facility licensees need to exercise caution when they reschedule examinations around the plants operating schedule to ensure they comply with the regulation by doing an operating test every calendar year.

Bottom line: The NRC expects facility licensees to comply with the requirements in 10 CFR 55.59 and their accredited training programs. The regulations do not appear to prohibit the use of test items covering topics outside the scope of the current requalification training cycle. You need to check to see what the facility licensees program requires.

99

100 ES-604 NRC requalification dynamic simulator tests 604.1 For requalification [examinations, do you] test how you normally staff?

Yes. As stated in Section D.2.a of ES-601 (in NUREG-1021), the NRC expects facility licensees to train and examine their operators in the same crew configurations with which they normally operate the plant.

604.2 Can an individual who fails in the simulator for a specific task be retested with a JPM, or must it be a scenario?

If an operator fails an annual operating exam scenario due to an independently performed competency, can a JPM be used as a retake exam?

If an operator fails any portion of an NRC-conducted operating test (initial or requalification), the retest will be in the same format as the part that was failed. If an operator fails a facility-conducted requalification examination, the facility licensee would be expected to administer the retest in accordance with its approved requalification program.

604.3 Can an individual failure [on the simulator operating test] be retested with surrogates, or must it be with a shift?

Surrogates would be acceptable for an NRC-conducted test, but the facility licensee would have to follow its program requirements if it conducts the test.

100

101 ES-605 License maintenance/conditions; renewals; requalification; appeals and hearings 605.1 NUREG-1021 allows postponement of requalification requirements for up to 2 years for off-site development assignments, such as INPO. We also have on-site development assignments, such as Work Control or Site Engineering, which are intensive from a workload standpoint. Why can't the requirements of requalification be suspended for an on-site/off-shift developmental assignment?

The Operator Licensing Program Office has a number of concerns regarding such a policy change (e.g., the quality of the make-up training and testing, limits on the number and duration of the assignments, public perception, NRC involvement and resource implications). The issue has been discussed during public meetings with the Nuclear Energy Institute's operator licensing focus group members, and everyone appeared to understand the basis for limiting the requalification suspension option to off-site assignments. Operators who wish to pursue on-site developmental opportunities can terminate their licenses, pursue other activities for up to two years without having to worry about attending requalification training, and then reapply for a license. In accordance with 10 CFR 55.47, the NRC can waive the requirement for an examination if the specified conditions are met. Refer to Section D.1.g of ES-204 for more information regarding such waivers.

605.2 Operator Medicals are required every 24 months with no grace [period]. This causes a need to schedule shift crews more often so 24 months not exceeded. With a fixed requalification schedule, 24-month refueling outage cycle, it would be nice to have medicals the same cycle every year. So, if critical equipment (RPS, etc.) surveillance frequencies can have grace [periods], why can't operator medicals?

As noted in Appendix F of NUREG-1021, a biennial requirement can extend beyond 730 days if the requirement is met during the anniversary month of the second year. For example, a biennial medical examination last performed on January 10, 1995, would be due again by January 31, 1997. This, in essence, provides a variable grace period of up to 30 days.

101

102 605.3 Notification of administrative suspension of licenses due to medical reasons.

In accordance with Section C.3.a of ES-605, the facility licensee does not need to notify the NRC if the medical condition is temporary and the operator is administratively prevented from performing licensed duties or otherwise restricted, as appropriate, during the period of his or her temporary disability. However, if the operators temporary medical condition also precludes the operator from maintaining currency with all of the facilitys 10CFR55.59 requalification program requirements i.e., the operators participation in all or part of the requalification training program is suspended, the facility shall notify the NRC as described in ES-605 C.1.c and C.3. unless the missed requalification training can be made up within the time limits specified in the facilitys Commission approved 10 CFR55.59 Requalification Training Program.

605.4 Can someone stand 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> of a normal 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> watch?

As discussed in Section C.2 of ES-605, the 10 CFR 55.53(e) requirement for licensed operators to maintain their proficiency may be satisfied with a combination of complete 8- and 12-hour shifts (in a position required by the plant's technical specifications) at sites having a mixed shift schedule. Watches shall not be truncated when the minimum quarterly requirement (56 hours6.481481e-4 days <br />0.0156 hours <br />9.259259e-5 weeks <br />2.1308e-5 months <br />) is satisfied. Overtime may be credited if the overtime work is in a position required by the plant's technical specifications. Overtime as an extra "helper" after the official watch has been turned over to another watch-stander does not count toward proficiency time.

605.5 Are there any unwritten restrictions for "no solo" license conditions?

No. The nature of the restriction, which is determined case-by-case based on the individual's medical status and the recommendation of the facility licensees physician, is clearly stated on the license. Section C.3.c of ES-605 (in NUREG-1021) describes some typical medical restrictions.

605.6 The regulations (specifically 10 CFR 55.55(b)) require license renewal applications to be filed at least 30 days before the expiration date of the existing license to ensure that the license does not expire while the Commission reviews the application? However, the regulation does not specify a "no earlier than" date for filing renewal applications. How early is too early?

In order for the NRC to have current information on which to base a renewal decision pursuant to 10 CFR 55.57(b), it is recommended that renewal applications be filed no more than 60 days before the existing license expires. If a facility licensee submits its operator license renewal applications more than 60 days in advance, the NRC regional office may contact the facility to determine whether it would prefer to have the licenses renewed immediately with a new effective date (the licenses will not be predated, nor will they exceed a six-year term) or to resubmit the applications within the 60-30 day window preceding the expiration date.

102

103 605.7 10 CFR 55.53(f)(2) requires that part of the 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> include a plant tour. Can the plant tour be performed alone or does it have to be with an active license holder?

The NRC staff's position, based on the wording of the regulation, is that the plant tour, being part of the 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> to be completed under the direction of an operator or senior operator (as appropriate), must be done in the company of an active watch stander. That way the active watch stander can ensure that the reactivating watch stander is made aware of on-going activities and abnormal situations in the plant.

605.8 In accordance with 10 CFR 55.53(f)(2), an operator who fails to maintain an active license must, before resuming licensed duties, complete a minimum of 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> of shift functions under the direction of an operator or senior operator, as appropriate, and in the position to which the individual will be assigned; for senior operators limited to fuel handling under 10 CFR 55.53(c), one shift must have been completed. In the case of senior operators limited to fuel handling (LSROs), when and where should they stand their under-direction shift, and what level of supervision is required?

Can an inactive SRO, whose license is NOT limited to the performance of fuel handling under 10 CFR 55.53(c), reactivate as a fuel handler by completing one shift under direction?

Can LSROs maintain an active license pursuant to 10 CFR 55.53(e) between refueling outages?

The answers to these questions have been incorporated in Section C.2.g of ES-605.

605.9 Is it acceptable for an operator with a no-solo license to stand watch with another no-solo operator; i.e., can two no-solo operators back each other up, or does the backup have to have an unrestricted license? If a no-solo operators backup has to leave the site unexpectedly to take care of a personal emergency, would the remaining operator be considered in noncompliance with his/her license condition or would this be covered by the temporary staffing deviation provision in the facilitys technical specifications (TS)?

The possibility that both no-solo operators standing watch together would become incapacitated at the same time is pretty remote. Therefore, yes, it would be acceptable for two no-solo operators to back each other up. If a backup operator has to leave unexpectedly (or is incapacitated while on watch), and prompt action is taken (per the facilitys TS) to restore compliance with control room staffing requirements, the remaining no-solo operator would not be subject to individual enforcement action.

103

104 605.10 In NUREG-1021, ES-605 C.3.c, there is a description of the "no-solo" restriction for SRO licenses. What are the actual restraints on the SRO that could have to leave the control room to perform an alternate shutdown due to fire in the control room and as part of those duties will have to manipulate controls locally in the plant. Also, if the STA on shift is licensed, could he or she be the extra licensed operator with the "no-solo" SRO when they perform the manipulations locally in the plant?

When the Operator Licensing Program Office last revised the wording of the no-solo license restriction, we tried to minimize the impact that it would have on the facilities and the individuals involved by distinguishing between those activities that require a license per 10 CFR 50.54 (i.e.,

manipulation of the controls that directly affect reactivity or power) and those that, with the knowledge and consent of a licensed operator or senior operator, can be performed by non-licensed personnel (i.e., apparatus and mechanisms other than controls that may affect reactivity or power). We also gave due consideration to the fact that most control manipulations are planned in advance and conducted in accordance with facility peer-check requirements and guidelines.

In the event that a control room evacuation becomes necessary, we expect that the operators would, under most circumstances, have sufficient time to trip/scram the reactor before relocating to the alternate/remote shutdown panel(s). This would generally mitigate the need to perform additional control manipulations locally in the plant and any concern regarding solo operations by a restricted operator. However, facility licensees should take reasonable measures to ensure that additional operators are available on-site to respond to an emergency scenario when no-solo operators are assigned the watch. For example, when a no-solo SRO is on watch, the other SRO on-site could be instructed to respond to the alternate shutdown panel if that is the location the no-solo SRO would respond to during such an emergency. Similarly, a no-solo RO could have another licensed individual on-site respond to the control room or "catch up" to the RO during their performance of the in-plant portion of the alternate shutdown procedure.

As written, the current no-solo restriction requires another licensed operator to be in view only when the restricted operator actually manipulates a control (i.e., that small subset of apparatus and mechanisms that directly affect reactivity or power) while in the control room or out in the plant; at all other times while performing SRO licensed duties, another SRO would have to be present on site. An STA would be acceptable to fill the role of the second operator if he or she has an active SRO license and is up-to-date in the licensed operator requalification program. If, despite the compensatory measures discussed above, a second licensed operator is not immediately available to oversee an emergency control manipulation in the control room or in the plant, we would expect the restricted operator to perform the necessary control manipulations to protect the plant in a timely manner even if it results in a failure to comply with their license condition.

104

105 605.11 What are the requirements with respect to a retired licensed operator (RO, SRO, or LSRO) returning to work as a licensed operator at the same facility after retirement?

• In accordance with 10 CFR 55.55, Expiration, an operators license expires upon termination of employment with the facility licensee. Therefore, if the facility wishes to retain the individuals license, it would have to execute a re-employment agreement with the individual before the retirement takes effect. If the operator actually terminates employment with the facility licensee without executing a re-employment agreement, the license would be considered expired and the individual would have to reapply for a new license in accordance with 10 CFR 55.31. However, the applicant may request and be able to justify a waiver of the examination and test requirements pursuant to 10 CFR 55.47 and Section D.1.g of ES-204 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors. In either case, the individual would be required to make up any requalification training and testing that might have been missed during any break in service and to reactivate the license, as necessary, per 10 CFR 55.53(f) before resuming licensed duties.

• The terms licensee, operator, and senior operator are defined in 10 CFR 55.4 and all refer to any individual licensed under 10 CFR Part 55 to manipulate the controls of the facility and, in the case of a senior operator, to additionally direct the licensed activities of licensed operators. Moreover, 10 CFR 55.31(a)(3), which addresses how to apply for a license, states that a license applicant shall submit a written request from an authorized representative of the facility licensee by which the applicant will be employed Although Part 55 does not define the term employed, a retired and subsequently rehired RO, SRO, or LSRO would be considered an employee of the facility regardless how the facility classifies the individual.

• Pursuant to 10 CFR 55.2(a), the regulations in Part 55 apply to any individual who manipulates the controls of any utilization facility licensed pursuant to 10 CFR Part 50, without regard to employment status. Consequently, a rehired operator would be subject to all the same license conditions specified in 10 CFR 55.53 (e.g., observe all applicable rules, regulations and orders of the Commission; maintain or re-establish proficiency; complete the requalification program; have a biennial medical examination; and comply with fitness for duty requirements) as a regular employee/operator. All other regulatory requirements and potential enforcement sanctions in Part 55 would also apply regardless of the individuals employment classification.

• Given that the responsibilities of senior operators, as defined in 10 CFR 55.4, include directing the licensed activities of licensed operators, the extent and nature of such direction may result in creating, under applicable state law, an employment relationship with those licensed operators (ROs). Therefore, facility licensees are advised to consult with their attorneys to ensure compliance with employment law requirements in the state in which their facility is located.

105

106 605.12 Can a shift (watch) that begins for an operator or senior operator at 1800 of the final day of the calendar quarter be counted for meeting the minimum shifts required by 10 CFR 55.53(e) for maintaining active license status even though the shift is completed at 0600 on the first day of the succeeding quarter?

10 CFR 55.53 (e) requires to maintain active status that every licensee actively perform the functions of an operator or senior operator on a minimum [emphasis added] of seven 8-hour or five 12-hour shifts per calendar quarter. There is no allowance provided in 10 CFR 55 or NUREG-1021 that permits counting a shift that will not be completed in the current calendar quarter for meeting 10 CFR 55.53(e) as long as it is started in the current calendar quarter. Thus, given the example provided in the question, the operators license would not be considered active as of 2400 on the last day of the calendar quarter since the operator/senior operator had not performed the functions of an operator or senior operator on a minimum

[emphasis added] five 12-hour shifts in the current calendar quarter.

106

107 605.13 The no-solo restriction described in ES-605, C.3.c is not explicit for an SRO supervising core alterations. Is there a difference in requirements for no-solo SROs and LSROs when supervising core alterations?

10 CFR 50.54m (2)(iv) requires an SRO or LSRO to be present to directly supervise any alterations of the core (including fuel loading or transfer) and to be assigned no other duties while supervising the alterations. ES-605, C.3.c describes an LSRO with a no-solo license restriction as requiring another individual capable of summoning assistance in view while the restricted LSRO is performing licensed duties (e.g. directly supervising core alterations). A no-solo SRO is required to have a licensed operator in view when the restricted SRO is performing control manipulations, however there is no explicit guidance on what type of restriction is required for a no-solo SRO supervising core alterations.

The requirement for a no-solo SRO performing control manipulations is intended to be more restrictive than for a no-solo LSRO or SRO who is directly supervising core alterations. It is expected that if an LSRO or SRO becomes incapacitated while supervising core alterations, the individuals performing the evolution will be able to safely stop the fuel movement and call for assistance. A requirement to have an additional licensed operator in view of a no-solo SRO when supervising core alterations would likely negate the reason to have the restricted SRO present in the first place. While supervising core alterations, a no-solo SRO is expected to adhere to the same restriction imposed on a no-solo LSRO and thereby ensure that another individual is in view who is capable of summoning assistance if needed.

It is important to realize that this clarification is intended to provide guidance that is consistent with current industry practices. No additional restrictions or changes to the methods currently being implemented are being established. However, for the purposes of clarification, the typical wording on all future SRO and LSRO licenses will be changed to read as the following:

SRO No-Solo:

Another licensed operator must be in view when you are performing control manipulations, and another senior operator must be present on-site at all other times while you are performing SRO licensed duties or someone capable of summoning assistance must be present in the control room at all other times while you are performing RO licensed duties. Another individual capable of summoning assistance must be in view when you are directly supervising core alterations.

LSRO No-Solo:

Another individual capable of summoning assistance must be in view when you are directly supervising core alterations.

No revisions or corrections will be made to current SRO and LSRO licenses. The new wording will be applied to new licenses as they are issued and to current licenses as they are renewed during the normal 6-year cycle. Currently licensed SROs with no-solo restrictions should understand that when directly supervising core alterations, their licenses require them to have an individual capable of summoning assistance in view at all times.

107

108 605.14 Revision 11 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, states that an amendment to an operators license requires a signature from the affected licensed operator. It also states that for medical condition license amendments the affected licensed operator can satisfy this requirement by signing the NRC Form 396, Certification of Medical Examination by a Facility Licensee.

Given that the current NRC Form 396 does not have a block for the signed statement of fact from the licensed operator, how should the facility licensee proceed to properly request a medical license amendment for a licensed operator?

The next planned revision to NRC Form 396 will include a block for the operators signature regarding the facility licensees submission of a medical condition license amendment.

However, the planned NRC Form 396 revision is currently still in progress and not in effect.

Therefore, in the interim, the NRC Form 396 submittal requesting a medical condition license amendment should be accompanied by a letter from the licensed operator acknowledging the proposed amendment. Please note that we anticipate the revised NRC Form 396 will be finalized and go into effect in early Fall 2017.

108

109 IP-71111.11 Requalification inspections IP.1 10 CFR 55.59 - the use of [systematic approach to training] SAT-based program vice regulatory based programs. Why do you have to track individual control manipulations if you have a SAT-based program?

10 CFR 55.59(c) allows licensees to substitute the appropriate SAT-based program elements (as defined in 10 CFR 55.4) for the requirements in sections (c)(2), (3), and (4) (i.e., lectures, on-the-job training, and evaluation). Record-keeping is not a SAT-based program element, and the NRC needs to know that each individual actually performed the requisite control manipulations.

While a SAT-based process can replace the requirements of 10 CFR 55.59(c)(3), it is still the NRC's expectation and requirement per 10 CFR 55.59(c)(5) that individual participation in the requalification program be recorded. How each utility chooses to do this should be clearly defined in its accredited SAT-based program.

Pursuant to 10 CFR 55.57(a)(4), an authorized representative of the facility licensee must provide a statement that each operator license renewal applicant at the facility has satisfactorily completed the requalification program. Making such a statement would be difficult if the facility licensee does not individually track and document each operator's participation in the program (e.g., classroom lecture attendance, completion of on-the-job training including control manipulations, and performance on examinations).

109

110 IP.2 "Control Manipulations" in Requal - a prior guidance from previous NRC meeting clearly indicated been counting control manipulation from the Denton letter was a thing of the past - SAT based requal training would naturally contain a large portion of the annual/biennial tasks and evolutions, therefore, program participants would be involved during simulator training/evaluation, and/or annual Op. Eval. JPMs; "individuals simulator critical tasks" went away and "crew critical tasks" were required. Teamwork/

communications, command & control/by the team was the most important. Bottom line -

the implied expectation expressed on 8/12/99 is not congruent with that provided in 1989 by Messrs. T.P., S. L., and others who provided us guidance. It appears that we are returning to the middle to early 80's again.

Reactivity Manipulations for [licensed operator continuing training?] LOCT: [The Institute of Nuclear Power Operation's] INPO's policy for tracking manipulations seems to be in conflict with NRC requirement (INPO doesn't require tracking on an individual basis).

The control manipulations conducted per 10 CFR 55.59(c)(3) or your SAT-based requalification program are individual, on-the-job training requirements, which are not to be confused with individual or crew critical tasks on the annual simulator operating test.

Pursuant to 10 CFR 55.57(a)(4), an authorized representative of the facility licensee must provide a statement that each operator license renewal applicant at the facility has satisfactorily completed the requalification program. Making such a statement would be difficult if the facility licensee does not individually track and document each operator's participation in the program (e.g., classroom lecture attendance, completion of on-the-job training including control manipulations, and performance on examinations).

110

111 IP.3 Is it required that each SRO be evaluated during the Emergency Operating Procedures

[EOPs]? Does their documentation for the evaluation need to be done in accordance with the requirements of conducting annual exams? If so, what is the basis for this requirement?

Although each SRO does not have to be evaluated during the EOPs on every annual operating test, every SRO should be at risk of being evaluated on all of the items in 10 CFR 55.45(a) during any test. The NRC does not differentiate between different levels of SROs, so the test-item sampling should be the same regardless whether or not the operator normally stands watch in an EOP-reader position. SROs would be considered "at risk" if the facility licensee holds them responsible for the actions of the EOP readers. However, they do not necessarily have to approve each and every action required by the EOPs.

Note that ES-604 does not require crew position rotation and states that an individual would pass the dynamic simulator test if the operating crew performs satisfactorily. The NUREG-1021 requalification examination crew-based grading methodology presumes that all individual crew members, including senior crew managers, are held accountable for all of the crews actions, and therefore are evaluated. Crew position rotation, if not required by the facility licensees requalification program, would only be considered if it was determined to be the only way to evaluate the scope and depth of a demonstrated individual performance deficiency. The facility licensees dynamic simulator requalification examination process is not required to be the same as that discussed in ES-604. However, if the facility licensee evaluates individual and crew performance consistent with the guidance of ES-604, then the test requirements of 10 CFR 55.59(a) would be met.

IP.4 Are requalification inspections conducted using NUREG-1021 as the standard (i.e., 600 series) for the inspection? Are facilities subject to violations because an aspect of NUREG-1021 is not utilized during a requalification exam or is it just the inspection plan (i.e., 71111-11 vs. ES-600)?

Requalification inspections are conducted using IP-71111.11. Facility licensees are not required to use the ES-600 series of NUREG-1021 to conduct their requalification examinations.

However, if a licensee's requalification program endorses or incorporates the NUREG-1021 examination process, the NRC will expect the facility to comply with its established program.

IP.5 Can I take credit for questions other than multiple choice questions in the LOR [licensed operator requalification] exam bank, including maintenance of the bank?

Yes. However, licensees are encouraged not to abandon their multiple choice question banks in case the NRC determines that a for-cause requalification examination is necessary. Facility licensees are expected to follow their own program guidelines for bank maintenance; the guidelines in ES-601 would only apply if the licensee has endorsed NUREG-1021 as part of its LOR program.

111

112 IP.6 How is the cognitive level determined if essay and short answer are used? (applies to operator requal exams)

As discussed in Section C.1.d of Appendix B of NUREG-1021, the NRC uses Bloom's Taxonomy to classify the cognitive level of test questions. That classification approach would apply regardless of the question format. Facility licensees are not obligated to use the same approach.

IP.7 What are the criteria (guidance) for test item reuse throughout a biennial [requalification]

cycle? ((i.e., 1) items used on more than 1 weekly quiz; 2) item used on weekly quizzes to be used on biennial exam). Need a number (upper limit) on requal test question reuse.

Subjective limits lead to variability in standards and enforcement. Suggest 20-25% limit.

What is the expectation or threshold on reuse of exam materials? During the Region I Conference the NRC stated that internal policy is <50% duplication of items between exams. We all agree we want to protect the validity of the exams. However, without clear expectations from the NRC, and subjective application by an evaluator, it will be difficult to predict acceptability.

Does ES-601 E.3.b(6) allow for subjective interpretation from examination to examination based on what the specific examiner "feels" is appropriate; can we not identify this internally and have the examiner base his decision on plant specific requirements?

Biennial requalification exam -- What is the standard for reusing exam questions from weekly exams from the last 2-year biennial training program?

The NRC does not have definitive criteria (i.e., regulations) regarding the number of test items that can be reused on weekly quizzes or biennial examinations. However, as stated in Section E.3.b(6) of ES-601, the amount of item duplication will be taken into consideration during the program evaluation because it could affect the discrimination validity and integrity of the examinations. Whenever test items are repeated, they should be selected in a distributed manner and approximately equally over all previous examinations to reduce predictability (if a large number of items were taken from the most recent examination). As always, facility licensees are expected to comply with their approved training program requirements, which would be expected to vary based on the licensee's specific circumstances. For example, the same level of question repetition would have less impact if the licensee does not distribute or post its examinations until after they are all complete. The NRC will evaluate every situation on its own merits; the same upper limit may not always be appropriate, nor would it be enforceable unless it was adopted as a regulatory requirement or licensee commitment.

NRC examiners and inspectors that document test item repetition as a weakness must demonstrate that the integrity of the examination was compromised or the discrimination validity of the examination was affected by inappropriate reuse of test items. In December 2003, the NRC revised IP-71111.11, the requalification program inspection procedure, to trigger a performance-based review if and when a facilitys comprehensive requalification examination repeats more than 50 percent of its test items from previously administered comprehensive 112

113 requalification examinations between and among crews undergoing the same requalification training program. The inspectors would apply the guidance in Appendix D of the IP to examine the crews average scores to determine whether they show any pattern of rise over successive crew examination administrations or any unexplained higher-than-expected crew mean scores.

Although the IP focuses specifically on the written examinations, the same 50 percent repetition philosophy would apply equally to the operating test.

IP.8 If a JPM exam is failed, can one of the failed JPM's be used in the retake examination?

It would certainly be appropriate to test the operator to determine if the remedial training was successful, and to include the failed material in that sample. However, the annual operating test given pursuant to 10 CFR 55.59 should consist of a new sample of test material to confirm the operator's overall competence.

In accordance with Appendix F of IP-71111.11, the requalification program inspection procedure, NRC inspectors will ensure that any test items that appeared on the original failed examination are not included as a part of the retake examination. Reusing the same items (missed or correct) from the original failed test on the retake examination is a flawed practice that would falsely bias the test results upward, inflating and distorting true retake performance.

Moreover, including any of the same items on the retake test amounts to little more than a review - not a test as it is operationally defined.

IP.9 During a recent inspection, the validation of a scenario did not match crew response.

The utility's examiner response was to remove the scenario from the exam. What and where are the standards for this?

If the NRC were administering the test, it would not replace the scenario because a crew did not perform as expected unless the scenario was found to contain a serious flaw. Rather, the examiners would document actions taken by each of the crews and later determine if they responded correctly under the given conditions. The examiners would also expect the facility licensee to determine whether the deviation could have resulted from a simulator fidelity problem.

In accordance with 10 CFR 55.4, a training program based on a systematic approach must be evaluated and revised based on the performance of the trained personnel in the job setting.

The fact that a crew deviates from a validated scenario suggests a problem in the training program that may not be fully understood if the scenario is replaced.

IP.10 If an instructor sees a scenario, trains [the] next crew, [then] administers same scenario

[to that crew] (doesn't know in advance), is this a problem?

Yes. This clearly raises a question regarding the validity of the second crew's operating test.

The facility licensee should probably administer an additional scenario to remove any question regarding the operators' competence.

113

114 The facility licensee should also evaluate its testing program to determine if corrective measures are necessary to preclude similar situations from recurring. If the facility licensee's program includes exam security restrictions similar to those endorsed by the NRC in Section D.6 of ES-601 (in NUREG-1021), then the instructor should not have been involved in training activities after gaining knowledge of the exam contents.

IP.11 Can the annual operating exam (simulator & JPMs) be split between two consecutive cycles (i.e., successive retraining weeks which is approximately every 5 weeks for a crew)? The licensed operators received annual JPMs in Nov./Dec. 1999 then received the annual simulator exams in Jan./Feb. 2000. The two together comprise the annual operating exam.

The answer to Question #354 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators Licenses," states that the annual operating test needs to be done at one time and provides an unacceptable example in which the parts of the test are separated by six months. However, your proposal to administer the dynamic simulator and walk-through portions of the operating test during consecutive requalification training weeks (nominally 5 weeks apart) is acceptable (and we understand from our Regional Offices is already being done at some facilities) subject to the following conditions:

- The regulation (10 CFR 55.59(a)(2)) requires each operator to pass an annual operating test.

Splitting the test such that the walk-through is given in one calendar year and the simulator test in the next (as in your example) may create a problem with regard to regulatory compliance.

- The operating test (scenarios and JPMs) must be comprehensive and conducted in accordance with the facility licensees approved, SAT (systematic approach to training) based training program.

- Any significant remedial training that is determined to be necessary should be completed in a timely manner and not deferred until the entire operating test has been administered. If an operator fails either portion of the operating test, this would include removal from licensed duties pending satisfactory completion of the required remedial training and retesting.

114

115 IP.12 The term "biennial" is defined in NUREG-1021, Appendix F as being: "In most instances, a period of time equal to 730 days and synonymous with the term "two years." Biennial requirements can extend beyond 730 days if the requirement is met during the anniversary month of the second year. For example, a biennial medical examination last performed on January 10, 1995, would be due again by January 31, 1997. January is seen as the anniversary month, the period of time between the two examinations is longer than 730 days, but the biennial requirement is satisfied."

This term (biennial) has often been used in discussing the requirement for a comprehensive written examination required as part of the 24 month continuous requalification program noted in 10 CFR 55.59(a).

QUESTION: Is the comprehensive written examination required at the end of the 24 month program to be completed for each licensee within 30 days of the anniversary date of their last written examination?

This issue has been addressed in Section C.1.a of ES-605 (in NUREG-1021).

IP.13 What are the requirements for sampling all items in 10 CFR 55.41 and §55.43 on the requalification exam?

As noted in response to a similar question related to the operating test (refer to Question 603.1),

the sample should be thorough or broad, but not every item listed in the regulation has to be covered on every examination. Moreover, the response to Question IP.3 indicates that operators should be at risk of being evaluated on all of the applicable items during any examination. Since the requalification examinations are part of a systems approach to training (SAT), they should emphasize the topics covered during the training cycle; however, the NRC expects that they would also cover topics from outside the requalification cycle in order to determine areas in which retraining is needed (refer to 10 CFR 55.59(c)(4)(i)).

IP.14 What happens if an individual is unable to successfully complete the requal exam prior to the end of the 2-year program cycle? He is already administratively restricted from standing watch.

As noted in response to Question #328 in NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses," it is only under extenuating circumstances (e.g., a special temporary assignment to a remote location, an extended illness, or enrollment in a degree program) that the NRC condones removing licensed operators from the requalification program. In such cases, the NRC generally invokes the provisions of 10 CFR 55.59(b), "Additional Training," to ensure that the affected operator is qualified prior to returning to licensed duties. Planned absences are processed as described in Section C.1.c of ES-605 of NUREG-1021. Unplanned incompletions and restorations should be documented and handled on a case-by-case basis in consultation with the NRC regional office.

115

116 IP.15 It is not uncommon to have on-shift crews staffed to beyond the minimum complement required by technical specifications. For this type of situation, is it acceptable to have a licensed operator participate in one scenario and still fulfill the requirement of completing an annual operating test (provided the facilities training program allowed this)? NUREG-1021, ES-604, is quite clear on crew dynamic simulator tests needing to be two scenarios but does not specify whether or not every crew member needs to be in an evaluated position for both scenarios.

As noted in the response to Question IP.4 above, facility licensees are not obligated to follow NUREG-1021 unless it is incorporated as part of their approved requalification program.

Although there is nothing in the regulations that dictates how many scenarios are required and whether every operator has to be in an evaluated position during each scenario, the guidelines in the NUREG are based on good practices and expectations that are widely practiced in the industry. For example, it is a good practice to train and test the crews in the same configuration as they operate in the control room; to do otherwise would run the risk of providing negative training. The fact that you have more than the minimum required number of operators on shift, does not mean that you should leave some of them on the bench during a simulator scenario or a real event in the control room. The NRC would expect you to construct your operating tests with a sufficient number of events and scenarios to ensure that every operator on the crew gets a meaningful evaluation.

116

117 Simulation Facilities Continued Assurance of Simulator Scope, Fidelity, and Testing Sim.1 Why is scenario-based-testing the simulators performance a challenge?

NRC operating test scenarios and scenarios used for performing control manipulations that affect reactivity to establish eligibility for an operators license may be used as simulator performance tests. Hence the term scenario based test (SBT). The overriding challenge to licensees is to conduct the SBT in a manner sufficient to ensure that simulator fidelity has been demonstrated (and met) so that significant control manipulations are completed without exceptions, simulator performance exceptions, or deviation from the approved training scenario.

Simulation facility licensees should consult RG 1.149, Revision 4, Regulatory Position No.3 which describes the staffs acceptance and endorsement of the SBT implementation guidance described in NEI-09-09, Revision 1. NEI-09-09, Revision 1, is an acceptable method for demonstrating compliance with the requirements of Section 3.4.3.2 and 4.4.3.2 of ANSI/ANS-3.5-2009 regarding simulator SBT.

Sim.2 What impact do computer upgrades and re-hosting have on performance tests?

Upgrades to licensee simulation facility plant-referenced simulator computer systems and re-hosting onto new computer platforms should not alter model performance characteristics. It is expected that similar results will be achieved when comparing performance test runs after an upgrade or re-host to the same test runs before the upgrade or re-host. Verification and validation testing shall be conducted, as required by Section 4.4.1 and 4.4.2 of the standard (2009, and 1998), following a system upgrade or re-host to confirm that model characteristics have not changed. Although not a requirement of the ANSI/ANS-3.5 standard or the 10 CFR 55.46 regulations, it is prudent to run the simulator operability tests (i.e., steady-state, and transient tests) following a computer upgrade or re-host to ensure or demonstrate no unintended consequences to models.

117

118 Sim.3 Are simulator design specifications required to be updated?

Plant-referenced simulators model systems of a reference plant. Reference plant is defined in 10 CFR 55.4 as the specific nuclear power plant from which a simulation facilitys control room configuration, system control arrangement, and design data are derived.

ANSI/ANS-3.5-2009 (1998), Section 5.1.2 Simulator Design Data Base Update, requires that the simulator design data base (i.e., design specifications) shall be periodically updated (i.e.,

within 18 months of the reference units commercial operation date or the simulators operational date, whichever is later; or following the initial update, new data shall be reviewed, and revised, once per calendar year). Maintaining the fidelity of the plant-referenced simulator includes updating the design specifications. The particular methodology for updating design specifications is determined, for the most part, by the facility licensees simulator configuration management control (i.e., ANSI/ANS-3.5 standard requires, among other criteria, that a means for establishing and maintaining a simulator design baseline shall be included in the configuration management).

Sim.4 What is actually required when documenting SBT (Scenario-Based Test)?

Please refer to NEI-09-09, Revision 1, as well as RG 1.149, Revision 1.

Sim.5 What is the periodicity for SBT?

SBT periodicity is not specifically addressed by the regulations or the industrys adopted standard ANSI/ANS-3.5-2009. Simulator scenario based tests (SBT) are uniquely developed as NRC operating tests (or in the case for which a scenario is developed for performing control manipulations that affect reactivity to establish eligibility for an operators license). That said, periodicity for a specific SBT per se is not appropriate since that SBT may or may not be used again in the future. However, should the specific SBT be used again (without alteration or modification), the expectation is that the specific SBT undergo performance testing again before it is used again to ensure that fidelity has not changed since the last time it was performed.

118

119 Sim.6 What is the staffs position with regard to installing modifications on the simulator before being installed on the referenced plant?

In general, the staff accepts and endorses industrys consensus standards (ANSI/ANS-3.5-2009

(-1998, -1993, and -1985)) through incorporation by reference in RG 1.149 (Revisions 4, 3, 2, and 1 respectively). Each revision of the ANSI/ANS-3.5 allows for simulator modifications to be completed either before or after the modifications in the reference plant. Decisions as to timing of the simulator modifications should be based on an analysis of training needs and must also take into consideration proposed uses of the simulator and the effect on operator actions.

When a plant-referenced simulator is used in NRC initial and or licensed operator requalification examinations (and, in some cases, for meeting eligibility requirements of 10 CFR 55.31),it must accurately reflect current design of the referenced plant and not produce negative training. In cases where a plant-referenced simulator differs from its reference plant as a result of plant modifications, the NRC expects differences training to compensate for deviations from the reference plant to preclude or compensate for any negative training. For example, if a reference plant modification is planned for completion in the last few weeks leading up to an initial license examination, it might be desirable to delay installation of the modification on the simulator until after the examination to avoid disrupting the orderly planning and administration of the exam.

However, this choice could call any licensing decision made using that simulator into question because the potential exists that skills demonstrated on the simulator would be different from what would be required in the plant for which a license is to be issued. In this case, a facility licensee could request in writing Commission approval to use the simulator while it differs from the reference plant. The request should address steps to be taken to prevent or compensate for negative training. The NRC has the option of granting such a request.

Several facility licenses have successfully implemented reference plant modifications (design changes), such as feed-water controls and digital EHC main turbine-generator controls, on the plant-referenced simulator without any regulatory approval or change in simulator status as a plant-reference simulator.

Sim.7 Will the staff determine whether or not a particular model is correct?

No, it is the responsibility of each facility licensee that maintains a simulation facility. Each facility licensee is expected to ensure their simulator adequately demonstrates expected plant response through appropriate testing. NRC staff evaluates and assesses whether or not the simulation facility complies with the scope and fidelity requirements describe in 10 CFR 55.46 during biennial baseline licensed operator requalification program inspections (IP-71111.11).

119

120 Sim.8 We have replaced some models with new models; What if the new model shows a different response than the old model? (With regard to malfunctions such as LOCAs and transients with no plant data).

Facility licensees that maintain a simulation facility must demonstrate continued assurance of simulator fidelity by conducting performance testing in a manner sufficient to ensure that simulator fidelity has been demonstrated and met (10 CFR 55.46). If the results of performance test are significantly different (e.g. does not meet the same acceptance criteria as before) a re-evaluation should be conducted to determine the extent of condition and whether or not a detailed engineering analysis is needed to resolve modeling discrepancies identified.

If the re-evaluation reveals significant modeling problems with the previous model and the model had been used to negatively train operators, then reactor safety may have been impacted. The facility licensees corrective action program would need to determine the extent to which the operators had been negatively trained. Retraining, if indicated, would follow.

Generally, NRCs licensed operator requalification program baseline inspections monitor performance in this area.

120

121 Sim.9 What constitutes an adequate degree of replication and if not adequate what is the safety significance?

The degree of replication depends on the type of evolution (steady state, transient/malfunction, normal evolution) and the applicable operability test acceptance criteria assuming adequate acceptance criteria have been established. For example, the ANSI/ANS-3.5 standard requires that certain steady state parameters meet a 2 percent tolerance. If there has been an identification of a fidelity issue in which the applicable parameter is beyond 2 percent, then the degree of replication is unacceptable since it would fail the steady state acceptance criterion.

For alarms and automatic action (or interlocks), the plants calibration and surveillance testing acceptance criterion (instrument tolerances) should be an adequate method for determining the degree of replication.

An ancillary question to the above is: What are the first order principles for NRC staff analysis in order to determine if a simulator fidelity performance deficiency is minor or not with respect to 10 CFR 55.46(c)(1) and what safety significance level could result? The issue is related to the human performance attribute in the three reactor safety cornerstones of initiating events, mitigation, and barrier controls per MC 0612, Appendix B. Performance deficiencies are more than minor and are of very low safety significance if they involve actual or potential impact on operator actions per MC 0609, Appendix I, [Blocks 13, 14, and 15, along with the basis statements for the questions in the blocks] (Note: This is a broader definition of negative training from that defined in ANSI/ANS 3.5 definitions section). These issues are not of greater significance because they did not have an adverse impact on operator actions such that safety related equipment was made or would have been made inoperable during normal operations or in response to a plant transient. If there was an effect to this degree, the performance deficiency would be analyzed per MC 0609, Appendix A (PRA basis). Minor performance deficiencies that have no effect or impact on operator actions are generally not documented in the inspection report.

121

122 Sim.10 Record retention: ... retained for four years after the completion of each performance test or until superseded by updated test results. How long can the or in this statement be - the life of the plant, for example?

Four Year Record Retention: do records older than four years have to be retained, such as acceptance tests from original certification, etc?

Per 10 CFR 55.46(d)(1), the performance test (as defined in 10 CFR 55.4) results are expected to be retained for four years after the completion of each performance test. Generally, simulator performance tests are conducted on a periodic basis in accordance with ANSI/ANS 3.5 and the facility licensees simulator testing schedule. The test results are subject to review by the NRC and a retention period of four years is prescribed so that an evaluation and comparison can be made for a given performance test over a period of time (up to four years) to ensure that simulator fidelity is being maintained. However, if a performance test is not repeated until a period of more than four years has passed, then the record of the performance test should be retained until superseded by the subsequent test. When a performance test is superseded before four years, then the four year period resets for the updated test. The rule still requires that the facility licensee conduct performance testing throughout the life of the simulation facility.

Keep in mind that the standard requires that: (A) in Section 4.4.1, that verification tests (i.e.,

software design documentation) be generated and be updated. (B) in Section 4.4.2, that validation test documentation is generated and that a record of the conduct of this test, the tests results, and the tests evaluation be maintained. It further requires that these tests be conducted prior to the simulators use in training and examination for the following situations: (1) completion of simulator initial construction; (2) whenever models are changed or modified in a way that potentially affects fidelity relative to the reference unit; and, (3) whenever there are changes which have the potential to affect simulator capabilities or repeatability. (C) in Section 4.4.3.1, that operability tests be conducted on a periodic basis and that a record of the conduct of this test and its evaluation be maintained. (D) in Section 4.4.3.2, that SBTs be tested before use for operator training or examination and that a record of the conduct of these tests, and the evaluation of the tests results be maintained. Implementing these standard requirements are measures acceptable to the staff for implementing the demonstration requirements of 10 CFR 55.46(c)(1).

Updating and maintaining tests documentation is ongoing. No relief is provided in the standard that allows cessation of maintaining the test records. Simulator test records provide evidence of simulator fidelity. If for no other reason, it would be prudent for licensees to retain all such records as a means of providing assurance of fidelity should it be brought into question by a future plant or industry event.

122

123 Sim.11 Core performance: what standards are being used to ensure the simulator performance replicates reference plant nuclear and thermal hydraulic operating characteristics, since there is a broad range of core models out there?

ANSI/ANS-3.5-2009 (1998)(1993)(1985) establishes the functional requirements for the plant-referenced simulator. It also establishes the criteria for the degree of simulation, performance, and functional capability. With regard to ensuring that the nuclear and thermal hydraulic characteristics are replicated appropriately, the standard, in Section 3.1, Simulator Capabilities, requires that the response of the simulator resulting from operator action, no operation action, improper operation action, automatic reference unit controls, and inherent operating characteristics shall be realistic and shall not violate the physical laws of nature.

Nuclear and thermal hydraulic characteristics are fundamental and must be consistent with the laws of nature. The standard (2009), in Section 4.1.3.2, requires that performance of procedures on the simulator, including core performance type procedures, shall be compared and demonstrated to represent correctly the response of the reference unit at the same power level consistent with the reference unit procedures and data availability. The standard establishes six acceptance criteria with regard to simulator response during the conduct of the performance tests: (1) be the same as the reference unit startup test procedure acceptance criteria; (2) be the same as the reference unit surveillance procedure acceptance criteria; (3) be the same as the reference unit normal operating procedure acceptance criteria; (4) require that the observable change in the parameters correspond in direction to those expected for a best estimate of normal unit operation; (5) require that the simulator shall not fail to cause an alarm or automatic action if the reference unit would have cause an alarm or automatic action under identical circumstances; and (6) require that the simulator shall not cause an alarm or automatic action if the reference unit would not cause an alarm or automatic action under identical circumstances. These standards are quite high when applying them to the nuclear and thermal hydraulic characteristics.

Sim.12 10 CFR 55.31 versus §55.46: If a candidate got some of his reactivity manipulations on a core in the plant that was then refueled and he then got additional manipulations, the earlier manipulations would still count and yet this is not the case with the simulator core load. Why?

Reactivity manipulations performed on the plant-reference simulator for an applicant to meet the experience eligibility requirements may be credited when the simulator, at the time of performance, meets the requirements of 55.46(c)(2)(i) and (ii). The rule requires that the plant-referenced simulator utilizes models relating to nuclear and thermal-hydraulic characteristics that replicate the most recent core load in the nuclear plant for which a license is sought; ... The Commission, in its response to public comments during the rule making process, interpreted most recent as the current core, or if in a refueling outage, the previous core. The intent is to ensure that the applicant has a like-kind experience as he would have in the reference plant. As is the case with reactivity manipulations conducted on the plant, any appropriate reactivity manipulation performed on the simulator may be credited provided the simulator replicates the most recent core at the time of the manipulation.

123

124 Sim.13 Please define the term replicate as found in 10 CFR 55.31 and §55.46.

SECY-01-0125, dated July 10, 2001, Analysis of Public Comments, Comment 3-3 Response addressed this question. The Commission believes that the terminology (in the proposed rule and subsequently in the final rule) is appropriate and consistent with ANSI/ANS-3.5-2009 (1998). It means that the plant-referenced simulators nuclear and thermal-hydraulic models operate within the tolerances specified in Section 4.1.3, Steady-State and Normal Evolutions, of the industry standard.

See also Sim.9.

Sim.14 Is core performance testing the same thing an operator would do in the course of his job?

No. The regulations, in 10 CFR 55.4, define performance testing as testing conducted to verify a simulation facilitys performance as compared to actual or predicted reference plant performance. The term Core refers to the nuclear reactor core, including but not limited to the design, configuration, and nuclear and thermal hydraulic characteristics of the core as well as the associated nuclear instrumentation that monitors or measures the various parameters which provide insight to the behavior and operating characteristics of the core.

Core performance testing means testing conducted to verify a simulation facilitys core performance replicates actual or predicted reference plant core performance. Core performance testing is not the same thing an operator may or may not do in the performance of his job. Absent conduct of the same core performance tests on the simulator as are performed on the plant and demonstration through such testing that the simulator meets actual or predicted plant performance within the acceptance criteria of the ANSI/ANS 3.5 standard, the NRC may not be able to confirm core replication in the simulator. This could adversely impact crediting of experienced gained on the simulator.

See also Sim.11.

Sim.15 Core vs. Thermal-hydraulics replication: weve talked a lot about core performance testing: how does the NRC propose how to test thermal-hydraulic performance?

Generally, the NRC does not prescribe how to conduct a performance test, but instead challenges a licensee to demonstrate that certain regulatory requirements are being met.

Thermal-hydraulic performance could be demonstrated by comparing simulator performance to actual plant performance during startup, power ascension, normal operation, and transient response. Startup test procedures and licensee event reports are good data sources.

124

125 Sim.16 Is it acceptable to do off-line testing of core performance (i.e., not use the actual simulator but instead a stand-alone system) to satisfy 10 CFR 55.46(d)(1)?

There is nothing to preclude core performance testing off-line for the sake of designing, debugging, and testing without other system interfaces to assure that the model is ready to be integrated into the simulated plant. However, fully integrated core performance testing on the plant-referenced simulator is expected and necessary to ensure that the appropriate input and output from and to other models are sufficient in scope and fidelity to ensure that the simulator responds as the reference plant would under the same operating conditions.

Sim.17 Updating models: is it encouraged to update our reactor vessel/core models to comply with 10 CFR 55.46?

The Commission in its statements of consideration during the rule making, emphasized that facility licensees would not be required to update their core models in order to comply with the requirements of 55.46. Refer to Regulatory Guide 1.149, Revision 3. This assumes that the simulator core model has been performance tested and the test results meet the appropriate acceptance criteria when compared to the reference plant performance or best estimate performance where actual performance data is not available.

Sim.18 If the reference plant undergoes a significant design change, such as a steam generator replacement or a power up-rate, that leads or lags the installation of the same change on its plant-referenced simulator, will it be necessary to obtain Commission approval, pursuant to 10 CFR 55.46(b)(1), to use other than a plant-referenced simulator to administer the operating tests required by the regulation?

The fact that a simulator may lead or lag design changes made to the reference plant is accommodated by ANSI/ANS-3.5 (2009, 1998, 1993, and 1985). Given that the purpose of making the design changes is, ultimately, to ensure that the simulator demonstrates reference unit response, any reasonable lead or lag in the process will not alter the simulators plant-referenced classification.

The Operator Licensing Program Office expects that (1) any simulator design changes would be tested in accordance with ANSI/ANS-3.5, (2) any simulator-to-plant differences resulting from a lead or lag situation would be appropriately addressed in the training program (as would any unit-to-unit differences at a multi-unit facility), and (3) that such differences will be resolved within the time frame specified in the ANSI standard. The NRC may refuse to administer operating tests on a simulator that has not been appropriately tested or if it is unable to meet the requirements of 10 CFR 55.46(c)((1)(i), i.e., if the simulator has not demonstrated the expected plant response for conditions to which it was designed to respond, the differences in response must be evaluated to confirm they do not interfere with the conduct of the operating tests.

Sim.19 If a comparison is made between actual core plant data (Dynamic Rod Worth for Control Bank D is measured at 910 pcm) verses engineering predicative core data (calculated at 125

126 1000 pcm) verses simulator core performance data (measured at1090 pcm) and the results show that the delta between the plant data verses predictive data is in an acceptable range, is there a performance issue with the plant-referenced simulator?

The situation described shows a deviation of more than 18% between the simulators rod worth and the reference plants rod worth for Control Bank D. Although the deviation between the simulator and predictive is less than 10%, the simulators performance is judged against the actual plant since actual data is available and measured. Additionally, the ANS-3.5 tolerances apply in this case. The simulators performance deviation would be considered a simulator performance exception as well as a modeling discrepancy identified from performance testing.

As a result, this type of modeling discrepancy could increase the potential for negative training and operator error.

126

127 10 CFR 55 Questions related to the operator licensing regulations CFR.1 How long does it take for an exemption request to be received and to be answered?

The time required will depend on the nature of the request and the quality of the licensees submittal. Plan at least two months to get an answer. If the NRC requires additional information to make a decision, it will probably take longer.

127

128 General Questions that do not fit within another category Gen.1 Is there some way to do a better distribution of clarifications/rulings from one site in the region to another? This would help all of us meet your expectations.

One of the NRC's goals in establishing this web site is to improve communications with facility licensees and to enhance consistency.

Gen.2 Will there be a revision to NUREG-1262 at any time soon? NUREG-1262 contains information that conflicts with NUREG-1021. Is there any intent to make NUREG-1262 current?

No. The NRC does not plan to revise NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses," which was published in November 1987. At the NRC staff's request, the Nuclear Energy Institute provided a list of questions and answers that appear to be out-of-date, but revising the NUREG remains a low priority. If there are conflicts between NUREG-1262 and any other guidance issued since then (including NUREG-1021 and the answers to these questions), the more recent guidance would take precedence.

Gen.3 Has the question been asked about the "intellectual rights" of the examination work product owner versus publish of examinations?

Examination authors are not prohibited from copyrighting their work. However, the NRC cannot accept copyrighted materials unless the holder of the copyright signs a release form to allow its publication. When those materials are placed in the public document room, users are permitted to make one copy for personal use. If additional copies are required, the user will have to obtain permission from the copyright holder.

128

129 Gen.4 Is the ES-601 definition of "low power" serious?

Low power - Is it really criticality to 5%?

Low power scenarios are defined as criticality to 5% reactor power. Is this the expectation to receive credit for a low power scenario?

Yes. The NRC staff's evaluation of shutdown and low-power operations at commercial nuclear power plants, which was reported in NUREG-1449, included operations with the reactor in the subcritical (i.e., shutdown) state and in transition between subcriticality and 5 percent power (i.e., low power). When NUREG-1021 was revised to place more emphasis on those operating conditions, it made more sense to use the same definition than to develop a new one. The definition, which has been incorporated in Appendix F of NUREG-1021, applies to both the initial and requalification examinations.

The NRC intends for the operating tests to sample the full range of operating conditions and power levels so they do not become predictable. It is unlikely that the NRC would deny credit for a scenario simply because it exceeded the power limit specified in a somewhat arbitrary definition.

Gen.5 What is/where do I find my "Commission Approved" training program?

As noted in the Statements of Consideration for the 1987 amendment to 10 CFR 55, a facility licensee's training program is considered Commission-approved when it becomes accredited by the National Nuclear Accrediting Board.

Gen.6 How familiar are, and what kind of training have the examiners received on the SAT process? How familiar (knowledgeable) are the headquarters management on the SAT process? What kind of training have they received?

The staff of the NRC Operator Licensing Program Office includes training and assessment specialists who are well-versed on SAT-based training processes and have many years of combined training experience. Issues and questions that come up regarding SAT-based training requirements and expectations are referred to one or more of those specialists for resolution. NRC examiners and managers having responsibilities in this area have received instruction on the SAT process during periodic operator licensing examiner training and conferences.

129

130 Gen.7 I would like to see the NRC go more toward an inspection process for plants that volunteer to write the exams. Have only one NRC examiner involved, allow the utility to administer all parts of the exam and use the resident if more oversight is needed during the exam administration. The NRC should continue to make the final licensing decision.

Comment noted. Although the NRC favors reducing unnecessary regulatory burden, the examination policies will only be changed if the NRC concludes that the changes will not have a negative impact on reactor safety, public confidence, efficiency and effectiveness. At the present time, the NRC sees significant benefit in continuing its current level of involvement in the operator licensing process.

Gen.8 NRC needs to understand that increased difficulty of exam process is a negative motivator and could be a distraction to competent board operators. Recommend survey to understand scope and potential impact on safe plant operations.

The examination process seems to be getting harder as compared to a few years ago.

Exam difficulty has gone beyond reason and is impacting the requal program. People are not willing to put up with the hassle and it does not result in better operators. It is impossible to meet question standards and avoid "tricky" questions, very knowledgeable operators can appear less that competent based on complexity of question rather than a test of knowledge.

As reported in Attachment 1 (Section 1) of SECY-98-266, the NRC has also noted a slight decrease in the average passing rates on both the written and operating portions of the facility-prepared examinations when compared with the passing rates on NRC-prepared examinations.

However, the decrease could be caused by a number of factors including variations in the average level of experience of the license applicants, changes in the quality of the training or the facility licensee's threshold for screening its applicants before they take the licensing examination, or variations in the average level of difficulty of the examinations. Although the staff did not intend for the level of difficulty or the failure rate on the examinations to increase, the examiners' efforts to achieve NRC standards regarding the cognitive level of questions and to improve the plausibility of the distracters may have improved the discrimination validity of the examinations. Consequently, those applicants who may have passed an examination containing lower cognitive level questions on which some of the distracters could be eliminated as implausible are now having more difficulty selecting the correct answers; in essence, their chances of passing the examination by guessing some of the correct answers have diminished.

Considering the historical fluctuation in the average examination passing rates and the other factors that could be responsible for some or all of the observed decline, the NRC has concluded that any increase in the level of difficulty is not significant.

The Operator Licensing Program Office will continue to monitor the applicants' performance for indications that the examinations are becoming too difficult. The initial operator licensing examination performance trends since 1991 are available for review on the Operator Licensing Process page.

130

131 Gen.9 The most common issue raised by Hot License Candidates and Requal license holders surround the issue of "trick questions" and operator written exams not being a fair test of operator knowledge.

The NRC exam has become an exercise in exam taking skills instead of a knowledge assessment.

The NRC goes to considerable lengths to ensure that its examinations measure what they are intended to measure, thereby enabling the NRC to distinguish between applicants who have and have not mastered the knowledge and abilities required to be safe nuclear power plant operators. The principles of fairness, validity, and safety have guided the NRC throughout the process of developing and implementing NUREG-1021. As stated in Attachment 1 of Appendix B of NUREG-1021, the NRC strives to minimize unnecessary difficulty, trickiness, and irrelevancy in its written examination questions. Authors and (multiple) reviewers are expected to identify and correct these psychometric deficiencies. Moreover, Section E.4 of ES-401 encourages facility licensees to peer-validate the written examination in a final effort to identify and correct deficiencies that might affect the validity of the examination.

Although the NRC has increased its emphasis on higher cognitive level questions and the plausibility of distracters in an effort to enhance the discrimination validity of the examinations, some may have misinterpreted these actions as an effort to trick or fool otherwise knowledgeable applicants. Truly knowledgeable applicants should be able to pass the examination regardless of their test-taking skills. Applicants who rely too much on their test-taking skills or their ability to guess the right answer after eliminating the implausible distracters should not be able to pass the licensing examination.

131

132 Gen.10 Guidelines shouldn't be open for individual examiner interpretation if it could show up as a weakness in the exam report. Example: Amount of question/operating test overlap on the requal exam from week to week.

There are still regional "requirements" (not NUREG interpretations) outside of NUREG-1021 such as ROI's [regional office interactions], etc. for example: "one scenario must have a computer failure." Why are these things still out there? Shouldn't they be in 1021 if they are required?

What is the NRC doing to ensure that the examiners are working to the same standards?

Comments noted.

The NRC's existing measures to maintain consistency in the examination process were summarized in Attachment 1 to SECY-98-266, "Final Rule - Requirements for Initial Operator Licensing Examinations." NRC examiners are expected to comply with the guidelines in NUREG-1021 and to exercise good judgment in those areas requiring a subjective evaluation.

The reviews and audits conducted by NRC regional management and the operator licensing program office and the continuing training program for examiners help minimize individual examiner interpretations and ensure consistency.

Section B of ES-201 requires the NRC Regional Offices to obtain approval from the operator licensing program office prior to knowingly deviating from the intent of the NUREG or implementing any initiative that has the potential to undermine examination consistency.

Gen.11 Need region workshops to calibrate us on future JPM direction.

We may want to have an exam writing workshop.

Who would be interested in putting together a utility sponsored exam question writing seminar?

Suggest national NUREG-1021 workshop twice a year with focus on facilities with upcoming exams (within 6-12 months).

The NRC has sponsored and participated in a number of examination workshops and, to the extent possible, will continue to work with facility licensees and industry training groups in this area. The NRC encourages facility licensees to pool their resources and work together to develop their examination-writing skills. The regional training organizations, Nuclear Energy Institute (NEI), and Institute of Nuclear Power Operations (INPO) might be able to provide support in this area.

Suggestion noted.

132

133 Gen.12 Will you "endorse" the Sonalyst Workshop?

The NRC reviewed the Sonalyst Workshop to ensure that it was consistent with NUREG-1021.

Legally, the NRC cannot endorse specific vendors or programs provided by them.

Gen.13 Install a bulletin board on the NRC web page for lessons learned as discussed in the workshop.

Suggestion noted. The operator licensing program office plans to use the Regulations, Guidance, and Communications page of its web site to promulgate lessons learned, guidance, policy clarifications, and interpretations that arise between revisions of NUREG-1021.

Gen.14 Can we get a copy of the two year NRC examining schedule?

The examination and inspection schedule (covering at least the next year) is posted on this web site. We expect to update the schedule at least quarterly.

Gen.15 Why did the NRC, INPO [Institute of Nuclear Power Operations], and NEI [Nuclear Energy Institute] meet [on December 16, 1999] to discuss future options for the exam process without involving industry representatives in the process?

The NRC issued a meeting notice on December 7, 1999, and members of the public and nuclear industry were welcome to attend. The NRC assumed that NEI would follow up with the appropriate operator licensing task force representatives as it had for previous meetings.

Gen.16 How will PRA [probabilistic risk assessment] need to be identified in future exams?

Section D.1.f of ES-301 (in NUREG-1021) requires examination authors to consider PRA insights (e.g., dominant accident sequences and risk-important operator actions) when preparing the operating tests. The Examination Outline Quality Checklist (Form ES-201-2) requires NRC examiners to assess whether plant specific priorities (including PRA and IPE insights) are covered in the appropriate exam section. Although there is currently no requirement to identify which test items address the PRA insights, the examination author should be able to explain to the chief examiner how those insights were covered. The NRC has no immediate plans to change this requirement.

133

134 Gen.17 How do we stabilize this process so that it won't have a detrimental effect on industry staffing needs? (Taking into account the huge demands that will be necessary due to the aging workforce.)

Many of the changes that have recently been made in the examination process can be directly attributed to industry requests. The NRC will continue to be responsive to its industry stakeholders as long the agency's goals related to safety, public confidence, efficiency, and effectiveness are not compromised. In that regard, the operator licensing program office will continue to work with the NEI operator licensing focus group in an effort to identify those changes that are in the best interest of the industry and the public.

Gen.18 Can facility licensees electronically submit NRC Form 398, "Personal Qualification Statement -- Licensee," and NRC Form 396, "Certification of Medical Examination by Facility Licensee?"

The NRC has permitted the electronic submittal of documents by outside participants since January 1, 2004, when the NRC rule governing electronic submittals ("E-Rule") took effect. This rulemaking expanded participation in electronic communication by giving all licensees, vendors, applicants and members of the public the option of submitting documents to the NRC in various electronic formats, including CD-ROM, e-mail, and a special Web-based interface, the Electronic Information Exchange ("EIE"). EIE has digital signature capabilities, and its use is explained at length in the guidance document accompanying the E-Rule Appendix A, United States Nuclear Regulatory Commission Guidance for Electronic Submissions to the Commission. The E-Rule and accompanying guidance can be found at http://www.nrc.gov/site-help/e-submittals.html.

Section 3.7 of the guidance document, entitled "How to Submit Documents," explains the two kinds of electronic signature available to the sender: one method is the digital signature, which uses a digital ID certificate; and the other method is a handwritten signature, which is then scanned and submitted via EIE and use of the digital ID. The way to obtain a digital ID certificate is explained in the guidance. Forms that have only one signature line may be submitted using either electronic signature method. However, forms with two or more signature lines (such as NRC Form 398, which must be signed by the applicant, the training coordinator, and the senior management representative on site) must rely on the handwritten scanned signature method because of EIE's limited digital signature capability per transmission. The facility licensee is not required to submit a paper copy of its electronic submittals. Section 3.7 of the guidance explains more fully the procedures for electronic submittal and the reasons for them.

134

135 Gen.19 During a recent inspection, it was noted that the facility licensee's UFSAR requires the control room operators to take Potassium Iodide (KI) pills under certain post-accident conditions to minimize long term consequences from potential exposure to radionuclides. Since Section 5.2.2 of ANSI/ANS-3.4-1983, Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants, requires operators to be free of any conditions that are considered by the designated medical examiner as significantly predisposing to incapacity for duty, including any treatment involving drugs, chemicals, diets, or other agents, is the licensee required to test its control room operators for sensitivity to KI as part of its periodic physical examinations?

The NRC staff has researched this issue and determined that true allergic or allergic-like reactions are rare in people who take KI and that testing for this is not practical. The current literature also suggests that allergy to seafood or radio-contrast material does not necessarily confer an elevated risk for allergy to KI, so the usual history of "iodine allergy" would not be helpful. Moreover, the experience in Poland (after Chernobyl) has shown that serious reactions to KI itself were extremely rare. Therefore, it appears that the administration of KI pills would not significantly predispose the operators to incapacity for duty. However, facility medical examiners should evaluate each operators specific circumstances in light of their facility licensees KI administration practices to determine if the operator is medically qualified or if some type of license restriction is necessary.

135

136 Gen.20 What is a permanent disability? The rule (10 CFR 55.25), Regulatory Guide 1.134, and ANSI/ANS-3.4 do not define it; they use terms like minimum conditions and disqualifying conditions.

You are correct; there is no formal definition of a permanent disability.

Section C.3 of ES-605 of NUREG-1021 and Information Notices 04-20, 94-14, and 91-08 (all of which are available via links from the operator licensing web page) provide additional information regarding the staffs expectations with respect to medical standards for licensed operators. With regard to permanence, Section C.3.a of ES-605 indicates that if an operator does not meet the specific minimum standards/requirements in the applicable version of ANSI/ANS-3.4 but is expected to meet those standards (without exception) again in the future, then the operators condition/disability is considered temporary and does not need to be reported to the NRC; however, the facility licensee is expected to administratively restrict the operators activities, as appropriate, during the term of the condition/disability. While most of the medical conditions/disabilities, including those that result in failure to meet the minimum requirements for medical qualification, identified in ANSI/ANS-3.4 are probably permanent, it is up to the examining physician to evaluate each operators situation on a case-by-case basis and assess whether the operator will be capable of meeting the standards in the foreseeable future.

For example, the facility should consider reporting a condition for an operator who requires medication to meet the minimum standard for blood pressure (i.e., less than or equal to 160/100 mmHg), unless the physician can reasonably determine that the condition will be controllable without medication in the foreseeable future. The NRC will review the facility licensees administrative controls and its physicians explanation for why the condition was considered temporary during the licensed operator requalification program inspections.

If an operator develops any permanent physical or mental condition that could adversely affect his or her performance of assigned operator job duties or cause operational errors endangering public health and safety, it must be reported to the NRC within 30 days of learning of the diagnosis (per 10 CFR 55.25 and §55.33(a)(1)). It does not matter whether the operator has tripped the specific minimum requirement or the related disqualifying condition threshold in ANSI/ANS-3.4 - all conditions, disabilities, and incapacities should be reported to the NRC for evaluation, regardless whether or not the facility has implemented compensatory measures. If an operator develops a condition that is not identified in the industry- and NRC-approved ANSI standard, but the examining physician believes that it could affect the operators performance or cause errors, then it would be prudent to report it anyway (or at least enquire whether it should be reported).

If the examining physician concludes that the operators condition, disability, or incapacity does not affect performance or safety, he or she can request and justify a waiver of the medical requirement; for example, a color-blind operator might be granted a waiver based on a satisfactory practical test. If the operators condition, disability, or incapacity can be safely accommodated by a restriction on the license (e.g., no-solo, more frequent monitoring, or requiring medication), then the physician should make an appropriate recommendation to the NRC on Form 396. However, if the operators condition, disability, or incapacity is such that it cannot be reasonably waived or accommodated, then the facility licensee should request the NRC to terminate the operators license.

136

137 Gen.21 In the past, if we had an individual on daily medications for hypertension and the condition was stable we would send an information letter to the NRC. If the condition was not stable, or there were multiple medications for treatment, or the condition was outside the regulation we would request a no solo license. How do we interpret this now that NRC Form 396 has been revised? Many physicians treat hypertension well below the 160/100 limit allowed by the ANSI standard. If an individual is treating with medications because his blood pressure is 148/88 and his physician is more aggressive, do we check the information only box or do we check the medication box on NRC Form 396?

The purpose for placing a take your medicine condition on operators licenses is to impress upon them the importance of maintaining their medical qualifications and to ensure that their medical condition and general health will not adversely affect their performance of assigned duties or cause operational errors endangering public health and safety (as required by 10 CFR 55.33). Presumably, if the examining physician directs an operator to take a prescription medication for whatever reason, it is to protect their general health and to prevent them from exceeding a threshold that would disqualify them from performing licensed duties - in this case the 160/100 mmHg blood pressure limit - or affect their job performance. If we fail to put a condition on an operators license at the time of initial diagnosis and treatment, we may likely not get another chance to do so, assuming the treatment is successful. Therefore, you should check the medication box on NRC Form 396 even if the threshold for disqualification has not yet been exceeded.

Gen.22 When we select the "Solo operation is not authorized" box on the revised NRC Form 396 there is no corresponding box to enter the wording of the restriction - will the wording be provided by the NRC based on the license holder's status (RO vs. SRO)?

Yes, the NRC will enter the standard wording based on the individuals license level. The instructions on the back of Form 396 refer to Section C.3.c of ES-605 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, which includes the standard wording for a number of operator license medical conditions.

137

138 Gen.23 If we previously submitted an "Information Only" letter for a medical condition that did not result in a license restriction (e.g., a well-controlled asthmatic or hypertensive on medication), do we need to submit a request for the new "Must take medication as prescribed to maintain medical qualifications" restriction based on the revised NRC Form 396?

Per 10 CFR 55.25, you do not need to submit a revised NRC Form 396 during the term of an operators license unless there is a permanent change in the operators medical condition that would cause him or her to fail to meet the requirements of 10 CFR 55.21. The next time the operators license is due for renewal, you would need to submit a new NRC Form 396 in accordance with 10 CFR 55.57(a)(6) and check Box 5 if the examining physician has determined that the operator must take a prescription medication to maintain his or her medical qualifications.

Gen.24 What are the qualifications/parameters for using the "Must take medication as prescribed to maintain medical qualifications?" (Diabetics, previous heart attack, organ transplant patients, asthmatics requiring DAILY use medication?)

The instructions on the back of NRC Form 396 indicate that Box 4 should be checked if, in the opinion of the examining physician, the applicant's medical qualification per the applicable ANSI standard is contingent on taking a prescription medication. It does not matter if the medication is administered on a daily, weekly, monthly, or as-needed basis; the license condition would simply require the operator to take the medication as prescribed.

Gen.25 If a licensed operator is already taking medication for hypertension and the physician prescribes either an increased dosage or a change in medication, would this have to be reported to the NRC?

As discussed in the response to Question Gen.23, 10 CFR 55.25 only requires facility licensees to submit a revised NRC Form 396 during the term of an operators license if there is a permanent change in the operators medical condition that would cause him or her to fail to meet the requirements of 10 CFR 55.21. The examining physician would have to make that determination based on the guidance in whichever version of ANSI/ANS-3.4 (1983 or 1996) the facility uses and then recommend a conditional license if he/she deems it necessary to accommodate any disability that the operator has developed. The next time the operators license is due for renewal, the facility licensee would need to submit a new NRC Form 396 in accordance with 10 CFR 55.57(a)(6) and check Box 4 if the examining physician has determined that the operator must take a prescription medication to maintain his or her medical qualifications. Simply increasing or changing a hypertensive operator's medication would not normally need to be reported unless the examining physician believes the operator's blood pressure is out of control to a point that it requires more frequent monitoring or a no-solo license (i.e., the addition of license condition # 4 or 6 on NRC Form 396). Refer to Questions Gen.21 and Gen.54 for additional guidance.

138

139 Gen.26 We recently received an amended license for one of our SROs as a result of reporting, for information only, a new diagnosis and medication for borderline type 2 diabetes. What is the purpose of amending an operator's license for that condition? What happens if his physician ceases that treatment? In the time required to issue another amended license, he is technically in violation of his license for not taking the medication.

If a previously healthy and unrestricted operator develops type 2 diabetes, which could conservatively be classified as "a permanent physical or mental condition that causes the licensee to fail to meet the requirements of 10 CFR 55.21," the facility licensee would be required (by 10 CFR 55.25) to notify the Commission within 30 days of learning of the diagnosis.

Since the physician has determined that the operator requires medication to control his or her diabetes (and remain medically qualified per ANSI/ANS-3.4), the facility licensee should check Box 5 on NRC Form 396 and provide appropriate medical evidence with the form (as required by 10 CFR 55.23(b)) for evaluation by the NRC; an information only report would not appear to be appropriate in this case. If the physician later determines that the operator no longer needs to take medication, the operator would not be in violation of his or her license given that the condition specifically states to "take medication as prescribed;" if it's no longer prescribed by the physician, the condition becomes irrelevant.

Gen.27 What are the reporting requirements if a licensed operator is newly diagnosed with and medicated for hypercholesterolemia (i.e., high cholesterol)?

High cholesterol is not, in and of itself, a condition that is addressed in ANSI/ANS-3.4; therefore, it would generally not disqualify an individual from having an unrestricted license, would not have to be reported to the NRC, and would not require a license condition. However, Section 5.4.9 of ANSI/ANS-3.4 (1996) indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would disqualify an operator.

Therefore, when the physician makes a new diagnosis and prescribes medication for hypercholesterolemia or any other disorder (whether or not it is addressed in the ANSI standard), the physician needs to consider the possible side effects to ensure (as required by 10 CFR 55.33(a)(1)) that they will not cause operational errors or affect the operator's capacity to safely perform licensed duties.

139

140 Gen.28 One of our SROs is undergoing chemotherapy treatments for multiple myeloma and is still off of licensed duties. Next month he will go for another round of inpatient treatment, but, after recovering from that, he is anticipating being able to return to his control room crew. Other than a list of medications that he will likely be taking coming out of that next treatment, what sort of information will your reviewer need? We're anticipating a condition on his license just for the medication, but other than a brief statement from his physician releasing him to return to in-plant activity, what else might be needed? We're not sure what to request from his physician on that point.

Despite the availability of effective treatments and the fact that the median survival of patients with multiple myeloma has improved from seven months (in the pre-chemotherapy era), to five years, it remains incurable, so it would clearly be a permanent disability that needs to be reported per 10 CFR 55.25. A physician familiar with ANSI 3.4 and the operators' job requirements needs to evaluate the individual's situation; the facility should not rely on the individual's personal physician who is thinking only of his patient and not the safety of the plant.

Some of the criteria in ANSI 3.4 that the NRC reviewer would need to see addressed include the operator's stamina while he is undergoing treatment, whether the condition or treatment could reasonably result in sudden or unexpected incapacitation, what medication(s) the operator will be taking, and whether they might result in incapacity. The facility needs to seriously consider whether other license conditions beyond the taking of medication (e.g., Item 4, No Solo, and/or Item 6, Medical Status Report, on NRC Form 396) might also be appropriate. A statement about general fitness for duty and overall prognosis would certainly be helpful.

Gen.29 How far back does a medical history have to go? For example, if an applicant was a blue baby would that have to be documented as part of his or her medical history?

Section 5.3 [5.4], Disqualifying Conditions, of ANSI/ANS-3.4-1983 [1996] states that a history or other indication of any disqualifying condition shall be considered disqualifying unless adequate supplemental findings demonstrate that no disqualifying condition [now] exists.

Although the standard provides specific time markers for certain conditions (e.g., Section 5.3.7(2) [5.4.7(3)] speaks of a history of disturbance of consciousness within the past five [two]

years), most conditions identified in the standard would require the examining physician to exercise judgment in determining how far back to take the medical history.

Blue baby is a non-specific term and usually refers to a number of congenital heart, lung, or hemoglobin abnormalities that can vary considerably in seriousness and may require treatment or may even go away spontaneously. If an applicant had blue baby syndrome as an infant, the physician would still have to make a determination that the applicant does not currently suffer from any disqualifying condition. Therefore, the condition should be documented in the medical history but would not need to be reported to the NRC unless a restricted license is requested due to a permanent medical condition.

Gen.30 Can the NRC clarify in writing whether it is acceptable for an operator to satisfy the near visual acuity requirement in one eye and the distant acuity requirement in the other (as might be the case if someone had lasik surgery)?

140

141 Such conditions have been reviewed on an individual basis and found to be acceptable.

Applicants who have uncorrected near visual acuity of at least 20/40 in one eye and uncorrected distant visual acuity of at least 20/40 in the other eye do not require a conditional license.

Gen.31 Can the NRC provide examples of actual operators who have been permanently disqualified?

Decisions to permanently disqualify an operator are generally, if not always, made by the facility licensee. If a facility licensee determines that a new license applicant is medically disqualified, the NRC would never see the individuals application or medical history. If an operators license is suspended by the facility licensee, the NRC might ask to be notified of and in agreement with the operators acceptable medical status before he or she returns to licensed duty.

However, we often do not get any further medical input and have to assume that the operator has been permanently disqualified.

Gen.32 Does the NRC expect us to report a diagnosis of high cholesterol with medication?

Please refer to Question Gen.27 above for a response.

Gen.33 Does the NRC expect us to report an episode of vasovagal syncope only during blood draw? Should an operator with such an episode have a no-solo license?

Section 5.3.7(2) [5.4.7(3)] of ANSI/ANS-3.4-1983 [1996] indicates that a history of disturbance of consciousness within the past five [two] years without a satisfactory medical explanation of the cause (emphasis added) shall be considered a disqualifying condition for operation without restriction. Moreover, Section 5.3 [5.4], Disqualifying Conditions, states that a history or other indication of any disqualifying condition shall be considered disqualifying unless adequate supplemental findings demonstrate that no disqualifying condition [now] exists (emphasis added). Accordingly, the facility should remove the operator from licensed duties until their physician can evaluate the episode to determine its underlying cause. If a satisfactory medical explanation cannot be identified, then the operator should be considered permanently disqualified and the facility licensee should report the incident and request a license restriction (most likely no-solo).

However, the simple faint or vasovagal syncope with blood draw is quite common and is considered a satisfactory explanation of the reason for loss of consciousness and does not require a no-solo license. It can usually be prevented by having the donor lie flat during the draw and for some short time afterward.

Gen.34 Does the NRC expect us to report when an operator is taking Viagra or Cialis (due to the side effects)?

141

142 The fact that an operator is taking either medication is not, in and of itself, reportable to the NRC. However, Section 5.3.9 [5.4.9] of ANSI/ANS-3.4-1983 [1996] indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would be considered disqualifying. Most medications have multiple side effects that may or may not present themselves depending upon the dosage and the takers individual body chemistry. Therefore, the facilitys physician would need to evaluate the effects of any medications (prescription and over-the-counter) that an operator is taking to determine if he or she is at risk of incapacitation. In the case of Viagra, for example, the internet lists changes or loss of vision and dizziness as possible side-effects, either one of which could be disqualifying if experienced by a licensed operator. Given that the underlying condition for which Viagra and Cialis are typically prescribed is not disqualifying, an operator who is experiencing undesirable side-effects from the medication may need to discontinue the treatment or seek a position that does not require a license.

Gen.35 We fill out NRC Form 396 after every biennial examination so compliance has the form on file, which causes about half of our submittals to be on the old version of the form. Is this a problem, or should we be filling out the forms as needed for a submittal?

It is only necessary to fill out and submit NRC Form 396 when applying for a license or reporting a change in medical status. Section B, Certification, should be signed and dated at the time of submittal; it should not be back-dated to coincide with the date of the last medical examination, which is entered in Section A of the form. Our preference would be for you to use the latest version of the form, which is available on the NRCs web page at http://www.nrc.gov/ reading-rm/doc-collections/forms/#NRC.

Gen.36 If an employees doctor restricts them with no overtime should we report this to the NRC?

Personal physicians are primarily concerned with their patients well-being; they are unlikely to be familiar with the requirements in ANSI/ANS-3.4 or to have an overriding interest in reactor safety. However, if they restrict one of your licensed operators to no overtime, they presumably have a medical basis for imposing such a restriction. Consequently, it would be prudent for the facilitys physician to evaluate the operators status to determine if any disqualifying conditions exist and whether a reportable change warranting a license restriction has occurred.

142

143 Gen.37 If an operator has one blood pressure reading over 160/100 followed by two readings that are lower, should we report the high reading to the NRC? Should we restrict the operator immediately or just refer him or her for treatment?

If this is a new condition and cannot be attributed to a measurement error or anomaly, then the conservative response would be to immediately restrict the individual from duties requiring a license until an evaluation can be performed to determine if the operator has developed a permanent physical condition that causes him or her to fail to meet the requirements of 10 CFR 55.33(a)(1). Under todays treatment guidelines, most physicians will begin medicating their patients for hypertension before they reach the160/100 mmHg threshold stated in ANSI/ANS-3.4, so, if that is the treatment regimen that is followed in this case, it would have to be reported to the NRC within 30 days after learning of the diagnosis using NRC Form 396. The form should include a recommendation to apply a take your medicine condition to the individuals license plus any other condition(s) that the examining physician might determine to be necessary based on his or her evaluation of the operators condition vis--vis the criteria in the ANSI standard. Refer also to Question Gen.21.

Gen.38 What fasting blood sugar level is deemed uncontrolled and in need of further evaluation? Is there a cutoff? Is there an A1C level cutoff?

Uncontrolled diabetes is a non-specific term, so it is up to the examining physician to use his or her judgment. For coding purposes, it has been described as elevated blood sugar with symptoms, and/or a blood sugar level of more than 300, or an A1C twice normal, or blood sugars that vacillate up and down considerably. Although these criteria are helpful, the NRC does NOT endorse or require that they be used.

Gen.39 The format and organization of NRC Form 396 is confusing when reporting new or changed medical conditions and recommending appropriate license amendments when the existing license already has other, unrelated restrictions.

NRC Form 396 is generally revised every three years when the associated OMB clearance is renewed and the public is given the opportunity to comment on the form and the information collection burden that it imposes on the industry. The Operator Licensing Program Office will take this recommendation into consideration the next time the form is revised.

143

144 Gen.40 If an operator is medically disqualified for licensed duties and awaiting a final determination from the doctor, is it acceptable to use the individual as a procedure reviewer if the facility requires that position to have an active license?

Although the NRCs regulations do not require procedure reviewers to have an active operators license, the NRC staff recognizes the benefits of using licensed operators in that capacity.

Nevertheless, the NRC does expect facility licensees to implement their procedures as written, so using medically disqualified operators to perform such duties could be a regulatory concern depending on how the facilitys administrative procedure is written. In the absence of a regulatory requirement, the facility would be free to change its administrative procedure to allow medically disqualified operators to perform that function.

Gen.41 If an operator takes a sleep aide (e.g., Ambient or Lunesta) how long does he/she need to wait before returning to licensed duties? There are reports that the effects can linger for up to 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />; do they need to wait that long?

Section 5.3.9 [5.4.9] of ANSI/ANS-3.4-1983 [1996] indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would be considered disqualifying. Most medications have multiple side effects that may vary considerably depending upon the dosage and the takers individual body chemistry. Therefore, the facilitys physician would need to determine, based on the operators history and physical exam, whether the effects of any medication (prescription and over-the-counter) that the operator is taking might disqualify him or her from performing licensed duty and for how long.

144

145 Gen.42 What are the reporting requirements if a licensed operator takes a prescribed medication (e.g., Provigil) as needed to improve wakefulness while on shift work?

Section 5.3.9 [5.4.9] of ANSI/ANS-3.4-1983 [1996] indicates that any medication taken in such a dosage that the taking or delay of taking might be expected to result in incapacity would be considered disqualifying. Most medications have multiple side effects that may vary considerably depending upon the dosage and the takers individual body chemistry. Given that dizziness, which is a possible disqualifying condition, is sometimes observed with the use of Provigil, the facilitys physician would need to evaluate the effects that the operator is experiencing to determine whether he or she might be disqualified when taking (or neglecting to take) the medication.

Assuming that the physician has concluded that no other mental or psychological (or physical) condition exists that might be impairing the operators alertness (which could be disqualifying per Section 5.3.8 [5.4.8] of ANSI/ANS-3.4-1983 [1996] and a reportable condition), prescribing Provigil (or a similar drug) as needed to promote wakefulness while on shift work would not be reportable to the NRC. Although mental alertness is identified as a general health requirement in Section 5.2 of the standard, the NRC staff understands that rotating shift work can affect sleep patterns, thereby leading to fatigue and diminished mental alertness. These effects would generally be considered transient in nature and not permanent physical or mental conditions that would need to be reported pursuant to 10 CFR 55.25.

Gen.43 Section B, Certification, of NRC Form 396 requires the name, title, and signature of the senior management representative on site. Who is that?

As stated in Section C.1.f of ES-202 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors, the term senior management representative on site is synonymous with authorized representative of the facility licensee, which includes examples such as the plant manager or site vice-president. In accordance with 10 CFR 55.31, How to Apply [for a license], that individual must certify when an applicant has completed all of the facility licensees requirements and commitments for the desired license level (e.g., experience, control manipulations, training, and medical fitness). That certification involves signing Block 19 of NRC Form 398 and Section B of NRC Form 396.

145

146 Gen.44 If we submitted a license renewal application in 2002 for an operator taking medication, but the NRC did not condition the license, do we need to provide supporting medical information when submitting the NRC Form 396 for a 2008 renewal if the operator is still taking the same medication but our physician currently does not believe a take your medicine condition is necessary?

If we reported a change in medical status with a letter in 2004 indicating that an operator was taking medication, but the NRC did not condition the license for this condition, do we have to provide additional supporting details with the Form 396 for a current renewal if the operators condition and treatment have not changed (e.g., same treating physician, same symptoms, same medicine, same dosage)?

If the NRC previously said that a medication was not a license condition, and the operator is still taking the same medication, do we have to report this medication every time the renewal occurs?

The license renewal process provides an opportunity, once every six years, for the NRC to review every licensed operators medical condition and general health (including any medications that the operator is taking) to ensure they will not adversely affect the performance of assigned operator duties or cause operational errors endangering public health and safety.

Because NRC Form 396 did not contain a must take medication restriction prior to 2006, a medical condition reported before the form was revised could, today, result in a determination that a must take medication restriction is warranted. Every time you submit NRC Form 396 for an operator, regardless whether you are reporting a change in medical status or renewing his or her license, you should check all the condition/restriction boxes that apply to that operator on the date that the certification is signed and provide supporting documentation, as necessary. For medical conditions that existed before the must take medication restriction was implemented, you should submit sufficient supporting documentation with NRC Form 396 to enable our physician to determine whether or not a restriction is warranted, even if a must take medication restriction was not imposed for the same condition in the past. If the supporting documentation was submitted with the previous application or status report and has not changed, an entry to that effect in the explanation field on the form would be sufficient.

Also refer to Questions Gen.21, Gen.23, Gen.24, and Gen.25 above for additional discussion of information-only reports and medication restrictions.

146

147 Gen.45 I received an RO and/or an SRO license from the NRC for a 10 CFR 50 facility licensee, but I am no longer licensed. Does the NRC keep records of those licenses issued and, if so, how can I obtain a physical copy? How do I obtain my just my license number?

This information is protected by the Privacy Act (PA) and, as such, you must file a formal Privacy Act Request with the Nuclear Regulatory Commission. The Privacy Act works as a companion with the Freedom of Information Act (FOIA), which establishes a presumption that any person has the right to request access to records in the possession of the Executive Branch of the Federal Government.

• Information pertaining to the Freedom of Information Act/Privacy Act can be found at:

https://www.nrc.gov/reading-rm/foia/foia-privacy.html.

• To view the Privacy Act Request Guide, please go to https://www.nrc.gov/reading-rm/foia/privacy-request.html#access.

• Follow the instructions provided in the Privacy Act Request Guide to properly submit a Privacy Act Request with the Nuclear Regulatory Commission.

• Freedom of Information/Privacy Team may be contacted at FOIA.resource@nrc.gov If an individual is requesting his/her license number only, the following provides guidance on obtaining that information:

• 10 CFR 55.5, Communications, provides specific information regarding the contact of the Regional Administrator in the Region where the nuclear power reactor is located.

• Alternatively, an individual may contact the Operator Licensing Branch in specific Region by filling out the information found at https://www.nrc.gov/reactors/operator-licensing/contact-us.html.

• Individuals requesting his/her license number will be required to submit proof of identity through a copy of driver license/state identification card, along with a notarized statement verifying their identity, or the following statement, in lieu of notarization:

o From 28 USC 1746 - I declare under penalty of perjury that the foregoing is true and correct. Executed on [date] and [signature]

Gen.46 What are the reporting requirements for an individual who has sleep apnea? What if they are using a Continuous Positive Airway Pressure (CPAP) machine for treatment?

The licensee needs to report the condition of sleep apnea since it is generally a permanent medical change that can affect the individual licensee's capacity to perform his/her required duties.

Depending on the severity of the condition and the requirement to use the CPAP treatment, the NRC medical authority may require a license restriction for taking the prescribed treatment, similar to the required taking of prescribed medication. In addition, the condition can cause additional medical problems including hypertension, which may be a disqualifying medical condition depending upon its severity. Refer to Section 5.2, Health Requirements, of ANSI/ANS-3.4-1996, Medical Certification and Monitoring of Personnel Requiring Operator 147

148 Licenses for Nuclear Power Plants; Section 5.2(1) specifically requires operators to have acuity of senses, and Section 5.2(3) requires metal alertness and emotional stability. Section 5.2, General Requirements, of ANSI/ANS-3.4-1983 contains the same requirements.

Gen.47 What is the NRCs policy regarding restricted operators (i.e., those who must take medications as prescribed to comply with a license condition) keeping medications on site in the event of an extended work stay that might be required during a hurricane, pandemic, or other emergency situation?

Although the NRC has no formal policy to address this specific situation, individual operators and facility licensees should consider the following regulatory requirements guidance in planning their response to emergency situations that may require operators to remain on-site for extended periods of time.

• Section 5.4 [5.3] of ANSI/ANS-3.4-1996 [1983], American National Standard for Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants, identifies a number of conditions that, unless adequately compensated for, shall disqualify an individual from licensed duty. Section 5.4[3].9 specifically states that "any medication taken in such a dosage that the taking or [temporary] delay of taking might be expected to result in incapacity [] would be disqualifying. Clearly, certain medications are more critical than others in ensuring that operators maintain their capacity to safely perform licensed duties; e.g., missing a dose of medication for diabetes or epilepsy would likely have a greater impact on an operators capacity than missing a dose of medication for hypertension.

• 10 CFR Part 26, Fitness for Duty [FFD] Programs, includes a number of requirements that appear pertinent under such situations. For example, FFD programs must: (1) provide reasonable assurance that individuals are not mentally or physically impaired from any cause; (2) describe the individual's responsibility to report FFD concerns; (3) describe the process that the licensee will use to ensure that individuals who are called in to perform an unscheduled working tour are fit for duty; (4) establish controls and conditions under which an individual can perform work, if called in, after reporting that he or she considers himself or herself to be unfit for duty for reasons including illness, fatigue, or other potentially impairing conditions; and (5) provide training to ensure that the individuals who are subject to the rule have knowledge of their responsibilities under the FFD program.

• Licensed operators are ultimately responsible for compliance with any conditions stated on their license. If possible, they should consider storing a one-week supply of necessary and critical medications at their work site, having such medications readily available at their homes to take to their work sites on short notice, and making contingency plans to have their medications brought to the site, if needed.

• If a licensed operator is required to remain at the facility for a period of time that exceeds the prescribed medication frequency with no medication available, he/she should inform the facility licensee and, if possible (*), discontinue licensed duties.

• Facility licensees are encouraged to accommodate and facilitate their operators compliance with medical requirements. In addition to training their operators on their FFD responsibilities, they should, if possible, provide any assistance they might need to safely and securely store their required medications on-site.

148

149

• If a licensed operator is required to remain at the facility for a period of time that exceeds the prescribed medication frequency with no medication available, the operator should be removed from licensed duties, if possible (*), and assigned to other work consistent with the operators diminished capacity. If possible, the facility should permit the operator to return home to retrieve the required medications or provide other assistance, as necessary, in procuring a supply.

(*) Note that performing operator duties in violation of a license condition or FFD requirement could result in enforcement action against the individual and/or the facility licensee.

However, the NRC may exercise discretion, in accordance with its Enforcement Policy, and mitigate or refrain from enforcement action based on the relevant circumstances of the particular case.

Gen.48 How does the NRC balance the medical reporting requirements for an operator's medical certification with the individual's rights for privacy of information under HIPAA laws?

Should there be a privacy agreement between the license holder and the NRC (also with all other groups within each utility that handle this confidential information)?

It is our understanding that the Health Insurance Portability and Accountability Act of 1996 (HIPAA) applies only to three types of entities, a covered health care provider, a health care clearinghouse, or a health plan. The NRC is none of those entities, so HIPAA requirements do not apply to the operator medical records that the NRC maintains. However, the Privacy Act does apply because they are records of individuals as defined by the Act. The limited medical information that the NRC obtains from individuals and facility licensees as part of the operator licensing and license renewal process is maintained in a Privacy Act system of records and afforded all of its protections. Please refer to http://www.nrc.gov/reading-rm/foia/privacy-systems.html for more information regarding the NRC's Privacy Act System of Records, including NRC-16, which covers all of the operator licensing records.

The NRC does not disseminate any of that medical information outside of the NRC and maintains it only for official use in making licensing and re-licensing decisions pursuant to 10 CFR 55.33 and §55.57. Those regulations permit the NRC to approve an initial/renewal application only if it finds that the applicant's medical condition and general health will not adversely affect the performance of assigned operator job duties or cause operational errors endangering public health and safety. In most cases, that finding is based on the facility licensee's certification with little or no transmission of personal medical information. However, in some instances, when an applicant's general medical condition does not meet the minimum standards, the NRC may still approve the application per 55.33(b) based on the facility's recommendation and supporting medical evidence provided by the licensee and the examining physician.

When it comes to the disclosure, protection, and exchange of private medical information, 45 CFR 164.512 identifies a number of situations when "covered entities" may disclose protected health information without the individual's written consent, and subsection (d) specifically allows disclosure for health oversight activities such as government regulatory programs for which health information is necessary to determine compliance with program standards. Although a separate privacy agreement between the license holder and the NRC should not be necessary, 149

150 the facility should take measures to ensure that personnel within their organization handle the information appropriately.

Gen.49 When a facility licensee submits a NRC Form 396 involving a license condition or restriction, the medical information is reviewed by the NRCs physician. What is the purpose of this review?

When making operator license decisions, the NRC considers all information certified by the facility senior management representative on site using NRC Form 398, Personal Qualification Statement - Licensee. Part of the personal qualifications being certified relates to medical qualification, and is certified via Form 396. Because evaluation of an individuals medical qualification often requires medical expertise not found within the staff, the NRC retains the services of contract physicians to perform a review to support making the licensing decision.

As stated on NRC Form 396, the overriding purpose of licensed operator medical qualification is that the individual would not be expected to cause operational errors endangering public health and safety. The guidance contained in industry consensus standards, specifically versions of ANS/ANSI-3.4 (power reactors) and 15.4 (non-power reactors), forms the basis in reaching this determination. In some cases, conditions or restrictions must be placed on an individuals license to compensate for a medical shortcoming relative to these standards to ensure safety.

Such conditions or restrictions are recommended by the facility licensees examining physician on Form 396 and must be supported by medical evidence. The purpose of the NRC physician review is to evaluate the Form 396 and supporting medical evidence to determine if the physical condition and general health of the applicant/operator are such that he or she would not be expected to cause operational errors which might endanger public health and safety. The NRC physician review is a confirmation that the facility physicians request regarding license conditions or restrictions is appropriate and that the applicant/operator will satisfy ANSI/ANS-3.4[15.4] requirements, or that a requested waiver (exception) is appropriate. The NRC physicians review is not for the purpose of re-diagnosing the individual.

150

151 Gen.50 In the NRC Form 396 block where the facility requests license conditions or restrictions on the basis of physician recommendations, the form states Provide explanation and attach supporting medical evidence for NRC review. What constitutes supporting medical evidence for the purposes of the NRC review?

Supporting medical evidence consists of the findings, laboratory data, examination results, diagnoses and treatment plans (such as prescribed medications, use of therapeutic devices and planned monitoring) that support a determination of whether or not an individual meets the physical condition and general health requirements to be licensed as an operator. The evidence must address the general health and disqualifying conditions contained in ANS/ANS-3.4[15.4]. Insights into the general prognosis as it relates to the need for more frequent monitoring (such as 3/6/12 month status reporting and no solo restrictions) are beneficial for the reviewing physician. The following specific examples are provided as illustrations:

• If the must take medication condition is recommended for hypertension, the name of the prescribed medications and dosages must be stated. Additionally, blood pressure readings from the most recent examination need to be reported so that the reviewing physician can confirm compliance with the ANSI/ANS-3.4[15.4] limits. Additionally, if information on the effectiveness of medications (how well are they controlling blood pressure) and any side effects (presence or absence) is available, it needs to be included to help the reviewing physician determine the individuals medical qualification status.

• Commonly reported conditions involving medications include diabetes and thyroid disease. The supporting medical evidence provided for these conditions should follow the same general form as the hypertension example. In the case of diabetes, the reviewing physicians would rely on fasting blood sugar and/or hemoglobin A1C laboratory data to determine if the disease was being controlled as required by the ANSI/ANS standard. Similarly, thyroid function study data is useful in confirming that the disease is controlled [ref. ANSI/ANS-3.4-1996, Section 5.4.3.(2)]. As is always the case when medications are involved (refer to Questions Gen.27, Gen.34, Gen.41, and Gen.42), an evaluation of side effects and the potential for incapacitation must be made.

The results of any such assessment should be shared with the NRC reviewing physicians to facilitate their determination of medical qualification.

• Certain cardiovascular conditions can be disqualifying. When reporting instances of coronary heart disease, available data (e.g., EKG or other test procedure or examination results) that indicate satisfactory cardiac function to consider an individual as medically qualified must be submitted. Information on medications, therapeutic devices, any co-morbidities (obesity, diabetes, hypertension, etc.) and the need for follow-up monitoring are useful to the reviewing physician in making a general assessment of the applicants/operators health and its potential effect on safe plant operation. A statement regarding the individuals physical capability to satisfactorily perform all assigned duties, including a brief description of any accommodations in place to assure capability to perform these duties, would facilitate the NRC physicians review to determine medical fitness for licensing.

151

152 In summary, for the NRC reviewing physicians to perform a meaningful review, some basic medical evidence/information relative to the following questions must be included:

• What is the medical problem/issue? (link to ANSI/ANS-3.4[15.4] disqualifying condition)

• What are the related medical examination results? (readings, laboratory data, physician observations)

• What is the diagnosis? (i.e., is the condition stable? being controlled? likely to result in incapacitation or eventual disqualification?)

• What are the treatment plan (medications, therapeutic devices, accommodations, monitoring) and proposed license restriction(s) to ensure ANSI/ANS-3.4[15.4] and not endanger public health and safety requirements are met?

Gen.51 There have been many cases where NRC Form 396s have been returned from the NRC physicians requesting additional information before a determination can be made. What suggestions can the NRC offer to help ensure the reviews can be completed more efficiently?

Some specific suggestions to facilitate a smooth review process based on recent experiences are:

• Make a clear link to ANSI/ANS-3.4[15.4] conditions/requirements in the submittal. Use the provided block on Form 396 to clearly link the proposed restriction to the ANSI/ANS-3.4[15.4] disqualifying condition.

• When proposing an other restriction or exception (Box 8 on Form 396), use the Proposed Wording of Restriction block to clearly state what the license condition should say in order to assure there is no misunderstanding.

• If medical information is being submitted as information only, indicate by checking Box 10 on Form 396. Use the explanation block or clearly indicate in attached correspondence whether or not there is a relationship to an ANSI/ANS-3.4[15.4]

disqualifying condition. If an ANSI/ANS-3.4[15.4] condition is involved, clearly state the impact (or absence thereof) of the information on the individuals qualification relative to that condition. If not related to a specific ANSI/ANS-3.4[15.4] disqualifying condition, a statement regarding possible side effects, if medication is involved, and the potential for incapacitation needs to be included when possible. (Refer to Questions Gen.27, Gen.34, Gen.41, and Gen.42)

• When making a submittal that involves a change in medication (although not necessarily required - see question Gen.25), it needs to contain a brief statement of the reason for the medication change, a confirmation that ANSI/ANS-3.4[15.4] requirements continue to be met, and that the existing license conditions remain adequate (e.g., the medical situation is stable such that more frequent monitoring or no solo changes are not warranted). This information will allow the NRC medical reviewer to have a more complete picture of the basis for the reported change and allow for an evaluation of the impact on overall medical qualification in accordance with the standard.

152

153

• If a no solo restriction is proposed, a simple statement identifying a specific ANSI/ANS-3.4[15.4] condition and why the no solo restriction will compensate is helpful.

• The NRC physicians do not maintain medical files on applicants/operators. Therefore, sufficient medical history/background must be contained in the Form 396 and supporting medical evidence such that any proposed license restrictions and overall conclusions relative to medical qualification for licensing are clearly supported. Each Form 396 submittal should stand on its own with enough information to give a clear picture of the individuals health and medical suitability for licensing. A brief history of medical status and changes since the last submittal (in the case of renewals) will enable the reviewing physician to make a more meaningful review of suitability for licensing within the context of the individuals overall health.

Gen.52 Ive noticed that the 1996 version of ANSI/ANS-3.4, Section 3.3, states that a physician may temporarily restrict an individuals duties as a licensed operator for a period as long as 30 days, without formal NRC notification. Does this imply a requirement to report temporary conditions lasting longer than 30 days to the NRC if a facility has committed to use the 1996 standard?

Please refer to Question Gen.20 above for a discussion of permanent disability.

The only related regulatory requirement, as stated in 10 CFR 55.25, is to report permanent physical or mental conditions within 30 days of learning of the diagnosis. As discussed in Question Gen.20, there is no strict time limit on when a condition must be considered permanent, and it is largely left to the licensee physicians judgment as to whether the operator will be capable of meeting the applicable medical standards in the foreseeable future. The NRC expects the facility licensee to administratively restrict the operators activities, as appropriate, during the term of the temporary condition. However, the NRC does not require formal notification for temporary conditions that exceed 30 days. The NRC only requires notification, within 30 days, when a developed physical or mental condition is determined to be permanent and causes a licensee to fail to meet the applicable ANSI/ANS-3.4 requirements/standards or could adversely affect performance of assigned operator job duties or cause operational errors endangering public health and safety. The provision for reporting temporary duty restrictions as stated in Section 3.3 of ANSI/ANS-3.4, 1996, is more restrictive and is not required to comply with the regulations.

153

154 Gen.53 Must detailed medical evidence be submitted with NRC Form 396 (associated with either an initial or license renewal application) for minor medical conditions related to corrective lenses or hearing aids?

The short answer is no. During the 10 CFR Part 55 rule change in 1987, the NRC received public comments that detailed medical evidence should not be required to be submitted for common conditions, such as corrective lenses and hearing aids. The staff agreed and modified Form 396 to require medical evidence only for restrictions other than corrective lenses or hearing aids. As NRC Form 396 was again modified in later years to accommodate new restrictions (such as no solo, shall take medication, and shall submit medical status report),

the requirement to attach supporting medical evidence for NRC review was moved to the section header for simplicity, rather than include it with each applicable license restriction in the list. This change was not meant to undo the 1987 change which excluded corrective lenses and hearing aids from the medical evidence requirement. The NRC will consider clarifying Form 396 during the next revision to remove any confusion.

Gen.54 If a medical condition was previously reported for hypertension to the NRC, but the operator had a dose increase but still meets the B/P standard, do you have to report the medication dose increase to the NRC?

No, as stated in Question Gen.25, simply increasing or changing a hypertensive operator's medication would not normally need to be reported unless the examining physician believes the operator's blood pressure is out of control to a point that it requires more frequent monitoring or a no-solo license (i.e., the addition of license condition #6 on NRC Form 396). This answer presumes that the operators license is already conditioned to take medication as prescribed to maintain medical qualifications (NRC Form 396, Box #4), as discussed in Question Gen.21.

Gen.55 When completing a license renewal, does the NRC want supporting medical documentation? For example, shall take medication restriction for blood pressure, do you want the most recent B/P readings or does the biennial exam suffice?

As stated in NUREG-1021, ES-605, D, License Renewal, the facility licensee must certify on NRC Form 396 that a physician has performed a medical examination within the previous 2 years, as required by 10 CFR 55.21, Medical Examination, and submit that form along with NRC Form 398. Therefore, the short answer to the first question is yes, the facility licensee is required to submit the most recent biennial medical examination with supporting medical evidence necessary to support any recommended medical license conditions. However, assuming the facility licensee has performed and forwarded the most recent biennial medical examination and it is anticipated that the license action will be completed before the time since the last medical examination exceeds 24 months, the facility is not required to submit additional blood pressure (B/P) readings unless recommended by the examining physician, i.e., the more recent B/P readings may be submitted at the discretion of the examining physician but are not required.

154

155 Gen.56 As a third party organization, and not a government agency, are we eligible to submit verification requests for a 10 CFR 50 facility licensee?

If a third-party organization requests an operators license information, then one of the two following provisions must be met:

• The NRC must verify that the individual operator consents to the disclosure in accordance with the Privacy Act. The consent must be notarized or otherwise provide a verifiable means of identification for the individual operator.

o Privacy Act Request Guide can be found at the following location:

https://www.nrc.gov/reading-rm/foia/privacy-request.html o If the third-party is not acting on behalf of a 10 CFR facility licensee, this option is to be exercised.

• The NRC must receive proof that the third-party organization is the authorized agent of a 10 CFR facility licensee, and that the operator license information being requested is within the scope of the services that the third-party organization has been authorized by the facility licensee to perform.

o The boundaries of the NRC Systems of Records Notices (SORN)

(https://www.gpo.gov/fdsys/pkg/FR-2016-11-17/pdf/2016-27652.pdf) dictate that the release of operator license information must be linked to a 10 CFR facility licensees request.

Depending on the type of consent provided (either individual consent or a request from a 10 CFR facility licensee), the following information/verification will be provided:

• The NRC will provide all available information as requested and as explicitly consented to by the individual, in compliance with the Privacy Act.

• The NRC will provide a verification (yes or no) to questions from a third-party organization for which the NRC has available information, as explicitly requested by a 10 CFR facility licensee, in accordance with the NRC SORN (https://www.gpo.gov/fdsys/pkg/FR-2016-11-17/pdf/2016-27652.pdf)

Gen. 57 Due to technological advances and the recent COVID-19 pandemic, can facilities use electronic signatures to sign the Form 398 and Form 396?

NUREG-1021, ES-202, C.1.f, states, in part, the following:

Forms that have only a single signature, such as NRC Form 396, may be submitted electronically using an electronic digital signature. However, forms with multiple signatures, such as NRC Form 398, must rely on handwritten optically scanned signatures because of the limited digital signature capability of the [Electronic Information Exchange (EIE)] system.

Note that Form 396 has since been revised and now includes multiple signatures.

155

156 The NRC has determined that the following options meet the intent of the NUREG-1021 statement regarding electronic signature of Forms 398 and 396 by facility licensee staff:

1. Any handwritten, optically scanned signature will continue to be accepted, regardless of transmission process used (hardcopy, Part 55 EIE, or e-mail)

2. Facility licensees may use any generally recognized form of electronic signature for forms being submitted through Part 55 EIE. The digital trail created by the EIE credential registration/issuance process, EIE system login, and other system database timestamps, offers sufficient documentation of authenticity.

3. Any utility desiring to use a digital certificate to sign the documents needs to obtain specific approval from the NRC by sending in a request, detailing how the facility meets the attributes in Table 1 below. The NRC may approve use of this certificate for signing and submitting Forms 398 and 396 through either e-mail or Part 55 EIE.

4. Facility licensees may NOT use electronic signature for forms submitted through email. The only possible exception is if the facility uses a digital certificate that the NRC has reviewed and determined to be acceptable per item 3 above. The document must contain the digital certificate information (it cannot digitally sign the document and then convert the file in such a way that the digital certificate information is lost). An EIE certificate is not sufficient to send documents using e-mail.

The NRC policy related to use of electronic signatures in place of a handwritten signature is to use the electronic signatures whenever practical and logical. Electronic signature is a broad term covering a multitude of technologies and methods, from digitizing a handwritten signature to be placed in document signature blocks, to competitive corporate solutions, all the way to signing with a Personal Identity Verification (PIV) card. Digital signature is a specific type of electronic signature, created through the use of Public Key Infrastructure (PKI) certificates, such as those encoded on government-issued PIV cards. Digital signature, especially when created by certificates residing on government-issued PIV or CAC cards is the most secure of electronic signature and is vastly superior to a handwritten signature in almost every way. It offers an incredibly high degree of non-repudiation/non-refutability.

That said, PKI certificates, especially those generated as the result of a vigorous, frequently-audited, and policy-constrained identity proofing, issuance, and activation process are not popular outside of government. This is primarily due to cost and complexity, as well as the availability and general suitability of other technologies for common use cases (e.g. use of One-time Passwords for logging in, instead of a PIV card). The NRC providing suitable certificates for use cases such as this would be technically complex and frustrating for users. This means that the best type of electronic signature isnt plausible for this case, so the staff evaluated other, less ideal means to determine their suitability for this application. There are differing factors in play - the need to support efficiency and effectiveness (supports use of electronic signature) and the need for signatures that are legally-binding when necessary (does not support use of electronic signature, especially the forms that offer lower levels of non-repudiation).

Even low assurance signatures can have process augmentations or other controls that can increase their suitability. Much like the Defense in Depth concept, this layering effect can be applicable here as well. Anything that can help offer a digital bread crumb trail or otherwise increase non-repudiation can be considered. A specific and highly relevant example is the use of the Part 55 EIE system to submit the relevant forms. EIE users are sent through an NRC 156

157 credentialing process to be issued the credential used for system log in. The system also logs user activity and has other means (such as database timestamps) that help document that a given action was initiated by a user. Even in cases where the EIE submitter is submitting documents on behalf of their company (e.g. they are not directly the signer), this still helps to create that digital chain in a significant way.

The required forms are completed by the facility licensee staff and then reviewed and signed by a senior member of the staff. They are submitted via paper (in which case consideration of electronic signature is not applicable) or electronically via email or Part 55 EIE. They are then reviewed and electronically signed by NRC staff. Some number of these forms are audited for accuracy. These forms are used to form the basis for making licensing decisions, meaning that there is some importance of non-repudiation and some risk significance to the use of the forms and the electronic signatures.

Table 1. Specific Case - Required Attributes for Facility Use of Root Certificate Authority and Subservient Issuing Certificate Authority Attribute Contain unique identifying attributes about the person it was issued to (name, email address, etc.)

Have a reasonable validity period (such as one year)

Be issued by a government entity, which is subject to a certain level of cybersecurity and process scrutiny by default Have the proper key usage defined 157