ML23059A216

DPO Case File: DPO-2021-002 --Redacted-Public

ML23059A216
Person / Time
Issue date:	09/27/2021
From:	Daniel Bacon, Jennivine Rankin Division of Reactor Safety II, Office of Nuclear Reactor Regulation
To:	Office of Nuclear Reactor Regulation
References
DPO-2021-002
Download: ML23059A216 (105)

Text

DPO Case File for DPO-2021-002 The following pdf represents a collection of documents associated with the submittal and disposition of a differing professional opinion (DPO) from an NRC employee involving the Operator Licensing Examination Standards for Power Reactors, NUREG-1021, Revision 12 (GFE issue).

Management Directive (MD) 10.159, The NRC Differing Professional Opinions Program, describes the DPO Program. https://www.nrc.gov/docs/ML1513/ML15132A664.pdf The DPO Program is a formal process that allows employees and NRC contractors to have their differing views on established, mission-related issues considered by the highest level managers in their organizations, i.e., Office Directors and Regional Administrators. The process also provides managers with an independent, three-person review of the issue (one person chosen by the employee). After a decision is issued to an employee, they may appeal the decision to the Executive Director for Operations (or the Commission, for those offices that report to the Commission).

Because the disposition of a DPO represents a multi-step process, readers should view the records as a collection. In other words, reading a document in isolation will not provide the correct context for how this issue was reviewed and considered by the NRC.

It is important to note that the DPO submittal includes the personal opinions, views, and concerns by NRC employees. The NRCs evaluation of the concerns and the NRCs final position are included in the DPO Decision.

The records in this collection have been reviewed and approved for public dissemination.

Document 1: DPO Submittal Document 2: Memo Establishing DPO Panel Document 3: DPO Panel Report Document 4: DPO Decision Document 5: DPO Appeal Submittal Document 6: Statement of Views Document 7: DPO Appeal Decision

Document 1: DPO Submittal

Official Use Only - Sensitive Internal Information U.S. NUCLEAR REGULATORY COMMISSION DPO Case Number DPO-2021-002 DIFFERING PROFESSIONAL OPINION Name and Title of Submitter Organization Daniel Bacon, Senior Operations Engineer

• See attached list for co-submitters and their signatures.

R-II/DRS/O81 Name and Title of Supervisor Organization Gerald McCoy, Branch Chief R-II/DRS/OB 1 Date Received 09/27/2021 Telephone Number (10 numeric digits)

(404) 997-4518 Telephone Number (10 numeric digits)

(404) 997-4551 When was the prevailing staff view, existing decision or stated position established and where can it be found?

Date 09/14/2021 Where (i.e., ADAMS ML#, if applicable):

ML21256A276 Subject of DPO Operator Licensing Generic Fundamentals Examination Summary of prevailing staff view, existing decision, or stated position.

The Operator Licensing Examination Standards for Power Reactors, NUREG-1021 Revision 12, (ML21256A276, September 2021) has eliminated the stand alone 50 question Generic Fundamentals Examination (GFE) that required a minimum passing score of 80% and reduced the required testing of generic fundamentals to as few as 6 to 8 questions with no required passing score for that section of the exam.

Reason for DPO, potential impact on mission, and proposed alternatives.

The reason for this DPO is that NUREG-1021 Revision 12 diverges from the Commission approved corrective actions taken with regard to operator licensing and training as a result of the accident at Three Mile lsland-2 (TMl-2). This change deemphasizes the importance of generic fundamentals and drastically reduces the ability of the operator licensing written examination to discriminate between an applicant that has an adequate knowledge of generic fundamentals and one that does not. This change also dilutes the sample of other topics that are currently tested on the operator licensing written examinations. When combined with industry initiatives that have reduced the length of operator license classes and reduced the educational requirements for eligibility as stated in ACAD 10-001 Revision 2 (ML21210A294), this change could result in less time spent in training preparation, less mental review and practice, more forgetting of factual details, less reinforcement and application of job concepts, and a gradual decline in the performance of control room licensed operators. The potential consequences of this simplistic GFE testing process specified in NUREG-1021 Revision 12 are clearly stated in Appendix B (section 8.1) of NUREG-1021 Revision 11 (ML17038A432).

Following the accident at TMl-2, SECY 79-330E (ML12236A763) provided recommendations for upgrading the qualifications of licensed operators and senior operators through a program that included increased training and testing in the areas of thermal hydraulics and reactor transient response, increased use of simulator training and testing, higher passing grades on licensing examinations, and increased emphasis on retraining and re-examination.

Clarification of TMl-2 Action Plan Requirements, NUREG-0737 (ML051400209), added a new category to the operator written examination entitled, "Principles of Heat Transfer and Fluid Mechanics," and a new category to the senior operator written examination entitled, "Theory of Fluids and Thermodynamics." The passing grade for the examinations was an 80% overall with at least 70% required in each category. These requirements were approved by the Commission.

Prior to splitting out the GFE from the licensing examination as described in SECY-89-292 (ML12251A650), the Examiners' Handbook for Developing Operator Licensing Examinations, NUREG/BR-0122 Revision 4 (ML20117N069), stated the reactor operator written examinations should normally sample: (1) 25% from the fundamentals area (reactor theory, thermodynamics, and component operation), (2) 48% from plant systems and plant-wide safety and administrative procedures, and (3) 27% from emergency and abnormal plant evolutions. For senior reactor operators, the written examination content should normally sample:

(1) 24% from fundamentals, (2) 33% from emergency and abnormal evolutions, and (3) 43% from plant systems and plant-wide safety and administrative procedures.

In accordance with NUREG-1021 Revision 5 (ML20235R399), applicants were required to obtain a score of at least 70% in each of the three sections with an 80% overall to pass the written examination.

The NRC staff developed NUREG/BR-0122 Revision 4 to improve the content validity of the operator licensing examinations.

Content validity was ensured through the performance of job/task analysis focusing on the delineation of essential knowledge NRC FORM 680 (09-2019)

Official Use Only - Sensitive Internal Information Page 1 of 4 NRC FORM 680

1. "ftiAfOt,(

(09-2019)

~~

o.,,"'

le n

NRC MD 10.159 0

~

~

~-$,.,

t-<S-"

NRC FORM 680 (09-2019)

NRC MD 10.159 Official Use Only - Sensitive Internal Information U.S. NUCLEAR REGULATORY COMMISSION DPO Case Number DPO-2021-002 DIFFERING PROFESSIONAL OPINION (Continued)

Date Received 09/27/2021 ana ao11mes. 1 ne eaucauona, memoao,ogy usea to aeve,op 111e comem or 11censIng exams was very we11 expIaInea in mis handbook.

It is important to recognize that Section 3 of a written examination developed using the handbook mentioned above is a combination of Tiers 2 and 3 of a written examination developed using NUREG-1021 Revision 11. From the time that the NUREG-0737 requirements were put into effect until NUREG-1021 Revision 6 was published in June 1990, an applicant could not receive a written exam score less than an 80% overall or 70% in any of the sections as described in the handbook and pass the written examination.

After the GFE was split out from the RO and SRO written examinations in NUREG-1021 Revision 6 (ML14346A092), a minimum score of 80% was required to pass the GFE. The operator licensing examinations were given separately and required a minimum overall score of 80% in order to pass the examination. Passing the GFE was also a prerequisite for being approved to take the operator licensing examination. These requirements remained in effect through NUREG-1021 Revision 11.

NUREG-1021 Revision 12 would allow an applicant to miss every question on generic fundamentals (receive a score of 0%) and still pass the written examination. NUREG-1021 Revision 12 would allow generic fundamentals to comprise as low as approximately 8% of the written exam vice 24-25% as was determined using job/task analysis and sound educational testing principles as described in NUREG/BR-0122 Revision 4. This is a step change reduction in the number of generic fundamentals test questions and the elimination of a performance requirement on generic fundamentals questions needed to pass the written examination. During a time when the NRC is attempting to use a risk-based approach for decision making, it does not make sense that NUREG-1021 Revision 12 reduces the number of written examination questions by approximately 82-88% and eliminates the passing requirement for the area of licensed operator knowledge that led to the only power reactor partial core melt in United States history. In addition, NUREG-1021 Revision 12 dilutes the sampling of other areas on the written examination.

Changes to generic fundamental testing in NUREG-1021 Revision 12 do not demonstrate the principles of good regulation because they reduce the reliability of the operator licensing examination process. These changes do not support the Safety Strategic Goal of the Fiscal Year 2018-2022 Strategic Plan because they reduce the ability of the operator licensing examination process to ensure the safe use of radioactive materials. These changes also do not meet the Strategic Objectives because they could lead to a reduction in the ability of licensed operators to prevent and mitigate accidents.

Events have occurred recently at operating power reactors that have demonstrated a lack of generic fundamentals knowledge and/or application by control room operators. One example would be an August 19, 2020, source range high flux automatic trip that occurred due to excessive control rod withdrawal by the control room operators. With events occurring that are caused by a weakness of generic fundamentals knowledge, the NRC operator licensing staff should be taking actions that strengthen generic fundamentals knowledge rather than ones that weaken it further. The drastic reduction of generic fundamentals questions on the operator licensing exam and elimination of a minimum score required to pass that portion of the exam will result in less time spent training and studying generic fundamentals.

We recommend adding a statistically significant number of generic fundamentals questions to the reactor operator written examination without diluting the other topics that were sampled in NUREG-1021 Revision 11. Also, recommend grading the GFE and site-specific written examinations separately while requiring a minimum cut score of 80% to pass each portion of the examination, as has been required prior to NUREG-1021 Revision 12. The mix of questions should be approximately the same percentages as was determined by job/task analysis for the handbook discussed above. If the length of the written examination given on one-day is of concern, the senior reactor operator and reactor operator examinations could be given on different days.

The different sections of the operating test are given over a one to two-week period.

Another option would be to give a GFE as a separate section of the written examination any time within thirty days of the start of the operator licensing examination while keeping the balance of the written examination the same as it is in NUREG-1021 Revision 11. This would be no different than administering and grading the different sections of the operating test during this thirty day period as is currently done in NUREG-1021 Revision 11. As stated in the previous paragraph, the GFE and site specific portions of the written examinations should be graded separately using the same 80% cut score for each section, so as not to dilute the site-specific written examination requirements or the GFE requirements.

A third option would be to specify the content of a GFE examination and allow the licensees to administer and grade the GFE as part of the eligibility and application process. This would likely require some changes to 10 CFR 55.

NRC FORM 680 (09-2019)

Official Use Only - Sensitive Internal Information Page 2 of 4

Official Use Only - Sensitive Internal Information NRC FORM 680 U.S. NUCLEAR REGULATORY COMMISSION DPO Case Number (09-2019)

DPO-2021-002 NRC MD 10.159 DIFFERING PROFESSIONAL OPINION (Continued)

Date Received 09/27/2021 1 ne opuons 11stea aoove wou,a ensure mat app11cams are at risK OT oeIng testea on a meanmg1 u, numoer ana variety OT generic fundamentals topics during either the application or operator licensing examination process and prevent the simplistic testing of generic fundamentals in NUREG-1021 Revision 12 that would lead to a reduction in the amount of training and study and an eventual degradation of performance as is described in Appendix B of NUREG-1021 Revision 11.

• See attached list for co-submitters and their signatures due to being unable to fit the number of co-submitters in the blocks provided on the form.

Describe the (a) importance of prompt action on the issue, (b) safety significance of the issue, and (c) the complexity of the issue.

Importance of prompt action on the issue: Prompt action is important because all operator licensing exams given six months after the publishing date of NUREG-1021 Revision 12 are required to be conducted in accordance with Revision 12.

Safety significance of the issue: This is safety significant because the large reduction in generic fundamentals test questions and elimination of a minimum score required to pass those questions could result in less time spent in training preparation, less mental review and practice, more forgetting of factual details, less reinforcement and application of job concepts, and a gradual decline in the performance of control room licensed operators as stated in Appendix B (Section 8.1 ) of NUREG-1021 Revision 11. Weaknesses in generic fundamentals were a contributor to the accident at TMl-2. Without a solid fundamental knowledge base, operators may not perform acceptably in situations that are not specifically addressed in procedures. Weaknesses in generic fundamentals knowledge continue to impact operator performance as recently shown in the August 19, 2020, automatic reactor trip discussed above.

Complexity of the issue: This issue is not complex. The written examination content for operator licensing examinations described in NUREG/BR-0122 Revision 4 (ML20117N069) and the written examination guidelines contained in Appendix B of NUREG-1021 Revision 11 were based on sound educational and testing principles. NUREG-1021 Revision 12 strays from these principles.

Do you believe the issue represents an immediate 0

No Yes, (Explain in box above with importance of prompt public health and safety concern?

action and safety significance.)

Is the issue directly relevant to a decision pending 0

No Yes, Reference Document before the Commission?

(i.e., ADAMS ML#)

0 Informal discussions took place (Identify with whom and time frame of Extenuating circumstances prevented informal discussions discussions)

(1) A January 24, 2019 email from a R-11 member of the generic fundamentals examination working group informed the group lead from Operator Licensing and Human Factors branch that the region was not in favor of reintegration of the generic fundamentals examination as it was proposed to the working group by the program office.

(2) After receiving training from the Operator Licensing and Human Factors branch on NUREG-1021 Revision 12 during the week of May 3, 2021, the DPO submitter brought up his concerns about the generic fundamentals examination to the DEDR and DEDM at a question and answer session during the R-11 inspector counterpart meeting the week of June 7, 2021.

(3) On May 16, 2021, the DPO submitter and other members of the R-11 Operations Branches attended a meeting with the R-11 Director for the Division of Reactor Safety to discuss our concerns with the insufficient sample size and elimination of a required passing score for the generic fundamentals portion of the operator licensing written exams.

(4) On July 16, 2021, the DPO submitter presented his concerns regarding the generic fundamentals examination to all staff attending the Bi-Weekly HQ/Regional Division Directors Meeting. The Operator Licensing and Human Factors Branch Chief and other members of IOLB were present for this meeting.

Proposed panel members are (in priority order):

1. Gary Callaway, Sr. Reactor Technology Instructor

3. Craig Kontz, Sr. Project Engineer

2. Tom Hedigan, Operations Engineer D No names of potential panel members will be provided.

List of area(s) of technical expertise needed to properly assess the issue (e.g., electrical engineering, operator licensing).

Operator Licensing NRC FORM 680 (09-2019)

Official Use Only - Sensitive Internal Information Page 3 of 4

NRC FORM 680 (09-2019)

NRC MD 10.159 Official Use Only - Sensitive Internal Information U.S. NUCLEAR REGULATORY COMMISSION DPO Case Number DPO-2021-002 DIFFERING PROFESSIONAL OPINION (Continued)

Date Received 09/27/2021 When the process is complete, I would like management to determine whether public release of the DPO case file (with or without redactions) is appropriate (Select "No" if you would like the DPO case file to be non-public):

[Z] Yes 0

No Please note that your DPO submittal may be shared on a need-to-know basis in an effort to resolve the concern, determine the most appropriate regulatory actions in response to the concern, and identify key agency resources to evaluate the concern.

Signature of Submitter:

Signature of Co-Submitter (If any):

Signature of DPO Program Manager:

NRC FORM 680 (09-2019)

Joseph Demarshall Digitally signed by Daniel M. Bacon Date: 2021.09.27 07:44:29 -04'00' Digitally signed by Joseph Demarshall Date: 2021.09.27 14:54:05 -04'00' I

Submit by E-mail:

I Gladys J. Figueroa Toledo D DPO returned Digitally signed by Gladys J. Figueroa Toledo Date: 2021.11.02 11 :33:38 -04'00'

[Z] DPO accepted Official Use Only - Sensitive Internal Information Page 4 of 4

Official Use Only - Sensitive Internal Information Operator Licensing Generic Fundamentals Examination DPO Co-submitter Signature Page Name Title Region Signature Joseph Demarshall Senior Operations Engineer I

Joseph Dem a rs ha 11 Digitally signed by Joseph Demarshall Date: 2021.09.27 15:01 :28 -04'00' Kevin Murphy Operations Engineer I

Ke Vin M M u rp h y Digitally signed by Kevin M. Murphy Date: 2021.09.28 07:51 :48 -04'00' Pete Ott Operations Engineer I

Peter J. Ott Digitally signed by Peter J. Ott Date: 2021.09.27 13:52:38 -04'00' Randy Baker Senior Operations Engineer 111 <~ 0--~

Digitally signed by Randal D. Baker Date: 2021.09.28 07:52:11 -05'00' Chuck Zoia Senior Operations Engineer 111 Charles D Zoia Digitally signed by Charles D. Zoia Date: 2021.09.27 13:06:03 -05'00' Bryan Bergeon Operations Engineer IV Bryan A. Bergeon Digitally signed by Bryan A. Bergeon Date: 2021.09.27 13:19:13-05'00' Mark Bates Senior Operations Engineer II Mark A. Bates Digitally signed by Mark A. Bates Date: 2021.10.04 06:19:42 -04'00' Michael Donithan Operations Engineer II M' h I G Donithan Digitally signed by Michael G. Donithan IC ae Date: 2021.09.2811:53:34-04'00' Andreas Goldau Operations Engineer II Andreas S. Goldau Digitally signed by Andreas S. Goldau Date: 2021.09.27 13:30:23 -04'00' Michael Kennard Senior Operations Engineer II Michael N. Kennard Digitally signed by Michael N. Kennard Date: 2021.09.27 12:25:17 -04'00' Operations Engineer II Kevin P. Kirchbaum Digitally signed by Kevin P. Kirchbaum Kevin Kirchbaum Date: 2021.09.27 13:00:29 -04'00' 1

Official Use Only - Sensitive Internal Information

2 Newton Lacy Operations Engineer II David Lanyi Senior Operations Engineer II Michael Meeks Senior Operations Engineer II Joseph Viera Senior Operations Engineer II Newton T. Lacy Digitally signed by Newton T. Lacy Date: 2021.09.27 14:43:37 -04'00' David R. Lanyi Digitally signed by David R. Lanyi Date: 2021.09.27 12:51:21 -04'00' Michael Meeks Digitally signed by Michael Meeks Date: 2021.09.30 11:52:39 -04'00' Joseph P. Viera Digitally signed by Joseph P. Viera Date: 2021.09.27 12:46:27 -04'00' Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Document 2: Memo Establishing DPO Panel

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 November 19, 2021 MEMORANDUM TO:

Blake Welling, Panel Chair Region I Tom Hedigan, Panel Member Region I Scott Egli, Panel Member Office of the Chief Human Capital Officer THRU:

Mark D. Lombard, Director Office of Enforcement FROM:

Gladys Figueroa-Toledo, Differing Views Program Manager Office of Enforcement

SUBJECT:

AD HOC REVIEW PANEL - DIFFERING PROFESSIONAL OPINION ASSOCIATED WITH THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

In accordance with Management Directive (MD) 10.159, The NRC Differing Professional Opinion Program; and in my capacity as the Differing Views Program Manager (DVP PM); and in coordination with Mark D. Lombard, Director, Office of Enforcement, Andrea Veil, Director, Office of Nuclear Reactor Regulation and the Differing Professional Opinion (DPO) submitters; you are appointed as members of a DPO Ad Hoc Review Panel (DPO Panel) to review a DPO submitted by a group of U.S. Nuclear Regulatory Commission (NRC) employees.

The DPO (Enclosure 1) involves the Operator Licensing Examination Standards for Power Reactors, NUREG-1021, Revision 12 concerning the Generic Fundamentals Examination. The DPO has been forwarded to Ms. Veil for consideration and issuance of a DPO Decision.

The DPO Panel plays a critical role in the success of the DPO Program. Your responsibilities for conducting the independent review and documenting your conclusions in a report are addressed in the handbook for MD 10.159 in Section II.F and Section II.G, respectively. The DPO Web site also includes helpful information, such as a Differing Views Best Practices Guide, tables with status information and timeliness goals for open DPO cases, and closed DPO case files (which include previous DPO panel reports). We will also send you additional information that should help you implement the DPO process.

CONTACT:

Gladys Figueroa-Toledo, OE (301) 287-9497 Tania Martinez Navedo Digitally signed by Tania Martinez Navedo Date: 2021.11.19 12:22:38 -05'00' Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

B. Welling, et al.

2 Timeliness is an important DPO Program objective. Thus, the disposition of this DPO should be considered an important and time sensitive activity. Although MD 10.159 identifies a timeliness goal of 75 calendar days for the DPO panel review and report and 21 additional calendar days for the issuance of a DPO Decision, the DPO Program also sets out to ensure that issues receive a thorough and independent review. Therefore, the overall timeliness goal will be based on the significance and complexity of the issues, schedule challenges, and the priority of other agency work. Process milestones and timeliness goals specific to this DPO will be discussed and established at a kick-off meeting.

Communication of expected timelines and status updates are important in the effectiveness and overall satisfaction with the Differing Views Program. If you need an extension beyond the timeliness goal, please send an e-mail to Mr. Lombard, Ms. Veil, the DPO submitters, and DPOPM.Resource@nrc.gov that includes the reason for the extension request and a proposed completion date.

An important aspect of our organizational culture includes maintaining an environment that encourages, supports, and respects differing views. As such, you should exercise discretion and treat this matter appropriately. To preserve privacy, minimize the effect on the work unit, and keep the focus on the issues, you should simply refer to the employees as the DPO submitters. Avoid conversations that could be perceived as hallway talk on the issue and refrain from behaviors that could be perceived as retaliatory or chilling to the DPO submitters or that could potentially create a chilled environment for others. It is appropriate for employees to discuss the details of the DPO with their co-workers as part of the evaluation; however, as with other predecisional processes, employees should not discuss details of the DPO outside the agency. If you have observed inappropriate behaviors, heard allegations of retaliation or harassment, or receive outside inquiries or requests for information, please notify the Office of Enforcement.

On an administrative note, please ensure that all DPO-related activities conducted by staff are charged to Activity Code ZG0007. Managers should report time to their Management/Supervisor Activity Code. Administrative Assistants should report time to their Secretary/Clerical Activity Code.

We appreciate your willingness to serve on the DPO Panel and your dedication to completing a thorough and objective review of this DPO. Successful resolution of the issues is important for the NRC and its stakeholders. If you have any questions or concerns, please feel free to contact me. We look forward to receiving your conclusions and recommendations.

Enclosures:

1.

DPO-2021-002 Submittal (ML21322A200 and ML21322A199) 2.

Process Milestones and Timeliness Goals (ML21323A002) cc: A. Veil, NRR A. Kock, NRR R. Taylor, NRR M. King, NRR D. Betancourt, NRR D. Bacon, RII Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

B. Welling, et al.

3 J. Demarshall, RI K. Murphy, RI P. Ott, RI R. Baker, RIII C. Zoia, RIV B. Bergeon, RIV M. Bates, RII M. Donithan, RII A. Goldau, RII M. Kennard, RII K. Kirchbaum, RII N. Lacy, RII D. Lanyi, RII M. Meeks, RII J. Viera, RII M. Lombard, OE T. Martinez-Navedo, OE D. Solorio, OE G. Figueroa-Toledo, OE Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

B. Welling, et al.

4

SUBJECT:

AD HOC REVIEW PANEL - DIFFERING PROFESSIONAL OPINION ASSOCIATED WITH THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

DATE: 11/19/2021 ADAMS Package: ML21323A006 MEMO: ML21323A001 - (ML21322A200 and ML21322A199) - ML21323A002 OE-011 OFFICE OE: DPO/PM OE: D NAME GFigueroaToledo MLombard (TMartinez Navedo for)

DATE 11/18/2021 11/19/2021 OFFICIAL RECORD COPY Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Document 3: DPO Panel Report

UNITED STATES NUCLEAR REGULATORY COMMISSION REGION I 2100 RENAISSANCE BOULEVARD, SUITE 100 KING OF PRUSSIA, PA 19406-2713 April 5, 2022 MEMORANDUM TO:

Andrea Veil, Director Office of Nuclear Reactor Regulation FROM:

Blake D. Welling, DPO Panel Chair Region I Scott Egli, DPO Panel Member Blake D.

Welling Richard Digitally signed by Blake D. Welling Date: 2022.04.05 14:37:11 -04'00' Digitally signed by Richard S. Egli Office of the Chief Human Capital Officer Thomas E. Hedigan, DPO Panel Member Region I S. Egli Thomas E.

Hedigan Date: 2022.04.05 13:58:51 -04'00' Digitally signed by Thomas E. Hedigan Date: 2022.04.04 14:15:29 -04'00'

SUBJECT:

DIFFERING PROFESSIONAL OPINION PANEL REPORT ON THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

In a memorandum dated November 19, 2021, we were appointed as members of a Differing Professional Opinion (DPO) Ad Hoc Review Panel (the Panel) to review a DPO associated with The Operator Licensing Examination Standards for Power Reactors, NUREG-1021, Revision 12, concerning the Generic Fundamentals Examination. The Panel has reviewed the DPO in accordance with the guidance in Management Directive 10.159, The NRC Differing Professional Opinion Program. The scope was limited to a review of the issues identified in the DPO as clarified through a Summary of Issues developed by the Panel and confirmed by the DPO Submitters. The panel evaluated the issues through interviews of knowledgeable NRC staff and a review of various documents including official agency records.

The results of the panels evaluation of the concerns raised in the DPO are detailed in the enclosed DPO Panel report. The following conclusions are based on our review of concerns raised in the DPO, information obtained in interviews, and our review of additional documents.

The Panel finds that the NUREG-1021 Operator Licensing Examination Standards for Power Reactors, Revision 12, method of testing generic fundamentals, namely, Reactor Theory, Thermodynamics, and Components, in the Reactor Operator (RO) licensing exam, does not determine if the applicant has mastered this subject matter. Since the purpose of a licensing exam is to determine if an applicant has mastered the subject matter and then to base the licensing decision on the exam score, the changes to the RO exam has diminished the NRCs ability to make accurate licensing decisions for license applicants.

The panel finds that reducing the number of generic fundamental questions has lowered the content validity of the exam. NUREG-1021, Rev. 12, defines content validity as The degree to Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

A. Veil 2

which a test measures the specific objectives or content. The low content validity of the exam for the generic fundamentals directly leads to the above finding that the licensing decision was adversely affected.

The Panel finds that the RO exam does meet 10 CFR 55.41, Written Examination: Operators.

This regulation requires that the written exam for an operator of a facility include a representative sample from 14 items to the extent applicable to the facility. Reactor Theory, Thermodynamics and Components are among the 14 items. Since licensing exams prepared using the guidance in NUREG-1021, Rev. 12, sample these three subjects, we concluded that the regulation is met.

The Panel offers the following recommendations for your consideration:

Establish a basis document for NRC operator exams. This basis document should include the national standards on which the program is based. It should specify the professional background in educational testing and measurement that would be necessary for personnel who design and revise NRC exams. Additionally, the basis document should define the qualifications for a Subject Matter Expert (SME) involved in the NRC exam process. Finally, the basis document should detail the process for determining the composition of the exams.

Once the resources are in place and there is agreement on the basis of the program, a working group with subject matter experts and representation from all four regions should revisit the RO written exam and evaluate all the key parameters, including the length of the exam, cut scores, weighting of different topics and whether there should be a separate generic fundamentals exam.

The Panel recommends that a more formal process be developed to obtain regional input into program office decisions. This could be a best practices document that provides a standard framework for working groups. This document should include practices for communications, seeking input, and resolving differing views within the process.

Enclosure:

DPO Panel Report Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

A. Veil 3

cc w/encl:

A. Veil, NRR J.Rankin, NRR D. Bacon, RII J. DeMarshall, RI K. Murphy, RI P. Ott, RI R. Baker, RIII C. Zoia, RIII B. Bergeon, RIV M. Bates, RII M. Donithan, RII A. Goldau, RII M. Kennard, RII K. Kirchbaum, RII N. Lacy, RII D. Lanyi, RII M. Meeks, RII J. Viera, RII M. Lombard, OE T. Martinez-Navedo, OE D. Solorio, OE G. Figueroa-Toledo, OE B. Welling, RI S. Egli, OCHCO T. Hedigan, RI Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

A. Veil 4

DIFFERING PROFESSIONAL OPINION PANEL REPORT ON THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO 2021-002) DATED APRIL 5, 2022 DOCUMENT: https://usnrc-my.sharepoint.com/personal/tmh_nrc_gov/Documents/Documents/DPO PANEL Report Copy 1.docx SUNSI REVIEW - MD 3.4 Non-Public A.7 OFFICE RI/DORS OCHCO RI/DRSS NAME Hediganr/th Egli/rse Welling/bdw DATE 04/04/22 04/0522 0405/22 Official Record Copy Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Richard S. Egli Digitally signed by Richard S.

Egli Date: 2022.04.05 13:59:59

-04'00' Differing Professional Opinion (DPO)

On Operator Licensing Examination Standards for Power Reactors, NUREG-1021, Revision 12 (DPO-2021-02)

DPO Panel Report Blake D.

Welling Digitally signed by Blake D. Welling Date: 2022.04.05 14:37:47

-04'00' Blake D. Welling, Panel Chair Scott Egli, Panel Member Thomas E.

Hedigan Digitally signed by Thomas E.

Hedigan Date: 2022.04.04 14:16:00

-04'00' Thomas E. Hedigan, Panel Member April 5, 2022 Date Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

ii Contents TABLE OF CONTENTS Introduction................................................................................................................................ 1 Summary of Issues.................................................................................................................... 1

Background:

Operator License Exam History for Generic Fundamentals...................................3

Background:

General Information on the Purpose of Licensing Exams...................................... 4 Content Validity of NRC Licensing Exams................................................................................... 6 Information on NUREG-1021, Rev. 12........................................................................................ 7 DPO Panel Evaluation of the Summary of Issues....................................................................... 8 DPO Submitters Recommendations......................................................................................... 17 Working Group Timeline and Structure..................................................................................... 18 Safety and Risk Significance.....................................................................................................18 Panel Findings..........................................................................................................................18 Panel Recommendations..........................................................................................................19 Appendix A - Documents Reviewed......................................................................................... A-1 Appendix B - NRC Staff Interviewed......................................................................................... B-1 Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

1 Introduction On September 27, 2021, several U.S. Nuclear Regulatory Commission (NRC) staff members filed a Differing Professional Opinion (DPO) in accordance with Management Directive 10.159 The NRC Differing Professional Opinions Program. The DPO involves changes to the reactor operator initial licensing exam, which is required by 10 CFR 55.41. The DPO submitters stated that NUREG-1021, Revision 12, deemphasizes the importance of generic fundamentals and drastically reduces the ability of the operator written licensing exam to discriminate between an applicant that has an adequate knowledge of generic fundamentals, and one who does not.

The NRCs Office of Enforcement accepted the DPO on November 2, 2021, and assigned the DPO case number 2021-002. By memorandum dated November 19, 2021, the Office of Enforcement established an Ad Hoc Review Panel (the Panel) to perform a review of the DPO.

The Panel developed a draft Summary of Issues (SOI) and shared it with the submitters on January 10, 2022. Following discussions and feedback from the submitters, a final SOI was agreed to by the DPO Panel and the submitters on January 25, 2022. The SOI is documented in the next section.

The Panel was tasked with reviewing the individual DPO issues and providing conclusions along with recommendations, as necessary. Following initial discussions with the submitters and development of the SOI, the Panel performed its review by reviewing documents and conducting interviews with NRC staff. A list of documents reviewed, and NRC staff interviewed are listed in Appendices A and B, respectively.

Summary of Issues Based on a review of the DPO submittal and associated references as well as an interview with the lead submitter, the Panel identified that the submitters concerns are all related to changes to the composition of the licensed operator initial exam as detailed in NUREG-1021, Rev. 12.

The issues raised by the submitters are summarized as follows by the Panel:

1.

NUREG-1021, Rev. 12, diverges from the Commission approved corrective actions taken with regard to operator licensing and training as a result of the accident at Three Mile Island-2 (TMI-2) as documented in NUREG-0737 (1980), which mandated that:

o A new category shall be added to the operator written examination entitled, Principles of Heat Transfer and Fluid Mechanics."

o A new category shall be added to the senior operator written examination entitled,

Theory of Fluids and Thermodynamics."

o The passing grade for the written examination shall be 80% overall and 70% in each category of the entire operator or senior operator examination.

Written license exams contained this new category with 24 to 25 questions requiring a passing score of 70% until 1989.

2.

In 1989, a standalone Generic Fundamentals Exam (GFE) which examined operator license candidates on their knowledge of Reactor Theory, Thermodynamics, Heat Transfer, Fluid Flow, & Plant Components was introduced by GL 89-17. This exam was 100 questions from 1989 to October 2003 and was reduced to 50 questions from March 2004 to present. Both GL 89-17 and NUREG-1021 Revision 6 established that a Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

2 minimum score of 80% was required to pass the GFE. Revision 12 to NUREG-1021 does not require a minimum cut score for the generic fundamentals questions, of which there are only 6 to 10. An operator license candidate could pass a license exam written to NUREG-1021, Rev. 12 standards even after incorrectly answering every fundamentals question. Additionally, the minimum educational requirement for an RO license is a high school diploma. Under Revision 12, it would be possible for a candidate for the RO and SRO-upgrade to obtain both licenses without answering any of the generic fundamentals questions correctly, and never receiving any formal education on GFE topics at a post-high school level.

3. Revision 12 to NUREG-1021 deemphasizes the importance of generic fundamentals by greatly reducing the required number of questions to be asked in the area of fundamentals. This is an 80-88% reduction and drastically reduces the ability of the operator licensing written examination to discriminate between an applicant who has an adequate knowledge of generic fundamentals and one who does not.

4. Revision 12 to NUREG-1021 also dilutes the sample of other topics that are currently tested on the operator licensing written examinations. Deleting 6 to 10 questions from the current sample plan to accommodate the fundamentals questions reduces the ability to discriminate between an applicant who has an adequate knowledge of systems and procedures and one who does not.

5. When combined with industry initiatives that have reduced the length of operator license classes and reduced the educational requirements for eligibility, the change to NUREG-1021 could result in less time spent in training preparation, less mental review, and practice, more forgetting of factual details, less reinforcement and application of job concepts, and a gradual decline in the performance of control room licensed operators.

Future operators may not have the foundational knowledge to address emergency situations that may not be fully covered in procedures.

6. The Examiners' Handbook for Developing Operator Licensing Examinations, NUREG/BR-0122 Revision 4, was prepared by the Professional Examination Service company for the NRC in 1988. It established the sample size for all the topics tested on RO and SRO exams. NUREG 1021 Rev. 12 did not utilize the information in NUREG/BR-0122 Revision 4, a professional education services company was not consulted, nor were other supporting bases used to establish the fundamentals sample size of the new RO exam. Additionally, no job task analysis or justification for the change/reduction was performed.

7. Previous changes were the result of TMI action items (NUREG 0737), Generic Letters, and NUREG/BR-0122. Revision 12 appears to have been approved/reviewed by a more truncated review process, which may not have been as thorough. The review process should have been more extensive.

8. Even with the training and testing that is currently being performed, several recent serious events have occurred on operating power reactors that could have been prevented by the proper application of generic fundamentals knowledge. Therefore, a reduction to the examination standards in generic fundamentals is not supported from a performance-based review.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

3

Background:

Operator License Exam History for Generic Fundamentals In the spring of 1955, the Atomic Energy Commission (AEC) drafted a regulation to implement its mandate on licensing reactor operators. The agency sought to keep its role to a minimum.

The regulation required applicants for operators licenses to pass a test and a medical examination. Otherwise, it left responsibility for training and evaluating operators to plant owners.

The regulation that the Commission approved in 1955 remained in effect without major revisions for a decade and a half. In the late 1960s, however, the regulatory staff revisited the issue of operator licensing. The staff proposed that new provisions be added to the regulations that would require licensees to provide requalification programs as a prerequisite for the renewal of individual operators' licenses. After public comment and Commission review, the new rules became effective in September 1973.

In 1975, the Nuclear Regulatory Commission (NRC) was formed and took over the regulatory responsibilities that the AEC had carried out in regard to operator licensing. Shortly thereafter in March 1979, the Three Mile Island accident occurred. The performance of reactor operators was a major contributor to the severity of the accident. The President's Commission on the Accident at Three Mile Island, chaired by John G. Kemeny, concluded that "the equipment was sufficiently good that, except for human failures, the major accident at Three Mile Island would have been a minor incident." The report to President Carter in October 1979 minced no words in citing shortcomings in operator training. Many of these shortcomings were related to lack of operator knowledge in the subject areas of reactor physics, thermodynamics, fluid flow, and heat transfer. These along with the subject area of power plant components would be grouped together and later called generic fundamentals.

NUREG-0737, Clarification of TMI Action Plan Requirements, was published in November of 1980 and included the following items related to generic fundamentals training and examination:

Training programs shall be modified, as necessary, to provide:

1)

Training in heat transfer, fluid flow and thermodynamics.

2)

Training in the use of installed plant systems to control or mitigate an accident in which the core is severely damaged.

3)

Increased emphasis on reactor and plant transients.

The scope of operator license exams was increased:

1)

A new category shall be added to the operator written examination entitled, Principles of Heat Transfer and Fluid Mechanics."

2)

A new category shall be added to the senior operator written examination entitled,

Theory of Fluids and Thermodynamics."

3)

The passing grade for the written examination shall be 80% overall and 70% in each category.

Shortly after this, the NRC license exams as described by early revisions of NUREG-1021, Operator Licensing Examination Standards for Power Reactors and NUREG/BR-0122, Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information Examiners' Handbook for Developing Operator Licensing Examinations, stated that the reactor operator written examinations should normally sample: (1) 25% from the fundamentals area (reactor theory, thermodynamics, and component operation), (2) 48% from plant systems and plant-wide safety and administrative procedures, and (3) 27% from emergency and abnormal plant evolutions. For senior reactor operators, the written examination content should normally sample: (1) 24% from fundamentals, (2) 33% from emergency and abnormal evolutions, and (3) 43% from plant systems and plant-wide safety and administrative procedures. An operator taking these exams had to achieve an 80% overall score and at least of 70% in each section to pass the exam. Exams of this description were used through 1988.

In 1989, the generic fundamentals subject areas were split out from the RO and SRO written examinations in NUREG-1021, Revision 6, and a minimum score of 80% was required to pass the new generic fundamental exam (GFE). The operator license examinations were given separately sometime after the GFE and required a minimum overall score of 80% in order to pass that examination. Passing the GFE was also a prerequisite for being approved to take the operator licensing examination. Initially, the GFE was a 100-question exam offered 3-times a year. In 2004, the exam questions were reduced to 50, and it was offered 4 times a year. This remained in effect until 2016, when the exam offerings were reduced to 3 times a year, and in 2017 the offerings were dropped to twice a year. The twice-a-year offering of the 50 question GFE exam remained in effect through Revision 11 of NUREG-1021.

Revision 12 of NUREG-1021 requires only 6 reactor theory and thermodynamics questions and up to 4 component questions, which allows generic fundamentals to comprise as low as approximately 8% of the written exam, vice 24-25% required on exams previous to 1989. Also, NUREG-1021, Rev. 12, does not grade the fundamental questions as a separate section, meaning a licensee candidate could get all of them all incorrect and still pass the exam.

Background:

General Information on the Purpose of Licensing Exams The following discussion will provide the reader with a general overview of the purpose of licensing exams. This will aid in understanding the discussions later in the report about the specifics of NRC licensing exams. The information presented is from the Standards for Educational and Psychological Testing, 2014 edition.1 1 American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, eds. 2014. Standards for Educational and Psychological Testing, Lanham, MD: American Educational Research Association. Pages 174-176.

4 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information 5

Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information Content Validity of NRC Licensing Exams NUREG-1021, Rev. 12 defines content validity as "The degree to which a test measures the specific objectives or content." From Appendix A, "For a test to be considered valid, it must be shown to measure that which it is intended to measure. In the case of the NRC licensing examination, the intent is to measure the candidate's knowledge and ability such that those who are licensed will perform the duties of the RO and SRO to ensure the safe operation of the plant. The general sequence of activities needed to establish the content validity of NRC licensure examinations are outlined below."

Appendix A outlines the method to establish the content validity of NRC exams. This method was outlined in NUREG/BR 0122 in 1988 and has been included in Appendix A of every NUREG-1021 revision since 1989.

The first step in the sequence of activities was to establish a link to job duties. A job task analysis (JTA) was performed by the Institute of Nuclear Power Operations (INPO) which identified approximately 28,000 knowledge and abilities (Kl As). The job content of special interest to the NRC is that subset of Kl As that are required for the safe operation of the nuclear plant. From these KIAs, the KIA catalogues were developed for boiling water reactors (BWRs) and pressurized water reactors (PWRs). There are approximately 8,000 Kl As in each of these catalogues. The individual KIAs were rated for importance to safety by both licensed SROs and license examiners. About 100 subject matter experts (SMEs) were used for each catalogue.

The next step is to develop a sample plan. Appendix A states, "Once the essential Kl As have been identified through the JTA, test specifications must be developed. The test specifications consist of a content outline or sample plan indicating what proportion of items or questions shall deal with each KIA. Because a single test cannot measure every KIA required to be a licensed 6

Official Use Only - Sensitive Internal Information

7 operator, it must sample the required knowledge or performance in a manner that allows inferences to be made about the examinees performance on the broader population of knowledge, even though the full body of knowledge was not tested. The sample must be evenly distributed and soundly based so that the NRC can confidently assume that the untested knowledge is proportionately known or not known in relation to the score on the sample. In other words, by testing performance on the sample, it is possible to make inferences concerning the broader area of knowledge not tested. This is referred to as a validity inference.

The sample plan is at the heart of making a validity inference. Research indicates that when samples are not chosen systematically and according to the sample plan, the sample is biased, and, therefore, its validity is reduced. When the sample is biased or skewed in a particular direction, it introduces some degree of sampling error, which makes it impractical to infer or generalize that the examinees have mastered the larger population of untested knowledge from which the sample was drawn.

Information on NUREG-1021, Rev. 12 The Abstract section of NUREG-1021, Rev. 12, states that the NRC issued the revision to:

(1) streamline information into topic-based sections for ease of use, (2) clarify instructions for the identification and grading of performance deficiencies on the operating test, (3) revise instructions for the selection of critical tasks and the assessment of critical and significant performance deficiencies, and (4) implement changes to support the testing of fundamentals topics on the site-specific initial licensing examination, in the place of a separate generic fundamentals examination. Item (4) is the focus of this review.

A Note to Commissioners Assistants dated July 28, 2020, (ML21119A862) provided information to the Commission on the reasons for the integration of the GFE into the RO initial licensing written examination. It stated that the initiative was started in response to industry feedback on the impacts of reducing the number of GFE administered per year. The goal was to improve the operational focus of questions on fundamental concepts and to provide increased schedule flexibility in support of industry training programs and the NRCs National Examination Schedule. In addition, the Note indicated that the change would be more efficient and cost effective as licensee and NRC resources needed for developing, administering, grading, and evaluating the GFE would be reduced. Thus, the staff incorporated the integration of generic fundamental concepts into the site-specific examination process in NUREG 1021, Rev. 12.

The Panel notes that the integrated examination process in NUREG-1021, Rev. 12, uses the previously existing job task analysis and the K/A catalogues for the RO written exam; however, the sample plan has changed. Specifically, it changes the sample plan by combining the 50 question GFE with the 75 question RO exam into a single 75 question exam, which is a notable deviation from the previous revision.

The 50 question GFE exam for a BWR was comprised of 14 Reactor Theory questions, 14 Thermodynamic questions, and 22 Component questions. The 14 Reactor Theory questions were distributed within 8 subtopics, Neutrons (1), Neutron Lifecycles (1), Reactor Kinetics and Neutron Sources (1), Reactivity Coefficients (2), Control Rods (2), Fission Product Poisons (2),

Fuel Depletion and Burnable Poisons (1), Reactor Operational Physics (4). The 14 Thermodynamics questions were distributed within 10 subtopics, Thermodynamic Units and Properties (1), Basic Energy Concepts (1), Steam (1), Thermodynamic Processes (1),

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

8 Thermodynamic Cycles (1), Fluid Statics and Dynamics (2), Heat Transfer (1), Thermal Hydraulics (2), Core Thermal Limits (3), Brittle Fracture and Vessel Thermal Stress (1). The 22 Component questions were distributed within 8 subtopics, Valves (3), Sensors and Detectors (4), Controllers and Positioners (2), Pumps (4), Motors and Generators (2), Heat Exchangers and Condensers (3), Demineralizers and Ion Exchangers (2), Breakers Relays and Disconnects (2). The GFE for PWRs had a similar distribution.

The NUREG-1021, Rev. 12, RO exam sample plan has 3 Reactor Theory, 3 Thermodynamics, and between 0-4 Component questions. Only 1 question can be selected for each subtopic. The Panel observes that not only is the number of questions testing fundamentals reduced, but the broad testing of all 26 subtopics, which were tested in the GFE, are not tested under NUREG-1021, Rev. 12.

The Panel notes that the method to determine how many questions the sample plan tests for each topic was not performed by personnel with educational testing or measurement experience. Also, the changes to the sample plan did not include a large group of SMEs. The NRC should consider having individuals with subject matter expertise evaluate the number of questions in the sample plan.

The Operator Licensing and Human Factors Branch (IOLB), Division of Reactor Oversight, NRR developed the sample plan for NUREG-1021, Rev. 12. The method that IOLB used to determine the new sample plan is described more fully in the Evaluation section below; however, the basic approach was that the importance to safety of the generic fundamental topics was equivalent to the importance to safety of a plant system. The K/A importance ratings for Reactor Theory and Thermodynamics were averaged and compared to average K/A ratings for several systems.

DPO Panel Evaluation of the Summary of Issues Issues 1 and 2 The Panel evaluated issues 1 and 2 together because they both point out historical reasons that the DPO submitters believe support their position that the RO exam testing of generic fundamentals should not have been significantly reduced. One point the DPO submitters are making is that the lessons learned and corrective actions due to the accident at Three Mile Island (TMI) should weigh heavily when considering the testing of fundamentals. Specifically, after the accident, new sections to the exam were added requiring testing of Principles of Heat Transfer and Fluid Mechanics," and Theory of Fluids and Thermodynamics." In 1989, when the GFE was established, both topics were tested in the Thermodynamics section of the GFE. The second point in issue 1 and 2 is that the applicant needed to pass an individual section prior to 1989, or pass the GFE since 1989, demonstrating that they had mastered the fundamental topics. Exams prepared using NUREG 1021, Rev. 12, will not require a separate passing score of the 6 to 10 fundamentals questions.

The Panel interviewed most staff in IOLB and the regional operator licensing branches.

Approximately 80% of the regional staff supported the views of the DPO submitters, in that the importance of testing fundamentals warrants more than 3 questions in Reactor Theory and 3 questions in Thermodynamics. Many of staff in this group referred to the TMI accident and related how the lessons learned informed their opinion. Most of the IOLB staff supported the method of testing fundamentals in Rev. 12. Of the 20% in the regions that did not support the Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

9 DPO position, about half were neutral and felt it would take time to judge whether the change was effective or not. The other half felt that it was a management decision to change the method of testing fundamentals, and people should accept the change.

The Panel agrees that the TMI accident corrective actions should weigh heavily when assessing the importance of operators knowledge of fundamentals. The Panel agrees with the DPO submitters that it is important that the applicants demonstrate their knowledge of the fundamentals, however the Panel notes that this could be achieved by either having a separate passing score for fundamentals, or by having a statistically large enough number of fundamental questions, such that an applicant could not pass the exam without mastering the fundamentals topics.

Issues 3, 4 and 6 The Panel evaluated issues 3 and 4 together because they both are associated with the reduction in the number of questions on the RO exam. Issue 3 covers the reduction of questions for Reactor Theory, Thermodynamics, and Components. Issue 4 is associated with the reduction of questions for Generic Knowledge and Abilities, Plant Systems and Emergency and Abnormal Plant Evolutions. Issue 6 was included in this section because it is related to issue 3 and 4, in that the DPO submitters stated that the method to determine the reductions in the number of questions did not follow NRC past practices, use a professional education services company, or establish a basis for the changes.

When the GFE was split out of the RO exam in 1989, a pilot was conducted prior to full implementation. In SECY 89-292, the staff described the method to construct the GFE exam:

The staff used the "Examiners' Handbook for Developing Operator Licensing Examinations,"

NUREG/BR-0122, in conjunction with the Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling and Pressurized Water Reactors, NUREG-1123 and 1122, to determine the depth of the examination and the areas to be covered. The sampling plan from the examiners handbook ensured that each subject received the correct emphasis on the examination. The Knowledge and Abilities Catalog provided the detailed subject areas and importance values from which questions were constructed.

In contrast to the process described in SECY 89-292, which supports the DPO submitters position, the following discussion will describe the method that was used to determine the composition of the sample plan for the NUREG-1021, Rev. 12, exam process.

In December 2018, a GFE working group (WG) was chartered to explore options for changing the GFE. The WG developed proposed changes to the GFE. IOLB then presented 3 options to IOLB division management and regional division management. Management then selected option 1, which was to integrate the GFE topics into the RO exam.

The Panel interviewed members of the GFE WG that developed the proposed changes to the RO exam, and the Panel reviewed several of the documents the WG used to determine the number of questions that should be on the exam. The WGs charter included three issues with the RO exam that the WG sought to address by proposing possible solutions, including Generic Fundamental Exam integration. One of the issues was the GFEs program currently provides 33% of all written examination test items for senior reactor operators and 40% of all written examination test items for reactor operators, indicating that fundamentals are being oversampled relative to the 14 items required by regulation.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information The Panel notes that IOLB decided that the composition of the GFE and RO exam constituted "oversampling" without an analysis, justification, or discussion within the examiner community, even though the composition of the exam had been the same since 2004. The 100 question GFE sample plan was based on NUREG/BR-0122. The Panel did not find a basis for the 50 question GFE that started in 2004. The Panel notes that "oversampling" was one of the 3 drivers for IOLB revising the RO exam, which does not have a definition in the Standards. However, the Standards does discuss A GFE integration talking point paper provided by IOLB stated the following to justify the reduction in questions on the RO exam, "The number of questions on this examination remain the same as the previously administered final examination, reducing the number of questions asked on the theory and component topics from 50 to at least 6. This brings the coverage of these topics on the test into alignment with other test topics of similar safety significance."

On the next page of this report is a slide from an IOLB PowerPoint presentation that was provided to the Panel. This slide indicates that the Reactor Theory and Thermodynamic Topics have the same importance to safety as a plant system. Copied from the chart are the following statements, "For below charts, the number above the bar indicates the average importance rating - the direct and indirect impact of the topic on ensuring personnel and public health and safety." Additionally, "Note: Although the reactor theory and thermodynamics topics have a similar importance to safety to other topics, they are tested 3-5 times more in the current process."

IOLB is stating that the importance rating (IR) 3.2 of Reactor Theory, as compared to the IR 3.5 of AC Distribution (one of the systems tested in Plant Systems), does not justify testing Reactor theory at 3 to 5 times the rate of testing AC Distribution. Similar comparisons are made for the Recirculation system, and High Drywell pressure (Emergency and Abnormal Plant Evolutions.)

Both Boiling Water Reactor (BWR) and Pressurized Water Reactor (PWR) comparisons are made in the slide.

The charts to the right side of the slide provide similar comparisons for the new proposed exam testing 3 Reactor Theory and 3 Thermodynamic questions. The following note is above the chart, "Note: The proposed outline would test reactor theory and thermodynamics at an equivalent rate to other topics with similar importance to safety."

The bottom portion of the slide compares the old outline to the proposed new outline with the following note at the bottom, "*Note: Added six theory questions by reducing the Generic Knowledge and Abilities by 4 questions and risk-informing a single question reduction in both "Group 2" categories, which have lower contribution to the public protection function of the examination and therefore have a lower testing emphasis."

2 American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, eds. 2014. Standards for Educational and Psychological Testing, Lanham, MD: American Educational Research Association. Page 12.

10 Official Use Only - Sensitive Internal Information

11 Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information The Panel reviewed the DPO submitters' concerns presented in issues 3, 4 and 6, and compared them to the methods that were used by IOLB to determine the new outline. Before proceeding with the analysis, the Panel notes that the DPO submitters are questioning the content validity of the new exam. From the DPO, "The NRC staff developed NUREG/BR-0122 Revision 4 to improve the content validity of the operator licensing examinations. Content validity was ensured through the performance of job/task analysis focusing on the delineation of essential knowledge and abilities. The educational methodology used to develop the content of licensing exams was very well explained in this handbook."

The crux of the matter is that "the sample of Kl As that are tested should cover all of the KIA categories in the catalog in a fashion that is consistent with their contribution to the public protection function of the examination," per Appendix A of NUREG-1021.

The Panel notes that the importance to safety of topics is a professional opinion. Some staff may overemphasize the importance, and some staff may underemphasize the importance. The use of a large group of SMEs to evaluate the new sample plan could have averaged out the high and low opinions and led to better-informed results on the importance to safety ratings.

This contrasts with the fact that a small group determined the importance to safety of the fundamental topics for the Rev. 12 exam process.

The method that IOLB used to determine the importance of the fundamental topics did not utilize SMEs. IOLB averaged the KIA importance ratings (IR) of a broad topic like Reactor Theory and compared it to the average IR of a plant system. The logic was if AC Distribution has an average importance rating of 3.5, then it has more importance to safety than Reactor Theory which has an average IR of 3.2. The next step in IOLB's logic is that the number of questions on the test for Reactor Theory should be the same as a plant system. The I Rs were not intended to be used this way. Each individual KIA was assigned an IR by a group of SMEs.

They rated the KIAs from 1 to 5, with 5 being essential and 1 of insignificant importance. The KIA catalogue explains that a rating of below 2.5 represents a statement of limited or insignificant importance for the safe operation of a plant. Such statements are generally considered as inappropriate content for NRC licensing examinations. In the KIA catalogues, there are a substantial amount of KIA statements that have low I Rs. You cannot average all the KIA's I Rs in a system or topic and make a logical conclusion that the overall system or topic has that importance to safety.

The Panel concludes that the logic used by IOLB to make the changes to the RO exam in Rev.

12 was not appropriate. By reducing the number of total questions, from 125 (GFE and RO exam under Rev. 11) to 75 (Rev. 12), without using SM Es, the content validity of the exam has been reduced. The Panel's opinion is that a low number of fundamental questions is a 3 American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, eds. 1985. Standards for Educational and Psychological Testing, Lanham, MD: American Educational Research Association. Page 11.

12 Official Use Only - Sensitive Internal Information

13 construct underrepresentation, which implies a narrowed meaning of test scores because the test does not adequately sample fundamentals.

The Panel notes that issue 6 specifically mentions that NUREG 1021, Rev. 12, did not utilize the information in NUREG/BR-0122 Revision 4, a professional education services company was not consulted, nor were other supporting bases used to establish the generic fundamentals sample size for the new RO exam. The Panel considers this issue to be a key statement. There are a number of historical documents that could be used as input into a basis for the operator licensing written exams. In fact, the NRC should pursue a basis that includes appropriate standards and background information for the operator licensing written exam process.

Thus, the Panel recommends that a formal basis document be created for NRC operator exams. This basis document should establish the national standards on which the program is based. Furthermore, the basis document should specify the professional background in educational testing and measurement required for personnel that design and revise NRC exams. Additionally, the basis document should define the qualifications for a SME. Finally, the basis document should provide details on the process to determine the composition of the exams.

The Panel agrees with the DPO submitters that both the reduction in fundamental questions and the reduction in questions for the original 75 question RO exam should have used the previous methods as described in Appendix A of NUREG-1021 and the Standards for Educational and Psychological Testing. Additionally, the Panel agrees that the lower content validity directly leads to a conclusion that the RO exams ability to discriminate between a competent operator and one who is not, has been adversely affected.

The Panel notes that the historical pass/fail rates of the GFE have been consistent since 1989.

Approximately 4% of people that have taken the GFE have failed. The average score on the exam has also been consistent at 91%. There is no data on how many of the people who failed a GFE went on to retake the exam and pass. Under the NUREG-1021, Rev. 12, process, there will be candidates who could not achieve a 80% passing score on the previous GFE, who will now be eligible to take the RO exam. Some percentage of this group will pass the exam even though they may not have mastered the fundamentals. Thus, the Panel concludes that the changes to the RO exam has diminished the NRCs ability to make accurate licensing decisions for license applicants.

Issues 5 and 8 Issues 5 and 8 are related, because they both refer to the potential for adverse effects on the future performance of licensed operators due to having a less content valid exam. Specifically, issue 5 is holistically looking at the combined changes in Rev. 12. Besides changing the RO exam, Rev. 12 lowered the amount of power plant experience and education requirements for a Senior Reactor Operator (SRO) applicant to take the licensing exam. Issue 8 is asserting that lowering the standards for the written exam will lead to more plant events due to declining performance by operators.

The Panel finds that it is difficult to assess how the changes in testing generic fundamentals in NUREG-1021, Rev. 12, will impact the future performance of operators; however, the Panel notes that the following from NUREG-1021, Appendix A section D, The Importance of the Written Exam, supports the views of the DPO submitters: The importance of knowledge testing should not be underestimated, because knowledge is the underpinning of professional Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

14 performance. The objectives of knowledge testing are varied; they may include assessing fundamental understanding, as well as testing more advanced levels of expertise. The most effective tests of knowledge include questions and test items that measure the application of knowledge that directly relates to an individuals job. In the case of operator licensing, the NRCs written examination yields a key measure that allows the agency to make a confident decision about the safety-significant performance of the individual seeking a license.

IOLB staff, during interviews, indicated that they planned on performing an effectiveness review of the Rev. 12 changes, which will include a review of generic fundamental changes.

The Panel agrees generally that over time a decline in performance by operators could be a result of lowering the content validity of the exam; however, the Panel finds that it would be difficult to quantify the effects due to many variables associated with operator performance during plant events.

Issue 7 The DPO submitters expressed a concern in issue 7 that the process for changing the RO exam and eliminating the GFE should have been more extensive. They point out that the previous changes were made through TMI action items (NUREG 0737), Generic Letters, and NUREG/BR-0122.

The DPO Panel reviewed available documents to determine the process used to eliminate the GFE. The following paragraphs provide a summary of that process.

In 2016 as part of Project AIM, the NRC proposed to the industry to reduce the offering of the GFE from 4 times per year to 2 times per year. The following is from a public meeting summary (ML16112A349) of the March 31, 2016, meeting with the Nuclear Energy Institutes Licensed Operator Focus Group (NEI LOFG).

The NRC discussed with the NEI LOFG a proposal to reduce the number of offerings of the GFE from 4 per fiscal year to 2 per fiscal year by 2018. The LOFG proposed 3 GFE offerings in calendar year 2017, because narrowing down the number of offerings too quickly may cause the industry problems with scheduling applicants for the subsequent site-specific portion of NRC operator licensing examinations. The NRC staff said that 3 GFE in calendar year 2017 may be acceptable, with 2 GFE offerings in fiscal year 2018 and thereafter. The LOFG commented that letting the industry administer the GFE program or eliminating the GFE entirely could be possible options for the NRC to consider. The NRC staff responded that eliminating the GFE entirely would require a rule change and that this rule change would probably be low priority.

Thus, the staff position in 2016 was that eliminating the GFE would require a rule change. In March 2017 at a public meeting (ML17102B020), the NEILOFG proposed as an alternative to the industry developing 2 GFE exams per year, the idea of a GFE test that is electronic and is available on demand. The NRC responded that the staff initiated an internal NRC analysis of the suitability of a bank-only GFE. Specifically, the staff stated that this analysis was initiated to answer two fundamental questions:

1. Can a bank-only GFE maintain test validity, integrity, and reliability?

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

15

2. If so, what is the minimum number of questions needed in the GFE banks to properly discriminate between competent and non-competent individuals as a part of the NRCs operator licensing process?

The staff further informed the NEI LOFG that the target completion date for this analysis was December 2017, and that the staff would determine if or how to communicate the results of this analysis to the LOFG. The Division of Inspection and Regional Support (DIRS; the current title is Division of Reactor Oversight) sent a memo on September 14, 2017, to Human Resources Training and Development (HRTD) tasking them with a Training Assistance Request for generic fundamentals examination support.

This memo stated, During a Commission Meeting in January 2017, (ML17018A047), the Nuclear Energy Institute (NEI) Licensed Operator Focus Group (LOFG) stated their interest in making changes to the GFE program, such as pursuing a question bank-only GFE and/or a computer-based GFE. In a subsequent public meeting in March 2017 (ML17102B020), NEI LOFG further outlined their recommendations for an approach to making such changes. In response, the NRC staff stated that an evaluation of the feasibility of both a bank-only examination and a computer-based GFE administration process would be explored. The Office of Research (RES) has accepted the task to evaluate a potential transition to a question bank only GFE. RES has also accepted the task to evaluate the potential transition from a hard-copy to a computer-based GFE. This memorandum provides the request for an assessment of a potential transition to a computer based GFE using the CLE (Current Learning Environment) tool.

HRTD responded to the DIRS request with a memo dated December 13, 2017, that the Technical Training Center (TTC) staff feels that use of the CLE tool for future GFE development, administration, and maintenance of the exam bank can be an equitable in house replacement for the services that we currently receive from the GFE contractor with minimal changes to the current examination standards found in NUREG-1021, Revision 11.

Sometime in 2018, IOLB briefed NRR management on eliminating the standalone GFE and incorporating the fundamental topics into the existing RO exam. In December 2018, a working group was chartered to explore options for changing the GFE. An email was sent to all the operator licensing (OL) branch chiefs in the regional offices requesting volunteers for a working group on the GFE. The kickoff meeting was expected for mid-January 2019. A charter was developed with deliverables and a timeline. The deliverables were:

a report with recommendations that includes the identification of the three possible solutions, a technical and regulatory analysis of each solution, and the working groups recommended solution will be developed.

Once the working group has decided on the recommended solution, the team will develop an informational SECY paper describing the recommended solution and the other options considered.

A public meeting was held on February 13, 2019, and the following is IOLBs position about the working group. Due to feedback from the industry about the effect on initial operator training, the staff discussed the reduction from four to two Generic Fundamentals Examinations (GFEs) administered per year due to the NRCs Project Aim 2020. The staff has established a working group to explore other options that will increase efficiency while satisfying regulations. The working group is considering several options, including the authoring and administering the GFE Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

16 using NRC staff, extending or eliminating the GFE expiration date for those who have passed the examination, and re-integrating the GFE with the site-specific initial licensing written examination. The working group will evaluate each options cost to the industry and the NRC, exam scheduling impact, regulatory alignment, and assurance that only safe and competent operators receive licenses. The working group plans to make a recommendation to management within the next 3 months.

The Panel notes that the working group charter included a timeline with milestones. The working group was to select an option by April 26, 2019, and submit a SECY paper by September 30, 2019. The working group met weekly starting on January 8, 2019, until February 27, 2019. At this time, the working group members, including regional participants, were instructed to submit their preferred option by email in the next week. IOLB indicated that the timeline was accelerated for several reasons, one of which was that IOLB desired to brief DIRS management in March and brief regional Division Directors on the GFE proposal and options in April.

The Panel noted that the working group did not complete the deliverables as described in the charter. The working group did not prepare a written report that documented why the options were chosen, and there was no informational SECY paper describing the recommended solution and the other options considered. The Panel noted there was a CA Note dated August 9, 2019, (ML19176A287) that provided information to the Commission on the status of the GFE integration effort. Also, the Panel noted that the on-demand GFE that TTC was working on had support in the WG but was not included in options presented to the regional Division Directors.

Additionally, the working group leadership did not provide feedback to the members on their emailed preferred options described above.

A meeting was held on April 8, 2019, between Regional Directors/Deputy Directors and DIRS management. Three options were presented. Option 1 was integrating the GFE into the RO exam. Option 2 was integrating the GFE into the RO exam, with the difference between option 1 and 2 being which questions would be removed from the Revision 11 sample plan to accommodate the new questions. Option 3 was keeping the GFE and RO exam the same and evaluating potential future changes. Option 1 was the option chosen. The next major effort was a pilot of the new format, which was performed at 4 reactor sites.

IOLB provided the Panel a briefing document that was called GFE Key Messages. The title of the paper is, The Operator Licensing Generic Fundamental Exam Process Changes. The document stated, The staff created a working group to evaluate this problem and develop a solution. Possible solutions were developed with associated challenges and benefits, which were presented to the DIRS and regional management. A management consensus was reached to proceed with the integration solution. Within the document is the following statement, NRR management, licensees and the public have been kept informed of the process over the last couple of years; and the changes have enjoyed strong support by staff and industry. IOLB staff was asked how they gauged staff support. IOLB could not provide information on how they concluded there was strong staff report.

The Panel agrees with the DPO submitters that the GFE changes did not receive the level of review commensurate with a change of this magnitude. Although there was regional license examiner input into the working group, the working group members were not afforded an opportunity to formally review and comment on the options presented to regional management.

Regional operator licensing Branch Chiefs were also not offered an opportunity to review and comment on the GFE options. Additionally, the working group did not prepare a final report, which may have articulated a basis for the change.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

17 Overall, the Panel observed that the working groups activities were accelerated, and the group did not complete some of the key activities described in their charter. The working group did not afford all members an opportunity to review and comment on the options presented to management. Most notably, the working group did not provide sufficient opportunities for key internal stakeholders, including regional branch chiefs, chief examiners, and examiners to review and comment on the proposed changes.

Therefore, the Panel recommends that NRR evaluate the processes that NRR uses to obtain regional input into program office decision making. There should be procedures or policies to establish constructive working groups that help shape NRC program office decisions. The regions have a wealth of knowledgeable and experienced people who want to have input into program office decisions that affect their ability to perform inspections and licensing actions.

DPO Submitters Recommendations The DPO submitters made three recommendations to address their concerns. The first recommendation was adding a statistically significant number of generic fundamentals questions to the reactor operator written examination without diluting the other topics that were sampled in NUREG-1021, Rev. 11. Also, recommend grading the GFE and site-specific written examinations separately while requiring a minimum cut score of 80% to pass each portion of the examination, as has been required prior to NUREG-1021 Rev. 12. The mix of questions should be approximately the same percentages as was determined by job/task analysis for the handbook discussed above. If the length of the written examination given on one-day is of concern, the senior reactor operator and reactor operator examinations could be given on different days. The different sections of the operating test are given over a one to two-week period.

The second recommendation was another option would be to give a GFE as a separate section of the written examination any time within thirty days of the start of the operator licensing examination while keeping the balance of the written examination the same as it is in NUREG-1021, Revision 11. This would be no different than administering and grading the different sections of the operating test during this thirty-day period as is currently done in NUREG-1021 Revision 11. As stated in the previous paragraph, the GFE and site-specific portions of the written examinations should be graded separately using the same 80% cut score for each section, so as not to dilute the site-specific written examination requirements or the GFE requirements.

The third option would be to specify the content of a GFE examination and allow the licensees to administer and grade the GFE as part of the eligibility and application process. This would likely require some changes to 10 CFR 55.

The DPO submitters conclusion on the three options was as follows: the options listed above would ensure that applicants are at risk of being tested on a meaningful number and variety of generic fundamentals topics during either the application or operator licensing examination process and prevent the simplistic testing of generic fundamentals in NUREG-1021 Rev. 12 that would lead to a reduction in the amount of training and study and an eventual degradation of performance as is described in Appendix B of NUREG-1021 Rev. 11.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

18 The Panel determined that there are many ways to address the reduction in fundamental questions. Any of the above recommendations would have merit and could be considered.

However, the Panel determined that it is appropriate to develop a basis document for the operator license examination process first. The Panel does not have the requisite experience in educational testing and measurement, and therefore choosing a method to resolve the DPO submitters concerns should be left to experts in the field and SMEs.

Working Group Timeline and Structure The working group charter had an aggressive timeline in the opinion of the Panel. However, the charter was thorough and identified deliverables for review that would have given the members opportunities to seek input from their represented groups and express their views. The Panel was not able to determine some of the reasons for the abbreviated working group timeline, but when the working groups activities ended abruptly, the regional input to the process was ended.

During interviews, the Panel learned that several regional staff members expressed concerns with the proposed GFE changes to their respective branch chiefs. Some of the branch chiefs had conversations with IOLB, but they were informed that the decision had been made and these conversations did not lead to IOLB reconsidering their position.

The Panel noted that the GFE working group did not establish clear expectations on communications with, and seeking input from, all appropriate internal stakeholders. Thus, some regional staff interviewed by the Panel stated they did not have an opportunity to comment on the proposed GFE changes or were unaware of the developments. Accordingly, they did not have clear opportunities to express their views or exercise the non-concurrence process. The working group did not fully evaluate input from the regions wealth of knowledgeable and experienced staff. The Panel recommends that a more formal process be developed to obtain regional input into program office decisions. This could be a best practices document that provides a standard framework for working groups. This document should include best practices for communications, seeking input, and resolving differing views within the process.

Safety and Risk Significance The Panel interviewed 45 staff members and many of them stated that NUREG-1021, Rev. 12 does not adequately test all the GFE topics and believe that there is a tie to safety. The Panel asked for input from Senior Risk Analysts on the risk insights due to operator errors. The Senior Risk Analysts stated that the Probabilistic Risk Assessment (PRA) models do NOT model errors of commission. Therefore, it is not possible to calculate an extrapolation of these errors on impact to Core Damage Frequency under other conditions which could occur, and operator decision making and actions. Human errors have the potential to increase Human Error Probability (HEP) for credited operator actions. The PRA models are highly sensitive to small variations in HEP changes. Human performance is a very large contributor to PRA uncertainty.

Panel Findings The Panel finds that the NUREG-1021 Operator Licensing Examination Standards for Power Reactors Rev. 12 method of testing Reactor Theory, Thermodynamics, and Components, in Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

19 the Reactor Operator (RO) licensing exam, does not determine if the applicant has mastered this subject matter. Since the purpose of a licensing exam is to determine if an applicant has mastered the subject matter, or not, and then to base the licensing decision on the exam score, the changes to the RO exam has diminished the NRCs ability to make accurate licensing decisions for license applicants.

The panel finds that reducing the number of generic fundamental questions, has lowered the content validity of the exam. The low content validity of the exam for the generic fundamentals directly leads to the above finding that the licensing decision was adversely affected.

The Panel finds that the RO exam does meet 10CFR 55.41, Written Examination: Operators.

The regulation requires that the written exam for an operator of a facility will include a representative sample from 14 items to the extent applicable to the facility. Reactor Theory, Thermodynamics and Components are among the 14 items. Since NUREG-1021, Rev. 12 exams do sample the three subjects, we concluded that the regulation is met.

Panel Recommendations Recommendation 1:

Establish a basis document for NRC operator exams. This basis document should include the national standards on which the program is based. It should specify the professional background in educational testing and measurement that would be necessary for personnel who design and revise NRC exams. Additionally, the basis document should define the qualifications for a Subject Matter Expert (SME) involved in the NRC exam process. Finally, the basis document should detail the process for determining the composition of the exams.

Recommendation 2:

Once the resources are in place and there is agreement on the basis of the program, a working group with subject matter experts and representation from all four regions should revisit the RO written exam and evaluate all the key parameters, including the length of the exam, cut scores, weighting of different topics and whether there should be a separate generic fundamentals exam.

Recommendation 3:

The Panel recommends that a more formal process be developed to obtain regional input into program office decisions. This could be a best practices document that provides a standard framework for working groups. This document should include practices for communications, seeking input, and resolving differing views within the process.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

A-1 Appendix A - Documents Reviewed DPO-2021-002 NRC DOCUMENTS Differing Professional Opinion (DPO) No. 2021-002 Differing Professional Opinion Associated with The Operator Licensing Examination Standards for Power Reactors, NUREG-1021, Revision 12, September 27, 2021, Accession Nos. ML21322A200 and ML21322A199.

Code of Federal Regulations (CFR) Title 10, Energy, Part 55 Operators Licenses, Subpart E, Written Examinations and Operating Tests, December 30, 2021.

The Operator Licensing Examination Standards for Power Reactors, NUREG-1021 Revision 12, September 2021, Accession No., ML21256A276.

The Operator Licensing Examination Standards for Power Reactors, NUREG-1021 Revision 11, February 2017, Accession No., ML17038A432, February 2017 The Operator Licensing Examination Standards for Power Reactors, NUREG-1021 Revision 6, June 1990, Accession No., ML14346A092.

The Operator Licensing Examination Standards for Power Reactors, NUREG-1021 Revision 5, January 1989, Accession No., ML20235R399.

Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water, September 2020, Reactors, NUREG-1122, Revision 3, ML20260H083.

Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors, September 2020, NUREG-1123, Revision 3, ML20260H086.

SECY 79-330E Qualification of Reactor Operators. July 30, 1979, Accession No. ML12236A763.

SECY-89-292 Final Report on NRC Pilot Testing of The Generic Fundamentals Examination Section (GFES) of The Operator Licensing Written Examination, September 18, 1989, Accession No. ML12251A650.

NUREG/BR-0122 Revision 4, Examiners' Handbook for Developing Operator Licensing Examinations, January 1988, Accession No. ML20117N069.

NUREG-0737, Clarification of TMI-2 Action Plan Requirements, Accession No. ML051400209.

Summary of the March 31, 2016, Public Meeting with The Nuclear Energy Institutes Licensed Operator Focus Group, May 4, 2016, Accession No. ML16112A349.

Summary of the March 23, 2017, Public Meeting with Industry Focus Group on Licensed Operator Topics, April 19, 2017, Accession No. ML17102B020.

Summary of the January 24, 2018, Public Meeting with Industry Focus Group on Licensed Operator Topics, February 21, 2018, Accession No. ML18046A122.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

A-2 Summary of the February 13, 2019, Public Meeting with Industry Operator Licensing Representatives, March 4, 2019, Accession No. ML19057A360.

Note to Commissioners Assistants dated August 9, 2019, EFFORT TO EVALUATE THE INTEGRATION OF THE GENERIC FUNDAMENTALS EXAMINATION INTO THE REACTOR OPERATOR INITIAL LICENSING WRITTEN EXAMINATION, Accession No. ML19176A287.

Note to Commissioners Assistants dated July 28, 2020, UPDATE ON THE INTEGRATION OF THE GENERIC FUNDAMENTALS EXAMINATION INTO THE REACTOR OPERATOR INITIAL LICENSING WRITTEN EXAMINATION, Accession No. ML21119A862.

GFE Key Messages. The Operator Licensing Generic Fundamental Exam Process Changes.

OTHER DOCUMENTS Standards for Educational and Psychological Testing, American Educational Research Association,1985 and 2014 editions.

ACAD 10-001 Revision 2, Guidelines for Initial Training and Qualification of Licensed Operators, May 31, 2022, Accession No. ML21210A294 NEI Letter, Generic Fundamentals Reintegration, dated March 4, 2020.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

B-1 Appendix B - NRC Staff Interviewed DPO-2021-002 To conduct its review, the Panel developed a list of staff to interview and solicited input from the DPO submitters on additional staff to interview. Through the course of the interviews, additional staff were identified given their potential role and knowledge of the subject associated with the DPO.

The Panel interviewed the following staff:

Region I:

Joe DeMarshall, Senior Operations Engineer Brian Dyke, Operations Engineer Sarah Elkhiamy, Operations Engineer Chris Lally, Senior Operations Engineer Kevin Murphy, Operations Engineer Pete Ott, Operations Engineer Tom Setzer, Senior Operations Engineer Region II:

Gerald McCoy, RII/DRS/OB1 Branch Chief Eugene Guthrie, RII/DRS/OB2, Branch Chief Daniel Bacon, Senior Operations Engineer Mark Bates, Senior Operations Engineer Bruno Caballero, Senior Operations Engineer Andreas Goldau, Operations Engineer Michael Kennard, Senior Operations Engineer Kevin Kirchbaum, Operations Engineer Newton Lacy, Operations Engineer David Lanyi, Senior Operations Engineer Michael Meeks, Senior Operations Engineer Joseph Viera, Senior Operations Engineer Region III:

Patricia Pelke, OL Branch Chief Gregory Roach, Senior Operations Engineer Charles Zoia, Senior Operations Engineer Lionel Rodriguez, Operations Engineer James Nance, Operations Engineer Randal Baker, Senior Operations Engineer John Robbins, Operations Engineer Theodore Wingfield, Operations Engineer Bruce Bartlett, Senior Operations Engineer Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information Region IV:

Greg Werner, R-IV/DORS/PBB Branch Chief (OL Branch Chief 2019)

Clyde Osterholtz, Senior Operations Engineer Matt Do le, O erations En ineer Russell Williams, Operations Engineer NRR/DRO/IOLB:

Chris Miller, NRR/DRO Division Director Lauren Nist, IOLB Branch Chief OGC:

Kevin Roach, Senior Attorney Jeremy Wachutka, Senior Attorney TIC:

Matthew Emrich, Reactor Technology Training Branch, Branch Chief Gary Callaway, Senior Reactor Technology Instructor B-2 Official Use Only - Sensitive Internal Information

Document 4: DPO Decision

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 MEMORANDUM TO:

Daniel Bacon, Senior Operations Engineer Joseph Demarshall, Senior Operations Engineer Kevin Murphy, Operations Engineer Peter Ott, Operations Engineer Randy Baker, Senior Operations Engineer Chuck Zoia, Senior Operations Engineer Bryan Bergeon, Operations Engineer Mark Bates, Senior Operations Engineer Michael Donithan, Operations Engineer Andreas Goldau, Operations Engineer Michael Kennard, Senior Operations Engineer Kevin Kirchbaum, Operations Engineer Newton Lacy, Operations Engineer David Lanyi, Senior Operations Engineer Michael Meeks, Senior Operations Engineer Joseph Viera, Senior Operations Engineer FROM:

Andrea D. Veil, Director Office of Nuclear Reactor Regulation

SUBJECT:

DIFFERING PROFFESIONAL OPINION DECISION ON THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

The purpose of the memorandum is to respond to your differing professional opinion (DPO) submitted on September 27, 2021, in accordance with Management Directive 10.159, The Nuclear Regulatory Commission [NRC] Differing Professional Opinions Program (Agencywide Documents Access and Management System (ADAMS) Accession No. ML18073A298). Your DPO, titled Operator Licensing Generic Fundamentals Examination (ADAMS Accession Nos.

ML21322A200 and ML21322A199), expressed concerns with changes to the composition of the licensed operator initial exam as issued in NUREG 1021, Operator Licensing Examination Standards for Power Reactors, Revision 12 (ADAMS Accession No. ML21256A276).

Specifically, you raised concerns that NUREG 1021, Revision 12 has eliminated the standalone 50 question Generic Fundamentals Exam (GFE) that required a minimum passing score of 80%

and reduced the required testing of generic fundamentals in the new integrated operator licensing exam to as few as 6 to 8 questions, with no required passing score for that section of the exam. With this change, you raised the concern that the NRCs ability to discriminate between an applicant that has adequate knowledge of generic fundamentals and one that does not, has been adversely impacted.

CONTACT: Diana Betancourt-Roldan, RIII 630-829-9507 May 20, 2022 Signed by Veil, Andrea on 05/20/22 Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information I found your DPO to be carefully researched and of high quality. I commend you for your commitment and dedication to the Nuclear Regulatory Commissions mission. Your willingness to raise concerns with your colleagues and managers and ensure that your concerns are heard and understood is admirable and vital to ensuring a healthy safety culture within the agency.

My response to the DPO is described in the enclosure.

Enclosures:

1.

Directors Decision for Differing Professional Opinion 2.

Attachment:

DPO Submitters Organizations Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Enclosure DIRECTORS DECISION FOR DIFFERING PROFESSIONAL OPINION (DPO) ON THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

Background

In DPO-2021-02, titled Operator Licensing Generic Fundamentals Examination (ADAMS Accession Nos. ML21322A200 and ML21322A199), concerns were raised regarding changes to the composition of the licensed operator initial exam as issued in NUREG 1021, Operator Licensing Examination Standards for Power Reactors, Revision 12 (ADAMS Accession No. ML21256A276). Specifically, the DPO asserts that NUREG 1021, Revision 12 has eliminated the standalone 50 question Generic Fundamentals Exam (GFE) that required a minimum passing score of 80% and reduced the required testing of generic fundamentals in the new integrated operator licensing exam to as few as 6 to 8 questions, with no required passing score for that section of the exam. With this change, the concern was raised that the NRCs ability to discriminate between an applicant that has adequate knowledge of generic fundamentals and one that does not, has been adversely impacted.

The DPO Ad Hoc Review Panel (the Panel) issued their report to me on April 5, 2022, after reviewing the applicable documents, conducting internal interviews with relevant individuals, and completing their deliberations (ADAMS Accession No. ML22095A246). I discussed the Panel Report with DPO Panel members on May 2, 2022.

To inform my decision regarding this DPO, I reviewed the submittal, the Panels report, NUREG/BR-0122, Examiners Handbook for Developing Operator Licensing Written Examinations, Revision 5 (ADAMS Accession No. ML090910709), and NUREG 1021, The Operator Licensing Examination Standards for Power Reactors, Revision 9 (ADAMS Accession No. ML0423320438). I also considered additional pertinent guidance and staff positions and information provided by subject matter experts. Further, I assigned an independent staff member to assist in my evaluation and the documentation of my decision which revealed additional pertinent information not included in the Panels report.

Summary of Issues The Panel agreed on a summary of issues identified in the DPO that accurately addressed the DPO submitters (the submitters) concerns. This summary of issues can be found on pages 1 and 2 of the Panels report. The Panel identified that the eight identified concerns could be grouped into four distinct areas for evaluation. My assessment of the Panel conclusions will follow the format delineated by the Panel.

My Assessment of the Panel Conclusions The Panel performed a thorough review of the DPO, and related technical areas. Their report provided substantial background regarding the origin and purpose of the reactor operator licensing exams. The report was well written and provided thoughtful conclusions from the Panel. I appreciate the Panels thoughtful assessment of the concerns raised in the DPO. The next several paragraphs, outline the basis for my decisions, and delineate where my conclusions support or differ from the Panels findings.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information Issues 1 and 2 As expressed by the Panel, these concerns are associated with historical reasons which the submitters believe support their position that, the operator licensing exam testing of generic fundamentals should not have been significantly reduced. In their report, the Panel agreed with the submitters that the Three Mile Island (TMI) corrective actions should weigh heavily when assessing the importance of operators knowledge of fundamentals. The Panel also agreed with the importance of applicants demonstrating their knowledge of fundamental concepts. The Panel noted that this could be achieved by either having a separate score for fundamentals, or by having a statistically large enough number of fundamental questions, such that an applicant could not pass the exam without mastering the fundamental topics.

I agree with the Panels assertion that, the TMI corrective actions should be taken into consideration when assessing the importance of operators knowledge of fundamentals. There were many lessons learned from the TMI accident. These lessons brought sweeping changes involving areas like emergency response planning, reactor operator training, human factors, and many other areas of nuclear power operations. It also caused the NRC to strengthen its regulatory oversight. The impact of the TMI lessons learned on our inspectors and technical staff cannot be understated; it is still felt today in their attention to detail and how they conduct their everyday work. As part of their review, the Panel surveyed 37 regional members, 41% of which were the submitters. The Panel noted that 80% of regional staff interviewed expressed support for the views of the submitters, in that the importance of testing fundamentals warranted more questions than three questions in Reactor Theory and three questions in Thermodynamics. Most of the staff in the surveyed group referenced the TMI accident and related lessons learned as informing their decision. However, it is my assessment that although the number of questions testing fundamentals was reduced, reactor theory and thermodynamics are still being tested. By still testing these areas, we are taking into consideration lessons learned from TMI and the requirements delineated in 10 CFR 55.41, Written Examination:

Operators. The key consideration is how much weight should be put on testing fundamentals.

This was part of the staffs assessment supporting reducing the number of questions focused on fundamentals. Although not mentioned in the Panels report, to arrive at their decision to reintegrate the GFE exam into the reactor operator exam, the staff relied on the judgment of qualified expert examiners, the use of risk insights, and review and consideration of available data, such as the results of an industry-led pilot. This is in line with how the agency implemented changes in 2004 that involved reducing the number of questions in the GFE from 100 to 50 questions. Additionally, although not mentioned in the Panels report, the staff utilized NUREG/BR-0122, Examiners Handbook for developing Operator Licensing Written Examinations Revision 5 to inform their work. My evaluation of the methodology used in evaluating the changes to the GFE is covered more fully in my response to Issues 3, 4, and 6.

I disagree with the Panels implied assertion that testing for required knowledge could only be achieved by either having a separate score for fundamentals, or by having a statistically large enough number of fundamental questions, such that an applicant could not pass the exam without mastering the fundamental topics. While I agree that an applicant cannot fail the operator licensing exam based solely on the questions dedicated to testing fundamentals, the same can be stated for other safety significant topics that we have determined to be important enough to test. Although not mentioned in the Panels report, as part of the assessment the staff considered that knowledge of reactor theory and thermodynamics is also implicitly tested through other questions on the written exam, particularly questions that require applicants to assess plant conditions to determine the appropriate response actions. Without the proper knowledge of reactor theory and thermodynamics, the applicant would be unable to properly diagnose these events and it would be more difficult for an individual to be able to pass the Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information examination. Finally, during my discussion with the Panel, I asked for clarification on whether the Panel considered the contribution of the other exam questions in determining if the examinee had the required knowledge of fundamentals, the Panel stated that they only evaluated the two proposed avenues that had been suggested by the submitters. I agree with the staffs assessment that fundamentals are inherently tested, to varying degrees, in other exam questions and was appropriately considered.

Issues 3, 4, and 6 The concern described in Issue 3 relates to the submitters belief that Revision 12 of NUREG 1021 deemphasizes the importance of generic fundamentals to the point that, it drastically reduces the ability of the operator licensing exam to discriminate between an applicant who has adequate knowledge of generic fundamentals, and one who does not. Issue 4 expresses the submitters concern that deleting 6 to 10 questions from the NUREG 1021, Revision 11 sample plan to accommodate fundamentals questions reduced the ability to discriminate between an applicant who has adequate knowledge of systems and procedures, and one who does not.

The concerns expressed in Issue 6 involved the submitters belief that the method used to determine the reductions in the number of questions did not follow NRC best practices, did not utilize a professional education services company, or establish a basis for the change. This included an assertion that the information provided in NUREG/BR-0122, Revision 4 (ADAMS Accession No. ML20117N069) was not utilized.

The Panel concluded that the logic used by the staff to make changes to the operator licensing exam in Revision 12 of NUREG 1021 was not appropriate. The Panel asserts that by reducing the number of total questions from 125 (GFE and RO exam under Revision 11) to 75 (Revision 12), without using subject matter experts, the content validity of the exam has been reduced.

Additionally, it was the Panels opinion that a low number of fundamental questions is a construct underrepresentation, which implies a narrowed meaning of test scores because the test does not adequately sample fundamentals.

The Panel also agreed with the submitters that both the reduction in fundamental questions and the reduction in questions for the original 75 question reactor operator exam should have used the methods as described in Appendix A of NUREG 1021, and the Educational and Psychological Testing, Revision 5 standards referenced in the NUREG. Additionally, the Panel agreed that the lower content validity directly leads to a conclusion that the operator licensing exams ability to discriminate between a competent operator and one who is not, has been adversely affected.

Although the Panel agreed with many of the submitters concerns, the Panel concluded that the operator licensing exam meets 10 CFR 55.41, Written Examination: Operators. The regulation requires that the written exam for an operator of a facility includes a representative sample from 14 items to the extent applicable to the facility. Since NUREG 1021, Revision 12 does sample the three areas that were subject of the submitters DPO (Reactor Theory and Thermodynamics and Components) the Panel concluded that there was a representative sample, as required by the regulation.

I agree with the Panels conclusion that the changes to the operators exam implemented in NUREG 1021, Revision 12 meet the requirements delineated in 10 CFR 55.41. Specifically, the staff included a representative sample of all 14 items required by the regulation in the sample plan. However, I disagree with the Panels conclusion that the logic used by the staff to reintegrate the GFE into the operator licensing exam was not appropriate. The NRCs examinations are not intended to distinguish among levels of competency or to identify the most Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information qualified individual but to make reliable and valid distinctions at the minimum level of competency that the agency has determined is necessary for the protection of public health and safety. My assessment is that the Panel did not provide sufficient objective evidence to show that reducing the number of questions by default means that the content validity of the examination has been reduced. I based my conclusions on the information presented below.

The methodology used by the NRC staff relied on: 1) the professional judgment of qualified expert examiners (subject matter experts); 2) risk insights to evaluate how much emphasis should be given to testing fundamentals topics such as reactor theory and thermodynamics and in the development of the sample plan; and 3) the review and consideration of available data, such as an industry-led pilot. Although not mentioned in the Panels report, the GFE reintegration concept was also evaluated by four facilities [2 Boiling Water Reactors (BWRs),

and two Pressurized Water Reactors (PWRs) owned by different utilities] that developed questions based on theory and component topics selected by NRC staff (ADAMS Accession No. ML20083F400). As stated in the Panels report, the NRC staff had assessed that the composition of the GFE and the operator licensing examination was oversampling in the area of fundamentals. This was in part based on the GFE examination as it existed in Revision 11 of NUREG 1021. The Revision 11 GFE examination included 33% of all written examination test items for senior reactor operators and 40% of all written examination test items for reactor operators, indicating that fundamentals were being oversampled relative to the 14 items required by regulation. This is a critical point, supported by data and a thorough examination of test questions prior to finalizing the development of Revision 12. It also supports the fact that the staff did not arbitrarily eliminate test items just to reduce the number of questions. As stated in the Panels report, the relative safety significance assigned to each of the required topics to be tested are based on subject matter expert opinion. The staff used the methodology described in the Panels report (pages 10-12), to risk-inform the importance to safety of all topics tested in the operator examination. Additionally, they used their experience as expert qualified licensed examiners (subject matter experts) to evaluate whether all topics were being tested consistent with their contribution to safety and their protection function. It was from this assessment that the staff concluded that, although the reactor theory and thermodynamics topics have a similar importance to safety to other topics, they were being tested 3-5 times more frequently. Although not mentioned in the Panels report, another factor that was taken into consideration for reintegrating the GFE into the operator licensing exam was that the GFE examination as administered until Revision 12 was focused on more theoretical plant configurations rather than plant specific configurations. This in combination with fundamentals being tested 18 months prior to the time of licensing allowed for possible unidentified degradation in concept knowledge and ability.

Regarding the content validity of the exam. As described in the Panels report, Appendix A of NUREG 1021 establishes the method to establish content validity of NRC exams. As stated in Appendix A:

The first step in the sequence of activities was to establish a link to job duties. A job task analysis (JTA) was performed by the Institute of Nuclear Power Operations (INPO) which identified approximately 28,000 knowledge and abilities (K/As).

As part of my review, I conferred with staff members to understand the rationale of not performing a new JTA for the reintegration effort. Although not mentioned in the Panels report, the staff based their determination that a new JTA was not needed due to their assessment that the original JTA was still valid. As stated in the Panels report, the original JTA was developed in the 1980s and was the original basis for the K/As. In Revision 12 of NUREG 1021, the staff did not change what areas are tested; rather the change involved how much sampling was Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information performed. Therefore, it was the staffs conclusion that the original JTA was still valid. This approach was similar to what was done in 2004 when the GFE test was changed in Revision 11 of NUREG 1021, as no new JTA was generated for the change in sampling.

The next step specified by Appendix A of NUREG 1021 is to develop a sample plan. As mentioned in the Panels report, the sample plan is an important part of making validity inference. Although not mentioned in the Panels report, when developing the sample plan the staff utilized NUREG/BR-0122, Examiners Handbook for Developing Operator Licensing Written Examinations, Revision 5. The staff used this document to determine the original bases for the written examination sample plan, and to assist them in determining potential impacts of the proposed changes to incorporate the reactor theory/thermodynamics topics. It was from this review that the staff developed a risk-informed approach for adding six theory and thermodynamics questions by removing questions from other parts of the examination. This was done in a way that balanced the question sampling to ensure that adequate coverage of safety-significant topics was retained. As mentioned by the submitters, no professional services company was used to make changes to the sampling plan delineated above. However, during my review I found that there is no requirement to use a professional educational services company for such changes. The information delineated above allowed me to determine that the content validity of the exam was not affected as it still meets the instructions and guidance contained under Content Validity in Appendix A of NUREG 1021.

The NRC is a learning and evidence-based organization. The NRC Principles of Good Regulation have led the agency in making regulatory decisions using available data. Although not mentioned in the Panels report, the GFE integration concept was evaluated by four facilities that developed questions based on theory and components topics selected by the NRC through an industry led pilot. The pilot included the development of eight plant specific questions using draft NRC guidance that mimic the new proposed testing methodology. The facilities then administered these questions to applicants in approximately the same period as the facilitys practice examination. The applicants were not given the opportunity to study the theory and components topics prior to question administration, nor where they given prior notification of the intent to test these areas. The facilities assessed the applicants performance on each question and determined the impact on the overall examination performance by replacing selected questions from the practice examination with these theory and components questions. The results were the following:

Average score on pilot questions Impact on audit examination if pilot questions used1 Station 1 85.7

+0.6 Station 2 81.8

-1.2 Station 3 82.1

-0.4 Station 4 58.3

-3.8 Based on the results of the industry pilot presented above, the changes in the sample plan produced slightly lower scores in three of the four facilities. The staff assessed that at least in one facility (Station 4) there was a need to better maintain theory knowledge throughout 1 The impact on audit examinations in this column refers to the change in average passing score upon substituting selected questions from the practice exam with the audit questions.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information training. This showed unidentified degradation in concept knowledge and ability that would likely not have been identified under Revision 11 of NUREG 1021. The staff took into consideration the results of the industry led pilot delineated above to assess the impact of the proposed changes to the operator licensing exam.

In summary, I found that reducing the number of generic fundamental questions has not adversely affected the NRCs ability to make accurate licensing decisions for operator license applicants. I also conclude that the operator licensing examinations ability to discriminate between a competent operator and one who is not, has not been adversely affected. This is based on my review of the three elements of the methodology that the staff relied on when performing the reintegration of the GFE into the operator licensing exam. While I am confident that the Office of Nuclear Reactor Regulation (NRR) staffs approach was valid, I also found that the basis for making changes to NUREG 1021 was not sufficiently documented by the staff.

Therefore, I am tasking the Division of Reactor Oversight (DRO) to document the basis for integrating the GFE examination into the reactor operator examination. This should include the methodology used to develop the new sampling plan, and the basis for determining that the operator licensing examinations ability to discriminate between a competent operator and one who is not, has not been adversely affected. A response by DRO regarding these recommendations will be tasked for completion by September 30, 2022.

Issues 5 and 8 These concerns are related to the potential adverse effects on the future performance of licensed operators due to having a less content valid exam. The Panel agreed with the submitters position that a decline in operator performance could result over time from lowering the content validity of the reactor operator exam. Importantly, the Panel also concluded that it would be difficult to quantify the effects due to many variables associated with operator performance during plant events. The basis for the Panels conclusion was statements included in NUREG 1021, Appendix A, Section D, The Importance of the Written Examination, which states:

The importance of knowledge testing should not be underestimated, because knowledge is the underpinning of professional performance. The objectives of knowledge testing are varied: they may include assessing fundamental understanding, as well as testing more advanced level of expertise. The most effective tests of knowledge include questions and test items that measure the application of knowledge that directly relates to an individuals job. In the case of operator licensing, the NRCs written examination yields a key measure that allows the agency to make a confident decision about the safety-significant performance of the individual seeking a license.

I agree with the Panel that it is difficult to assess how the changes in testing generic fundamentals in NUREG 1021, Revision 12, will impact the future performance of operators.

However, as stated in my response to Issues 3, 4, and 6, I found insufficient evidence to state that the new lower number of fundamentals questions results in the content validity of the reactor operator examination being reduced to the point of not ensuring that operators have mastered the skills necessary to safely operate the plant.

Regarding the long-term impact of the changes to NUREG 1021, Revision 12. The NRC does not rely on a one-time examination to maintain proper knowledge on how to respond to emergencies. In accordance with 10 CFR 55.59 (a), Requalification requirements, each Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information licensed operator must pass a comprehensive (biennial) written examination and an annual operating test. This allows the NRC to have continued assurance that the operators have the proper knowledge to respond to emergencies. Additionally, the requalification standards implemented by the licensees contain expectations to include refresher training on fundamental topics, such that, among other things, it increases the level of understanding of applied fundamentals presented in initial training.

Moreover, one strength of the NRC regulatory framework is that it provides for needed oversight through inspection. Inspection Procedure (IP) 71111.11, License Operator Requalification Program and Licensed Operator Performance, provides for quarterly samples involving licensed operator performance in the control room during periods of heightened activity or risk.

The procedure provides guidance on evaluating diverse aspects of operator performance which include, in part: operator compliance and use of plant procedures; use and interpretation of plant instruments, indications, and alarms; and diagnosis of plant conditions based on instruments, indications, and alarms diagnosis. Additionally, as part of ongoing oversight, the NRC analyzes licensee event reports, inspection findings, and industry databases to determine whether adverse trends in performance exist. The combination of these activities would allow the NRC to determine if additional actions are warranted to mitigate a negative trend in operator performance, and to ensure continued safe operation. I am confident that the ongoing implementation of the Reactor Oversight Process will continue to ensure that reactor operators safely operate their respective nuclear plants.

As part of being a learning organization, NRR performs periodic reviews of certain programs.

As such, DRO has an on ongoing effort to perform an Effectiveness Review for Revision 12 of NUREG 1021. This Effectiveness Review was referenced in the Panels report and will review the changes made in the latest revision. One of the items identified for review is to: Verify that generic fundamental knowledge is being maintained through initial and continuing learning programs. This Effectiveness Review is currently in the approval process and was provided to regional branch chiefs for review and comment. The planned review will cover a period ranging from April 2022 and extend at least through April 2024. Therefore, I determined that a new tasking was unnecessary.

Issue 7 This concern was related to the submitters assertion that the process used for incorporating the GFE into the operator licensing exam appeared to have been approved/reviewed by a truncated review process. In their report, the Panel agreed with the submitters in that the incorporation of the GFE to the operator licensing examination did not receive a level of review commensurate with the implemented change. This conclusion was reached based on regional working group members, regional Branch Chiefs, chief examiners, and examiners, not being afforded an opportunity to formally review and comment on the GFE options presented to regional management. Additionally, the Panel referenced the working group not preparing a final report, which may have articulated the basis for the change, as part of their decision. Moreover, the Panel noted that the GFE working group did not establish clear expectations on communications with, and seeking input from, all appropriate stakeholders.

I disagree with the assertion that the GFE change was not thoroughly reviewed by the agency prior to issuance. As mentioned in the Panels report, changes to the GFE have been discussed internally and with the industry as early as 2016. Summaries of four public meetings between the NRC and industry representatives between March 2016 and February 2019 detail ongoing conversations on potential GFE changes. The proposed changes evolved due to internal and external stakeholder input. All of the summaries included regional Branch Chiefs or Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information their representatives in the distribution and would allow for awareness on the topics that were discussed. I would like to note an important clarification/correction to the Panels report. The report stated that the staffs position in 2016 was that eliminating the GFE would require a rule change. The statement references the complete elimination of testing generic fundamentals.

As the change in NUREG 1021, Revision 12 was the integration of fundamental reactor theory and thermodynamics into the facility-specific operator licensing exam it would not require a change to 10 CFR 55.41. The Commission was kept appraised of the proposed changes to NUREG 1021 using an internal process (Note to Commissioner Assistants; ADAMS Accession Nos ML19176A287 and ML21119A862). I also note that the changes to NUREG 1021 were subject to a review and concurrence process that included all four Regional Offices, and NRR, as well as the issuance of a draft for public comment. Additionally, the GFE integration concept was evaluated by four facilities that developed questions based on theory and component topics selected by the NRC staff. Based on the above discussion, I concluded that the changes to NUREG 1021, Revision 12 were reviewed and approved in accordance with NRC processes and not by a truncated review.

I agree with the Panel that in accordance with NRC best practices, the working group members should have been allowed to formally review and comment on what was presented to their management. However, although not mentioned in the Panels report, a discussion with the working group lead revealed that the working group members were requested to share their views on the proposed options with their regional management prior to the decision-making meeting between Regional Directors/Deputy Directors and the Division of Inspection and Regional Support (DIRS; the previous title of the Division of Reactor Oversight). This request was confirmed by a regional member of the working group. In addition, I reviewed a briefing document called GFE Key messages referenced by the Panels report. The document contained 11 concerns raised by internal stakeholders and how they were addressed. Two of the concerns listed where:

Because the number of theory and basic component questions is decreased, this integrated approach may no longer adequately test these concepts.

An applicant can fail all theory questions and still be licensed.

The above-mentioned concerns are part of the concerns discussed in the DPO submittal.

Based on the above, I conclude that although the working group members were not provided the opportunity to formally review and comment on what was presented to their management, they were encouraged to share their views in an informal manner with their management prior to their decision. Additionally, as evidenced by the information presented in the briefing document the internal concerns were documented for management review. This provided an opportunity for working group members to express their views.

I agree with the Panels conclusion that the working group did not establish clear expectations on communications. The Panel report includes multiple examples where communications with working group members regarding changes to the working groups scope, and deliverables lacked clarity and transparency. Although it is not uncommon for the scope of a working group and the deliverables to change, working group members should be kept informed of activities which impact the scope of their work. As part of the agencys values, we seek diverse views and promote open communications that maximize and improve our regulatory decision-making.

I agree with the Panel that a best practices document that provides a standard framework for working groups is one way to improve communication and governance. It may also clarify scope and content of products as well as roles and responsibilities of working group members and their management before a formal review is finalized. For this reason, I am tasking DRO to Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information develop a best practices document to provide additional clarity, and an operating framework for working groups. This will include communications expectations for both NRR and regional participants (both staff and management), and best practices for seeking input from diverse groups and addressing different opinions. A response from DRO regarding these recommendations will be tasked for completion by September 30, 2022.

Response to Recommendations Panel Recommendation 1 (Issues 3, 4 & 6): Establish a basis document for NRC operator exams. This basis document should include the national standards on which the program is based. It should specify the professional background in educational testing and measurement that would be necessary for personnel who design and revise NRC exams. Additionally, the basis document should define the qualifications for a Subject Matter Expert (SME) involved in the NRC exam process. Finally, the basis document should detail the process for determining the composition of the exam.

I disagree with Recommendation 1. Part of the basis for this recommendation as expressed in the Panels report stem from the following:

The Panel notes that issue 6 specifically mentions that NUREG 1021, Rev.12 did not utilize the information in NUREG/BR-0122 Revision 4, a professional educational services company was not consulted, nor were other supporting bases used to establish the generic fundamentals sample size for the [reactor operator] exam. The Panel considers this issue to be a key statement. There are a number of historical documents that could be used as input into a basis for the operator licensing written exams. In fact, the NRC should pursue a basis that includes appropriate standards and background information for the operator written exam process.

As outlined in my evaluation of issues 3, 4 and 6, the staff used NUREG/BR-0122, Revision 5 in the development of NUREG 1021, Revision 12. Additionally, when the NRC issued Revision 9 of NUREG 1021 in 2004, it was established that the related guidance that was previously incorporated in the Examiners Handbook for Developing Operator Licensing Examinations (NUREG/BR-0122, Revision 5, dated March 1990) was incorporated into the NUREG.

Therefore, NUREG/BR-0122 was no longer in effect. With this change, NUREG-1021 includes background information about the written examination for licensing purposes. Furthermore, Appendix A of NUREG 1021 already outlines the method to establish content validity of NRC examinations and, as discussed previously, was followed in the development of Revision 12 to NUREG 1021. Specifying additional requirements on the professional background in educational testing and measurement that would be necessary for personnel who design, and revise NRC examinations would impose a stricter standard than has been in place since 1988, and is unwarranted.

Furthermore, the staff has created documents to record the Operating Licensing Program History and Chronology (ADAMS Accession No. ML16084A726 and ML16060A123). They are also available on the NRC website (https://www.nrc.gov/reactors/operator-licensing/related-documents.html#history).

Panel Recommendation 2 (Issues 3, 4 & 6): Once the resources are in place and there is agreement in the basis of the program, a working group with subject matter experts and Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information representation from all four regions should revisit the RO written exam, cut scores, weighting of different topics and whether there should be a separate generic fundamentals exam.

I disagree with Recommendation 2. As determined by the Panel, the reactor operator examination as implemented in NUREG 1021, Revision 12 meets the requirements delineated in 10 CFR 55.41. During my review, I concluded that reducing the number of generic fundamental questions has not adversely affected the NRCs ability to make accurate licensing decisions for license applicants. Therefore, there is no basis to revisit the reactor operator written examination, cut scores, or weighting of different topics and whether there should be a separate generic fundamentals examination. While I am confident that the NRR staffs approach for incorporating the GFE into the reactor operating examination was valid; I also found that the basis for making changes to NUREG 1021 was not sufficiently documented by the staff. Therefore, I am tasking DRO to document the basis for integrating the GFE examination into the operator licensing examination, the methodology used to develop the new sampling plan, and the basis for determining that the reactor operator examinations ability to discriminate between a competent operator and one who is not, has not been adversely affected.

As part of being a learning organization, DRO has an on ongoing effort to perform an Effectiveness Review for Revision 12 of NUREG 1021. Based on the results of this Effectiveness Review we can determine whether additional changes are warranted after the review is complete.

Panel Recommendation 3 (Issue 7): The Panel recommends that a more formal process be developed to obtain regional input into the program office decisions. This could be a best practices documents that provides a standard framework for working groups. This document should include practices for communications, seeking input, and resolving differing views within the process.

I agree with the Panel that a best practices document that provides a standard framework for working groups is one way to improve communication and governance. It may also clarify scope and content of products as well as roles and responsibilities of working group members and their management before a formal review is finalized. For this reason, I am tasking DRO to develop a best practices document to provide additional clarity, and an operating framework for working groups. This will include communications expectations for both NRR and regional participants (both staff and management), and best practices for seeking input from diverse groups and addressing different opinions.

Concluding Remarks The submitters positions were of notable technical merit and well documented in the submittal. I commend all of you for your commitment and dedication to the NRCs mission.

I want to thank the submitters for raising this DPO and for their active participation in the process. I also want to thank the Panel for their thoughtful assessment of the concerns raised by the submitters.

While I do not agree with the submitters assertions that the changes to NUREG 1021 diminished the NRCs ability to make accurate licensing decisions for operator license applicants, their willingness to raise concerns is admirable and vital to ensuring a healthy safety culture within our agency.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information A summary of the DPO will be included in the Weekly Information Report (when the case is closed) to advise employees of the outcome.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Enclosure ATTACHMENT: DPO SUBMITTERS ORGANIZATIONS Daniel Bacon, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Joseph Demarshall, Senior Operations Engineer Operations Branch Division of Operating Reactor Safety Region I Kevin Murphy, Operations Engineer Operations Branch Division of Operating Reactor Safety Region I Peter Ott, Operations Engineer Operations Branch Division of Operating Reactor Safety Region I Randy Baker, Senior Operations Engineer Operations Branch Division of Operating Reactor Safety Region III Chuck Zoia, Senior Operations Engineer Operations Branch Division of Operating Reactor Safety Region III Bryan Bergeon, Operations Engineer Operations Branch Division of Operating Reactor Safety Region IV Mark Bates, Senior Operations Engineer Operations Branch 2 Division of Operating Reactor Safety Region II Michael Donithan, Operations Engineer Operations Branch 1 Division of Operating Reactor Safety Region II Andreas Goldau, Operations Engineer Operations Branch 2 Division of Operating Reactor Safety Region II Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information Michael Kennard, Senior Operations Engineer Operations Branch 2 Division of Operating Reactor Safety Region II Kevin Kirchbaum, Operations Engineer Operations Branch 2 Division of Operating Reactor Safety Region II Newton Lacy, Operations Engineer Operations Branch 1 Division of Operating Reactor Safety Region II David Lanyi, Senior Operations Engineer Operations Branch 1 Division of Operating Reactor Safety Region II Michael Meeks, Senior Operations Engineer Operations Branch 1 Division of Operating Reactor Safety Region II Joseph Viera, Senior Operations Engineer Operations Branch 1 Division of Operating Reactor Safety Region II Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

%

 <1*.4*13

 $" (  ""#% #! !#

! !""    #  !#! " '# "#!"

#

! &! !#!" $!



 !%"

    

 

*=    

"#!$#    

 

*,65 !!"

.4*890*33 !!"

:: !!"

!*2.8 !!"

)61* !!"

.8/.65 !!"

*:.9 !!"

651:0*5 !!"

63-*; !!"

.55*8- !!"

18,0+*;4 !!"

*,= !!"

*5=1 !!"

..29 !!"

%1.8* !!"

.:*5,6;8:!63-*5 !! !

%.13 !!

15/ !!

!#*=368 !!

6,2 !!

!*5215 !!

&.3315/ !!

!/31 !#"##"

1. .-1/*5 !!"

1/;.86*#63.-6 !

"636816 !

64+*8- 

" ,,.99165 6 







!!



%.13

#

*=    

 !!  (

 $" (  ""#% #! !#



Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Document 5: DPO Appeal Submittal

Official Use Only - Sensitive Internal Information NRC FOAM 1911 U.S. NUCLEAR REGULATORY COMMISSION DPO C.e Number (GN015) {¥)

OP0,,2021-002 NACIID 10.159 DIFFERING PROFESSIONAL OPINION -APPEAL Data Appeal ReceMHI

~..... ~

Nim* and TIiie or Submitter Orpnlullon Telephone Number(10 nufflllffc dlglll)

Daniel Bacon, Senior Operations Engineer R-IIIDRS/081 (404)997-4518 Nim* and TIiie of Supervisor Orpnlzatlon Telephone Number (10 runeric digits)

Tom Stephen, Branch Chief R-II/DRSIOB1

( 404) 997-4703 Balla for Mng appeaL Focus 1hcn,d be on perceived ftawa In the DPO Declllon and wfly the agency lhould come to a dlffer9nt conclualon. (UH contlnudon page or laac:h Word document)

This appeal is being submitted because several aspects of the methodobgy ~

logic used to reintegrate the generic fundamentals examination (GFE) into the reactor operator (RO) written examination specified in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Revision 12 (ML21256A276), as descri>ed in the NRR Office Oirec:to(s (NRR OD) decision (ML22139A175), were fundamentally flawed.

Additionaly, the NRR OD stated that the panel did not provide sufficient objective evidence to show that reducing the number of questions by default means that the content vafldily of the examination has been reduced. This appeal wil descrile each flaw and provide the objective evidence necessary to prove that the reduction in the number of each category of generic fundamentals questions to the act\\al number incorporated into the RO written examination specified in NUREG-1021 Revision 12 is a reduction in the content valicity of the examination. Fl'l&lly, this appeal will give additional reasons why the agency should come to a different conclusion.

This appeal contains the following sections:

A. NUREG,9R-0122 Revision 5 was incone~ used as a basis for reintegrating the GFE into the RO written examination as specified in NUREG-1021 Revision 12. Revision 4 of NUREGIBR-0122 is the appropriate version to use.

B. Content validity for the ROwrittenexaminationin NUREG-1021 Revision 12 has been lost C. The methodology used to risk-inform the development of the NUREG-1021 Revision 12 sample plan was flawed.

D. Meeting 10 CFR 55.41, ~Written Examination: Operators", does 11Ql, by itself, ensure a reliable content valid examination.

E. The GFE Reintegration Pibt Program was of very narrow scope and cid not provide a statistically meaningful justification for reducing the number cl generic fundamentals questions to the level spectfled in MJREG-1021 Revision 12.

F. Inspection Procedure (IP) 71111.11, does not provide a method for the NRC to evaluate the generic fundamer1als knowledge cl individual licensed operators.

G. Reasons why the agency shcud cane to a different conclusion.

SIGNATURE OF SUBMIT1ER Digitally signed by Daniel M. Bacon DATE Daniel M. Bacon Date: 2022.06.21 12:05:59 -04'00' S1GNA TURE OF CO-SUBMITTI:R (If lftJ)

DATE See page 22 for co-submitter signatures.

SCAN THE SIGNED AND DATED FORM (INCLUDING CONTINUATION PAGES OR WORD DOCUMENTS) AND EMAIL TO: DPOPM.Resource@..nrc.aov SIGNATURE OF DPO PROGRAM MANAGER DATE

~

06/27/2022

/

[N DPO Pf)NI accepted DPO lppctll nttumed

[ 0ellte Conllnu1tlon P_!D* I r

,Adel Cont""'-tlclft Page

• 1 NRC FORM 890 (011-20111)

Page1 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal A. NUREG/BR-0122 Revision 5 was incorrectly used as a basis for reintegrating the GFE into the RO Written Examination as specified in NUREG-1021 Revision 12. Revision 4 of NUREG/BR-0122 would have been the appropriate version to use.

The NRR OD stated on pages 4-5 of the decision document:

"Regarding the content validity of the exam. As described in the Panel's report, Appendix A of NUREG 1021 establishes the method to establish content validity of NRC exams. As stated in Appendix A:"

"The first step in the sequence of activities was to establish a link to job duties. A job task analysis (JTA) was performed by the Institute of Nuclear Power Operations (INPO) which identified approximately 28,000 knowledge and abilities (K/As)."

"As part of my review, I confe"ed with staff members to understand the rationale of not performing a new JT A for the reintegration effort. Although not mentioned in the Panel's report, the staff based their determination that a new JTA was not needed due to their assessment that the original JTA was still valid. As stated in the Panel's report, the original JTA was developed in the 1980s and was the original basis for thf3 K/As. In Revision 12 of NUREG 1021, the staff did not change what areas are tested; rather the change involved how much sampling was performed. Therefore, it was the staff's conclusion that the original JTA was still valid. This approach was similar to what was done in 2004 when the GFE test was changed in Revision 11 of NUREG 1021, as no new JTA was generated for the change in sampling."

"The next step specified by Appendix A of NUREG 1021 is to develop a sample plan. As mentioned in the Panel's report, the sample plan is an important part of making validity inference. Although not mentioned in the Panel's report, when developing the sample plan the staff utilized NUREGIBR-0122, "Examiners' Handbook for Developing Operator Licensing Written Examinations," Revision 5. The staff used this document to determine the original bases for the written examination sample plan, and to assist them in determining potential impacts of the proposed changes to incorporate the reactor theory/thermodynamics topics. It was from this review that the staff developed a risk-informed approach for adding six theory and thermodynamics questions by removing questions from other parts of the examination. This was done in a way that balanced the question sampling to ensure that adequate coverage of safety*

significant topics was retained. As mentioned by the submitters, no professional services company was used to make changes to the sampling plan delineated above. However, during my review I found that there is no requirement to use a professional educational services company for such changes. The information delineated above allowed me to determine that the content validity of the exam was not affected as it still meets the instructions and guidance contained under "Content Validity" in Appendix A of NUREG 1021."

The GFE was "split out" from the RO and SRO written examinations by removing the reactor theory, thermodynamics, and components categories from those examinations and testing those categories with a separate stand-alone examination. NUREG/BR-0122, Examiners' Handbook for Developing Operator licensing Written Examinations, Revision 4 (ML20117N069) is the revision that was in effect just prior to splitting out the GFE. This revision had the generic fundamentals topics included in the examination outline models for the RO and SRO written examinations with the percentage of generic fundamental questions required to construct a reliable content valid examination. Revision 4 should have been used to develop the sample plan for NUREG-1021 Revision 12.

Page 2 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal NUREG/BR-0122 Revision 5 (ML090910709) did not have the generic fundamentals topics included in the examination outline model and was not the revision that should have been used to develop the sample plan for NUREG-1021 Revision 12. Revision 5 of NUREG/BR-0122 was developed as a result of splitting out the GFE from the RO and senior reactor operator (SRO) written examinations as described in SECY-89-292 (ML12251A650) to reduce the resource demand on NRC examiners in preparing the written examination.

The method for constructing an examination is the same in Revision 4 and Revision 5. A job/task analysis (JTA) is used to produce knowledge and abilities (K/As) that are necessary to assure safe operation of the plant. The JTA used for Revision 4 of NUREG/BR-0122 is the same as Revision 5. The NRR OD decision states that the previous JT A is still valid. The DPO submitters agree because the job of a licensed operator has not changed. Next test specifications are developed based on validation data. These test specifications are used to generate the examination outline model. The JT A is used in conjunction with the examination outline model to select the individual K/As that will be tested on the examination. The examination outline model specifies the percentage of questions needed on each section of the examination to produce a content valid examination.

Although the process used for constructing an examination had not changed with Revision 5, the resulting examination outline model developed using Revision 5 will be significantly different because the generic fundamentals topics are not included in the examination outline model.

As stated in the Examiner's Handbook on page 2-15 of Revision 5: "The topic areas listed above in Section 2.1.5 [Components] and 2.1.6 [Theory] are covered by the Generic Fundamentals Exam Section. They are listed here for information only and will not be included in the exam construction discussion."

The examination outline model in Revision 4 of NUREG/BR-0122 specifies that RO written examinations should normally sample 25% from fundamentals (7% reactor theory, 7%

thermodynamics, and 11 % components).

Information Notice No. 88-40 also states that the RO written examination should normally sample 25% of its content from the fundamentals area (reactor theory, thermodynamics, and components). This information notice was describing Revision 4 of NUREG/BR-0122.

The examination outline model developed by Revision 4 was prepared by a professional education services company using the 1985 revision of the Standards for Educational and Psychological Testing published by the American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education.

Although there is no requirement to use a professional education services company when making changes to the written exam sample plan, it does not seem appropriate that the staff disregarded the work done by a professional education services company using established standards, and c::lrastically reduced the sampling in the fundamentals area to 4% reactor theory, 4% thermodynamics, and 0-2% components of an exam with 25% fewer questions than was in effect prior to splitting out the GFE without a valid basis to do so. This reduced the sampling in the fundamentals area by a factor of three overall from the. guidance that was in effect before the GFE was split out from the written exams. This also reduced the sampling of plant-wide generics by 40%.

Page 3 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal Flaws in the methodology used to develop the sample plan in NUREG-1021 Revision 12 will be discussed later in this document (See sections B - E.).

The resulting differences in the examination outline models are illustrated with the following two figures comparing the sample plans for a Reactor Operator at a PWR facility.

NUREG/BR*0122 Rev. 5 {no generic fundamentals items on exam)

SAMPLE PLAN PWR - REACTOR OPERATOR 13%

PLANT-WlDE GENERICS Page4 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal NUREG/BR-0122 Rev. 4 (generic fundamentals included on examination)

Pt.ANT SYSTEM$

1M PLANT* WIDE G£NEAlC$

NUREG/BR-0122 was incorporated into Revision 8 of NUREG-1021 well after the GFE was split out from the RO and SRO written examinations. For this reason, it also does not make sense that the staff used Revision 9 of NUREG-1021 to inform their work on development of the sample plan specified in Revision 12 of NUREG-1021.

Page 5 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal

8. Content validity for the RO written examination in NUREG-1021 Revision 12 has been lost.

Content validity is the degree to which a test measures the specific objectives or content.

The submitters evaluated the changes in the number of fundamentals questions based on using the targeted coverage of a 75-question reactor operator (RO) exam prepared in accordance with NUREG/BR-0122 Revision 4 as opposed to the actual coverage on a 75-question exam developed using NUREG-1021 Revision 12. This evaluation uses the same methodology that an NRC testing expert used to evaluate a 1993 LaSalle operator licensing examination (ADAMS ML19351B976). This document is also included in the NUREG-1021 Toolbox on Nuclepedia.

Fundamentals Target Value Actual Value Comments on Balance of Topic Area NUREG/BR-0122 NUREG-1021 Sampling Revision 4 Revision 12 Reactor Theorv 7%

4%

Only slightly greater than half Thermodynamics 7%

4%

Only slightly Qreater than half Comoonents 11%

0-2%

Balance of sampling lost Overall Comments Balance of sampling is lost in the components topic. The sampling on this topic was reduced by a factor of ten or more. It is impossible to make an inference of competency in the topic area of components if zero or even two questions are asked. The sampling of the fundamental topic areas overall was reduced by a factor of three. Sampling of reactor theory and thermodynamics is only slightly greater than half (by percentage) of what it was prior to the GFE being split out. Additionally, sampling of the plant-wide generic knowledge and abilities was reduced by 40%. It should also be noted that there are 25% fewer questions on the RO written exam specified in NUREG-1021 Revision 12 than there was prior to when the GFE exam was solit out.

This evaluation proves that the content validity of a NUREG-1021 Revision 12 examination has been lost. Balance of sampling does not remain intact for two of reasons. First, only zero to two component questions could be randomly selected on the exam. This is a reduction of component sampling by approximately a factor of ten. Second, the sampling of reactor theory, thermodynamics, and plant-wide generic K/As is only slightly over half (by percentage) of the sampling specified in NUREG/BR-0122 Revision 4. It should also be noted that there are 25% fewer questions on the RO written exam specified in NUREG-1021 Revision 12 than there was prior to when the GFE exam was split out.

Based on generating sample plans and reviewing written exams submitted by licensees on a frequent basis, the DPO submitters do not believe that the NRC can rely on the generic fundamental topics being tested in a reliable way on questions that are developed for other areas of the examination. This also does not provide the reliability needed for making licensing decisions.

The NRR OD stated on pages 2-3 of the decision document:

"Although not mentioned in the Panel's report, as part of the assessment the staff considered that knowledge of reactor theory and thermodynamics is also implicitly tested through other questions on the written exam, particularly questions that require applicants to Page 6 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal assess plant conditions to determine the appropriate response actions. Without the proper knowledge of reactor theory and thermodynamics, the applicant would be unable to properly diagnose these events and it would be more difficult for an individual to be able to pass the examination. Finally, during my discussion with the Panel, I asked for clarification on whether the Panel considered the contribution of the other exam questions in determining if the examinee had the required knowledge of fundamentals, the Panel stated that they only evaluated the two proposed avenues that had been suggested by the submitters. I agree with the staff's assessment that fundamentals are inherently tested, to varying degrees, in other exam questions and was appropriately considered."

It is important to note that operators are required to know symptoms-based entry conditions and the major mitigating strategies for abnormal and emergency procedures. Therefore, knowledge of generic fundamentals is generally not needed to answer these types of questions.

If the staff is relying on other sections of the written examination to make up for a drastic reduction in the number of theory and component questions being sampled, then a comprehensive analysis of a reasonable number of written examinations that had been developed under the previous revisions of NUREG-1021 should have been conducted to verify that a sufficient number of questions tested reactor theory, thermodynamics, and component knowledge in these sections on a consistent basis prior to making the changes. The submitters, some of whom were on the working group, have been unable to obtain any objective evidence that that type of analysis occurred. The DPO submitters would be interested to know if or how an assessment was performed.

Relying on "chance" that reactor theory, thermodynamics, and component questions will be developed from other sections of the examination does not provide the statistical reliability needed for a license examination that is based on random sampling to make an inference of competency in each section. It is not possible to do an effective risk analysis without some type of systematic evaluation of the risk factors.

It should also be noted that each knowledge and ability (KIA) category could be sampled only twice in Tier 1 and Tier 2 of the RO examination. Only some of the KIA categories in Tier 1 and Tier 2 even lend themselves to the possibility of developing questions that test theory or components. Tier 1 and Tier 2 Kl As were not designed for the purpose of testing reactor theory, thermodynamics. or components. Those topics have their own sections in the KIA catalog.

Although the original JTA is still valid, the staff is trying to force the testing of the generic fundamentals topics with KIAs that were designed to test other areas of knowledge and placed in other sections of the written examination.

Page 7 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal C. The methodology used to risk-inform development of the NUREG-1021 Revision 12 sample plan was fundamentally flawed.

The NRR OD stated on page 4 of the decision document:

"The staff used the methodology described in the Panel's report (pages 10-12), to risk-infonn the importance to safety of all topics tested in the operator examination. Additionally, they used their experience as expert qualified licensed examiners (subject matter experts) to evaluate whether all topics were being tested consistent with their contribution to safety and their protection function. It was from this assessment that the staff concluded that, although the reactor theory and thennodynamics topics have a similar importance to safety to other topics, they were being tested 3-5 times more frequently. Although not mentioned in the Panel's report, another factor that was taken into consideration for reintegrating the GFE into the operator licensing exam was that the GFE examination as administered until Revision 12 was focused on more theoretical plant configurations rather than plant specific configurations. This in combination with fundamentals being tested 18 months prior to the time of licensing allowed for possible unidentified degradation in concept knowledge and ability" The submitters do not believe that the method described is a valid method to determine how the GFE was reintegrated back into the reactor operator written exam for the following reasons:

An Importance Rating (IR) is an average of individual KIA Catalog Panel members' opinions of direct and indirect impacts of the KIA statement on safe plant operation, using a subjective scale from 1 to 5.

RO and SRO Impo11ance Ratings Ratine rnworrance for safe operation 5

Essential 4

Very important 3

Fairly important 2

Of lilllited importance I

Insignificant Itnponance h1dicates variability in rhe responses The purpose of I Rs is to be used as a qualitative measure to determine if a KIA is important enough to be used on an operator licensing examination. NUREG-1021 specifies that only KIAs with an IR of 2.5 or higher are normally selected for use on an initial operator licensing written examination. IRs were never intended to be used quantitatively.

IOLB averaged all the KIA importance ratings (I Rs) for Reactor Theory to obtain an Average-Rx-Theory-lR "value." The same process was then performed for the AC Electrical topic to obtain an Average-AC-Electrical-lR "value." IOLB compared these two "values" (i.e., Average-Rx-Theory-lR was 3.2 and Average-AC-Electrical-lR was 3.5) to justify that the Reactor Theory topic could be tested at the same rate as the AC Electrical topic.

Comparing the two "average values" is flawed because 1) taking the "average of an average" is only mathematically correct when the number of KIA statements is the same, which is not the case for the Reactor Theory and AC Electrical topics and 2) the I Rs were a subjective average that, in many cases, the KIA Catalog Panel members could not agree on.

Page 8 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal IOLB's "average IR" methodology is not consistent throughout the ES-4.1 written exam outline model for sampling frequencies. Specifically, the ES-4.1 written exam outline model includes Tier 1 and Tier 2, where each Tier includes two Groups, i.e., Tier 1 Group 1 and Group 2, Tier 2 Group 1 and Group 2. Group 1 topics are sampled more frequently than Group 2 topics in the ES-4.1 written exam outline model even though the "average IR" for some Group 2 topics is HIGHER than the "average IR" for some Group 1 topics. Consider the following example:

007 Pressurizer Relief/Quench Tank (PWR Tier 2, Group 1) "average IR": 3.12 016 Non-Nuclear Instrumentation (PWR Tier 2, Group 2) "average IR": 3.5 Based on the sample plan specified in NUREG-1021 Revision 12, System 007 being T2G1 is guaranteed to be sampled once and has a 5/23 chance of being sampled twice. System 016 being T2G2 has only a 9/24 chance of being sampled once.

The submitters do not believe that the importance to safety of theory and component K/As can be directly compared to the importance to safety of individual system Kl As based on Importance Ratings. This is because a single theory or component KIA may apply to numerous safety related systems throughout the nuclear plant. A lack of knowledge in a reactor theory, thermodynamics, and component topic could result in the inability to diagnose problems or property operate multiple systems.

It is very important for a licensed operator to have a solid fundamental knowledge of theory and components to be able to respond properly when the plant is not operating as designed or in situations that procedures do not specifically address.

It was stated above that the staff used their experience as expert qualified license examiners (subject matter experts) to evaluate whether all topics were being tested consistent with their contribution to safety and their protection function. The sixteen DPO appeal submitters are also expert qualified license examiners (subject matter experts) and strongly believe that the reintegrated RO written examination has lost content validity and will lead to a reduction in component, reactor theory, and thermodynamics knowledge among licensed operators over the long term.

Three or four of the program office (IOLB) subject matter experts are qualified chief examiners.

The DPO appeal submitters estimate that IOLB chief examiners have performed the duties of a chief examiner for a combined total of less than five examinations.

To compare the experience level of the IOLB subject matter experts with that of the DPO appeal submitters, consider that:

Fourteen of the sixteen submitters have held either an RO or SRO license at one or more facilities.

Two of the sixteen submitters have held SRO training certifications.

Fifteen of the sixteen submitters are qualified chief examiners.

• The submitters have administered well over 500 examinations.

The submitters have been chief examiners for well over 150 examinations.

Submitters' previous experience, in addition to the above includes:

o U.S. Navy Engineering Officers, o

U.S. Navy prototype training instructors, o

U.S. Navy Engineering Officers of the Watch, o

U.S. Navy Engineering Watch Supervisors, Page 9 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal o

Licensee initial license training instructors, o

Licensee Operations Shift Managers, o

Licensee system engineers, o

Licensee reactor engineers, and o

Licensee SROs on-loan to the INPO Training and Accreditation Department.

In the short term, it is impossible to know when an event will occur where a licensed operator will be required to use knowledge of reactor theory, thermodynamics, or components to ensure the safety of the nuclear plant and public. An RO or SRO could be licensed using NUREG-1021 Revision 12 and never get a reactor theory, thermodynamics, or components question correct on their examination.

While it is true that an applicant could also fail all the questions on a particular system ( could only be one question or no questions for a particular system) and still pass their written examination, they would still have to get most questions correct on other systems so that an overall inference of competency could be made. Any single question could be answered incorrectly because the question itself was misinterpreted and not because there was a lack of knowledge.

Although the decision report stated that generic fundamentals topics may have been oversampled while the GFE was split out from the written examination, that in no way justifies drastic under sampling from this point forward.

It is not appropriate that IOLB removed one question each from Tier 1 Group 2 and Tier 2 Group 2 and four questions from the plant-wide generics in order to make room for reactor theory and thermodynamics questions to keep the total number of questions on the RO written examination at 75. See the table below.

The information included in the table below was taken from the RO PWR Examination Outlines contained in ES-401 of NUREG-1021 Revision 11 and ES-4.1 of NUREG-1021 Revision 12.

Tier/Group

1. of questions

1. of APEs/EPEs/ # of questions

1. of APEs/EPEs/

to pick in Systems to pick to pick in Systems to pick NUREG-1021 from in NUREG-NUREG-1021 from in NUREG-Rev. 11 1021 Rev. 11 Rev. 12 1021 Rev. 12 Tier 1 Group 1 18 23 18 23 Tier 1 Group 2 9

36 8

36 Tier 2 Group 1 28 23 28 23 Tier 2 Group 2 10 24 9

24 For Revision 11, Each Tier 1 Group 1 topic has a 18/23 (78%) chance of being sampled once.

For Revision 11, Each Tier 1 Group 2 topic has a 9/36 (25%) chance of being sampled once.

For Revision 11, Each Tier 2 Group 1 topic has a 28/28 ( 100%) chance of being sampled once and a 5/28 (17%) of being sampled a second time.

For Revision 11, Each Tier 2 Group 2 topic has a 10/24 (41%) chance of being sampled once.

Page 10 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal For a sample plan that is supposed to be constructed to produce an inference of competency by randomly sampling a large body of knowledge, it is not appropriate to remove questions from Tiers and Groups with topic areas that have only a 25-78% chance of being sampled once rather than a Tier and Group that has a 100% chance of being sampled at least once and approximately a 21 % chance of being sampled twice.

Removing six questions from the other sections of the examination ~lso reduced the content validity of other sections of the examination. See section B of this document.

Although previously administering the GFE approximately 18 months prior to the time of licensing allowed for potential degradation of that knowledge, the previous GFE required the applicants to learn the concepts and required a passing score of 80% to be eligible apply for a license. The drastically reduced testing of these concepts in NUREG-1021 Revision 12 has already caused a reduction in the amount of training provided and will allow applicants to receive a license without demonstrating any knowledge in these areas on the RO written exam.

Page 11 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal D. Meeting 10 CFR 55.41, "Written Examination: Operators", does fil?!, by itself, ensure a reliable content-valid examination.

The NRR OD stated on page 3 of the decision document:

"Although the Panel agreed with many of the submitters' concerns, the Panel concluded that the operator licensing exam meets 10 CFR 55.41, "Written Examination: Operators." The regulation requires that the written exam for an operator of a facility includes a representative sample from 14 items to the extent applicable to the facility. Since NUREG 1021, Revision 12 does sample the three areas that were subject of the submitters' DPO (Reactor Theory and Thetmodynamics and Components) the Panel concluded that there was a representative sample, as required by the regulation."

"I agree with the Panel's conclusion that the changes to the operator's exam implemented in NUREG 1021, Revision 12 meet the requirements delineated in 10 CFR 55.41. Specifically, the staff included a representative sample of all 14 items required by the regulation in the sample plan. However, I disagree with the Panel's conclusion that the logic used by the staff to reintegrate the GFE into the operator licensing exam was not appropriate."

With that logic, it could be said that an exam with 15 questions that had a representative sample of all 14 items would meet 1 O CFR 55.41. It would be unconscionable to believe that a 15-question exam would be adequate to reliably test the body of knowledge needed to be a safe nuclear operator or make an inference of competency on any of the examination sections. It would, however, meet the code.

Because a single test cannot measure every KIA required to be a competent licensed operator, it must sample the required knowledge or abilities in a manner that allows inferences to be made about the examinees' performance on the broader population of knowledge, even though the full body of knowledge was not tested.

The sample must be evenly distributed and soundly based so that the NRC can confidently assume that the untested knowledge is proportionately known or not known in relation to the score on the sample. In other words, by testing performance on the sample, it is possible to make inferences concerning the broader area of knowledge not tested. This is referred to as a "validity inference."

10 CFR 55.40, "Implementation", states that the Commission shall use the criteria in NUREG-1021 to prepare the written examinations and operating tests. Therefore, it is imperative that NUREG-1021 contains a sample plan that will result in an examination that contains the proper proportion and number of questions in each section (Tier) of the examination.

10 CFR 55.40 and 10 CFR 55.41 work together to develop an operator licensing written examination and should not be considered separately. Neither regulation by itself will result in a reliable content-valid examination that will meet the intent of "validity inference" testing.

The reactor operator written examination sample plan contained in NUREG-1021 Revision 12 does not have sufficient dedicated sampling in components, reactor theory, or thermodynamics to make a reliable validity inference for licensing decisions.

Page 12 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal E. The GFE Reintegration Pilot Program was of very narrow scope and did not provide statistically meaningful Ju$tification for reducing the number of generic fundamentals questions to the level specified in NUREG-1021 Revision 12.

The NRR OD decision also stated on page 4 of the decision document:

"The methodology used by the NRC staff relied on: 1) the professional judgment of qualified expert examiners (subject matter experts); 2} risk insights to evaluate how much emphasis should be given to testing fundamentals topics such as reactor theory and thermodynamics and in the development of the sample plan; and 3) the review and consideration of available data, such as an industry-led pilot. Although not mentioned in the Panel's report, the GFE reintegration concept was also evaluated by four facilities [2 Boiling Water Reactors (BWRs),

and two Pressurized Water Reactors (PWRs) owned by different utilities] that developed questions based on theory and component topics selected by NRC staff (ADAMS Accession No. ML20083F400)."

Flaws in the methodology contained in items 1 and 2 in the paragraph above have been addressed in previous sections of this document. The remainder of this section will discuss the flaws in the pilot program described in item 3.

The NRR OD decision stated on pages 5-6 of the decision document:

"The NRC is a learning and evidence-based organization. The NRC Principles of Good Regulation have led the agency in making regulatof'/ decisions using available data. Although not mentioned in the Panel's report, the GFE integration concept was evaluated by four facilities that developed questions based on theof'/ and components topics selected by the NRC through an industry led pilot. The pilot included the development of eight plant specific questions using draft NRC guidance that mimic the new proposed testing methodology. The facilities then administered these questions to applicants in approximately the same period as the facility's practice examination. The applicants were not given the opportunity to study the theof'/ and components topics prior to question administration, nor where they given prior notification of the intent to test these areas. The facilities assessed the applicants' performance on each question and detennined the impact on the overall examination performance by replacing selected questions from the practice examination with these theof'/ and components questions. The results were the following:"

Average score on Impact on audit examination if pilot questions pilot questions used1 Station 11 85.7

+0.6 Station 2 81.8

-1.2 Station 3 82.1

-0.4 Station 4 58.3

-3.8 "Based on the results of the industf'/ pilot presented above, the changes in the sample plan produced slightly lower scores in three of the four faciHties. The staff assessed that at least in Page 13 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal one facilfty (Station 4) there was a need to better maintain theory knowledge throughout training. This showed unidentified degradation in concept knowledge and ability that would likely not have been identified under Revision 11 of NU REG 1021. The staff took into consideration the results of the industry led pilot delineated above to assess the impact of the proposed changes to the operator licensing exam."

"In summary, I found that reducing the number of generic fundamental questions has not adversely affected the NRC's ability to make accurate licensing decisions for operator license applicants. I also conclude that the operator licensing examination's ability to discriminate between a competent operator and one who is not, has not been adversely affected. This is based on my review of the three elements of the methodology that the staff relied on when performing the reintegration of the GFE into the operator licensing exam."

The pilot program is of very narrow scope in that a total of only 2 PWRs and 2 BWRs were included, and each performed only one examination. The DPO submitters did not have access to any information that described the number of applicants who participated in this single examination or the individual applicant scores.

The pilot program that was used to inform splitting out the GFE (described in SECY-89-292 (ML12251A650}) consisted of two pilot examinations. The first pilot examination was administered to 209 BWR candidates from 19 facilities. The second examination was administered to 364 PWR and 161 BWR candidates from 58 facilities. The pilot program for splitting out the GFE was much more extensive than the one for reintegrating (77 facilities vs. 4 facilities, 734 applicants vs. "?" applicants).

The only information that had any description concerning the specifics of the conduct of the pilot program used to "informn the decision to reintegrate the GFE was a letter from a Senior Project Manager, Regulatory Affairs from the Nuclear Energy Institute (NEI} (ADAMS Accession No. ML20083F400).

The NEI letter stated that the DRO/NRR provided eight randomly sampled K/As for development of the generic fundamentals questions. Only three of the four stations used these K/As. The fourth station selected their own K/As which did not sample any component K/As.

The NEI letter also stated that only three of the four stations submitted their questions for review by NRR. Given this, NRR has no idea of the quality or substance of 25% of the questions administered for the pilot project.

The DPO appeal submitters have the following concerns concerning this pilot project:

Did NRR/IOLB do an independent review of the data described in the NEI letter on an individual participant and individual question basis?

Was this data incorrectly evaluated using an average of an ~verage of different size number sets just as the method of risk informing the changes to the sample plan was?

Was this very limited set of data which was used to support a major change to the operator licensing written examination comparing "apples to oranges" because 25% of the questions were different and 25% of the questions were not reviewed for quality by NRR?

Was the station that was the outlier in the table above either the station that selected the different K/As or the one that did not submit their questions to NRR for review? Or both?

Page 14 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal This would be relevant because that is the only set of data in the table that is statistically significant.

Were all questions developed for the pilot project of sufficient quality. The DPO appeal submitters believe that at least some were not. (See below.)

What statistical analysis was performed by NRR that would suggest that a 32-question pilot program would provide meaningful data to support allowing applicants to get zero questions correct on generic fundamentals material and be issued a license? Compare this to the sample size in SECY-89-292.

A major flaw in the pilot program is that no individuals participated that had not already been through a full GFE training program and examination. The DPO appeal submitters concern is about the future performance of applicants after going through a generic fundamentals training program that has been truncated due to the reduced emphasis on generic fundamentals in NUREG-1021 Revision 12.

On the Nuclepedia website in the NUREG-1021 Toolbox, IOLB has listed two examples of Tier 4 Theory and Thermodynamics questions used on the NEI pilot examination. Example 1 of those questions is listed below.

Example1 This example is from an NEI pilot exam and Is based on KIA 193005, "ThennOdynamic Cycles,* K1.03, *oescrtbe ho Unit 2 is operatillg at 100% power. The following occurreo:

• The 6A Feedwater Heater Normal Level Control Valve failed closed.

• The 6A Feedwater Heater High Level Conlfol Valve Is seized dosed.

With NO operator action, reactor power will _(1 )___, and tile thermal efficiency of the steam plant will _(2}_

A. (1) rise (2) increase

8. (1) rise (2) decrease

c. (1) lower (2) increase D. (1) lower (2) decrease This question does not meet the quality guidelines of NUREG-1021 due to having two non-credible distractors. It is not credible to believe that two automatic "control valves" either failing closed or being seized closed in any system would cause an increase in efficiency when control valves modulate to maintain the normal operations and efficiency of a system. Choices A and C are not plausible.

This situation easily removes two answer choices and makes this a 50/50 question with a low level of difficulty. This question has low discriminatory ability and most experienced chief examiners would not allow it on an examination.

Page 15 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal F. Inspection Procedure (IP) 71111.11, does not provide a method for the NRC to evaluate generic fundamentals knowledge of individual licensed operators.

The NRR OD stated in the decision document:

"Regarding the long-term impact of the changes to NUREG 1021, Revision 12. The NRG does not rely on a one-time examination to maintain proper knowledge on how to respond to emergencies. In accordance with 10 GFR 55.59 (a), "Requalification requirements," each licensed operator must pass a comprehensive (biennial) written examination and an annual operating test. This allows the NRG to have continued assurance that the operators have the proper knowledge to respond to emergencies. Additionally, the requalification standards implemented by the licensees contain expectations to include refresher training on fundamental topics, such that, among other things, it increases the level of understanding of applied fundamentals presented in initial training.

Moreover, one strength of the NRG regulatory framework is that it provides for needed oversight through inspection. Inspection Procedure (IP) 71111.11, "License Operator Requalification Program and Licensed Operator Performance," provides for quarterly samples involving licensed operator performance in the control room during periods of heightened actMty or risk.

The procedure provides guidance on evaluating diverse aspects of operator performance which include, in part: operator compliance and use of plant procedures; use and interpretation of plant instruments, indications, and alarms; and diagnosis of plant conditions based on instruments, indications, and alarms diagnosis. Additionally, as part of ongoing oversight, the NRG analyzes licensee event reports, inspection findings, and industry databases to determine whether adverse trends in performance exist. The combination of these activities would allow the NRG to determine if additional actions are warranted to mitigate a negative trend in operator performance, and to ensure continued safe operation. I am confident that the ongoing implementation of the Reactor Oversight Process will continue to ensure that reactor operators safely operate their respective nuclear plants."

The DPO appeal submitters do not agree with these statements for the following reasons.

There are events that have occurred at operating power reactors in the recent past where resident inspectors or IP71111.11 inspections did not identify an adverse trend in generic fundamental knowledge prior to the event occurring.

In one event a unit started up with one nuclear instrument that had not been operating properly for a significant amount of time and had not been identified as being inoperable by the licensee. During this startup the reactor operator added an excessive amount of reactivity that resulted in a high source range flux trip while the control rods were still being withdrawn. The two senior reactor operators that were directly responsible for supervising the startup and the reactivity addition did not identify either of these problems prior to the reactor trip.

Requalification training does not ensure an adequate level of understanding of applied fundamentals presented in initial training.

Although generic fundamentals knowledge training is probably contained on the two-year training "backbone" of most sites, that does not mean that it will be tested on the biennial written exam. There are approximately 30-35 questions on biennial written examinations. The questions are only a sample of the training that was received over a Page 16 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal two-year period. Many written examinations reviewed for IP71111.11 B inspections do not contain any generic fundamental or component questions.

The biennial written examination and annual operating tests are not required to be constructed or conducted in accordance with NUREG-1021. 10 CFR 55.59, "Requalification", allows the licensees to manage their lectures, on-the-job training, and evaluation in accordance with a systems approach to training (SAT) process. The SAT based training process in effect at a site is not inspected during IP71111.11.

The NRC normally only reviews the quality of one of generally five biennial written examinations during IP 71111.11. A second examination may be reviewed if the first one is determined to be greater than 20% flawed.

IP 71111.11 does not require the inspector to review the success rates on individual test questions and therefore has no way of identifying trends of any topic area.

The annual operating test (scenario portion) does not provide a reliable method for evaluating knowledge of generic fundamentals for an individual because it is a crew exam. A crew member that lacks knowledge can be covered by others that do possess the knowledge.

The individual control room/simulator, in-plant, and administrative JPMs frequently do not test any generic fundamentals knowledge because it is not a requirement.

The generic licensed operator operations checklists (Appendix H) which are performed by the resident inspectors during their observations may only identify a generic fundamentals weakness on two items which are categorized under interpretation, diagnosis, and understanding. These are:

o Were off-normal trends recognized in a timely fashion?

o. Did the crew demonstrate an understanding of the manner in which the plant, systems, and components operated and interacted, including the knowledge of setpoints, interlocks, and automatic functions?

Although these two items are in an area that could possibly identify a general fundamentals weakness, they are not a reliable evaluation of individual operator knowledge or trends in generic fundamentals. Because this checklist is performed about four times per site per year, it will only cover a few operators per occurrence and there may be no abnormal trends or evolutions during performance of the checklist that would test this knowledge.

Page 17 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal G. Reasons why the agency should come to a different conclusion.

1. In March of 1979 the accident at TMl-2 occurred. NUREG-0585 (ML061430367), the TMl-2 Lessons Learned Task Force Final Report, stated that although the accident at TMl~2 stemmed from many sources, the most important lessons learned fall in the general area of operational safety. This general area includes the topics of human factors engineering; qualifications and training of operations personnel; integration of the human element in the design, operation, and regulation of system safety; and quality assurance of operations. Specifically, the primary deficiency in reactor safety technology was the inadequate attention paid by all levels and all segments of the technology to the human element and the fundamental role of operators in both the prevention of accidents and the response to accidents.

NUREG~0737(ML051400209), Clarification of TMl-2 Action Plan Requirements, required new categories added to the operator and senior operator written exal'.J'linations. These were "Principles of Heat Transfer and Fluid Mechanics" for reactor operators and

'Theory of Fluids and Thermodynamics" for senior operators. NUREG-0737 also directed that the passing grade for the written examination shall be 80% overall and 70%

in each category.

In October of 1988, NUREG-1021 Revision 5 (ML20235R399) was issued. The written examination was divided into three categories: Approximately 24-25% of the exam was "Reactor PrinciplesfThermodynamics/Components.

• Approximately 27-33% of the exam was "Emergency and Abnormal Plant Evolutions." Approximately 43-48% of the exam was "Plant Systems and Plant-Wide Generic Responsibilities." Instructions for preparing the exam were contained in NUREG/BR-0122 Revision 4, the Examiners' Handbook for Developing Operator Licensing Examinations. The passing grade remained 70% in each category with an overall final grade of at least 80%.

These content valid licensing examinations were endorsed by the testing industry in the 1985 revision of the Standards for Educational and Psychological Testing published by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education.

In October of 1989, the GFE program was formally implemented by Generic Letter 89-17 and the GFE exam was split from the rest of the written exam. The GFE exam required 80% to pass and was not averaged into the remainder of the written exam. Passing the GFE exam was now a prerequisite to applying for a license. This was a 100-question exam.

In 2004, the GFE exam was reduced from 100 questions to 50 questions. A minimum grade of 80% was still required to pass the examination.

In September 2021, Revision 12 reintegrated the GFE topics back into the remainder of the written examination. This reduced the number of reactor theory and thermodynamics questions to approximately 8 percent of the RO written examination with no minimum cut score on the GFE questions. This also reduced the number of dedicated component questions to O to 2%.

Page 18 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal NUREG-1021 Revision 12 has effectively reversed all the changes to the operator licensing written examination that were put in place as corrective actions following the accident at Three Mile Island.

2. NUREG-1021 Revision 12, Appendix A, Section D, The Importance of the Written Exam, states, in part, "Deemphasizing or sidestepping knowledge testjng through careless or simplistic testing processes or treating it secondarily to other portions of the examination that are more operationally oriented could affect subsequent job performance. Failing to focus on testing the individual operator's cognitive abilities (i.e., comprehension, problem-solving, and decision making) or paying insufficient attention to the operator's fundamental understanding of job content (e.g., systems, components, and procedures) may ultimately place job performance at risk of gradual degradation. When the demand for disciplined learning and study declines or the level of knowledge {depth of application} required for the iob is reduced, it could lead to less time spent in training preparation. less mental review and practice. more forgetting of factual details. less reinforcement and application of job concepts. and a gradual decline in performance.

Moreover, without a solid fundamental knowledge base, operators may not perform acceptably in situations that are notspecifically addressed in procedures. Since every performance has an underlying knowledge component, that knowledge and its depth need development and assessment to ensure the operators' competence on the job.

Recent studies assessing mental performance in cognitively demanding emergencies point out that higher level cognitive thought (such as event diagnosis and response planning) are important in responding to safety-related events."

NUREG-1021 Revision 12 changes made to the RO written examination sample plan

!!£ deemphasizing or sidestepping knowledge testing through careless or simplistic testing processes as described above.

NUREG-1021 Revision 12 changes made to the RO written examination are already resulting in less time training in generic fundamentals. A Region I licensee Training Manager informed the Region I Operator Licensing Branch Chief and a member of IOLB that the site has already cut the generic fundamentals training for initial operator licensing applicants in half. Training Managers at Region II sites have informed chief examiners that there will be a reduction In the amount of time spent training on generic fundamentals due to the changes in NUREG-1021 Revision 12.

Licensees are in the business of making money and they will spend no more time on generic fundamentals training than is required to ensure that their applicants pass whatever version of examination that is in effect by the current revision of NUREG-1021.

Some operating nuclear fleets have recently reduced the length of initial operator license class from approximately 18 months to 12 months.

3. ~sto

---

have reduced the education, experience, and eligibility

~ments for initial licensed operator class. NUREG-1021 Revision 12-no longer require 6 months on-site or 3 months as an extra person on-shift. The degree requirements for senior reactor operators have also been reduced.

Page 19 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal

4. An RO applicant could pass the written examination and get a license without getting a single question on generic fundamentals or components correct. That same operator could then come back to class as an upgrade senior reactor operator (SRO) a year later and get an SRO license still without ever passing a single generic fundamentals question on a licensing examination. An instant SRO could get a license without ever getting a single question on generic fundamentals or components correct. Based on the latest degree requirements neither of these SROs may have any post-high school formal education on some generic fundamentals and components topics. Either of these SROs could also be assigned as the Shift Technical Assistant (or equivalent title) on shift.

From 1989 to 2003 an applicant was required to answer 80 generic fundamentals questions correctly to be issued a license.

From 2004 to 2021 an applicant was required to answer 40 generic fundamentals questions correctly to be issued a license.

Since September 2021, an applicant is not required to answer any generic fundamentals questions correctly to be issued a license.

5. The importance of reactor theory, thermodynamics, and components knowledge cannot be understated. The only power reactor to have a core melt accident in United States history was at least partially due to a lack of component and generic fundamental knowledge.

6. The reduction of education, eligibility, and experience requirements combined with the reduction in training and testing that is already occurring is not in the best interest of public health and safety. A reduction in performance due to these issues could result in an accident that could be prevented by maintaining proven standards. These concerns are also magnified because the fleet of current nuclear plants is aging, and many are operating closer to limits due to power uprates. Although the GFE program prior to NUREG-1021 Revision 12 was not perfect, the testing should not have been carelessly reduced to its current level.

7. The primary strategy for ensuring reactor safety is both redundancy and diversity.

Eliminating the GFE (or failing to sample a meaningful number of generic fundamentals topics on the RO examination after reintegration) does not support this strategy. The.

generic fundamental topics of Reactor Theory, Thermodynamics, and Components establish the foundational knowledge which supports the understanding necessary for the safe operation of a nuclear power plant.

WASH-1400, "Reactor Safety Study: An Assessment of Accident Risks in U.S.

Commercial Nuclear Power Plants," issued October 1975, details PRA studies have recognized the importance of dependent failure as a means of defeating designed-in redundancy and diversity. WASH-1400 employed the term common-mode failure, defined as follows:

Multiple failures which are dependent, thereby causing the joint failure probabUity to increase. The multiple failures are common mode or dependent because they result from a single initiating cause, where cause is used in its broadest context.

Page 20 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal Consider an applicant who has a generic weakness in either Reactor Theory, Thermodynamics, or Components as a single initiating cause. As a result of this weakness, multiple failures may be attributed to this initiating cause. For example, a weakness in understanding failure mechanisms such as cavitation or runout associated with the operation of a centrifugal pump will have ties to multiple safety related systems with pumps. Additionally, understanding the bases for emergency operating procedures actions associated with pump operation will be challenged. Additional examples involving reactor theory, thermodynamic operations, and multiple components can be easily postulated.

8. Drawing an analogy from 10CFR50.59, Changes, Tests and Experiments; the elimination of the GFE (or failing to sample a meaningful number of generic fundamentals topics on the RO examination after reintegration) is a "departure from a method of evaluation".

Previously an applicant demonstrated their mastery of generic fundamental topics by successfully passing a 50-question examination as a prerequisite to apply for an operator license. By eliminating the separate GFE and integrating the topics into a single examination without a separate assessment of their mastery of the generic fundamentals topics or adequate sampling, a departure from a method of evaluation has occurred. Change is inevitable, and just as a licensee uses the 50.59 process to perform a thorough review and evaluation of changes, tests and experiments, the elimination of the GFE should be subject to the same type of analysis with respect to reactor safety. Unfortunately, the level of scrutiny applied to the elimination of the stand-alone GFE fell short. Numerous factors contributed to an incomplete and incorrect evaluation regarding elimination of the stand-alone GFE and-reintegration into the RO written examination. This has resulted in the loss of content validity for operator licensing written examinations.

9. An in-depth analysis of what was needed to develop content valid operator licensing written examinations that adequately tested reactor theory, thermodynamics, and components was performed decades ago using nationally recognized standards.

The staff itself has stated that the JT A used to develop those examinations has not changed, yet they have drastically departed from the sample plan without a valid basis to do so. The DPO submitters have provided objective evidence in this appeal document to prove that the methodology that the staff used to reintegrate the GFE into the RO written examination is fundamentally flawed. The NRG should restore the content validity of the RO written examination in order to make accurate and reliable licensing decisions.

Ensuring that only competent operators receive a license is in the best interest of public health and safety.

Page 21 of 22 Official Use Only - Sensitive Internal Information

Official Use Only - Sensitive Internal Information DPO-2021-002 Appeal Generic Fundamentals Examination DPO Appeal Additional Co-submitter Signature Page Name Title Region Signature Joseph Demarshall Senior Operations Engineer I

Digitally signed by Joseph Joseph Demarshall Demarshall Date: 2022.06.15 15:54:32 -04'00' Kevin Murphy Operations Engineer I

K, M M h

Digitally signed by Kevin M.

8Vln Urp y Murphy Dete: 2022.06.1612:36:18 --04'00' Pete Ott Operations Engineer I

Peter J. Ott Digitally signed by Peter J. Ott Date: 2022.06.17 13:45:38

-04'00' Mark Bates Senior Operations Engineer II Mark A Bates DlgltallyslgnedbyMarkA. Bates Date: 2022.06.15 09:49:28 -04'00' Digitally signed by Michael G.

Michael Donithan Operations Engineer II Michael G. Donithan Donllhan Date: 2022.06.15 09:18:40 -04'00' Andreas Goldau Operations Engineer II D

, I M B Digitally signed by Daniel M.

ante 8CQn Bacon

-,.,, 1',lec f-"\\

Date: 2022.06.21 06:57:54 -04'00' Digitally signed by Michael N.

Michael Kennard Senior Operations Engineer II Michael N. Kennard Kennard Date: 2022.06.15 14:00:09 -04'00' Digitally ~ gned by Kevin P.

Kevin Kirchbaum Operations Engineer II Kevm p. Kirchbaum Kirchbaum Date: 2022.06.21 09:56:22 -04'00' Newton Lacy Operations Engineer II Newton T Lacy Dlgltalfy signed by Newton T, Lacy Date: 2022.06.15 14:05:38 -04'00' David Lanyi Senior Operations Engineer II David R Lanyi OigitallysignedbyDavidR.Lanyi Date: 2022.06.1615:07:39 -04'00' M. h I K M Digitally signed by Michael K.

Michael Meeks Senior Operations Engineer II IC ae eeks Meeks Date: 2022.06.15 14:08:56 -04'00' Joseph Viera Senior Operations Engineer J

h p v*

Digitally signed by Joseph P, II 0Sep lef8 Viera Date: 2022.06.16 08:20:39 -04'00' Randy Baker Senior Operations Engineer Ill

~J.Jt 0-~

Digttally signed by Randal o. Baker Date: 2022.06.15 15:04:37 -05'00' Chuck Zoia Senior Operations Engineer Ill Charles D Zoia Digitally signed by Charles D. Zoia Date: 2022.06.21 06:45:14 -05'00' Bryan Bergeon Operations Engineer IV B

A B Digitally signed by Bryan A ryan. ergeon Bergeon Date: 2022.06.21 07:47:43 -05'00' Page 22 of 22 Official Use Only - Sensitive Internal Information

Document 6: Statement of Views on DPO Appeal Submittal

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 MEMORANDUM TO:

Daniel H. Dorman Executive Director for Operations FROM:

Andrea D. Veil, Director Office of Nuclear Reactor Regulation

SUBJECT:

STATEMENT OF VIEWS REGARDING APPEAL OF DIFFERING PROFFESIONAL OPINION DECISION INVOLVING THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

The purpose of the memorandum is to provide you with my statement of views on the appeal of the differing professional opinion (DPO)-2021-002, titled Operator Licensing Generic Fundamentals Examination (Agency Documents Access and Management System (ADAMS)

Accession Nos. ML21322A200 and ML21322A199). The DPO was submitted on September 27, 2021, and raised concerns regarding changes to the composition of the licensed operator initial examination as issued in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Revision 12 (ML21256A276).

On November 19, 2021, a DPO Ad Hoc Review Panel (the Panel) was established and tasked to meet with the submitter, review the DPO submittal, and issue a DPO Panel report, including conclusions and recommendations. On April 5, 2022, after reviewing the applicable documents, conducting internal interviews with relevant individuals, and completing their deliberations, the Panel issued their report to me, the Office of Nuclear Reactor Regulation (NRR) Office Director (ML22095A246). On May 20, 2022, I issued the Directors Decision memorandum to the DPO submitters documenting my assessment and decision regarding the DPO (ML22139A175). On June 27, 2022, the DPO submitters sent an appeal to you, the Executive Director for Operations, and expressed their views on the Directors Decision memorandum.

In the appeal, the submitters specified seven areas of disagreement, referred to as flaws.

Given the importance of the reactor operator written examination, I took the necessary time to assess the appeal. Specifically, I considered additional pertinent guidance, staff CONTACT: Diana Betancourt-Roldan, RIII 630-810-4373 August 17, 2022 Signed by Veil, Andrea on 08/17/22 Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information positions and information provided by subject matter experts to inform whether any additional actions were warranted.

In the appeal the submitters delineated their seven areas of disagreement as A-G. My views on those seven areas are as follows:

A. Basis for Using NUREG/BR-0122, Examiners Handbook for Developing Operator Licensing Written Examinations, Revision 5 In my Directors Decision memorandum, I reference the use of NUREG/BR-0122, Revision 5 (ML090910709) in the development of the sample plan. As stated in my memorandum, the staff used Revision 5 of NUREG/BR-0122 to determine the original bases for the reactor operator written examination sample plan, and to assist them in determining potential impacts of the proposed changes to incorporate the reactor theory/thermodynamics topics. The staffs decision to use Revision 5 to inform their work was based on the fact that it shows how the sample plan was constructed once the Generic Fundamentals Examination (GFE) was originally split from the operator reactor examination. The staff used NUREG/BR-0122, Revision 5, in order to understand the basis and supporting information for the original 75 question examination. This review allowed the staff to develop a risk-informed approach for adding six theory and thermodynamics questions by removing questions from other parts of the examination. This was done in a way that balanced the question sampling to ensure adequate coverage of safety-significant topics. Based on my review, I determined that the staffs basis for using NUREG/BR-0122 Revision 5 to inform their work continues to be appropriate.

I would also like to note an important clarification/correction to a statement contained on Page 5 of the submitters appeal. The appeal states that Revision 9 of NUREG-1021 was used to inform the development of the sample plan as specified in Revision 12 of NUREG-1021. Revision 9 of NUREG-1021 was not utilized in the development of the sample plan. As specified on Page 9 of my Directors Decision memorandum, the document was referenced to clarify that the content of NUREG/BR-0122 was integrated into NUREG-1021.

B. Content Validity for the Reactor Operator Written Examination in NUREG-1021 Revision 12 has been lost In their appeal, the submitters stated, in part, that based on the reduction of component sampling by approximately a factor of 10, and the sampling of reactor theory and thermodynamics being reduced in half (by percentage) it was their belief that the content validity of NUREG-1021 has been lost. Additionally, the submitters stated that the staff may be inappropriately relying on other sections of the examination to make up for the reduction in generic fundamental questions being asked.

As stated in my Directors Decision memorandum, the staff assessed that, although the reactor theory and thermodynamics topics have a similar importance to safety as other topics, they were being tested 3-5 times more frequently than those other topics. A reduction in the number of questions (or percentage coverages) by default does not mean that the content validity of the examination has been reduced to the point where the operator licensing examinations can no longer discriminate between a competent operator and one who is not competent. The staff did not arbitrarily eliminate test items just to reduce the number of questions. Instead, the staff used a combination of risk-informing the importance of safety topics, and their experience as expert qualified licensed examiners (subject matter experts) to evaluate whether all topics were being tested consistent with their contribution to safety and their protection function. The methodology Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information used to accomplish this is described on pages 3 to 6 of my Directors memorandum. I would also note that the changes to NUREG 1021 were subject to a regional review and comment process that leveraged the combined knowledge of Branch Chiefs and Division Directors from all four Regional Offices, and NRR. The technical points raised by the submitters were discussed as part of this process and given proper consideration.

I would also like to note an important clarification/correction to a statement contained on Page 7 of the submitters appeal. The appeal states that the staff is relying on other sections of the reactor operator written examination to make up for a drastic reduction in the number of theory and component questions being sampled in NUREG-1021. This statement was made in reference to information contained on pages 2 and 3 of my Directors Decision memorandum. It references a statement that says that fundamentals are inherently tested, to varying degrees, in other questions. The staff is not relying on other areas of the examination for testing reactor theory and thermodynamics; rather, the statement was meant to acknowledge that there is some level of defense-in-depth in the fact that, without proper knowledge of reactor theory and thermodynamics, an applicant may not be able to effectively assess plant conditions to determine the appropriate response.

Regarding the reductions to the number of questions related to the components topic, as part of their work, the staff reviewed the requirements contained in Title 10 of the Code of Federal Regulations (10 CFR) 55.41, Written Examinations: Operators. This regulation requires that the reactor operator written examination contains a representative sample from among 14 items, to the extent applicable to the facility. Through their review, the staff determined that, although components have historically been tested as a stand-alone topic, components is not listed as a stand-alone testable item in the regulation. Instead, it is listed under 10 CFR 55.41 items (3), (6), (7), and (8), as shown below:

10 CFR 55.41(b)(3) - Mechanical components and design features of the primary system

10 CFR 55.41(b)(6) - Design, components, and functions of reactivity control mechanisms and instrumentation

10 CFR 55.41(b)(7) - Design, components, and functions of control and safety systems, including instrumentation, signals, interlocks, failure modes, and automatic and manual features

10 CFR 55.41(b)(8) - Components, capacity, and functions of emergency systems Under each of these items, components is referenced in the context of specific systems, and in no case is there a stand-alone component testable item. This formed the basis for the decision that just testing generic components (for example, how does a motor operated valve work) is not required by regulation but testing a representative sampling of components in the context of a system is required. Therefore, the staff determined that it was not necessary to have a separate section or Tier of the examination to cover component knowledge and that it was instead more consistent with the regulation to include these component knowledge and abilities (K/As) into testing as part of Tier 2 (Systems). This was the logic used as part of the sample plan development to allow for zero to four component questions to be tested. As part of the review for this appeal, the staff performed a review of a sample of the current K/A catalogs.

The review showed that components are being tested in the context of a system.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information C. The methodology used to risk-inform the development of NUREG-1021 Revision 12 sample plan was flawed In their appeal, the submitters based their conclusion that the methodology used to risk-inform the development of the sample plan was flawed, in part, on: 1) utilizing the importance rating (IR) in a different way than their original intent; 2) the averaging of IRs performed by the staff would lead to comparing an average of an average, which is only mathematically correct when the number of items compared is the same; 3) the belief that the importance to safety of theory and components K/As cannot be directly compared to the importance to safety of individual systems K/As; and 4) the belief that removing questions from other sections of the reactor operator examination results in content validity of the other sections being reduced.

Regarding the utilization of IRs as part of the staffs analysis, as stated by the submitter, the IRs were originally developed to provide a qualitative measure to determine if a K/A is important enough to be used on a reactor operator licensing examination. Although the original intent was for IRs to provide a qualitative assessment, that does not negate the fact that IRs can provide risk-insights on the importance to safety of a topic. As an evidence-based organization we make regulatory decisions using available data. The data provided from analysis of the IRs, in conjunction with the knowledge of subject matter experts provides for a balanced, evidence-based, risk-informed process that is described in pages 3 to 6 of my Directors Decision memorandum. With regard to the submitters statement on averaging IRs, I agree with the submitters that to effectively compare an average of an average the number of items being compared would have to be similar. A discussion with the person responsible for the evaluation revealed that this was taken into consideration and the numbers were normalized (i.e., a weighted average was used to account for changes to the number of questions when conducting the qualitative assessment.)

The submitters contend that the logic used for comparing the importance of safety systems and for removing questions from other areas of the examination was inappropriate. As stated in my Directors Decision memorandum, the staff used a risk-informed approach for adding six theory and thermodynamics questions by removing questions from other parts of the examination that were deemed to have the least relative importance to safety. The examination is constructed in a way that the most important areas are tested at higher frequency and the areas deemed less important are sampled less frequently. For this reason, the staff elected to reduce questions from areas that were deemed not as important to safety, even if they are not sampled 100 percent of the time.

D. Meeting 10 CFR 55.41, Written Examinations: Operators, does not by itself, ensure a reliable content valid examination The submitters stated that the logic expressed in my Directors Decision memorandum could infer that an examination with 15 questions that contain a representative sample of all 14 items would meet 10 CFR 55.41. Although the NRC has not developed a definition of what constitutes a representative sample, an NRC staff position1 states that a "representative sample," should include a reasonably complete, thorough, balanced, and varied cross section of the items in the 1 This position is documented under item 3.1.1 of a of a publicly available document called Operator Licensing Program Feedback, which is a program that provides an aid for understanding elements of the operator licensing program (ML22153A477).

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information population to be sampled. As stated by the submitters, 10 CFR 55.40, Implementation, NUREG-1021 is required to be used to prepare the reactor operator written examination required by 10 CFR 55.41. Through the development of the new sampling plan contained in NUREG-1021, the staffs intent was to construct a plan that will result in a reasonably complete, thorough, balanced, and varied cross section of the items in the population, hence a representative sample. As described in my Directors Decision memorandum, this was done in a way that balanced the question sampling to ensure adequate coverage of safety-significant topics. Therefore, the logic used in my Directors Decision memorandum does not imply that an examination with 15 questions that contain a representative sample of all 14 items would meet 10 CFR 55.41. Instead, it supports balanced representation of safety-significant topics that meet 10 CFR 55.41. I would also note that the DPO Ad Hoc Review Panel also reached the conclusion that the staffs approach meets 10 CFR 55.41.

E. The GFE reintegration Pilot Program was of very narrow scope and did not provide a statistical justification for reducing the number of generic fundamentals questions to the level specified in NUREG-1021 Revision 12 In their appeal, the submitters stated that the industry-led pilot was of very narrow scope as it only evaluated four facilities in contrast to the pilot that was used to inform splitting the GFE from the operator reactor examination, which included 19 facilities. They also expressed concerns with the level of review of the information provided by the industry-led pilot.

Not every change performed by the agency is evaluated through a pilot. As part of the reintegration of the GFE into the reactor operator examination the staff considered whether an NRC-led full-scale pilot was necessary to justify the proposed changes. After careful consideration, the staff concluded that the reintegration of the GFE into the reactor operator examination could be performed without the performance of a full-scale pilot like the one described in SECY-89-292 (ML12251A650). This decision was reached, in part, because the operator licensing program is a mature program, and the NRC has over 30 years of experience in the performance and administration of examinations. The information provided by the limited scope industry-led pilot did not constitute the sole basis of acceptability of the change. As stated in my Directors Decision memorandum, the methodology used by the staff relied on various factors. The information provided by the industry-led pilot was used to assess the impact of the proposed changes to the operator licensing examination and was done in the context of an audit examination, and for validation. It was not intended to be the sole basis to demonstrate the feasibility of reintegration of the GFE into the reactor operator examination.

Regarding the staffs level of review of the information provided by the industry-led pilot; three of the four stations submitted their questions for review by NRR staff. Station 4, whose results showed that there was a need to better maintain theory knowledge throughout training, was one of the three submitters. During their review of submitted questions, qualified examiners followed the same process they would use during a standard reactor operator examination to determine if the questions met the standards for quality. Additionally, the staff determined that not having the four sets of questions submitted for review did not invalidate the data provided by the industry, as there was no indication that the data was manipulated or skewed. For example, there was no case for which all the examinees got a question right or wrong. This was acceptable as the information utilized did not constitute the sole basis of acceptability of the changes to the reactor operator examination.

Regarding the example brought up by the submitters, which they considered most chief examiners would not allow in an examination, there is a certain level of subjectivity when Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information evaluating whether a question meets the guidelines of NUREG-1021. Some examiners may find a question acceptable while some may not. Through a conversation with qualified examiners, they deemed the question as meeting the guidelines of NUREG-1021. Additionally, I would note that this question was also similar in constitution to questions that have been allowed in previous versions of the GFE.

F. Inspection Procedure (IP) 71111.11 does not provide a method for the NRC to evaluate the generic fundamentals knowledge of individual licensed operators In their appeal, the submitters state that IP 71111.11 does not provide a method for the NRC to evaluate generic fundamentals knowledge. They stated that on at least one occasion, inspectors did not identify a negative trend in generic fundamental knowledge prior to an event occurring. Additionally, they stated that it is their view that the biennial portion of the procedure also does not allow for a reliable method for evaluating generic fundamentals knowledge.

In my Directors Decision memorandum, I noted that the NRC does not rely on a one-time examination to maintain proper knowledge on how to respond to emergencies. I also stated that a combination of ongoing oversight through inspections such as IP 71111.11, and the NRCs analyses of licensees event reports, inspection findings, and industry databases would allow for the NRC to determine if additional actions were warranted to mitigate a negative trend in operator performance. The Reactor Oversight Process is a robust oversight program that remains effective in enabling the NRC to independently review reactor operator performance and respond to safety-significant issues in a timely manner. One example where inspectors did not document a negative trend prior to an event does not indicate a deficiency in the entire program; however, I acknowledge that enhanced guidance in IP 71111.11 may provide an opportunity for additional defense-in-depth through ongoing firsthand observations of trends in generic fundamental knowledge, beyond the initial licensing examination.

G. Reasons why the Agency should come to a different conclusion In their appeal, the submitters enumerated nine reasons to support their views that the agency should not have reintegrated the GFE into the reactor operator licensing program as described in Revision 12 of NUREG-1021. The information provided is the basis of their DPO submittal, has been evaluated by the DPO Panel, and is addressed in my Directors Decision memorandum. Although my memorandum addresses how the concerns were evaluated, there are some items that I would like to highlight in this discussion. In their appeal, the submitters stated, in part, that the changes made in Revision 12 have effectively reversed all the changes to operator licensing written examination put in place as part of the Three Mile Island (TMI) lessons learned. NUREG-0737, Clarification of TMI Action Plan Requirements, established, in part, the following corrective actions relating to generic fundamentals:

A new category shall be added to the operator examination entitled, Principles of Heat Transfer and Fluid Mechanics.

A new category shall be added to the senior reactor operator entitled, Theory of Fluids and Thermodynamics.

As stated in my Directors Decision memorandum, the corrective actions for TMI (referenced above) are still being tested under reactor theory and thermodynamics and were reintegrated into the reactor operator examination. Therefore, the changes in Revision 12 of NUREG-1021 did not eliminate corrective actions put in place as part of the TMI lessons learned. The key consideration is how much weight should be put on testing fundamentals. This consideration Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information was part of the staffs assessment in support of reducing the number of questions focused only on fundamentals when reintegrating the GFE into the reactor operator examination.

Regarding operator training, the NRC does not specify the length of time a licensee should spend training an operator in fundamentals. The standards delineating what an operator should master in the area of fundamentals are contained in ACAD 10-001, Guidelines for Initial Training and Qualification of Licensed Operators. Those standards have not changed from what was required prior to Revision 12 of NUREG-1021. Operators are still being taught fundamentals and the changes implemented as part of NUREG-1021 do not interfere with the NRCs ability to assess the cognitive capabilities of an operator (e.g., comprehension, problem solving, and decision-making). The NRC monitors licensee performance and accreditation of the operator training programs. If a program loses its accreditation or if operator performance declines at a facility, the NRC would be able to take appropriate actions such that operators can safely operate a plant.

Additional Actions by the Office of Nuclear Reactor Regulation Based on my review of the appeal I conclude that the submitters did not raise any issues that fundamentally impact the conclusions documented in my Directors Decision memorandum.

However, the information presented in the appeal justified the following additional actions to be tasked within 14 days following the issuance of the statement of views memorandum.

1.

I am tasking NRRs Division of Reactor Oversight to: 1) evaluate if additional guidance and/or training is required to ensure that inspectors would be able to recognize degradation in reactor operator knowledge of generic fundamental topics through IP 71111.11; and 2) provide for a mechanism for documenting weaknesses noted in the area of generic fundamentals during performance of IP 71111.11 such that they can be reviewed as part of the planned Effectiveness Review of Revision 12 of NUREG-1021.

I want to also take this opportunity to thank the submitters once again for raising this DPO and for their active participation in the process. While I do not agree with the submitters assertions that the changes to NUREG 1021 diminished the NRCs ability to make accurate licensing decisions for operator license applicants, their willingness to raise concerns is admirable and vital to ensuring a healthy safety culture within our agency.

Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information Letter to Daniel Dorman from Andrea Veil dated August 17, 2022

SUBJECT:

STATEMENT OF VIEWS REGARDING APPEAL OF DIFFERING PROFFESIONAL OPINION DECISION INVOLVING THE OPERATOR LICENSING EXAMINATION STANDARDS FOR POWER REACTORS, NUREG-1021, REVISION 12 (DPO-2021-002)

ADAMS Accession Number: ML22228A068 Publicly Available Non-Publicly Available Sensitive A7 Non-Sensitive To receive a copy of this document, indicate in the concurrence box "C" = Copy without attach/encl "E" = Copy with attach/encl "N" = No copy OFFICE RIII RIII RIII NAME DBetancourt AVeil DATE 8/15/22 8/17/22 OFFICIAL RECORD COPY Official Use Only - Sensitive Internal Information Official Use Only - Sensitive Internal Information

Document 7: DPO Appeal Decision

0FFIOAL USE ONLY-SENSmVE INTERNAL INFORMATION MEMORANDUM TO:

FROM:

SUBJECT:

UNITED STATES NUCLEAR REGULA TORY COMMISSION WASHINGTON, D.C. 20555-0001 February 6, 2023 Daniel Bacon, Senior Operations *Engineer Joseph Demarshall, Senior Operations Engineer Kevin Murphy, Operations Engineer Peter Ott, Operations Engineer Randy Baker, Senior Operations Engineer Chuck Zoia, Senior Operations Engineer Bryan Bergeon, Operations Engineer Mark Bates, Senior Operations Engineer Michael Donithan, Operations Engineer Andreas Goldau, Operations Engineer Michael Kennard, Senior Operations Engineer Kevin Kirchbaum, Operations Engineer Newton Lacy, Operations Engineer David Lanyi, Senior Operations Engineer Michael Meeks, Senior Operations Engineer Joseph Viera, Senior Operations Engieer Daniel H. Dorman 'l'v. * -.. ~ ~~

Executive Directorfo~

DIFFERING PROFESSIONAL OPINION APPEAL DECISION INVOLVING DP0-2021-002 The purpose of this memorandum is to inform you of my considerations and condusions regarding the Differing Professional Opinion (DPO) appeal you submitted on June 21, 2022 (Agencywide Documents Access and Management System Accession No. ML22230A850). The appeal raised concerns regarding the U.S. Nuclear Regulatory Commission's (NRC's) alleged flawed methodology and logic used to reintegrate the generic fundamentals examination (GFE) into the Reactor Operator (RO) written examination specified in NUREG-1021, "Operator Licensing Examination Standards for Power Reactors," Revision 12 (ML21256A276).

Your DPO appeal raised seven specific issues. A paraphrased summary of issues and my conclusions for each are as follows:

Issue 1: NUREG-1021, Revision 12, diverges from requirements specified in NUREG-0737, Clarification of Three Mile Island Action Plan Requirements. Specifically, the requirement for a minimum passing score for each category of the written examination was removed by NUREG-1021, Revision 12. The two Commissioners Assistant Notes ( CA Notes) used to communicate the changes made by NUREG-1021, Revision 12, to the Commission did not mention this fact.

There was no SECY paper associated with this major change in testing requirements.

CONTACT: Hector Rodriguez-Luccioni, OEDO 301-415-6004

D. Bacon, et al.

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION Answer 1: Based on my review, the changes made to NUREG-1021, Revision 12, to reintegrate the generic fundamentals questions into the RO written examination were acceptable because the requirements for the written examination are contained in Title 10 of the Code of Federal Regulations (10 CFR) part 55 and not in NUREG-0737. The two CA Notes informed the Commission of the reintegration of the GFE back into the RO written exam and there is no requirement for a SECY paper.

Issue 2: Content validity and test reliability for the RO written examination in NUREG-1021 Revision 12 has been lost. This can be proved by using the same method that Dr. George M.

Usova used to evaluate a LaSalle RO/SRO written exam that had numerous questions removed in 1993. This evaluation is listed on Nuclepedia.

Answer 2: Based on my review, the reduction in the number of questions being tested does not mean that the content validity of the examination has been lost.

Issue 3: NUREG/BR-0122 Revision 5 was incorrectly used as a basis for reintegrating the GFE into the RO written examination as specified in NUREG-1021 Revision 12. Revision 4 of NUREG/BR-0122 is the appropriate version to use because it was in effect prior to splitting out the GFE from the RO/SRO written examinations and contained the sample plans that had the proper percentage of questions from each category based on the initial Job Task Analysis that is still in effect Revision 4 was developed by a professional educational services company using recognized standards.

Answer 3: Based on my review, the NRC used the guidance contained in NUREG-1021 as the basis for reintegrating the GFE back into the RO written examination. The Operator Licensing and Human Factors branch used information from NUREG/BR-0122, Revision 5, to inform their work, but it was not used as the basis for the final sample plan.

Issue 4: The methodology used to risk-inform the development of the NUREG-1021 Revision 12 sample plan was flawed. A lack of Generic Fundamentals knowledge led to the only core melt of a power reactor in the U.S.

Answer 4: Based on my review, the NRC used available information and relied upon subject matter experts to develop a methodology to implement changes to the RO written examination, and there is not sufficient evidence to conclude that methodology is flawed.

Issue 5: Meeting 1 O CFR 55.41 does not, by itself, ensure a reliable content valid examination. The use of NUREG-1021 to develop the examinations is specified in 10 CFR 55.40. An improper change to the sample plan contained in NUREG-1021 can, and has, led to a loss of reliability and content-validity of the RO written examination.

Answer 5: Based on my review, I have concluded that the changes to the sample plan contained in NUREG 1021 were not improper and did not result in a loss of reliability and content-validity of the examination.

Issue 6: The GFE Reintegration Pilot Program was of very narrow scope, was not uniformly administered, and did not provide a statistically meaningful justification for reducing the number of generic fundamentals questions to the level specified in NUREG-1021 Revision 12.

D. Bacon, et al.

OFFICIAL USE ONLY-SENSITIVE INTERNAL INFORMATION Answer 6: Based on my review, I agree that the scope of this GFE Reintegration Pilot Program was narrow when compared to the pilot program that was used to inform splitting out the GFE as described in SECY-89-292 (ML12251A650). However, the GFE Reintegration Pilot Program was not the sole basis used to reintegrate the GF questions into the RO exam and therefore the methodology used was reasonable and did not violate any requirements. Furthermore, the methodology results in an exam that still tests the requisite sample of generic fundamentals questions.

Issue 7: IP 71111.11, does not provide a method for the NRC to evaluate the generic fundamentals knowledge of individual licensed operators. Additionally, only approximately 20%

of the licensee crews are evaluated during a biennial requalification program inspection. With crews being comprised of 5-7 operators containing a shift manager, shift technical assistant, and possibly more than one SRO, an operator on the crew with a solid knowledge of fundamentals could easily compensate for and prevent the identification of others that do not.

Answer 7: Based on my review, I agree that enhanced guidance in IP 71111.11 may provide an opportunity for additional defense-in-depth. The Director of the office of Nuclear Reactor Regulation (NRR) also acknowledged that IP 71111.11 could use enhanced guidance and directed the Division of Reactor Oversight to revise IP 71111.11 to add enhanced guidance for inspectors to aid in identifying deficiencies in generic fundamental knowledge.

After careful consideration of your appeal, I concluded that the methodology and logic for reintegrating the GFE into the RO written examination specified in NUREG-1021, Revision 12, are acceptable. The written examination is prepared in accordance with NU REG 1021 as required by 10 CFR 55.40 and NUREG 1021, Revision 12, meets the requirements in 10 CFR part 55.41. However, there are enhancements needed to the NRC's effectiveness review and inspection guidance to monitor for any unintended consequences resulting from the changes.

As such, I direct NRR to do the following:

1. Revise the Effectiveness Review Plan for NUREG-1021, Revision 12, Operator Licensing Examination Standards for Power Reactors (ML22138A409). Specifically, Task 2a should:

a. have a sufficient duration to ensure that meaningful data is collected for generic fundamentals questions from applicants who had NOT previously taken a stand-alone NRC Generic Fundamentals Examination;

b. include guidance to review any questions that may indirectly/implicitly test generic fundamentals and were incorrectly answered and include that data in the effectiveness review; and

c. contain specific thresholds for performance that will trigger revisiting the sample plan distribution contained in NUREG-1021, Revision 12.

I want to thank you for bringing your concerns to my attention. I appreciate you taking the time to document and share your concerns. Your willingness to raise concerns through the DPO process is consistent with our organizational values of Openness and Commitment. Our agency relies on dedicated professionals, such as yourself, who are willing to raise concerns that could

D. Bacon, et al.

OFFICIAL USE ONLY-SENSITIVE INTERNAL INFORMATION impact the NRC mission. More in depth analysis of each of the issues you raised is provided in the enclosure.

In accordance with Management Directive 10.159, "NRC Differing Professional Opinion Program," a summary of this appeal decision will be included in the Weekly Information Report posted on the NRC's public website to advise interested employees and members of the public of the outcome.

Enclosure:

Deputy Executive Director for Reactor Programs-Led Appeal Review Team Analysis cc: D. Roberts, OEDO H. Rodriguez-Luccioni, OEDO A. Veil, NRR D. Betanourt-Roldan, RIii B. Orlikowski, RIii G. Callaway, OCHCO T. Martinez Navedo, OE G. Figueroa Toledo, OE

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMAITON Enclosure DEPUTY EXECUTIVE DIRECTOR FOR REACTOR PROGRAMS-LED APPEAL REVIEW TEAM ANALYSIS To better understand your concerns, I assigned the Deputy Executive Director for Reactor Programs (DEDR), an Executive Technical Assistant from my office, a subject matter expert from the Office of Chief Human Capitol Officer, and a subject matter expert from Region III. This DEDR-led appeal review team gathered information through discussions with you, the Director of the Office of Nuclear Reactor Regulation (NRR), the Differing Professional Opinion (DPO)

Panel, members of The Operator Licensing and Human Factors branch (IOLB), and other knowledgeable staff who reviewed documents pertinent to your appeal. The appeal review team also gathered additional information through their own independent reviews of the NUREG-1021 revision and other agency documents. The information collected provided independent insights and perspectives for my consideration.

On September 27, 2021, you submitted a DPO on Operator Licensing Generic Fundamentals Examination (Agencywide Documents Access and Management System Accession Nos.

ML21322A200 and ML21322A199). On October 19, 2021, an Ad Hoc Review Panel was formed and tasked by the U.S. Nuclear Regulatory Commission (NRC) Differing Views Program to review your DPO. The DPO Panel subsequently issued their findings report to the Director of NRR on April 5, 2022 (ML22095A246). With respect to the concerns discussed in your appeal, the DPO Panel concluded that: 1) the NUREG-1021 Operator Licensing Examination Standards for Power Reactors, Revision 12, method of testing generic fundamentals, namely, Reactor Theory, Thermodynamics, and Components, in the Reactor Operator (RO) licensing exam, does not determine if the applicant has mastered this subject matter; 2) reducing the number of generic fundamental questions has lowered the content validity of the exam; and 3) the RO exam does meet Title 10 of the Code of Federal Regulations (10 CFR) 55.41, Written Examination: Operators.

On May 20, 2022, the Director of NRR issued their decision regarding the DPOs concerns as informed by the DPO Panel report and their own review (ML22139A175). The Director did not agree with the submitters assertion that the changes to NUREG-1021 diminish the NRCs ability to make accurate licensing decisions for operator license applicants. Although the Director is confident that the staffs approach for incorporating the generic fundamentals examination (GFE) into the reactor operating examination was valid, the Director found that the basis for making changes to NUREG-1021 was not sufficiently documented by the staff. For this reason, the Director tasked the staff to document the basis for integrating the GFE examination into the operator licensing examination, the methodology used to develop the new sampling plan, and the basis for determining that the reactor operator examinations ability to discriminate between a competent operator and one who is not, has not been adversely affected. The Director agreed with the DPO panel that a best practices document that provides a standard framework for working groups is one way to improve communication and governance. The Director tasked the staff to develop a best practices document to provide additional clarity and an operating framework for working groups.

On June 27, 2022, you submitted an appeal (ML22230A850). This appeal stated seven continuing issues:

Issue 1: NUREG-1021, Revision 12, diverges from requirements specified in NUREG-0737, Clarification of Three Mile Island (TMI) Action Plan Requirements. Specifically, the requirement for a minimum passing score for each category of the written examination was removed by NUREG-1021, Revision 12. The two Commissioners Assistant Notes (CA Notes) used to

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION 2

communicate the changes made by NUREG-1021, Revision 12, to the Commission did not mention this fact. There was no SECY paper associated with this major change in testing requirements.

Issue 2: Content validity and test reliability for the RO written examination in NUREG-1021 Revision 12 has been lost. This can be proved by using the same method that Dr. George M.

Usova used to evaluate a LaSalle RO/SRO written exam that had numerous questions removed in 1993. This evaluation is listed on Nuclepedia.

Issue 3: NUREG/BR-0122 Revision 5 was incorrectly used as a basis for reintegrating the GFE into the RO written examination as specified in NUREG-1021 Revision 12. Revision 4 of NUREG/BR-0122 is the appropriate version to use because it was in effect prior to splitting out the GFE from the RO/SRO written examinations and contained the sample plans that had the proper percentage of questions from each category based on the initial Job Task Analysis that is still in effect. Revision 4 was developed by a professional educational services company using recognized standards.

Issue 4: The methodology used to risk-inform the development of the NUREG-1021 Revision 12 sample plan was flawed.A lack of Generic Fundamentals knowledge led to the only core melt of a power reactor in the U.S.

Issue 5: Meeting 10 CFR 55.41 does not, by itself, ensure a reliable content valid examination.The use of NUREG-1021 to develop the examinations is specified in 10 CFR 55.40. An improper change to the sample plan contained in NUREG-1021 can and has led to a loss of reliability and content-validity of the RO written examination.

Issue 6: The GFE Reintegration Pilot Program was of very narrow scope, was not uniformly administered, and did not provide a statistically meaningful justification for reducing the number of generic fundamentals questions to the level specified in NUREG-1021 Revision 12.

Issue 7: IP 71111.11, does not provide a method for the NRC to evaluate the generic fundamentals knowledge of individual licensed operators. Additionally, only approximately 20%

of the licensee crews are evaluated during a biennial requalification program inspection. With crews being comprised of 5-7 operators containing a shift manager, shift technical assistant, and possibly more than one senior reactor operator, an operator on the crew with a solid knowledge of fundamentals could easily compensate for and prevent the identification of others that do not.

The appeal review teams assessment is limited to these issues. The Director of NRR issued their statement of views on August 17, 2022 (ML22228A068).

Response to Issue 1 Your appeal stated that NUREG-1021, Revision 12, diverges from requirements specified in NUREG-0737, Clarification of TMI Action Plan Requirements. Specifically, the requirement for a minimum passing score for each category of the written examination was removed by NUREG-

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION 3

1021, Revision 12. Additionally, you stated that there was no SECY paper associated with this divergence.

By way of background, NUREG-0737 incorporates in one document, all TMI-related items approved for implementation by the Commission as of October 31, 1980. Most of the items had already been issued as requirements by previous correspondence. The relevant requirements for written examinations repeated in NUREG-0737 are from Enclosure 1 Criteria for Reactor Operator Training and Licensing, of a letter from H. R. Denton, Director, NRR, to All Power Reactor Applicants and Licensees, dated March 28, 1980,

Subject:

Qualifications of Reactor Operators. Enclosure 1 to the March 28, 1980, letter stated in part that a new category shall be added to the operator written examination entitled, Principles of Heat Transfer and Fluid Mechanics, and that a new category shall be added to the senior operator written examination entitled, "Theory of Fluids and Thermodynamics." At the time, the regulations governing the content of operator and senior operator written examinations (then in 10 CFR §§ 55.21 and 55.22) did not have those categories. Further, Enclosure 1 to the March 28, 1980, letter stated that The passing grade for the written examination shall be 80%

overall and 70% in each category.

Significantly, however, in the March 28, 1980, letter, the Director states: Commission review in the area of operator training and qualification is continuing and can be expected to result in additional criteria. Finally, requirements will be established through rule making proceedings (emphasis added). In other words, the March 28, 1980, letter, which is repeated in NUREG-0737, forecast that it would be superseded. As explained below, the Commission did not include all the 1980 requirements when the Commission subsequently set for TMI-related changes to operator licensing.

In 1984, the Commission published proposed amendments to its regulations to, among other things, revise the requirements and scope of written examinations for operators and senior operators. Proposed Rule, Operator's Licenses and Conforming Amendment, 49 Fed. Reg.

46,428 (November 26, 1984). The Commission stated the revisions were needed to, among other things, reflect upgraded requirements for licensed operator selection, training, and requalification program resulting from the accident at TMI-2. Id. at 46,429. Under the proposal, the Commission would develop written examinations by using the topics in new 10 CFR §§ 55.41 and 55.43 (instead of 10 CFR §§ 55.21 and 55.22), information from industry task analyses, experts and studies in the field, and information from the facility licensees and from within NRC. Id. In 1987, the NRC finalized the rule and promulgated §§ 55.41 and 55.43. Final Rule, Operators Licenses and Conforming Amendments, 52 Fed. Reg. 9453 (March 25, 1987).

The final rule contained some items similar to the March 28, 1980, letter and enclosures, but not all items from the letter became final regulations. For example, the March 28, 1980, letter stated to add a category to the reactor operator written examination entitled Principles of Heat Transfer and Fluid Mechanics and the final rule in 10 CFR 50.55.41(b)(14) says that exams sample the similar category of Principles of heat transfer thermodynamics and fluid mechanics. However, the final rule in 55.43 did not include the March 28, 1980, requirement to include for the senior operator a category of Theory of Fluids and Thermodynamics.

Significantly, the 1987 rulemaking did not include a requirement to achieve 70% in each category (i.e., 70% for each item sampled and listed in 10 CFR 55.41(b)(1)-(14) (for operator tests) and 55.41(b)(1)-(14) and 55.43(b)(1)-(7) (for senior operator tests)). If the Commission wished to maintain the requirements set forth in the March 28, 1980, letter, as reflected in

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION 4

NUREG-0737, then the Commission would not have set forth different requirements in the March 25, 1987, final rule.

The DPO appeal review team reviewed the two CA Notes used to communicate the changes made by NUREG-1021, Revision 12, to the Commission. While neither note explicitly stated that the GFE questions were no longer being graded as a separate category, there is no evidence that the CA Notes intentionally excluded this fact. The CA Notes clearly stated that the general fundamental topics would be integrated into the facility-specific reactor operator NRC administered initial exam. The NRC followed Management Directive 3.7, NUREG-Series Publications, NUREG-0650, Preparing NUREG Series Publications, and OLMC-110, Control of Operator Licensing Guidance Documents to prepare NUREG-1021, Revision 12. These documents outline the process for revising a NUREG and include guidance for required reviews and necessary authorizations for publication.

Response to Issue 2 Your appeal stated that content validity and test reliability for the RO written examination in NUREG-1021 Revision 12 has been lost. In your argument you compared the number of fundamental questions based on the targeted coverage of a 75-question RO exam prepared in accordance with NUREG/BR-0122 Revision 4 to the coverage of a 75-question exam developed using NUREG-1021 Revision 12.

Content validity is the degree to which a test measures the specific objectives or content. In the 1993 LaSalle operator licensing examination validity determination memo you referenced in your DPO appeal, Dr. Usova states that Validity is a matter of degree. It does not exist on an all or none basis. He also states that No exam, even under the best circumstances, can test all of what needs to be known or performed. Because of this limitation, we must sample knowledge and abilities.

The DPO appeal review team has determined that a reduction in the number of questions being tested does not mean that the content validity of the examination has been lost. In developing NUREG-1021, Revision 12, the NRR staff used information available and their experience and knowledge as subject matter experts to implement a methodology that meets the requirements of 10 CFR 55.41. Specifically, the written examinations are developed and verified to contain questions on the knowledge, skills, and abilities needed to perform licensed operator duties and include a representative sample from among the 14 items listed in 10 CFR 55.41(b).

Response to Issue 3 Your appeal stated that NUREG/BR-0122 Revision 5 was incorrectly used as the basis for reintegrating the GFE into the RO written examination as specified in NUREG-1021 Revision

12.

The Division of Reactor Oversight (DRO) was tasked with documenting the basis for integrating the generic fundamentals examination into the operator licensing examination. The tasking stated that the documentation should include the methodology used to develop the new sampling plan and the basis for determining that the reactor operator examinations ability to discriminate between a competent operator and one who is not, has not been adversely affected. DRO documented this basis in a memorandum from The Director, DRO, to the Director, NRR (ML22271A851).

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION 5

The DPO appeal review team has determined that the methodology used by DRO to reintegrate the GFE into the RO written examination did not rely solely on NUREG/BR-0122 Revision 5.

NUREG-1021 contains the guidance, including the sampling plans, for developing the RO written examinations. IOLB used the existing sampling plans in NUREG-1021 as the starting point for developing a revised sample plan that incorporated generic fundamental questions into the RO written examination. IOLB evaluated the sampling plans to determine where to reduce questions to make room for the generic fundamentals questions without significantly impacting the validity of the examination. IOLB used the information in NUREG/BR-0122, Revision 5, to gain insights on the various Groups and Tiers of questions and their relative nexus to safety.

With this information, IOLB modified the NUREG-1021 sample plans to incorporate generic fundamental questions to maintain content validity of the RO written examination.

The DPO appeal review team reviewed the effectiveness review plan (ML22145A456) that will assess the impact of the GFE reintegration. The DPO appeal review team identified several enhancements that could be incorporated to the effectiveness review plan. For this reason, I am directing NRR to increase the duration of the effectiveness review to gather sufficient data from applicants who have not taken a stand-alone GFE, expanding the data review to include questions that implicitly test generic fundamentals knowledge, and defining specific thresholds for performance that will trigger revisiting the sample plan distribution contained in NUREG-1021, Revision 12.

Response to Issue 4 Your appeal stated that the methodology used to risk-inform the development of the NUREG-1021 Revision 12 sample plan was flawed. Specifically, you took exception to the method used by IOLB to determine how the GFE was reintegrated back into the reactor operator written exam. The methodology involved the use of Importance Ratings of the Knowledge and Abilities (K/As) to determine what areas of the exam would include few questions to add GFE questions to maintain a total of 75 questions on the RO exam.

The Director of NRR agreed with your statement that the Importance Ratings (IRs) were originally developed to provide a qualitative measure to determine if a K/A is important enough to be used on a reactor operator licensing examination. The Director further explained that the IRs can provide risk-insights on the importance to safety of a topic, and when those insights are used in conjunction with the knowledge of subject matter experts, the methodology used provided for a balanced, evidence-based, risk-informed process.

The DPO appeal review team agrees that the NRC used available information and relied upon subject matter experts to develop a methodology to implement changes to the RO written examination, and there is not sufficient evidence to conclude that methodology is flawed. The DPO review team reviewed the Effectiveness Review Plan for NUREG 1021, Revision 12, and determined that it contains specific tasks requiring IOLB to gather data and review the changes incorporated in Revision 12 to determine if there are unintended consequences such as a loss of content validity regarding the GFE topics.

Response to Issue 5 Your appeal stated that 10 CFR 55.41 does not, by itself, ensure a reliable content-valid examination. You further stated that an improper change to the sample plan contained in

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION 6

NUREG-1021 can and has led to a loss of reliability and content-validity of the RO written examination.

The DPO review team agrees that there is more to examination validity than just meeting the requirements of 10 CFR 55.41, which states the content requirements for the written examination for operators. 10 CFR 55.41 requires that the written examination for an operator contain a representative selection of questions on the knowledge, skills, and abilities needed to perform licensed operator duties. 10 CFR 55.41 also requires the written examination include a representative sample from the list of 14 items in 55.41(b).

The written examination is prepared in accordance with NUREG 1021 as required by 10 CFR 55.40. NUREG 1021 includes detailed guidance for developing exams, including guidance for the sample plan ensuring that the written examination include a representative sample of items specified in 10 CFR part 55.41.

The DPO review team has concluded that the changes to the sample plan contained in NUREG 1021 were not improper and did not result in a loss of the reliability and content-validity of the examination. The CFR requires a representative sampling of the items listed in 10 CFR 55.41(b). While it is true that NUREG 1021 Revision 12 has reduced the number of written examination questions on which an applicant will be tested (since there is no longer a separate 50-question GFE), Revision 12 still tests a representative sample of required items.

Response to Issue 6 Your appeal stated the GFE Reintegration Pilot Program was of very narrow scope, was not uniformly administered, and did not provide a statistically meaningful justification for reducing the number of generic fundamentals questions to the level specified in NUREG-1021 Revision

12.

The DPO appeal review team agrees that the scope of this GFE Reintegration Pilot Program was narrow when compared to the pilot program that was used to inform splitting out the GFE as described in SECY-89-292. The NRR Directors Decision memorandum discussed the methodology used by the staff. The pilot program for splitting out the GFE consisted of two pilot examinations conducted at 77 facilities with 734 applicants. The GFE Reintegration Pilot Program was conducted at 4 facilities with an unspecified number of applicants, which reflected a significant reduction of the sample used to pilot splitting out the GFE.

In the Director of NRRs statement of views memorandum to the Executive Director for Operations, the Director States that not every change performed by the agency is evaluated through a pilot. As part of the reintegration of the GFE into the RO exam the staff determined that a full-scale pilot like the one described in SECY-89-292 was not needed. The pilot that was performed concluded that reintegrating GF questions/topics into the final NRC written exam is viable. The DPO appeal review team determined that this was a reasonable scope for this pilot given the stated goal.

The DPO appeal review team concluded that the GFE Reintegration Pilot Program was not the sole basis used to reintegrate the GF questions into the RO exam. The DPO appeal review team determined that the methodology used was reasonable and did not violate any

OFFICIAL USE ONLY - SENSITIVE INTERNAL INFORMATION 7

requirements. Furthermore, the methodology results in an exam that still tests the requisite sample of generic fundamentals questions.

The DPO appeal team recognizes there could be unintended consequences related to performing the GFE reintegration. To help identify any unintended consequences, IOLB will complete the Effectiveness Review Plan for NUREG 1021, Revision 12, to determine if there are unintended consequences such as a loss of content validity regarding the GFE topic.

Response to Issue 7 Your appeal stated that NRC Inspection Procedure 71111.11, does not provide a method for the NRC to evaluate the generic fundamentals knowledge of individual licensed operators. You then lay out an argument with many valid points to bolster your position.

The DPO review team has determined that IP 71111.11 is not the only inspection activity that would be used to identify a weakness in licensed operator generic fundamental knowledge. The NRCs baseline inspection program includes other inspection activities that observe and review operator performance during reactor operations. Additionally, the NRC analyzes and reviews license event reports, inspection findings, corrective action program documents, and industry databases to identify potential adverse trends or degraded knowledge and performance. NRC inspectors are trained to look for, recognize, and evaluate any deficiencies. The aggregation of these inspection results informs periodic plant performance reviews conducted under the Reactor Oversight Process, which in turn can be used to inform focused inspections of operator performance as needed.

In an August 17, 2022, letter from Ms. Andrea Veil to me, Ms. Veil acknowledged that enhanced guidance in IP 71111.11 may provide an opportunity for additional defense-in-depth through ongoing firsthand observations of trends in generic fundamental knowledge, beyond the initial licensing examination. Therefore, I am directing NRR to revise IP 71111.11 to add enhanced guidance for inspectors to aid in identifying deficiencies in generic fundamental knowledge.

Upon review of the case file for releasability, it was noted that a few ADAMS Accession numbers could be clarified to reference the public and/or the correct version of the document. See below.

SECY 89-292: ML19351A245 NUREG/BR-0122, Revision 5: ML20069B817 NUREG-1021, Revision 9: ML042320438