ML19350C551

DPO-2017-007, DPO Case File (Public)

ML19350C551
Person / Time
Issue date:	01/10/2020
From:	Eugene Guthrie NRC/RGN-II
To:
Figueroa G
References
DPO-2017-007
Download: ML19350C551 (173)

Text

DPO Case File for DPO-2017-007 The following pdf represents a collection of documents associated with the submittal and disposition of a differing professional opinion (DPO) from NRC employees involving Operator Licensing Written Examinations- Tier 1 Test Items.

Management Directive (MD) 10.159, NRC Differing Professional Opinions Program, describes the DPO Program. https://www.nrc.gov/docs/ML1513/ML15132A664.pdf The DPO Program is a formal process that allows employees and NRC contractors to have their differing views on established, mission-related issues considered by the highest level managers in their organizations, i.e., Office Directors and Regional Administrators. The process also provides managers with an independent, multi-person review of the issue (one person chosen by the employee). After a decision is issued to an employee, he or she may appeal the decision to the Executive Director for Operations (or the Commission, for those offices that report to the Commission).

Because the disposition of a DPO represents a multi-step process, readers should view the records as a collection. In other words, reading a document in isolation will not provide the correct context for how this issue was reviewed and considered by the NRC.

It is important to note that the DPO submittal includes the personal opinions, views, and concerns by NRC employees. The NRCs evaluation of the concerns and the NRCs final position are included in the DPO Decision.

The records in this collection have been reviewed and approved for public dissemination.

Document 1: DPO Submittal Document 2: Memo Establishing DPO Panel Document 3: DPO Panel Report Document 4: DPO Decision Document 5: DPO Appeal Document 6: Statement of Views Document 7: DPO Appeal Decision

Document 1: DPO Submittal

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Summary of existing NRC decision or agencys stated position NRR/IOLB determined that Tier 1 written examination test items that test plant system design features, interlocks, and system operation adequately test applicant knowledge of emergency and abnormal evolutions on the site-specific written exam. This determination was documented in Record of Interaction (ROI) 17-09, NUREG 1021, ES-401 Tier 1 Written Exam Test Items (ML17165A579), and was disseminated to industry stakeholders in Operator Licensing Feedback Item 401.55 (ML17249A961).

Operator Licensing Feedback Item 401.55 included two examples of Tier 1 written examination test item topics, one for PWR K/A Abnormal Evolution Pressurizer Pressure Control Malfunctions, and the other for BWR K/A Emergency Evolution High Drywell Pressure, and NRR/IOLBs feedback to industry stated that written examination test items were acceptable for these Tier 1 topics, if the test item solely tested how the system worked (i.e., Tier 2, Plant Systems knowledge aspect).

Reason for DPO The writers of this DPO are identified on Page 17; the purpose of this DPO is to require NRR/IOLB to revise its recent policy determination for writing and assessing Tier 1 written examination test items. NRR/IOLBs position has the potential to undermine the 10 CFR 55.41 requirement that the written examination contain a representative selection of questions on the knowledge, skills, and abilities needed to perform licensed operator duties. Specifically, NRR/IOLBs policy interpretation will result in fewer questions that test the operators knowledge of abnormal and emergency procedures, in accordance with 10 CFR 55.41(b)(10), on the site-specific written examination.

Title 10 of the Code of Federal Regulations (10 CFR) Part 55, Operators Licenses, requires that applicants for reactor operator (RO) and senior reactor operator (SRO) licenses pass a written examination. The regulation at 10 CFR 55.40(b) allows power reactor facility licensees to prepare the site-specific written examinations, provided that the facility licensee prepares the site-specific written examination in accordance with the criteria contained in NUREG-1021, Operator Licensing Examination Standards for Power Reactors.

In accordance with NUREG-1021, ES-401, Preparing Initial Site-Specific Written Examination, the site-specific written exam must be comprised of three parts:

Tier 1: Emergency/Abnormal Plant Evolutions Tier 2: Plant Systems Tier 3: Generic Knowledges & Abilities Interaction between Region II and NRR/IOLB to collaborate on clarifying the intent of Tier 1 Written Exam test items is documented in ROI 17-09 (March-June 2017 time frame). Also a teleconference was held on August 24, 2017, prior to disseminating the Operator Licensing Feedback Item 401.55 to industry stakeholders. NRR/IOLB program office Branch Chief began the teleconference by stating that opinions regarding the final resolution to ROI 17-09 (on 6 17) were not going to be discussed during the teleconference, and said the purpose of the phone call was to discuss enhancements for the phrasing or wording to the response to Operator Licensing Feedback Item 401.55, prior to disseminating to industry stakeholders.

Page 1 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

NRR/IOLBs response to Region II in ROI 17-09 (ML17165A579) stated that testing applicants knowledge of the emergency or abnormal procedure content for Tier 1 written examination test items was more restrictive than currently called for by NUREG-1021, and that this was a change in policy which would require a revision or supplement to NUREG-1021. During the 8/24/17 teleconference, Region II asked the program office to explain the difference between Tier 1 and 2 written test items. The program office staff said they did not know what the authors of NUREG-1021 intended when constructing the two tiers, and did not attempt to justify or explain the difference between the Tiers during the teleconference.

The NRR/IOLB response in both ROI 17-09 and Operator Licensing Feedback Item 401.55 included: Tier 1 test items dont need to reference a procedure and Tier 1 test items dont require EOP/AOP entry, both which may be true, depending on a test items particular construction. However, NRR/IOLBs response did not address the fundamental question regarding the intent of Tier 1 Emergency/Abnormal Evolution test items on the plant-specific written examination, nor did it explain the intended difference between Tier 1 and Tier 2 test items.

Impact on Agencys Mission The impact of NRR/IOLBs policy interpretation will result in less testing of applicants knowledge of abnormal and emergency procedures on the site-specific NRC written exam; a representative selection of 10 CFR 55.41(b)(10) may not be ensured on the NRC site-specific written examination. Although the NRC does not train operators, the NRC is responsible for testing operators after they complete their training program. Facility licensees make adjustments to their initial training program based, in part, on the NRC exam content.

NRR/IOLBs policy interpretation means that the site-specific written exam will test less abnormal/emergency procedure knowledge; therefore, it is likely that licensee training programs may be inappropriately adjusted to reflect the NRR/IOLB policy determination. Actual consequences may occur when operator procedure knowledge declines, and plant events are not properly mitigated by operators because of inadequate procedure knowledge. See the DPO Section titled Public Health and Safety Concern for a disturbing trend identified by INPO in 2017 regarding operator knowledge of abnormal procedures. The impact of NRR/IOLBs policy determination, in turn, could affect the agencys Strategic Plan (ML14246A439), Safety Objective 1, Prevent and Mitigate Accidents and Ensure Radiation Safety, because licensed operators mitigate accidents during abnormal and emergency events.

The impetus for ROI 17-09 was actual inconsistencies in facility licensee interpretations of how to develop Tier 1 written exam test items; the ROI presented three differing viewpoints that currently exist. During the Region II 2017 Office Assessment, NRR/IOLB identified that several Tier 1 draft test items, as submitted by the facility licensee for the 2016 Brunswick NRC examination, were categorized as deficient (by the examiner) because the questions did not test applicant knowledge of emergency/abnormal procedures. The NRR/IOLB assessment (ML17095A958) concluded that NUREG-1021 did not require Tier 1 test items to test applicant knowledge of emergency/abnormal procedure content, and that the examiners evaluation (that the draft test items did not match the intent of the K/A) was wrong.

Page 2 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Proposed Alternative As previously documented in ROI 17-09, the submitters of this DPO propose the following alternative, including its basis, instead of NRR/IOLBs position regarding Tier 1 written test items.

A Tier 1 written test item should, whenever possible, within the wording of the K/A statement, test the applicants knowledge of the abnormal condition or emergency procedure content, for example:

o an immediate operator action, o an important subsequent manual operator action, or o overall mitigative strategy of the off-normal or emergency procedure.

Tier 1 test items where the stem of the question mentions an ongoing abnormal/emergency evolution, but where the test item can be answered solely using Tier 2 (Plant Systems) knowledge, contain, in a sense, window dressing; these test items should be assessed as K/A mismatches, but assessed as enhancement required, in accordance with ES-401-9, Written Exam Worksheet.

If testing knowledge of the abnormal condition or emergency procedure content is not possible given the wording of the K/A, then accept the question as meeting the K/A as long as all other aspects of the K/A are met.

The benefit of this proposed alternative is that testing procedure knowledge is promoted, there is no penalty for the exam writer, and the random sample initially drawn is preserved.

Basis for Proposed Alternative NRR/IOLBs response to Region II in ROI 17-09 (ML17165A579) stated that testing applicants knowledge of the emergency or abnormal procedure content for Tier 1 written examination test items was more restrictive than currently called for by NUREG-1021, and that this was a change in policy which would require a revision or supplement to NUREG-1021.

Testing applicants knowledge of EOPs and AOPs on the site-specific written exam is not a change in policy; there has always been precedence for testing applicant knowledge of EOPs and AOPs on the site-specific written examination. For example, even the original 1983 version of NUREG-1021, ES-203, Structure of Written Examination Administered to Reactor Operators

- Power Reactors (ML15027A434) stated:

In general, the candidate must demonstrate complete knowledge and understanding of the symptoms, automatic actions, and immediate action steps specified by abnormal and emergency procedures.

The original (1983 version) of ES-203 written exam cover page included Category 4, as shown below:

Page 3 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Testing applicants knowledge of EOPs and AOPs in Tier 1 of the site-specific written exam is not more restrictive than currently called for in NUREG-1021 because the current versions of NUREG-1122, Rev. 2 and NUREG-1123, Rev. 2 (PWR & BWR Knowledge and Abilities Catalogs, respectively) Section 1.10, Emergency and Abnormal Evolutions, contain the following definition of an emergency and abnormal evolution:

EMERGENCY EVOLUTION: An emergency plant evolution is any condition, event, or symptom which leads to entry into the EOPs.

ABNORMAL EVOLUTION: An abnormal plant evolution is any degraded condition, event, or symptom not directly leading to an EOP entry condition, but nonetheless, adversely affecting a safety function.

The current version of NUREG-1021 (Rev. 11), includes Form ES-401-1 (BWR Written Exam Outline) and Form ES-401-2 (PWR Written Exam Outline), these forms identify Tier 1 as EMERGENCY and ABNORMAL PLANT EVOLUTIONS.

NRR/IOLBs policy interpretation does not ensure that the site-specific written exam tests vendor-specific (Westinghouse, Babcock & Wilcox, Combustion Engineering) EOPs, as required by NUREG-1021.

To illustrate this point, consider Westinghouse procedure ECA-2.1, Uncontrolled Depressurization of All Steam Generators, listed in the Westinghouse Owners Group EOP list of procedures:

Page 4 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

NUREG-1021, Form ES-401-2, identifies 000040 Steam Line Rupture - Excessive Heat Transfer (W E12) as an emergency/abnormal plant evolution topic. The WE12 designator on Form ES-401-2 for this topic refers to PWR K/A Catalog Section 4.5, Westinghouse Emergency Plant Evolutions, E12: Uncontrolled Depressurization of all Steam Generators. (See PWR K/A Catalog Section 4.5 listed below, item listed on page 4.5-31.)

Page 5 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

The intent of the 000040 Steam Line Rupture - Excessive Heat Transfer (W E12) topic is to test the applicants knowledge of the Westinghouse ECA-2.1 procedure. The vendor-specific AOPs and EOPs in the K/A Catalogs, and in ES-401, mirror the actual vendor procedures because the intent was to test the content of these important procedures.

Therefore, the NRR/IOLB policy determination, that applicants knowledge of EOPs for Tier 1 test items like this ECA-2.1 example is adequately tested using test items that test plant system design features, interlocks, and system operation, does not ensure that the site-specific written exam tests vendor-specific (Westinghouse, Babcock & Wilcox, Combustion Engineering) EOP content, which is required by NUREG-1021, ES-401.

Instead, the writers of this DPO contend that the site-specific written exam Tier 1 test items, like the vendor-specific EOPs example discussed above, should, whenever possible, test the applicants knowledge of:

an immediate operator action, an important subsequent manual operator action, or overall mitigative strategy associated with the emergency procedure During the 8-24-17 teleconference with Region II, when asked to explain the difference between Tier 1 and Tier 2 site-specific written exam items, NRR/IOLBs response was they did not know what the authors intended when constructing two Tiers, and subsequently did not attempt to justify or explain their policy determination for Tier 1 test items.

The original Examiner Standards Handbook included a graphic representation of the intended content of the generic and site-specific portions of the written examination: (see next page)

Page 6 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

This Examiners Standard Handbook representation depicts the middle Tier 1 slice as EPEs and APEs, i.e., emergency and abnormal plant evolutions. The Tier 1 slice is represented as a different slice compared to Tier 2; Tier 2 is plant systems knowledge. NRC Information Notice 88-40 (Examiners Handbook for Developing Operator Licensing Examinations) indicated that approximately 27% of the RO exam (40 % of the SRO exam) should normally sample emergency and abnormal plant evolutions in Tier 1. This graphical representation explained how the 10 CFR 55.41 (a) requirement for a representative selection was being met.

Based on the preceding discussion of historical precedence, definitions in the K/A catalogs, and ES-401 requirements for two Tiers, the writers of this DPO contend the following items:

Page 7 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Tier 2 site-specific written exam items are intended to test the applicants knowledge of plant systems, and to a lesser degree, some procedure knowledge as it relates to A.2 K/As. The intent of Tier 2 is to test an applicants mastery of how the plant works.

Tier 1 site-specific written exam items are intended to test the applicants mastery of how to operate the plant in accordance with the abnormal (i.e., off-normal) and emergency operating procedures, during abnormal and emergency evolutions.

NRR/IOLBs response to Region II during the 8-24-17 teleconference was that systems knowledge and Emergency/Abnormal procedure knowledge cannot be separated.

The writers of this DPO contend that systems knowledge and procedure knowledge can be separated. Systems knowledge includes, for example, plant system design features, interlocks, flow paths, actuation logic, and set points. Abnormal/emergency procedures knowledge includes, for example, required immediate operator actions, important subsequent actions, and overall mitigative strategy for off-normal and emergency evolutions. Consider the following before and after examples listed below, for the same Tier 1 K/A. In the before example, a fault inside containment causes pressure to rise above the Main Steam Trip Valve auto-isolation set point. In the after example, the applicants knowledge of the overall mitigative procedure strategy is tested.

Before:

Page 8 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

After:

As can be seen by this example, it is not difficult to avoid testing abnormal condition procedure content, and still meet the lower level wording of a Tier 1 K/A statement. Systems knowledge and procedures knowledge form the basis for the NRCs confidence when issuing a license to an operator applicant - the NRC issues a license to an applicant who knows 1) how the plant works (system knowledge) and 2) how to operate the plant, in accordance with procedures, during abnormal/emergency situations (procedure knowledge).

During the 8-24-17 teleconference with Region II, NRR/IOLB stated that changes to the yet unpublished K/A Catalog Emergency/Abnormal Stem Statements would be made (for the future Revision 3, which was published in the Federal Register in April 2017), commensurate with the recent NRR/IOLB policy determination in ROI 17-09 and Operator Licensing Feedback Item 401.55.

The last sentence of ROI 17-09 (Recommended Action/Resolution Section) referred to the new K/A Catalog (Revision 3) industry effort. The new BWR and PWR catalogs were the result of a joint agency-industry effort, compiled of stakeholder teams with years of industry experience, Page 9 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17) and were published in the Federal Register in April 2017. The new catalogs, as published in the Federal Register in April 2017, included basis statements for the emergency/abnormal knowledge and ability stem statements, which were intended to provide insight for the intent of Tier 1 topics. (See Table 4 below).

If NRR/IOLB removes or alters these basis statements for the upcoming Revision 3 K/A Catalogs, then this refutes the experience and wisdom of the industry effort, and creates an inconsistency between the operating fleet K/A catalogs and the AP-1000 K/A Catalog, NUREG-2103. (See the following pages.)

Page 10 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Note that the AP-1000 abnormal/emergency procedure numbers and titles are included on Form ES-401N-2 (See following page)

Page 11 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 12 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Lastly, based on NRR/IOLBs policy interpretation, there is no need to even have separate Tier 1 and Tier 2 categories since the NRR/IOLB policy interpretation relies only on the lower wording of the K/A statement, and discounts the Tier in which the K/A was selected. It is not reasonable to assume that the authors of ES-401 arbitrarily designed two tiers that were not different. The reasonable assumption is that there was a reason for separate Tier 1 and Tier 2 categories. Tier 2 is Plant Systems, which was meant to test the configuration of the plant systems and their design. The void is filled by Tier 1, which was meant to test how to operate the plant in accordance with plant procedures when confronted with abnormal/emergency situations that challenge safety.

Some may say that, since the site-specific written exam is not the only portion of the NRC exam that tests abnormal and emergency procedures, it doesnt matter if the quantity of test items requiring knowledge of the content of abnormal/emergency procedures is reduced.

Although the operating portion of the NRC exam does test abnormal and emergency procedure knowledge, the control room team and open-reference operating exam format should not be relied upon to satisfy the intent of 10 CFR 55.41(b) which requires a representative selection of the fourteen items, for each individual on a written examination.

Form ES-201-2, Examination Outline Quality Checklist, contains the following items that the facility licensee and Chief Examiner must assess for each NRC exam:

Written Exam Item 1.c: Assess whether the outline overemphasizes any systems, evolutions, or generic topics.

General Item 4.b: Assess whether the 10 CFR 55.41, 55.43, and 55.45 sampling is appropriate General Item 4.e: Check the entire exam for balance of coverage.

The Form ES-201-2 Quality Check Items (1.c, 4.b, and 4.e) will not be met because the result of NRR/IOLBs policy is that the number of site specific written exam test items that test abnormal/emergency procedure content will be lower, especially for RO exams, which leads to overemphasis of testing plant systems knowledge on the plant specific written exam.

To illustrate this point, a reactor operator (RO) written exam sample plan was reviewed to identify how many test items would test knowledge of abnormal/emergency procedure content when NRR/IOLBs policy determination was implemented. Specifically, the criteria used in this review was, unless the K/A statement wording specifically included the word procedure, the test item was assumed to solely test a plant system design feature, interlock, or system operation aspect. The review results identified that only three of the 27 Tier 1 test items would test abnormal/emergency procedure content. Out of all three Tiers, (75 questions on the RO sample plan), a total of only NINE items would test abnormal/emergency procedure knowledge when the NRR/IOLB policy determination was implemented. This review meant that only 12%

of the RO test items would be required to test abnormal/emergency procedure knowledge when NRR/IOLBs policy determination was implemented.

Based on this initial review, three additional RO written exam sample plans were independently reviewed, using the same criteria listed above, to verify the first review results.

Page 13 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Percentage of 75 RO test items that would test abnormal/emergency procedure knowledge Second Sample Plan Reviewed: 9% (7/75)

Third Sample Plan Reviewed: 10% (8/75)

Fourth Sample Plan Reviewed: 9% (7/75)

Based on these review results, implementation of NRR/IOLBs policy determination will result in less site-specific RO written exam test items that test abnormal/emergency procedure content, and is a departure from NRC Information Notice 88-40, which described the original methodology for fulfilling the 10 CFR 55.41 representative selection requirement for the site-specific written exam, including 10 CFR 55.41(b)(10).

To counter the opposing view that the operating test may somehow compensate for a reduced number of RO abnormal/emergency procedure test items on the site-specific written exam, during the scenario portion of the operating examination, SRO applicants knowledge of EOPs is evaluated more so than RO applicants knowledge of EOPs, since the SRO applicants direct RO emergency procedure actions. The site-specific written exam provides the only opportunity to evaluate knowledge of EOPs for RO applicants on an individual basis. The control room team and open-reference operating exam format should not be relied upon to satisfy the intent of 10 CFR 55.41(b) which requires a representative selection of the fourteen items, for each individual on a written examination.

One must look no further than pass rates on the operating portion of the examination. Pass rates on the dynamic simulator portion of the examination approach 100%. The percentage of applicants that receive satisfactory scores on more than 80% of the Systems Job Performance Measures (JPMs) also approaches 100%. The majority of operating test failures occur due to the contribution of failing scores on Administrative JPMs; citing a lower overall JPM pass rate for the purpose of justifying a reduced number of abnormal/emergency written test items is misleading because the majority of JPM Section failures occur as a result of the Administrative JPM contribution to the JPM Section failure. Therefore, although the dynamic simulator scenarios and JPMs test applicant knowledge of emergency and abnormal procedures, these sections of the operating exam do not discriminate at the same level as the site-specific RO written exam.

It is noteworthy that, during operating exam scenarios, applicants who are examined in the RO (a.k.a operator-at-the-controls - OATC) and Balance of Plant (BOP) positions are typically provided direction from the SRO (a.k.a control room supervisor, CRS), who reads steps from the emergency procedures. When the SRO provides the directives, the OATC and BOP applicants then carry out the required emergency actions. This operating exam format tests the OATC and BOP applicants ability to perform emergency actions, but it does not test the same in-depth emergency procedure knowledge that was intended to be tested in the Tier 1 portion of the site-specific written exam.

Page 14 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Public Health & Safety Concern On September 7, 2017, NRR/IOLB published Operator Licensing Feedback Item 401.55, which means that facility licensees will likely develop Tier 1 plant-specific written test items that test less abnormal and emergency operating procedure knowledge. Facility licensees may inappropriately make adjustments to their initial license training program based, in part, on NRR/IOLBs response to Operator Licensing Feedback Item 401.55; therefore, actual consequences may occur as operator procedure knowledge continues to decline (see next paragraph) and plant events are not properly mitigated by operators due to lack of procedure knowledge. Because of NRR/IOLBs policy determination, applicant knowledge of the content of abnormal and emergency procedures, in accordance with 10 CFR 55.41(b)(10), will not be adequately tested on the site-specific written examination, prior to issuance of operator licenses.

On June 1, 2017, INPO identified operator weaknesses in the implementation of abnormal operating and off normal alarm response procedures in INPO IER 17-5, Line of Sight to the Core, (ML17171A309):

Reviews of noteworthy events and evaluation data indicate that weaknesses exist in implementation of abnormal operating and alarm response procedures. In the majority of these events, abnormal plant conditions satisfied the entry conditions for multiple procedures, and operators chose implementation paths that resulted in inappropriate operator responses. Gaps in knowledge of abnormal and alarm response procedures led to a perception that it was allowable to be selective concerning procedural steps.

Further, several of the events revealed procedural deficiencies resulting from inadequate revisions that, in turn, led to flawed rationale by operators.

On March 28, 2010 at H. B. Robinson a 4KV cable fault caused a fire and damaged a transformer, and a second electrical fault and fire was caused by the operators when they inappropriately attempted to reset the generator lockout relay without first ensuring the cause of the lockout was cleared. The part of the event that is relevant to this DPO occurred while the crew was implementing the EOP response following the reactor trip; the operators came very close to losing a RCP seal, due to thermally shocking the seal, because they failed to implement Step 19 (Check RCP Seal Cooling) correctly. Specifically, the operators opened the RCP seal injection valve even though thermal barrier cooling was lost for 39 minutes coincident with no seal injection for 10 to 15 minutes. The operators complied with the procedure step for verifying that a charging pump was operating, when they made the determination that adequate seal injection existed, but did not comply with the intent of the step to verify that seal cooling had been maintained. The March 2010 H.B. Robinson event is a strong example of why testing abnormal/emergency procedure knowledge on the site-specific written exam is important. The final significance determination for the two WHITE findings was documented in NRC Inspection Report 05000261/2011008 (ML110310469).

Page 15 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Summary of Concern NRR/IOLBs policy interpretation, and subsequent communication to industry stakeholders, will result in fewer items that test applicant knowledge of abnormal and emergency procedure content on the site-specific written examination; abnormal and emergency procedures are required to be tested on the site-specific written exam in accordance with 10 CFR 55.41(b)(10).

The control room team and open-reference operating exam format should not be relied upon to satisfy the intent of 10 CFR 55.41(b) which requires a representative selection of the fourteen items, for each individual on a written examination.

NRR/IOLBs policy interpretation does not ensure that the site-specific written exam tests vendor-specific (Westinghouse, Babcock & Wilcox, Combustion Engineering) EOPs, as required by NUREG-1021.

Facility licensees may inappropriately adjust training programs to reflect the NRR/IOLB policy determination.

Tier 1 site-specific written exam items are intended, whenever possible, to test the applicants mastery of how to operate the plant during abnormal and emergency evolutions, in accordance with the abnormal/emergency procedures. NRR/IOLBs communication to industry in Operator Licensing Feedback Item 401.55 means Tier 1 site-specific written exam test items will likely become an extension of Tier 2 test items. Tier 2 site-specific written exam items are intended to test the applicants systems knowledge of how the plant works, and to a lesser degree, some procedure knowledge as it relates to A.2 K/As.

If NRR/IOLB removes or alters the basis statements for the upcoming Revision 3 K/A Catalogs, as published in the Federal Register on April 2017, to facilitate their policy determination, then the experience and wisdom of the industry effort for the Catalogs will be lost, and an unnecessary inconsistency between the operating fleet K/A catalogs and the AP-1000 K/A Catalog, NUREG-2103 will exist.

Page 16 of 30

Diffenng Professional Opinion: Operator Licensing Written Exam Tier I Test Items (Document Date: 10-10-17)

Signatures The writers of this DPO are Region II examiners, with extensive industry training and/or operating experience, many of which previously held SRO NRC licenses.

[Name (printed) I Signature I Date]

uuo C4EkLL.eL.o /&w(o 7,0-10

- /

M. &cø/

/ /3-_

& I I D - 10 - (7

%L k

)

,rL:d /e/ /

LATIYr I L%%tZ.

Page 17 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

The following pages contain important documents associated with this Differing Professional Opinion. Although these documents may be available in ADAMS, the writers of this DPO included these documents as part of the DPO.

Page 18 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 19 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 20 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 21 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 22 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 23 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 24 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 25 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 26 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 27 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 28 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 29 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 30 of 30

Document 2: Memo Establishing DPO Panel November 6, 2017 MEMORANDUM TO: Raymond K. Lorson, Panel Chairperson Region I Matthew P. Emrich, Panel Member Office of the Chief Human Capital Officer Charles D. Zoia, Panel Member Region III THRU: Anne T. Boland, Director /RA/

Office of Enforcement FROM: Renée M. Pedersen /RA/

Sr. Differing Professional Views Program Manager Office of Enforcement

SUBJECT:

AD HOC REVIEW PANEL - DIFFERING PROFESSIONAL OPINION ON OPERATOR LICENSING WRITTEN EVALUATIONS - TIER 1 TEST ITEMS (DPO-2017-007)

In accordance with Management Directive (MD) 10.159, The NRC Differing Professional Opinion Program; and in my capacity as the Differing Professional Opinion (DPO) Program Manager; and in coordination with Anne Boland, Director, Office of Enforcement, Brian Holian, Acting Director, Office of Nuclear Reactor Regulation; and the DPO submitters; you are being appointed as members of a DPO Ad Hoc Review Panel (DPO Panel) to review a DPO submitted by several NRC employees.

The DPO (Enclosure 1) raises concerns about the recent policy on writing and assessing Tier 1 written examination test items. The DPO has been forwarded to Mr. Holian for consideration and issuance of a DPO Decision.

CONTACTS: Renée Pedersen, OE (301) 287-9426 Gladys Figueroa-Toledo, OE (301) 287-9497

The DPO Panel has a critical role in the success of the DPO Program. Your responsibilities for conducting the independent review and documenting your conclusions in a report are addressed in the handbook for MD 10.159 in Section II.F and Section II.G, respectively.

The DPO Web site also includes helpful information, including interactive flow charts, frequently asked questions, and closed DPO cases, including previous DPO Panel reports. We will also be sending you additional information that should help you implement the DPO process.

Because this process is not routine, we will be meeting and communicating with all parties during the process to ensure that everyone understands the process, goals, and responsibilities.

Disposition of this DPO should be considered an important and time sensitive activity. The timeliness goal for issuing a DPO Decision is 120 calendar days from the day the DPO is accepted for review. In this case, the DPO was accepted for review on October 16, 2017. The timeliness goal for issuing this DPO Decision is February 13, 2018.

Process Milestones and Timeliness Goals for this DPO are included as Enclosure 2. The timeframes for completing process milestones are identified strictly as goalsa way of working towards reaching the DPO timeliness goal of 120 calendar days. The timeliness goal identified for your DPO task is 75 calendar days from the date of this memorandum (January 20, 2018).

Although timeliness is an important DPO Program objective, the DPO Program also sets out to ensure that issues receive a thorough and independent review. The overall timeliness goal should be based on the significance and complexity of the issues and the priority of other agency work. Therefore, if you determine that your activity will result in the need for an extension beyond the overall 120-day timeliness goal, please send an e-mail to (OD/RA) with a copy to DPOPM.Resource@nrc.gov and include the reason for the extension request and a proposed completion date for your work and a proposed timeliness goal for issuance of a DPO Decision. Mr. Holian is responsible for subsequently forwarding the request for a new DPO timeliness goal to the EDO for approval.

An important aspect of our organizational culture includes maintaining an environment that encourages, supports, and respects differing views. As such, you should exercise discretion and treat this matter appropriately. Documents should be distributed on an as-needed basis. In an effort to preserve privacy, minimize the effect on the work unit, and keep the focus on the issues, you should simply refer to the employees as the DPO submitters. Avoid conversations that could be perceived as hallway talk on the issue and refrain from behaviors that could be perceived as retaliatory or chilling to the DPO submitters or that could potentially create a chilled environment for others. It is appropriate for employees to discuss the details of the DPO with their co-workers as part of the evaluation; however, as with other predecisional processes, employees should not discuss details of the DPO outside the agency. If you have observed inappropriate behaviors, heard allegations of retaliation or harassment, or receive outside inquiries or requests for information, please notify me.

On an administrative note, please ensure that all DPO-related activities are charged to Activity Code ZG0007.

We appreciate your willingness to serve and your dedication to completing a thorough and objective review of this DPO. Successful resolution of the issues is important for NRC and its stakeholders. If you have any questions or concerns, please feel free to contact me or Gladys.

We look forward to receiving your independent review results and recommendations.

Enclosures:

1. DPO-2017-007

2. Process Milestones and Timeliness Goals cc:

B. Holian, NRR M. Evans, NRR B. McDermott, NRR C. Haney, RII B. Caballero, RII D. Bacon, RII P. Capehart, RII M. Meeks, RII D. Lanyi, RII M. Bates, RII D. Lew, RI S. Rutledge, OCHCO R. Orlikowski, RIII A. Boland, OE G. Figueroa-Toledo, OE

SUBJECT:

AD HOC REVIEW PANEL - DIFFERING PROFESSIONAL OPINION ON OPERATOR LICENSING WRITTEN EVALUATIONS - TIER 1 TEST ITEMS (DPO-2017-007) DATE: November 6, 2017 ADAMS Package: ML17307A046 MEMO: ML17307A053 - ML17290A536 - ML17307A059 OE-011 OFFICE OE: DPO/PM OE: DPO/PM OE: D NAME GFigueroa RPedersen ABoland DATE 11/ 06 /2017 11/ 06 /2017 11/ 06 /2017 OFFICIAL RECORD COPY

Document 3: DPO Panel Report March 19, 2018 MEMORANDUM TO: Brian E. Holian, Acting Director Office of Nuclear Reactor Regulation FROM: Jeffrey A. Clark, DPO Panel Chair /RA/

Matthew P. Emrich, DPO Panel Member Charles D. Zoia, DPO Panel Member

SUBJECT:

DIFFERING PROFESSIONAL OPINION PANEL REPORT ON OPERATOR LICENSING WRITTEN EXAMINATIONS - TIER 1 ITEMS (DPO-2017-007)

In a memorandum, dated November 21, 2017, we were appointed as members of a Differing Professional Opinion (DPO) Ad Hoc Review Panel (DPO Panel) to review a DPO regarding Operator Licensing written examinations; Tier 1 Items. The DPO Panel has reviewed the DPO in accordance with the guidance in Management Directive 10.159, The NRC Differing Professional Opinion Program.

The results of the DPO Panels evaluation of the concerns raised in the DPO are detailed in the enclosed DPO Panel Report and is submitted for your consideration. Based on our review of concerns raised in the DPO, the DPO Panel made a recommendation, with two additional considerations, if implemented.

Please do not hesitate to contact us if you have any questions regarding the enclosed report.

CONTACT: Jeffrey A. Clark, RIV/DRS 817-200-1180

Enclosure:

DPO Panel Report

Statement of Issue (SOI)

The Operator Licensing and Training Branch (IOLB) of the Office of Nuclear Reactor Regulation (NRR) determined that a test item developed for Tier 1 of the site-specific written exam matches the intent of its knowledge or ability (K/A) statement if the test item solely tests plant systems knowledge, such as a design feature, interlock, or automatic operation. IOLB determined it was inappropriate to evaluate Tier 1 test items as enhancement required or inappropriate on Form ES-401-9, Written Examination Review Worksheet, when the test item did not test knowledge of emergency or abnormal procedures. This determination was documented in Record of Interaction (ROI) 17-09, NUREG 1021, ES-401 Tier 1 Written Exam Test Items (ML17165A579); was disseminated to industry stakeholders in Operator Licensing Feedback Item 401.55 (ML17249A961); and was communicated during an operator licensing examiner training session conducted by IOLB staff on October 19, 2017.

Based on the Panels review of the Differing Professional Opinion (DPO) submittal and associated references, and interview and follow-up discussion with the submitters, we determined the following issue was expressed:

Some NRC staff are concerned that the IOLB policy determination conflicts with the purpose of Tier 1 test items to test emergency and abnormal operating procedural knowledge on the site-specific Reactor Operator (RO) written examination, which is required in accordance with 10 CFR 55.41 (b)(10). The staff members contend that the number of RO questions that test abnormal and emergency procedures on the site-specific written exam should not be reduced because the operating exam does not test individual applicants procedure knowledge to the same extent as the written exam because:

• Dynamic scenarios are administered in a team, open-book environment where the SRO reads or directs emergency/abnormal operating procedure steps to RO applicants, and systems Job Performance Measures (JPMs) are administered by directing the applicant to perform a task in accordance with a specific procedure.

These staff members are concerned that the IOLB policy determination precludes the Chief Examiner from evaluating a Tier 1 test item as enhancement required on Form ES-401-9, when the proposed test item does not test abnormal or emergency procedure knowledge relevant to the K/A statement wording, which would ensure overall exam balance of coverage for abnormal and emergency operating procedures.

Panel Review Summary and Recommendation In response to DPO Case Number DPO-2017-007, its associated summary of concern (SOI),

and after careful consideration of input obtained through independent research, interviews with the DPO submitters, and interviews with personnel from the IOLB, the DPO Panel offers the following recommendations:

1. A Chief Examiner (CE) should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e. Enhancements may be assessed per Form ES-401-9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak. Amplifying information is discussed below for how procedural relevance can be determined. If the only flaw for a test item being assessed is its link to a relevant procedure, then the question should not be assessed as unsatisfactory.

2. If recommendation #1 is implemented, IOLB should determine how to promulgate this policy change regarding the assessment of Tier 1 questions to both agency and industry stakeholders. Additionally, IOLB should consider coordinating with the staff at the Technical Training Center (TTC) to include appropriate modifications to the examiner training course (G-107).

3. If recommendation #1 is implemented, IOLB should also assess the necessity of changes to NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The DPO Panel does not recommend any substantive changes to the current revision of NUREG-1021, as the current guidance supports recommendation #1.

However, the next revision of NUREG-1021 may include clarifying guidance or examples providing the basis for the assessment of Tier 1 written exam questions (similar to what was included in previous revisions regarding SRO only written exam questions).

Supporting Information The information below provides the basis from which the DPO Panel arrived at the recommendation outlined above. Based on the summary of concern outlined in the DPO, the DPO panel analyzed the applicable sections of the current revision of NUREG-1021 related to written examination construction and evaluation. As a result of this analysis, the DPO Panel noted the following:

1. Per ES-401 of NUREG-1021, Revision 11 (underlined text indicates emphasis added):

The content of the written licensing examinations for ROs and SROs is dictated by 10 CFR 55.41, Written Examination: Operators, and 10 CFR 55.43, Written Examination:

Senior Operators, respectively. Each examination shall contain a representative selection of questions concerning the knowledge and abilities (K/As) and skills needed to perform duties at the desired license level. Both the RO and SRO examinations will sample the 14 items specified in 10 CFR 55.41(b), and the SRO examination will also sample the 7 additional items specified in 10 CFR 55.43(b).

Except as noted in Section D.1.b of this examination standard, NUREG-1122, Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors, and

NUREG-1123, Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors, available in the Agencywide Documents Access and Management System (ADAMS), provide the basis for developing content-valid operator licensing examinations.

Each K/A stem statement has been linked to an applicable item number in 10 CFR 55.41 and/or 10 CFR 55.43. Preparing the license examination using the appropriate K/A catalog, in conjunction with the instructions in this NUREG-series report, will ensure that the Examination includes a representative sample of the items specified in the regulations.

==

Conclusion:==

Following the sample plan methodology as described in NUREG-1021 (and the applicable K/A catalog) ensures that the NRC written exam meets the content requirements as outlined in the items from 10CFR55.41 and 10CFR55.43.

2. From 10 CFR 55.41:

(b) The written examination for an operator for a facility will include a representative sample from among the following 14 items, to the extent applicable to the facility.

(1) Fundamentals of reactor theory, including fission process, neutron multiplication, source effects, control rod effects, criticality indications, reactivity coefficients, and poison effects.

(2) General design features of the core, including core structure, fuel elements, control rods, core instrumentation, and coolant flow.

(3) Mechanical components and design features of the reactor primary system.

(4) Secondary coolant and auxiliary systems that affect the facility.

(5) Facility operating characteristics during steady state and transient conditions, including coolant chemistry, causes and effects of temperature, pressure and reactivity changes, effects of load changes, and operating limitations and reasons for these operating characteristics.

(6) Design, components, and functions of reactivity control mechanisms and instrumentation.

(7) Design, components, and functions of control and safety systems, including instrumentation, signals, interlocks, failure modes, and automatic and manual features.

(8) Components, capacity, and functions of emergency systems.

(9) Shielding, isolation, and containment design features, including access limitations.

(10) Administrative, normal, abnormal, and emergency operating procedures for the facility.

(11) Purpose and operation of radiation monitoring systems, including alarms and survey equipment.

(12) Radiological safety principles and procedures.

(13) Procedures and equipment available for handling and disposal of radioactive materials and effluents.

(14) Principles of heat transfer thermodynamics and fluid mechanics.

Note: Items 1 and 14 from the above list are not part of the site-specific NRC licensing examination as they are covered on the Generic Fundamentals Exam.

3. From 10 CFR 55.43:

(b) The written examination for a senior operator for a facility will include a representative sample from among the following seven items and the 14 items specified in § 55.41 of this part, to the extent applicable to the facility:

(1) Conditions and limitations in the facility license.

(2) Facility operating limitations in the technical specifications and their bases.

(3) Facility licensee procedures required to obtain authority for design and operating changes in the facility.

(4) Radiation hazards that may arise during normal and abnormal situations, including maintenance activities and various contamination conditions.

(5) Assessment of facility conditions and selection of appropriate procedures during normal, abnormal, and emergency situations.

(6) Procedures and limitations involved in initial core loading, alterations in core configuration, control rod programming, and determination of various internal and external effects on core reactivity.

(7) Fuel handling facilities and procedures.

4. Sample plan methodology / Knowledge and Ability (K/A) Stem Links:

The site-specific written exam sample plan (or exam outline) is divided into 3 distinct Tiers:

Tier 1 Emergency and Abnormal Plant Evolutions Tier 2 Plant Systems Tier 3 Generic K/A Categories As can be observed from Table 1, a properly constructed exam outline (per the requirements of ES-401 and Form ES-401-1) is designed to include K/A stem statements that are linked to all of the items (55.41(b)(2) through 55.41(b)(13)) required by regulation for a Reactor Operator written examination AND all of the items (55.43(b)(1) through 55.43(b)(7)) for a Senior Reactor Operator written examination.

Following the above (which was taken directly from NUREG-1021; ES-401), the K/A catalogs provide the basis for developing content-valid operator licensing examinations AND each K/A stem statement has been linked to an applicable item number in 10 CFR 55.41 and/or 10 CFR 55.43. With that in mind, consider the following 10 CFR 55.41 and 10 CFR 55.43 links for the K/A statements associated with Tier 1 (Emergency and Abnormal Plant Evolutions) questions on the exam outline (Table 1) {Bold Italics added for emphasis}:

• Reactor Operator (5) Facility operating characteristics during steady state and transient conditions, including coolant chemistry, causes and effects of temperature, pressure and reactivity changes, effects of load changes, and operating limitations and reasons for these operating characteristics.

(6) Design, components, and functions of reactivity control mechanisms and instrumentation.

(7) Design, components, and functions of control and safety systems, including instrumentation, signals, interlocks, failure modes, and automatic and manual features.

(8) Components, capacity, and functions of emergency systems.

(9) Shielding, isolation, and containment design features, including access limitations.

(10) Administrative, normal, abnormal, and emergency operating procedures for the facility.

• Senior Reactor Operator (1) Conditions and limitations in the facility license.

(2) Facility operating limitations in the technical specifications and their bases.

(3) Facility licensee procedures required to obtain authority for design and operating changes in the facility.

(5) Assessment of facility conditions and selection of appropriate procedures during normal, abnormal, and emergency situations.

From the two lists above, only item (10) for the Reactor Operator, and items (3) and (5) for the Senior Reactor Operator specifically mention procedures. The blue-shaded blocks on Table 1 indicate the K/A stem statements associated with these items for the Tier 1 section of the exam outline. As highlighted on Table 1, the K/A categories with 10 CFR links to items that refer to procedures are K1, A2, and G.

Conversely, the 10 CFR links associated with Tier 2 questions that refer to procedures (blocks highlighted in green on Table 1) only include the G K/A category. An exception to this observation is the A2 category, which consists of K/A stem statements that specifically call out procedure selection as part of the required knowledge and ability of the applicant (see example below):

Ability to (a) predict the impacts of the following on the RHR/LPCI: INJECTION MODE; and (b) based on those predictions, use procedures to correct, control, or mitigate the consequences of those abnormal conditions or operations:

Based on these observations, The DPO Panel believes that Tier 1 questions in the K1, A2, and G categories should have some procedural aspect associated with them in order to appropriately test the 10 CFR item(s) they are linked to, and be in conformance with the existing regulation. While not explicit (i.e. shall), disregarding the 10 CFR links listed in the K/A stem statements risks not testing in a manner to ensure all of the CFR 55.41 and 10 CFR 55.43 items are sampled.

5. From NUREG-1123 (underlined text indicates emphasis added):

The K/As were linked to their applicable 10CFR55 item numbers. SRO level K/As were identified by 10CFR55.43 item numbers. (Taken from NUREG-1123 abstract)

The linkage of K/As to the 10 CFR 55.41, 43, and 45 requirements was done to help ensure that the examinations include a representative sample from among the applicable items (Taken from NUREG-1123, Rev. 2, Supp.1 - Summary of Significant Changes)

All knowledge and abilities (K/As) in this catalog are directly linked by item number to 10 CFR 55.

These statements agree with ES-401, which states that, Each K/A stem statement has been linked to an applicable item number in 10 CFR 55.41 and/or 10 CFR 55.43.

Also, from NUREG-1123 related to the Knowledge and Ability Stem Statements for Emergency and Abnormal Plant Evolutions:

Table 4 Knowledge and Ability Stem Statements for Emergency and Abnormal Plant Evolutions

E/AK1 Knowledge of the operational implications of the following concepts as they apply to the (EMERGENCY OR ABNORMAL PLANT EVOLUTION):

(CFR: 41.8 to 41.10)

E/AK2 Knowledge of the interrelations between (EMERGENCY OR ABNORMAL PLANT EVOLUTION) and the following:

(CFR: 41.7 / 45.8)

E/AK3 Knowledge of the reasons for the following responses as they apply to (EMERGENCY OR ABNORMAL PLANT EVOLUTION):

(CFR: 41.5 / 45.6)

E/AA1 Ability to operate and / or monitor the following as they apply to (EMERGENCY AND ABNORMAL PLANT EVOLUTION):

(CFR: 41.7 / 45.6)

E/AA2 Ability to determine and interpret the following as they apply to (EMERGENCY AND ABNORMAL PLANT EVOLUTION):

(CFR: 41.10 / 43.5 / 45.13)

Note: Similar statements are found related to the structural layout of the exam outline and associated K/A stem statement links to 10 CFR 55 for PWR written examinations in NUREG-1122, Revision 2, Supplement 1, and the Advanced Reactor designs (AP-1000 and ABWR) in NUREG-2103 and 2104. Also refer to Tables 2, 3, and 4 (highlighted in the same manner as described above for Table 1) for a comparison of the related K/A stem statement links to 10 CFR 55 for each of the K/A categories that comprise the exam outlines for the additional reactor designs.

6. From NUREG-1021, Appendix B (underlined text indicates emphasis added):

Failing to focus on testing the individual operators cognitive abilities (i.e., comprehension, problem-solving, and decision-making) or paying insufficient attention to the operators fundamental understanding of job content (e.g., systems, components, and procedures) may ultimately place job performance at risk of gradual degradation.

==

Conclusion:==

The random and systematic sampling process used when generating the exam outline in accordance with ES-401 of NUREG-1021 ensures that the written examination is content valid. The job content (for the written examination) that is being tested per existing regulation are those items identified in 10 CFR 55.41 and 55.43. The K/A catalogs link to the knowledges and abilities prescribed in 10 CFR 55 in a specific manner (refer to Tables 1 and 2.)

Failing to ensure that written exam questions developed for NRC examination test applicants to meet the intent of the 10 CFR 55 links, creates a risk that all of the 10 CFR 55 items may not be sampled appropriately, and therefore the validity inference that our process was designed to have may become skewed. For example, for those Tier 1 K/A categories previously mentioned, if there was not a procedural aspect for questions in categories K1, K2 (PWR), A2, and G, then the exam may overemphasize plant systems and under-emphasize plant procedures. The chief examiners quality assurance checks, per Form ES-201-2 items 4.b and 4.e, specifically direct verifying that the 10 CFR 55.41, 43, and 45 sampling is appropriate and exam coverage is balanced (see the form on the next page.)

7. From NUREG-1021, ES-201 (underlined text indicates emphasis added):

There are no minimum or maximum limits on the number or scope of changes the NRC may direct the facility licensee to make to its proposed examinations, provided that they are necessary to make the examinations conform with established acceptance criteria or to attain an appropriate level of examination difficulty.

==

Conclusion:==

The Chief Examiner is responsible for ensuring that the 10CFR55.41, 55.43, and 55.45 sampling is appropriate and exam coverage is balanced for the entire exam. Thus, if the Chief Examiner believes that the licensees proposed exam under-emphasizes procedural knowledge, it is not only prudent, but required by the QA checklist in ES-201 for the Chief Examiner to ensure that this issue is corrected prior to approving the exam for administration.

This may be performed by either providing unsatisfactory or enhancement required comments to the licensee via Form ES-401-9.

Final Conclusion From the information outlined above, the DPO panel recommends that IOLB partially implement the proposed alternative outlined in the DPO submittal, by implementing the above recommendation. Due to the current guidance in the K/A catalogs and NUREG-1021, the panel also feels that a Chief Examiner would be justified in asking a facility licensee to enhance Tier 1 written examination questions to test a procedural concept in K/A categories K1, K3 (PWR only), A2, and G as relevant to the respective K/A statement, or any other situation where the exam coverage becomes skewed. This approach helps ensure that NRC site-specific written examinations reflect the representative sample of 10CFR55 items required by regulation, exams will not over-emphasize plant systems knowledge, or under-emphasize plant procedural knowledge. Test items that appropriately solicit knowledge of plant system operation/response, or design associated with an emergency or abnormal event in categories K2, K3 (BWR only),

and A1, should be deemed satisfactory if their K/A statement was met with no other psychometric flaws.

ML18078A009 OFFICE RIV/DRS ADHRTD/TTC RIII/DRS NAME JClark MEmerich CZoia SIGNATURE /RA/ /RA-E/ /RA-E/

DATE 3/19/18 3/14/18 3/14/18 Document 4: DPO Decision UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 April 27, 2018 MEMORANDUM TO: Bruno L. Caballero, Senior Operations Engineer Operations Branch 2 Division of Reactor Safety Region II Daniel M. Bacon, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II David R. Lanyi, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Phillip G. Capehart, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Michael K. Meeks, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II FROM: Brian E. Holian, Acting Director /RA/

Office of Nuclear Reactor Regulation

SUBJECT:

DIFFERING PROFESSIONAL OPINION INVOLVING OPERATOR LICENSING WRITTEN EXAMINATIONS - TIER 1 ITEMS (DPO 2017-007)

On October 10, 2017, in accordance with Management Directive 10.159, The NRC Differing Professional Opinions Program, you submitted a differing professional opinion (DPO) involving operator licensing written examinations (DPO-2017-007). Specifically, your DPO raises concerns that the recent policy determination made by NRR staff for writing and assessing Tier 1 written examination test items has the potential to undermine the 10 CFR 55.41 that the CONTACT: Trent L. Wertz, NRR 301-415-1568

B. Caballero, et al. written examination should contain a representative selection of questions on the knowledge, skills, and abilities needed to perform licensed operator duties. Specifically, the policy interpretation will result in fewer questions that test the operators knowledge of abnormal and emergency procedures. The purpose of this memorandum is to respond to your DPO.

On January 21, 2017, a DPO Ad Hoc Review Panel (the Panel) was established and tasked to meet with you, review your DPO submittal, and issue a DPO report, including conclusions and recommendations to me regarding the disposition of the issues presented in your DPO. On March 19, 2018, after reviewing the applicable documents, completing internal interviews of relevant individuals and completing their deliberations, the Panel issued their report to me.

On April 19, 2018, I talked to you by telephone to discuss the Panels report and to get your insights and comments. On April 25, 2018, you provided me additional insights into your concerns and your thoughts for resolving the issue.

In order to make a decision with regard to your DPO, I reviewed your DPO submittal, the Panels report, met with the headquarters operator licensing staff, talked with you, and then re-considered your comments to me. In addition, on April 26, 2018, I discussed these issues with the DPO Panel Chair Statement of Concern The Operator Licensing and Training Branch (IOLB) of the Office of Nuclear Reactor Regulation (NRR) determined that a test item developed for Tier 1 of the site-specific written exam matches the intent of its knowledge or ability (K/A) statement if the test item solely tests plant systems knowledge, such as a design feature, interlock, or automatic operation. IOLB determined it was inappropriate to evaluate Tier 1 test items as enhancement required or inappropriate on Form ES-401-9, Written Examination Review Worksheet, when the test item did not test knowledge of emergency or abnormal procedures. This determination was documented in Record of Interaction (ROI) 17-09, NUREG 1021, ES-401 Tier 1 Written Exam Test Items (ML17165A579); was disseminated to industry stakeholders in Operator Licensing Feedback Item 401.55 (ML17249A961); and was communicated during an operator licensing examiner training session conducted by IOLB staff on October 19, 2017.

Based on the Panels review of the Differing Professional Opinion (DPO) submittal and associated references, and interview and follow-up discussion with the submitters, the panel determined the following issue was expressed:

Some NRC staff are concerned that the IOLB policy determination conflicts with the purpose of Tier 1 test items to test emergency and abnormal operating procedural knowledge on the site-specific Reactor Operator (RO) written examination, which is required in accordance with 10 CFR 55.41 (b)(10). The staff members contend that the number of RO questions that test abnormal and emergency procedures on the site-specific written exam should not be reduced because the operating exam does not test individual applicants procedure knowledge to the same extent as the written exam because:

• Dynamic scenarios are administered in a team, open-book environment where the SRO reads or directs emergency/abnormal operating procedure steps to RO applicants, and systems Job Performance Measures (JPMs) are administered by directing the applicant to perform a task in accordance with a specific procedure.

B. Caballero, et al. These staff members are concerned that the IOLB policy determination precludes the Chief Examiner (CE) from evaluating a Tier 1 test item as enhancement required on Form ES-401-9 when the proposed test item does not test abnormal or emergency procedure knowledge relevant to the K/A statement wording, which would ensure overall exam balance of coverage for abnormal and emergency operating procedures.

Panel Recommendations The Panel concluded and recommended the following:

1. A CE should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e. Enhancements may be assessed per Form ES-401- 9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak.

Amplifying information is discussed below for how procedural relevance can be determined. If the only flaw for a test item being assessed is its link to a relevant procedure, then the question should not be assessed as unsatisfactory.

2. If recommendation #1 is implemented, IOLB should determine how to promulgate this policy change regarding the assessment of Tier 1 questions to both agency and industry stakeholders. Additionally, IOLB should consider coordinating with the staff at the Technical Training Center (TTC) to include appropriate modifications to the examiner training course (G-107).

3. If recommendation #1 is implemented, IOLB should also assess the necessity of changes to NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The DPO Panel does not recommend any substantive changes to the current revision of NUREG-1021, as the current guidance supports recommendation #1. However, the next revision of NUREG-1021 may include clarifying guidance or examples providing the basis for the assessment of Tier 1 written exam questions (similar to what was included in previous revisions regarding SRO only written exam questions).

After considering all the information, I essentially agree with the recommendations provided by the DPO panel. They have thoroughly and conscientiously endeavored to address your well-thought out and articulated concerns. I have the following comments/clarifications to the recommendations.

Regarding Recommendations 1 and 3, I directed my staff to revise the Operator Licensing Program Feedback response (Question 401.55) to clarify that CEs are allowed to make reasonable changes for balance of coverage throughout the examination. I agree that Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge. However, when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not basis to remove it from the test. I considered the amplifying information referenced in Recommendation 1 and find there is no need to revise the current guidance that written examination questions should, but are not required to, test one of the 10 CFR 55 written examination items that the K/A is linked to, or to a facility learning objective. This is, in part,

B. Caballero, et al. because the linkage of K/A items to 10 CFR 55 written examination items does not represent an exhaustive list. I have assigned the above task to DIRS, NRR, to be completed by June 30, 2018.

Regarding Recommendation 2, the acknowledgement that CEs may request enhancements to written examination items, including the failure to test a relevant procedural concept will be promulgated through a revision to ROI 17-09 and an update to Question 401.55 on the Operator Licensing Program Feedback webpage. This action is also assigned to DIRS, NRR, to be completed by June 30, 2018.

A summary of the DPO will be included in the Weekly Information Report (when the case is closed) to advise interested employees of the outcome.

Thank you for raising your DPO and for your active participation in the DPO process. An open and thorough exploration of how we carry out our regulatory processes is essential to keeping these programs effective. Your willingness to raise concerns with your colleagues and managers and ensure that your concerns are heard and understood is admirable and vital to ensuring a healthy safety culture within the Agency.

Enclosure:

DPO Panel report, dated March 19, 2018 cc: R. Lorson, NRR M. Evans, NRR A. Boland, OE G. Figueroa-Toledo, OE C. Haney, RII C. Miller, NRR M. Johnson, OEDO

Pkg: ML18117A132; DPO: ML18117A079;

Enclosure:

ML18079A001 OFFICE NRR NAME BHolian DATE 4/27/18 Document 5: DPO Appeal

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items CONTENTS OF THIS APPEAL I. Reason for the Appeal II. Safety Significance of this Issue III. Executive Summary A. How We Got Here B. Objective Evidence for the Intent of Tier 1 Written Test Items C. Flaws with NRRs Guidance to Exam Writers on the NRCs Public Webpage D. DPO Submitters Proposed Solution E. Interactions with NRR F. DIRS Action Memo G. Closing Argument IV. History of the DPO 2017-007 Decision V. Analysis of the NRR DPO Decision Memo VI. Analysis of OL Feedback Item 401.55 VII. Analysis of the DIRS Action Memo A. Why stating the Intent of Tier 1 K/A Statements is NOT Creating a new requirement B. Why waiting until the next revision of NUREG-1021 is not necessary C. Why the NRCs Examination Process is NOT Subject to Backfit Considerations VIII. Timeline of DPO 2017-007 Events IX. Attachments and ML Numbers 1

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items I. REASON FOR THE APPEAL The reason for this appeal is that the corrective action taken for DPO-2017-007 was inadequate. The corrective action was to revise NRR-IOLBs answer to a stakeholders question, on the NRC public webpage, hereafter referred to as OL Feedback Item 401.55.

The revised answer to the stakeholders question does not state the intent of Tier 1 (Emergency and Abnormal Plant Evolutions) written examination questions, is misleading, contradictory, and incorrect. Clear and correct guidance to internal and external stakeholders is crucial for the content of operator licensing written exams.

The DPO Submitters did not initially appeal the DPO-2017-007 decision because they trusted that the full spirit and intent of the DPO Panel Recommendations and the NRR Decision Memo would be included in the revised answer to OL Feedback Item 401.55. The NRR Decision Memo stated agreement with the intent of Tier 1 test items, and provided general guidance on how to address Tier 1 items that did not test emergency and abnormal procedure knowledge. If the spirit and intent of the words in the DPO ad hoc Panel Recommendations and the NRR Decision Memo had been implemented, this issue would have been concluded long ago.

This appeal will show how the information on the NRCs public website contradicts itself, the intent of Tier 1 as stated in existing published NUREGs used for developing initial license exams, the DPO ad hoc Panel recommendations, and the NRR Decision Memo. This appeal also provides additional information supporting our disagreement, and will provide an acceptable alternative to the inadequate corrective action taken by NRR.

II. SAFETY SIGNIFICANCE OF THIS ISSUE Getting adequate resolution to our DPO Appeal is important because it is a nuclear safety imperative that reactor operator (RO) applicants knowledge of abnormal and emergency procedures be adequately tested on the written exam before the NRC issues a license. A recent industry report contained an analysis of significant operational events; the analysis showed that a key contributor to the events was weaknesses in the operators implementation of abnormal operating and alarm response procedures. Details concerning the nuclear industry organizations significant event report are not included in this DPO Appeal due to the proprietary information contained in the report; however, a redacted version is available in ADAMS (ML17171A309).

The safety significance of this issue is that allowing the current version of OL Feedback Item 401.55 to remain on the public webpage may result in the NRC issuing licenses to operators who have not been adequately examined in emergency and abnormal procedure knowledge on the RO written exam. This is a safety concern because the published resolution on the webpage ignores requirements that are already in place to ensure that emergency and abnormal condition procedures are adequately tested on the RO written exams. The original DPO (ML17290A536) should be referenced to understand this safety concern, because it objectively showed that an RO written exam could contain an extremely limited number of abnormal and emergency condition procedure questions if NRRs guidance to exam writers is not corrected. The original DPO also showed why testing emergency and abnormal procedure knowledge on the operating test was not a basis for inadequately testing emergency and abnormal procedure test items on the RO written exam.

2

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items III. EXECUTIVE

SUMMARY

A. How We Got Here The DPO submitters are all of the Region II Senior Examiners, with collective experience that includes multiple reactor operator and senior reactor operator licenses and instructor certifications, as well as decades of collective NRC chief examiner experience.

The Region II Senior Examiners engaged NRR before DPO 2017-007 was submitted, because NRR-IOLBs initial answer to a stakeholders question (Attachment 1) on the NRCs public webpage was misleading and incorrect. After the DPO was resolved, the DPO submitters accepted the results of the DPO decision, as documented in the NRR Decision Memo (Attachment 2, ML18117A079), because the statements in the NRR Decision Memo largely agreed with the intent of the Ad Hoc DPO Panel Recommendations (Attachment 3, ML18079A001).

The corrective action was to revise OL Feedback Item 401.55 (Attachment 4); however, the DPO submitters concluded that the NRR DIRS Operator Licensing Program Office did not adequately address important and necessary aspects of the NRR Decision Memo, and the intent of the DPO Panel Recommendations in the revision. Again, the DPO submitters engaged the NRR DIRS Operator Licensing Program Office on the inadequate implementation of the NRR Decision Memo and the DPO Panel Report Recommendations. This interaction caused the Acting Director of DIRS to write a memo to the Acting Director of NRR on August 15, 2018; hereafter referred to as the DIRS Action Memo (Attachment 5, ML18225A149), which changed the scope and intent of the corrective actions that were originally published in the NRR Decision Memo.

In response to the DIRS Action Memo, the DPO submitters spoke to two non-management members of the Ad Hoc DPO Panel about the revision to OL Feedback Item 401.55. These Panel Members agreed with the DPO Submitters, that the corrective action taken to revise OL Feedback Item 401.55 did not fully address the recommendations in the DPO Panel Report, and did not address what the DPO submitters and two-thirds of the Ad Hoc Panel believed to be the intent of the NRR Decision Memo. After confirming that the consulted DPO Panel members agreed that the NRR DIRS Operator Licensing Program Office did not fully implement corrective actions of the NRR Decision Memo, as recommended in the DPO Panel Report, this appeal of DPO 2017-007 was submitted.

B. Objective evidence for the Intent of Tier 1 RO Written Test Items The NRC written exam for RO applicants contains three sections, or Tiers:

Tier 1 (Emergency and Abnormal Plant Evolutions)

Tier 2 (Plant Systems)

Tier 3 (Generic plant-wide knowledge).

The following is a list of objective evidence that proves the intent of the Tier 1 portion of the RO written exam is, and has always been, to test applicants knowledge of how to respond to emergency and abnormal conditions in accordance with plant procedures.

The NRR-DIRS/IOLB Offices position is that it is permissible for Tier 1 questions to only 3

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items test systems knowledge, even though Tier 2 is intended to test how plant systems are designed to operate. The original DPO submittal (ML17290A536) presented Items 1 - 6 below.

1. The authors of NUREG-1021 and the K/A Catalogs, had a reason for why the written exam outline included Tiers 1 and 2, and a reason for why a random and systematic selection process was required to ensure adequate distribution from each of these Tiers. IF (as stated by NRR-DIRS/IOLB) Tier 1 questions were intended to be answered using only systems knowledge, THEN there is absolutely no reason for NUREG-1021 to contain a process that separates Tier 1 and Tier 2; every topic from the current Tiers 1 and 2 could be placed into a single tier if that were the case. There is a reason for Tier 1.

2. Tier 1 topics correlate to the titles of abnormal condition and emergency procedures, for each of the reactor technologies. This is not a coincidence.

3. Information Notice 88-40, Examiners Handbook for Developing Operator Licensing Examinations, (Attachment 6) is longstanding evidence, from the beginning of the operator licensing examination process, that there is an intended proportion of questions required to test systems versus emergency and abnormal evolutions. These proportions still exist in the current version of NUREG-1021.

4. The 1983 version of the NUREG-1021 Examiner Standard (Attachment 7) stated, In general, the candidate must demonstrate complete knowledge and understanding of the symptoms, automatic actions, and immediate action steps specified by abnormal and emergency procedures. Testing applicant knowledge of abnormal and emergency procedures on the written exam is not a new requirement, it has always been the intent.

5. Table 4, Knowledge and Ability Statements for Emergency and Abnormal Evolutions, (Attachment 8) in the AP-1000 (NUREG-2103) and ABWR (NUREG-2104) K/A Catalogs contain basis statements which state the intent to test procedure knowledge.

6. The industrys proposed Revision 3 to the operating fleet BWR (NUREG-1123) and PWR (NUREG-1122) K/A Catalogs, includes bases statements in Table 4, Knowledge and Ability Statements for Emergency and Abnormal Evolutions, which emphasize the intent to test procedure knowledge. (Attachment 9)

We have additional objective evidence, already published in the NUREGs used for exam development, which further substantiates that the intent of Tier 1 is to test emergency and abnormal procedure knowledge. In fact, not testing emergency and abnormal procedure knowledge on the RO written exam, is a violation of the current examination requirements. The additional evidence is presented in Items 7 - 10 below.

7. NUREG-1021, Revision 11, Form ES-401-6, Written Exam Quality Checklist, Item 9 (Attachment 10), is a check to see if a question is appropriate for the Tier to which it is assigned. There would be no reason to even have an Item 9 check 4

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items if there was no difference between the Tiers - see Item 1 above which states that the original authors must have had a reason to construct Tier 1 and 2.

8. Revision 11 of NUREG-1021, ES-401, states: Preparing the license examination using the appropriate K/A Catalog, in conjunction with the instructions in this NUREG-series report, will ensure that the examination includes a representative sample of the items specified in the regulations.

The approved K/A Catalogs for new reactors [AP-1000 (NUREG-2103) and ABWR (NUREG-2104)], as well as the industrys proposed Revision 3 for the PWR (NUREG-1122) and BWR (NUREG-1123) K/A Catalogs include Table 4, Knowledge and Ability Statements for Emergency and Abnormal Evolutions. Recall that Table 4 in these catalogs includes basis statements that point to testing procedures.

The title of Tier 1 of the written exam outlines contained in ES-401 (and ES-401N) is Emergency and Abnormal Plant Evolutions. The title of Table 4 in these K/A Catalogs is Knowledge and Ability Statements for Emergency and Abnormal Evolutions. The fact that these titles are the same is not a coincidence.

9. SECY-12-0151 from the NRR Director to the Commissioners (Attachment 11, ML12278A258), states that Revision 3 of the operating plant K/A catalogs will follow the same process that was used for the AP-1000 and ABWR catalogs, both of which contain basis statements for Table 4, Knowledge and Ability Statements for Emergency and Abnormal Evolutions.

10. NEI and members of industry have communicated to NRR DIRS Operator Licensing Branch Chief via official letter (Attachment 12, ML18180A121), that their submittals for Revision 3 of the operating plant K/A catalogs have both appropriate technical justification and our full support. Testing abnormal and emergency procedure knowledge in Tier 1 is not a new requirement. The Revision 3 K/A catalog submittals referred to by NEI contain the basis statements for Table 4, Knowledge and Ability Statements for Emergency and Abnormal Evolutions. Revision 3 of the catalogs have a section titled, Summary of Significant Changes, and those basis statements were not listed as a significant change in the revision 3 submittals.

The collective evidence means that the longstanding intent of the Tier 1 portion of the written exam is, and always was, to test emergency and abnormal procedure knowledge. It is imperative, from a safety perspective, to validate a licensed operators knowledge on how to operate the plant using procedures during emergency and abnormal conditions, prior to entrusting them with protecting the public health and safety, as well as the environment. Clearly communicating this requirement to test the intent of Tier 1 is in keeping with the NRCs principles of good regulation.

5

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items C. Flaws with NRRs Guidance to Exam Writers on the NRCs Public Webpage There are four flaws with OL Feedback Item 401.55 on the NRCs public webpage.

These flaws are analyzed in detail in Section VI of this appeal.

1. The current guidance does not state the intent of Tier 1 questions.

2. The current guidance is not clear for how to handle a situation where a Tier 1 question does not test abnormal or emergency procedure knowledge, or when it is not possible to write a question because of the difficulty with the randomly selected K/A.

3. The current guidance does not define adequate balance of coverage, which is now necessary to define because the NRR Decision Memo used this phrase; the NRR Decision Memo states that chief examiners can make changes to questions that do not meet Tier 1 when adequate balance of coverage does not exist.

4. The current guidance contradicts itself.

NRR-IOLB authorized flawed guidance to be posted on the public webpage in OL Feedback Item 401.55, even though the DPO submitters used every opportunity to engage and negotiate before each posting. When the current (6-27-18) version of OL Feedback Item 401.55 was posted to the public web page (Attachment 4), the Acting Director of DIRS stated, This is the best we can do. In spite of this, the DPO submitters have continued to pursue the issue because we believe that we, as an Agency, can do much better, serve all stakeholders better, and at the same time ensure that safe operators staff the controls of our nuclear power plants.

D. DPO Submitters Proposed Solution The Region II DPO submitters propose that the following paragraphs should be used to replace the current version of OL Feedback Item 401.55, in order to provide guidance to stakeholders for (1) the intent of Tier 1, (2) how to address Tier 1 questions that do not test emergency or abnormal procedure knowledge, and (3) what adequate balance of coverage means.

Questions written to test knowledge of a Tier 1 K/A should test abnormal/emergency procedure knowledge in order to meet the intent of a Tier 1 question. However, it is recognized that other portions of the exam may contribute to testing abnormal/emergency procedure knowledge; therefore, if a Tier 1 question does not test abnormal/emergency procedure knowledge, this should not be used as the basis to modify or remove the question IF (emphasis added) proper balance is exhibited within the written exam.

Chief examiners may request enhancements to written examination questions to ensure adequate abnormal/emergency procedure knowledge is tested. It may be prudent for examination authors to try to meet the intent of Tier 1 K/As by writing questions that test abnormal/emergency procedure knowledge, when relevant, to ensure balance of coverage, which may reduce the amount of re-6

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items work that could surface during the review process if adequate balance of coverage is not present due to inadequate abnormal/emergency procedure knowledge being tested.

Proper balance, as discussed in this response, can be analytically determined by looking at the NUREG-1021 requirements for generating a random and systematic sample plan.

The minimum number of questions required to test emergency and abnormal condition procedure knowledge is determined by the number of RO Tier 1 K/As (e.g., 27 for PWR) plus the minimum number of Tier 2 A2 K/As (2 because each category in each tier requires a minimum of 2). The A2 K/As require testing an applicants knowledge on predicting the impact of a malfunction or operation and using procedures to correct, control or mitigate; therefore, the number of Tier 1 K/As in the sample plan should be increased by 2 for making a determination on whether the exam contains the minimum number of emergency and abnormal procedure questions to claim adequate balance. For the example of a PWR RO Exam, the minimum number of questions testing emergency and abnormal procedure knowledge would be 29 (27 + 2).

For simplicity, this DPO Appeal solely focuses on adequate resolution to the current OL Feedback 401.55 revision on the NRCs public webpage. However, it is the DPO Submitters expectation that a similar revision to ROI 17-09R will follow.

E. Interactions with NRR After the DPO decision, the NRR/IOLB Branch Chief emailed the proposed revision to OL Feedback Item 401.55 to the Region II Operator Licensing Branch Chiefs for review on May 31, 2018. The Region II Branch Chiefs forwarded the proposal to the Region II Examiners. The examiners read the proposal, which triggered Mark Bates email to the NRR/IOLB Branch Chief on June 14, 2018 (Attachment 13). The Bates June 14 email reiterated many of the technical points that had already been discussed, as well as how the proposed revision did not fully implement the intent of the DPO Panel Report or the NRR Decision Memo. The NRR/IOLB Branch Chief replied to the June 14 Bates email on June 27, 2018, stating that they were moving forward with posting the proposed revision to OL Feedback Item 401.55 despite the concerns. Mark Bates forwarded the NRR/IOLB Branch Chiefs email response to one of the original DPO Panel Members to re-confirm that the Region II DPO submitters understanding of the intent of Tier 1 K/As, as well as the DPO submitters conclusions that the corrective actions, did not adequately address key aspects of the recommendations of the panel and the DPO decision memo. The DPO Panel members reply (Attachment 14) was: I think your group in Region 2 and I are in alignment on the fact that the revised OL Feedback Item response does not answer the mail based on the DPO panel recommendations and the tasking memo from Mr. Acting NRR Director.

During the June - July, 2018 timeframe, the Region II DPO submitters made many attempts to have a meaningful dialogue with the Acting NRR Director and the Acting DIRS Director on this issue. The Acting NRR Director, in a July 3 email (Attachment 15),

stated, I know the Acting DIRS Director has reviewed the DPO closely and has talked 7

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items with DPO panel chair. However, on a later phone call between Dan Bacon and the Acting DIRS Director, the Acting DIRS Director stated to Dan Bacon that he had not read the entire DPO. In the July 3 Acting NRR Director email, the Acting NRR Director also stated, Im aware of the intent not to diminish the number of procedure questions - and the flexibility that should be the Chief Examiners At the same time I would not intend to revise what is currently requiredpreferring consistency and only making changes that are well-vetted with all stakeholders. In rebuttal to the Acting NRR Directors statement in his July 3 email, one should consider these points:

1. Bases statements (Attachment 9) are already part of the draft Revision 3 catalogs for operating reactors, which were submitted by NEI, representing the industry, and a letter from NEI to the Operator Licensing Branch Chief on November 2, 2017, states, Notwithstanding our previous comments, all other changes contained in both documents have both appropriate technical justification and our full support.

Furthermore, both the new reactor K/A catalogs and revision 3 of the operating fleet catalogs are also published as NUREGs and have progressed through the public comment period, and in the case of the AP-1000 catalog, are being used for initial license examinations. In other words, testing emergency and abnormal procedure knowledge is already required; however, clarification is warranted to ensure consistency throughout all exams in every region.

2. NUREG-1021 states that the K/A catalogs, in conjunction with NUREG-1021 itself, provide the basis for developing content-valid operator licensing exams by ensuring that a representative sample of items is included on every exam. The combination of NUREG-1021 and the applicable K/A catalog determines the representative sample of test items. The K/A catalogs have already been through the public comment period, the authors of the catalogs attest to the technical justification of the revisions; therefore, the stakeholders have already vetted the content of Revision 3 of the operating fleet K/A catalogs, as well as the AP-1000 and ABWR catalogs.

3. The Acting NRR Director stated that consistency was a concern. If the intent of Tier 1 is not communicated to stakeholders in OL Feedback Item 401.55, then inconsistency between regions will be allowed to exist, and more re-work could result for exam writers. Also, if OL Feedback Item 401.55 is not adequately revised, an inconsistency will exist between what was submitted by NEI (Rev 3 of the operating fleet K/A Catalogs) and OL Feedback Item 401.55. Furthermore, IF the basis statements are deleted from Revision 3 of the operating fleet K/A Catalogs, which was suggested by NRR-IOLB, then an inconsistency will exist between the new reactor catalogs and the operating fleet catalogs.

4. The Acting NRR Director stated that he did not intend to revise what is currently required. Even if one considered stating the intent of Tier 1 as a change (which it is NOT), then consider the statement in NUREG-1021: Because licensees are not required to prepare examinations, changes made to the criteria used to prepare them are not imposed upon them; therefore, changes to NUREG-1021 for initial examination development do not meet the definition of backfitting. This is relevant because, the Region II DPO submitters do not view stating the intent of Tier 1 as a new requirement; the intent of Tier 1 is a longstanding requirement, which is supported by the basis statements not being listed as a significant change within 8

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items Revision 3 of the catalogs. Furthermore, even if adding the basis statements were viewed as a change, the backfitting discussion in NUREG-1021 makes it clear that making a change to NUREG-1021 is not imposing a new requirement because facilities are not required to write their own exams.

F. DIRS Action Memo On August 15, 2018, the Acting DIRS Director, as he has done several times throughout this process, called each of the DPO submitters individually to tell us that he was going to send a memo to the Acting NRR Director and copy the DPO submitters. Following these individual conversations between Dan Bacon, Mark Bates, and Acting DIRS Director, Mark Bates and Dan Bacon both sent emails (Attachment 16 & 17) to the Acting DIRS Director to memorialize their conversations. Prior to sending these emails, Mark Bates recapped the Acting DIRS Directors concerns verbally before the email was sent at the conclusion of the phone call. The Acting DIRS Director did not reply to either email. The Acting DIRS Director verbally described his concerns with Mark Bates and Dan Bacon as follows:

NRR viewed stating the intent of Tier 1 topics, as described in the Acting NRR Director DPO Decision memo, was adding a new requirement to the exam process.

The reason NRR did not revise OL Feedback Item 401.55 to include the intent of a Tier 1 K/A is that a NUREG change process must be followed in order to revise, or add a new requirement to the exam process. NRR believed that clarifying the intent of Tier 1, as described in the Acting NRR Director DPO Decision memo, would constitute a new requirement.

The Acting DIRS Director, on the phone call said that he was going to recommend in the memo he would send to the Acting NRR Director, that NUREG-1021 be revised to address this issue during the next revision, but did not provide any specifics on what the revision would contain. On the phone call, the Acting DIRS Director stated that NRR did not intend on making any further revisions to OL Feedback Item 401.55.

The Acting DIRS Directors memo to the Acting NRR Director, hereafter known as the DIRS Action Memo (Attachment 5, ML18225A149), was sent on August 15, 2018, soon after the individual phone calls with the DPO submitters. The Acting DIRS Director stated in this memo, As NUREG-1021 is binding upon both the staff and the regulated industry, I was cautious to ensure that the answer to question 401.55 did not create a new requirement or a new regulatory position for licensees that prepare written examinations. The DPO submitters believe that NRR current guidance to stakeholders in OL Feedback Item 401.55 actually does exactly what the Acting DIRS Director is supposedly concerned with, which is changing a NUREG requirement without the proper processes being followed and involving all internal and external stakeholders. The objective evidence supporting that the intent of Tier 1 K/As is to test procedure knowledge is overwhelming, as well as evidence that indicates that there is a difference between Tier 1 and Tier 2 K/As, so NRRs response to OL Feedback Item 401.55 actually changes the exam content requirements as defined by NUREG-1021 and the K/A catalogs. Refer to the four numbered points in Section III.C of this appeal.

Objective evidence and precedence already exists, so that providing the intent of Tier 1 9

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items to stakeholders does not create a new requirement; it is a longstanding requirement, has already been through the NUREG revision process, and the basis statements for the new K/A catalogs were written by and submitted by the industry. This leaves us confused to understand how the Acting DIRS Director could be concerned with imposing a new requirement on the industry, when the industry wrote the requirement and attested to the technical justification of the requirement.

It is the NRCs job to ensure safe operators receive licenses. One key component to ensuring this is to adequately test applicant knowledge on how to operate the plant in accordance with emergency and abnormal procedures. The Acting DIRS Director claims to be concerned with all stakeholders. The public may have the largest stake in nuclear safety of all the possible stakeholders. The Senior Examiners in Region II, as well as members of the nuclear industry who submitted the K/A catalogs containing Tier 1 bases statements, are stakeholders too, and we believe that adequately testing emergency and abnormal procedure knowledge is important and the right thing to do to ensure that safe operators staff the control rooms of nuclear power plants.

The Acting DIRS Director, in his August 15, 2018 memo, continues with, Having reviewed the applicable documents, and through discussions with you (Acting NRR Director) and a number of the DPO submitters, I have concluded that the current revision to question 401.55 meets the intent of your direction. It should be re-iterated at this point that the DPO submitters discussed the revised response to OL Feedback Item 401.55 with two of the three DPO panel members and they both informed us that the revision did not fully implement the recommendations in the DPO Panel Report and the NRR Decision Memo. This begs the question how the Acting DIRS Director believes that the full intent of the NRR Decision Memo had been implemented, when the majority of the DPO panel itself did not believe that to be the case.

The Acting DIRS Director also stated in his memo that he was recommending that the next NUREG-1021 revision should address the intent of Tier 1 questions and to take up the issue of balance of coverage in the written examinations. The Acting DIRS Director continues to say, By addressing both of these topics in NUREG-1021, in a forum open to internal and external stakeholder participation and comment, the agency and licensed community should benefit from increased clarity (leading to increased efficiency) in binding guidance. In response to this, the DPO submitters have provided a proposed solution to replace NRRs response to OL Feedback Item 401.55. Because the intent of the Tier 1 K/As has been systematically justified within this appeal process, a logical thought process can be applied to the random and systematic sampling process that will ensure adequate balance of coverage.

G. Closing Argument According to Management Directive MD 10.159, NRC Differing Professional Opinion Program, the scope of the DPO appeal must be limited to the originally agreed upon Summary of Issues (SOI) Statement; therefore, this appeal does not address any potential safety culture issue. Within this appeal, the reader may draw conclusions on how the integrity of the DPO process itself can be undermined, but that is not the purpose of this appeal. The purpose of this appeal is the technical safety issue of Tier 1 written exam items. For the technical safety issue associated with DPO-2017-007, the 10

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items Acting Director of NRR stated that he directed NRR staff to revise OL Feedback Item 401.55 to state that chief examiners could request licensees to make enhancements to the exams to obtain the proper balance of coverage. He continued to state that he agreed that Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge. He continued to state that OL Feedback Item 401.55 would be revised to acknowledge that chief examiners can request enhancements to written examination items, including the failure to test a relevant procedure concept. The NRR/IOLB program office literally complied with the direction to revise OL Feedback Item 401.55 to say that chief examiners could request licensees to make changes to the exams to obtain the proper balance of coverage, but the NRR/IOLB program office did not revise OL Feedback Item 401.55 to explain the intent of Tier 1 questions, and the revision did not clearly communicate that chief examiners could request enhancements to Tier 1 questions that did not test emergency and abnormal procedure knowledge.

It is not the intent of this appeal to ask for a determination on whether the NRR Decision Memo was intentionally written to direct one small action, while at the same time stating agreement with the DPO Panel Recommendations, with no intent to ever fully incorporate those recommendations into the final corrective actions. At the time the NRR Decision Memo was issued, the DPO submitters believed that the full spirit and intent of the DPO Panel Recommendations and the NRR Decision Memo would be included in the corrective action to revise OL Feedback Item 401.55. Therefore, the DPO decision was not initially contested. The DIRS Action Memo, altered the implementation of both the DPO Panel Recommendations and the original NRR Decision Memo. Because the DIRS Action Memo essentially changed the DPO decision, the appeal process was made available to the DPO submitters and this appeal was drafted. The two non-management representatives on the Ad Hoc DPO Panel confirmed, one via email, and the other via telephone, that the corrective actions did not fully implement the DPO Panel Recommendations.

It is important to recognize that the justification provided by the Acting DIRS Director for not correcting OL Feedback Item 401.55 is refuted point-by-point in Section VII of this Appeal. The Acting DIRS Directors justification included a concern of imposing a new requirement on the licensee without going through an official NUREG revision process.

This concern is without merit because the licensee wrote and submitted the bases statements for Revision 3 of the K/A Catalogs and those basis statements require testing procedure knowledge for Tier 1 topics. The concern about imposing a new requirement without undergoing an official NUREG revision process is also without merit, because the basis statements provided by industry have been through the NUREG revision process. The Acting DIRS Directors concern with imposing a new requirement on licensees is without merit, because NUREG-1021 itself states that the requirements for exam development can change and it does not impose a new requirement on the licensee because the licensees are not required to write their own exams (it is just an option for them to write the exam).

Lastly, it is also important to note that SECY-12-0151 (Attachment 11) informed the Commission that the same process that developed the AP-1000 and ABWR K/A catalogs would be followed for Revision 3 of the operating fleet K/A catalogs. The industry followed the guidance in SECY-0151 and submitted Revision 3 of the K/A catalogs to the NRC for review via the NUREG revision process, which included 11

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items submitting basis statements for the Tier 1 topics. Then NEI, in a letter to the NRR/IOLB Branch Chief (Attachment 12), stated that Revision 3 of the catalogs had both appropriate technical justification and their full support. This series of events does not support the concern of the Acting DIRS Director with respect to imposing a new requirement on the industry, for exams that the industry is not even required to write themselves, when it was the industry itself that wrote Revision 3 of the K/A catalogs, and endorsed their technical justification in separate correspondence sent to the Operator Licensing Branch Chief.

12

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items IV. HISTORY OF THE DPO 2017-007 DECISION Six experienced Senior Operations Engineers in Region II submitted Differing Professional Opinion (DPO) 2017-007, Operator Licensing Written Exam Tier 1 Test Items, (ML17290A536) on October 10, 2017. The DPO submitters contended the following points:

A Tier 1 written test item should, whenever possible, within the wording of the K/A statement, test the applicants knowledge of the abnormal condition or emergency procedure content, for example:

o an immediate operator action, o an important subsequent manual operator action, or o overall mitigative strategy of the off-normal or emergency procedure.

Tier 1 test items where the stem of the question mentions an ongoing abnormal/emergency evolution, but where the test item can be answered solely using Tier 2 (Plant Systems) knowledge, contain, in a sense, window dressing; these test items should be assessed as K/A mismatches, but assessed as enhancement required, in accordance with ES-401-9, Written Exam Worksheet.

If testing knowledge of the abnormal condition or emergency procedure content is not possible given the wording of the K/A, then accept the question as meeting the K/A as long as all other aspects of the K/A are met.

The Independent (ad hoc) Panel Report On March 19, 2018, the ad hoc DPO Panel tasked with reviewing DPO 2017-007 issued its report (Attachment 3, ML18079A001), which endorsed, in part, the concerns and recommendations of the DPO 2017-007 submitters. The ad hoc DPO Panel stated the following as their Final

Conclusion:

From the information outlined above, the DPO panel recommends that IOLB partially implement the proposed alternative outlined in the DPO submittal, by implementing the above recommendation. Due to the current guidance in the K/A catalogs and NUREG-1021, the panel also feels that a Chief Examiner would be justified in asking a facility licensee to enhance Tier 1 written examination questions to test a procedural concept in K/A categories K1, K3 (PWR only), A2, and G as relevant to the respective K/A statement, or any other situation where the exam coverage becomes skewed. This approach helps ensure that NRC site-specific written examinations reflect the representative sample of 10CFR55 items required by regulation, exams will not over-emphasize plant systems knowledge, or under-emphasize plant procedural knowledge. [emphasis added] Test items that appropriately solicit knowledge of plant system operation/response, or design associated with an emergency or abnormal event in categories K2, K3 (BWR only), and A1, should be deemed satisfactory if their K/A statement was met with no other psychometric flaws.

The three recommendations made by the ad hoc DPO panel mentioned above are as follows:

1. A Chief Examiner (CE) should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e. Enhancements may be assessed per Form ES-401- 9 13

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure

[emphasis added], or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak. Amplifying information is discussed below for how procedural relevance can be determined. If the only flaw for a test item being assessed is its link to a relevant procedure, then the question should not be assessed as unsatisfactory.

2. If recommendation #1 is implemented, IOLB should determine how to promulgate this policy change regarding the assessment of Tier 1 questions to both agency and industry stakeholders. Additionally, IOLB should consider coordinating with the staff at the Technical Training Center (TTC) to include appropriate modifications to the examiner training course (G-107).

3. If recommendation #1 is implemented, IOLB should also assess the necessity of changes to NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The DPO Panel does not recommend any substantive changes to the current revision of NUREG-1021, as the current guidance supports recommendation #1. However, the next revision of NUREG-1021 may include clarifying guidance or examples providing the basis for the assessment of Tier 1 written exam questions (similar to what was included in previous revisions regarding SRO only written exam questions).

The NRR Decision Memo On April 27, 2018, the Acting Director of the NRC Office of Nuclear Reactor Regulation (NRR) issued a memorandum (NRR Decision Memo) to the submitters of DPO 2017-0071 (Attachment 2, ML18117A079) detailing the agency resolution to the submitted DPO and the ad hoc DPO Panel Report mentioned above. Under the section of the DPO Decision Memo entitled Panel Recommendations, the acting NRR Director re-stated the three recommendations quoted above, and then continued as follows:

After considering all the information, I essentially agree with the recommendations provided by the DPO panel. They have thoroughly and conscientiously endeavored to address your well-thought out and articulated concerns. I have the following comments/clarifications to the recommendations.

Regarding Recommendations 1 and 3, I directed my staff to revise the Operator Licensing Program Feedback response (Question 401.55) to clarify that CEs are allowed to make reasonable changes for balance of coverage throughout the examination. I agree that Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge. However, when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not basis to remove it from the test. I considered the amplifying information referenced in Recommendation 1 and find there is no need to revise the current guidance that written examination questions should, but are not required to, test one of the 10 CFR 55 written examination items that the K/A is linked to, or to a facility learning objective.

This is, in part, because the linkage of K/A items to 10 CFR 55 written examination 1

Note that in an apparent clerical error, the memo from the Acting Director, NRR, is only addressed to five of the six submitters of DPO 2017-007. The individual omitted from the memo was, and is, still fully participating in the DPO 2017-007 process. No explanation has yet been provided as to why the individual was omitted from the DPO Decision Memo.

14

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items items does not represent an exhaustive list. I have assigned the above task to DIRS, NRR, to be completed by June 30, 2018.

Regarding Recommendation 2, the acknowledgement that CEs may request enhancements to written examination items, including the failure to test a relevant procedural concept will be promulgated through a revision to ROI 17-09 and an update to Question 401.55 on the Operator Licensing Program Feedback webpage. This action is also assigned to DIRS, NRR, to be completed by June 30, 2018.

15

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items V. ANALYSIS OF THE NRR DECISION MEMO A careful reading of the contents of the DPO Decision Memo is required to fully understand the direction that is given to the staff to implement.

After quotingverbatimthe three recommendations of the ad hoc DPO Panel, the NRR Director stated: I essentially agree with the recommendations provided by the DPO panel.

He then stated: I have the following comments/clarifications to the recommendations. Note that the NRR Director is not changing the recommendations; the NRR Director is not modifying the recommendations; in fact, the NRR Director is endorsing the recommendations of the DPO Panel, with the proviso that his specified comments/clarifications are followed.

The NRR Directors next sentence explicitly directs the staff to revise OL Feedback Item 401.55 to align with recommendations 1 and 3: Regarding Recommendations 1 and 3, I directed my staff to revise the Operator Licensing Program Feedback response (Question 401.55) to clarify that CEs [Chief Examiners] are allowed to make reasonable changes for balance of coverage throughout the examination. What constitute reasonable changes for balance of coverage? Reasonable changes was already clearly defined by recommendation 1, stated above.

Conclusion:

NRC staff was directed by the NRR Director to revise Question 401.55 in the language of recommendation 1, specifically to state that:

A Chief Examiner (CE) should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e. Enhancements may be assessed per Form ES-401- 9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak.

The NRR Director then continues as follows: I agree that Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge.

The above sentence linked with the next sentence is crucial in understanding the DPO Decision.

The NRR Director then stated: However, when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not basis to remove it from the test.

Although it is not explicitly stated, the NRR Director is assuming that IF there is NOT proper balance in the overall exam, THEN Chief Examiners may assess Tier 1 written questions as Enhancement required, and direct these items removed from the test to ensure the appropriate balance of coverage in Abnormal and Emergency procedure knowledge. The reason for this allowance is because the intent of Tier 1 questions is, when relevant, to test abnormal/emergency procedure knowledge.

16

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items

Conclusion:

NRC staff was further directed by the NRR Director to revise Question 401.55, specifically to state that:

Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge.

A CE can require a Tier 1 question that does not tie to a procedure be removed from the test when there is not a proper balance in the overall exam.

However, when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not a basis to remove it from the test.

The NRR Director then reiterated his directives in the next paragraph, which stated:

Regarding Recommendation 2, the acknowledgement that CEs may request enhancements to written examination items, including the failure to test a relevant procedural concept will be promulgated through a revision to ROI 17-09 and an update to Question 401.55 on the Operator Licensing Program Feedback webpage. []

Therefore, the NRR Decision Memo mandated that OL Feedback Item 401.55 be modified to direct the following points:

Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge.

A Chief Examiner (CE) should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e.

Enhancements may be assessed per Form ES-401- 9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak.

A Chief Examiner (CE) can require a Tier 1 question that does not tie to a procedure be removed from the test when there is not a proper balance in the overall exam.

However, when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not a basis to remove it from the test.

17

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items VI. ANALYSIS OF OL FEEDBACK ITEM 401.55 The original August 25, 2017 answer to OL Feedback Item 401.55 (Attachment 1, ML17249A961) stated, in part, the following:

[] However, NUREG-1021, Operator Licensing Examination Standards for Power Reactors, does not require Tier 1 written examination questions to reference a procedure. []

If a question meets its specific K/A statement in its entirety, then it meets the intent of the Tier category it is within, even if it does not specifically test procedural knowledge for a Tier 1 question. []

In summary, rating a question as an unacceptable or deficient K/A mismatch, i.e.

unsatisfactory or in need of enhancement, because it can be answered based on plant system knowledge as it relates to the referenced Emergency or Abnormal Plant Evolution, is not supported by NUREG-1021. []

From the original answer to OL Feedback Item 401.55, stakeholders got the following WRONG messages:

WRONG MESSAGE #1: The intent of Tier 1 written examination questions is not to test procedure knowledge.

WRONG MESSAGE #2: IF a Tier 1or any other Tierwritten examination question meets its specific K/A statement, THEN it meets the intent of that questions Tier. A Chief Examiner is not allowed to remove a Tier 1 written examination from a test to increase procedure knowledge balance-of-coverage, since the intent of Tier 1 is already met by meeting the specific K/A statementirrespective of whether or not the proposed question tests systems knowledge or abnormal/emergency procedure knowledge.

WRONG MESSAGE # 3: Moreover, rating a question as either unacceptable or enhancement due to a lack of abnormal/emergency procedure knowledge is not supported by NUREG-1021. In other words, Chief Examinerswho are bound to follow NUREG-1021 to conduct an examare not allowed to rate a Tier 1 question deficient of procedure knowledge as enhancement; if a Chief Examiner were to do so, it would be a practice not supported by NUREG-1021.

In the revised version of OL Feedback Item 401.55 (Attachment 2, ML18178A581), all of the above language was left unchanged. The only modification that was made was to add the following paragraph to the end of the existing text:

As a clarification, Chief Examiners are allowed to make reasonable changes that are necessary to ensure an adequate balance of coverage throughout the examination. In addition, the resolution to this question is not intended to restrict the Chief Examiners ability to request reasonable enhancements to any written examination items when necessary, including those that do not test a relevant procedural concept.

18

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items The result of adding this language to the end of OL Feedback Item 401.55, without modifying any of the preceding text, was to officially promulgate contradictory and confusing guidance. Here are the reasons why OL Feedback Item 401.55 still remains incorrect, misleading, and contradictory to the NRR Decision Memo.

1. The NRR Decision Memo stated that Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge.

Contrary to the DPO Decision Memo, the revised OL Feedback Item 401.55 states that NUREG-1021 does not require Tier 1 written examination questions to reference a procedure.

Conclusion:

The current revised version of OL Feedback Item 401.55 does not support the NRR Decision Memo that the intent of Tier 1 written exam questions is to test abnormal or emergency procedure knowledge whenever relevant.

2. The NRR Decision Memo stated that Enhancements may be assessed per Form 401-9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure.

Contrary to the DPO Decision Memo, the revised OL Feedback Item 401.55 states that rating a question as in need of enhancement, because it can be answered based on plant system knowledge is not supported by NUREG-1021.

Conclusion:

The current revised version of OL Feedback Item 401.55 does not support the NRR Decision Memo because it states that Tier 1 questions that do not test relevant procedure knowledge can not be rated as Enhancement required on Form 401-9 (arguably depending on which section of OL Feedback Item 401.55 the reader references, as shown by point 5. below).

3. The NRR Decision Memo stated that when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not basis to remove it from the test. It therefore must be logically true that when there is not a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is basis to remove it from the test.

Contrary to the DPO Decision Memo, the revised OL Feedback Item 401.55 states that If a question meets its specific K/A statement in its entirety, then it meets the intent of the Tier category it is within, even if it does not specifically test procedural knowledge for a Tier 1 question.

Conclusion:

The current revised version of OL Feedback Item 401.55 does not support the NRR Decision Memo because it states that there is no reason to remove a Tier 1 question from the test, even if it does not specifically test procedural knowledge, because as long as any question meets its specific K/A statement, then it meets the intent of the Tier category it is within.

19

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items

4. Section ES-401 D.2.a of NUREG-1021 states, in part: [] Ensure that the questions selected for Tier 3 maintain their focus on plant-wide generic K/As and do not become an extension of Tier 2. Furthermore, NUREG-1021 Form 401-6, Written Exam Quality Checklist, Item 9 requires that CEs certify that Question content conforms to specific K/A statements in the previously approved examination outline and is appropriate for the tier to which they are assigned; deviations are justified. [emphasis added] It is therefore clear that NUREG-1021 requires CEs to consider both the specific K/A statement and (in addition) the tier to which the question is assigned.

Contrary to NUREG-1021, the revised OL Feedback Item 401.55 states that If a question meets its specific K/A statement in its entirety, then it meets the intent of the Tier category it is within, even if it does not specifically test procedural knowledge for a Tier 1 question.

Conclusion:

The current revised version of OL Feedback Item 401.55 contradicts current NUREG-1021 requirements.

5. The current revision of OL Feedback Item 401.55 is self-contradictory.

The revised version of OL Feedback Item 401.55 states: the resolution to this question is not intended to restrict the Chief Examiners ability to request reasonable enhancements to any written examination items when necessary, including those that do not test a relevant procedural concept.

However, contrary to its own guidance, the revised version of OL Feedback Item 401.55 also states the following: rating a question as in need of enhancement, because it can be answered based on plant system knowledge is not supported by NUREG-1021.

Conclusion:

The current revised version of OL Feedback Item 401.55 allows a Chief Examiner to rate a question an Enhancement when necessary, including a question that do[es] not test a relevant procedural concept; whilst, and at the same time, the current revised version of OL Feedback Item 401.55 also states that a Chief Examiner should not rate a question as Enhancement because it can be answered based on plant system knowledge, because such an action is not supported by NUREG-1021.

This is an example of obvious and blatant self-contradiction contained in the currently promulgated guidance of OL Feedback Item 401.55.

There is no other way possible: either a CE can rate a Tier 1 question as Enhancement due to lack of procedural knowledge content, or a CE cannot rate a Tier 1 question as Enhancement due to lack of procedural knowledge content. The current agency guidance as expressed in the revised version of OL Feedback Item 401.55 is an exercise in illogical confusion and represents a violation of the basic fundamental principle of self-contradiction.

20

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items VII. ANALYSIS OF THE DIRS ACTION MEMO Following the publication of the revision to OL Feedback Item 401.55 detailed above, the DPO Submitters engaged NRC staff and management in an attempt to correct the inadequate implementation of the DPO decision. On August 15, 2018, the Acting DIRS Director issued a memorandum to the Acting NRR Director entitled ACTION ON DIFFERING PROFESSIONAL OPINION INVOLVING OPERATOR LICENSING WRITTEN EXAMINATIONS - TIER 1 ITEMS (DPO 2017-007). (Attachment 5, ML#18225A149) In the DIRS Action Memo, the Acting DIRS Director stated, in part, the following:

[] Per your direction in that memorandum, Operator Licensing Program Feedback response 401.55 has been updated. []

In assessing the need for a further revision to the answer to question 401.55, I considered the language of your written direction and the report submitted by the DPO panel (see memorandum from J. Clark to yourself, "Differing Professional Opinion Panel Report on Operator Licensing Written Examinations-Tier 1 Items (DP0-2017-007)", ML18117A132). In considering these memoranda, I was also mindful that 10 CFR 55 requires that the Commission "shall use the criteria in NUREG-1021, 'Operator Licensing Examination Standards for Power Reactors,' in effect six months before the examination date to prepare the written examinations required by§§ 55.41 and 55.43."

As NUREG-1021 is binding upon both the staff and the regulated industry, I was cautious to ensure that the answer to question 401.55 did not create a new requirement or a new regulatory position for licensees that prepare written examinations.

Having reviewed the applicable documents, and through discussions with you and a number of the DPO submitters, I have concluded that the current revision to question 401.55 meets the intent of your direction. Specifically, your statement "when relevant,"

when taken in the context of both the "Statement of Concern" in your memorandum ("

... when relevant to the K/A statement wording ... ") and the conclusion of the DPO panel led me to conclude that Tier 1 questions do not, by definition, require a procedural basis for all KA categories, and that implying an overall statement of intent for Tier 1 questions in the answer to feedback question 401.55 might create a new or revised regulatory position. The current revision of the answer to feedback question 401.55 also makes it clear that the balance of coverage for an examination remains the responsibility of the CE and that the CE has the authority to request reasonable changes to insure that balance of coverage exists.

While I find that the current answer to feedback question 401.55 is consistent with NUREG-1021, revision 11 (the revision currently in force), increased clarity (on the topics of Tier 1 questions and examination coverage) in the NUREG should be pursued. Consequently, I have directed Operator Licensing Branch (IOLB) staff to include two new items into the preparation of an upcoming revision to NUREG-1021.

First, IOLB has been directed to take up the issue of the intent of Tier 1 questions. In this effort, staff is to consider whether a general statement of intent should be explicitly defined or discussed in the standard and, if so, to develop such language. Second, IOLB has been directed to take up the issue of "balance of coverage" in written examinations. The goal of this effort would be to define the phrase as clearly as possible in NUREG-1021, mindful of the complexity of the issue, in order to provide a clear standard to both industry and NRC staff. As the DPO authors have indicated, this topic was the subject of an Information Notice (IN) in 1988 (see IN 88-04), which may provide a basis for this work. By addressing both of these topics in NUREG-1021, in a forum open to internal and external stakeholder participation and comment, the agency 21

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items and licensed community should benefit from increased clarity (leading to increased efficiency) in binding guidance.

There are three contentions raised in the DIRS Action Memo that are incorrect, without merit, and fail to implement the DPO 2017-007; the three contentions are:

(1) The current revision to OL Feedback Item 401.55 fully complies with and meets all the requirements of the NRR Decision Memo. No further revisions shall be made.

(2) To state the intent of Tier 1 written exam questions would be to create a new requirement or a new regulatory position.

(3) Due to the danger of creating a new requirement or new regulatory position, we must wait until the next revision of NUREG-1021 to determine what the intent of Tier 1 questions are, and to define what constitutes a proper balance of coverage in written examinations.

In answer to the first contention stated above, Part VI, Inadequate Revision to OL Feedback Item 401.55, of this DPO Appeal proves that, contrary to the DIRS Action Memo claims, the revised version of OL Feedback Item 401.55 (Attachment 4) is inadequate, contradictory and confusing, and certainly not in accordance with the direction of the NRR Decision Memo.

It is a very important point to understand why, contrary to the DIRS Action Memo, stating the intent of Tier 1 written exam questions is not creating a new requirement or regulatory position.

A. Why stating the Intent of Tier 1 K/A Statements is NOT Creating a new requirement It is imperative that reviewers of this appeal educate themselves on the content of the original DPO (ML17290A536). Within the original DPO, and in the Executive Summary of this Appeal, objective evidence exists to support that the intent of Tier 1 K/A topics is, and always has been, to test abnormal and emergency condition procedure knowledge.

The following paragraphs explain the new supporting evidence, and shows that NUREG-1021 states that it is the combination of NUREG-1021 and the K/A catalogs which define the scope of an initial license exam.

NUREG-1021 endorses the use of the draft AP-1000 K/A catalog, which contains Tier 1 basis statements (Attachment 8) requiring procedure knowledge to be tested; these AP-1000 K/A catalogs have been used for multiple exams and are a key part of the current initial operator licensing exam process. SECY-12-0151 (Attachment 11) shows that NRR itself informed the Commission that the latest revision (Revision 3) of the K/A catalogs would be based on the new draft AP-1000 and ABWR K/A catalogs. The industry submitted Revision 3 of the operating fleet K/A catalogs (Attachment 9) and specifically attested to the technical basis within those catalogs and explicitly stated that those catalogs have their full support (Attachment 12), with one exception that is not related to the basis, or intent, of Tier 1 K/A statements. The submitted Revision 3 catalogs, as well as the AP-1000 and ABWR catalogs from which they were based, all contain basis statements that adequately define the intent of Tier 1 K/A statements.

22

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items NUREG-1021, ES-401 and ES-401N (Attachment 1 and 2), both state that preparing the license examination using the appropriate K/A catalog, in conjunction with the instructions in this NUREG-series report, will ensure that the examination includes a representative sample of the items in the regulations and is the basis for developing content valid exams. NUREG-1021 provides Item 9 on Forms ES-401-6 and ES-401N-6 (Attachment 10) to ensure that the question content is appropriate for the Tier to which they are assigned. Item 9 on these Forms is evidence that a K/A residing in a specific Tier means something. If NRRs position is applied, the Tiers to which a K/A resides is meaningless. The existence of Item 9 is an indicator that there is a difference between Tier 1 and Tier 2.

NUREG-1021, ES-401N, Section B, Page 1 of 53, states that the AP-1000 K/A catalog issued in October 2011, and the ABWR K/A catalog issued December 2011, will provide the basis for developing content-valid operator licensing examinations. These new reactor K/A catalogs both contain basis statements that clearly communicate the intent of Tier 1 K/A statements (Attachment 8). In the case of the NUREG-2104, for AP-1000 reactors, these NUREG-level catalogs have already been used for multiple exams. The basis statements from Table 4 of the AP-1000 catalog are as follows (the word procedure has been underlined for emphasis):

EK 1 Lists the systems required to be monitored and/or operated by the procedure.

EK 2 Lists the operationally based theoretical concepts applicable to the procedure. These items typically came from the procedure bases, PRA, OE, procedure notes and cautions.

EK 3 Lists the actions and bases taken in the procedure.

EA 1 Lists the systems and/or components required to be monitored and/or operated by the procedure.

EA 2 Lists the parameters and/or conditions that are monitored to verify successful implementation of the procedure.

The basis statements from Table 4 of the ABWR catalog are as follows (again, the word procedure has been underlined for emphasis):

E/AK 1 Lists the operational implications applicable to the procedure. These items can come from the procedure bases, PRA, OE, procedure notes and cautions.

E/AK 2 Lists the systems required to be monitored and/or operated by the procedure.

E/AK 3 Lists the actions and bases taken in the procedure.

EA 1 Lists the systems and/or components required to be monitored and/or operated by the procedure.

EA 2 Lists the parameters and/or conditions that are monitored to verify successful implementation of the procedure.

23

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items SECY-12-0151 from the Director NRR, to the Commissioners, on November 2, 2012 (Attachment 12, ML12278A258), stated that the NRC will work with the NEI Licensed Operator Focus Group (LOFG) to revise the NRCs Knowledge and Abilities (K/A)

Catalogs for Nuclear Power Plant Operators and the K/A statement selection process for the written examinations. Within this SECY paper it states that an industry working group be established, with NRC participation, to revise the operating fleet K/A Catalogs (NUREG-1122 and NUREG-1123) to better reflect the licensed operator knowledge requirements, skills, and abilities required in the current operating environment. The SECY also states that the revision process would be modeled after the processes, including lessons learned, used to develop Draft NUREG-2103 and NUREG-2104, which are the K/A Catalogs for AP-1000 and ABWR reactor designs. It is worth reinforcing the fact that the AP-1000 and ABWR K/A Catalogs contain basis statements that describe how testing procedure knowledge is necessary to meet the Tier 1 K/A statements.

Draft Revision 3 of the operating plant K/A catalogs were submitted by NEI and these revisions contained basis statements for the Tier 1 K/As (Attachment 9), clearly describing the intent of the Tier 1 K/As.

NEI, sent a letter to the NRR-IOLB Branch Chief on November 2, 2017 (Attachment 12),

to request the removal of unintended changes contained in NUREG-1122, revision 3 and NUREG-1123, revision 3. NEI described in this letter that these unintended changes were associated with changes to the generic fundamentals K/As, listed in Sections 5 and 6 of those catalogs (issues which are unrelated to this DPO). The NEI letter continued to state, Notwithstanding our previous comments, all other changes contained in both documents have both appropriate technical justification and our full support. It is worth reinforcing the fact that among those other changes, were the basis statement additions for the Tier 1 K/As which were modeled after the AP-1000 and ABWR K/A Catalogs.

The basis statements for Tier 1 of revision 3 of the PWR K/A Catalog, NUREG-1122, are as follows (as before, the word procedure has been underlined for emphasis):

E/AK1 Lists the operationally based theoretical concepts applicable to the procedure. These items can come from the procedure bases, probabilistic risk assessment, operating experience, procedure notes and cautions.

E/AK2 Lists the systems required to be monitored and/or operated by the procedure.

E/AK3 Lists the reasons responses and/or actions are taken in the procedure.

E/AA1 Lists the systems and/or components required to be monitored and/or operated by the procedure. E/AA1 may include systems from E/AK2.

EA 2 Lists the parameters and/or conditions that are monitored to verify successful implementation of the procedure.

Similarly, the basis statements for Tier 1 of revision 3 of the BWR K/A Catalog, NUREG-1123, are (the word procedure has been underlined for emphasis):

24

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items E/AK1 Lists the operationally based theoretical concepts applicable to the procedure. These items can come from the procedure bases, PRA, OE, procedure notes and cautions.

E/AK2 Lists the systems required to be monitored and/or operated by the procedure.

E/AK3 Lists the reasons responses and/or actions are taken in the procedure.

E/AA1 Lists the systems and/or components required to be monitored and/or operated by the procedure. EA1 may include systems from EK2.

EA 2 Lists the parameters and/or conditions that are monitored to verify successful implementation of the procedure.

Using the above information, one can conclude that the intent, or the basis, of Tier 1 written exam items is to test abnormal or emergency procedure knowledge. The intent of Tier 1 has already been used for AP-1000 exams and is clearly stated in both draft K/A catalogs for AP-1000 and ABWR, which are endorsed for use within NUREG-1021, ES-401N. NRR directed the Revision 3 of the operating fleet K/A catalogs to be based on the AP-1000 and ABWR catalogs that are already endorsed by NUREG-1021 and have been used. NEI LOFG, consisting of a large population of industry representatives, states that the revision 3 of the operating fleet K/A catalogs, as submitted by the LOFG, have appropriate technical justification and their full support, with the exception of inadvertent generic fundamental K/A changes. Therefore, the intent of Tier 1 is not a new requirement. When NEI submitted Revision 3 of the operating fleet catalogs, they did not even list the addition of the basis statements as a significant change (potentially because it is not viewed as a change).

As one additional point to make regarding the issue of whether the intent of Tier 1 is a new item, the ad hoc panels report clearly stated: The DPO Panel does not recommend any substantive changes to the current revision of NUREG-1021, as the current guidance supports recommendation #1. [emphasis added]

B. Why waiting until the next revision of NUREG-1021 is not necessary For the sake of disproving the third argument raised in the DIRS Action Memo, this section will demonstrate two examples of changes made to the NRC exam process outside the NUREG-1021 revision cycle. In other words, even if one believed that defining the intent of Tier 1 written exam questions was a new regulatory position or requirement, this section will demonstrate that changes can and have been made using other methods than directly revising NUREG-1021.

First Example: May 15, 2014 Interim Guidance Memo On May 15, 2014, the DIRS Director issued a memorandum to the four regional DRS Directors with the subject INTERIM GUIDANCE RELATED TO THE CONDUCT OF INITIAL OPERATOR LICENSING EXAMINATIONS. In this memorandum, the DIRS 25

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items Director imposed requirements upon all regional examiners that changed the existing requirements in the then-current revision of NUREG-1021 (which was Revision 9, Supplement 1ADAMS ML15198A190).

For example, ES-204 section c.2.b (third paragraph) stated the following:

The region does not require written concurrence from NRR to deny an applicants waiver request, but it should discuss its decision with the operator licensing program office before informing the applicant; formal concurrence may be desirable in some cases.

The above direction given in NUREG-1021 was changed by the Interim Guidance memo, which stated:

Regional management shall discuss with the NRR program office its decision to deny an applicants request for waiver of a test based on previously passing that portion of the examination. Concurrence shall be obtained using the ROI process.

So, it is clear that the NRC examination process as specified in NUREG-1021 was, and can be, changed/modified and superseded by memorandum.

A further example in the Interim Guidance memo was the discussion of simulator grading policies as it related to applicants who committed two non-critical errors in one simulator rating factor. In this case, revision 9, supplement 1 of NUREG-1021 stated the following (section ES-303, D.2.b. fourth bulleted paragraph):

If an applicant makes two errors related to a rating factor, circle an RF Score of 1 for that rating factor unless a score of 2 can be justified (and documented as discussed in Section D.3, below) based on correctly performing another activity (or activities) related to the same rating factor; three or more errors generally require a score of 1, regardless of the applicants compensatory actions.

Section D.3.d of ES-303 further clarified the simulator grading policy of revision 9, supplement 1 of NUREG-1021 as follows:

As noted in Section D.2, above, deviations from the nominal grading criteria must be explained in detail. For example, an examiner may conclude that an applicants performance is acceptable despite exhibiting deficiencies that would normally result in an unsatisfactory grade (e.g., committing two or more errors related to the same simulator rating factor ).

The above grading allowance was changed by the Interim Guidance memo, which stated, in part:

the exercise of this provision is not subject to examiner discretion. If an applicant commits two non-critical errors in a rating factor, but performs another activity correctly related to that same rating factor, the rating factor score must be increased from 1 to 2.

26

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items Again, it is obvious that the previously stated Interim Guidance Memo changed the requirements that then existed in NUREG-1021. As an aside, the current requirements of Revision 11 to NUREG-1021 have dis-allowed the practice of increasing rating factor scores due to correct performance.

Second Example: OL Feedback Item 401.53 As a second example of changing (or providing clarifying information) to the exam process, consider the answer NRR provided in OL Feedback Item 401.53 (available in the current OL Feedback document (ML18178A581). This OL Feedback Item answer provided new guidance concerning acceptable number of open reference questions allowable on an NRC written examination as follows:

In this regard, the following ranges are provided regarding the allowable use of references on initial license examinations consistent with the principles discussed in Question 401.42. Note that these quantitative ranges are not absolute limitations, nor should they be construed as goals or requirements. You should also note that NUREG-1021 does not permit any "direct lookup" questions or questions with references that provide an advantage in answering other "closed-reference" questions on the initial licensing examination.

RO (75 items) = up to ~5% or 4 questions SRO (25 items) = up to ~20% - 25% or 5 - 6 questions Note that this new guidance on the recommended number of allowable open reference questions was not included as a change to NUREG-1021. As shown above, providing new guidance in the OL Feedback format is seemingly an acceptable method to make changes to the NRC initial operator examination processformally revising NUREG-1021 is not required.

C. Why the NRCs Examination Process is NOT Subject to Backfit Considerations The current revision 11 of NUREG-1021 (ML17038A432), page xiii, states that revisions to NUREG-1021 are not considered a backfit because licensees are not mandated to prepare initial operating exams required by the CFR. Specifically, under the section Backfitting and Issue Finality, NUREG-1021 stated the following, in part:

Revision 11 to NUREG-1021 does not represent backfitting as that term is defined in 10 CFR 50.109, Backfitting, and is not inconsistent with the issue finality provisions in 10 CFR Part 52. Current holders of operating licenses under 10 CFR Part 50 or combined licenses under 10 CFR Part 52 are not mandated to prepare the written examinations required by 10 CFR 55.41 and 10 CFR 55.43 and the operating tests required by 10 CFR 55.45, which must be prepared using the criteria in NUREG-1021 in effect 6 months before the examination date.1 Because licensees under 10 CFR Part 50 and 10 CFR Part 52 are not required to prepare the 10 CFR 55.41 and 10 CFR 55.43 examinations and 10 CFR 55.45 tests, changes to the criteria used to prepare the examinations and tests are not imposed upon them; therefore, these 27

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items changes do not meet the definition of backfitting in 10 CFR 50.109 and are not inconsistent with the finality provisions in 10 CFR Part 52.

In other words, because licensees are not required to prepare examinations, changes made to the criteria used to prepare them are not imposed upon them; therefore, changes to NUREG-1021 for initial examination development do not meet the definition of backfitting. (This is relevant because the August 15, 2018, memorandum claims that clarifying the intent of Tier 1 K/As, as suggested by the DPO submitters, would constitute a new requirement being imposed on the exact same licensees that submitted revision 3 of the K/A catalogs.)

28

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items VIII. Timeline of Events for DPO 2017-007 March 30, 2017: Region II submitted ROI 17-09 to NRR-IOLB for Tier 1 Test Items June 7, 2017: NRR-IOLB issued resolution to ROI 17-09 August 24, 2017: Teleconference between Region II and NRR-IOLB to discuss proposed wording OL Feedback Item 401.55 August 25, 2017: Operator Licensing Feedback Item 401.55 posted on public web October 10, 2017: DPO 2017-007 submitted by Region II Chief Examiners October 19, 2017: NRR-IOLB provided negative training to all examiners on Tier 1 test items March 19, 2018: Independent DPO Panel Report issued April 27, 2018: NRR Decision Memo issued May 30, 2018: ROI 17-09R (revision) issued May 31, 2018: NRR-IOLB distributed proposed revision to OL Feedback Item 401.55 to regional Branch Chiefs June 14, 2018: DPO Submitters emailed NRR-IOLB Branch Chief that the proposed revision to OL Feedback Item 401.55 was incorrect and contradicted itself.

June 27, 2018: OL Feedback Item 401.55 revision posted on the public webpage.

July 3, 2018: Conference between DPO Submitters and NRR to attempt to resolve differences.

August 1-13, 2018: DIRS Acting Branch Chief contacted DPO Submitters individually to discuss differences.

August 15, 2018: DPO Submitters provided Acting DIRS Director (via email) with the K/A Catalog Rev. 3 information regarding basis for emergency abnormal stem statements, including industrys previous acknowledgement.

August 15, 2018: DIRS Action Memo issued.

October 25, 2018: DPO Appeal Submitted 29

DPO Appeal of the Inadequate Implementation of DPO 2017-007: Operator Licensing Written Exam Tier 1 Test Items IX. Attachments and ML Numbers (each attachment provided after the next page)

1. The 8/25/17 (original) version of OL Feedback Item 401.55 (ML17249A961)

2. NRR Decision Memo (ML18117A079)

3. Independent DPO Panel Report (ML18079A001)

4. The 6/27/18 (revised) version of OL Feedback Item 401.55

5. DIRS Action Memo (ML1822A149)

6. Information Notice 88-40, Examiners Handbook For Developing Operator Licensing Examinations

7. NUREG-1021, 1983 Version (ML15027A414)

8. AP-1000 and ABWR K/A Catalog Table 4, Knowledge and Ability Statements for Emergency and Abnormal Procedures

9. Operating Fleet PWR and BWR K/A Catalog, Revision 3, Table 4, Knowledge and Ability Statements for Emergency and Abnormal Procedures

10. Form ES-401-6, Written Exam Quality Checklist

11. SECY-12-0151, From Eric Leeds to the Commissioners (ML12278A258)

12. NEI Letter from Gregory R. Cameron to Nancy Salgado (ML18180A121)

13. Mark Bates June 14, 2018 email to NRR/IOLB Branch Chief

14. DPO Independent Panel member June 27, 2018 email to Mark Bates

15. Acting NRR Director July 3, 2018 email to Bruno Caballero

16. Mark Bates August 15, 2018 email to Acting DIRS Director

17. Dan Bacon August 15, 2018 email to Acting DIRS Director Other ML numbers (not included as Attachments)

ROI 17-09 (ML#17165A579)

Entire OL Feedback Package on 8/25/17 (ML17249A961)

Entire OL Feedback Package on 6/27/18 (ML18178A581)

Entire DPO Case File (ML18150A469)

NUREG-2103, AP-1000 K/A Catalog (ML11307A3670)

NUREG-2104, ABWR K/A Catalog (ML11354A280)

NUREG-1122, PWR K/A Catalog, Rev. 2, Supplement 1 (ML102571881)

NUREG-1122, PWR K/A Catalog, Rev. 3 Draft (ML17097A204)

NUREG-1123, BWR K/A Catalog, Rev. 2, Supplement 1 (ML13086A115)

NUREG-1123, BWR K/A Catalog, Rev. 3 Draft (ML17097A214)

NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Rev. 11 (ML17038A432) 30

Document 6: Statement of Views UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 January 22, 2019 MEMORANDUM TO: Margaret M. Doane Executive Director for Operations FROM: Ho K. Nieh, Director /RA/

Office of Nuclear Reactor Regulation

SUBJECT:

STATEMENT OF VIEWS REGARDING APPEAL OF DIFFERING PROFESSIONAL OPINION CONCERNING OPERATOR LICENSE DPO-2017-007 Introduction The purpose of this memorandum is to provide you with my statement of views regarding the appeal of Differing Professional Opinion (DPO)-2017-007. 1 The appeal raises concerns that corrective actions taken by the Office of Nuclear Reactor Regulation (NRR) staff in response to the associated NRR Directors memoranda 2 were inadequate. Specifically, the appealers believe that NRRs revised answer to Operator License (OL) Program Feedback item 401.55 (ML18178A581) is misleading, contradictory, and incorrect. The appealers hold the position that the intent of Tier 1 (Emergency and Abnormal Plant Evolutions) written examination questions is to test the applicants knowledge of abnormal and emergency procedures.

Since I was not involved with the review of the original DPO, and, given the programmatic importance of the issues raised by the appealers, it was necessary to take sufficient time to have discussions with the appealers, my predecessor, and staff involved with evaluating the original DPO. Furthermore, my assessment of the appeal required the review of existing OL program guidance and discussions with headquarters OL program staff. All of these steps were needed to provide this statement of views and to determine if any additional corrective actions are needed to better address the underlying issues raised by the appealers.

CONTACT: Jeanne D. Johnston, NRR 301-415-1349 1

On October 10, 2017, several Senior Operations Engineers from RII submitted a differing professional opinion regarding operator licensing (OL) written examination test items for abnormal and emergency plant evolutions. Specifically, the submitters are concerned that a NRR policy interpretation on the adequacy of certain written examination questions will result in inadequate testing of operators knowledge of emergency and abnormal procedures.

2 On April 17, 2018, the acting NRR Director, Brian Holian, issued a Directors Decision that essentially agreed with the Ad Hoc DPO panel recommendations and offered additional clarifications. On April 27, 2018, Brian Holian tasked NRR staff to implement corrective actions consistent with the Ad Hoc DPO Panel recommendations with additional clarifications (ML18117A079).

My views on the appeal The appealers expressed a safety concern because the answer to OL Program Feedback item 401.55 does not ensure that emergency and abnormal condition procedures are adequately tested in reactor operator exams. I disagree with this assertion because it is unfounded in fact.

Based on a sampling, the headquarters OL program has assessed that reactor operator exams in all four regions sufficiently test knowledge of procedures in accordance with NUREG-1021.

As a general matter of principle, I believe that the Nuclear Regulatory Commissions (NRC) examination process for a control room operator should test for knowledge of plant systems, integrated response and procedures. The operating test is an essential and significant component of the operator licensing process that provides a major opportunity to test an applicants knowledge of procedures. The written examination, which is the focus of this DPO and appeal, provides a complementary measure of an applicants overall level of knowledge of plant systems, integrated response, and procedures.

On page 6 of the appeal, the appealers specified four flaws with the revised answer to OL Program Feedback item 401.55. My views on the four flaws are as follows:

1. The current guidance does not state the intent of Tier 1 questions. I think the current answer to feedback item 401.55 sufficiently addresses the intent of Tier 1 questions. Regarding the appealers position that all Tier 1 questions should test abnormal or emergency procedure knowledge, the headquarters OL program position is that each Tier 1 written examination question is not required to test procedural knowledge. A Tier 1 question can test system level knowledge without a reference to a specific procedure, so long as the question directly pertains to the selected emergency and abnormal evolutions knowledge or abilities (K/A) statement. The headquarters position is currently supported by the most current version of NUREG-1021, which is the guidance document used to implement the operator licensing requirements in Title 10 of the Code of Federal Regulations Part 55.

2. The current guidance is not clear on how to handle a situation where a Tier 1 question does not test abnormal or emergency procedure knowledge or when it is not possible to write a question because of the difficulty with the randomly selected K/A. I believe that the current guidance attempted to answer the specific question that was being asked related to rating Tier 1 questions that do not reference a specific procedure. I do consider that the answer could be improved for better clarity.

3. The current guidance does not define adequate balance of coverage, which is now necessary to define because the NRR decision memo used this phrase. I agree with appealers view that balance of coverage is not defined. Given the currently available guidance that exists today, NRC Chief Examiners in the regions have fairly broad discretion to apply their experience and judgment in preparing a written examination. As such, the NRC is susceptible to regional variations in exam content.

Thus far, no significant regional consistency concerns with written examinations have been identified by the headquarters OL program. Notwithstanding, balance of coverage should be better defined.

4. The current guidance contradicts itself. I agree, in part, with the appealers view.

In stepping through the four paragraphs of the revised answer to OL Program Feedback item 401.55, the first two paragraphs are intended to provide the context and intent of

Tier 1 questions as they relate to emergency and abnormal plant evolutions. The third paragraph is intended to apply to the rating of a specific question by itself. The fourth, final paragraph, is intended to apply to the exams overall balance of coverage. It is my impression from discussions with the appealers that they do not support the view of a two-part approach where a Tier 1 question is rated by itself, then considered as part of a holistic review of the exam for adequate balance of coverage. While it is arguable whether the current guidance is contradictory, I do consider that the revised answer could be improved for better clarity. Specifically, the guidance in item 401.55 could be enhanced to clarify the expectation for the Chief Examiner to review an exam outline for balance of coverage with the facility at the beginning of the examination development process. Such a review would be intended to preclude a licensees need to replace a significant number of questions after the draft exam has been written.

It should be noted that none of the other regional offices has taken exception to the revised answer to OL Program Feedback item 401.55. This apparent fact was even acknowledged by several of the appealers. Nevertheless, I found that the revised answer could be improved. In this regard, I reviewed the proposed solution offered by the appealers to replace the current version of the OL Program Feedback item 401.55 (see page 6 of the appeal).

I found the appealers proposed solution to be too restrictive, particularly with their proposal to specify a minimum number of questions to test procedure knowledge in Tier 1 and Tier 2 questions. My concern is that such guidance could potentially result in the balance of coverage being skewed too heavily toward procedure knowledge. For example, applying the appealers proposal could result in a reactor operators written examination having upwards of two-thirds of the 75 questions being procedures-based. This could have unintended consequences such as narrowing the scope of the written examination to procedural content included in abnormal and emergency operating procedures. These potential consequences and other possible implications have not been be evaluated with a diverse group of stakeholders.

Additional actions to be taken by NRR The additional information presented in the appeal justifies further corrective actions to be taken by NRR. I have directed the staff in NRRs Division of Inspection and Regional Support, Operator Licensing Branch, to do the following:

1. Clarify the answer to OL Program Feedback item 401.55 to be more clear on the intent of Tier 1 questions and rating of Tier 1 question that do not reference specific procedures or require procedural knowledge to answer;

2. Work with the OL staff in all regions to identify the most effective way to provide guidance on what constitutes adequate balance of coverage, with consideration of insights from an analysis of the balance of coverage in past exams from all regions;

3. Include the intent and rating of Tier 1 questions and balance of coverage as discussion topics for the upcoming biennial OL examiners conference in April 2019 so that input from all regions can be obtained; and

4. Incorporate into the next revision to NUREG-1021 guidance for adequate balance of coverage.

NRR will commit to revising the OL Program Feedback item 401.55 to better clarify the intent and rating of Tier 1 questions by no later than June 30, 2019.

NRR will commit to developing draft guidance for adequate balance of coverage by September 30, 2019. This draft guidance will be brought into the periodic update process for NUREG-1021.

ML19015A010 OFFICE NRR NRR NAME JJohnston HNieh DATE 1/22/2019 1/22/2019 Document 7: DPO Appeal Decision UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 December 10, 2019 MEMORANDUM TO: Bruno L. Caballero, Senior Operations Engineer Operations Branch 2 Division of Reactor Safety Region II Phillip G. Capehart, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II David R. Lanyi, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Mark A. Bates, Senior Operations Engineer Operations Branch 2 Division of Reactor Safety Region II Michael K. Meeks, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Daniel M. Bacon, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II FROM: Margaret M. Doane /RA/

Executive Director for Operations

SUBJECT:

DIFFERING PROFESSIONAL OPINION APPEAL INVOLVING OPERATOR LICENSING WRITTEN EXAMINATIONS - TIER 1 ITEMS (DPO-2017-007)

The purpose of this memorandum is to inform you of my considerations and conclusions regarding the Differing Professional Opinion (DPO) appeal you submitted on October 25, 2018.

The appeal raised concerns with the decision issued by the Acting Office Director of the Office of Nuclear Reactor Regulation (NRR) on April 27, 2018, regarding operating licensing written CONTACT: Christopher Cook, OEDO (301) 415-6397

Caballero, et al. 2 examinations (Agencywide Documents Access and Management System (ADAMS) Accession No.: ML18117A132). I evaluated the issues you raised in the appeal using the process described in Management Directive 10.159, The NRC Differing Professional Opinions Program, to ensure appropriate agency action in this matter.

After careful consideration of your appeal, I have concluded the following:

1) Operator licensing examinations must continue to ensure that future licensed operators demonstrate proficiency of knowledge and ability to execute emergency and abnormal evolutions, including the site-specific abnormal and emergency operating procedures.

2) The Chief Examiner (CE) remains the responsible individual for ensuring the 10 CFR 55.41 and 55.43 items are appropriately sampled and that written examination coverage is balanced.

3) Neither the applicable regulations nor the history of the written examinations require a strict interpretation that each Tier 1 question include a specific reference to an abnormal or emergency plant procedure in order to be deemed fully satisfactory.

4) It is permissible for a CE to rate a Tier 1 question in need of editorial enhancement if the question does not reference a procedure when the CE determines that, on balance, the examination lacks sufficient testing of the abnormal or emergency operating procedures.

Conversely, a CE may accept a Tier 1 question that does not reference an abnormal or emergency operating procedure when the CE determines that, on balance, the examination includes sufficient testing of these procedures.

5) The current version of Operator Licensing Feedback Item 401.55 (updated June 13, 2019; ADAMS Accession No.: ML19169A208) does not represent a consensus view between CEs in all Regional Offices and NRR. This Feedback Item should be updated to remove ambiguity regarding acceptable balance of coverage for testing Tier 1 procedures.

To this end, I will task NRR and the regional offices to work together to revise Feedback Item 401.55 (ADAMS Accession No.: ML19169A208) and related NRC guidance, as necessary, for the purpose of defining appropriate balance for use by CEs to evaluate coverage of site-specific abnormal and emergency operating procedures in Tier 1 of the written examination.

Thank you for taking the time to raise your concerns to me and for the detailed information you provided to support your position and my review. Your willingness to raise concerns through the DPO process is consistent with our organizational values of Openness and Commitment.

EXECUTIVE DIRECTOR FOR OPERATIONS REVIEW AND DECISION PROCESS In order to better understand your concerns, I assigned the Deputy Executive Director for Materials, Waste, Research, State, Tribal, Compliance, Administration, and Human Capital Programs (DEDM), an Executive Technical Assistant from my office, and an attorney from the Office of the General Counsel, to review the issues. The DEDM review team gathered information through discussions with you, the Deciding Official, and the NRR Director who wrote the Statement of Views, and reviewed documents pertinent to this appeal. The information collected provided independent insights and perspectives for my consideration.

Caballero, et al. 3

Background

On October 10, 2017, you submitted a DPO related to the Operator Licensing Written Examinations - Tier 1 Test Items. The main documents in dispute were the Report on Interaction-17-09, NUREG-2014, ES-401 Tier 1 written examination test items (ADAMS Accession No.: ML17165A579) and Operator Licensing Feedback Item 401.55 (ADAMS Accession No.: ML17249A961). In response to your concerns, a panel was formed and tasked by the NRC DPO Program Managers to review your DPO. The DPO panel issued their report on March 19, 2018, to the Deciding Official, the acting director of NRR, and provided recommendations (ADAMS Accession No.: ML18078A009). Of note is Recommendation 1, which states that a CE should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the CEs responsibility to ensure balance of coverage throughout the entire exam. Recommendation 1 also states that a question should not be assessed as unsatisfactory if the only flaw is its lack of a link to an abnormal or emergency operating procedure.

On April 27, 2018, the Deciding Official issued his decision regarding the DPOs concerns as informed by the DPO Panel report and his own review (ADAMS Accession No.: ML18117A132).

He essentially agreed with the submitters regarding Recommendation 1, and he directed his staff to revise Operator Licensing Feedback Item 401.55 to clarify that CEs are allowed to make reasonable changes for balance of coverage throughout the examination. He agreed that Tier 1 test items should when relevant test abnormal/emergency procedure knowledge. However, he also stated that when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not a basis to remove the question from the test.

On August 15, 2018, the acting Director of the Division of Inspection and Regional Support (DIRS), NRR, issued a letter to the acting Director of NRR with the subject, Action of Differing Professional Opinion involving Operator Licensing Written Examinations - Tier 1 Items (ADAMS Accession No.: ML18225A149). The letter expanded on the Deciding Officials decision in several key ways that are germane to the DPO appeal. First, the context surrounding use of the phrase when relevant was discussed in several paragraphs. The DIRS Director expressed concern that a new or revised regulatory position could be implied if the Deciding Officials letter was interpreted as implying a statement of intent for Tier 1 questions in the answer to Feedback Question 401.55. To clarify, the DIRS Director stated that Tier 1 items do not, by definition, require a procedural basis for all knowledge and ability (K/A) categories.

Second, the DIRS letter stated that the current version of Feedback Item 401.55 meets the intent of the Deciding Officials letter. Third, the DIRS Director stated that the balance of coverage for an examination remains the responsibility of the CE and that the CE has the authority to request reasonable changes to ensure that balance of coverage exists.

In the appeal you submitted on October 25, 2018, you state that the reason for this appeal is that the corrective action taken for DPO-2017-007 following issuance of the Deciding Officials letter was inadequate. In your view, the corrective action, which was to revise Operator Licensing Feedback Item 401.55, did not properly state the intent of Tier 1-written examination questions and that the response remains incorrect, misleading, and contradictory. You also state that if the spirit and intent of the words in the DPO panel report and the Deciding Officials letter had been implemented, this appeal would not have been needed.

In response to your appeal, on January 22, 2019, the permanent Director of NRR issued his Statement of Views letter (ADAMS Accession No.: ML19015A010). He organized his letter

Caballero, et al. 4 around the four specific flaws you identified with Operator Licensing Feedback Item 401.55.

These four flaws (in bold) addressed by the NRR Director are as follows:

1. The current guidance does not state the intent of Tier 1 questions.

I think the current answer to feedback item 401.55 sufficiently addresses the intent of Tier 1 questions. Regarding the appealers position that all Tier 1 questions should test abnormal or emergency procedure knowledge, the headquarters OL program position is that each Tier 1 written examination question is not required to test procedural knowledge. A Tier 1 question can test system level knowledge without a reference to a specific procedure, so long as the question directly pertains to the selected emergency and abnormal evolutions knowledge or abilities (K/A) statement. The headquarters position is currently supported by the most current version of NUREG-1021, which is the guidance document used to implement the operator licensing requirements in Title 10 of the Code of Federal Regulations (10 CFR) Part 55.

2. The current guidance is not clear on how to handle a situation where a Tier 1 question does not test abnormal or emergency procedure knowledge or when it is not possible to write a question because of the difficulty with the randomly selected K/A.

I believe that the current guidance attempted to answer the specific question that was being asked related to rating Tier 1 questions that do not reference a specific procedure. I do consider that the answer could be improved for better clarity.

3. The current guidance does not define adequate balance of coverage, which is now necessary to define because the NRR decision memo used this phrase.

I agree with appealers view that balance of coverage is not defined. Given the currently available guidance that exists today, NRC Chief Examiners in the regions have fairly broad discretion to apply their experience and judgment in preparing a written examination. As such, the NRC is susceptible to regional variations in exam content. Thus far, no significant regional consistency concerns with written examinations have been identified by the headquarters OL program. Notwithstanding, balance of coverage should be better defined.

4. The current guidance contradicts itself.

I agree, in part, with the appealers view. In stepping through the four paragraphs of the revised answer to OL Program Feedback item 401.55, the first two paragraphs are intended to provide the context and intent of Tier 1 questions as they relate to emergency and abnormal plant evolutions. The third paragraph is intended to apply to the rating of a specific question by itself. The fourth, final paragraph, is intended to apply to the exams overall balance of

Caballero, et al. 5 coverage. It is my impression from discussions with the appealers that they do not support the view of a two-part approach where a Tier 1 question is rated by itself, then considered as part of a holistic review of the exam for adequate balance of coverage.

While it is arguable whether the current guidance is contradictory, I do consider that the revised answer could be improved for better clarity. Specifically, the guidance in item 401.55 could be enhanced to clarify the expectation for the Chief Examiner to review an exam outline for balance of coverage with the facility at the beginning of the examination development process. Such a review would be intended to preclude a licensees need to replace a significant number of questions after the draft exam has been written.

On June 13, 2019, NRR updated the response to Operator Licensing Feedback Item 401.55 and published it on the NRC web site (ADAMS Accession No.: ML19169A208). The updated Feedback Item implements changes directed by the NRR Director in his Statement of Views letter. The updated feedback states that the intent of Tier 1 questions is to test an applicants knowledge of emergency and abnormal evolutions. It also states, citing the K/A catalogs, that an emergency plant evolution is defined as any condition, event or system which leads to entry into emergency operating procedures (EOPs), and an abnormal plant evolution is defined as any degraded condition, event, or symptom not directly leading to an EOP entry condition, but nonetheless, adversely affecting a safety function.

The updated Feedback Item also states that guidance in NUREG-1021 does not specifically require Tier 1 written examination questions reference a procedure or that the content of a Tier 1 written examination question be limited to only the information embedded, or contained within, the sites plant procedures. It states that a Tier 1 question should not be rated as unacceptable (i.e., unsatisfactory) only because the Tier 1 question does not reference a procedure or test knowledge of information that is contained within the facilitys plant procedure.

Likewise, the Tier 1 question should not be rated in need of editorial enhancement for the same reasons, unless there is a concern with balance of coverage within the exam.

To this end, the updated Feedback Item states that this Feedback Item is not intended to restrict the CEs ability to request reasonable changes to any written examination items when necessary, including those that do not test a relevant procedural concept. The CE may require reasonable changes, such as editorial enhancement, to the examination to achieve this balance, despite the relative merit of individual questions which may be replaced. The updated Feedback Item concludes that the CE and examination author should discuss expectations for adequate balance of testing procedures, along with the other 10 CFR 55.41 and 55.43 sample items, during the development, review, and approval of the written examination outline.

Review of Operator Licensing Examination Regulations, SECYs, and Guidance Documents In evaluating this appeal, the following key documents were considered, as well as NRC guidance and other documents related to Operator Licensing Examinations, including:

Title 10 of the Code of Federal Regulations (10 CFR) Part 55, Operators Licenses, requires that applicants for reactor operator (RO) and senior reactor operator (SRO) licenses pass both a written examination and an operating test (both initially and for requalification). Moreover, the regulations mandate that the license examinations must be developed and administered in

Caballero, et al. 6 accordance with 10 CFR 55.41, Written Examination: Operators, and 10 CFR 55.45, Operating Tests, for ROs or 10 CFR 55.43, Written Examination: Senior Operators, and 10 CFR 55.45 for SROs. The regulation at 10 CFR 55.40(a) states the following: The Commission shall use the criteria in NUREG-1021, Operator Licensing Examination Standards for Power Reactors, in effect 6 months before the examination date to prepare the written examinations required by §§ 55.41 and 55.43 and the operating tests required by § 55.45. The Commission shall also use the criteria in NUREG-1021 to evaluate the written examinations and operating tests prepared by power reactor facility licensees.

Pursuant to 10 CFR 55.40(b)(1), power reactor facility licensees may prepare, proctor, and grade the written examinations required by 10 CFR 55.41 and 10 CFR 55.43 and may prepare the operating tests required by 10 CFR 55.45 as long as they prepare the required examinations and tests in accordance with the criteria in NUREG-1021. NUREG-1021 establishes the policies, procedures, and practices for administering the required initial and requalification written examinations and operating tests. These standards describe the provisions of the Atomic Energy Act of 1954, as amended, and the regulations on which the operator licensing program is based. They also ensure the equitable and consistent administration of examinations to all applicants and licensed operators at all licensee facilities that are subject to the regulations.

Pursuant to 10 CFR 55.40, Implementation, [t]he Commission shall use the criteria in NUREG-1021in effect 6 months before the examination date to prepare the written examinations required by [10 CFR] 55.41 and [10 CFR] 55.43 and the operating tests required by [10 CFR] 55.45. The Commission shall also use the criteria in NUREG-1021 to evaluate the written examinations and operating tests prepared by power reactor facility licensees pursuant to paragraph (b) of this section.

The rulemaking documents for the 1986 Amendments to 10 CFR Part 55 includes SECY-84-76, SECY-84-76A, SECY-84-76B, SECY-86-123, SRM-86-123, the proposed rule at 49 FR 46428, and the final rule at 52 FR 9453. These documents cover the promulgation of the rules, the contemporary understanding of the rules, and the responses to public comments for 10 CFR Part 55, including 10 CFR 55.41 and 10 CFR 55.43. While these records document the rationale for written exams, they do not include a discussion on a specific makeup of the written exams aside from the rules requirement that the examination constitute a representative sample of the listed items.

Information Notice 88-40 is the Examiners' Handbook for Developing Operator Licensing Examinations (https://www.nrc.gov/reading-rm/doc-collections/gen-comm/info-notices/1988/in88040.html). This information notice provided addressees copies of NUREG/BR-0122, Examiners' Handbook for Developing Operator Licensing Examinations and provided additional guidance that the format of the operator licensing examinations administered under 10 CFR 55.41 or 55.43 should reflect the sampling plans contained in the handbook. However, since that time, the NRC has promulgated NUREG-1021 which stated, in Revision 9, that it explicitly supersedes those standards and that NUREG/BR-0122 is no longer in effect. Additionally, the information notice and the handbook utilize the same term, evolution, with the same definition, when describing the content of Tier 1 questions. The handbook further stated that the outline provided is not intended to dictate how many K/As or which K/As should be included in any outline but that the handbook, other guidance, plant-specific priorities, and the examiners knowledge contribute to making such decisions.

Caballero, et al. 7 SECY-98-266, Final Rule-Requirements for Initial Operator Licensing Examinations, along with its associated Voting Record, Staff Requirements Memorandum, and the proposed and final rules (62 FR 42426; 64 FR 19868). As was identified in your supplemental email, the background section in the SECY and the final rule describe the site-specific written examination as covering plant systems, emergency and abnormal plant procedures, and plant-wide generic knowledge and abilities. While these categories roughly correspond to the tiers that were discussed in your appeal, I did not see this as dispositive on the makeup of exams. This discussion is only included in the general background section and is not tied to the rules that were being promulgated at the time. As a result, I did not find that this was a clear and deliberate statement as to the required makeup of an examination, and even so, that it could not be changed through later guidance.

NUREG-1021, Revision 11, Chapter ES-401 (pg. 1; ADAMS Accession No.: ML17038A432) states:

The content of the written licensing examinations for ROs and SROs is dictated by 10 CFR 55.41, Written Examination: Operators, and 10 CFR 55.43, Written Examination: Senior Operators, respectively. Each examination shall contain a representative selection of questions concerning the knowledge and abilities (K/As) and skills needed to perform duties at the desired license level. Both the RO and SRO examinations will sample the 14 items specified in 10 CFR 55.41(b), and the SRO examination will also sample the 7 additional items specified in 10 CFR 55.43(b).

Three tiers of information comprise the licensed operator written examination, as defined and used in NUREG-1021. In particular, Section D.1.b of the NUREG discusses preparation of the examination outline, and describes each tier as follows:

Systematically and randomly select specific K/A statements (e.g., K1.03 or A2.11) from NUREG-1122 (for PWRs) or NUREG-1123 (for BWRs) to complete each of the three tiers (i.e., Tier 1, Emergency and Abnormal Plant Evolutions; Tier 2, Plant Systems; and Tier 3, Generic Knowledge and Abilities) of the applicable examination outline. (pg. 3)

The specific K/A topics, which must be examined in order for the examination to meet requirements in 10 CFR 55, are detailed in Form ES-401 of NUREG-1021, with the -1 version of the form specific to boiling water reactors and the -2 version specific to pressurized water reactors. Footnotes on Form ES-401 discuss usage of the form, as does Attachment 1 to Section ES-401. Attachment 1 provides one acceptable sampling methodology for randomly selecting K/As within the structure of the written examination outline. Throughout Form ES-401 and Attachment 1, Tier 1 topics are consistently referred to as emergency and abnormal plant evolutions.

The current version of NUREG-1122 (Rev. 2, Supp. 1; ML102571881), Section 1.10, Emergency and Abnormal Plant Evolutions, defines an emergency plant evolution (EPE) as any condition, event, or symptom which leads to entry into Emergency Operating Procedures (EOPs). Likewise, an abnormal plant evolution (APE) is any degraded condition, event, or symptom not directly leading to an EOP entry condition. The NUREG goes on to state that for each condition, there are degrees of severity. The EOP entry conditions were used as the bases for classifying a condition either as an EPE or an APE. Any abnormal condition which

Caballero, et al. 8 degrades as to threaten the plant safety will result in entry into the EOPs is treated as an emergency condition.

While the authors original prerogative for writing the title of Tier 1 of ES-401, NUREG-1021, to focus on K/A topics for evolutions versus procedures may be lost to the passage of time, and although these two words do mean different things, the NRCs practical use of these terms inseparably ties the two words together. For example, EOP entry conditions were used as the bases for classifying a condition either as an EPE or an APE (pg. 1.10, NUREG-1122, Rev 2).

Regardless, the goal of the Tier 1 portion of the written examination remains unchanged; namely to reliably test the ROs and SROs knowledge of, and ability to, operate the power reactor safely during emergency and abnormal conditions.

EXECUTIVE DIRECTOR FOR OPERATIONS ASSESSMENT The CE is responsible for confirming the 10 CFR 55.41 and 55.43 items are appropriately sampled and that coverage of the K/A categories in the written examination is balanced. This is practically executed through the CEs concurrence on NUREG-1021, Form ES-201-2, as part of preparation activities to develop the examination outline. During the written examination review, the CE may also assess a question to need editorial enhancement through the application of NUREG-1021, Forms ES-401-9 or ES-401N-9. If the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to a specified K/A statement of the test item is weak, the CE should not be prohibited from assessing the question as needing enhancement for any reasonable situation.

The documents discussed above do not support the interpretation that each Tier 1 test question must include a specific reference to an abnormal or emergency plant procedure. As discussed earlier, the three tiers discussed in NUREG-1021, Revision 11, are defined as: Tier 1, Emergency and Abnormal Plant Evolutions; Tier 2, Plant Systems; and Tier 3, Generic Knowledge and Abilities Categories. While definitions for evolutions and procedures can be debated, the safety objective of Tier 1 is clear: demonstrated knowledge and proficiency of how to safety operate the plant during emergency or abnormal conditions. These facility evolutions are a significant aspect of operating a nuclear power plant safely, and knowledge of these evolutions, including the use of abnormal and emergency procedures as appropriate, must be robustly understood by each licensed operator.

As a result, it is permissible for a CE to assess a question as in need of editorial enhancement if the question does not reference a procedure if the CE determines that, on balance, the examination lacks sufficient testing of abnormal or emergency operating procedures. This is because the CE has the responsibility to evaluate each examination separately, based on the examiners knowledge of the plant. The CE is therefore allowed to emphasize procedures in Tier 1 if the CE determines this emphasis is important to safety based on the plant-specific complexity of the emergency or abnormal operating procedures. Conversely, a CE may accept a Tier 1 question that does not reference an abnormal or emergency procedure if the CE determines, on balance, the examination sufficiently tests these procedures.

Licensing examinations must continue to ensure power reactor operators have demonstrated proficiency of knowledge in elements required by NRC regulations, including robust knowledge of, and the ability to, execute site-specific abnormal and emergency operating procedures when called upon. As discussed above, CEs play a critical safety role in development of the operator licensing examination. As evidenced by your appeal however, the current version of Operator Licensing Feedback Item 401.55 (ML19169A208) is not satisfactory. In particular, this

Caballero, et al. 9 Feedback Item does not represent a consensus view between CEs across the NRC regarding how to determine balance of coverage, particularly of Tier 1 procedures. Defining appropriate balance of coverage is necessary for the Feedback Item to be aligned with our Principles of Good Regulation. In particular, the principle of clarity discusses how there should be a clear nexus between regulations and agency goals and objectives whether explicitly or implicitly stated. Agency positions should be easily understood and easily applied. This DPO appeal decision demonstrates that the current version of Operator Licensing Feedback Item 401.55 should be improved to clarify its use of balance of coverage.

The enclosure tasks the Director of NRR and the four Regional Administrators to revise this guidance document. The revised guidance should provide CEs with a methodology to openly and reliably determine if an examination has appropriate balance of coverage of the site-specific abnormal and emergency operating procedures. The new guidance should also provide sufficient clarity so that independent CEs will consistently agree that an examination is or is not balanced in its coverage of Tier 1 procedures. Licensing examinations must continue to ensure that future licensed operators demonstrate a proficiency of knowledge and the ability to safely operate the power reactor under all foreseeable conditions.

In accordance with MD 10.159, a summary of this appeal decision will be included in the Weekly Information Report posted on the NRCs public Web site to advise interested employees and members of the public of the outcome.

Enclosure:

OEDO Ticket Assignment Form

ML19333B979 *via email Non-Sensitive Publicly Available SUNSI Review Sensitive Non-Publicly Available OFFICE OEDO/AO OGC DEDM EDO NAME CCook *JGillespie SWest MDoane DATE 12/02/19 12/02/19 12/10 /19 12/ 10 /19 OEDO TICKET ASSIGNMENT REQUEST FORM Task Due Date: TBD NRR (lead)

RII (co-lead)

Assigned Office(s): RI RIII RIV CC Routing:

N/A (Mark N/A if None)

DEDM alignment brief prior to briefing the EDO regarding actions to clarify guidance discussing balance of coverage associated Description of Task:

with abnormal and emergency plant procedures in Tier 1 of the operator license written examination.

Background:

DPO-2017-007-Appeal: EDO Decision dated December XX, 2019 (MLYYDDDAYYY).

Revise the content of Feedback Item 401.55 (ADAMS Accession No.: ML19169A208) and Special Instructions for Assigned Office(s): related NRC guidance, as necessary, for the purpose of defining appropriate balance for use by CEs to evaluate coverage of site-specific abnormal and emergency operating procedures in Tier 1 of the written examination.

Type of Response (memo, email, etc.) EDO briefing NOTE: Please attach a copy of any original incoming correspondence that will be needed as part of the ticket action.

Enclosure 1