ML18150A469

DPO Case File: DPO-2017-007 (Public)

ML18150A469
Person / Time
Issue date:	10/10/2017
From:	Eugene Guthrie NRC/RGN-II
To:	Pedersen R NRC/OE
Figueroa G
References
DPO-2017-007
Download: ML18150A469 (59)

Text

DPO Case File for DPO-2017-007 The following pdf represents a collection of documents associated with the submittal and disposition of a differing professional opinion (DPO) from NRC employees involving Operator Licensing Written Examinations-Tier 1 Test Items.

Management Directive (MD) 10.159, NRC Differing Professional Opinions Program, describes the DPO Program. https://www.nrc.gov/docs/ML1513/ML15132A664.pdf The DPO Program is a formal process that allows employees and NRC contractors to have their differing views on established, mission-related issues considered by the highest level managers in their organizations, i.e., Office Directors and Regional Administrators. The process also provides managers with an independent, multi-person review of the issue (one person chosen by the employee). After a decision is issued to an employee, he or she may appeal the decision to the Executive Director for Operations (or the Commission, for those offices that report to the Commission).

Because the disposition of a DPO represents a multi-step process, readers should view the records as a collection. In other words, reading a document in isolation will not provide the correct context for how this issue was reviewed and considered by the NRC.

It is important to note that the DPO submittal includes the personal opinions, views, and concerns by NRC employees. The NRCs evaluation of the concerns and the NRCs final position are included in the DPO Decision.

The records in this collection have been reviewed and approved for public dissemination.

Document 1: DPO Submittal Document 2: Memo Establishing DPO Panel Document 3: DPO Panel Report Document 4: DPO Decision

Document 1: DPO Submittal

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 1 of 30

Summary of existing NRC decision or agencys stated position NRR/IOLB determined that Tier 1 written examination test items that test plant system design features, interlocks, and system operation adequately test applicant knowledge of emergency and abnormal evolutions on the site-specific written exam. This determination was documented in Record of Interaction (ROI) 17-09, NUREG 1021, ES-401 Tier 1 Written Exam Test Items (ML17165A579), and was disseminated to industry stakeholders in Operator Licensing Feedback Item 401.55 (ML17249A961).

Operator Licensing Feedback Item 401.55 included two examples of Tier 1 written examination test item topics, one for PWR K/A Abnormal Evolution Pressurizer Pressure Control Malfunctions, and the other for BWR K/A Emergency Evolution High Drywell Pressure, and NRR/IOLBs feedback to industry stated that written examination test items were acceptable for these Tier 1 topics, if the test item solely tested how the system worked (i.e., Tier 2, Plant Systems knowledge aspect).

Reason for DPO The writers of this DPO are identified on Page 17; the purpose of this DPO is to require NRR/IOLB to revise its recent policy determination for writing and assessing Tier 1 written examination test items. NRR/IOLBs position has the potential to undermine the 10 CFR 55.41 requirement that the written examination contain a representative selection of questions on the knowledge, skills, and abilities needed to perform licensed operator duties. Specifically, NRR/IOLBs policy interpretation will result in fewer questions that test the operators knowledge of abnormal and emergency procedures, in accordance with 10 CFR 55.41(b)(10), on the site-specific written examination.

Title 10 of the Code of Federal Regulations (10 CFR) Part 55, Operators Licenses, requires that applicants for reactor operator (RO) and senior reactor operator (SRO) licenses pass a written examination. The regulation at 10 CFR 55.40(b) allows power reactor facility licensees to prepare the site-specific written examinations, provided that the facility licensee prepares the site-specific written examination in accordance with the criteria contained in NUREG-1021, Operator Licensing Examination Standards for Power Reactors.

In accordance with NUREG-1021, ES-401, Preparing Initial Site-Specific Written Examination, the site-specific written exam must be comprised of three parts:

Tier 1: Emergency/Abnormal Plant Evolutions Tier 2: Plant Systems Tier 3: Generic Knowledges & Abilities Interaction between Region II and NRR/IOLB to collaborate on clarifying the intent of Tier 1 Written Exam test items is documented in ROI 17-09 (March-June 2017 time frame). Also a teleconference was held on August 24, 2017, prior to disseminating the Operator Licensing Feedback Item 401.55 to industry stakeholders. NRR/IOLB program office Branch Chief began the teleconference by stating that opinions regarding the final resolution to ROI 17-09 (on 6 17) were not going to be discussed during the teleconference, and said the purpose of the phone call was to discuss enhancements for the phrasing or wording to the response to Operator Licensing Feedback Item 401.55, prior to disseminating to industry stakeholders.

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 2 of 30

NRR/IOLBs response to Region II in ROI 17-09 (ML17165A579) stated that testing applicants knowledge of the emergency or abnormal procedure content for Tier 1 written examination test items was more restrictive than currently called for by NUREG-1021, and that this was a change in policy which would require a revision or supplement to NUREG-1021. During the 8/24/17 teleconference, Region II asked the program office to explain the difference between Tier 1 and 2 written test items. The program office staff said they did not know what the authors of NUREG-1021 intended when constructing the two tiers, and did not attempt to justify or explain the difference between the Tiers during the teleconference.

The NRR/IOLB response in both ROI 17-09 and Operator Licensing Feedback Item 401.55 included: Tier 1 test items dont need to reference a procedure and Tier 1 test items dont require EOP/AOP entry, both which may be true, depending on a test items particular construction. However, NRR/IOLBs response did not address the fundamental question regarding the intent of Tier 1 Emergency/Abnormal Evolution test items on the plant-specific written examination, nor did it explain the intended difference between Tier 1 and Tier 2 test items.

Impact on Agencys Mission The impact of NRR/IOLBs policy interpretation will result in less testing of applicants knowledge of abnormal and emergency procedures on the site-specific NRC written exam; a representative selection of 10 CFR 55.41(b)(10) may not be ensured on the NRC site-specific written examination. Although the NRC does not train operators, the NRC is responsible for testing operators after they complete their training program. Facility licensees make adjustments to their initial training program based, in part, on the NRC exam content.

NRR/IOLBs policy interpretation means that the site-specific written exam will test less abnormal/emergency procedure knowledge; therefore, it is likely that licensee training programs may be inappropriately adjusted to reflect the NRR/IOLB policy determination. Actual consequences may occur when operator procedure knowledge declines, and plant events are not properly mitigated by operators because of inadequate procedure knowledge. See the DPO Section titled Public Health and Safety Concern for a disturbing trend identified by INPO in 2017 regarding operator knowledge of abnormal procedures. The impact of NRR/IOLBs policy determination, in turn, could affect the agencys Strategic Plan (ML14246A439), Safety Objective 1, Prevent and Mitigate Accidents and Ensure Radiation Safety, because licensed operators mitigate accidents during abnormal and emergency events.

The impetus for ROI 17-09 was actual inconsistencies in facility licensee interpretations of how to develop Tier 1 written exam test items; the ROI presented three differing viewpoints that currently exist. During the Region II 2017 Office Assessment, NRR/IOLB identified that several Tier 1 draft test items, as submitted by the facility licensee for the 2016 Brunswick NRC examination, were categorized as deficient (by the examiner) because the questions did not test applicant knowledge of emergency/abnormal procedures. The NRR/IOLB assessment (ML17095A958) concluded that NUREG-1021 did not require Tier 1 test items to test applicant knowledge of emergency/abnormal procedure content, and that the examiners evaluation (that the draft test items did not match the intent of the K/A) was wrong.

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 3 of 30

Proposed Alternative As previously documented in ROI 17-09, the submitters of this DPO propose the following alternative, including its basis, instead of NRR/IOLBs position regarding Tier 1 written test items.

A Tier 1 written test item should, whenever possible, within the wording of the K/A statement, test the applicants knowledge of the abnormal condition or emergency procedure content, for example:

o an immediate operator action, o an important subsequent manual operator action, or o overall mitigative strategy of the off-normal or emergency procedure.

Tier 1 test items where the stem of the question mentions an ongoing abnormal/emergency evolution, but where the test item can be answered solely using Tier 2 (Plant Systems) knowledge, contain, in a sense, window dressing; these test items should be assessed as K/A mismatches, but assessed as enhancement required, in accordance with ES-401-9, Written Exam Worksheet.

If testing knowledge of the abnormal condition or emergency procedure content is not possible given the wording of the K/A, then accept the question as meeting the K/A as long as all other aspects of the K/A are met.

The benefit of this proposed alternative is that testing procedure knowledge is promoted, there is no penalty for the exam writer, and the random sample initially drawn is preserved.

Basis for Proposed Alternative NRR/IOLBs response to Region II in ROI 17-09 (ML17165A579) stated that testing applicants knowledge of the emergency or abnormal procedure content for Tier 1 written examination test items was more restrictive than currently called for by NUREG-1021, and that this was a change in policy which would require a revision or supplement to NUREG-1021.

Testing applicants knowledge of EOPs and AOPs on the site-specific written exam is not a change in policy; there has always been precedence for testing applicant knowledge of EOPs and AOPs on the site-specific written examination. For example, even the original 1983 version of NUREG-1021, ES-203, Structure of Written Examination Administered to Reactor Operators

- Power Reactors (ML15027A434) stated:

In general, the candidate must demonstrate complete knowledge and understanding of the symptoms, automatic actions, and immediate action steps specified by abnormal and emergency procedures.

The original (1983 version) of ES-203 written exam cover page included Category 4, as shown below:

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 4 of 30

Testing applicants knowledge of EOPs and AOPs in Tier 1 of the site-specific written exam is not more restrictive than currently called for in NUREG-1021 because the current versions of NUREG-1122, Rev. 2 and NUREG-1123, Rev. 2 (PWR & BWR Knowledge and Abilities Catalogs, respectively) Section 1.10, Emergency and Abnormal Evolutions, contain the following definition of an emergency and abnormal evolution:

EMERGENCY EVOLUTION: An emergency plant evolution is any condition, event, or symptom which leads to entry into the EOPs.

ABNORMAL EVOLUTION: An abnormal plant evolution is any degraded condition, event, or symptom not directly leading to an EOP entry condition, but nonetheless, adversely affecting a safety function.

The current version of NUREG-1021 (Rev. 11), includes Form ES-401-1 (BWR Written Exam Outline) and Form ES-401-2 (PWR Written Exam Outline), these forms identify Tier 1 as EMERGENCY and ABNORMAL PLANT EVOLUTIONS.

NRR/IOLBs policy interpretation does not ensure that the site-specific written exam tests vendor-specific (Westinghouse, Babcock & Wilcox, Combustion Engineering) EOPs, as required by NUREG-1021.

To illustrate this point, consider Westinghouse procedure ECA-2.1, Uncontrolled Depressurization of All Steam Generators, listed in the Westinghouse Owners Group EOP list of procedures:

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 5 of 30

NUREG-1021, Form ES-401-2, identifies 000040 Steam Line Rupture - Excessive Heat Transfer (W E12) as an emergency/abnormal plant evolution topic. The WE12 designator on Form ES-401-2 for this topic refers to PWR K/A Catalog Section 4.5, Westinghouse Emergency Plant Evolutions, E12: Uncontrolled Depressurization of all Steam Generators. (See PWR K/A Catalog Section 4.5 listed below, item listed on page 4.5-31.)

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 6 of 30

The intent of the 000040 Steam Line Rupture - Excessive Heat Transfer (W E12) topic is to test the applicants knowledge of the Westinghouse ECA-2.1 procedure. The vendor-specific AOPs and EOPs in the K/A Catalogs, and in ES-401, mirror the actual vendor procedures because the intent was to test the content of these important procedures.

Therefore, the NRR/IOLB policy determination, that applicants knowledge of EOPs for Tier 1 test items like this ECA-2.1 example is adequately tested using test items that test plant system design features, interlocks, and system operation, does not ensure that the site-specific written exam tests vendor-specific (Westinghouse, Babcock & Wilcox, Combustion Engineering) EOP content, which is required by NUREG-1021, ES-401.

Instead, the writers of this DPO contend that the site-specific written exam Tier 1 test items, like the vendor-specific EOPs example discussed above, should, whenever possible, test the applicants knowledge of:

an immediate operator action, an important subsequent manual operator action, or overall mitigative strategy associated with the emergency procedure During the 8-24-17 teleconference with Region II, when asked to explain the difference between Tier 1 and Tier 2 site-specific written exam items, NRR/IOLBs response was they did not know what the authors intended when constructing two Tiers, and subsequently did not attempt to justify or explain their policy determination for Tier 1 test items.

The original Examiner Standards Handbook included a graphic representation of the intended content of the generic and site-specific portions of the written examination: (see next page)

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 7 of 30

This Examiners Standard Handbook representation depicts the middle Tier 1 slice as EPEs and APEs, i.e., emergency and abnormal plant evolutions. The Tier 1 slice is represented as a different slice compared to Tier 2; Tier 2 is plant systems knowledge. NRC Information Notice 88-40 (Examiners Handbook for Developing Operator Licensing Examinations) indicated that approximately 27% of the RO exam (40 % of the SRO exam) should normally sample emergency and abnormal plant evolutions in Tier 1. This graphical representation explained how the 10 CFR 55.41 (a) requirement for a representative selection was being met.

Based on the preceding discussion of historical precedence, definitions in the K/A catalogs, and ES-401 requirements for two Tiers, the writers of this DPO contend the following items:

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 8 of 30

Tier 2 site-specific written exam items are intended to test the applicants knowledge of plant systems, and to a lesser degree, some procedure knowledge as it relates to A.2 K/As. The intent of Tier 2 is to test an applicants mastery of how the plant works.

Tier 1 site-specific written exam items are intended to test the applicants mastery of how to operate the plant in accordance with the abnormal (i.e., off-normal) and emergency operating procedures, during abnormal and emergency evolutions.

NRR/IOLBs response to Region II during the 8-24-17 teleconference was that systems knowledge and Emergency/Abnormal procedure knowledge cannot be separated.

The writers of this DPO contend that systems knowledge and procedure knowledge can be separated. Systems knowledge includes, for example, plant system design features, interlocks, flow paths, actuation logic, and set points. Abnormal/emergency procedures knowledge includes, for example, required immediate operator actions, important subsequent actions, and overall mitigative strategy for off-normal and emergency evolutions. Consider the following before and after examples listed below, for the same Tier 1 K/A. In the before example, a fault inside containment causes pressure to rise above the Main Steam Trip Valve auto-isolation set point. In the after example, the applicants knowledge of the overall mitigative procedure strategy is tested.

Before:

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 9 of 30

After:

As can be seen by this example, it is not difficult to avoid testing abnormal condition procedure content, and still meet the lower level wording of a Tier 1 K/A statement. Systems knowledge and procedures knowledge form the basis for the NRCs confidence when issuing a license to an operator applicant - the NRC issues a license to an applicant who knows 1) how the plant works (system knowledge) and 2) how to operate the plant, in accordance with procedures, during abnormal/emergency situations (procedure knowledge).

During the 8-24-17 teleconference with Region II, NRR/IOLB stated that changes to the yet unpublished K/A Catalog Emergency/Abnormal Stem Statements would be made (for the future Revision 3, which was published in the Federal Register in April 2017), commensurate with the recent NRR/IOLB policy determination in ROI 17-09 and Operator Licensing Feedback Item 401.55.

The last sentence of ROI 17-09 (Recommended Action/Resolution Section) referred to the new K/A Catalog (Revision 3) industry effort. The new BWR and PWR catalogs were the result of a joint agency-industry effort, compiled of stakeholder teams with years of industry experience,

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 10 of 30

and were published in the Federal Register in April 2017. The new catalogs, as published in the Federal Register in April 2017, included basis statements for the emergency/abnormal knowledge and ability stem statements, which were intended to provide insight for the intent of Tier 1 topics. (See Table 4 below).

If NRR/IOLB removes or alters these basis statements for the upcoming Revision 3 K/A Catalogs, then this refutes the experience and wisdom of the industry effort, and creates an inconsistency between the operating fleet K/A catalogs and the AP-1000 K/A Catalog, NUREG-2103. (See the following pages.)

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 11 of 30

Note that the AP-1000 abnormal/emergency procedure numbers and titles are included on Form ES-401N-2 (See following page)

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 12 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 13 of 30

Lastly, based on NRR/IOLBs policy interpretation, there is no need to even have separate Tier 1 and Tier 2 categories since the NRR/IOLB policy interpretation relies only on the lower wording of the K/A statement, and discounts the Tier in which the K/A was selected. It is not reasonable to assume that the authors of ES-401 arbitrarily designed two tiers that were not different. The reasonable assumption is that there was a reason for separate Tier 1 and Tier 2 categories. Tier 2 is Plant Systems, which was meant to test the configuration of the plant systems and their design. The void is filled by Tier 1, which was meant to test how to operate the plant in accordance with plant procedures when confronted with abnormal/emergency situations that challenge safety.

Some may say that, since the site-specific written exam is not the only portion of the NRC exam that tests abnormal and emergency procedures, it doesnt matter if the quantity of test items requiring knowledge of the content of abnormal/emergency procedures is reduced.

Although the operating portion of the NRC exam does test abnormal and emergency procedure knowledge, the control room team and open-reference operating exam format should not be relied upon to satisfy the intent of 10 CFR 55.41(b) which requires a representative selection of the fourteen items, for each individual on a written examination.

Form ES-201-2, Examination Outline Quality Checklist, contains the following items that the facility licensee and Chief Examiner must assess for each NRC exam:

Written Exam Item 1.c: Assess whether the outline overemphasizes any systems, evolutions, or generic topics.

General Item 4.b: Assess whether the 10 CFR 55.41, 55.43, and 55.45 sampling is appropriate General Item 4.e: Check the entire exam for balance of coverage.

The Form ES-201-2 Quality Check Items (1.c, 4.b, and 4.e) will not be met because the result of NRR/IOLBs policy is that the number of site specific written exam test items that test abnormal/emergency procedure content will be lower, especially for RO exams, which leads to overemphasis of testing plant systems knowledge on the plant specific written exam.

To illustrate this point, a reactor operator (RO) written exam sample plan was reviewed to identify how many test items would test knowledge of abnormal/emergency procedure content when NRR/IOLBs policy determination was implemented. Specifically, the criteria used in this review was, unless the K/A statement wording specifically included the word procedure, the test item was assumed to solely test a plant system design feature, interlock, or system operation aspect. The review results identified that only three of the 27 Tier 1 test items would test abnormal/emergency procedure content. Out of all three Tiers, (75 questions on the RO sample plan), a total of only NINE items would test abnormal/emergency procedure knowledge when the NRR/IOLB policy determination was implemented. This review meant that only 12%

of the RO test items would be required to test abnormal/emergency procedure knowledge when NRR/IOLBs policy determination was implemented.

Based on this initial review, three additional RO written exam sample plans were independently reviewed, using the same criteria listed above, to verify the first review results.

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 14 of 30

Percentage of 75 RO test items that would test abnormal/emergency procedure knowledge Second Sample Plan Reviewed:

9% (7/75)

Third Sample Plan Reviewed:

10% (8/75)

Fourth Sample Plan Reviewed:

9% (7/75)

Based on these review results, implementation of NRR/IOLBs policy determination will result in less site-specific RO written exam test items that test abnormal/emergency procedure content, and is a departure from NRC Information Notice 88-40, which described the original methodology for fulfilling the 10 CFR 55.41 representative selection requirement for the site-specific written exam, including 10 CFR 55.41(b)(10).

To counter the opposing view that the operating test may somehow compensate for a reduced number of RO abnormal/emergency procedure test items on the site-specific written exam, during the scenario portion of the operating examination, SRO applicants knowledge of EOPs is evaluated more so than RO applicants knowledge of EOPs, since the SRO applicants direct RO emergency procedure actions. The site-specific written exam provides the only opportunity to evaluate knowledge of EOPs for RO applicants on an individual basis. The control room team and open-reference operating exam format should not be relied upon to satisfy the intent of 10 CFR 55.41(b) which requires a representative selection of the fourteen items, for each individual on a written examination.

One must look no further than pass rates on the operating portion of the examination. Pass rates on the dynamic simulator portion of the examination approach 100%. The percentage of applicants that receive satisfactory scores on more than 80% of the Systems Job Performance Measures (JPMs) also approaches 100%. The majority of operating test failures occur due to the contribution of failing scores on Administrative JPMs; citing a lower overall JPM pass rate for the purpose of justifying a reduced number of abnormal/emergency written test items is misleading because the majority of JPM Section failures occur as a result of the Administrative JPM contribution to the JPM Section failure. Therefore, although the dynamic simulator scenarios and JPMs test applicant knowledge of emergency and abnormal procedures, these sections of the operating exam do not discriminate at the same level as the site-specific RO written exam.

It is noteworthy that, during operating exam scenarios, applicants who are examined in the RO (a.k.a operator-at-the-controls - OATC) and Balance of Plant (BOP) positions are typically provided direction from the SRO (a.k.a control room supervisor, CRS), who reads steps from the emergency procedures. When the SRO provides the directives, the OATC and BOP applicants then carry out the required emergency actions. This operating exam format tests the OATC and BOP applicants ability to perform emergency actions, but it does not test the same in-depth emergency procedure knowledge that was intended to be tested in the Tier 1 portion of the site-specific written exam.

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 15 of 30

Public Health & Safety Concern On September 7, 2017, NRR/IOLB published Operator Licensing Feedback Item 401.55, which means that facility licensees will likely develop Tier 1 plant-specific written test items that test less abnormal and emergency operating procedure knowledge. Facility licensees may inappropriately make adjustments to their initial license training program based, in part, on NRR/IOLBs response to Operator Licensing Feedback Item 401.55; therefore, actual consequences may occur as operator procedure knowledge continues to decline (see next paragraph) and plant events are not properly mitigated by operators due to lack of procedure knowledge. Because of NRR/IOLBs policy determination, applicant knowledge of the content of abnormal and emergency procedures, in accordance with 10 CFR 55.41(b)(10), will not be adequately tested on the site-specific written examination, prior to issuance of operator licenses.

On June 1, 2017, INPO identified operator weaknesses in the implementation of abnormal operating and off normal alarm response procedures in INPO IER 17-5, Line of Sight to the Core, (ML17171A309):

Reviews of noteworthy events and evaluation data indicate that weaknesses exist in implementation of abnormal operating and alarm response procedures. In the majority of these events, abnormal plant conditions satisfied the entry conditions for multiple procedures, and operators chose implementation paths that resulted in inappropriate operator responses. Gaps in knowledge of abnormal and alarm response procedures led to a perception that it was allowable to be selective concerning procedural steps.

Further, several of the events revealed procedural deficiencies resulting from inadequate revisions that, in turn, led to flawed rationale by operators.

On March 28, 2010 at H. B. Robinson a 4KV cable fault caused a fire and damaged a transformer, and a second electrical fault and fire was caused by the operators when they inappropriately attempted to reset the generator lockout relay without first ensuring the cause of the lockout was cleared. The part of the event that is relevant to this DPO occurred while the crew was implementing the EOP response following the reactor trip; the operators came very close to losing a RCP seal, due to thermally shocking the seal, because they failed to implement Step 19 (Check RCP Seal Cooling) correctly. Specifically, the operators opened the RCP seal injection valve even though thermal barrier cooling was lost for 39 minutes coincident with no seal injection for 10 to 15 minutes. The operators complied with the procedure step for verifying that a charging pump was operating, when they made the determination that adequate seal injection existed, but did not comply with the intent of the step to verify that seal cooling had been maintained. The March 2010 H.B. Robinson event is a strong example of why testing abnormal/emergency procedure knowledge on the site-specific written exam is important. The final significance determination for the two WHITE findings was documented in NRC Inspection Report 05000261/2011008 (ML110310469).

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 16 of 30

Summary of Concern NRR/IOLBs policy interpretation, and subsequent communication to industry stakeholders, will result in fewer items that test applicant knowledge of abnormal and emergency procedure content on the site-specific written examination; abnormal and emergency procedures are required to be tested on the site-specific written exam in accordance with 10 CFR 55.41(b)(10).

The control room team and open-reference operating exam format should not be relied upon to satisfy the intent of 10 CFR 55.41(b) which requires a representative selection of the fourteen items, for each individual on a written examination.

NRR/IOLBs policy interpretation does not ensure that the site-specific written exam tests vendor-specific (Westinghouse, Babcock & Wilcox, Combustion Engineering) EOPs, as required by NUREG-1021.

Facility licensees may inappropriately adjust training programs to reflect the NRR/IOLB policy determination.

Tier 1 site-specific written exam items are intended, whenever possible, to test the applicants mastery of how to operate the plant during abnormal and emergency evolutions, in accordance with the abnormal/emergency procedures. NRR/IOLBs communication to industry in Operator Licensing Feedback Item 401.55 means Tier 1 site-specific written exam test items will likely become an extension of Tier 2 test items. Tier 2 site-specific written exam items are intended to test the applicants systems knowledge of how the plant works, and to a lesser degree, some procedure knowledge as it relates to A.2 K/As.

If NRR/IOLB removes or alters the basis statements for the upcoming Revision 3 K/A Catalogs, as published in the Federal Register on April 2017, to facilitate their policy determination, then the experience and wisdom of the industry effort for the Catalogs will be lost, and an unnecessary inconsistency between the operating fleet K/A catalogs and the AP-1000 K/A Catalog, NUREG-2103 will exist.

Diffenng Professional Opinion: Operator Licensing Written Exam Tier I Test Items (Document Date:

10-10-17)

Signatures The writers of this DPO are Region II examiners, with extensive industry training and/or operating experience, many of which previously held SRO NRC licenses.

[Name (printed) I Signature I Date]

uuo C4EkLL.eL.o/&w(o7,0 /

M. &cø/

/ /3-_,

I I

D (7

%L k

)

,rL:d

/e/ /

LATIYr I

L%%tZ.

Page 17 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 18 of 30

The following pages contain important documents associated with this Differing Professional Opinion. Although these documents may be available in ADAMS, the writers of this DPO included these documents as part of the DPO.

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 19 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 20 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 21 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 22 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 23 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 24 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 25 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 26 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 27 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 28 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 29 of 30

Differing Professional Opinion: Operator Licensing Written Exam Tier 1 Test Items (Document Date: 10-10-17)

Page 30 of 30

Document 2: Memo Establishing DPO Panel

November 6, 2017 MEMORANDUM TO:

Raymond K. Lorson, Panel Chairperson Region I Matthew P. Emrich, Panel Member Office of the Chief Human Capital Officer Charles D. Zoia, Panel Member Region III THRU:

Anne T. Boland, Director /RA/

Office of Enforcement FROM:

Renée M. Pedersen /RA/

Sr. Differing Professional Views Program Manager Office of Enforcement

SUBJECT:

AD HOC REVIEW PANEL - DIFFERING PROFESSIONAL OPINION ON OPERATOR LICENSING WRITTEN EVALUATIONS - TIER 1 TEST ITEMS (DPO-2017-007)

In accordance with Management Directive (MD) 10.159, The NRC Differing Professional Opinion Program; and in my capacity as the Differing Professional Opinion (DPO) Program Manager; and in coordination with Anne Boland, Director, Office of Enforcement, Brian Holian, Acting Director, Office of Nuclear Reactor Regulation; and the DPO submitters; you are being appointed as members of a DPO Ad Hoc Review Panel (DPO Panel) to review a DPO submitted by several NRC employees.

The DPO (Enclosure 1) raises concerns about the recent policy on writing and assessing Tier 1 written examination test items. The DPO has been forwarded to Mr. Holian for consideration and issuance of a DPO Decision.

CONTACTS: Renée Pedersen, OE (301) 287-9426 Gladys Figueroa-Toledo, OE (301) 287-9497

R. Lorson, et al.

2 The DPO Panel has a critical role in the success of the DPO Program. Your responsibilities for conducting the independent review and documenting your conclusions in a report are addressed in the handbook for MD 10.159 in Section II.F and Section II.G, respectively.

The DPO Web site also includes helpful information, including interactive flow charts, frequently asked questions, and closed DPO cases, including previous DPO Panel reports. We will also be sending you additional information that should help you implement the DPO process.

Because this process is not routine, we will be meeting and communicating with all parties during the process to ensure that everyone understands the process, goals, and responsibilities.

Disposition of this DPO should be considered an important and time sensitive activity. The timeliness goal for issuing a DPO Decision is 120 calendar days from the day the DPO is accepted for review. In this case, the DPO was accepted for review on October 16, 2017. The timeliness goal for issuing this DPO Decision is February 13, 2018.

Process Milestones and Timeliness Goals for this DPO are included as Enclosure 2. The timeframes for completing process milestones are identified strictly as goalsa way of working towards reaching the DPO timeliness goal of 120 calendar days. The timeliness goal identified for your DPO task is 75 calendar days from the date of this memorandum (January 20, 2018).

Although timeliness is an important DPO Program objective, the DPO Program also sets out to ensure that issues receive a thorough and independent review. The overall timeliness goal should be based on the significance and complexity of the issues and the priority of other agency work. Therefore, if you determine that your activity will result in the need for an extension beyond the overall 120-day timeliness goal, please send an e-mail to (OD/RA) with a copy to DPOPM.Resource@nrc.gov and include the reason for the extension request and a proposed completion date for your work and a proposed timeliness goal for issuance of a DPO Decision. Mr. Holian is responsible for subsequently forwarding the request for a new DPO timeliness goal to the EDO for approval.

An important aspect of our organizational culture includes maintaining an environment that encourages, supports, and respects differing views. As such, you should exercise discretion and treat this matter appropriately. Documents should be distributed on an as-needed basis. In an effort to preserve privacy, minimize the effect on the work unit, and keep the focus on the issues, you should simply refer to the employees as the DPO submitters. Avoid conversations that could be perceived as hallway talk on the issue and refrain from behaviors that could be perceived as retaliatory or chilling to the DPO submitters or that could potentially create a chilled environment for others. It is appropriate for employees to discuss the details of the DPO with their co-workers as part of the evaluation; however, as with other predecisional processes, employees should not discuss details of the DPO outside the agency. If you have observed inappropriate behaviors, heard allegations of retaliation or harassment, or receive outside inquiries or requests for information, please notify me.

On an administrative note, please ensure that all DPO-related activities are charged to Activity Code ZG0007.

R. Lorson, et al.

3 We appreciate your willingness to serve and your dedication to completing a thorough and objective review of this DPO. Successful resolution of the issues is important for NRC and its stakeholders. If you have any questions or concerns, please feel free to contact me or Gladys.

We look forward to receiving your independent review results and recommendations.

Enclosures:

1. DPO-2017-007

2. Process Milestones and Timeliness Goals cc:

B. Holian, NRR M. Evans, NRR B. McDermott, NRR C. Haney, RII B. Caballero, RII D. Bacon, RII P. Capehart, RII M. Meeks, RII D. Lanyi, RII M. Bates, RII D. Lew, RI S. Rutledge, OCHCO R. Orlikowski, RIII A. Boland, OE G. Figueroa-Toledo, OE

R. Lorson, et al.

4

SUBJECT:

AD HOC REVIEW PANEL - DIFFERING PROFESSIONAL OPINION ON OPERATOR LICENSING WRITTEN EVALUATIONS - TIER 1 TEST ITEMS (DPO-2017-007) DATE: November 6, 2017 ADAMS Package: ML17307A046 MEMO: ML17307A053 - ML17290A536 - ML17307A059 OE-011 OFFICE OE: DPO/PM OE: DPO/PM OE: D NAME GFigueroa RPedersen ABoland DATE 11/ 06 /2017 11/ 06 /2017 11/ 06 /2017 OFFICIAL RECORD COPY

Document 3: DPO Panel Report

March 19, 2018 MEMORANDUM TO:

Brian E. Holian, Acting Director Office of Nuclear Reactor Regulation FROM:

Jeffrey A. Clark, DPO Panel Chair /RA/

Matthew P. Emrich, DPO Panel Member Charles D. Zoia, DPO Panel Member

SUBJECT:

DIFFERING PROFESSIONAL OPINION PANEL REPORT ON OPERATOR LICENSING WRITTEN EXAMINATIONS - TIER 1 ITEMS (DPO-2017-007)

In a memorandum, dated November 21, 2017, we were appointed as members of a Differing Professional Opinion (DPO) Ad Hoc Review Panel (DPO Panel) to review a DPO regarding Operator Licensing written examinations; Tier 1 Items. The DPO Panel has reviewed the DPO in accordance with the guidance in Management Directive 10.159, The NRC Differing Professional Opinion Program.

The results of the DPO Panels evaluation of the concerns raised in the DPO are detailed in the enclosed DPO Panel Report and is submitted for your consideration. Based on our review of concerns raised in the DPO, the DPO Panel made a recommendation, with two additional considerations, if implemented.

Please do not hesitate to contact us if you have any questions regarding the enclosed report.

CONTACT:

Jeffrey A. Clark, RIV/DRS 817-200-1180

Enclosure:

DPO Panel Report

1 Enclosure Statement of Issue (SOI)

The Operator Licensing and Training Branch (IOLB) of the Office of Nuclear Reactor Regulation (NRR) determined that a test item developed for Tier 1 of the site-specific written exam matches the intent of its knowledge or ability (K/A) statement if the test item solely tests plant systems knowledge, such as a design feature, interlock, or automatic operation. IOLB determined it was inappropriate to evaluate Tier 1 test items as enhancement required or inappropriate on Form ES-401-9, Written Examination Review Worksheet, when the test item did not test knowledge of emergency or abnormal procedures. This determination was documented in Record of Interaction (ROI) 17-09, NUREG 1021, ES-401 Tier 1 Written Exam Test Items (ML17165A579); was disseminated to industry stakeholders in Operator Licensing Feedback Item 401.55 (ML17249A961); and was communicated during an operator licensing examiner training session conducted by IOLB staff on October 19, 2017.

Based on the Panels review of the Differing Professional Opinion (DPO) submittal and associated references, and interview and follow-up discussion with the submitters, we determined the following issue was expressed:

Some NRC staff are concerned that the IOLB policy determination conflicts with the purpose of Tier 1 test items to test emergency and abnormal operating procedural knowledge on the site-specific Reactor Operator (RO) written examination, which is required in accordance with 10 CFR 55.41 (b)(10). The staff members contend that the number of RO questions that test abnormal and emergency procedures on the site-specific written exam should not be reduced because the operating exam does not test individual applicants procedure knowledge to the same extent as the written exam because:

Dynamic scenarios are administered in a team, open-book environment where the SRO reads or directs emergency/abnormal operating procedure steps to RO applicants, and systems Job Performance Measures (JPMs) are administered by directing the applicant to perform a task in accordance with a specific procedure.

These staff members are concerned that the IOLB policy determination precludes the Chief Examiner from evaluating a Tier 1 test item as enhancement required on Form ES-401-9, when the proposed test item does not test abnormal or emergency procedure knowledge relevant to the K/A statement wording, which would ensure overall exam balance of coverage for abnormal and emergency operating procedures.

Panel Review Summary and Recommendation In response to DPO Case Number DPO-2017-007, its associated summary of concern (SOI),

and after careful consideration of input obtained through independent research, interviews with the DPO submitters, and interviews with personnel from the IOLB, the DPO Panel offers the following recommendations:

2

1. A Chief Examiner (CE) should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e. Enhancements may be assessed per Form ES-401-9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak. Amplifying information is discussed below for how procedural relevance can be determined. If the only flaw for a test item being assessed is its link to a relevant procedure, then the question should not be assessed as unsatisfactory.

2. If recommendation #1 is implemented, IOLB should determine how to promulgate this policy change regarding the assessment of Tier 1 questions to both agency and industry stakeholders. Additionally, IOLB should consider coordinating with the staff at the Technical Training Center (TTC) to include appropriate modifications to the examiner training course (G-107).

3. If recommendation #1 is implemented, IOLB should also assess the necessity of changes to NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The DPO Panel does not recommend any substantive changes to the current revision of NUREG-1021, as the current guidance supports recommendation #1.

However, the next revision of NUREG-1021 may include clarifying guidance or examples providing the basis for the assessment of Tier 1 written exam questions (similar to what was included in previous revisions regarding SRO only written exam questions).

Supporting Information The information below provides the basis from which the DPO Panel arrived at the recommendation outlined above. Based on the summary of concern outlined in the DPO, the DPO panel analyzed the applicable sections of the current revision of NUREG-1021 related to written examination construction and evaluation. As a result of this analysis, the DPO Panel noted the following:

1. Per ES-401 of NUREG-1021, Revision 11 (underlined text indicates emphasis added):

The content of the written licensing examinations for ROs and SROs is dictated by 10 CFR 55.41, Written Examination: Operators, and 10 CFR 55.43, Written Examination:

Senior Operators, respectively. Each examination shall contain a representative selection of questions concerning the knowledge and abilities (K/As) and skills needed to perform duties at the desired license level. Both the RO and SRO examinations will sample the 14 items specified in 10 CFR 55.41(b), and the SRO examination will also sample the 7 additional items specified in 10 CFR 55.43(b).

Except as noted in Section D.1.b of this examination standard, NUREG-1122, Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors, and

3 NUREG-1123, Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors, available in the Agencywide Documents Access and Management System (ADAMS), provide the basis for developing content-valid operator licensing examinations.

Each K/A stem statement has been linked to an applicable item number in 10 CFR 55.41 and/or 10 CFR 55.43. Preparing the license examination using the appropriate K/A catalog, in conjunction with the instructions in this NUREG-series report, will ensure that the Examination includes a representative sample of the items specified in the regulations.

==

Conclusion:==

Following the sample plan methodology as described in NUREG-1021 (and the applicable K/A catalog) ensures that the NRC written exam meets the content requirements as outlined in the items from 10CFR55.41 and 10CFR55.43.

2. From 10 CFR 55.41:

(b) The written examination for an operator for a facility will include a representative sample from among the following 14 items, to the extent applicable to the facility.

(1) Fundamentals of reactor theory, including fission process, neutron multiplication, source effects, control rod effects, criticality indications, reactivity coefficients, and poison effects.

(2) General design features of the core, including core structure, fuel elements, control rods, core instrumentation, and coolant flow.

(3) Mechanical components and design features of the reactor primary system.

(4) Secondary coolant and auxiliary systems that affect the facility.

(5) Facility operating characteristics during steady state and transient conditions, including coolant chemistry, causes and effects of temperature, pressure and reactivity changes, effects of load changes, and operating limitations and reasons for these operating characteristics.

(6) Design, components, and functions of reactivity control mechanisms and instrumentation.

(7) Design, components, and functions of control and safety systems, including instrumentation, signals, interlocks, failure modes, and automatic and manual features.

(8) Components, capacity, and functions of emergency systems.

(9) Shielding, isolation, and containment design features, including access limitations.

(10) Administrative, normal, abnormal, and emergency operating procedures for the facility.

(11) Purpose and operation of radiation monitoring systems, including alarms and survey equipment.

(12) Radiological safety principles and procedures.

(13) Procedures and equipment available for handling and disposal of radioactive materials and effluents.

(14) Principles of heat transfer thermodynamics and fluid mechanics.

Note: Items 1 and 14 from the above list are not part of the site-specific NRC licensing examination as they are covered on the Generic Fundamentals Exam.

3. From 10 CFR 55.43:

(b) The written examination for a senior operator for a facility will include a representative sample from among the following seven items and the 14 items specified in § 55.41 of this part, to the extent applicable to the facility:

(1) Conditions and limitations in the facility license.

4 (2) Facility operating limitations in the technical specifications and their bases.

(3) Facility licensee procedures required to obtain authority for design and operating changes in the facility.

(4) Radiation hazards that may arise during normal and abnormal situations, including maintenance activities and various contamination conditions.

(5) Assessment of facility conditions and selection of appropriate procedures during normal, abnormal, and emergency situations.

(6) Procedures and limitations involved in initial core loading, alterations in core configuration, control rod programming, and determination of various internal and external effects on core reactivity.

(7) Fuel handling facilities and procedures.

4. Sample plan methodology / Knowledge and Ability (K/A) Stem Links:

The site-specific written exam sample plan (or exam outline) is divided into 3 distinct Tiers:

Tier 1 Emergency and Abnormal Plant Evolutions Tier 2 Plant Systems Tier 3 Generic K/A Categories As can be observed from Table 1, a properly constructed exam outline (per the requirements of ES-401 and Form ES-401-1) is designed to include K/A stem statements that are linked to all of the items (55.41(b)(2) through 55.41(b)(13)) required by regulation for a Reactor Operator written examination AND all of the items (55.43(b)(1) through 55.43(b)(7)) for a Senior Reactor Operator written examination.

Following the above (which was taken directly from NUREG-1021; ES-401), the K/A catalogs provide the basis for developing content-valid operator licensing examinations AND each K/A stem statement has been linked to an applicable item number in 10 CFR 55.41 and/or 10 CFR 55.43. With that in mind, consider the following 10 CFR 55.41 and 10 CFR 55.43 links for the K/A statements associated with Tier 1 (Emergency and Abnormal Plant Evolutions) questions on the exam outline (Table 1) {Bold Italics added for emphasis}:

Reactor Operator (5) Facility operating characteristics during steady state and transient conditions, including coolant chemistry, causes and effects of temperature, pressure and reactivity changes, effects of load changes, and operating limitations and reasons for these operating characteristics.

(6) Design, components, and functions of reactivity control mechanisms and instrumentation.

(7) Design, components, and functions of control and safety systems, including instrumentation, signals, interlocks, failure modes, and automatic and manual features.

(8) Components, capacity, and functions of emergency systems.

(9) Shielding, isolation, and containment design features, including access limitations.

(10) Administrative, normal, abnormal, and emergency operating procedures for the facility.

Senior Reactor Operator (1) Conditions and limitations in the facility license.

5 (2) Facility operating limitations in the technical specifications and their bases.

(3) Facility licensee procedures required to obtain authority for design and operating changes in the facility.

(5) Assessment of facility conditions and selection of appropriate procedures during normal, abnormal, and emergency situations.

From the two lists above, only item (10) for the Reactor Operator, and items (3) and (5) for the Senior Reactor Operator specifically mention procedures. The blue-shaded blocks on Table 1 indicate the K/A stem statements associated with these items for the Tier 1 section of the exam outline. As highlighted on Table 1, the K/A categories with 10 CFR links to items that refer to procedures are K1, A2, and G.

Conversely, the 10 CFR links associated with Tier 2 questions that refer to procedures (blocks highlighted in green on Table 1) only include the G K/A category. An exception to this observation is the A2 category, which consists of K/A stem statements that specifically call out procedure selection as part of the required knowledge and ability of the applicant (see example below):

Ability to (a) predict the impacts of the following on the RHR/LPCI: INJECTION MODE; and (b) based on those predictions, use procedures to correct, control, or mitigate the consequences of those abnormal conditions or operations:

Based on these observations, The DPO Panel believes that Tier 1 questions in the K1, A2, and G categories should have some procedural aspect associated with them in order to appropriately test the 10 CFR item(s) they are linked to, and be in conformance with the existing regulation. While not explicit (i.e. shall), disregarding the 10 CFR links listed in the K/A stem statements risks not testing in a manner to ensure all of the CFR 55.41 and 10 CFR 55.43 items are sampled.

5. From NUREG-1123 (underlined text indicates emphasis added):

The K/As were linked to their applicable 10CFR55 item numbers. SRO level K/As were identified by 10CFR55.43 item numbers. (Taken from NUREG-1123 abstract)

The linkage of K/As to the 10 CFR 55.41, 43, and 45 requirements was done to help ensure that the examinations include a representative sample from among the applicable items (Taken from NUREG-1123, Rev. 2, Supp.1 - Summary of Significant Changes)

All knowledge and abilities (K/As) in this catalog are directly linked by item number to 10 CFR 55.

These statements agree with ES-401, which states that, Each K/A stem statement has been linked to an applicable item number in 10 CFR 55.41 and/or 10 CFR 55.43.

Also, from NUREG-1123 related to the Knowledge and Ability Stem Statements for Emergency and Abnormal Plant Evolutions:

Table 4 Knowledge and Ability Stem Statements for Emergency and Abnormal Plant Evolutions

6 E/AK1 Knowledge of the operational implications of the following concepts as they apply to the (EMERGENCY OR ABNORMAL PLANT EVOLUTION):

(CFR: 41.8 to 41.10)

E/AK2 Knowledge of the interrelations between (EMERGENCY OR ABNORMAL PLANT EVOLUTION) and the following:

(CFR: 41.7 / 45.8)

E/AK3 Knowledge of the reasons for the following responses as they apply to (EMERGENCY OR ABNORMAL PLANT EVOLUTION):

(CFR: 41.5 / 45.6)

E/AA1 Ability to operate and / or monitor the following as they apply to (EMERGENCY AND ABNORMAL PLANT EVOLUTION):

(CFR: 41.7 / 45.6)

E/AA2 Ability to determine and interpret the following as they apply to (EMERGENCY AND ABNORMAL PLANT EVOLUTION):

(CFR: 41.10 / 43.5 / 45.13)

Note: Similar statements are found related to the structural layout of the exam outline and associated K/A stem statement links to 10 CFR 55 for PWR written examinations in NUREG-1122, Revision 2, Supplement 1, and the Advanced Reactor designs (AP-1000 and ABWR) in NUREG-2103 and 2104. Also refer to Tables 2, 3, and 4 (highlighted in the same manner as described above for Table 1) for a comparison of the related K/A stem statement links to 10 CFR 55 for each of the K/A categories that comprise the exam outlines for the additional reactor designs.

6. From NUREG-1021, Appendix B (underlined text indicates emphasis added):

Failing to focus on testing the individual operators cognitive abilities (i.e., comprehension, problem-solving, and decision-making) or paying insufficient attention to the operators fundamental understanding of job content (e.g., systems, components, and procedures) may ultimately place job performance at risk of gradual degradation.

==

Conclusion:==

The random and systematic sampling process used when generating the exam outline in accordance with ES-401 of NUREG-1021 ensures that the written examination is content valid. The job content (for the written examination) that is being tested per existing regulation are those items identified in 10 CFR 55.41 and 55.43. The K/A catalogs link to the knowledges and abilities prescribed in 10 CFR 55 in a specific manner (refer to Tables 1 and 2.)

Failing to ensure that written exam questions developed for NRC examination test applicants to meet the intent of the 10 CFR 55 links, creates a risk that all of the 10 CFR 55 items may not be sampled appropriately, and therefore the validity inference that our process was designed to have may become skewed. For example, for those Tier 1 K/A categories previously mentioned, if there was not a procedural aspect for questions in categories K1, K2 (PWR), A2, and G, then the exam may overemphasize plant systems and under-emphasize plant procedures. The chief examiners quality assurance checks, per Form ES-201-2 items 4.b and 4.e, specifically direct verifying that the 10 CFR 55.41, 43, and 45 sampling is appropriate and exam coverage is balanced (see the form on the next page.)

7

7. From NUREG-1021, ES-201 (underlined text indicates emphasis added):

There are no minimum or maximum limits on the number or scope of changes the NRC may direct the facility licensee to make to its proposed examinations, provided that they are necessary to make the examinations conform with established acceptance criteria or to attain an appropriate level of examination difficulty.

==

Conclusion:==

The Chief Examiner is responsible for ensuring that the 10CFR55.41, 55.43, and 55.45 sampling is appropriate and exam coverage is balanced for the entire exam. Thus, if the Chief Examiner believes that the licensees proposed exam under-emphasizes procedural knowledge, it is not only prudent, but required by the QA checklist in ES-201 for the Chief Examiner to ensure that this issue is corrected prior to approving the exam for administration.

This may be performed by either providing unsatisfactory or enhancement required comments to the licensee via Form ES-401-9.

Final Conclusion From the information outlined above, the DPO panel recommends that IOLB partially implement the proposed alternative outlined in the DPO submittal, by implementing the above recommendation. Due to the current guidance in the K/A catalogs and NUREG-1021, the panel also feels that a Chief Examiner would be justified in asking a facility licensee to enhance Tier 1 written examination questions to test a procedural concept in K/A categories K1, K3 (PWR only), A2, and G as relevant to the respective K/A statement, or any other situation where the exam coverage becomes skewed. This approach helps ensure that NRC site-specific written examinations reflect the representative sample of 10CFR55 items required by regulation, exams will not over-emphasize plant systems knowledge, or under-emphasize plant procedural knowledge. Test items that appropriately solicit knowledge of plant system operation/response, or design associated with an emergency or abnormal event in categories K2, K3 (BWR only),

and A1, should be deemed satisfactory if their K/A statement was met with no other psychometric flaws.

8

9

10

11

12

ML18078A009 OFFICE RIV/DRS ADHRTD/TTC RIII/DRS NAME JClark MEmerich CZoia SIGNATURE

/RA/

/RA-E/

/RA-E/

DATE 3/19/18 3/14/18 3/14/18

Document 4: DPO Decision

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 April 27, 2018 MEMORANDUM TO:

Bruno L. Caballero, Senior Operations Engineer Operations Branch 2 Division of Reactor Safety Region II Daniel M. Bacon, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II David R. Lanyi, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Phillip G. Capehart, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II Michael K. Meeks, Senior Operations Engineer Operations Branch 1 Division of Reactor Safety Region II FROM:

Brian E. Holian, Acting Director

/RA/

Office of Nuclear Reactor Regulation

SUBJECT:

DIFFERING PROFESSIONAL OPINION INVOLVING OPERATOR LICENSING WRITTEN EXAMINATIONS - TIER 1 ITEMS (DPO 2017-007)

On October 10, 2017, in accordance with Management Directive 10.159, The NRC Differing Professional Opinions Program, you submitted a differing professional opinion (DPO) involving operator licensing written examinations (DPO-2017-007). Specifically, your DPO raises concerns that the recent policy determination made by NRR staff for writing and assessing Tier 1 written examination test items has the potential to undermine the 10 CFR 55.41 that the CONTACT: Trent L. Wertz, NRR 301-415-1568

B. Caballero, et al.

written examination should contain a representative selection of questions on the knowledge, skills, and abilities needed to perform licensed operator duties. Specifically, the policy interpretation will result in fewer questions that test the operators knowledge of abnormal and emergency procedures. The purpose of this memorandum is to respond to your DPO.

On January 21, 2017, a DPO Ad Hoc Review Panel (the Panel) was established and tasked to meet with you, review your DPO submittal, and issue a DPO report, including conclusions and recommendations to me regarding the disposition of the issues presented in your DPO. On March 19, 2018, after reviewing the applicable documents, completing internal interviews of relevant individuals and completing their deliberations, the Panel issued their report to me.

On April 19, 2018, I talked to you by telephone to discuss the Panels report and to get your insights and comments. On April 25, 2018, you provided me additional insights into your concerns and your thoughts for resolving the issue.

In order to make a decision with regard to your DPO, I reviewed your DPO submittal, the Panels report, met with the headquarters operator licensing staff, talked with you, and then re-considered your comments to me. In addition, on April 26, 2018, I discussed these issues with the DPO Panel Chair Statement of Concern The Operator Licensing and Training Branch (IOLB) of the Office of Nuclear Reactor Regulation (NRR) determined that a test item developed for Tier 1 of the site-specific written exam matches the intent of its knowledge or ability (K/A) statement if the test item solely tests plant systems knowledge, such as a design feature, interlock, or automatic operation. IOLB determined it was inappropriate to evaluate Tier 1 test items as enhancement required or inappropriate on Form ES-401-9, Written Examination Review Worksheet, when the test item did not test knowledge of emergency or abnormal procedures. This determination was documented in Record of Interaction (ROI) 17-09, NUREG 1021, ES-401 Tier 1 Written Exam Test Items (ML17165A579); was disseminated to industry stakeholders in Operator Licensing Feedback Item 401.55 (ML17249A961); and was communicated during an operator licensing examiner training session conducted by IOLB staff on October 19, 2017.

Based on the Panels review of the Differing Professional Opinion (DPO) submittal and associated references, and interview and follow-up discussion with the submitters, the panel determined the following issue was expressed:

Some NRC staff are concerned that the IOLB policy determination conflicts with the purpose of Tier 1 test items to test emergency and abnormal operating procedural knowledge on the site-specific Reactor Operator (RO) written examination, which is required in accordance with 10 CFR 55.41 (b)(10). The staff members contend that the number of RO questions that test abnormal and emergency procedures on the site-specific written exam should not be reduced because the operating exam does not test individual applicants procedure knowledge to the same extent as the written exam because:

Dynamic scenarios are administered in a team, open-book environment where the SRO reads or directs emergency/abnormal operating procedure steps to RO applicants, and systems Job Performance Measures (JPMs) are administered by directing the applicant to perform a task in accordance with a specific procedure.

B. Caballero, et al.

These staff members are concerned that the IOLB policy determination precludes the Chief Examiner (CE) from evaluating a Tier 1 test item as enhancement required on Form ES-401-9 when the proposed test item does not test abnormal or emergency procedure knowledge relevant to the K/A statement wording, which would ensure overall exam balance of coverage for abnormal and emergency operating procedures.

Panel Recommendations The Panel concluded and recommended the following:

1. A CE should not be prohibited from assessing written examination questions as enhancement required for any reasonable situation, since it is the Chief Examiners responsibility to ensure balance of coverage throughout the entire exam, as stated on Form ES-201-2, Item 4.e. Enhancements may be assessed per Form ES-401-9 of NUREG-1021, in the opinion of the CE, if the submitted question does not test a relevant concept of the applicable abnormal or emergency operating procedure, or if the questions link to the specified knowledge and ability (K/A) statement of the test item is weak.

Amplifying information is discussed below for how procedural relevance can be determined. If the only flaw for a test item being assessed is its link to a relevant procedure, then the question should not be assessed as unsatisfactory.

2. If recommendation #1 is implemented, IOLB should determine how to promulgate this policy change regarding the assessment of Tier 1 questions to both agency and industry stakeholders. Additionally, IOLB should consider coordinating with the staff at the Technical Training Center (TTC) to include appropriate modifications to the examiner training course (G-107).

3. If recommendation #1 is implemented, IOLB should also assess the necessity of changes to NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The DPO Panel does not recommend any substantive changes to the current revision of NUREG-1021, as the current guidance supports recommendation #1. However, the next revision of NUREG-1021 may include clarifying guidance or examples providing the basis for the assessment of Tier 1 written exam questions (similar to what was included in previous revisions regarding SRO only written exam questions).

After considering all the information, I essentially agree with the recommendations provided by the DPO panel. They have thoroughly and conscientiously endeavored to address your well-thought out and articulated concerns. I have the following comments/clarifications to the recommendations.

Regarding Recommendations 1 and 3, I directed my staff to revise the Operator Licensing Program Feedback response (Question 401.55) to clarify that CEs are allowed to make reasonable changes for balance of coverage throughout the examination. I agree that Tier 1 test items should, when relevant, test abnormal/emergency procedure knowledge. However, when there is a proper balance in the overall exam, not having a Tier 1 question tie to a procedure is not basis to remove it from the test. I considered the amplifying information referenced in Recommendation 1 and find there is no need to revise the current guidance that written examination questions should, but are not required to, test one of the 10 CFR 55 written examination items that the K/A is linked to, or to a facility learning objective. This is, in part,

B. Caballero, et al.

because the linkage of K/A items to 10 CFR 55 written examination items does not represent an exhaustive list. I have assigned the above task to DIRS, NRR, to be completed by June 30, 2018.

Regarding Recommendation 2, the acknowledgement that CEs may request enhancements to written examination items, including the failure to test a relevant procedural concept will be promulgated through a revision to ROI 17-09 and an update to Question 401.55 on the Operator Licensing Program Feedback webpage. This action is also assigned to DIRS, NRR, to be completed by June 30, 2018.

A summary of the DPO will be included in the Weekly Information Report (when the case is closed) to advise interested employees of the outcome.

Thank you for raising your DPO and for your active participation in the DPO process. An open and thorough exploration of how we carry out our regulatory processes is essential to keeping these programs effective. Your willingness to raise concerns with your colleagues and managers and ensure that your concerns are heard and understood is admirable and vital to ensuring a healthy safety culture within the Agency.

Enclosure:

DPO Panel report, dated March 19, 2018 cc: R. Lorson, NRR M. Evans, NRR A. Boland, OE G. Figueroa-Toledo, OE C. Haney, RII C. Miller, NRR M. Johnson, OEDO

Pkg: ML18117A132; DPO: ML18117A079;

Enclosure:

ML18079A001 OFFICE NRR NAME BHolian DATE 4/27/18