ML20083F400
| ML20083F400 | |
| Person / Time | |
|---|---|
| Site: | Nuclear Energy Institute |
| Issue date: | 03/04/2020 |
| From: | Riti T Nuclear Energy Institute |
| To: | Cowdrey C NRC/NRR/DRO/IOLB |
| Cowdrey C, NRR/DRO, 415-3085 | |
| References | |
| Download: ML20083F400 (7) | |
Text
TIMOTHY RITI Senior Project Manager, Regulatory Affairs 1201 F Street, NW, Suite 1100 Washington, DC 20004 P: 202.739.8137 txr@nei.org nei.org March 4, 2020 Mr. Christian Cowdrey Chief, Operator Licensing & Human Factors Branch Division of Reactor Oversight Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, DC 20555-0001
Subject:
Generic Fundamentals Reintegration Project Number: 689
Dear Mr. Cowdrey:
The Nuclear Energy Institute (NEI), 1 on behalf of its industry members, appreciates the opportunity to submit for your consideration and use, results of an industry led pilot conducted to evaluate a NUREG-1021, Operator Licensing Examination Standards for Power Reactors draft proposal to re-incorporate generic fundamentals (GF) questions into the Reactor Operator (RO) section of the final NRC license written examination. NEIs Licensed Operator Focus Group (LOFG) sponsored this initiative and wants to recognize the support received by the industry and NRC. Four nuclear stations - APS-Palo Verde, Duke-McGuire, Entergy-Grand Gulf and Exelon-Limerick volunteered to assist in this effort. Your staff also assisted by providing eight randomly sampled Knowledge and Abilities (K/As) for the industrys use in question development. The industry intends to use the pilot information to aid in the development of change management if the NRC moves forward with the proposed change. The methods used and results of the pilot are summarized below.
Background
The pilot required exam authors at the four stations to develop operationally focused plant-specific GF questions using draft NRC guidance. The questions were administered to Initial License Training (ILT) class students around the time of their written audit/certification exam. Student performance on the pilot 1
The Nuclear Energy Institute (NEI) is responsible for establishing unified policy on behalf of its members relating to matters affecting the nuclear energy industry, including the regulatory aspects of generic operational and technical issues. NEIs members include entities licensed to operate commercial nuclear power plants in the United States, nuclear plant designers, major architect and engineering firms, fuel cycle facilities, nuclear materials licensees, and other organizations involved in the nuclear energy industry.
Mr. Christian Cowdrey March 4, 2020 Page 2 program questions was used for information only; it did not impact their actual exam scores and had no impact on the decision-making process for taking the NRC exam. Their performance on the pilot questions was used to determine a composite written audit exam score by substituting the scores of the pilot questions for the actual exam question scores, based on the draft sample plan methodology.
Process At our request, the NRC (Operator Licensing Branch, DRO/NRR) provided eight randomly sampled K/As for development of the GF questions. Of the eight K/As, six were associated with a proposed new Tier 4 for generating written exam sample plans, with three each from the Reactor Theory and Thermodynamics sections of the PWR and BWR K/A Catalogs (NUREG-1122, -1123). In the proposed sample plan methodology, these six K/As would replace 6 K/As in the existing Tier 3, Generic Knowledge and Abilities, portion of the sample plan.
The remaining two K/As of the eight provided by the NRC were from the Components section of the PWR and BWR K/A Catalogs. They were designated as replacements for generic K/As in the existing Tier 2, Plant Systems, portion of the sample plan; one each from Group 1 and Group 2.
NOTE: One station selected their own K/As and only selected six (3 reactor theory and 3 thermodynamics).
They did not sample component K/As and therefore did not develop any component-based questions. This stations results are still considered valid in the analysis since it is possible for the proposed K/A sampling methodology to not select any component K/As.
Exam authors at all four stations developed eight (or six) operationally focused plant-specific questions based on the NRC provided (or station selected) K/As. The NRCs GF exam bank was used as a starting point for many of the questions. Since the NRC requires questions to match the K/A statement for initial license exams, that same criterion was followed for the eight (or six) GF questions. However, the ability to develop operationally focused GF questions is largely dependent on the specific wording of the K/A statement.
Section ES-401 of NUREG-1021 provides guidance for rejecting and resampling K/As if certain criteria are met. Since this guidance does not specifically apply to the GF exam/questions, and since the GF pilot program guidance did not have any provision for rejecting and resampling K/As, station exam authors developed questions that matched the K/A statement and did the best they could in terms of operational focus. One of the key learnings from this pilot is the need for clear NRC guidance for rejecting GF K/As when the ability to create operationally focused questions is in doubt (see rec. 2 in Appendix 1).
Following question development, the normal validation process at each station was followed. Three of four stations also submitted questions for review by NRR to simulate the process of gaining NRC approval for the associated questions.
Mr. Christian Cowdrey March 4, 2020 Page 3 Results Specific station results, in no particular order are shown on Attachment 1. Note that these results are without any ongoing GF-focused training during the programs and with little or no advance warning to the ILT students that they would be tested on these pilot GF questions. Actual results of reintegration following facility program and student adjustments would likely yield improved scores.
Overall, the results indicate that reintegrating GF questions/topics into the final NRC written exam is viable.
With proper industry and NRC change management, such as ongoing reinforcement and testing of GF concepts during initial license training programs, the LOFG believes there will be no significant change in initial license applicant pass rates.
Recommendations for NRC consideration are summarized in Appendix 1. Student feedback is summarized in Appendix 2 and will be used to inform change management at the stations.
In conclusion, the LOFG supports the NRC moving forward with reintegrating GF topics into the RO section of the final written licensing examination. In addition, the LOFG requests the NRC develop clear guidance to address the recommendations described in Appendix 1.
If there are any questions on this matter, please contact me at (202) 739-8137 or txr@nei.org.
Sincerely, Timothy Riti Enclosure c: Mr. Brian Tindell, NRR/DRO/IOLB, NRC Mr. Kostas Dovas, LOFG Co-Chair
Mr. Christian Cowdrey March 4, 2020 Page 4 Appendix 1: Recommendations
- 1. There is a wide spectrum of what is operationally focused/relevant, especially within the context of GF topics and associated K/A statements. Since current GFE bank questions do not offer guidelines, criteria or examples of what will be acceptable for the new style of questions, additional guidance is needed for the industry to avoid submitting unsatisfactory questions. The LOFG is willing to work with the NRC to develop example questions for inclusion in the NUREG or other similar guideline if the NRC choses this option.
- 2. K/A matching criteria for GFE questions has not always been the same as that required for licensing exam questions. Existing ES-401 guidance should be updated to specifically address K/A matching vs.
rejecting and resampling of GF K/As when the ability to develop operationally focused questions is in doubt.
- 3. In addition to items 1 and 2 above, measures to ensure regional alignment and consistency for what is acceptable for GF questions is necessary to ensure consistent implementation throughout the industry.
- 4. Develop guidelines and/or criteria for when using bank GFE questions as is is appropriate.
- 5. Provide criteria for what technical references will be required or acceptable for GF questions.
- 6. Provide guidance on how SRO upgrade applicants will be affected by this change, if at all.
- 7. Develop implementation guidance to clarify how license applicants that have already passed the GF exam will be affected by reintegration (e.g., do not sample GF topics on their final license exam).
Mr. Christian Cowdrey March 4, 2020 Page 5 Appendix 2: Student Feedback Summary
- Need a refresher on formulas/calculations prior to exam
- Questions were fair and most were tied to plant operations
- Questions are an improvement over existing Tier 3 generic questions
- Felt unprepared since GF questions are not normally given on audit exam
- The volume of testable GF material would significantly impact study plans
- Some K/As require specific knowledge of fundamental concepts and could have low operational validity
- The additional time needed to study GF material will impact time available to study the other tested material
- There are concerns about the amount of additional material that will be tested (must be known) adding GF knowledge with no reduction in admin topic knowledge
Mr. Christian Cowdrey March 4, 2020 Page 6 Attachment 1: Pilot Results (page 1 of 2)
Station 1:
Audit Exam Average w/o Question Individual Question Score GF GF-1 88 GF-2 88 83.1 GF-3 92 GF-4 88 GF-5 73 Audit Exam Average w/ GF GF-6 85 GF-7 N/A 83.7 GF-8 N/A ( 0.6)
Average: 85.7 Station 2:
Audit Exam Average w/o Question Individual Question Score GF GF-1 67 GF-2 75 92.2 GF-3 92 GF-4 92 GF-5 75 Audit Exam Average w/ GF GF-6 87 GF-7 83 91.0 GF-8 83 ( 1.2)
Average: 81.8
Mr. Christian Cowdrey March 4, 2020 Page 7 Attachment 1: Pilot Results (page 2 of 2)
Station 3:
Audit Exam Average w/o Question Individual Question Score GF GF-1 67 GF-2 78 85.4 GF-3 100 GF-4 100 GF-5 67 Audit Exam Average w/ GF GF-6 78 GF-7 67 85.0 GF-8 100 ( 0.4)
Average: 82.1 Station 4:
Audit Exam Average w/o Question Individual Question Score GF GF-1 33 GF-2 44 93.0 GF-3 22 GF-4 44 GF-5 89 Audit Exam Average w/ GF GF-6 56 GF-7 89 89.2 GF-8 89 ( 3.8)
Average: 58.3