NUREG-1021, Revised Effectiveness Review Plan for NUREG-1021 Revision 12 (Public)
| ML23236A119 | |
| Person / Time | |
|---|---|
| Issue date: | 08/24/2023 |
| From: | Theresa Buchanan NRC/NRR/DRO/IOLB |
| To: | |
| References | |
| Download: ML23236A119 (15) | |
Text
Revised Effectiveness Review Plan for NUREG-1021, Revision 12, Operator Licensing Examination Standards for Power Reactors
Background
In September 2021, the Operator Licensing (OL) program office (NRR/DRO/IOLB) issued a major revision to NUREG-1021, Operator Licensing Examination Standards for Power Reactors.
Revision 12 was issued for the following reasons:
- 1. Re-incorporate generic fundamentals questions into the reactor operator section of the site-specific written examination and discontinue the standalone NRC Generic Fundamentals Examination (GFE).
- 2. Incorporate changes to the critical task (CT) methodology and the grading of critical performance deficiencies (CPDs) on the simulator operating test and introduce a new category for significant performance deficiencies (SPDs).
- 3. Acknowledge use of the National Academy for Nuclear Trainings ACAD-10-001, Guidelines for Initial Training and Qualification of Licensed Operators, Revision 2, as the standard set of eligibility requirements for accredited training programs and discontinue the process for NRC waivers of facility licensees training program eligibility requirements.
ACAD-10-001, Revision 2, modified experience requirements for eligibility for initial license training programs. Specifically, a facility licensee board will determine if senior reactor operator (SRO)-instant applicants with related sciences degrees have adequate math/science coursework to enter initial license training, and the number of months of previous power plant experience for reactor operator applicants has decreased across the categories of experience (i.e., for both nuclear and non-nuclear plant experience)
- 4. Structure the NUREG into topic-based sections for ease-of-use and improve existing guidance by incorporating recommendations from lessons-learned, clarifications from interactions with NRC region offices, and feedback received from industry.
Purpose The purpose of this effectiveness review is to monitor the application of new instructions and guidance in NUREG-1021, Revision 12, and determine if additional actions are needed, such as additional training or clarifications to support implementation of the major changes made in the revision. This review will also include an assessment of the implementation and outcomes of each major change. The review should address the following questions for the changes introduced in Revision 12: Once implemented, did the changes occur the way that the OL program office envisioned, what was the impact of the changes, and were the actual results and outcomes worth the resources invested?
2
Duration Most of the review will start just after the first Revision 12 examination is administered in April 2022 and continue for at least 2 years (i.e., at least through April 2024), while the GFE related review will continue through 2027. The OL program office may extend the duration of the review to collect additional data, if necessary, to better understand a potential issue. The OL program office will develop a report to capture the findings from this review.
3
Tasks Task 1: Review the integration of generic fundamentals topics into the site-specific examination and verify that the instructions in NUREG-1021 are being used as intended; identify any areas for improving clarity in the instructions/guidance related to these "new" questions.
Subtask Sample Duration Deliverables/Notes Check for consistent and adequate sampling of generic fundamentals topics on written examination outlines and a sample of examinations by doing the following:
Deliverable: exam review memos IAW OLMC-320.
- Verify that written examination All GF questions on outline reflects adequate the exams (i.e., 8-10 Issue a memo from IOLB Branch Chief sampling of generic written examinations to examination auditors with instructions 1a fundamentals topics on the RO CY 2022 and 2023 per year) selected to support this effectiveness review.
portion in Tier 2 and Tier 4 and for program office complies with the instructions in annual audit. Statistics and observations for review ES-4.1 B steps 5a, 5f and 5h.
reports.
- Record the total number of generic fundamentals questions, including the number of K/As from Section 5 and Section 6 of the applicable K/A catalog for each examination.
4
Check the quality of all generic fundamental questions against the All GF questions on Deliverable: exam review memos IAW criteria on Form 2.3-5 Written the exams (i.e., 8-10 OLMC-320.
Examination Review Worksheet written examinations 1b and observe how chief examiners CY 2022 and 2023 per year) selected Draft memo from IOLB Branch Chief to are reviewing proposed questions for program office examination auditors with instructions to against the criteria in NUREG-1021 annual audit. support this effectiveness review as documented on the written examination quality checklists.
Track how many appeals are associated with generic fundamentals topic questions (RO Section, Tier 2/4) including the Statistics and observations for review All written 1c number of licensing decisions May 2022- May 2024 examination appeals reports.
upheld and overturned. If overturned, determine reason for overturning (e.g., due to psychometric flaws).
5
Task 2: Verify that generic fundamental knowledge is being maintained through initial and continuing training programs.
Subtask Sample Duration Deliverables/Notes Track performance on Tier 2 and Tier 4 generic fundamentals questions and compare to a baseline on GFE performance over 2018 and 2019 (average score).
For each written exam, tabulate April 2022- September individual applicant scores on GF 2027*:
questions. Revisit the sample plan distribution for generic Pre-evaluation: April fundamentals in NUREG-1021, 2022-September 2023 Rev. 12 if any of the following (most applicants during performance thresholds are this period will have crossed at the end of a 2-year previously taken a GFE) evaluation period on an industry level: Record # of GF questions and individual Evaluation period 1:
- 1. >10% of applicants score applicant scores on "GF questions" per All written October 2023-September 2a less than 70% on GF exam. (e.g., RO1 6/6, RO2 4/6, SROI1 examinations 2025 questions (Tier 4 and Tier 2 3/6, etc.) Consider using PowerBI/Power Section 5 G K/As) automate for data analysis.
Evaluation period 2:
- 2. >5% of applicants score October 2025-50% or less on GF September 2027 questions o If 5% is exceeded,
- End date may be 50% still passed extended as necessary the RO portion of the to ensure accurate exam (i.e., 80%)
trending with sufficient
- 3. >10% of facilities have an justification and approval average score of less than 70% on GF questions on an examination OR have an identified weakness in generic fundamentals training 6
o Screening criteria:
A weakness in the GF portion of the training program identified by the facility or INPO, high miss GF question that the NRC determined was related to a training issue, an operational event that was determined to have GF knowledge as either a contributing or root cause, or as determined by a majority of chief examiners or the IOLB program office as documented in an inspection report.
May 2022-May 2024*
- because initial results Document OpE with training-related Monitor operating experience NRC OpE may vary and 2022/2023 issues, include date and event for 2b (OpE) trends communications applicants may have additional analysis (NRR/DRO/IOEB already taken the GFE in staff are a resource).
2022, this subtask may extend past 2024 7
Obtain a copy of the facility examination Evaluate high miss questions to performance analysis (See NUREG-determine if the questions implicitly 1021 ES-4.4 Section B.7), if performed.
test* GF and if there are indications Identify any high miss questions (50%
that the high miss rate were due to miss rate) and evaluate those questions an issue in the training program to determine:
associated with GF
- 1) was the question psychometrically sound?
- In this context, an implicit GF
- 2) are there indications that there is an question is one that requires GF All written September 2023 -
2c issue with the facilitys training program knowledge to answer and, on the examinations September 2027 portion associated with teaching GFE basis of generic fundamental concepts that caused the high miss rate knowledge alone, at least two
- 3) determine if the question implicitly answer choices were able to be tests GF eliminated; two answer choices rendered significantly less This information will then be used as plausible; or one answer choice part of the threshold metric regarding was eliminated and one rendered issues with the facilities training program significantly less plausible.
(see Task 2.a.3).
8
Task 3: Determine if the instructions for identifying critical tasks are clear and verify proper and consistent application of the critical task methodology and grading of critical performance deficiencies; identify any areas for improving clarity in the instructions/guidance related to critical tasks and critical performance deficiencies.
Subtask Sample Duration Deliverables/Notes All CTS on the Deliverable: exam review memos IAW exams (i.e., 8-10 Check for consistent and adequate OLMC-320.
written examinations critical tasks by reviewing each per year) selected critical task against the critical task Draft memo from IOLB Branch Chief to for program office methodology in ES-3.3. If a critical examination auditors with instructions to 3a annual audit. If one CY 2022 and CY 2023 task does not meet the support this effectiveness review or more audits methodology, document specifics reveal problems with (provide task and what element(s) Statistics and observations for review critical tasks, then not met and why). reports.
consider expanding sample size.
Ask regions to report any CPDs at OL biweekly meetings. The OL program office will confer with the cognizant chief Verify that each CPD meets criteria All CPDs over a 12 examiner to get the information needed 3b April 2022-April 2023 for grading CPDs in ES-3.6 month period to analyze each CPD.
Statistics and observations for review reports.
Track how many appeals are associated with critical task and what aspects of critical tasks are All simulator Statistics and observations for review contested. Track the number of 3c operating test May 2022- May 2024 licensing decisions upheld and reports.
appeals overturned due to critical tasks. If overturned, determine reason for overturning.
9
- Track number of times that alternative boundary conditions All CTS on the are used exams (i.e., 8-10 written examinations Draft memo from IOLB Branch Chief to
- Analyze alternative boundary per year) selected examination auditors with instructions to conditions for arbitrary for program office support this effectiveness review.
3d conditions annual audit. If one CY 2022 and CY 2023 or more audits Statistics and observations for review
- Across all CTs that were reveal problems with reports.
audited, check alternative critical tasks then boundary conditions are consider expanding consistent/similar for the sample size.
same/similar CTs 10
Task 4: Verify that the critical task methodology is adequate for AP1000 simulator operating tests Subtask Sample Duration Deliverables/Notes Around the timelines Track ROIs or other issues for the first three Statistics and observations for review 4a reported to OL program office for Revision 12 AP1000 TBD reports.
CTs for AP1000. examinations/
Review the critical tasks on first three Rev 12 AP1000 examinations Coordinate with the chief examiner to as follows: review the operating test scenarios.
First three Revision
- Each CT meets critical task 12 AP1000 4b methodology in ES-3.3 TBD examinations Statistics and observations for review
- There is adequate variation in reports.
CTs from exam to exam
- Limits for scenario event repeatability in ES-3.4 are met 11
Task 5: Verify that the instructions for identifying and grading significant performance deficiencies (SPDs) are clear and verify proper and consistent application of SPD grading; identify any areas for improving clarity in the instructions/guidance related to significant performance deficiencies.
Subtask Sample Duration Deliverables/Notes Have regions report any SPDs at OL biweekly meetings. The OL program office will confer with the cognizant chief Verify that each SPD meets criteria examiner to get the information needed All SPDs over a 12 5a for identifying and grading SPDs in April 2022- April 2023 to analyze each SPD.
month period ES-3.5 and ES-3.6 Statistics and observations for review reports.
Track how many appeals are associated with SPDs and what aspects of SPDs are contested. All simulator Statistics and observations for review 5b Track the number of licensing operating test May 2022-May 2024 reports.
decisions upheld and overturned appeals due to SPDs. If overturned, determine reason for overturning.
12
Task 6: Determine the overall impact of the Revision 12 changes on the pass/fail rate for NRC initial licensing examinations.
Subtask Sample Duration Deliverables/Notes April 2022-March 2023, April 2023 - September 2023 See Duration input.
Analyze for any change to the pass Evaluation period 1: SRO/RO written examination pass-rates Represent and rates and average scores for October 2023-September and written examination average grades analyze important written examinations and operating 2025 and SRO/RO operating test pass rates statistical data for test under the Revision 12 data for last 10 years is available on the trends of SRO/RO 6a examination standards by Evaluation period 2: NRC public website for baseline written examination comparing written/operating test October 2025- comparison.
pass-rates and pass rates for two years to the pass September 2027 written examination rates under Revision 11 Statistics and observations for review average grades and examination standards. *End date may be reports.
SRO/RO operating extended as necessary test pass rates to ensure accurate trending with sufficient justification and approval 13
Task 7: Assess the use of new allowances in ACAD-10-001 for SRO-instant applicants to be eligible for licensing class by possessing a "related sciences degree." Assess any reductions to time spent on shift as under-instruction. Determine the impact, if any, of the changes made to eligibility requirements.
Subtask Sample Duration Deliverables/Notes NRC chief OL biweekly call and region OL chiefs examiners or region biweekly calls with IOLB. Add to Review observations from audits of chiefs share agenda. Can also track any related SRO-instant applications when this observations of use ROIs.
7a allowance is used; this includes June 2022 - June 2024 of the new process tracking any ROIs related to this for assessing Statistics and observations for review topic (e.g., ROI 2019-03).
"related sciences" reports.
degrees Chief examiners OL biweekly call and region OL chiefs collect information Started in mid-January biweekly calls with IOLB.
Record any reductions to time-7b from on-site 2023 (new item based on spent on shift validation week feedback) - June 2024 Statistics and observations for review application audits reports.
14
Task 8: Assess the ease-of use of NUREG-1021 Revision 12 as a result of changes made for streamlining and value Subtask Sample Duration Deliverables/Notes Trend total numbers of hours charged for examination development and Compare average hour spent on documentation (by CY 2022 and CY 2023 Statistics and observations for review examination development and region) in 2022 and 8a and first 6 months of CY reports.
documentation under Revision 12 2023 and first 6 2024 with averages under Revision 11. months of 2024 compare with a set of average hours charged in 2018 and 2019 (by region).
Assessment to include total # ROIs received, appeals, and hearings, and Statistics and observations for review compare to total reports.
8b Other assessment of value number from date of TBD for comparison Rev 11 being effective to March 17, 2022 (date Rev 12 became effective) 15
Level of Effort Team members are expected to plan specifics for their taskings, share the specifics (e.g.,
methodology and data collection format) with the team lead and team on how they will complete the taskings; provide periodic updates to the team lead, roll-up findings from taskings and participate in periodic team meetings. A SharePoint page has been developed for storing and sharing task related information and project status.
16