NUREG-1021, Revised Effectiveness Review Plan for NUREG-1021 Revision 12 (Public)
| ML23236A119 | |
| Person / Time | |
|---|---|
| Issue date: | 08/24/2023 |
| From: | Theresa Buchanan NRC/NRR/DRO/IOLB |
| To: | |
| References | |
| Download: ML23236A119 (15) | |
Text
2 Revised Effectiveness Review Plan for NUREG-1021, Revision 12, Operator Licensing Examination Standards for Power Reactors
=
Background===
In September 2021, the Operator Licensing (OL) program office (NRR/DRO/IOLB) issued a major revision to NUREG-1021, Operator Licensing Examination Standards for Power Reactors.
Revision 12 was issued for the following reasons:
- 1. Re-incorporate generic fundamentals questions into the reactor operator section of the site-specific written examination and discontinue the standalone NRC Generic Fundamentals Examination (GFE).
- 2. Incorporate changes to the critical task (CT) methodology and the grading of critical performance deficiencies (CPDs) on the simulator operating test and introduce a new category for significant performance deficiencies (SPDs).
- 3. Acknowledge use of the National Academy for Nuclear Trainings ACAD-10-001, Guidelines for Initial Training and Qualification of Licensed Operators, Revision 2, as the standard set of eligibility requirements for accredited training programs and discontinue the process for NRC waivers of facility licensees training program eligibility requirements.
ACAD-10-001, Revision 2, modified experience requirements for eligibility for initial license training programs. Specifically, a facility licensee board will determine if senior reactor operator (SRO)-instant applicants with related sciences degrees have adequate math/science coursework to enter initial license training, and the number of months of previous power plant experience for reactor operator applicants has decreased across the categories of experience (i.e., for both nuclear and non-nuclear plant experience)
- 4. Structure the NUREG into topic-based sections for ease-of-use and improve existing guidance by incorporating recommendations from lessons-learned, clarifications from interactions with NRC region offices, and feedback received from industry.
Purpose The purpose of this effectiveness review is to monitor the application of new instructions and guidance in NUREG-1021, Revision 12, and determine if additional actions are needed, such as additional training or clarifications to support implementation of the major changes made in the revision. This review will also include an assessment of the implementation and outcomes of each major change. The review should address the following questions for the changes introduced in Revision 12: Once implemented, did the changes occur the way that the OL program office envisioned, what was the impact of the changes, and were the actual results and outcomes worth the resources invested?
3 Duration Most of the review will start just after the first Revision 12 examination is administered in April 2022 and continue for at least 2 years (i.e., at least through April 2024), while the GFE related review will continue through 2027. The OL program office may extend the duration of the review to collect additional data, if necessary, to better understand a potential issue. The OL program office will develop a report to capture the findings from this review.
4 Tasks Task 1: Review the integration of generic fundamentals topics into the site-specific examination and verify that the instructions in NUREG-1021 are being used as intended; identify any areas for improving clarity in the instructions/guidance related to these "new" questions.
Subtask Sample Duration Deliverables/Notes 1a Check for consistent and adequate sampling of generic fundamentals topics on written examination outlines and a sample of examinations by doing the following:
Verify that written examination outline reflects adequate sampling of generic fundamentals topics on the RO portion in Tier 2 and Tier 4 and complies with the instructions in ES-4.1 B steps 5a, 5f and 5h.
Record the total number of generic fundamentals questions, including the number of K/As from Section 5 and Section 6 of the applicable K/A catalog for each examination.
All GF questions on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit.
CY 2022 and 2023 Deliverable: exam review memos IAW OLMC-320.
Issue a memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review.
Statistics and observations for review reports.
5 1b Check the quality of all generic fundamental questions against the criteria on Form 2.3-5 Written Examination Review Worksheet and observe how chief examiners are reviewing proposed questions against the criteria in NUREG-1021 as documented on the written examination quality checklists.
All GF questions on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit.
CY 2022 and 2023 Deliverable: exam review memos IAW OLMC-320.
Draft memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review 1c Track how many appeals are associated with generic fundamentals topic questions (RO Section, Tier 2/4) including the number of licensing decisions upheld and overturned. If overturned, determine reason for overturning (e.g., due to psychometric flaws).
All written examination appeals May 2022-May 2024 Statistics and observations for review reports.
6 Task 2: Verify that generic fundamental knowledge is being maintained through initial and continuing training programs.
Subtask Sample Duration Deliverables/Notes 2a Track performance on Tier 2 and Tier 4 generic fundamentals questions and compare to a baseline on GFE performance over 2018 and 2019 (average score).
For each written exam, tabulate individual applicant scores on GF questions. Revisit the sample plan distribution for generic fundamentals in NUREG-1021, Rev. 12 if any of the following performance thresholds are crossed at the end of a 2-year evaluation period on an industry level:
- 1. >10% of applicants score less than 70% on GF questions (Tier 4 and Tier 2 Section 5 G K/As)
- 2. >5% of applicants score 50% or less on GF questions o If 5% is exceeded, 50% still passed the RO portion of the exam (i.e., 80%)
- 3. >10% of facilities have an average score of less than 70% on GF questions on an examination OR have an identified weakness in generic fundamentals training All written examinations April 2022-September 2027*:
Pre-evaluation: April 2022-September 2023 (most applicants during this period will have previously taken a GFE)
Evaluation period 1:
October 2023-September 2025 Evaluation period 2:
October 2025-September 2027
- End date may be extended as necessary to ensure accurate trending with sufficient justification and approval Record # of GF questions and individual applicant scores on "GF questions" per exam. (e.g., RO1 6/6, RO2 4/6, SROI1 3/6, etc.) Consider using PowerBI/Power automate for data analysis.
7 o Screening criteria:
A weakness in the GF portion of the training program identified by the facility or INPO, high miss GF question that the NRC determined was related to a training issue, an operational event that was determined to have GF knowledge as either a contributing or root cause, or as determined by a majority of chief examiners or the IOLB program office as documented in an inspection report.
2b Monitor operating experience (OpE) trends NRC OpE communications May 2022-May 2024*
- because initial results may vary and 2022/2023 applicants may have already taken the GFE in 2022, this subtask may extend past 2024 Document OpE with training-related issues, include date and event for additional analysis (NRR/DRO/IOEB staff are a resource).
8 2c Evaluate high miss questions to determine if the questions implicitly test* GF and if there are indications that the high miss rate were due to an issue in the training program associated with GF
- In this context, an implicit GF question is one that requires GF knowledge to answer and, on the basis of generic fundamental knowledge alone, at least two answer choices were able to be eliminated; two answer choices rendered significantly less plausible; or one answer choice was eliminated and one rendered significantly less plausible.
All written examinations September 2023 -
September 2027 Obtain a copy of the facility examination performance analysis (See NUREG-1021 ES-4.4 Section B.7), if performed.
Identify any high miss questions (50%
miss rate) and evaluate those questions to determine:
- 1) was the question psychometrically sound?
- 2) are there indications that there is an issue with the facilitys training program portion associated with teaching GFE concepts that caused the high miss rate
- 3) determine if the question implicitly tests GF This information will then be used as part of the threshold metric regarding issues with the facilities training program (see Task 2.a.3).
9 Task 3: Determine if the instructions for identifying critical tasks are clear and verify proper and consistent application of the critical task methodology and grading of critical performance deficiencies; identify any areas for improving clarity in the instructions/guidance related to critical tasks and critical performance deficiencies.
Subtask Sample Duration Deliverables/Notes 3a Check for consistent and adequate critical tasks by reviewing each critical task against the critical task methodology in ES-3.3. If a critical task does not meet the methodology, document specifics (provide task and what element(s) not met and why).
All CTS on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit. If one or more audits reveal problems with critical tasks, then consider expanding sample size.
CY 2022 and CY 2023 Deliverable: exam review memos IAW OLMC-320.
Draft memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review Statistics and observations for review reports.
3b Verify that each CPD meets criteria for grading CPDs in ES-3.6 All CPDs over a 12 month period April 2022-April 2023 Ask regions to report any CPDs at OL biweekly meetings. The OL program office will confer with the cognizant chief examiner to get the information needed to analyze each CPD.
Statistics and observations for review reports.
3c Track how many appeals are associated with critical task and what aspects of critical tasks are contested. Track the number of licensing decisions upheld and overturned due to critical tasks. If overturned, determine reason for overturning.
All simulator operating test appeals May 2022-May 2024 Statistics and observations for review reports.
10 3d Track number of times that alternative boundary conditions are used Analyze alternative boundary conditions for arbitrary conditions Across all CTs that were audited, check alternative boundary conditions are consistent/similar for the same/similar CTs All CTS on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit. If one or more audits reveal problems with critical tasks then consider expanding sample size.
CY 2022 and CY 2023 Draft memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review.
Statistics and observations for review reports.
11 Task 4: Verify that the critical task methodology is adequate for AP1000 simulator operating tests Subtask Sample Duration Deliverables/Notes 4a Track ROIs or other issues reported to OL program office for CTs for AP1000.
Around the timelines for the first three Revision 12 AP1000 examinations/
TBD Statistics and observations for review reports.
4b Review the critical tasks on first three Rev 12 AP1000 examinations as follows:
Each CT meets critical task methodology in ES-3.3 There is adequate variation in CTs from exam to exam Limits for scenario event repeatability in ES-3.4 are met First three Revision 12 AP1000 examinations TBD Coordinate with the chief examiner to review the operating test scenarios.
Statistics and observations for review reports.
12 Task 5: Verify that the instructions for identifying and grading significant performance deficiencies (SPDs) are clear and verify proper and consistent application of SPD grading; identify any areas for improving clarity in the instructions/guidance related to significant performance deficiencies.
Subtask Sample Duration Deliverables/Notes 5a Verify that each SPD meets criteria for identifying and grading SPDs in ES-3.5 and ES-3.6 All SPDs over a 12 month period April 2022-April 2023 Have regions report any SPDs at OL biweekly meetings. The OL program office will confer with the cognizant chief examiner to get the information needed to analyze each SPD.
Statistics and observations for review reports.
5b Track how many appeals are associated with SPDs and what aspects of SPDs are contested.
Track the number of licensing decisions upheld and overturned due to SPDs. If overturned, determine reason for overturning.
All simulator operating test appeals May 2022-May 2024 Statistics and observations for review reports.
13 Task 6: Determine the overall impact of the Revision 12 changes on the pass/fail rate for NRC initial licensing examinations.
Subtask Sample Duration Deliverables/Notes 6a Analyze for any change to the pass rates and average scores for written examinations and operating test under the Revision 12 examination standards by comparing written/operating test pass rates for two years to the pass rates under Revision 11 examination standards.
See Duration input.
Represent and analyze important statistical data for trends of SRO/RO written examination pass-rates and written examination average grades and SRO/RO operating test pass rates April 2022-March 2023, April 2023 - September 2023 Evaluation period 1:
October 2023-September 2025 Evaluation period 2:
October 2025-September 2027
- End date may be extended as necessary to ensure accurate trending with sufficient justification and approval SRO/RO written examination pass-rates and written examination average grades and SRO/RO operating test pass rates data for last 10 years is available on the NRC public website for baseline comparison.
Statistics and observations for review reports.
14 Task 7: Assess the use of new allowances in ACAD-10-001 for SRO-instant applicants to be eligible for licensing class by possessing a "related sciences degree." Assess any reductions to time spent on shift as under-instruction. Determine the impact, if any, of the changes made to eligibility requirements.
Subtask Sample Duration Deliverables/Notes 7a Review observations from audits of SRO-instant applications when this allowance is used; this includes tracking any ROIs related to this topic (e.g., ROI 2019-03).
NRC chief examiners or region chiefs share observations of use of the new process for assessing "related sciences" degrees June 2022 - June 2024 OL biweekly call and region OL chiefs biweekly calls with IOLB. Add to agenda. Can also track any related ROIs.
Statistics and observations for review reports.
7b Record any reductions to time-spent on shift Chief examiners collect information from on-site validation week application audits Started in mid-January 2023 (new item based on feedback) - June 2024 OL biweekly call and region OL chiefs biweekly calls with IOLB.
Statistics and observations for review reports.
15 Task 8: Assess the ease-of use of NUREG-1021 Revision 12 as a result of changes made for streamlining and value Subtask Sample Duration Deliverables/Notes 8a Compare average hour spent on examination development and documentation under Revision 12 with averages under Revision 11.
Trend total numbers of hours charged for examination development and documentation (by region) in 2022 and 2023 and first 6 months of 2024 compare with a set of average hours charged in 2018 and 2019 (by region).
CY 2022 and CY 2023 and first 6 months of CY 2024 Statistics and observations for review reports.
8b Other assessment of value Assessment to include total # ROIs received, appeals, and hearings, and compare to total number from date of Rev 11 being effective to March 17, 2022 (date Rev 12 became effective)
TBD for comparison Statistics and observations for review reports.
16 Level of Effort Team members are expected to plan specifics for their taskings, share the specifics (e.g.,
methodology and data collection format) with the team lead and team on how they will complete the taskings; provide periodic updates to the team lead, roll-up findings from taskings and participate in periodic team meetings. A SharePoint page has been developed for storing and sharing task related information and project status.