ML22138A409
| ML22138A409 | |
| Person / Time | |
|---|---|
| Issue date: | 05/18/2022 |
| From: | Maurin Scheetz NRC/NRR/DRO/IOLB |
| To: | |
| Maurin Scheetz, NRR/DRO/IOLB, 4142758 | |
| References | |
| Download: ML22138A409 (8) | |
Text
1 Effectiveness Review Plan for NUREG-1021, Revision 12, Operator Licensing Examination Standards for Power Reactors
=
Background===
In September 2021, the Operator Licensing (OL) program office (NRR/DRO/IOLB) issued a major revision to NUREG-1021, Operator Licensing Examination Standards for Power Reactors.
Revision 12 was issued for the following reasons:
- 1. Re-incorporate generic fundamentals questions into the reactor operator section of the site-specific written examination and discontinue the standalone NRC Generic Fundamentals Examination (GFE).
- 2. Incorporate changes to the critical task (CT) methodology and the grading of critical performance deficiencies (CPDs) on the simulator operating test and introduce a new category for significant performance deficiencies (SPDs).
- 3. Acknowledge use of the National Academy for Nuclear Trainings ACAD-10-001, Guidelines for Initial Training and Qualification of Licensed Operators, Revision 2, as the standard set of eligibility requirements for accredited training programs and discontinue the process for NRC waivers of facility licensees training program eligibility requirements.
ACAD-10-001, Revision 2, modified experience requirements for eligibility for initial license training programs.
- 4. Structure the NUREG into topic-based sections for ease-of-use and improve existing guidance by incorporating recommendations from lessons-learned, clarifications from interactions with NRC region offices, and feedback received from industry.
Purpose The purpose of this effectiveness review is to monitor the application of new instructions and guidance in NUREG-1021, Revision 12, and determine if additional actions are needed, such as additional training or clarifications to support implementation of the major changes made in the revision. This review will also include an assessment of the implementation and outcomes of each major change. The review should address the following questions for the changes introduced in Revision 12: Once implemented, did the changes occur the way that the OL program office envisioned, what was the impact of the changes, and were the actual results and outcomes worth the resources invested?
Duration Most of the review will start just after the first Revision 12 examination is administered in April 2022 and continue for at least two years (i.e., at least through April 2024). The OL program office may extend the duration of the review to collect additional data, if necessary, to better understand a potential issue. The OL program office will develop a report to capture the findings from this review.
2 Tasking Task 1: Review the integration of generic fundamentals topics into the site-specific examination and verify that the instructions in NUREG-1021 are being used as intended; identify any areas for improving clarity in the instructions/guidance related to these "new" questions.
Subtask Sample Duration Deliverables/Notes 1a Check for consistent and adequate sampling of generic fundamentals topics on written examination outlines and a sample of examinations by doing the following:
Verify that written examination outline reflects adequate sampling of generic fundamentals topics on the RO portion in Tier 2 and Tier 4 and complies with the instructions in ES-4.1 B steps 5a, 5f and 5h.
Record the total number of generic fundamentals questions, including the number of K/As from Section 5 and Section 6 of the applicable K/A catalog for each examination.
All GF questions on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit.
CY 2022 and 2023 Deliverable: exam review memos IAW OLMC-320.
Issue a memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review.
Statistics and observations for review reports.
1b Check the quality of all generic fundamentals questions against the criteria on Form 2.3-5 Written Examination Review Worksheet and observe how chief examiners are reviewing proposed questions against the criteria in NUREG-1021 as documented on the written examination quality checklists.
All GF questions on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit.
CY 2022 and 2023 Deliverable: exam review memos IAW OLMC-320.
Draft memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review
3 1c Track how many appeals are associated with generic fundamental topic questions (RO Section, Tier 2/4) including the number of licensing decisions upheld and overturned. If overturned, determine reason for overturning (e.g., due to psychometric flaws).
All written examination appeals May 2022-May 2024 Statistics and observations for review reports.
Task 2: Verify that generic fundamental knowledge is being maintained through initial and continuing training programs.
Subtask Sample Duration Deliverables/Notes 2a Track performance on Tier 2 and Tier 4 generic fundamentals questions and compare to a baseline on GFE performance over 2018 and 2019 (average score). For each written exam, tabulate individual applicant scores on GF questions.
All written examinations April 2022-April 2024* (or longer based on applicant pool for first few Rev 12 exams in the year-they may have taken GFE)
- Based on initial results and applicant that already took GFE in 2022, may extend past 2024.
Record #of GF questions and individual applicant scores on "GF questions" per exam. (e.g., RO1 6/6, RO2 4/6, SROI1 3/6, etc.) Consider using PowerBI/Power automate for data analysis.
2b Monitor operating experience (OpE) trends and NRC inspection findings NRC OpE communications and NRC inspection reports May 2022-May 2024*
- Because initial results may vary and 2022/2023 applicants may have already taken the GFE in 2022, this subtask may extend past 2024 Document NRC inspection findings and OpE with training-related issues, include date and event for additional analysis (NRR/DRO/IOEB staff are a resource).
Task 3: Determine if the instructions for identifying critical tasks (CTs) are clear and verify proper and consistent application of the critical task methodology and grading of critical performance deficiencies (CPDs); identify any areas for improving clarity in the instructions/guidance related to critical tasks and critical performance deficiencies.
Subtask Sample Duration Deliverables/Notes
4 3a Check for consistent and adequate critical tasks (CTs) by reviewing each critical task against the critical task methodology in ES-3.3. If a critical task does not meet the methodology, document specifics (provide task and what element(s) not met and why).
All CTs on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit. If one or more audits reveal problems with critical tasks, then consider expanding sample size.
CY 2022 and CY 2023 Deliverable: exam review memos IAW OLMC-320.
Draft memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review Statistics and observations for review reports.
3b Verify that each CPD meets criteria for grading CPDs in ES-3.6 All CPDs over a 12-month period April 2022-April 2023 Ask regions to report any CPDs at OL biweekly meetings.
The OL program office will confer with the cognizant chief examiner to get the information needed to analyze each CPD.
Statistics and observations for review reports.
3c Track how many appeals are associated with critical task grading and what aspects of critical tasks are contested. Track the number of licensing decisions upheld and overturned due to critical tasks. If overturned, determine reason for overturning.
All simulator operating test appeals May 2022-May 2024 Statistics and observations for review reports.
3d Track number of times that alternative boundary conditions are used Analyze alternative boundary conditions for arbitrary conditions All CTs on the exams (i.e., 8-10 written examinations per year) selected for program office annual audit. If one or more audits reveal problems with critical tasks, then CY 2022 and CY 2023 Draft memo from IOLB Branch Chief to examination auditors with instructions to support this effectiveness review
5 Across all CTs that were audited, check alternative boundary conditions are consistent/similar for the same/similar CTs consider expanding sample size.
Statistics and observations for review reports.
Task 4: Verify that the critical task methodology is adequate for AP1000 simulator operating tests Subtask Sample Duration Deliverables/Notes 4a Track ROIs or other issues reported to OL program office for CTs for AP1000.
Any ROIs submitted for the first three Revision 12 examinations Around the timelines for the first three Revision 12 AP1000 examinations Statistics and observations for review reports.
4b Review the critical tasks on first three Rev 12 AP1000 examinations as follows:
Each CT meets critical task methodology in ES-3.3 There is adequate variation in CTs from exam to exam Limits for scenario event repeatability in ES-3.4 are met First three Revision 12 AP1000 examinations First three Revision 12 examinations Coordinate with the chief examiner to review the operating test scenarios.
Statistics and observations for review reports.
Task 5: Verify that the instructions for identifying and grading significant performance deficiencies (SPDs) are clear and verify proper and consistent application of SPD grading; identify any areas for improving clarity in the instructions/guidance related to significant performance deficiencies.
Subtask Sample Duration Deliverables/Notes 5a Verify that each SPD meets criteria for identifying and grading SPDs in ES-3.5 and ES-3.6 All SPDs over a 12-month period April 2022-April 2023 Have regions report any SPDs at OL biweekly meetings.
The OL program office will confer with the cognizant chief examiner to get the information needed to analyze each SPD.
6 Statistics and observations for review reports.
5b Track how many appeals are associated with SPDs and what aspects of SPDs are contested. Track the number of licensing decisions upheld and overturned due to SPDs. If overturned, determine reason for overturning.
All simulator operating test appeals May 2022-May 2024 Statistics and observations for review reports.
Task 6: Determine the overall impact of the Revision 12 changes on the pass/fail rate for NRC initial licensing examinations.
Subtask Sample Duration Deliverables/Notes 6a Analyze for any change to the pass rates and average scores for written examinations and operating test under the Revision 12 examination standards by comparing written/operating test pass rates for two years to the pass rates under Revision 11 examination standards.
A 2-year and two 1-year trends of SRO/RO written examination pass-rates and written examination average grades and SRO/RO operating test pass rates April 2022-March 2023 and April 2023 - March 2024 SRO/RO written examination pass-rates and written examination average grades and SRO/RO operating test pass rates data for last 10 years is available on the NRC public website for baseline comparison.
Statistics and observations for review reports.
7 Task 7: Determine the impact, if any, of the modifications made to eligibility requirements.
Subtask Sample Duration Deliverables/Notes 7a Review observations from audits of applications, including tracking any ROIs related to this topic (e.g., ROI 2019-03).
NRC chief examiners or region chiefs share observations June 2022 - June 2024 OL biweekly call and region OL chiefs biweekly calls with IOLB. Track any related ROIs.
Statistics and observations for review reports.
Task 8: Assess the ease-of use of NUREG-1021 Revision 12 as a result of changes made for streamlining and value Subtask Sample Duration Deliverables/Notes 8a Compare average hour spent on examination development and documentation under Revision 12 with averages under Revision 11.
Trend total numbers of hours charged for examination development and documentation (by region) in 2022 and 2023 and first 6 months of 2024 compare with a set of average hours charged in 2018 and 2019 (by region).
CY 2022 and CY 2023 and first 6 months of CY 2024 Statistics and observations for review reports.
8 8b Other assessment of value Assessment to include total # ROIs received, appeals, and hearings, and compare to total number from date of Rev 11 being effective to 3/17/2022 (date Rev 12 became effective)
TBD for comparison Statistics and observations for review reports.
Level of Effort Team members are expected to plan specifics for their taskings, share the specifics (e.g.,
methodology and data collection format) with the team lead and team on how they will compete the taskings; provide periodic updates to the team lead, roll-up findings from taskings and participate in periodic team meetings. A SharePoint page will be used for storing and sharing task related information and project status.