ML22138A409

From kanterella
Jump to navigation Jump to search
Effectiveness Review Plan for NUREG 1021 Revision 12 May 2022
ML22138A409
Person / Time
Issue date: 05/18/2022
From: Maurin Scheetz
NRC/NRR/DRO/IOLB
To:
Maurin Scheetz, NRR/DRO/IOLB, 4142758
References
Download: ML22138A409 (8)


Text

Effectiveness Review Plan for NUREG-1021, Revision 12, Operator Licensing Examination Standards for Power Reactors

Background

In September 2021, the Operator Licensing (OL) program office (NRR/DRO/IOLB) issued a major revision to NUREG-1021, Operator Licensing Examination Standards for Power Reactors.

Revision 12 was issued for the following reasons:

1. Re-incorporate generic fundamentals questions into the reactor operator section of the site-specific written examination and discontinue the standalone NRC Generic Fundamentals Examination (GFE).
2. Incorporate changes to the critical task (CT) methodology and the grading of critical performance deficiencies (CPDs) on the simulator operating test and introduce a new category for significant performance deficiencies (SPDs).
3. Acknowledge use of the National Academy for Nuclear Trainings ACAD-10-001, Guidelines for Initial Training and Qualification of Licensed Operators, Revision 2, as the standard set of eligibility requirements for accredited training programs and discontinue the process for NRC waivers of facility licensees training program eligibility requirements.

ACAD-10-001, Revision 2, modified experience requirements for eligibility for initial license training programs.

4. Structure the NUREG into topic-based sections for ease-of-use and improve existing guidance by incorporating recommendations from lessons-learned, clarifications from interactions with NRC region offices, and feedback received from industry.

Purpose The purpose of this effectiveness review is to monitor the application of new instructions and guidance in NUREG-1021, Revision 12, and determine if additional actions are needed, such as additional training or clarifications to support implementation of the major changes made in the revision. This review will also include an assessment of the implementation and outcomes of each major change. The review should address the following questions for the changes introduced in Revision 12: Once implemented, did the changes occur the way that the OL program office envisioned, what was the impact of the changes, and were the actual results and outcomes worth the resources invested?

Duration Most of the review will start just after the first Revision 12 examination is administered in April 2022 and continue for at least two years (i.e., at least through April 2024). The OL program office may extend the duration of the review to collect additional data, if necessary, to better understand a potential issue. The OL program office will develop a report to capture the findings from this review.

1

Tasking Task 1: Review the integration of generic fundamentals topics into the site-specific examination and verify that the instructions in NUREG-1021 are being used as intended; identify any areas for improving clarity in the instructions/guidance related to these "new" questions.

Subtask Sample Duration Deliverables/Notes Check for consistent and adequate sampling of generic fundamentals topics on written examination outlines and a sample of examinations by doing the following: Deliverable: exam review memos IAW

  • Verify that written OLMC-320.

examination outline reflects adequate Issue a memo from sampling of generic All GF questions on IOLB Branch Chief fundamentals topics on the exams (i.e., 8-10 to examination the RO portion in Tier 2 written examinations CY 2022 and auditors with 1a and Tier 4 and per year) selected 2023 instructions to complies with the for program office support this instructions in ES-4.1 B annual audit. effectiveness steps 5a, 5f and 5h. review.

  • Record the total Statistics and number of generic observations for fundamentals review reports.

questions, including the number of K/As from Section 5 and Section 6 of the applicable K/A catalog for each examination.

Check the quality of all generic fundamentals Deliverable: exam questions against the review memos IAW criteria on Form 2.3-5 OLMC-320.

Written Examination All GF questions on Review Worksheet and the exams (i.e., 8-10 Draft memo from observe how chief written examinations CY 2022 and 1b IOLB Branch Chief examiners are reviewing per year) selected 2023 to examination proposed questions for program office auditors with against the criteria in annual audit.

instructions to NUREG-1021 as support this documented on the written effectiveness review examination quality checklists.

2

Track how many appeals are associated with generic fundamental topic questions (RO Section, Tier 2/4) including the Statistics and All written May 2022- May observations for 1c number of licensing examination appeals 2024 review reports.

decisions upheld and overturned. If overturned, determine reason for overturning (e.g., due to psychometric flaws).

Task 2: Verify that generic fundamental knowledge is being maintained through initial and continuing training programs.

Subtask Sample Duration Deliverables/Notes April 2022- April 2024* (or longer Record #of GF based on Track performance on Tier questions and applicant pool for 2 and Tier 4 generic individual applicant first few Rev 12 fundamentals questions scores on "GF exams in the and compare to a baseline questions" per year- they may on GFE performance over All written exam. (e.g., RO1 2a have taken GFE) 2018 and 2019 (average examinations 6/6, RO2 4/6, SROI1 score). For each written 3/6, etc.) Consider

  • Based on initial exam, tabulate individual using results and applicant scores on GF PowerBI/Power applicant that questions. automate for data already took GFE analysis.

in 2022, may extend past 2024.

May 2022-May 2024*

Document NRC inspection findings

  • Because initial and OpE with Monitor operating NRC OpE results may vary training- related experience (OpE) trends communications and and 2022/2023 2b issues, include date and NRC inspection NRC inspection applicants may and event for findings reports have already additional analysis taken the GFE in (NRR/DRO/IOEB 2022, this staff are a resource).

subtask may extend past 2024 Task 3: Determine if the instructions for identifying critical tasks (CTs) are clear and verify proper and consistent application of the critical task methodology and grading of critical performance deficiencies (CPDs); identify any areas for improving clarity in the instructions/guidance related to critical tasks and critical performance deficiencies.

Subtask Sample Duration Deliverables/Notes 3

Deliverable: exam review memos IAW OLMC-320.

Check for consistent and All CTs on the adequate critical tasks exams (i.e., 8-10 Draft memo from (CTs) by reviewing each written examinations IOLB Branch Chief critical task against the per year) selected to examination critical task methodology in for program office CY 2022 and CY auditors with 3a ES-3.3. If a critical task annual audit. If one 2023 instructions to does not meet the or more audits support this methodology, document reveal problems with effectiveness review specifics (provide task and critical tasks, then what element(s) not met consider expanding Statistics and and why). sample size.

observations for review reports.

Ask regions to report any CPDs at OL biweekly meetings.

The OL program office will confer with the cognizant chief Verify that each CPD All CPDs over a 12- April 2022- April examiner to get the 3b meets criteria for grading month period 2023 information needed CPDs in ES-3.6 to analyze each CPD.

Statistics and observations for review reports.

Track how many appeals are associated with critical task grading and what aspects of critical tasks are contested. Track the All simulator Statistics and May 2022- May observations for 3c number of licensing operating test 2024 review reports.

decisions upheld and appeals overturned due to critical tasks. If overturned, determine reason for overturning.

  • Track number of times All CTs on the Draft memo from that alternative exams (i.e., 8-10 IOLB Branch Chief boundary conditions written examinations to examination are used per year) selected CY 2022 and CY auditors with 3d for program office 2023 instructions to
  • Analyze alternative annual audit. If one support this boundary conditions for or more audits effectiveness review arbitrary conditions reveal problems with critical tasks, then 4
  • Across all CTs that consider expanding Statistics and were audited, check sample size. observations for alternative boundary review reports.

conditions are consistent/similar for the same/similar CTs Task 4: Verify that the critical task methodology is adequate for AP1000 simulator operating tests Subtask Sample Duration Deliverables/Notes Around the Any ROIs submitted timelines for the Statistics and Track ROIs or other issues for the first three first three observations for 4a reported to OL program Revision 12 Revision 12 review reports.

office for CTs for AP1000.

examinations AP1000 examinations Review the critical tasks on first three Rev 12 AP1000 Coordinate with the examinations as follows: chief examiner to

  • Each CT meets critical review the operating task methodology in test scenarios.

First three Revision First three ES-3.3 4b 12 AP1000 Revision 12

  • There is adequate Statistics and examinations examinations variation in CTs from observations for exam to exam review reports.
  • Limits for scenario event repeatability in ES-3.4 are met Task 5: Verify that the instructions for identifying and grading significant performance deficiencies (SPDs) are clear and verify proper and consistent application of SPD grading; identify any areas for improving clarity in the instructions/guidance related to significant performance deficiencies.

Subtask Sample Duration Deliverables/Notes Have regions report any SPDs at OL biweekly meetings.

Verify that each SPD The OL program meets criteria for office will confer with All SPDs over a 12- April 2022- April 5a identifying and grading the cognizant chief month period 2023 SPDs in ES-3.5 and ES- examiner to get the 3.6 information needed to analyze each SPD.

5

Statistics and observations for review reports.

Track how many appeals are associated with SPDs and what aspects of SPDs Statistics and are contested. Track the All simulator May 2022- May observations for 5b number of licensing operating test 2024 review reports.

decisions upheld and appeals overturned due to SPDs. If overturned, determine reason for overturning.

Task 6: Determine the overall impact of the Revision 12 changes on the pass/fail rate for NRC initial licensing examinations.

Subtask Sample Duration Deliverables/Notes SRO/RO written examination pass-rates and written Analyze for any change to examination the pass rates and A 2-year and two 1-average grades and average scores for written year trends of SRO/RO operating examinations and SRO/RO written April 2022-March test pass rates data operating test under the examination pass-2023 and April for last 10 years is 6a Revision 12 examination rates and written 2023 - March available on the standards by comparing examination 2024 NRC public website written/operating test pass average grades and for baseline rates for two years to the SRO/RO operating comparison.

pass rates under Revision test pass rates 11 examination standards.

Statistics and observations for review reports.

6

Task 7: Determine the impact, if any, of the modifications made to eligibility requirements.

Subtask Sample Duration Deliverables/Notes OL biweekly call and region OL chiefs Review observations from biweekly calls with NRC chief audits of applications, IOLB. Track any examiners or region June 2022 - June 7a including tracking any related ROIs.

chiefs share 2024 ROIs related to this topic observations (e.g., ROI 2019-03). Statistics and observations for review reports.

Task 8: Assess the ease-of use of NUREG-1021 Revision 12 as a result of changes made for streamlining and value Subtask Sample Duration Deliverables/Notes Trend total numbers of hours charged for examination Compare average hour development and spent on examination documentation (by CY 2022 and CY Statistics and development and region) in 2022 and 2023 and first 6 observations for 8a documentation under 2023 and first 6 months of CY review reports.

Revision 12 with averages months of 2024 2024 under Revision 11. compare with a set of average hours charged in 2018 and 2019 (by region).

7

Assessment to include total # ROIs received, appeals, and hearings, and Statistics and compare to total TBD for observations for 8b Other assessment of value number from date of comparison review reports.

Rev 11 being effective to 3/17/2022 (date Rev 12 became effective)

Level of Effort Team members are expected to plan specifics for their taskings, share the specifics (e.g.,

methodology and data collection format) with the team lead and team on how they will compete the taskings; provide periodic updates to the team lead, roll-up findings from taskings and participate in periodic team meetings. A SharePoint page will be used for storing and sharing task related information and project status.

8