ML21222A045

OLMC-330- Evaluation of the Operator Licensing Program

ML21222A045
Person / Time
Issue date:	08/05/2021
From:	Lauren Nist NRC/NRR/DRO/IOLB
To:
Nist L
References
OLMC-330
Download: ML21222A045 (6)

Text

OLMC-330 August 2021 EVALUATION OF THE OPERATOR LICENSING PROGRAM A. Purpose To establish the procedures and guidelines necessary to evaluate the Operator Licensing Program and ensure that it consistently implements the requirements in Title 10 of the Code of Federal Regulations (10 CFR) Part 55, Operators Licenses; the guidance in NUREG-1021, Operator Licensing Examination Standards for Power Reactors; and other policy documents.

B. Overview Annual evaluation of the Operator Licensing Program involves audits of one or two written examinations and operating tests in each NRC region to ensure consistent quality, level of difficulty, administration, and grading. Examination audits are conducted in accordance with OLMC-320, Review of Initial Licensing Examinations. Guidelines for developing the annual audit schedule are in Section C.1. of this OLMC.

The evaluation also includes a detailed review of the operator licensing function of one regional office each year so that each region is reviewed once every 4 years. During the other 3 years, the regions perform a similar self-assessment. The detailed reviews assess nine functional areas: (1) examination administrative requirements, (2) written examinations, (3) operating tests, (4) operator requalification program, (5) regional operations, (6) resource utilization, (7) regional and program office communications, (8) regional differences, and (9) cross-regional examination participation. Regional office reviews are conducted in accordance with OLMC-310, Regional Office Review Procedure. Guidelines for scheduling the regional office review and selecting the team are in Section C.2 of this OLMC.

The results of examination audits, the regional office review and the regional self-assessment, and informal reviews (i.e., administrative reviews) are reviewed annually in accordance with the guidance in Section C.3 of this OLMC to identify topics for training and potential changes to the examination standards.

This OLMC also provides guidelines for cross-regional examination participation and addressing regional differences in Section C.4.

C. Procedures 1

OLMC-330 August 2021

1. Examination Audits

a. The Operator Licensing and Human Factors Branch (IOLB) in NRR will develop a draft audit schedule for the upcoming year, typically by no later than mid-December of the current year. Examination audits will be performed for two examinations from each regional office each year; however, under extenuating circumstances, as determined by the IOLB branch chief in consultation with the Director of the Division of Reactor Oversight, one examination review may be performed in a given year for a regional office. IOLB will use the examination schedule information in RRPS and the following guidelines to determine the examination samples for the coming year and assign reviewers:

• IOLB will limit its review to exams in which there are no members of IOLB serving as the chief examiner.

• When feasible, IOLB will avoid auditing the same chief examiner two years in a row. Each chief examiner should receive approximately the same number of reviews. Similarly, onsite audits at the same facility should not be conducted two years in a row (remote audit is acceptable).

• Availability of the reviewers.

• Qualifications of the reviewers. The assigned reviewers must be qualified as an examiner for the technology of the plant where the examination will be administered.

• On-site examination reviews are preferred to remote reviews. At least one on-site audit will be performed during examination administration, preferably at the region that is scheduled for the regional office review.

b. Once IOLB has developed a draft audit schedule for the upcoming year, the schedule will be sent to the regional operator licensing branch chiefs for review.

c. The regional operator licensing branch chiefs may request selection of different examinations for review. Selection of another examination will be based on the regions justification for the request (i.e., already a large number of observers, resource strain) and current resource and scheduling availabilities.

d. When finalized, the Chief, IOLB will ensure the audit schedule is provided to the assigned reviewers, regional branch chiefs, and chief examiners via email and posted on SharePoint.

i. Once finalized, it is acceptable to revise the final schedule based on examination performance, scheduling conflicts, or other 2

OLMC-330 August 2021 unforeseen circumstances. Revisions to the schedule will be provided via email and posted on the SharePoint site.

ii. In order to minimize the burden on the facility licensee, the region and IOLB should avoid arranging for additional NRC observers to attend examinations scheduled for an onsite audit.

e. The Chief, IOLB may identify focus areas for the reviewers at any time during the year.

f. Each examination review will be performed and documented in accordance with OLMC-320.

2. Regional Office Review

a. Generally by no later than December 1st, the Chief, IOLB will identify the team leader for the next years regional office review. The team leader will generally be a senior examiner from IOLB and will ensure that the office review is scheduled, conducted, and documented in accordance with OLMC- 310.

b. Generally by no later than December 15th, the team leader shall contact the appropriate regional branch chief to arrange an optimal time for the review. The date of the review should be selected to coincide with a time when most examiners are in the office and/or available for telephone, electronic, or in-person interviews.

c. The Chief, IOLB and team leader will identify IOLB staff to participate in the next years regional office review. When possible, the team will include an experienced chief examiner from another regional office and an operator licensing assistant (OLA) from IOLB or another regional office. The Chief, IOLB or team leader should request an experienced chief examiner from a different region to participate in the office review.

d. The Chief, IOLB or team leader will request a DRO management representative to participate in the office review and update the DRO calendar with the dates of the office review.

3. Annual Review

a. During the first quarter of the calendar year, the Chief, IOLB will assign one or more reviewers to read the audit reports, informal review (appeal) review results, office review report from the previous year and the regions self-assessment 3

OLMC-330 August 2021 report1. The purpose of the review is to identify strengths and weaknesses in the Operator Licensing Program, which may be used to identify (1) focus areas for upcoming audits and office reviews, (2) training topics for the biennial operator licensing conference, and (3) necessary revisions to program guidance documents.

b. The reviewer(s) will read the audit reports, informal review (appeal) review results, the office review report from the previous year, and the regions self-assessment report and should use Excel or another tool to categorize audit and office review findings (for example, issues requiring corrective actions) and observations, including strengths, and to identify common themes or trends. The following categories may be useful for categorizing findings and observations.

They are not meant to be all-inclusive; additional categories may also be identified.

i. Written Examination

1. Sample plan

2. Examiner comments

3. KA mismatch

4. Question appropriate for license level

5. Question appropriate for assigned tier

6. Level of difficulty

7. Appropriate use of references

8. Bank limits

9. Resolution of facility post-exam comments

10. Number of questions deleted from exam or modified after administration ii. Operating Test - JPMs

1. Onsite administration

2. Examiner comments

3. JPM outlines

4. JPMs appropriate for license level

5. Level of difficulty

6. Alternate path criteria

7. Task standard

8. Critical steps 1

In accordance with OLMC-310, the team leader for the region office review will obtain the regions self-assessment report as part of the records collected and reviewed during the office review.

4

OLMC-330 August 2021 iii. Operating Test - Simulator Scenarios

1. Varied initial conditions

2. Varied transients and malfunctions

3. Number of transients and events for each applicant

4. Scenario quantitative attributes

5. Resolution of facility post-exam comments

6. Critical Task criteria iv. Grading

1. Written examination grading sheets properly annotated

2. Appropriate documentation of performance deficiencies

3. Appropriate assignment of performance deficiencies to rating factors

4. Grading tabulation accurate

v. Other Documents and Records

1. Documentation of examination changes between examination submittal and approval

2. Completeness of examination package in ADAMS

3. Correct release dates for exam records

4. Necessary signatures provided

5. Completeness of operator docket files

6. Completeness of RPS-OL

c. The reviewer(s) will identify and document strengths, weaknesses, and themes and provide recommendations in a memo to the Chief, IOLB, with the regional OL chiefs on distribution (although the appeal results reports may contain personally identifiable information (PII), do NOT include PII in the annual review report).

d. The review results memo should be placed in the ADAMS Main Library under NRR/NRR-DRO/NRR-DRO/IOLB/Annual Review Folder. The document should be designated as Non-Publicly Available, sensitivity classification B.1: Non-Sensitive, and NRC Users = Viewer box checked. Additionally, inform the examiners of the availability of the document (e.g., via the biweekly teleconference).

4. Cross-Regional Examination Participation Guidelines 5

OLMC-330 August 2021

a. Each regional office is encouraged to support another regional office examination at least once per year as resources allow (refer to OLMC-120, National Examination Schedule). Ideally, the regional offices would rotate support to other regional offices from year to year.

b. If significant regional differences are identified by an examiner while participating in an examination in a different region than the one the examiner is from, then the examiner should prepare a report on interaction (ROI) for review and resolution in accordance with OLMC-160, Report on Interaction Process. The ROI should describe the difference(s), the potential impact on the Operator Licensing Program, and any recommendations for resolution.

c. Examiners are encouraged to identify and share best practices identified from other regions with the rest of the examiners (e.g., by discussing during the biweekly teleconference).

6