ML24157A376

Olmc 310 Regional Office Review June 2024

ML24157A376
Person / Time
Issue date:	07/03/2024
From:	James Anderson NRC/NRR/DRO/IOLB
To:	Matthew Endress, Heather Gepford, Diane Jackson, Tom Stephen, Paul Zurawski NRC/NRR/DRO/IOLB
References
OLMC-310
Download: ML24157A376 (1)

Text

OLMC-310 June 2024

REGIONAL OFFICE REVIEW PROCEDURE

A. Purpose

To establish a procedure and criteria to review the administration of the operator licensing functions in the regional offices and to review the Operator Licensing and Human Factors Branch (IOLB) support of the regional offices in carrying out the operator licensing functions delegated to the regions. IOLB has the following objectives for the regional office review:

• Evaluate how the region consistently implements the regulations, Examination Standards, and other guidance. This is done by assessing examination administration, quality, level of difficulty, and grading.

• Provide feedback to both NRR and regional management about the adequacy of the operator licensing functions delegated to the regional office.

• Evaluate examiner feedback and the responsiveness of IOLB to that feedback, along with any other identified issues based on observations and interactions.

• Determine the adequacy of IOLB support provided to the regional offices to carry out their functions.

• Ensure licensing decisions are implemented in accordance with 10 CFR Part 55 and NUREG-1021, Operator Licensing Examination Standards for Power Reactors.

B. Background

NRR is responsible for regulatory policy and licensing operators pursuant to 10 CFR Part 55. To monitor the execution of this responsibility, IOLB shall oversee the operator licensing functions delegated to the regional offices. The oversight function shall include a quadrennial office review.

C. Implementation Procedure

Note: Attachment 3 is an activity tracker for the team leader to use as an aid in coordinating and implementing the office review.

1. General Guidance

a. OLMC-330, Evaluation of the Operator Licensing Program, contains procedures related to planning for the regional office review.

b. The office review will be performed by a team consisting of the necessary expertise to adequately evaluate the assessment areas. The review of written examinations and operating tests will be performed by one or more qualified examiners. The team leader and team members are selected in accordance with OLCM-330.

c. During the regional office review, IOLB evaluates the administration of operator licensing functions delegated to the region by using the assessment guidance in Attachment 2. IOLB also evaluates the support and guidance provided to the

1 OLMC-310 June 2024

regional office. The assessment of many of the elements that make up the regional operator licensing functions will be both objective and subjective in nature. For those licensing functions where definitive requirements have been previously established, the ratings will be based on objective measurements of performance against those requirements. The specific ratings are defined as follows:

SATISFACTORY (S):

• Goals and requirements are consistently met or exceeded.

• Schedules are realistic and consistently met.

• Initiatives are implemented with positive results.

NEEDS IMPROVEMENT (NI):

• Goals and requirements are consistently not met.

• Schedules are frequently not met.

• Regional management attention is warranted.

• Potential vulnerability or deficiency.

• Observations or deficiencies that are significant, widespread, or repetitive as defined in OLMC-320, IOLB Review of Initial Licensing Examinations (these will require regional office response).

NOT APPLICABLE (NA): Not applicable or not evaluated (use of this rating must be justified).

Additional reviews may be scheduled if necessary to meet the objectives of the review program.

However, a region should receive no more than one office review in any fiscal year unless unforeseen or special circumstances arise. If an additional office review is necessary, then IOLB will document the circumstances requiring the additional review in the associated quadrennial operator licensing program review report.

2. Scheduling the Office Review

Note: For hybrid reviews, the following timeline is recommended:

• T-4 months (from in-person portion) - planning memorandum submitted

• T-3 months - virtual entrance meeting; remote portions of the office review start

• T-2 weeks - remote portion of review complete

• T In-person portion of the review; regional management debrief

• T+1 month - report issued

a. For hybrid reviews, at a minimum, the team leader, IOLB branch chief, and DRO director should attend the in-person portion of the review.

b. Based upon the dates established for the office review in accordance with OLMC-330, the assigned team leader will make detailed arrangements, including entrance and pre-exit/exit meeting times and locations.

• Ensure a calendar scheduler is sent to all needed attendees (include a MS Teams Meeting link for personnel who are not attending in person).

2 OLMC-310 June 2024

o Attendees are listed in Sections 7 and 9 below.

• The date(s) of the review should be selected to coincide with a time when most examiners are available for telephone, MS Teams, or in-person interviews.

o The review dates should be scheduled to accommodate NRR Program Office manager (i.e., division director or division deputy director and IOLB chief) interviews with region OL staff and participation in the exit briefing.

• For fully in-person office reviews, the office review period is typically a week, and document reviews may commence remotely beforehand.

• For hybrid and fully remote reviews, the office review period can be approximately three months long. An initial entrance meeting with regional management should be conducted virtually prior to commencing the remote portion of the review. Subsequently, a follow-on entrance meeting should then be conducted, in person, at the commencement of the office visit.

• The office review should be included on both the DRO calendar and the DRO division directors personal calendar (contact the divisional administrative assistant as needed).

3. Scheduling Interviews

The DRO director or deputy director should conduct interviews with all available regional examiners, the regional OLA, the regional operator licensing (OLC) branch chief (BC), the regional division director/deputy director, and the regional administrator/deputy regional administrator. The IOLB chief should conduct interviews with the regional OL branch chief. The IOLB chief and the IOLB OLA should also interview the regional OLA. Request DRO AA assistance in scheduling interviews for the DRO director or deputy director.

4. Coordinate the Office Review Visit

a. Schedule a time to meet with the regional OL BC to discuss specific review items (e.g., items in Attachment 2, Section 5.1, Region/Headquarters Interaction).

b. Schedule a meeting with the DORS/DRS director, regional OL BC, and IOLB BC (via telecon or Teams, if necessary) to discuss observations prior to the exit meeting.

c. Ensure office space (with hardwired network or WiFi access) is available for the team members attending in-person, including the DRO management representative.

d. Ensure regional office building access (i.e., badges can access door card readers).

e. Ensure access to the Part 55 dockets in ODD and/or ADAMS; the regional OLA, ADAMS IM, and/or ODD support can assist with this)

3 OLMC-310 June 2024

f. Ensure facility exam files have the information the team members will need for the review prior to arriving onsite (if a fully in-person review will be conducted).

g. Review both the Regions self-assessment report and the report memo from IOLBs previous quadrennial review of the Region.

• In general, regions will implement a self-assessment during the year leading up to the IOLB office review. The IOLB review team should evaluate this report to determine whether the Region is adequately self-identifying deficiencies.

• In reviewing the results of the prior quadrennial office review, the review team should consider whether any previously identified issues are repetitive in nature and have gone uncorrected since the previous review.

h. Periodic updates between the regional OL branch chief, team leader, and IOLB chief on assessment results should be scheduled during the remote portions of the review.

5. Prepare a Review Plan

a. The team leader shall prepare a review plan and provide for review by the IOLB chief. The review plan should consider the IOLB examination audit schedule and the results of previous reviews/audits. Issues from previous reviews, regional self-assessments, annual program reviews, and examination audits are generally of particular interest. It should be noted that not all review areas of OLMC-310 need to be implemented during each office review, nor does every element of the selected review areas need to be completed. Rather, the team leader, in coordination with the IOLB chief, may select focus areas for the review based upon insights into regional performance, areas of potential concern, or to compensate for staffing constraints that limit available review team resources.

• Attachment 3, Team Leader Activity Tracker, lists the activities that need to occur in advance of the office review. The team leader will assign the assessment areas among the team members based on their experience and expertise.

b. The team leader should select approximately four initial examinations to evaluate during the review (fewer or more examinations may be sampled depending on availability of resources and scope of review of each selected examination).

• Select the initial exams to be included in the sample by considering facility licensees and chief examiners that were reviewed during audits conducted over the four-year period.

• Try to get a broad sample of facilities and chief examiners. Consider selecting exams that had post-exam comments, written or operating test failures, findings/violations, or simulator issues to see how the exam team responded to these situations.

4 OLMC-310 June 2024

• The team leader should also consider the exams that the region selected for the sample of exams reviewed for its self-assessment (for example, it may be informative to include within the sample one of the exams that was self-assessed to evaluate the quality of the self-assessment).

• Do not select initial exams that were either developed by an IOLB examiner (such as for qualifications) or had an IOLB chief examiner lead the exam (however, exams where IOLB examiners participated on the exam team can still be selected for the review).

c. Select IP 71111.11 requalification inspections for review. Typically, two biennial requalification inspections are selected for a detailed review, with all of the requalification inspection activities that occurred during the quadrennial review period being, at a minimum, reviewed for appropriate direct inspection hours usage.

d. Approximately three months prior to the review, conduct a brief with the review team and regional support team members to discuss the assignments, expectations, and roles. Travel plans should be coordinated for team members that will be participating in an in-person visit.

• Ensure the team members understand how to perform their review activities and have access to the Part 55 dockets for the region if necessary to perform their review activities.

• Discuss which activities might be able to be done remotely; this may be particularly helpful if the team will be onsite for a relatively short amount of time.

• Provide the team members with a list of focus areas based on the results of the regions audits over the four-year period, the results of the regions self-assessment, the annual program review, the results of other recent office reviews/audits, and input from the IOLB branch chief. For example, this may include items like adequacy of JPM task standards or whether SRO written exam questions appropriately test to the SRO license level.

• The team leader should establish a means for reviewers to provide their observations in a common electronic location during the review.

6. Submit Memorandum of Office Review

a. No later than 30 days prior to the scheduled review date, the team leader will send a memorandum in the format of Attachment 1 to the regional office being reviewed.

Place the memorandum in ADAMS under NRR/NRR-DRO/IOLB/OFFICE REVIEWS/REGION X/YEAR.

7. Entrance Meeting

5 OLMC-310 June 2024

a. Attendees: At a minimum, the team leader and IOLB chief should conduct an entrance meeting with the Regional OL BC. The other team members, regional examiners, OLA, DRO management representative, and the DORS or DRS (as applicable) management representative are optional attendees.

• Recommended entrance meeting topics include the following:

o The purpose of the visit and that the team will be using the guidance in OLMC-310 to conduct the review. Emphasize that the team will assess the effectiveness of the regions implementation of the OL program and collect feedback about the effectiveness of the program offices support of the regions implementation of the OL program.

o Introduce the team members and explain what each member will be reviewing. This is also a good time for the team members to ask who the points of contact will be for their review topics, if necessary.

o Identify the exams included in the office review sample.

o Confirm the schedule and which interviews will occur in-person.

o Explain how the results will be documented. The team leader will develop a draft report using input from the team members. The report will document the results of the teams review of the functional areas, whether those areas are satisfactory or need improvement, whether additional management attention is warranted, and/or whether a response to the program office is required. The team leader will send the draft report to the Region OL BC for comments. The team leader will consider comments from the region before issuing the final report, which will be transmitted via memo from the DRO Director to the DORS/DRS Director.

o The timing of team debriefs and that the team will inform the Region OL BC of any significant issues.

o Ask if any of the attendees have any questions or additional comments.

8. Office Review Activities

a. The scope of the assessment is detailed in Attachment 2, Assessment of the Regional Operator Licensing Program Summary Sheet. A detailed review of each of the following assessment areas or all aspects of an assessment area is not required during each review. These areas include the following sections of Attachment 2:

• Section 1.0, the examination administrative requirements assessment, which should be completed in conjunction with the written and operating test examination packages selected for review.

6 OLMC-310 June 2024

• Sections 2.0 and 3.0, the operating and written examination assessments, should be used to review selected portions of examination packages. Note that these examination packages are in addition to those audited per OLMC-320.

Since each regional office is responsible for quality assurance of examinations, no distinction shall be made between examinations written by the NRC or by the facility licensee. The examination package shall be reviewed for compliance with the revision of NUREG-1021 in effect at the time the examination was administered.

• Section 4.0, the operator requalification program assessment, should be used to review the biennial requalification program inspections or NRC-administered requalification examinations, if applicable.

• Section 5.0, the regional operations assessment, should be used to review the region's method of disseminating guidance from IOLB, scheduling examinations, handling waiver requests, reviewing medical evaluation forms, processing license denial appeals, docket management, and Reactor Program System Operator Licensing (RPS-OL) entries.

• Section 6.0, the resource utilization assessment, should be used to assess the effectiveness of the regions examiner training program and the efficiency of examiner resource utilization.

• Section 7.0, the regional and program office communications section, should be used to assess the interaction between the regional office and IOLB.

• Section 8.0, the regional differences assessment, should be used to assess the regions ability to identify regional differences and implement IOLB resolutions.

• Section 9.0, the cross-regional examination participation assessment, should be used to assess the regions participation in cross-regional examinations and implementation of OLMC-120, National Exam Schedule.

b. The team leader should have team members collect the following reports, as applicable to their areas of review, covering the period since the last review in the regional office. NOTE: RPS-OL reports may contain personally identifiable information (PII) and should be handled in accordance with the Privacy Act and NRC guidance.

• The previous Regional Office Review Report

• Annual program reviews completed since the previous quadrennial review

• Regional operator licensing self-assessment reports or a summary of trends and any notable findings from the self-assessment(s)

• RPS-OL Report 14, License Restriction Report

• RPS-OL Report 18, Proposed Denials / Appeals Status 7

OLMC-310 June 2024

• RPS-OL Report 19, Waiver Tracking

• RPS-OL Report 25, Fitness for Duty

• RPS Inspections and Inspection Scheduling, Tracking and Reporting (ISTAR)

Report IR 4, Advanced Findings Violations Search, for 71111.11A, 71111.11B, and 71111.11Q examination audit reports, or a summary of trends and any notable findings from the audits conducted IAW OLMC-320

• RPS ISTAR Report IR 6ROP6, ROP Sample Completion Hours Charged - Min Nom Max, for IP-71111.A11A, IP-71111.11B, and IP-71111.11Q for each in-scope cycle year of the office review

• CAC Labor Summary Report for the following CACS: 000703, FB-OR-IP-7111111A-LICENSED OPERATOR REQUALIFICATION PROGRAM-ANNUAL REVIEW; 000704, FB-OR-IP-7111111B-LICENSED OPERATOR REQUALIFICATION PROGRAM; and 000705, FB-OR-IP-7111111Q-LICENSED OPERATOR REQUALIFICATION PROGRAM.

o Requal Inspections Time Usage: Enter CACS 000703 and 000704, which are for the IP71111.11A and B inspections, respectively. Select the desired time period and select the desired region. Refer to the resource estimates identified in IP 71111.11 to assess whether hours charged are reasonable.

Refer to screenshot below:

o Exam Time Usage: Using the exams selected for the office review sample, assess whether the hours charged are reasonable. To do this, enter the following information in CACS: Yes for Fee billable, 50 for Docket Part Number (or 52 if the facility licensee is a Part 52 / new reactor licensee),

8 OLMC-310 June 2024

the docket number of the facility licensee under Docket, 11 for Business Line (11 = operating reactors, 17 = new reactors); 4 for Product Line; 165 for Product, and the date range of interest (this should be broad enough to account for pre-exam activities and post-exam doc). Run the same report for the other dockets/units at a multi-unit site to get the total hours for exam because time is typically charged to more than one unit when the exam is being given to license operators at more than one unit at a site (e.g., PVNGS Units 1, 2, and 3). Save/export the report(s) as a CSV file to sort by CAC to see hours / activity, and sum all hours to get total hours spent on the exam. For budgeting purposes, an exam for 12 applicants and 4 examiners will take about 714 hours0.00826 days <br />0.198 hours <br />0.00118 weeks <br />2.71677e-4 months <br /> if the facility wrote it and 300-400 hours more if the NRC staff wrote it. If the hours for an exam are significantly higher (>20%), then assess whether there is appropriate justification (e.g., the exam had a large class size which increased the time spent on administration and doc; there were violations that increased the doc time; there were multiple failures that increased doc time; or the submittal was of poor quality, which led to increased exam review or development time). Refer to the screenshot below (PVNGS Unit 1 shown) as example:

9 OLMC-310 June 2024

• OLMC-120, National Exam Schedule, also has guidance for scheduling exams and assigning examiners to exams to increase the efficiency of examiner resource utilization. The CACS reports and NES in RPS (Report IP21A) can also be used to assess the regions implementation of the guidance in OLMC-120.

9. Exit Meeting

a. Prior to the exit meeting, the team leader will discuss the team's findings with the appropriate regional manager(s).

b. An exit meeting shall be scheduled at the end of the review. Regional staff may attend the exit meeting at regional management discretion. The team leader will ensure a Teams meeting is available, as needed, for individuals who are unable to attend in person. The team leader will ensure that regional staff are either afforded an opportunity to attend the exit meeting or are otherwise provided with a suitable debrief on the team's observations and findings.

• Attendees: The team leader, regional OL BC, IOLB BC, the other review team members (as needed), available regional examiners, the regional OLA, DRO management representative, and the DORS/DRS management representative.

Consideration should be given to inviting to the Regional Administrator and/or their deputy.

• Recommended exit meeting topics include the following:

o Provide an overview of the areas assessed by the team and summarize the review results for each of the nine areas. Provide specific examples of any items needing improvement, requiring a response to be provided to IOLB, activities that the region should continue, and any actions IOLB will be taking as a result of the review.

10 OLMC-310 June 2024

o State that the team leader will issue a written report (in the form of a memorandum) summarizing the teams findings and that a draft will be sent to the regional OL BC for review and comment, and comments will be considered in finalizing the report.

o Ask if any of the attendees have any questions or additional comments.

10.Final Report

a. A report, in the form of a memorandum, shall be prepared by the team leader upon completion of the office review (typically within 30 days of completing the exit meeting). Using input from the team members, document areas needing additional management attention for both IOLB and the region, as well as areas that require the region to provide a response to IOLB regarding corrective actions. This report shall summarize the findings of the assessment team regarding the adequacy and consistency of program implementation by the regional office and areas where IOLB must provide additional support for the regional office.

b. The memorandum containing the report shall be from the Director, Division of Reactor Oversight (DRO), to the appropriate regional Division Director and other appropriate regional office personnel.

c. Prior to final distribution of the report, a preliminary copy should be sent to the regional office for review and comment. The regional office's comments will be considered for inclusion in the final report.

d. The report should be placed in ADAMS in the appropriate region and year folder in the Office Reviews folder in NRR NRR-DRO IOLB.

List of Attachments: : Example Letter to Region Scheduling Quadrennial Office Review : Assessment of the Regional Operator Licensing Program : Optional Team leader Planning Aid

11 OLMC-310 June 2024

ATTACHMENT 1

EXAMPLE LETTER TO REGION SCHEDULING QUADRENNIAL OFFICE REVIEW

MEMORANDUM TO: [First Last Division Director Name], Director

[Division of Operating Reactor Safety (DORS) or Division of Reactor Safety (DRS)], Region ##

FROM: [First Last Division Director Name], Director Division of Reactor Oversight (DRO)

Office of Nuclear Reactor Regulation (NRR)

SUBJECT:

(Year) QUADRENNIAL PROGRAM REVIEW OF THE OPERATOR LICENSING FUNCTION IN REGION ##

The Operator Licensing and Human Factors Branch (IOLB) is planning to perform its quadrennial Region ## office review [during the [the period of (Date - Date), followed by an in-person office visit on (Date - Date)] or [in person during the period of (Date - Date)]. (____BC NAME____) of IOLB has discussed this visit with (___REGION CONTACT___) of your staff and confirmed the dates. (TEAM LEADER), IOLB, will lead the review.

Enclosed for your information is the planned agenda for (this review/the in-office visit).

(___NAME AND TITLE ___) will be the NRR management representative and will discuss the operator licensing programs with you and your staff, review the team's findings, and participate in the exit briefing.

We appreciate the cooperation of you and your staff for this review. I believe that the success of this important review function depends on a high level of regional office cooperation. If you or your staff have any questions pertaining to this review, please call me or [NAME], Chief, IOLB at (301) 415-xxxx or (301) 415-xxxx, respectively.

Attachment:

Agenda CONTACT: (IOLB BC), NRR/DRO 301-415-XXXX

Distribution:

Regional BC (___NAME___)

IOLB OLA Regional OLA Team Members

Concurrence: Team leader, IOLB BC, DRO DD

1 Attachment 1 OLMC-310 June 2024

ATTACHMENT AGENDA (Fully in-person)

DAY ONE Assemble and brief team/region staff on schedule for visit

Brief region management and staff on schedule for visit

Begin assessments and interviews DAY TWO Continue assessments and interviews

Meet with regional operator licensing staff to discuss assessments, as needed

DAY THREE Continue assessments and interviews

Meet with regional operator licensing staff to discuss assessments, as needed

Brief NRR and DRS managers on team findings

DAY FOUR Exit meeting

(OR)

AGENDA (Hybrid)

DAY ONE Brief region management and staff on schedule for visit

Begin in-office portions of assessments

Begin interviews

Meet with regional operator licensing staff to discuss assessments DAY TWO Complete in-office portions of assessments

Complete interviews

Meet with regional operator licensing staff to discuss assessments

Brief NRR and DORS managers on team findings

Exit meeting

2 Attachment 1 OLMC-310 June 2024

ATTACHMENT 2

ASSESSMENT OF THE REGIONAL OPERATOR LICENSING PROGRAM

ASSESSMENT

SUMMARY

SHEET

1.0 EXAMINATION ADMINISTRATIVE REQUIREMENTS ASSESSMENT S/NI/NA

2.0 OPERATING TEST ASSESSMENT S/NI/NA

3.0 WRITTEN EXAMINATION ASSESSMENT S/NI/NA

4.0 OPERATOR REQUALIFICATION PROGRAM ASSESSMENT S/NI/NA

5.0 REGIONAL OPERATIONS ASSESSMENT S/NI/NA

6.0 RESOURCE UTILIZATION ASSESSMENT S/NI/NA

7.0 REGIONAL AND PROGRAM OFFICE COMMUNICATIONS S/NI/NA

8.0 REGIONAL DIFFERENCES ASSESSMENT S/NI/NA

9.0 CROSS-REGIONAL EXAMINATION PARTICIPATION ASSESSMENT S/NI/NA

3 Attachment 2 OLMC-310 June 2024

1.0 EXAMINATION ADMINISTRATIVE REQUIREMENTS ASSESSMENT

1.1 License applications (NRC Form 398 & NRC Form 396) were reviewed for completeness and accuracy and approved by an examiner. (ES-2.2, Section C.1).

Comments:

1.2 An ADAMS Facility Examination File exists for each facility, containing the pertinent written correspondence, checklists, and forms listed in ES-5.1,Section I.

1.2.1 ADAMS files have public release dates consistent with program office delayed-release guidance (i.e., ROI 23-07 (ML23130A152)).

Comments:

1.2.2 Pre-and Post-Examination Security Agreements were executed in accordance with the Examination Standards. (ES-2.1, Section C.2; Form 1.3-1).

Comments:

1.2.3 The Examination Report was prepared in accordance with ES-5.1,Section I.

Comments:

2.0 OPERATING TEST ASSESSMENT

Examinations assessed: ____________________

2.1 Adequacy of the Operating Test Reviews:

2.1.1 Perform a review of the Operating Tests assigned to the office review in accordance with OLMC-320, with the following revisions/additions:

• A sampling of, at a minimum, 2 dynamic simulator scenarios per examination is an adequate sample.

• A sampling of, at a minimum, 5 JPMs per examination is an adequate sample.

• If the nominal sample identifies significant errors or issues, consult with the team leader to determine if the sample size should be expanded.

Comments:

2.2 Examination Record Retention

4 Attachment 2 OLMC-310 June 2024

2.2.1 Sample 2-3 individual docket files, including those of operators who failed the exam1, to ensure the files contain the following:

2.2.1.1Examination Reports (Forms 5.1-2 and Forms 3.6-2 through 4, Form 3.6-5, as-run Form 3.3.1, for each operator, along with all correspondence with the applicant. (ES-5.1,Section I).

Comments:

2.2.1.2For any operator who failed the operating test of the NRC exam, docket file contains all failed portions of the exam, including Form 3.3.1 if the simulator operating test portion was failed, in addition to the forms identified in item 2.2.1.1 (ES-5.1,Section I). Verify that the operating test comments justify the grades assigned per ES-3.6.

Comments:

2.2.2 The Facility Examination File contains the as-run forms for each scenario set (ES 4.1) and operating test outlines (Forms 3.2-2 and 3.2.3) administered during the examination.

Comments:

3.0 WRITTEN EXAMINATION ASSESSMENT

Examinations assessed: ____________________

3.1 Adequacy of the Written Examination reviews:

3.1.1 Perform a review of the written examinations assigned to the office review in accordance with OLMC-320, with the following revisions/additions:

• If the nominal sample identifies significant errors or issues, consult with the Team leader to determine if the sample size should be expanded.

Comments:

4.0 OPERATOR REQUALIFICATION PROGRAM ASSESSMENT

Inspections assessed: ____________________

4.1 For two (or more) facilities selected for sampling, requalification program inspections were conducted in accordance with IP 71111.11, including the quarterly, annual, and biennial samples. RPS and Inspection Reports data match, and any deviations from the IP are appropriately documented.

1 In the event that the sample of examinations selected for review does not include any applicants who failed the exam, the review team should expand the sample to include an additional partial-scope examination review of, at a minimum, the grading documentation for a written examination or operating test failure.

5 Attachment 2 OLMC-310 June 2024

Comments:

4.2 For two (or more) facilities selected for sampling, the regional office announced its intent to conduct inspections through the annual assessment letter (https://www.nrc.gov/reactors/operating/oversight/listofasmrpt.html).

Comments:

4.3 For two (or more) facilities selected for sampling, how many inspectors are typically dispatched to the site? How long are the inspectors at the site? How many direct inspection effort hours have been required to complete the inspections in comparison to the ISTAR? (Refer to RPS ISTAR Report IR 6.)

Comments:

4.4 Inspection findings are written against the regulation or against program specific commitments (i.e. site procedures that implement the accredited training program, or committed standards (e.g., ANS-3.1, 3.4, or 3.5), and not written against NUREG-1021 Examination Standards).

Comments:

4.5 Inspection findings are well supported and consistent with IOLB guidance. Conclusions are reasonable and based on facts discussed in the report.

Comments:

4.6 The NRC inspectors are qualified to perform the inspection.

Comments:

5.0 REGIONAL OPERATIONS ASSESSMENT

5.1 Region/Headquarters Interactions

5.1.1 An effective method is used to disseminate information (e.g., ROIs, conference call notes, appeal results, manual chapter changes) to all examiners in the region. (Note: this item should be assessed during interviews with the region examiners.)

Comments:

5.1.2 Verify examiners are capable of accessing the ADAMS document file for ROIs.

(This is required by OLMC-160, Regional Office Interactions.) (Note: this item could be assessed during interviews with the region examiners.)

Comments:

6 Attachment 2 OLMC-310 June 2024

5.1.3 Summarize regional views on level of IOLB support and timeliness of response to requests for support. Include planned or completed actions taken to address regional comments. (Note: this item should be assessed during interviews with the region examiners, OLA and chief.)

Comments:

5.2 Waivers

5.2.1 Using RPS-OL Report 19 and the IOLB ROI log, determine how many applications requested waivers (which includes waivers, deferrals, and excusals) during the review period. _______

5.2.2 The regional offices resolution of waivers, deferrals, excusals, and exemptions conforms to NRC policy (ES-2.2.).

Comments:

5.2.3 USING RPS-OL Report 19 and ADAMS operator docket files, verify that a sample (10 - 20%) of the waiver decisions are documented on NRC Form 398 and in correspondence to the applicant (ES-2.2, Section 4).

Comments:

5.3 Review of medical examinations

5.3.1 The region has an effective medical examination tracking system to monitor the status and location of medical reviews.

Comments:

5.3.2 Select several ADAMS operator docket files to ensure NRC Form 396 and, if appropriate, the NRC doctor's review is filed in each docket.

Comments:

5.3.3 Verify that only current medical examinations have been accepted. (ES-2.2, ES-5.3).

Comments:

5.3.4 Using RPS-OL Report 14, check several ADAMS operator docket files for which the medical evaluations require license restrictions and verify the restrictions are stated on the license.

Comments:

7 Attachment 2 OLMC-310 June 2024

5.3.5 Using RPS-OL Report 25 and current enforcement information (the regional enforcement specialist can supply information for cases against operators), verify that fitness for duty cases are documented in the ADAMS operator docket files and RPS-OL.

Comments:

5.4 License denial and appeal process

5.4.1 Number of failures overturned:

5.4.1.1By the regional office: _______

5.4.1.2By IOLB: _______

5.4.2 Assess whether the regional office should have reasonably been expected to identify and correct the flawed test items prior to examination administration.

Comments:

5.4.3 The region, as applicable, supported the appeal process in accordance with ES-5.1, ES-5.2, and OLMC-500, as applicable.

Comments:

5.4.4 Using RPS-OL Report 18, verify that any appellants ADAMS operator docket files include documentation from DRO on the appeal results.

Comments:

5.4.5 For any appeal that resulted in examination grading changes, confirm that the data in RPS-OL was updated, as appropriate, for the appellants.

Comments:

5.5 Docket Management and Office Procedures

5.5.1 Applications are logged in as received.

Comments:

5.5.2 Applications are processed within 30 days of receipt.

Comments:

5.5.3 ADAMS operator docket files are maintained and up to date.

Comments:

8 Attachment 2 OLMC-310 June 2024

5.5.4 Completeness of ADAMS operator docket file contents verified by a random sample of docket files for completeness required per ES-511,Section I. Track a minimum of one application through the complete process from date of receipt to issuance of license or denial. Ensure that docket files have the appropriate security to ensure the privacy of applicants.

Comments:

5.6 Status of RPS-OL Entries/Licensing Assistant Interaction with RPS-OL

5.6.1 RPS-OL entries are up to date within 1 week of the review date. Enforcement against operators are entered in RPS-OL.

Comments:

5.6.2 ADAMS operator docket files align with their RPS-OL entries.

Comments:

5.6.3 Document any notable issues experienced with RPS-OL and Operator Digital Docket (ODD) after the last review and convey the recent RPS-OL and ODD developments to the regional OLA.

Comments:

6.0 RESOURCE UTILIZATION ASSESSMENT

6.1 Examiner training and development

6.1.1 The training program is implemented for new examiners. (Inspection Manual Chapter [IMC] 1245, Appendix C-10.)

Comments:

6.1.2 Perform a sample review of examiner training journals to determine compliance with Appendix C-10 of IMC-1245, if applicable. The sample should generally include 1-2 qualification journals with an emphasis upon newly qualified examiners and examiners who have recently completed further qualification as chief examiners.

Comments:

6.1.3 Each examiner has been observed during an operating test or inspection on an annual basis in accordance with IMC-0102.

Comments:

9 Attachment 2 OLMC-310 June 2024

6.1.4 Each examiner attended a reactor technology and an examination techniques refresher training course at 3-(calendar) year intervals (IMC-1245, Appendix D-1).

Comments:

6.1.5 Each examiner maintained proficiency by administering one complete operating test per calendar year (IMC-1245, Appendix D-1).

Comments:

6.1.6 Each examiner attended the National Examiners Conference or received the training related information that was presented at the conference via alternate means.

Comments:

6.2 Staff Utilization

6.2.1 During the four-year period of the review, has the region interacted with regional industry representatives on operator licensing issues? For example, did the regional office staff participate in meetings with utility training staffs other than routine meetings associated with specific examination assignments, such as training conferences or workshops held by utility organizations? (Note: meetings with industry representatives focused on operator licensing topics or training help to improve the quality of facility-developed examination materials.)

Comments:

6.2.2 Review examiner time usage and evaluate appropriateness of charged time for examinations and requalification program inspections by assessing FTE utilization and budgeted FTE (request information from the DRO TA; prior year budgets are in cuff reports).

6.2.2.1Assess FTE utilization for fee billable OL work:

• Select CAC Labor Summary under Reports on cacs.nrc.gov

• Select Yes for fee-billable

• Select the region under Office

• Select 11 - Operating Reactors under Business Line

• Select 4 - Licensing under Product Line

• Select 165 - Operator Licensing under Product,

• Enter from and to dates for the most recent full fiscal year of the review period

• Click View Report

• Save report as a CSV file

• Sum all regular and non-regular hours of fee-billable work

10 Attachment 2 OLMC-310 June 2024

• Convert hours from Step (10) to FTE used (1 FTE = 1160 hours0.0134 days <br />0.322 hours <br />0.00192 weeks <br />4.4138e-4 months <br />)

• Compare FTE used for the fiscal year to fee-billable OL FTE budgeted for the business line for the fiscal year

• Repeat for each fiscal year of the review period

• NOTE: If the region is budgeted for new reactor work, perform all the same steps EXCEPT at (4), select 17 - New Reactors under Business Line.)

Comments:

6.2.2.2Assess FTE utilization for non-fee billable OL work:

• Select CAC Labor Summary under Reports on cacs.nrc.gov

• Select No for fee-billable

• Select the region under Office

• Select 11 - Operating Reactors under Business Line

• Select 4 - Licensing under Product Line

• Select 165 - Operator Licensing under Product,

• Enter from and to dates for the most recent full fiscal year of the review period

• Click View Report

• Save report as a CSV file

• Sum all regular and non-regular hours of non-fee-billable work

• Convert hours from Step (10) to FTE used (1 FTE = 1160 hours0.0134 days <br />0.322 hours <br />0.00192 weeks <br />4.4138e-4 months <br />)

• Compare FTE used for the fiscal year to non-fee-billable OL FTE budgeted for the business line for the fiscal year

• Repeat for each fiscal year of the review period

• NOTE: If the region is budgeted for new reactor work, perform all the same steps EXCEPT at (4), select 17 - New Reactors under Business Line.)

Comments:

7.0 REGIONAL AND PROGRAM OFFICE COMMUNICATIONS

7.1 Feedback from Regions

7.1.1 Do regional office staff, other than the branch chief, regularly participate in the biweekly phone call with IOLB and the other regions? Are these calls effective and useful? (Note: this should be assessed during interviews with regional examiners.)

Comments:

7.1.2 Has IOLB participated in regular communication with the regional office and been responsive to questions/concerns from regional staff since the last regional office review? (Note: this should be assessed during interviews with regional examiners.)

Comments:

11 Attachment 2 OLMC-310 June 2024

8.0 REGIONAL DIFFERENCES ASSESSMENT

8.1 During the four-year period, did the regional office identify any regional differences and inconsistencies with other regional offices? If so, were these addressed by an ROI or during a biweekly call or at the biennial conference? (Note: this may be assessed by reviewing ROIs submitted by the region during the review period and by discussing with regional staff during interviews.)

Comments:

8.2 Were any new regional differences identified as a result of the review? If so, ensure that these differences are documented for review by IOLB.

Comments:

9.0 CROSS-REGIONAL EXAMINATION PARTICIPATION ASSESSMENT

9.1 How many exams in other regions did the regional office support? (Note: The goal is for each regional office to support another regional office examination at least once per year. Regional offices can provide examination support to any other regional office, but it is preferred that the regional offices rotate support to other regional offices from year-to-year to allow for best exposure to different regional practices.)

Comments:

9.2 How much cross-regional support was provided to the regional office? (See note for 9.1 above.)

Comments:

9.3 Does the region meet the goals discussed in OLMC-120, National Examination Schedule?

Comments:

12 Attachment 2 ATTACHMENT 3 - OPTIONAL TEAM LEADER PLANNING AID

OLMC-310 Task Team Members Due Status / Rating Section Date

C.1.a Planning procedures of OLMC-330 complete (e.g., determine if review is hybrid or fully in office or remote, determine dates)

C.1.b Schedule/send invites for the following:

Entrance meeting Exit Meeting

Date(s) of the review

C.1.b Add meetings/office review to DRO Calendar and DDs personal calendar

C.2.a Schedule NRR Mgmt. interviews with region staff

3 Conf room, computers, file access, bldg access

3.g Review Region's Self-Assessment Results

C.4.a Prepare and submit a review plan

C.4.b Select exams for review

C.4.c Pick BRQ inspections for review

C.5.d Brief team members on assignments.

C.5 Submit office review memo (Attachment 1)

C.6 Entrance meeting

C.7.b Pull requal hours (CACS/Excel Spreadsheet)

C.7.b Pull RPS-OL reports 14, 18, 19 and 25

1 Attachment 3 C.7.b Pull exam resource utilization reports (CACS)

Att 2, Admin Requirements Section 1.0

1.1 License Applications

1.2 ADAMS Facility Examination File

Att 2, Operating Test Assessment Exams: Exams:

Section 2.0

2.1 Adequacy of the facility walk-through

2.2 Examination Record Retention

Att 2, Written Examination Exams: Exams:

Section 3.0 Assessment

3.1 Adequacy of the written examination reviews

3.2 Examination Record Retention

Att 2, Operator Requal Program Section 4.0 Assessment

4.1 Conducted in accordance with IP 71111.11B

4.2 Regional Office announces its intent

4.3 Inspection hours (RPS ISTAR Report IR 6)

4.4 Inspection findings written against approved regulations or commitments

4.5 Inspection findings well supported

4.6 NRC inspectors qualified

2 Attachment 3 Att 2, Regional Operations Section 5.0 Assessment

5.1 Region/Headquarters Interactions

5.2 Waivers

5.3 Review of medical examinations

5.4 License denial and appeal process

5.5 Docket Management and Office Procedures

5.6 Status of RPS-OL Entries

Att 2, Resource Utilization Section 6.0 Assessment

6.1 Examiner training and development

6.2 Staff Utilization

Att 2, Regional and Program Office Section 7.0 Communications

7.1 Feedback from Regions

Att 2, Regional Differences Section 8.0 Assessment

8.1 Regional office identifying differences

8.2 New regional differences

Att 2, Cross-Regional Exam Section 9.0 Participation Assessment

9.1 Regional Office participated

9.2 Cross-regional support was provided

9.3 Implementation of NES Goals

Exit/Doc Finalize assessments 3 Attachment 3 Compile issues and discuss findings

Document findings

8.a Brief Region DRS Director on findings

8.b Attend exit meeting

9 Issue final report

Focus Areas:

4 Attachment 3