ML22216A198

From kanterella
Jump to navigation Jump to search
OLMC-320, Iolb Review of Initial Licensing Examinations
ML22216A198
Person / Time
Issue date: 08/31/2022
From: Bernard Litkett
NRC/NRR/DRO/IOLB
To:
Bernard Litkett, NRR/DRO/IOLB 415-3657
References
OLMC-320
Download: ML22216A198 (25)
v • d • e


Text

OLMC-320 August 2022 IOLB REVIEW OF INITIAL LICENSING EXAMINATIONS A. Purpose To provide guidance for reviewing the regions preparation, administration, grading, and documentation of initial operator licensing written examinations and operating tests.

These reviews will emphasize examination consistency, quality, and compliance with the guidance in NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The reviews may include the observation of on-site activities but will also allow for post-examination audits that can be conducted from the program office. By conducting these audits and reviews, the Operator Licensing (OL) program office (IOLB) will verify that examination reliability, and therefore examination validity, remains at acceptable levels.

B. Background

1. IOLB is responsible for ensuring that the regional operator licensing examinations are administered in accordance with the applicable revision of NUREG-1021 (i.e.,

the Examination Standards (ES)). The applicable ES for administering initial examinations are as follows:

  • ES-2.1, "Preparing for Operator Licensing Initial Examinations"
  • ES-2.2, Applications, Medical Requirements, and Waiver and Excusal of Examination and Test Requirements
  • ES-2.3, "Reviewing and Approving Operator Licensing Initial Examinations"
  • ES-3.1, Overview of the Operating Test for Operator Licensing Initial Examinations
  • ES-3.3, General Testing Guidelines for Dynamic Simulator Scenarios
  • ES-3.5, "Administering Operating Tests"
  • ES-3.6, "Grading and Documenting Operating Tests"
  • ES-4.1, Developing Written Examination Outlines
  • ES-4.2, Developing Written Examinations
  • ES-4.3, "Administering Written Examinations"
  • ES-4.4, "Grading and Documenting Written Examinations;"
  • ES-5.1, "Issuing Operator Licenses and Other Post Examination Activities"
2. The reviewer must evaluate the preparation, administration, grading, and documentation of the examinations to the extent required to confirm compliance 1

OLMC-320 August 2022 with NUREG-1021, and to assess the consistency and quality of examinations among the regions. The reviewer must also evaluate NRC examination changes in order to verify that compliance with NUREG-1021 has not been compromised and to ensure consistency and uniformity of initial operator examinations.

3. Secondary goals of the IOLB initial examination reviews include providing feedback to individual examiners and the regions relative to the intent and objectives of the instructions and guidelines in NUREG-1021, and promoting more efficient implementation of the operator licensing process.
4. The results of the initial examination reviews shall be incorporated with the findings of the quadrennial regional assessment (per OLMC-310, Regional Office Review Procedure). The reviews shall not be for the purpose of personnel performance appraisals, which are the responsibility of regional management.
5. On-site examination activities may be observed during the examination week and/or the preparatory/validation week. Most other reviews will be conducted post-examination from the program office to allow for a larger sample of examinations. The IOLB reviewer will make appropriate arrangements with the chief examiner and/or regional operator licensing branch chief ahead of time for any needed assistance.

C. Implementation Procedures Specific information and requirements regarding sample selection can be found in OLMC-330, Operator Licensing Oversight Program. Note that reviews are limited to examinations for which there are no members of the IOLB program office serving in a chief examiner capacity. If on-site activities will be observed, the IOLB reviewer will contact the chief examiner approximately 90 days (if possible) before the scheduled start of the examination observation to plan the review. This should be sufficient time to ensure the IOLB reviewer completes all necessary documentation for unescorted access to the facility, if necessary.

1. Applicable portions of Attachment 1 are performed for any examination assessment that is performed on-site.
2. Applicable portions of Attachment 2 are performed for all examination assessments.
3. Start the review of off-site records (Attachment 2) no later than 90 days after the issuance of the examination report. Complete the entire review and documentation, including sending draft comments to the respective region, within 90 days of starting the review.

2

OLMC-320 August 2022

4. For on-site reviews, follow-up discussions between the IOLB reviewer and the examiner(s) being reviewed will take place to provide feedback to the examiner(s) and to improve the reviewers understanding of the practices applied in the administration of the operating tests. The reviewer may also note those aspects of the simulator examination that he/she thought were particularly well-designed and implemented. The discussions must not be used to review the performance of the applicant(s), and the reviewer must not discuss with the examiner the pass/fail decision(s) to be made prior to the licensing decision.

D. Documentation Required

1. Any amplifying remarks or comments should be documented in the examination assessment memorandum. See Attachment 3 for an example.
2. The IOLB reviewer should also document any feedback related to consistent and efficient examination administration in the examination assessment memorandum.
3. Observations or deficiencies that require regional office response will be documented in the applicable enclosures to the examination assessment memorandum. Items that are significant, widespread, or repetitive require regional office response.
a. Significant - Single observation or group of observations that would have required program office consultation if it had been identified before licensing decisions were made.
b. Widespread - Multiple observations across all facets of the examination that would have required post-examination changes or changes to operating test grading.
c. Repetitive - An observation that would have required post-examination changes or changes to operating test grading that has occurred multiple times in multiple audits since the last office review.
4. If this review is part of the quadrennial regional assessment visit, the IOLB reviewer will forward all comments and remarks to the Team Leader assigned for the quadrennial regional assessment.
5. If this review is not part of the quadrennial regional assessment visit, a memorandum using the template in Attachment 3 will be completed for transmittal to the appropriate regional management.

3

OLMC-320 August 2022

6. These reports, either the quadrennial regional assessment or the individual plant initial operator licensing examination assessment, will generally be signed by the Director (or Deputy) of the NRR Division to which the OL program office is currently assigned.
7. For observations requiring regional response, the region may document any corrective actions taken in a memorandum from the regional branch chief to the IOLB branch chief.
8. The examination assessment memorandum must be placed in ADAMS and profiled as follows: Non-publicly Available, B.1 Non-Sensitive with Limited Document Security: Owners: DPC, IOLB reviewer, IOLB Administrative Assistant and IOLB branch chief. Viewers: NRR-Operator Licensing Group.
9. File the examination assessment memorandum and enclosures in the following ADAMS file location (file in the associated regions folder): /NRR/NRR-DRO/NRR-DRO/IOLB/EXAMINATION AUDITS/EXAM AUDITS/REGION XX E. Attachments
1. On-Site Review of Examination
2. Off-Site Examination Records Review
3. Initial Licensing Examination Assessment Template 4

OLMC-320 August 2022 Attachment 1: On-Site Review of Examination Region: Chief Examiner:

Facility: Examination Dates:

NOTE: The presence of an asterisk (*) indicates a comment related to that item in Section C, below.

A. Written Examination Activities a) If NRC proctored the written examination and the IOLB reviewer was present during administration, mark each of the following. Otherwise, mark each of the following as not applicable.

a) Each examinee was briefed on the policies and guidelines for taking NRC written examinations according to Guidelines for Taking NRC Examinations. (ES-1.2)

YES NO N/A b) The NRC verified each applicants identity and examination level against the approved List of Applicants. (Form ES-2.2-1)

YES NO N/A c) All applicant questions regarding specific written examination test items and all statements of clarification were documented (verbatim if possible) for reference by the NRC in resolving grading conflicts. (ES-4.3 C.1)

YES NO N/A d) All applicants signed their exam cover sheet before leaving the examination room (ES-4.3 B.4.a)

YES NO N/A e) The NRC chief examiner periodically monitored the facilitys examination administration. (ES-4.3 A.2.b)

YES NO N/A 1 Attachment 1

OLMC-320 August 2022 b) If the IOLB reviewer is present at validation week, mark each of the following.

Otherwise, mark each of the following as not applicable.

a) The chief examiner reviewed the facilitys examination security and administration policies for compliance with the guidelines in NUREG-1021. (ES-1.2, ES-1.3, and ES-4.3 B)

YES NO N/A b) The NRC chief examiner inspected the examination facilities during validation week to ensure their adequacy (ES-4.3 A.2.b)

YES NO N/A B. Operating Test Activities

1. If the IOLB reviewer is present during the validation activities (validation week), mark each of the following. Otherwise, mark each of the following as not applicable.

a) An NRC examiner briefed the applicant(s) in accordance with parts A, B, C, D, and E of ES-1.2, Guidelines for Taking NRC Examinations, before beginning the operating test. (ES-3.5 A.8)

YES NO N/A b) The chief examiner confirmed with the facility licensee that the instructors station, programmers tools, and external connections cannot compromise operating test security while conducting examinations. (ES-3.5 A.6)

YES NO N/A c) Each scenario and each job performance measure (JPM) were validated on the simulator. (ES-3.5 A.5 and ES-3.1 B.14)

YES NO N/A d) The examiners reviewed the scenarios together and discussed the required procedures, technical specifications, and special circumstances related to the scenarios. (ES-3.5 G.6)

YES NO N/A 2 Attachment 1

OLMC-320 August 2022 e) Forms 3.3-1, Scenario Outline, and 3.3-2, Required Operator Actions, reflect any changes made to the scenario events or the expected operator actions as a result of scenario validation runs. (ES-2.3 C.2 and C.3.a)

YES NO N/A

2. If the IOLB reviewer is present during operating test administration, mark each of the following. Otherwise, mark each of the following not applicable.

a) The same JPMs and simulator scenarios were not repeated on subsequent days.

(ES-3.1 B.2 and ES-3.5 A.9)

YES NO N/A b) Applicants applying for a multiunit license at a multiunit facility that has a simulator modeled after only one of the units were tested on differences between the units during the in-plant JPM portion of the operating test. (ES-3.1 B.13.c)

YES NO N/A c) Simulator operating tests for each applicant were not divided among examiners.

(ES-3.5 A.3)

YES NO N/A d) Every applicant received a complete set of JPMs. (ES-3.5 A.10)

YES NO N/A e) Any JPMs subsequently altered or replaced after on-site validation were validated before use on the operating test. (ES-3.5 A.5 and ES-3.1 B.14)

YES NO N/A f) Follow-up questions during any part of the operating test complied with the Instructions for the Use of Followup Questions, provided in ES-3.5 section E.

Examiners only asked JPM follow-up questions when warranted. (ES-3.5 E) 3 Attachment 1

OLMC-320 August 2022 YES NO N/A g) The simulator setup matches the conditions specified for each JPM. (ES-3.5 H.4)

YES NO N/A h) The chief examiner had the simulator operator record selected parameters on the facilitys safety parameter display system and retained the recordings. (ES-3.5 G.8)

YES NO N/A i) The facility licensee simulator operator advanced any strip chart recorders and reset digital recorders before beginning each scenario. (ES-3.5 G.11)

YES NO N/A j) The examiners reviewed the scenarios with the simulator operator before beginning the simulator tests. This review included the examiner cautioning the simulator operator to only provide information specifically requested by the applicants and not to compromise the integrity of the examination. The simulator operator was provided with an up-to-date copy of Form ES-D-1 and Form 3.3-2 (or the scenario guide).

(ES-3.5 G.6-7)

YES NO N/A k) If time compression was used during the operating test, applicants were informed during the briefing or during the scenario. (ES-3.5 G.10)

YES NO N/A l) The examiner in charge of each scenario ensured communications with the simulator operator to insert scenario events did not cue applicants. (ES-3.5 G.9)

YES NO N/A m) Examiners monitored conversations between the simulator operator and the applicants to identify inappropriate cuing. (ES-3.5 G.7)

YES NO N/A 4 Attachment 1

OLMC-320 August 2022 n) After completing each scenario, the examiners conferred to compare notes and to verify that each applicant performed the required number of transients and events, determine the status of the scenario critical tasks including any post-scenario critical tasks, identify any significant performance deficiencies, and determine what follow-up questions need to be asked. Planned events were not removed inappropriately, and additional events/scenarios were run as necessary. (ES-3.5 G.16)

YES NO N/A o) Examiners collected any sketches, flow paths, or other illustrations made by the applicants when answering follow-up questions for JPMs and/or scenarios.

Examiners had the simulator operator provide and retain copies of any logs, charts, or other materials to assist in documenting the applicants performance when necessary. (ES-3.5 G.18)

YES NO N/A p) No facility staff, other than simulator operators, observed an operating test without the chief examiners permission. No applicant was allowed to witness another applicants operating test. Surrogate operators were used in accordance with NUREG-1021 and use of shift technical advisors was consistent with in-plant practice. Both surrogates and shift technical advisors were briefed on their expected actions. (ES-3.5 C and D and ES-3.4 B)

YES NO N/A q) The chief examiner obtained operator licensing program office approval for outside organizations (e.g., Institute of Nuclear Power Operators representatives) to observe any portion of an operating test. Observers activities were controlled in accordance with program office guidance. (ES-3.5 C.5)

YES NO N/A r) Any simulator fidelity or performance issues identified during the operating tests were documented and attached to the final examination report. (OLMC-510 Operator Licensing Examination Reports)

YES NO N/A 5 Attachment 1

OLMC-320 August 2022 C. Comments and Notes on the On-Site Activities Section and Item #: (Provide comment or evaluation for the associated item number) 6 Attachment 1

OLMC-320 August 2022 Attachment 2: Off-Site Records Review Region: Chief Examiner:

Facility: Examination Dates:

NOTE: The presence of an asterisk (*) indicates a comment related to that item in Section D, below.

A. Examination Documentation Review

1. Confirm that the ADAMS examination package contains properly completed and signed copies of the following with the appropriate release date, as applicable (OLMC-520, Operator Licensing Examination Records and Documentation):

o Letter 2.1-1, Corporate Notification Letter o Letter 2.3-1, Examination Approval Letter, with updated Form 2.2-1, List of Applicants o Form 1.3-1, Examination Security Agreement o Form 2.1-1, Examination Preparation Checklist o The written examination outlines: Form 4.1-BWR, BWR Examination Outline, Form 4.1-ABWR, ABWR Examination Outline, Form 4.1-PWR, PWR Examination Outline, or Form 4.1-AP, AP-1000 Examination Outline and Form 4.1-COMMON, Common Examination Outline o Form 4.1-1, Record of Rejected Knowledge and Abilities o The operating test outlines: Form 3.2-1, Administrative Topics Outline, Form 3.2-2, Control Room/In-Plant Systems Outline, and Form 3.3-1, Scenario Outline o Form 2.3-1, Examination Outline Quality Checklist o The proposed NRC- or facility-developed operating tests and written examination (including comments made by the facility licensee or the NRC, as applicable) o Form 2.3-4, Written Examination Quality Checklist o Form 2.3-5, Written Examination Review Worksheet, and any reference handouts o Form 2.3-2, Operating Test Quality Checklist o Form 2.3-3, Operating Test Review Worksheets o The final written examination and answer key with all changes incorporated (the pen-and-ink corrections made for the applicants while the examination was administered may be changed to typewritten corrections; however, all changes shall be annotated in such a way that they are evident) o The as-administered operating examination (all record copies should reflect the as-run test conditions; pen-and-ink markups of the original, approved forms are acceptable) including:

Form 3.2-1, Administrative Topics Outline Form 3.2-2, Control Room/In-Plant Systems Outline JPMs for each walkthrough test Form 3.3-1, Scenario Outline for each scenario administered Form 3.3-2, Required Operator Actions for each scenario administered Form 3.4-1, Events and Evolutions Checklist 1 Attachment 2

OLMC-320 August 2022 o Form 3.3-1, Scenario Outline, and Form 3.3-2, Required Operator Actions, for the spare scenario, if the spare scenario or portions of it were not administered during the operating test o Form 5.1-1, Post examination Check Sheet o Form 5.1-3, Power Plant Examination Results Summary o The final Examination Report with all enclosures o The facility licensees and applicants specific comments and recommended changes regarding the operating and written examinations that were administered. Applicant names and any other personally identifiable information should be redacted in the publicly available version of this record.

YES NO

2. As documented in the examination report, no more than 30 days elapsed between the written examination and operating tests without obtaining NRR OL program office concurrence (ES-2.1 section D, step 15)

YES NO N/A

3. As documented in the examination report, the examiners audited approximately 10% of the license applications. (ES-2.2 section G)

YES NO

4. Quality of the submitted written examination and operating test, generic strengths and weaknesses, significant written examination grading deficiencies, examination security issues, and any examination delays or time extensions are appropriately documented in the examination report. (OLMC-510, Operator Licensing Examination Reports, section C.)

YES NO N/A B. Written Examination Review

1. The written examination outline does not overemphasize any systems, evolutions, or generic topics and the justifications for deselected or rejected knowledge and ability (K/A) statements are appropriate. (Form 2.3-1, Examination Outline Quality Checklist)

YES NO 2 Attachment 2

OLMC-320 August 2022

2. Perform an independent review on 10 - 20% of the written examination questions, including reactor operator / senior reactor operator questions, a sampling of bank, modified and new questions, and a sampling of questions identified as U, E, and S on Form 2.3-5, Written Examination Review Worksheet.

a) NRC examiners comments are reasonable and appropriate changes were made to address identified deficiencies. (ES-2.3 C.3.b. and D.3)

YES NO b) The final version of each sampled question is without latent flaws and meets requirements for level of difficulty, level of knowledge, K/A match, and license level.

References do not aid in distractor elimination or cue the correct answer. (ES-4.2 sections B and C)

YES NO c) Changes made to modified questions were sufficient to meet NUREG-1021 criteria for a significant modification of a bank question. (ES-4.2 B.4)

YES NO d) If more than four RO or two SRO questions were repeated from the last two NRC licensing examinations, including full retake examinations (100 questions), IOLB was informed. (Form 2.3-4, Written Examination Quality Checklist)

YES NO e) Bank use meets limits. (ES-4.2 B.4)

YES NO f) A technical reference and cross-reference to the facilitys examination bank is noted as applicable for each question. (ES-4.2 B.6)

YES NO g) The master copy of the examination was annotated to reflect changes made to questions during administration of the examination. (ES-4.4 B.6)

YES NO N/A 3 Attachment 2

OLMC-320 August 2022 h) Within 20 calendar days after examination administration, the facility submitted all examination administration or post examination review comments made by the facility licensee and the applicants (regarding the written examination) after the written examination and operating tests. The comments, as part of the post examination package, were concurred on by facility licensee management, through a cover letter or other form of correspondence. (ES-4.4 B.7)

YES NO N/A i) The facility comments were properly evaluated and resolved with the resolution and justification documented in the examination report and the master examination file revised accordingly. (ES-4.4 B.7-8, ES 4.4 C and OLMC-510, Operator Licensing Examination Reports),

YES NO N/A j) If two or more SRO and/or seven or more RO questions were deleted during the grading process, the regional office evaluated the remainder of the examination to ensure it satisfies the test outline sampling requirements in ES-4.1 and consults with the operator licensing program office if the validity of the examination is in question.

(ES-4.4 D.2)

YES NO N/A k) There was no opportunity for examiners to identify and correct psychometric flaws that led to post-administration question changes/deletions during examination development and review. (ES-2.3 C.3.b)

YES NO N/A C. Operating Test Review

1. The Regions (facilitys) draft operating test comments were reasonable and appropriately resolved in the final operating test. (ES-2.3 C.3.a)

YES NO N/A

2. Each applicant was evaluated on the required number of administrative topics and at the required license level (Instructions for Form 3.2-1, Administrative Topics Outline, step No. 1) 4 Attachment 2

OLMC-320 August 2022 YES NO

3. Based on the applicant's license level, each applicant was evaluated on the required number of systems from the safety function groupings identified in the applicable K/A Catalog (BWR or PWR) during the test: (Instructions for Form 3.2-2, Control Room/In-Plant Systems Outline, step No.1)

YES NO

4. Each control room system JPM selected for RO and instant SRO (SROI) applicants evaluates a different safety function (for PWRs, two systems, one primary and one secondary, may be selected from Safety Function 4). The same system is not used to evaluate more than one safety function in each location. The task for each JPM supports, either directly or indirectly and in a meaningful way, the successful fulfillment of the associated safety function. (Instructions for Form 3.2-2, Control Room/In-Plant Systems Outline, steps 2 and 3)

YES NO N/A

5. The five systems selected for an upgrade SRO applicant evaluate five different safety functions. The same system is not used to evaluate more than one safety function. The task for each JPM supports, either directly or indirectly and in a meaningful way, the successful fulfillment of the associated safety function. (Instructions for Form 3.2-2, Control Room/In-Plant Systems Outline, steps 2 and 3)

YES NO N/A

6. The criteria requirements listed in the instructions in step 3 on Form 3.2-1, Administrative Topics Outline, and step 4 on Form 3.2-2, Control Room/In-Plant Systems Outline, are within limits for each type of applicant. (Instructions for Forms 3.2-1 and 3.2-2)

YES NO

7. All JPMs, individually and as a group, have meaningful performance requirements that provide a basis for evaluating the applicants understanding of and ability to operate the plant. The level of difficulty of each JPM and the JPM set is acceptable. The JPMs do not have latent flaws that would impact the ability to evaluate the applicants competency. (ES-3.2: A.4, B.2, and D.1 a-d) 5 Attachment 2

OLMC-320 August 2022 YES NO

8. Applicants applying for a multiunit license at a multiunit facility that has a simulator modeled after only one of the units were tested on differences between the units during the in-plant JPM portion of the operating test. (ES-3.1 B.13.c) Mark this as not applicable if the on-site review was performed.

YES NO N/A

9. Alternate path tasks meet the criteria specified in ES-3.2, section E.

YES NO

10. The initial conditions, normal operations, malfunctions, and major transients are varied among the scenarios and periodically include startup, low power, and full power situations. (ES-3.3 A.2 and ES-3.4 section A)

YES NO

11. The scenario set exercises the applicants on the types and quantities of evolutions, failures, technical specification (TS) evaluations, and transients, identified for each applicant's license level, as described in Table 3.4-2, "Events and Evolutions for License Level," with each event only counted once per applicant. (ES-3.4 C.2)

YES NO

12. For each event listed on Form 3.3-1, Scenario Outline, Form 3.3-2, "Required Operator Actions," describes the expected communications, actions, and reference material to be used by each operating position on the crew. (ES-3.3 A.4 and ES-3.4 D.2)

YES NO

13. The scenario quantitative attributes are within the target bands established in Table 3.4-1, Target Quantitative Attributes per Scenario/Scenario Set. (ES-3.4 C.1)

YES NO 6 Attachment 2

OLMC-320 August 2022

14. The scenario set allowed for evaluation of each applicants performance on each competency and rating factor. (ES-3.4 section A)

YES NO

15. Each scenario and scenario set have meaningful performance requirements that provide a basis for evaluating the applicants competency in the control room and is of an appropriate level of difficulty. The scenarios do not have latent flaws that would impact the ability to evaluate the applicants competency. (ES-3.1 B.7)

YES NO

16. The walk-through and simulator tests are not redundant, nor do they duplicate material on the written examination. (ES-3.1 B.3)

YES NO

17. The final simulator scenarios were revised to reflect the as run conditions. (ES-3.6 B.7 and OLMC-520 C.2.o)

YES NO N/A

18. Within 20 calendar days after examination administration, the facility submitted all examination administration or post examination review comments made by the facility licensee and the applicants (regarding the operating test) after the written examination and operating tests. The comments, as part of the post examination package, were concurred on by facility licensee management, through a cover letter or other form of correspondence. (ES-3.6 A.1 and ES-4.4 B.7)

YES NO N/A

19. The facility comments were properly evaluated and resolved with the resolution and justification documented in the examination report and the master examination file revised accordingly. There was no opportunity for examiners to identify and correct psychometric flaws during operating test development and review that led to post-administration changes/deletions (ES-4.4 B.7 and OLMC-510, Operator Licensing Examination Reports)

YES NO N/A 7 Attachment 2

OLMC-320 August 2022

20. Using the examination schedule, mark each of the following. If an on-site review was performed during operating test administration, mark each of the following as N/A:
  • Confirm that the same JPMs and simulator scenarios are not repeated on subsequent days. (ES-3.1 B.2 and ES-3.5 A.9)

YES NO N/A

  • Simulator operating tests for each applicant were not divided among examiners.

(ES-3.5 A.3)

YES NO N/A

  • Every applicant received a complete set of JPMs. (ES-3.5 A.10)

YES NO N/A

  • No NRC examiner is scheduled to administer more than four complete simulator operating tests in any one week. (ES-2.1 C.3)

YES NO N/A D. Grading Activities

1. Each applicants original answer sheet was graded and annotated to indicate which questions were answered incorrectly, the correct answer and which questions (if any) were deleted. (ES-4.4 B.3)

YES NO

2. The results of an independent review of the written examination agrees with the regions results. (ES-4.4 C.3.i)

YES NO

3. Unsatisfactory JPM grading evaluations based solely on follow-up questions are documented in accordance with ES-3.6. (ES-3.6 B.2.b)

YES NO N/A 8 Attachment 2

OLMC-320 August 2022

4. Any material generated or used by the applicant that contributes to a failure of any portion of the operating test is cross-referenced to the applicable performance deficiency and attached to the examination results package. (ES-3.5 B.5 and ES-3.6 B.1 and B.8)

YES NO N/A

5. All of the examiners observations are consistent and mutually supportive. A performance deficiency shared by more than one applicant is appropriately documented for those applicants. Documentation for applicants on the same operating crew is consistent. Performance deficiencies that involved more than one applicant in an operating crew are noted by all involved evaluating examiners. (ES-3.6 A.3)

YES NO N/A

6. The validity and technical accuracy of any performance-based questions and any simulator test unexpected events or actions have been verified. (ES-3.6 B.1)

YES NO N/A

7. All required rating factors were evaluated for every applicant. If an SRO upgrade applicant performed a scenario in a position that operated plant controls, then they were graded on SRO Competency 3, Control Board Operations, even if they were not individually evaluated during that scenario. Rating factors that were not observed were limited and properly justified. (ES-3.5 G.16.a and ES-3.6 B.5 and Forms 3.6-5/6)

YES NO

8. Each documented simulator operating test performance deficiency is assigned to no more than two rating factors. (ES-3.6 B.4.b)

YES NO

9. Performance deficiencies, significant performance deficiencies and critical performance deficiencies were properly tabulated when determining rating factor scores and competency grades. (ES-3.6 B.4, B.5 and B.6)

YES NO 9 Attachment 2

OLMC-320 August 2022

10. Any performance deficiency contributing to a failure of one or more portions of the operating test is supported with detailed documentation. (ES-3.6 C.1)

YES NO N/A

11. Any deviation from the nominal grading criteria in ES-3.6 was explained in detail and approved in writing by the NRR OL program office. (ES-3.6 B.7)

YES NO N/A

12. An independent pass or fail recommendation is made by the chief examiner or designee and reviewed by the licensing official. (ES-5.1 sections A and B and Form 5.1-2, Individual Examination Report)

YES NO E. Comments and Notes on the Off-Site Records Review Section and Item #: (Provide comment or evaluation for the associated item number) 10 Attachment 2

OLMC-320 August 2022 MONTH DAY, 20XX MEMORANDUM TO: [FIRST M. LAST], Director Division of [Operating] Reactor Safety, Region X FROM: [FIRST M. LAST], Deputy Director Division of Reactor Oversight Office of Nuclear Reactor Regulation

SUBJECT:

[INSERT STATION NAME] INITIAL OPERATOR LICENSING EXAMINATION ASSESSMENT This memorandum refers to the audit conducted by [REVIEWER NAME] of my staff of the initial operator licensing examinations administered by Region [X] staff from [DATE RANGE], at the

[STATION NAME]. [The audit consisted of on-site observation of the administration of the operating test OR written examination, and a post-examination review of the written examination and other examination records.] OR [The audit consisted of a post-examination review of the written examination, operating test, and other examination records]

Include if the examination was determined to be of good quality - [In general, the examination was determined to be of good quality and in compliance with Revision [XX] of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors." I have enclosed some observations and feedback for your consideration. {Corrective actions to the noted observations shall be provided to the program office for review - Include if necessary}]

Include if the examination had deficiencies requiring regional response - [The audit results identify {multiple instances where the examination did not} {a significant failure to}{a repetitive failure to} meet the requirements of Revision [XX] of NUREG-1021, Operator Licensing Examination Standards for Power Reactors. The program office has determined that no evidence exists to suggest that the licensing decisions are invalid; however, these deficiencies must be corrected for future examinations. Corrective actions to the noted deficiencies shall be provided to the program office for review. These deficiencies and additional comments for consideration are enclosed.]

If you have any questions, please contact [FIRST LAST NAME], Chief, Operator Licensing and Human Factors Branch, at (301) 415-XXXX.

1 Attachment 3

OLMC-320 August 2022

Enclosures:

1. Written Examination Review Findings
2. Operating Test Review Findings CONTACT: [BC FIRST LAST NAME], NRR/DRO 301-415-XXXX 2 Attachment 3

OLMC-320 August 2022

SUBJECT:

[INSERT STATION NAME] INITIAL OPERATOR LICENSING EXAMINATION ASSESSMENT DISTRIBUTION:

{Distribution includes the regional branch chief and members of the examination team}

[FLast, RX FLast, RX FLast, RX FLast, RX]

Accession No.: MLXXXXXXXX OFFICE DRO/IOLB BC:DRO/IOLB DD: DRO NAME DATE OFFICIAL RECORD COPY

{Items in blue should be updated by the IOLB reviewer. Items in green should be updated/included by the IOLB reviewer as needed. Items in fuchsia should be deleted as these are included to assist in the template formatting and should not be part of the final document. Items in yellow are updated by the administrative assistant.}

3 Attachment 3

OLMC-320 August 2022 Written Examination Review Findings Deficiencies Requiring Corrective Actions Submitted to Program Office

{In this section put in any items that require corrective actions on the part of the regions be submitted to the program office. Include the applicable portion of the NUREG-1021 that was not met and the basis for requiring corrective action - i.e., significance, how widespread, and/or repetitive nature of the errors in accordance with Section D.2 of this OLMC. Discuss any impact on the examination quality and/or licensing decisions, if necessary. Recommend that this be a bulleted list with reference to the audit worksheet items. If there are no items, insert NONE below the title.}

Feedback for Consideration

{In this section put in any feedback for the regions consideration, including feedback regarding consistent and efficient examination administration. These are items that might or might not be in accordance with the NUREG, but whose impact is minor and while corrective actions may be performed by the region, it is not necessary for the program office to be informed of those actions. Recommend that this be a bulleted list, with reference to the applicable NUREG section if necessary. Note that this is also the location to provide any positive comments on the examination.}

1 Enclosure 1

OLMC-320 August 2022 Operating Test Review Findings Deficiencies Requiring Corrective Actions Submitted to Program Office

{In this section put in any items that require corrective actions on the part of the regions be submitted to the program office. Include the applicable portion of the NUREG-1021 that was not met and the basis for requiring corrective action - i.e., significance, how widespread, and/or repetitive nature of the errors in accordance with Section D.2 of this OLMC. Discuss any impact on the examination quality and/or licensing decisions, if necessary. Recommend that this be a bulleted list with reference to the audit worksheet items. If there are no items, insert NONE below the title.}

Feedback for Consideration

{In this section put in any feedback for the regions consideration, including feedback regarding consistent and efficient examination administration. These are items that might or might not be in accordance with the NUREG, but whose impact is minor and while corrective actions may be performed by the region, it is not necessary for the program office to be informed of those actions. Recommend that this be a bulleted list, with reference to the applicable NUREG section if necessary. Note that this is also the location to provide any positive comments on the examination.}

1 Enclosure 2

Retrieved from "https://kanterella.com/w/index.php?title=ML22216A198&oldid=3459552"