IP 71111.11, Licensed Operator Requalification Program and Licensed Operator Performance

Licensed Operator Requalification Program and Licensed Operator Performance

ML14217A409

Issue Date: 09/24/14 1 71111.11 Effective Date: 01/01/2015 ATTACHMENT 71111.11 INSPECTABLE AREA: Licensed Operator Requalification Program and Licensed Operator Performance CORNERSTONES: Initiating Events (10%) Mitigating Systems (70%) Barrier Integrity (10%) Emergency Preparedness (10%) INSPECTION BASES: This inspection evaluates licensed operator1 performance during the conduct of facility-administered requalification examinations required by 10 CFR 55.59, other examinations, facility training exercises, and during selected evolutions conducted in the actual plant/main control room. This inspection also reviews the ability of the facility licensee to identify and correct problems associated with licensed operator performance. Poor licensed operator performance results in an increase in risk due to: (1) an increase in human errors which cause initiating events, and (2) an increase in human errors for taking timely and correct mitigating actions after an event. Licensed operator errors can also impact barrier integrity and emergency preparedness. In order to utilize requalification examinations for measuring licensed operator performance, it is necessary that facility licensees properly develop and administer these examinations, such that they are effective tools for evaluating licensed operators. Therefore, this inspection evaluates the facility licensee’s ability to develop and administer these examinations. In addition, this inspection checks that licensed operators are properly maintaining their licenses by meeting requirements for training attendance, on-watch proficiency, and medical fitness. Finally, a portion of all licensed operator training and examination is conducted on the control room simulator, and in order for these activities to be conducted in a realistic fashion, it is necessary that the control room simulator properly model expected plant performance. Therefore, this inspection also evaluates control room simulator performance, and the facility licensee’s testing and maintenance of the control room simulator.

1 Throughout this inspection procedure, “licensed operator” is used to collectively refer to both licensed reactor operators and licensed senior reactor operators. Issue Date: 09/24/14 2 71111.11 Effective Date: 01/01/2015 LEVEL OF EFFORT: Quarterly Review by Resident Staff A review of (1) licensed operator performance during requalification testing and training, (2) the ability of the facility licensee to conduct requalification testing and training, and (3) simulator performance, will be conducted quarterly by the resident staff. In addition, licensed operator performance in the actual plant/main control room will be observed and assessed quarterly/annually2 by the resident staff during periods of heightened activity or risk. At the discretion of Regional Management, this review may also be performed by a regional operator licensing examiner. Annual Review by Regional Specialist A review of licensed operator performance, as determined from requalification examination results, will be conducted annually by a regional operator licensing examiner. This review is expected to be conducted in-office by contacting the facility licensee. Biennial Review by Regional Specialist A review of (1) licensed operator performance during requalification examinations, (2) the ability of the facility licensee to properly develop and administer requalification examinations, (3) the maintenance of individual operator licenses, (4) the performance of the control room simulator, and (5) the ability of the facility licensee to identify and resolve problems related to licensed operator performance, will be conducted biennially by an inspection team consisting of (1) a team leader who is a qualified operator licensing examiner on the facility licensee’s vendor type, who, at a minimum, is also qualified as basic certified inspector; and (2) one or more additional examiners/inspectors 3 . This review will also include the annual review of requalification examination results. The biennial review is expected to require a one-week on-site visit, plus additional in-office review. At the discretion of Regional Management, the biennial review may also include observation of licensed operator performance in the actual plant/main control room, provided there is an activity or evolution that occurs in the actual

2 The frequency of control room observations is explained in detail in Section 03.11.b 3 It is the expectation that the inspection team leader be an examiner who is a fully qualified inspector. In rare cases the inspection team leader may be an examiner qualified as a basic certified inspector, as approved by the Region’s Operations Branch Chief. For the second member of the team, it is preferred that this individual be an examiner qualified on the facility’s vendor type, with or without inspector qualifications. Additional examiners/inspectors may be used at the Region’s discretion. Issue Date: 09/24/14 3 71111.11 Effective Date: 01/01/2015 plant/main control room that is suitable to observe during the biennial inspection week. If such an observation is performed during the biennial review, the inspection effort should be charged to Inspection Procedure (IP) 71111.11 Quarterly (Q).

71111.11-01 INSPECTION OBJECTIVES

01.01 To evaluate licensed operator performance during the conduct of requalification examinations, other examinations, training exercises, and in the actual plant/main control room. 01.02 To assess the facility licensee’s ability to evaluate the performance of their licensed operators during the conduct of requalification examinations, other examinations, and training exercises. 01.03 To assess the facility licensee’s ability to properly develop and administer requalification annual operating tests and biennial written examinations. 01.04 To assess the performance of the control room simulator, and the facility licensee’s testing and maintenance of the control room simulator. 01.05 To assess the facility licensee's effectiveness in ensuring that the individuals who are licensed to operate the facility satisfy the conditions of their licenses as specified in 10 CFR 55.53 and 10 CFR 55.59. 01.06 To assess the facility licensee’s effectiveness in identifying and resolving problems related to licensed operator performance. All of the above objectives will serve to verify that that the facility licensee's requalification program for licensed operators is effective in ensuring safe power plant operation.

71111.11-02 INSPECTION REQUIREMENTS

02.01 Inspection Composition. For biennial reviews, the inspection will include the inspection team specified above in the Level of Effort section, where, at a minimum, two examiners/inspectors will be utilized during the facility licensee’s administration of the requalification annual operating test. For the annual review of requalification examination results, one operator licensing examiner should be utilized. For quarterly reviews, one or more of the resident staff should be utilized, using the applicable portions of this procedure. 02.02 Requalification Examination Results. Following the completion of each requalification annual operating test and each biennial written examination, review licensed operator examination failure rates. Examination failures indicate that one or more licensed operators do not possess the necessary knowledge and abilities to pass requalification examinations required by 10 CFR 55.59(a)(2), and a corresponding increase in plant risk associated with poor operator performance. Additionally, examination failures may indicate deficiencies associated with the quality of the requalification training received, as specified in 10 CFR 55.59(c) and in accordance with a systems approach to training (SAT) as defined in 10 CFR 55.4. 02.03 Biennial Requalification Written Examination Quality. Assess the adequacy of the facility licensee’s biennial written examination. This section checks that the facility licensee has developed written examinations consistent with the expected quality of requalification examinations required by 10 CFR 55.59(a)(2). Quality written examinations are also necessary to ensure that the facility licensee is effectively evaluating their licensed operators for mastery of training objectives, as specified in Element (4) of a SAT as defined in 10 CFR 55.4. 02.04 Annual Requalification Operating Test Quality. Assess the adequacy of the facility licensee’s annual requalification operating test. This section checks that the facility licensee has developed operating tests consistent with the expected quality of requalification examinations required by 10 CFR 55.59(a)(2). Quality operating tests are also necessary to ensure that the facility licensee is effectively evaluating their licensed operators for mastery of training objectives, as specified in Element (4) of a SAT.

02.05 Licensee Administration of an Annual Requalification Operating Test. Observe an annual requalification operating test that is in progress and assess: (1) licensed operator performance, (2) the facility licensee’s ability to administer the annual operating test, (3) the facility licensee’s ability to evaluate the performance of their licensed operators, and (4) simulator performance. This section checks the effectiveness of the facility licensee in administering requalification operating tests required by 10 CFR 55.59(a)(2) and that the facility licensee is effectively evaluating their licensed operators for mastery of training objectives, as specified in Element (4) of a SAT. Additionally, this section checks for compliance with the simulator performance requirements contained in 10 CFR 55.46. 02.06 Requalification Examination Security. Observe an annual requalification operating test that is in progress and assess the ability of the facility licensee to safeguard examination material, such that the examination is not compromised. Review examinations used during a requalification test cycle for excessive test item repetition. This section checks that the facility licensee effectively implements proper controls to protect the integrity of requalification examinations, in accordance with 10 CFR 55.49. This section also checks for incidents of examination compromise, as defined in 10 CFR 55.49. 02.07 Licensee Remedial Training and Re-examinations. For licensed operators who do not pass an NRC-required requalification examination, assess the effectiveness of remedial training conducted by the facility licensee, and review the adequacy of re-examinations. This section checks that re-examinations meet expected quality standards for requalification examinations required by 10 CFR 55.59(a)(2), and checks the effectiveness of any remedial training, which is specified in 10 CFR 55.59(b). Effective remediation and quality re-examinations are also associated with Elements (4) and (5) of a SAT. 02.08 Conformance with Operator License Conditions. Review licensed operator records to assess compliance with 10 CFR 55.53 and 10 CFR 55.59 regarding training attendance, maintaining an active license, and medical fitness. Assess the facility licensee’s program for ensuring that licensed operators meet the conditions of their licenses.

02.09 Simulator Performance. Assess the adequacy of the facility licensee’s control room simulator in modeling the actual plant, and for meeting the requirements contained in 10 CFR 55.46. Assess the facility licensee’s periodic testing of the simulator, installation of simulator modifications, and for identifying and correcting simulator discrepancies. 02.10 Problem Identification and Resolution. Assess the facility licensee’s ability to identify, evaluate, and resolve licensed operator performance problems which occur in the actual plant/main control room. As part of this assessment, this section checks the effectiveness of the licensee in evaluating and revising requalification training based on licensed operator performance, as specified in Element (5) of a SAT. 02.11 Resident Inspector Quarterly Review of Licensed Operator Requalification and Licensed Operator Performance. At least once each quarter, observe the facility licensee’s requalification testing and training for licensed operators, identify any deficiencies and discrepancies, assess licensed operator performance and evaluators critique, and assess simulator performance. In addition, observe licensed operator performance in the actual plant/main control room during periods of heightened activity or risk.

71111.11-03 INSPECTION GUIDANCE

General Guidance As demonstrated by the accidents at Three Mile Island and Chernobyl, other plant events, and from probabilistic risk analyses, licensed operator performance can be a significant contributor to overall plant risk. For this IP, licensed operator performance in the actual plant/main control room will be observed and assessed during periods of heightened plant activity or risk, and inplant licensed operator performance will be inferred from their performance on facility-conducted requalification examinations, other examinations, and facility training exercises. Regarding requalification examinations, each licensed operator at a facility must, in accordance with 10 CFR 55.59(a), pass a comprehensive (biennial) written examination and an annual operating test4 . However, in order for the requalification examinations to be an effective tool for evaluating licensed operator performance, it is necessary that facility licensees develop and administer these examinations in accordance with accepted standards5 .

4 The periodicity of these examinations has been interpreted by the NRC as: annual operating tests are to be administered at least once in every calendar year, and the comprehensive written examination is to be administered at least once within each 24-month requalification program preferably near the end of each program. 5 Industry or self-imposed standards for requalification examinations can be found, as applicable, in facility licensee procedures, industry accreditation guidelines, NUREG-1021, and 10 CFR 55. The standard(s) applicable to a particular facility licensee do not necessarily constitute regulatory requirements; however, failure to meet a standard may establish the basis for a performance deficiency. The Appendices of this IP, where performance deficiencies are identified based upon not meeting certain metrics, were developed from these standards. Issue Date: 09/24/14 6 71111.11 Effective Date: 01/01/2015 Each region is responsible for knowing their facilities’ requalification examination schedules, and the biennial inspection should be scheduled through and announced to each facility licensee. Typically, facility licensees divide up each 24-month requalification program into a number of training cycles. During each requalification training cycle, training and/or examination activities are repeated on subsequent weeks for different groups or operating crews of licensed operators. Typically, facility licensees will conduct an annual operating test for licensed operators during a training cycle near the mid-point of each 24-month requalification program, and conduct both an annual operating test and a biennial written examination during a training cycle near the end of each 24-month requalification program. With such a schedule, it is preferred that the IP be performed in the manner presented in the following table. TABLE 03-1 PREFERRED INSPECTION SCHEDULE (1) For the training cycle which contains only an annual requalification operating test, a regional operator licensing examiner will review the examination results upon completion of this training cycle, and complete inspection requirement 02.02. The resident inspector staff will observe a portion of the annual requalification operating test while it is in-progress. (2) For the training cycle which contains both an annual requalification operating test and a biennial requalification written examination, the region will complete inspection requirements 02.02 through 02.10, utilizing an operator licensing examiner plus one or more additional examiners/inspectors. For facility licensees that do not employ this typical schedule, or for facilities in which the NRC has historically completed this IP in a different way, the overall requirements are for the region to: (1) Complete inspection requirement 02.02 each time a facility licensee completes a NRCrequired annual requalification operating test or a biennial written examination. The resident inspector staff will observe a portion of the annual requalification operating test not observed by the NRC during the biennial inspection. (2) Complete inspection requirements 02.03 through 02.10 biennially, utilizing the inspection team specified in the Level of Effort section for biennial reviews. For the biennial inspection, it is expected that the inspection team can complete inspection requirements 02.02 through 02.10 with a combination of a one-week on-site visit and additional inspection activities conducted off-site, such as reviewing examination results and examination quality. In particular, a minimum of two examiners/inspectors shall be on-site to observe the facility licensee’s administration of the annual operating test to a minimum of one group or operating crew of licensed operators, where observing an actual operating crew is preferred. As an efficiency measure, it is recommended that the NRC request that the facility licensee submit specific examinations, and other information prior to the on-site portion of the biennial inspection, such that inspection requirements 02.03, 02.04, and portions of inspection requirements 02.06 and 02.10, can be completed and discussed with the facility licensee while on-site. In accordance with 10 CFR 55.59(c), facility licensees are required to make these Issue Date: 09/24/14 7 71111.11 Effective Date: 01/01/2015 examination records available for NRC review, and these records should be requested in advance of the on-site inspection. A list of the typical documents reviewed during this IP is presented in Appendix A. Generally, only the inspection requirements of this procedure will need to be conducted. However, regional managers will consider overall facility performance, allegations related to licensed operator requalification, findings from this IP, and any traditional enforcement actions taken as a result of this IP in determining whether additional activities should be performed. Additional activities include observation of additional groups or crews of licensed operators during an annual requalification operating test, the performance of IP 41500, “Training and Qualification Effectiveness,” and the performance of an NRC-conducted licensed operator requalification examination, in accordance with 10 CFR 55.59(a)(2)(iii) and NUREG-1021, “Operator Licensing Examination Standards for Power Reactors,” ES-600 sections. Additional activities should be considered when any of the following conditions exist: Observe Additional Crews. At the discretion of the NRC Region, this activity could be performed if a dynamic simulator scenario crew failure is observed during the regularly scheduled inspection, or as a possible response to allegations associated with licensed operator performance or requalification training. Perform IP 41500. This activity should be performed if the NRC is concerned with the quality of licensed operator requalification training at a facility. Initiators for this activity include: (1) significant in-plant operator performance issues that have requalification training quality as a root cause, (2) a failure rate on a requalification examination of greater than or equal to 50 percent6 , (3) indications of a breakdown in the systems approach to training at the facility, or (4) as a possible response to allegations associated with licensed operator performance or requalification. Prior to initiating this activity, consult with the operator licensing program office and obtain concurrence from the affected Region’s Regional Administrator. Perform an NRC-conducted licensed operator requalification examination. This activity should be performed if the NRC has lost confidence in the facility licensee’s ability to conduct its own examinations. Initiators for this activity include: (1) white findings in both written examination quality (02.03) and operating test quality (02.04); (2) in response to an actual examination compromise as defined in Manual Chapter 0609, Appendix I, “Licensed Operator Requalification Significance Determination Process;” or (3) as a possible response to allegations associated with licensed operator requalification examinations. Prior to initiating this activity, consult with the operator licensing program office and obtain concurrence from the affected Region’s Regional Administrator. Most issues that meet the threshold as defined in this IP and in IMC 0612, Appendix B, for assessment using the significance determination process (SDP) will relate to mitigating activities and should be assigned to the Mitigating Systems cornerstone. Should the finding clearly relate to the breech of a barrier, it should be assigned to the Barrier Integrity cornerstone. Should the

6 To calculate the requalification examination failure rate, take the number of licensed operators who failed any portion of the examination that was administered (e.g., written, job performance measures, simulator scenarios) and divide by the total number of licensed operators who were administered the examination. Issue Date: 09/24/14 8 71111.11 Effective Date: 01/01/2015 finding clearly relate to an error by the operator that would initiate an event had it been on the actual plant, it should be assigned to the Initiating Events cornerstone. In all cases, the inspector should provide a rationale for the cornerstone assignment. Specific Guidance

03.01 Inspection Composition. Refer to paragraph 02.01.

03.02 Requalification Examination Results. This inspection activity should be performed by a regional operator licensing examiner who is aware of the facility licensee’s schedule for conducting annual requalification operating tests and biennial requalification written examinations. After the facility licensee has completed any training cycle which contains an NRC-required licensed operator requalification annual operating test or biennial written examination, and after the facility licensee has graded those examinations, contact the facility licensee and: a. Determine the composition of the examinations administered, i.e., written examinations, simulator scenarios, job performance measures (JPMs). b. Collect the examination results in order to complete the following table: TABLE 03.02-1 EXAMINATION RESULTS 1. Total number of licensed operators. 2. Number of licensed operators administered a requalification examination required by 10 CFR 55.59(a). 3. Number of individual licensed operators who failed any portion of a requalification examination (written, JPM, or individual simulator scenario failures). 4. Divide line 3 by line 2 to obtain the individual requalification examination failure rate. Line 3/Line 2. % 5. Number of crews administered simulator scenarios as part of a requalification examination required by 10 CFR 55.59(a). 6. Number of crews who performed unsatisfactorily on the simulator scenarios. 7. Divide line 6 by line 5 to obtain the crew simulator scenario failure rate. Line 6/Line 5. % c. If any individual licensed operator failed any portion of a requalification examination, as indicated in line 3 above, or any crew failed the simulator scenarios, as indicated in line 6 above, determine from the facility licensee whether remediation and re-examinations occurred, and that the re-examinations were passed. If the inspector identifies any concerns with re-examinations which could impact whether or not licensed operators Issue Date: 09/24/14 9 71111.11 Effective Date: 01/01/2015 were fulfilling a condition of their license (i.e., 10 CFR 55.53(h) via 10 CFR 55.59(a)(2) requires each licensed operator to pass a comprehensive written examination and an annual operating test) refer to section 03.08 of this IP, “Conformance with Operator License Conditions.” d. If any examination failure rate (individual or crew) exceeds 20% as identified above (line 4 or line 7), then this shall be considered a performance deficiency against the expected knowledge and abilities of licensed operators as demonstrated during the requalification examinations required by 10 CFR 55.59(a)(2), and as a possible finding (typically without a regulatory violation). Consult IMC 0612, Power Reactor Inspection Reports, for additional information regarding the screening of inspection results. e. Obtain from the region’s Operator Licensing Assistant the total number of licensed operators at the site. Using this number, discuss with the facility licensee any differences in the total number of licensed operators at the site with the total number of licensed operators who took a requalification examination. Discuss with the facility licensee any licensed operator who did not take a complete examination, and any plans the licensee has for making-up any missed examinations. Determine whether each licensed operator is or is not taking the required requalification examinations, as set forth in 10 CFR 55.59(a)(2). f. Discuss with the facility licensee when the next requalification examinations will be administered. This will allow the region to plan for future inspections, and when the next requalification examination results will be reviewed. 03.03 Biennial Requalification Written Examination Quality. Typically, facility licensees develop multiple similar versions7 of the biennial written examination, with different versions administered on subsequent weeks for different groups or operating crews of licensed operators. To complete this inspection activity, a minimum of one complete version of the requalification biennial written examination shall be reviewed in detail, using the checklists and guidance provided in Appendix B. If greater than 20% of the written examination questions reviewed are determined to be flawed, then a second complete version of the written examination from the same training cycle shall be reviewed. If this IP is conducted in accordance with Table 03-1, “Preferred Inspection Schedule” then the biennial requalification written examination reviewed will be an examination from the licensee’s current or upcoming training cycle in which the examination has been or will be administered.8

If this IP is conducted using an alternate schedule, then the biennial requalification written examination reviewed should be from an examination which occurred since the last NRC biennial review, such that this IP reviews at least one version of the biennial requalification written examination from each of the licensee’s successive 24-month requalification programs.

7 Multiple versions of the biennial requalification written examination are necessary to prevent any potential examination compromise, due to the different dates on which the examination is administered. See the inspection guidance contained in section 03.06. 8 When reviewing an exam yet to be administered, inspectors shall be careful to protect the examination from inadvertent disclosure to unauthorized facility personnel, and adhere to the facility licensee’s examination security procedures and policies. Issue Date: 09/24/14 10 71111.11 Effective Date: 01/01/2015 It is also recommended that this review be completed prior to the on-site portion of this inspection, such that any observations regarding written examination quality can be discussed with the facility licensee while on-site. Upon the completion of this review, any written examination quality issue shall be discussed with the facility licensee, to assist in confirming the issue. However, the inspector should not interfere with the facility licensee’s requalification examination process by suggesting modifications to test items or examination schedules. If there are significant concerns with the quality of the written examinations reviewed, contact regional management as soon as possible. In evaluating any written examination quality issues, refer to Appendix B and IMC 0612, Power Reactor Inspection Reports. 03.04 Annual Requalification Operating Test Quality. Typically, facility licensees develop multiple similar versions of the annual requalification operating test, with different versions administered on subsequent weeks for different groups or operating crews of licensed operators. To complete this inspection activity, a minimum of ten (10) job performance measures (JPMs) and four (4) simulator scenarios associated with an annual requalification operating test required by 10 CFR 55.59(a)(2) shall be reviewed in detail, using the checklists and guidance provided in Appendix C. If greater than 20% of the JPMs are determined to be flawed, then an additional ten (10) JPMs from the same training cycle shall be reviewed. If greater than 20% of the simulator scenario events are determined to be flawed, then an additional four (4) scenarios from the same training cycle shall be reviewed.

Since it is expected that this inspection activity will be conducted along with the on-site observation of the annual requalification operating test (inspection requirement 02.05), the JPMs and scenarios reviewed should include those that will be observed during the on-site portion of this inspection, with any remaining JPMs and scenarios reviewed selected from other weeks within the same training cycle.9

It is recommended that these reviews be completed prior

to the on-site portion of this inspection to familiarize the inspectors with the operating test that will be observed, and to facilitate discussing with the facility licensee any observations made regarding examination quality while on-site. In addition, during this review for operating test quality and as an efficiency measure, the inspector should check for excessive test item repetition between operating tests administered during different weeks within a training cycle (see section 03.06, Requalification Examination Security). Upon the completion of this review, any operating test quality issue shall be discussed with the facility licensee, to assist in confirming the issue. However, the inspector should not interfere with the facility licensee’s requalification examination process by suggesting modifications to test items or examination schedules. If there are significant concerns with the quality of the operating tests reviewed, contact regional management as soon as possible. In evaluating any operating test quality issues, refer to Appendix C and IMC 0612, Power Reactor Inspection Reports.

9 When reviewing any operating test items yet to be administered, inspectors shall be careful to protect the test items from inadvertent disclosure to unauthorized facility personnel, and adhere to the facility licensee’s examination security procedures and policies. Issue Date: 09/24/14 11 71111.11 Effective Date: 01/01/2015 03.05 Licensee Administration of an Annual Requalification Operating Test. Observing an inprogress annual requalification operating test is the most performance based activity in this IP. As such, a minimum of two examiners/inspectors as specified in the Level of Effort section for biennial reviews will be utilized to maximize the amount of the annual requalification operating test that is observed. This inspection activity should be conducted during a one-week on-site visit, although if desired, the region may re-visit the site and observe the annual requalification operating test for a different group or crew of licensed operators during a subsequent week in the training cycle. This inspection activity should be performed by having the examiner and the examiner/inspector observe the simulator scenarios together as a team, and by having the examiner and the examiner/inspector each individually observe different licensed operators and facility evaluators during the conduct of JPMs. One option for completing these activities would be to assign the examiner to observe a complete annual requalification examination for one licensed operator, and to assign the examiner/inspector to observe a complete annual requalification examination for another licensed operator, assuming these two licensed operators are part of the same crew, such that the examiner and the examiner/inspector can observe the simulator scenarios together as a team. The basic methodology for performing this inspection activity is to observe licensed operator performance during an annual requalification operating test, in parallel but not to interfere with, the facility licensee’s personnel who are administering the examination, and to observe postsimulator scenario critiques and other facility licensee operating test grading activities. It is important to note that during this inspection activity it is expected that multiple items will be assessed, often simultaneously, including: a. Licensed operator performance, including:  Crew performance in terms of clarity and formality of communication  Ability to take timely action in the safe direction  Prioritizing, interpreting, and verifying alarms  Correct use and implementation of procedures, including the alarm response procedures  Timely control board operation and manipulation, including high-risk operator actions  Oversight and direction provided by the shift supervisor, including ability to identify and implement appropriate technical specifications actions such as reporting and emergency plan actions and notifications  Group dynamics involved in crew performance b. The facility licensee’s ability to administer the annual requalification operating test. c. The facility licensee’s ability to assess the performance of their licensed operators. d. The adequacy of plant procedures. e. The quality of the annual requalification operating test scenario guides and JPMs (see section 03.04). Issue Date: 09/24/14 12 71111.11 Effective Date: 01/01/2015 f. Examination security (see section 03.06). g. Simulator performance (see section 03.09). Refer to Appendix D for a checklist of items to assess during this inspection activity. During this inspection activity and upon its conclusion, any operating test administration issue shall be discussed with the facility licensee, to assist in confirming the issue. However, the inspector should not interfere with the facility licensee’s requalification examination process. If there are significant concerns with the facility licensee’s administration of the operating test, contact regional management as soon as possible. In evaluating any operating test administration issues, refer to Appendix D and IMC 0612, Power Reactor Inspection Reports. 03.06 Requalification Examination Security. Examination security encompasses all practices taken by the facility licensee to ensure compliance with 10 CFR 55.49, which states, in part: “Applicants, licensees, and facility licensees shall not engage in any activity that compromises the integrity of any application, test, or examination required by this part. The integrity of a test or examination is considered compromised if any activity, regardless of intent, affected, or, but for detection, would have affected the equitable and consistent administration of the test or examination.” For this IP, the inspection activity for requalification examination security focuses on facility licensee practices to prevent examinees from gaining any specific knowledge of requalification examination content prior to taking the requalification examination. This inspection activity reviews requalification examination security using three basic methods: (1) checking for excessive test item repetition between requalification examinations administered during different weeks of a training cycle, (2) observation of the facility licensee’s implementation of examination security during their administration of an annual requalification operating test, and (3) determining through the facility licensee if any licensed operator has been pre-exposed to examination information via validation, exam development, or other activities. When checking for excessive test item repetition, the inspector should discuss this with the facility licensee, who should have spreadsheets and/or test outlines which show the written examination questions, JPMs, and scenarios to be used during the entire requalification examination testing cycle. In addition to these reviews, the inspector shall also discuss with the facility licensee any requalification examination security incidents which have occurred since the last biennial inspection. Refer to Appendix E for a checklist of items to assess during this inspection activity. During this inspection activity and upon its conclusion, any examination security issue shall be discussed with the facility licensee, to assist in confirming the issue. However, the inspector should not interfere with the facility licensee’s requalification examination process. If there are significant concerns regarding examination security, contact regional management as soon as possible. In evaluating any examination security issues, refer to Appendix E and IMC 0612, Power Reactor Inspection Reports. Issue Date: 09/24/14 13 71111.11 Effective Date: 01/01/2015 03.07 Licensee Remedial Training and Re-examinations. This inspection activity includes the following: a. Since the last biennial inspection, determine from the facility licensee if any individual licensed operators or crews failed any portion (written, JPM, or simulator scenario examination) of an NRC requalification examination required by 10 CFR 55.59(a)(2), and determine what type of failure(s) occurred - individual failures or crew failures; written, JPM or simulator scenario examination failures. b. For each examination failure identified above, determine if the associated reexamination administered was commensurate with the original failure. c. For each re-examination identified above, check for duplication of test items from the examination that was originally failed, and check for excessive duplication from other examinations administered during the training cycle when the original examination was failed. Although it is appropriate for re-examinations to test material that was failed on the original examination, re-examinations shall not contain any test items which exactly duplicate test items from the original examination. Additionally, as an examination security measure, re-examinations shall not duplicate more than 50 percent of test items previously administered during the training cycle when the original examination was failed. Refer to Appendix F for additional guidance and a checklist for conducting this inspection activity. Upon completion of this review, any remedial training and re-examination issue shall be discussed with the facility licensee, to assist in confirming the issue. If there are significant concerns with the licensee’s performance in this area, contact regional management as soon as possible. In evaluating any remedial training and/or re-examination issues, refer to Appendix F and IMC 0612, Power Reactor Inspection Reports. 03.08 Conformance with Operator License Conditions. Operator license conditions are contained in 10 CFR 55.53 and 10 CFR 55.59, and include requalification training attendance, maintaining an active license, and medical fitness. This inspection activity should be performed primarily during the on-site portion of this inspection. However, prior to the on-site review, the inspector should (1) obtain Operator Licensing Tracking System (OLTS) report number 14 (available from the Region’s Operator Licensing Assistant), which lists all licensed operators at the site and their associated medical restrictions, if any; (2) preselect licensed operators for review of their medical records, (refer to paragraph d below); and (3) review the selected licensed operators’ medical information contained in their 10 CFR Part 55 individual docket files (also available from the Operator Licensing Assistant). Additional guidance for performing this inspection activity is presented in NUREG-1021, “Operator Licensing Examination Standards for Power Reactors,” Section ES-605. a. For one complete operating crew of licensed operators10, review the following:

10 For multi-unit sites, review records for all licensed operators assigned to a shift for all of the units. Issue Date: 09/24/14 14 71111.11 Effective Date: 01/01/2015 1. Records that indicate the participation of licensed operators in the facility licensee’s requalification program (i.e., training attendance records) (10 CFR 55.53(h), 10 CFR 55.59(a)(1), 10 CFR 55.59(c)(5)(i)). Determine if all requalification training is completed on schedule or made up in accordance with the facility's program. 2. Records that indicate the performance of licensed operators on annual requalification operating tests and biennial requalification written examinations (10 CFR 55.59(a)(2), 10 CFR 55.59(c)(5)(i)). 3. Records that indicate that licensed operators are properly maintaining an active license (10 CFR 55.53(e)). Include in this activity a determination of which shift crew positions the facility licensee credits towards maintaining an active license. 4. NRC Form 398, “Personal Qualification Statement – Licensee,” that is in effect at the time of the inspection and any supporting documentation, that indicate operators’ licenses were properly issued or renewed. b. Determine from the facility licensee if any licensed operator with an inactive license reactivated their license since the last NRC biennial inspection. If license reactivation did occur, select one or more licensed operators who reactivated their license(s), and review records which indicate that their license(s) was (were) properly reactivated in accordance with 10 CFR 55.53(f). c. Determine from the facility licensee if any licensed operator did not pass a biennial requalification written examination or annual requalification operating test required by 10 CFR 55.59(a)(2) since the last NRC biennial inspection. If examination failure did occur, select one or more licensed operators who failed an examination, and review records which indicate that the operator(s): (1) received remedial training, (2) was (were) re-examined, and (3) passed their re-examination(s). d. Review a sample (approximately 10%) of licensed operator medical records, including any medical records maintained in the regional office’s Part 55 docket system. Approximately 50% of this sample, if possible, should include licensed operators whose licenses contain medical restrictions, as determined by the facility licensee and OLTS report number 14. In performing this review, check that: 1. The required medical examinations are being conducted biennially (10 CFR 55.53(i)). 2. The results of medical examinations are in agreement with any license medical restrictions (10 CFR 55.23(b)). 3. NRC Form 396, “Certification of Medical Examination by Facility Licensee,” that is in effect at the time of the inspection, is accurate and complete, including any supporting medical documentation. Issue Date: 09/24/14 15 71111.11 Effective Date: 01/01/2015 4. Any medical status reports, if required by block 6 of NRC Form 396, are submitted to the regional office in a timely fashion. In performing this review, determine which version of Regulatory Guide 1.134, “Medical Evaluation of Licensed Personnel at Nuclear Power Plants,” and the associated version of ANSI/ANS-3.4, “Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants,” that the facility licensee has committed to. During this inspection activity and upon its conclusion, any issue associated with the conformance with operator license conditions shall be discussed with the facility licensee, to assist in confirming the issue. If it is determined that any licensed operator has not properly conformed with the conditions of their license, perform, as applicable, the following supplemental actions: 1. Discuss with the facility licensee any immediate actions to take, such as removing the affected licensed operator(s) from on-shift licensed operator duties, and notifying the regional office. 2. Discuss with the facility licensee any plans for restoring compliance with operator license conditions. Assess all issues associated with non-conformance with operator license conditions using IMC 0612, Power Reactor Inspection Reports. Typically, these issues may result in a violation of one or more NRC requirements (refer to 10 CFR 55.53, 10 CFR 55.9, 10 CFR 50.9, and 10 CFR 50.74), and often times these issues impact the regulatory process (e.g., a licensed operator performing licensed duties while in noncompliance with his or her license, or had the NRC known that the operator’s license had not been complied with, the NRC would have conditioned the license in some way or not renewed the license). Therefore, for these issues, traditional enforcement typically applies and the current Enforcement Policy, Section 6.4, “Licensed Reactor Operators,” has some examples. 03.09 Simulator Performance. This inspection activity should be performed during the on-site portion of this inspection, and consists of the following elements: a. Observe simulator modeling and performance during an in-progress annual operating test, and note any simulator modeling or performance deficiencies. b. Review a listing of open simulator deficiencies as maintained by the facility licensee in their simulator corrective action program. Sample the facility licensee’s proposed corrective actions for any open simulator deficiencies. c. Review a listing of simulator deficiencies closed by the facility licensee since the last biennial inspection. Sample the corrective actions taken by the facility licensee in closing simulator deficiencies. d. Review a sample of records associated with any simulator testing performed by the facility licensee since the last biennial inspection. Issue Date: 09/24/14 16 71111.11 Effective Date: 01/01/2015 e. Review a sample of records associated with any simulator modifications made by the facility licensee since the last biennial inspection. In performing this inspection activity, it will be necessary to determine which version of Regulatory Guide 1.149, “Nuclear Power Plant Simulation Facilities for Use in Operator [Training and] License Examinations,” and the associated version of ANSI/ANS-3.5, “Nuclear Power Plant Simulators for Use in Operator Training [and Examination],” that the facility licensee is committed to, and whether the facility licensee is using the simulator to meet the control manipulation eligibility requirements contained in 10 CFR 55.31(a)(5). In performing this inspection activity, the inspector should answer the following three questions to his or her satisfaction: 1. Is simulator modeling and performance satisfactory? 2. Does the facility licensee effectively correct identified simulator deficiencies? 3. Does the facility licensee properly perform required simulator testing? In answering this question, the inspector should check for the correct simulator testing periodicity, proper evaluations of the simulator against the reference plant or best-estimate data, and the correct documentation and retention of simulator test records. 10 CFR 55.46 provides additional details towards answering these questions, and a checklist for 10 CFR 55.46 is presented in Appendix G. Additional details are also available in the applicable versions of Regulatory Guide 1.149 and ANSI/ANS-3.5. During this inspection activity and upon its conclusion, any simulator performance issue shall be discussed with the facility licensee, to assist in confirming the issue. If there are significant concerns regarding simulator performance, contact regional management as soon as possible. In evaluating any simulator performance issues, refer to Appendix G, 10 CFR 55.46, the applicable versions of Regulatory Guide 1.149 and ANSI/ANS-3.5, and IMC 0612, Power Reactor Inspection Reports. 03.10 Problem Identification and Resolution (PI&R). This inspection activity evaluates the ability of the facility licensee to identify, evaluate, and resolve problems associated with licensed operator performance, for problems in performance which occur in the actual plant/main control room. The intent of this inspection activity is not to duplicate efforts associated with IP 71152, “Problem Identification and Resolution,” and other completed NRC inspections, but to add a specific focus to licensed operator errors and/or other licensed operator performance problems which have occurred in the actual plant/main control room, and how these errors or problems were identified and resolved by the facility licensee. The guidance provided in this IP section is similar, although abbreviated and focused specifically on licensed operator performance, to that provided in IP 71152 sections 03.04 through 03.07 for performing and documenting the biennial PI&R team inspection. For additional guidance, please refer to IP 71152. Issue Date: 09/24/14 17 71111.11 Effective Date: 01/01/2015 a. Review records which document licensed operator performance issues. Similar to sections 03.04 through 03.05 of IP 71152, the inspector should obtain and review records which will provide a source for sampling. Records to review include:  The most recently completed NRC biennial PI&R team inspection report.  NRC inspection report findings, including cited and non-cited violations.  Previously completed NRC End of Cycle and Mid Cycle reports.  The NRC-maintained Plant Issues Matrix.  Licensee-identified issues, such as issues from licensee-conducted audits, assessments, and issues documented in condition/problem identification reports (including issues identified through industry operating experience). In addition, the inspector should discuss with the resident inspection staff their impressions of licensed operator performance as observed in the actual plant/main control room. When performing the above:  Review records since the last biennial licensed operator requalification inspection was performed.  Screen issues for sampling based upon those issues associated with licensed operator errors or other licensed operator performance problems that have occurred in the actual plant/main control room. For licensee-identified issues (e.g., condition/problem identification reports, industry operating experience, audit and assessment findings) the licensee should be able to provide a prescreened list of issues associated with licensed operator or human performance errors or performance problems. As an efficiency measure, these reviews should be conducted prior to the on-site portion of this inspection. b. Conduct an in-depth review of a sampling of licensee corrective actions and requalification training records associated with licensed operator errors or other licensed operator performance problems. Since the number and type of these issues may vary from plant to plant, no minimum number of samples is specified. However, the following guidance is provided:  The sample for in-depth review should include those licensed operator errors or performance problems considered as most significant by the inspector. When assessing significance, the inspector should consider the nature of the error, the safety/risk significance of the error, whether the error caused any equipment inoperability or damage or endangered plant personnel, whether the error caused an unnecessary plant transient, whether the error complicated emergency response or plant recovery, and whether the error resulted in an avoidable entry Issue Date: 09/24/14 18 71111.11 Effective Date: 01/01/2015 into a facility’s technical specifications or other avoidable non-compliance with the facility’s license.  The sample for in-depth review should not duplicate efforts associated with IP 71152 and other completed NRC inspections. However, it is appropriate to follow-up on corrective actions taken by the licensee as a result of past NRC inspections, such as checking that the corrective actions were completed and assessing their effectiveness.  The sample for in-depth review should include one or more examples of issues where as a corrective action, additional training for licensed operators was conducted. As part of the in-depth review, the inspector should assess the quality of this additional training, and verify that this training was actually conducted (e.g., by reviewing licensee training schedules and/or training attendance records).  The sample for in-depth review should include one or more examples of issues where additional training was not conducted as a result of the issue. In performing this review, the inspector should verify that the facility licensee’s decision not to conduct additional training as a result of the issue was consistent with the nature of the issue, and consistent with the licensee’s licensed operator requalification training and corrective action programs. When conducting the in-depth review, the inspector should apply the guidance presented above, and review each issue sampled using IP 71152, “Problem Identification and Resolution,” Section 03.06, “Performance Attributes.” c. Assessment and Documentation. Assess the ability of the facility licensee to identify, evaluate, and resolve problems associated with licensed operator performance, for problems in performance which occur in the actual plant/main control room. In performing this inspection activity, it is expected that the inspector will assess:  The ability of the facility licensee to identify these types of issues at an appropriate threshold, and that the facility licensee is entering them into their PI&R (corrective action) program.  The ability of the facility licensee to develop and implement corrective actions for these types of issues, including any additional training for licensed operators. Included in this assessment should be a check that the corrective actions were completed, and that the corrective actions were commensurate with the significance of the issues.  The effectiveness of the corrective actions for these types of issues. One way to assess effectiveness is to check that licensed operator errors and performance problems of the same type do not show a pattern of repeating themselves after the completion of corrective actions. Issue Date: 09/24/14 19 71111.11 Effective Date: 01/01/2015 The results of this assessment should be discussed with the facility license at the exit briefing. Documentation of this section of the IP shall be in accordance with MC0612C Appendix D, “Guidance for Documenting Inspection Procedure 71152 Identification and Resolution of Problems.” 03.11 Resident Inspector Quarterly Review of Licensed Operator Requalification and Licensed Operator Performance. At least once each calendar quarter, observe the facility licensee’s requalification testing and training for licensed operators, identify any deficiencies and discrepancies, assess licensed operator performance and evaluators’ critique, and assess simulator performance. In addition, at least once each calendar quarter, licensed operator performance in the actual plant/main control room will be observed and assessed during periods of heightened activity or risk.

a. Review of licensed operator requalification activities As a part of the quarterly reviews of requalification activities, the resident inspector staff shall observe a portion of an in-progress annual requalification operating test11 required by 10 CFR 55.59 administered during a training cycle in which it will not be observed by the NRC during the biennial portion of this IP. Scheduling this activity may be determined from the facility licensee and the region’s Operations Branch. By including this requirement, the facility licensee’s administration of the annual requalification operating test will include some type of NRC observation, alternating years between the resident staff and the biennial inspection team. For inspection activities performed in calendar quarters when not observing a portion of an in-progress annual requalification operating test, the resident staff should prioritize inspection activities, consistent with the facility licensee’s schedule, as follows: 1. The highest priority should be to observe an evaluated simulator scenario of an operating crew, including the evaluators’ critique, if the facility licensee conducts an evaluated scenario. 2. If the facility licensee does not conduct an evaluated scenario, then observe simulator training for an operating crew, including any evaluators’ critique. Prioritize this observation to simulator training conducted in response to licenseeidentified corrective actions, plant modifications, or upcoming plant evolutions. 3. At the discretion of the resident staff, other requalification training maybe observed, for example, classroom training and in-plant training. Prioritize this observation to training conducted in response to licensee-identified corrective actions, plant modifications, or upcoming plant evolutions.

11 Preferably, the resident staff should observe the administration of one or more simulator scenarios administered to an operating crew, including the evaluators’ critique, during an in-progress annual requalification operating test. Additional simulator scenarios and/or JPMs may also be observed, if desired. Issue Date: 09/24/14 20 71111.11 Effective Date: 01/01/2015 If a portion of an annual requalification operating test required by 10 CFR 55.59 is observed, the basic methodology for performing this inspection activity is presented in section 03.05 of this IP, “Licensee Administration of an Annual Requalification Operating Test,” with a focus upon completing the applicable elements of Appendix D of this IP. Based upon the limited observation time of the resident staff and their variable experience with operator licensing, it is not expected that the resident staff will conduct in-depth reviews of operating test quality (section 03.04) or examination security (section 03.06). During this inspection activity and upon its conclusion, any annual requalification operating test issue shall be discussed with the facility licensee, to assist in confirming the issue. However, the inspector should not interfere with the facility licensee’s requalification examination process. If there are any significant concerns with the annual requalification operating test, contact regional management as soon as possible. In evaluating any annual requalification operating test issues for operating tests required by 10 CFR 55.59(a)(2) refer to the applicable appendix of this IP. If observing licensed operator evaluations, including evaluated scenarios, which are not a part of an annual requalification operating test required by 10 CFR 55.59, the inspector should use a methodology similar to that used above for observing a portion of an annual requalification operating test. In particular, the inspector should assess, without interfering with the facility licensee’s evaluation process:  licensed operator performance  the ability of the facility licensee to administer the evaluations  the quality of any post-scenario critiques  follow-up actions taken by the facility licensee for any licensed operator who failed an evaluation (e.g., removal from shift duties, remediation, re-examination) Note that 10 CFR 55.59 requires only a requalification operating test to be conducted annually, and any additional evaluation of licensed operators is up to the facility licensee, in accordance with their accredited systems approach to training (SAT) program. Given the optional nature of additional licensed operator evaluations, issues identified by the resident inspector during this activity typically will not result in documented findings, except, for example, if the issue is associated with conformance with operator license conditions or simulator performance. In particular, the checklists contained in Appendices B, C, D, E, and F concerning examination quality, administration, security, and remedial training and re-examinations are not applicable for examinations that are not required per 10 CFR 55.59. However, observations made during this inspection activity should be provided to the facility licensee as feedback. If observing licensed operator requalification training, no specific guidance is offered in this IP, except to assess the quality of the training received, and to provide feedback to the facility licensee, as appropriate. In assessing the quality of the training received, detailed guidance may be available via facility licensee training observation forms. Since licensed operator requalification training is conducted in accordance with the facility licensee’s accredited SAT program, issues identified by the resident inspector Issue Date: 09/24/14 21 71111.11 Effective Date: 01/01/2015 during this activity typically will not result in documented findings, except, for example, if the issue is associated with conformance with operator license conditions or simulator performance. However, observations made during this inspection activity should be provided to the facility licensee as feedback. Simulator Performance Regardless of what kind of licensed operator requalification activity is observed (i.e., annual requalification operating test, other licensed operator evaluations, training), if the activity is performed in the control room simulator, then the modeling and performance of the control room simulator shall be evaluated. In particular, the resident inspector staff shall review simulator physical modeling (i.e., the degree of similarity between the simulator and the reference plant control room, such as physical location of panels, equipment, instruments, controls, labels, and related form and function) and simulator performance, especially regarding recent modifications implemented in the control room, during the observation of requalification activities. Refer to section 03.09 of this IP, “Simulator Performance,” for additional information regarding this inspection activity, however, based upon the limited observation time of the resident staff and their variable experience with simulators, it is not expected that the resident staff will review any facility licensee simulator records (e.g., simulator corrective action or testing records). During this inspection activity and upon its conclusion, any simulator performance issue shall be discussed with the facility licensee, to assist in confirming the issue. If there are any significant concerns with simulator performance, contact regional management as soon as possible. In evaluating any simulator performance issues, refer to section 03.09 and Appendix G of this IP and IMC 0612, Power Reactor Inspection Reports. b. Review of licensed operator performance in the actual plant/main control room At least once each calendar quarter, licensed operator performance in the actual plant/main control room will be observed and assessed during periods of heightened activity or risk. Although the intent is to conduct control room observations approximately 4 hours per calendar quarter, such that the resident staff monitors operator performance on a fairly periodic basis and can note any trends, observations may be deferred based upon the plant’s schedule and resident inspector judgment, such that approximately 16 hours of observation are conducted per calendar year. For example, suppose in given a calendar year a plant were in an extended outage followed by outage exit activities and a plant startup. Under these circumstances, it may be advantageous to perform little or no control room observations during the calendar quarters of the outage (if few activities are occurring in the control room), but instead spend most or all of the 16 hours for that given calendar year performing observations during the outage exit and plant startup. Regardless of whether quarterly observations are performed or deferrals are used, all inspection hours for this effort should be charged to IP 71111.11Q.

This activity will typically be performed by the resident inspector; however, at the

discretion of Regional Management, it may be advantageous for licensed operator examiners to be utilized from time to time, with some control room activities observed Issue Date: 09/24/14 22 71111.11 Effective Date: 01/01/2015 by examiners during the biennial requalification inspection, provided there is an activity or evolution that occurs in the actual plant/main control room that is suitable to observe during the biennial inspection week. If such an observation is performed during the biennial review, the inspection effort should be charged to IP 71111.11Q. Prior to conducting this inspection activity, the resident staff (or examiner) shall familiarize themselves with the licensee’s procedures, expectations, and policies including:  Operator compliance and use of plant procedures, including procedure entry and exit, performing procedure steps in the proper sequence, procedure place keeping, and technical specification entry and exit.  Control board/in-plant component manipulations.  Communications between crew members.  Use and interpretation of plant instruments, indications, and alarms; diagnosis of plant conditions based on instruments, indications, and alarms.  Use of human error prevention techniques, such as pre-job briefs and peer checking.  Documentation of activities, including initials and sign-offs in procedures, control room logs, technical specification entry and exit, entry into out of service logs.  Management and supervision of activities, including risk management and reactivity management.  Pre-job briefs. For more details on items the inspector should be familiar with, refer to Appendix H, which contains two generic observation checklists for licensed operator performance and pre-job briefs. The licensee’s policies for these areas are typically contained in various operations’ administrative procedures, with titles such as “Conduct of Operations for Shift Personnel,” “Reactivity Management,” “Control Room Conduct and Control Room Shift Activities,” “Risk Management,” etc.; consult with the facility licensee to determine which procedures are applicable at a particular plant. Once familiar with the licensee’s policies for licensed operator performance, it is important that licensed operator performance be observed and assessed during periods of heightened plant activity or plant risk, such that most of the elements listed in Appendix H can be observed, and where the activities could impact plant safety. Actual plant/main control room activities to observe should be selected by reviewing plant activity and/or work schedules, attending daily briefs, shift turnover briefs, refueling outage planning meetings, maintenance planning meetings, and via discussions with the facility licensee, and consideration should be given to observing the following activities:  Plant startups, shutdowns, and mode changes.  Reactor power and turbine load changes, especially when licensee reactivity management policies will be in effect.  Infrequent plant evolutions.  Unplanned transients and off-normal events, including post-scram response. Issue Date: 09/24/14 23 71111.11 Effective Date: 01/01/2015  Surveillance testing.  Post-maintenance testing of safety-related structures, systems, and components.  Pre-startup equipment line-ups, operational checks, and functional checks.  Changes to the line-ups or modes of operation of safety related systems, structures, and components.  Refueling outage preparations, such as filling the reactor cavity or entering midloop operations.  Reactor refueling activities. In particular and as an efficiency measure, observing licensed operator performance per this IP can be performed in conjunction with other resident inspector activities, including plant status reviews (IMC 2515 Appendix D), and inspection of post maintenance testing (IP 71111.19), refueling and outage activities (IP 71111.20), and surveillance testing (IP 71111.22). The activities to be observed shall involve licensed operators, typically observed from the main control room (although in-plant activities can be observed as well) and it is expected that 4 hours per calendar quarter or 16 hours annually be utilized to complete this inspection element, where quarterly, periodic observations are preferred. When observing the activities, if possible, the inspector should also observe any pre-job briefs held prior to the activity. Prior to the observation period, the inspector should also become familiar with any plant operating or testing procedures which will be used by the operators during the observed evolutions, including any precautions and limitations associated with the procedures. It is expected that different inspectors will use slightly different techniques when observing licensed operator performance; however, Appendix H contains two generic observation checklists to use as guidance, where consideration should be given to modifying these checklists based on the licensee’s particular policies at a given plant. Regardless of the differing techniques used, inspectors shall refrain from interfering with the performance of the licensed operators being observed unless interference is warranted due to a significant safety concern. As a part of not interfering with the licensed operators, questions and discussions initiated by the inspector during plant activities should be limited to prevent unnecessary distractions to the licensed operators. When performing the observation of licensed operator performance, the two checklists contained in Appendix H may be used for guidance. Upon completion of the observed activity, the inspector will discuss the overall assessment of licensed operator performance with appropriate licensee management. When communicating with the facility licensee, inspector comments should be objective and supported with examples when possible. Regarding licensed operator performance, the inspector should, in particular, comment on the licensed operators’ adherence to plant procedures, including adherence to the licensee’s conduct of operations procedures and policies. Issue Date: 09/24/14 24 71111.11 Effective Date: 01/01/2015

71111.11-04 RESOURCE ESTIMATE

It is estimated that approximately 96 hours, on average, of direct inspection effort (DIE) will be required to conduct the biennial review of the licensee’s requalification program, regardless of the number of units at the site. The effort includes a regional specialist’s in-office review of materials prior to and/or after the on-site observation period. It is expected that the actual hours required to complete the inspection may vary from the estimate. The hours expended during an inspection should be tailored for the facility licensee and accurately recorded. Depending on availability, resident staff members may assist the regional specialist during the biennial review. It is estimated that approximately 1 hour per year will be required to discuss, collect, review, and document the annual review of requalification examination results. For the resident staff, it is estimated that 4 hours per calendar quarter will be required to review licensed operator requalification activities, with an additional 4 hours per calendar quarter (or 16 hours per calendar year) estimated for observing licensed operator performance in the actual plant/control room.

71111.11-05 COMPLETION STATUS

Inspection of the minimum sample size will constitute completion of this procedure in the Reactor Program System (RPS). The minimum sample size consists of:  Observing licensed operator requalification activities once per calendar quarter.  Observing licensed operator performance in the actual plant/main control room approximately 4 hours per calendar quarter, or 16 hours per calendar year.  Reviewing requalification examination results once per calendar year, and  Completing the biennial inspection of the licensee’s licensed operator requalification program approximately once every two years, consistent with the licensee’s requalification examination schedule.

71111.11.06 REFERENCES

1. IMC 0609, Appendix I, “Licensed Operator Requalification Significance Determination Process” 2. IMC 0612, “Power Reactor Inspection Reports” 3. NUREG-1021, “Operator Licensing Examination Standards for Power Reactors” 4. Regulatory Guide 1.134, “Medical Evaluation of Licensed Personnel at Nuclear Power Plants” Issue Date: 09/24/14 25 71111.11 Effective Date: 01/01/2015 5. ANSI/ANS-3.4, “Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants” 6. Regulatory Guide 1.149, “Nuclear Power Plant Simulation Facilities for Use in Operator [Training and] License Examinations” 7. ANSI/ANS-3.5, “Nuclear Power Plant Simulators for Use in Operator Training [and Examination]” 8. IP 71152, “Problem Identification and Resolution” 9. IMC 0612, Appendix D, “Guidance for Documenting Inspection Procedure 71152 Identification and Resolution of Problems.” END

Appendices:

A. Typical Documents Reviewed During Biennial Inspection B. Biennial Requalification Written Examination Quality C. Annual Requalifcation Operating Test Quality D. Operating Test Administration Checklist E. Requalification Examination Security Checklist F. Remedial Training and Re-Examination Checklist G. Checklist for Evaluating Plant-Referenced Simulators Operating Under 10 CFR 55.46(C) And (D) H. Generic Licensed Operator Observation Checklists Attachment: Revision History Issue Date: 09/24/14 A-1 71111.11 Effective Date: 01/01/2015 APPENDIX A Typical Documents Reviewed During Biennial Inspection 1*. One complete version of a biennial requalification written examination administered during the training cycle. 2*. Ten (10) job performance measures (JPMs) and four (4) simulator scenarios associated with an annual requalification operating test required by 10 CFR 55.59(a)(2). The JPMs and scenarios to be reviewed should include those that will be or have been observed during the onsite portion of this inspection, with any remaining JPMs and scenarios reviewed selected from other weeks within the same training cycle. 3*. Spreadsheets and/or test outlines which show the usage of written examination questions, JPMs, and simulator scenarios. 4*. A list (or lists) of licensee-identified issues associated with licensed operator errors or other licensed operator performance problems which have occurred in the actual plant/main control room since the last biennial requalification program, their associated corrective actions, and if these issues were incorporated into requalification training . (These issues are likely documented in condition/problem identification reports, licenseeconducted audits and assessments, and in industry operating experience reports, etc.) 5*. Facility licensee procedures for licensed operator requalification training and examination, including examination security procedures. 6*. A schedule of the licensee’s examination activities during the on-site inspection week. 7**. Operator Licensing Tracking System Report #14 and selected individual licensed operator’s 10 CFR 55 docket files. 8**. NRC records which document licensed operator performance issues and the facility licensee’s corrective action program performance since the last biennial requalification inspection. These records include: the most recently completed NRC biennial PI&R team inspection report, NRC inspection findings, Plant Issues Matrix, and NRC operating experience information. 9. Licensed operator pass/fail statistics for any NRC-required requalification examination that has been completed (see Table 03.02-1, Examination Results). 10. An overall schedule of the facility licensee’s requalification program since the last biennial inspection (cycle weeks, training topics, etc.) 11. A list and descriptions of any examination security problems since the last biennial inspection (likely documented in training department condition/problem identification reports). Issue Date: 09/24/14 A-2 71111.11 Effective Date: 01/01/2015 12. Records for licensed operator requalification training attendance, licensed operator performance in requalification, licensed operator remedial training, individual licensed operator medical records, and records for maintaining an active license and license reactivation. 13. Simulator testing, maintenance, modification, and performance records. Simulator corrective action records, including lists of open/closed simulator deficiencies, and corrective actions taken.

• It is recommended that the NRC request that the facility licensee submit items 1 through 6

listed above prior to the on-site portion of the biennial inspection, such that portions of the biennial inspection can be reviewed and discussed with the facility licensee while on-site, and for inspection planning purposes.

• • Items 7 and 8 should be reviewed prior to the on-site portion of the biennial inspection.

Issue Date: 09/24/14 B-1 71111.11 Effective Date: 01/01/2015 APPENDIX B Biennial Requalification Written Examination Quality Biennial Requalification Written Examination Quality Checklist Written examination reviewed or date administered: ________ YES NO 1. Biennial requalification written examinations are administered approximately every 2 years, such that each 24-month requalification program contains at least one biennial requalification written examination (10 CFR 55.59(a)(2)). 2. Review the biennial requalification written examination questions from at least one complete written examination using the Requalification Written Examination Questions Review Worksheet (see page B-2), and determine how many questions had flaws, how many questions had no flaws, and determine the percent of questions that had flaws. Note: if a complete examination contains fewer than 30 questions, add an additional flaw based upon the number of questions that the exam has that is less than 30 (i.e., a 28 question exam would be assigned 2 flaws just for having 2 questions less than 30). If greater than 20% of the questions from the initial review conducted above had flaws, review at least one additional written exam, and evaluate for flaws. From the initial review, and if necessary the additional review, record the total number of questions without flaws, the total number of questions with flaws, and the percent of all questions reviewed that were flawed. Total # of Qs without flaws: Total # of Qs with flaws: % of all Qs reviewed with flaws: 3. From line 2, less than or equal to 20% of the total number of reviewed written examination questions contained flaws. If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected quality standards for a licensed operator requalification written examination required by 10 CFR 55.59(a)(2), and as a possible finding. Consult IMC 0612, Power Reactor Inspection Reports, for additional information for screening of inspection results. Issue Date: 09/24/14 B-2 71111.11 Effective Date: 01/01/2015 Requalification Written Examination Question Review Worksheet Written examination reviewed or date administered: ________ Q# 1. LOD (1-5) 2. Question Flaws 3. Q flawed or not flawed? (F/NF) 4. Explanation LOD Direct L/U Correct Answer Cues T/F Cred. Dist. Instructions: 1. Enter the level of difficulty (LOD) of each question using a 1-5 (easy – difficult) rating scale. 2. Check the appropriate block if a question flaw is identified:  The question’s level of difficulty is inappropriate (LOD = 1 too easy; LOD = 5 too hard).  The question is a direct look-up, which only requires an examinee to recall where to find the answer (see NUREG-1021, ES-602 Attachment 1 for additional guidance).  The question has no correct answer or more than one correct answer. Short-answer questions must contain objective scoring, with clear guidance on granting partial and full credit. In addition, check the following items for all multiple choice questions (see NUREG-1021, Appendix B):  The stem or answer choices contain cues as to the correct answer (i.e., clues, specific determiners, phrasing, length, etc.)  The answer choices are a collection of unrelated true false statements such that the question can be answered correctly without reading the question stem.  Two or more distractors are not credible. For open-reference questions, distractors should be judged as if the question was closed-reference. 3. Based on review item 2 above, the question as written is either flawed (F) or not flawed (NF) . 4. Provide a brief explanation for all questions that are determined to be flawed (F). Issue Date: 09/24/14 B-3 71111.11 Effective Date: 01/01/2015 Requalification Written Examination Questions Review Worksheet (Continued) Written examination reviewed or date administered: ________ Q# 1. LOD (1-5) 2. Question Flaws 3. Q flawed or not flawed? (F/NF) 4. Explanation LOD Direct L/U Correct Answer Cues T/F Cred. Dist. Issue Date: 09/24/14 C-1 71111.11 Effective Date: 01/01/2015 APPENDIX C Annual Requalifcation Operating Test Quality Annual Requalification Operating Test Quality Checklist Operating test # or date(s) administered: _______ YES NO 1. An annual requalification operating test is administered at least once every calendar year (10 CFR 55.59(a)(2). 2. The operating test consists of at least five job performance measures (JPMs) for each individual. 3. Review at least 10 JPMs using the Requalification Job Performance Measures Review Worksheet (see page C-2), and determine how many JPMs had flaws, how many JPMs had no flaws, and the percent of JPMs with flaws. If greater than 20% of the JPMs from this initial review had flaws, review at least 10 additional JPMs for flaws. From the initial review of 10 JPMs, and if necessary the additional review of 10 more JPMs, record the total number of JPMs without flaws, the total number of JPMs with flaws, and the percent of all JPMs reviewed that were flawed. Total # of JPMs without flaws: Total # of JPMs with flaws: % of all JPMs reviewed with flaws: 4. From line 3, less than or equal to 20% of the reviewed JPMs contained flaws.

5. The mix of JPMs per JPM set is appropriate: 2 simulator or control room JPMs, 2 in-plant JPMs, and 1 JPM may be of any kind – simulator, in-plant, or administrative. 6. At least 40% (e.g., two out of five) of the JPMs per JPM set are alternate path. 7. The operating test contains an adequate number of simulator scenarios, such that each licensed operator is evaluated using at least two simulator scenarios. 8.a. Review at least four (4) scenarios using the Simulator Scenario Review Worksheet (see page C-4), and determine for each scenario the number of events that each scenario is deficient by. For example, if a scenario contains the minimum number of events (or more than the minimums) in all categories, then this figure would be zero for that scenario. However, if a scenario contained only 3 malfunctions, two (2) flaws would be assigned, since it is less than the minimum number of malfunctions by two. If this same scenario also did not exercise technical specifications, that would count as one additional flaw. After all the scenarios have been reviewed, add up the number of deficient events and enter the total number of deficient events as flaws. Total # of scenario flaws from deficient # of events: Issue Date: 09/24/14 C-2 71111.11 Effective Date: 01/01/2015 Annual Requalification Operating Test Quality Checklist (continued) YES NO 8.b. Review each of the scenario events from the four (or more) scenarios selected in step 8.a above using the Simulator Scenario Review Worksheet (see page C-4), and identify how many scenario events had flaws. To the number of events with flaws add in line 8.a for the total number of deficient events (i.e., one deficient event is treated as one event flaw) and record it. Then determine and record the number of scenario events that had no flaws. From the total number of events reviewed, determine the percent of scenario events with flaws.

1. of scenario

events without flaws:

1. of

scenario events with flaws (add in any deficit from line 8a): % of scenario events reviewed with flaws: 9. If greater than 20% of the scenario events from line 8.b had flaws, review at least 4 additional scenarios per lines 8.a and 8.b above. From the initial review of four scenarios, and if necessary the additional review of four more scenarios, add up and record the total number of scenario events without flaws, the total number of scenario events with flaws (including deficient number of events). From the total number of events reviewed, determine the percent of all scenario events reviewed that were flawed. Total # of scenario events without flaws: Total # of scenario events with flaws (add in any deficit from line 8a): Total % of scenario events reviewed with flaws: 10. From line 9, less than or equal to 20% of the reviewed scenario events contained flaws. If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected quality standards for a licensed operator requalification operating test required by 10 CFR 55.59(a)(2), and as a possible finding. Consult IMC 0612, Power Reactor Inspection Reports, for additional information for screening of inspection results. Issue Date: 09/24/14 C-3 71111.11 Effective Date: 01/01/2015 Requalification Job Performance Measure Review Worksheet Operating test # or date(s) administered: _______ Instructions: 1. Enter the type of JPM – (S)imulator, (P)lant, or (O)ther. 2. Enter (Y)es or (N)o whether the JPM is alternate path or not. 3. Enter the level of difficulty (LOD) of each JPM using a 1-5 (easy – difficult) rating scale. (LOD > 1 and < 5 are acceptable). Also see below. 4. Check the appropriate block if a JPM flaw is identified:  The JPM’s level of difficulty is inappropriate. Simple one-step JPMs, or a JPM that tests solely for recall or memorization, or a JPM which requires directly looking-up a single correct answer is likely LOD = 1 and too easy. Conversely, a JPM with over 30 steps or that takes in excess of 45 minutes to complete is likely LOD = 5 and too hard.  The JPM lacks adequate initial conditions or lacks an adequate initiating cue.  The JPM lacks adequate evaluator cues to allow the examinee to complete the task.  The JPM lacks adequate performance standards and/or contains errors in designating critical steps.  The JPM lacks an appropriate validation time or lacks a time for completion standard. 5. Based on the review of item 4 above, the JPM as written is either flawed (F) or not flawed (NF). 6. Provide a brief explanation for all JPMs that are determined to be flawed (F). JPM # or title 1. Type? (S/P/O) 2. Alt. Path? (Y/N) 3. LOD (1-5) 4. JPM Flaws 5. JPM flawed or not flawed? (F/NF) 6. Explanation LOD IC Cues Perf. Stds

&

crit.steps Time Limit Issue Date: 09/24/14 C-4 71111.11 Effective Date: 01/01/2015 Simulator Scenario Review Worksheet 1. Scenario ID (e.g., # or title) Week Used: Total Malf. Malf. After EOPs ABNs MTs EOPs beyond scram CTs TS 2. Minimum number of events (see page C-5) 5 1 2 1 1 2 1 5. Total deficit: Explanation: 3. Actual number and type of events in the scenario 4. Deficit 6. Scenario Event ID/Name: 7. Scenario event flawed (F) or not flawed (NF)? 8. Explanation Performance Standards Critical Task Instructions: 1. Enter the scenario identifier and week within the training cycle that the scenario will be (or has been) administered. 2. Minimum number of events per scenario from page C-5. 3. Enter the actual number and types of events in the scenario. 4. Subtract each column in line 3 from line 2. Enter zero if the minimum number of events for that event type is equaled or exceeded. 5. Add up the columns from line 4, and determine the total number of events less than the minimums that the scenario has. Any deficit will be used in computing line 8.a of the Requalification Operating Test Quality Checklist (pages C-1, C-2), when totaled with the deficits from other scenarios. 6. Enter the scenario event name and description. 7. Review the individual events contained in each scenario, and evaluate for event flaws:  The scenario guide event description lacks adequate crew/operator performance standards.  The scenario guide event description incorrectly designates an event as a critical task (i.e., a non-critical task labeled as critical or a critical task labeled as non-critical). The number of scenario events that are flawed and not flawed will be used in computing line 8.b of the Requalification Operating Test Quality Checklist (pages C-1, C-2), when totaled with the flawed/not flawed events from other scenarios. 8. Provide a brief description for any scenario event determined to be flawed (F). Abbreviations: Malf. = Malfunction EOPs = Emergency Operating Procedures ABNs = Abnormals MTs = Major Transients CTs = Critical TasksTS = Technical Specifications Issue Date: 09/24/14 C-5 71111.11 Effective Date: 01/01/2015 MINIMUM NUMBER OF EVENTS* PER SCENARIO 1. Total malfunctions inserted: 5 2. Malfunctions that occur after emergency operating procedure (EOP) entry: 1 3. Abnormal events: 2 3. Major transients: 1 4. EOPs used beyond primary scram response EOP: 1 5. Crew critical tasks: 2 6. Technical Specifications are exercised during the scenario

• In order for a scenario event to be counted towards the above listed minimums, the scenario

event must contain verifiable operator action(s) and an associated performance standard(s) to mitigate or address the event. Issue Date: 09/24/14 D-1 71111.11 Effective Date: 01/01/2015 APPENDIX D Operating Test Administration Checklist Date(s) operating test observed: _______ If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected standards for the administration of a licensed operator requalification operating test required by 10 CFR 55.59(a)(2), and as a possible finding. Consult IMC 0612, Power Reactor Inspection Reports, for additional information for screening of inspection results. 1 It is understood that JPMs and/or scenarios may not run “as planned” due to examinee errors. The checklist items here refer to JPMs and/or scenarios that do not run “as planned” due to errors made by licensee personnel who are administering the examinations, i.e., the simulator booth operator, examinee evaluators, and other licensee personnel who are assisting in administering the examination. YES NO 1. JPMs are administered with one facility evaluator per examinee. Exception: administrative JPMs can be administered in a group format. 2. There were no uncorrected examination administration errors observed during the performance of the JPMs and all of the JPMs were conducted as planned1 . Examples of examination administration errors include improper simulator set-up, improper action taken by the simulator booth operator in response to an examinee’s action(s), or improper evaluator cuing of the examinee. 3. There were no uncorrected examination administration errors observed during the performance of the simulator scenarios and all of the scenarios were conducted as planned1 . Examples of examination administration errors include improper simulator set-up, improper action taken by the simulator booth operator in response to the crew’s action(s), or improper evaluator cuing of one or more crew members. 4. Examinee performance errors during JPMs are detected by facility evaluators, such that you agree with all of the pass/fail determinations for JPMs that you observed. 5. Examinee performance errors during simulator scenarios are detected by facility evaluators, such that you agree with all of the pass/fail determinations for scenarios that you observed. When evaluating this item, the inspector should consider whether the facility licensee utilized an adequate number of evaluators for the given crew size. Issue Date: 09/24/14 E-1 71111.11 Effective Date: 01/01/2015 APPENDIX E Requalification Examination Security Checklist If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected standards for examination integrity and as a possible finding, typically associated with a regulatory violation of 10 CFR 55.49. If the equitable and consistent administration of an examination was actually affected, traditional enforcement should also be considered. Consult IMC 0612, Power Reactor Inspection Reports, for additional information for screening of inspection results. YES NO 1. Biennial requalification written examinations administered to different licensed operators during different weeks within a training cycle repeat < 50% of examination questions that have previously been administered during that same training cycle. 2. Annual requalification operating tests administered to different licensed operators during different weeks within a training cycle repeat < 50% of JPMs that have previously been administered during that same training cycle. 3. Annual requalification operating tests administered to different licensed operators during different weeks within a training cycle repeat < 50% of scenario events that have previously been administered during that same training cycle. In addition, operators tested during different weeks within a training cycle are exposed to a variety of major transients. 4. During the annual requalification operating test, some examinees may have completed a test item or items prior to other examinees during the testing week. In such instances, the facility licensee implemented proper controls (e.g., sequestering, monitoring) which prevented communication of examination information between examinees who had completed a test item or items from those examinees who had yet to complete the same test item or items. 5. During the annual requalification operating test, examination materials - e.g., JPMs, scenario guides, hand-outs to examinees, procedures marked-up or used by examinees, logs kept by examinees - were properly controlled by the facility licensee, such that examinees were not exposed to any examination materials prior to exam administration. 6. During the annual requalification operating test, access to the control room simulator was properly controlled by the facility licensee (e.g., posted signs, locked doors), such that examinees were not exposed to any examination information prior to exam administration. 7. During this inspection and since the last biennial inspection, no incidents of examination compromise have occurred. As defined in 10 CFR 55.49, the integrity of a test or examination is considered compromised if any activity, regardless of intent, affected, or but for detection, would have affected the equitable and consistent administration of the test or examination. 8. If licensed individuals were used to develop or validate requalification examinations, or to administer requalification examinations to other licensed operators, then those licensed individuals’ requalification examinations contained no duplication of test items that they developed, validated, or administered to others. Issue Date: 09/24/14 F-1 71111.11 Effective Date: 01/01/2015 APPENDIX F Remedial Training and Re-Examination Checklist YES NO 1. Re-examinations administered by the facility licensee are commensurate with the original failures. To be considered commensurate, determine the nature of the original failure (written, JPM, or scenario examination) and apply the criteria contained in Appendix B of this IP for written re-examinations (i.e., a minimum of 30 questions) and Appendix C of this IP for JPM and simulator scenario re-examinations (i.e., a minimum of 5 JPMs, at least 40% of JPMs are alternate path, each licensed operator evaluated using at least two simulator scenarios, each simulator scenario contains at least the minimum number of events). Under appropriate circumstances, an individual failure during a simulator crew examination may be retested with one or more JPMs. 2. Re-examinations do not contain any test items which exactly duplicate test items from the original examination. [Similar test items may be used on re-examinations, but they must be modified from original test items as shown below: - Written examination questions used on re-examinations must include a change to at least one pertinent condition in the stem and a change to at least one answer choice when compared to questions used on the original examination. - JPMs used on re-examinations must include a substantive change to at least one condition, such that the course of action differs when compared to JPMs used on the original examination. - Simulator scenarios used on re-examinations must include a substantive change to each scenario event such that the course of action for each event differs when compared to events used from the scenarios that comprised the original examination.] 3. Re-examinations repeat < 50% of test items previously administered during the training cycle. If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected standards (either quality, integrity, or operator performance) associated with licensed operator requalification examinations required by 10 CFR 55.59(a)(2), and as a possible finding. Consult IMC 0612, Power Reactor Inspection Reports, for additional information for screening of inspection results. Issue Date: 09/24/14 G-1 71111.11 Effective Date: 01/01/2015 APPENDIX G Checklist for Evaluating Plant-Referenced Simulators Operating Under 10 CFR 55.46 (C) and (D) YES NO OVERALL QUESTIONS: 1. Is simulator modeling and performance satisfactory? 2. Does the facility licensee effectively correct identified simulator deficiencies? 3. Does the facility licensee properly perform required simulator testing?  At the correct simulator testing periodicity?  With proper evaluations of the simulator against the reference plant or best-estimate data?  With the correct documentation and retention of simulator test records? DETAILED QUESTIONS FROM 10 CFR 55.46 1. If the plant-referenced simulator is used for the administration of NRC reactor operator and senior operator operating test, does the plant-referenced simulator demonstrate expected plant response to operator input and to normal, transient, and accident conditions to which the simulator has been designed to respond? ['55.31(a)(5) and '55.46(c)(1)] 2. Is the plant-referenced simulator sufficient in scope and fidelity with the reference plant to allow conduct of the evolutions listed in 10 CFR 55.45(a)(1) through (13),as applicable to the reference plant? ['55.46(c)(1)(i)] 3. Is the plant-referenced simulator sufficient in scope and fidelity with the reference plant to allow conduct of the evolutions listed in 10 CFR 55.59(c)(3)(i)(A) through (AA), as applicable to the reference plant? ['55.46(c)(1)(i)] 4. Is the plant-referenced simulator designed and implemented in a manner that allows for the completion of control manipulations for operator license applicants? ['55.46(c)(1)(ii)] 5. If the plant-referenced simulator is used to meet experience requirements for applicants for operator and senior operator licenses, does the plant-referenced simulator utilize models relating to nuclear and thermal-hydraulic characteristics that replicate the most recent core load in the nuclear power reference plant for which a license is being sought? The phrase "most recent" means the current core or if the reference plant is in a refueling outage, the core just previous to the outage. (This question can be answered with a brief inquiry to the facility licensee, unless an issue is identified.) ['55.31(a)(5)], ['55.46(c)(1), and '55.46(c)(2)(i)] 6. Has the plant-referenced simulator fidelity been demonstrated so that significant control manipulations are completed without procedure exceptions, simulator performance exceptions, or deviation from the approved training scenario sequence? ['55.46(c)(2)(ii)] 7. There has been no lapse in the facility licensee conducted simulator performance testing throughout the life of the simulation facility ['55.46(d)(1)] Issue Date: 09/24/14 G-2 71111.11 Effective Date: 01/01/2015 YES NO 8. Are the results of performance testing retained for four years after the completion of each performance test or until superseded by updated test results? ['55.46(d)(1)] 9. Are modeling and hardware discrepancies and discrepancies identified from scenario validation and from performance testing being corrected? ['55.46(d)(2)] 10. Are results of any uncorrected performance test failures that may exist at the time of the operating test or requalification program inspection available for NRC review? ['55.46(d)(3)] 11. Has simulator fidelity been maintained such that license application, examination, and test integrity are consistent with 10 CFR 55.49 requirements? ['55.46(d)(4)] If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the standards associated with 10 CFR 55.46, “Simulation facilities,” and shall be processed as potential findings typically with a regulatory violation against 10 CFR 55.46. Consult IMC 0612, Power Reactor Inspection Reports, for additional information for screening of inspection results. Issue Date: 09/24/14 H-1 71111.11 Effective Date: 01/01/2015 APPENDIX H Generic Licensed Operator Observation Checklists Control Room/In Plant Observation Checklist The following is presented as generic guidance for observing licensed operators in the main control room or in the plant. The listed items are not regulatory requirements. For additional plant-specific guidance, refer to the licensee’s conduct of operations policies. YES NO COMPLY WITH AND USE PROCEDURES INCLUDING TECHNICAL SPECIFICATIONS 1. Were the appropriate procedures used and referenced in a timely manner? 2. Were the procedures used correctly, including following procedure steps in the correct sequence, abiding by the precautions and limitations, selecting the correct procedure paths on decision blocks, and correctly transitioning between procedures? 3. If procedure steps were skipped or marked “N/A”, these instances were appropriately justified. 4. Was place-keeping in the procedures effective and in accordance with licensee practices for procedure place-keeping? 5. Were Technical Specifications appropriately entered, exited, and complied with? CONTROL BOARD/IN-PLANT COMPONENT MANIPULATIONS 1. Were the components/controls located efficiently and accurately by the operators ? 2. Were the components/controls manipulated accurately and in a timely fashion? COMMUNICATIONS 1. Did the crew exchange complete and relevant information in a clear, easily understood, and accurate manner? 2. Did the crew appropriately keep personnel outside the control room informed of plant status, and were the required communications made outside of the control room in accordance with licensee policy? 3. Did the crew ensure the receipt of clear, easily understood communications from the crew and others? 4. Did the crew consistently use repeat backs, three-way communications, and the phonetic alphabet in accordance with licensee policy? INTERPRETATION, DIAGNOSIS, AND UNDERSTANDING 1. Were plant alarms and indications properly utilized and interpreted to correctly assess and diagnose plant conditions? 2. Were off-normal trends recognized in a timely fashion? 3. Did the crew demonstrate an understanding of the manner in which the plant, systems, and components operated and interacted, including the knowledge of setpoints, interlocks, and automatic functions? 4. Did the crew demonstrate an understanding of how their actions (or inaction) affected system and plant conditions? Issue Date: 09/24/14 H-2 71111.11 Effective Date: 01/01/2015 YES NO USE OF HUMAN ERROR PREVENTION TECHNIQUES 1. Pre-job and status briefs were well-conducted and in accordance with licensee policy (See separate Activity Briefing Checklist contained in this Appendix) 2. Were peer checks conducted in accordance with licensee policy? DOCUMENTATION OF ACTIVITIES 1. Were plant procedures, testing procedures, and other documents used during the activity properly initialed and signed? 2. Was the documentation of Technical Specification entries (and exits), and entries into equipment out of service logs/degraded equipment logs in accordance with licensee policy? 3. Were control room logs properly maintained in accordance with licensee procedures? 4. If a problem did occur during the activity, did the crew properly open and document the issue in a condition report? MANAGEMENT AND SUPERVISION OF ACTIVITIES 1. Did shift management ensure that the crew adhered to plant procedures (e.g., administrative, system operating, surveillance, and alarm response procedures; operations policies and management expectations)? 2. Did shift management properly consider plant safety, including performing a risk assessment if necessary, prior to and during the activity? 3. Did shift management demonstrate the ability to make sound decisions, applying conservative decision making where appropriate? 4. Did shift management demonstrate the ability to properly prioritize tasks and effectively use available personnel resources? 5. Was shift management well aware of the crew’s actions and plant conditions, and in a position to allow proper crew oversight? 6. Did shift management effectively solicit crew feedback? 7. Were reactivity manipulations conducted in accordance with the licensee’s policy for reactivity management? Issue Date: 09/24/14 H-3 71111.11 Effective Date: 01/01/2015 The following is presented as generic guidance for observing activity briefs, and is primarily suited for assessing pre-job briefs, although aspects of this checklist may also apply to status briefs conducted during an evolution. The listed items are not regulatory requirements. For additional plant-specific guidance, refer to the licensee’s briefing policies. YES NO Overall, did the brief adequately address the task? Task items to check include:  Was the task adequately described?  Were task and individual step completion criteria presented?  Were key/critical steps identified and discussed?  Were communication methods discussed?  Were roles, responsibilities, and specific steps identified and assigned to specific individuals?  Was the sequence of steps and events discussed?  Were technical specifications, operability, and out of service log entries discussed (as applicable)?  Was the use of any special equipment discussed (as applicable)?  Were expected results, trends and, plant/system/component responses discussed?  Where criteria and methods for stopping and “hold points” discussed?  Where concerns and anticipated problems discussed, including contingencies and abort criteria if problems arose? Overall, did the brief adequately address human performance elements associated with the task? Human performance elements include:  Were error likely situations discussed?  Were irreversible actions discussed?  Was the need for self-checking and peer-checking discussed?  Was procedure adherence discussed? Other items  Was the briefing well-led, with sufficient management involvement?  Did individuals at the brief actively participate?  Were safety concerns and affect on plant risk discussed?  Were previous lessons learned, industry events, and operating experience associated with this activity discussed? Issue Date: 09/24/14 H-4 71111.11 Effective Date: 01/01/2015 DATE(S): ______________ EVOLUTION(S) OBSERVED: ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ NOTES AND COMMENTS: ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ Issue Date: 09/24/14 Att1-1 71111.11 Effective Date: 01/01/2015 ATTACHMENT 1 - Revision History - IP 71111.11 Commitment Tracking Number Accession Number Issue Date Change Notice Description of Change Description of Training Required and Completion Date Comment and Feedback Resolution Accession Number N/A ML012420444 08/16/2001 CN 01-015 Revised to clarify the original intent of the procedure as it relates to sample size selection. None N/A N/A ML022320730 08/20/2002 CN 02-031 Revised to reflect the amended 10 CFR Part 55, "Operators' Licenses," regarding operator license eligibility and the use of simulation facilities in operator licensing (66 FRN 52657, dated October 17, 2001). This revision provides specific guidance to inspector when assessing conformance with simulator requirements specified in 10 CFR 55.46. None N/A N/A ML040210317 12/16/2003 CN 03-041 Revised to include an additional section that inspects excessive test item repetition among comprehensive requalification exams that are taken by crews undergoing the same training program cycle. Excessive item repetition adversely affects validity of the exam. Clarify the original intent of the procedure as it relates to sample size selection. None N/A Issue Date: 09/24/14 Att1-2 71111.11 Effective Date: 01/01/2015 Commitment Tracking Number Accession Number Issue Date Change Notice Description of Change Description of Training Required and Completion Date Comment and Feedback Resolution Accession Number N/A ML053490168 01/05/2006 CN 06-001 Inspection resource was increased to 4 hrs/quarter (net increase of 4 hours/year) to more accurately reflect the time spent by resident inspectors during their quarterly observation of operator requalification activities. Completed historical CN search. None N/A N/A ML113270192 12/06/11 CN 11-040 Complete rewrite of document. Added 4 hrs/quarter for Resident Inspectors to observe operators in the control room. Replaced Operating History section with Problem Identification and Resolution section. Added an Examination Security section, taken from other parts of the existing IP. Clarified biennial inspection requirements and updated assessment methods to current practices. Training held by teleconference with Regional examiners on 11/30/11 ML113250476 N/A ML121560358 08/27/12 CN 12-018 Minor typographical errors corrected, clarified who can perform the biennial inspection, added reference to industry standards for requalification examinations, changed wording of appendices regarding performance deficiencies, and removed number of licensee evaluators used during scenarios as a metric from Appendix D. None Closed FF: 71111.11-1756 ML12240A228 1245-1757 Ml12240A210 Issue Date: 09/2414 Att1-3 71111.11 Effective Date: 01/01/2015 Commitment Tracking Number Accession Number Issue Date Change Notice Description of Change Description of Training Required and Completion Date Comment and Feedback Resolution Accession Number N/A ML14217A409 09/24/14 CN 14-022 Flexibility added for the frequency of main control room observations, clarified individual examination failure rate, changed the methodology for assessing simulator scenario quality (including a new worksheet), eliminated the 10% re-take exam failure rate metric None Closed FF: 71111.11-1850, ML14266A099 71111.11-1920, ML14266A101 71111.11-1950 ML14266A104