ML18086A737
| ML18086A737 | |
| Person / Time | |
|---|---|
| Issue date: | 03/27/2018 |
| From: | Kelly Clayton NRC Region 4 |
| To: | |
| Cushing E, NRR/DIRS, 415-1064 | |
| References | |
| Download: ML18086A737 (19) | |
Text
Lessons Learned from the Most Recent NRC Authored Exams in R4 Kelly Clayton Sr. Operations Engineer, RIV
Overview
- Requirements for NRC Written Exams
- Communications between NRC and Licensee
- Format and Revision Control
- Reference submittal
- Dates and validations (plural here)
- Misc Observations
- Conclusions/Questions 2
Requirements for NRC Initial Exams
§ 55.40 Implementation.
(a) The Commission shall use the criteria in NUREG-1021, "Operator Licensing Examination Standards for Power Reactors,"in effect six months before the examination date to prepare the written examinations required by §§ 55.41 and 55.43 and the operating tests required by § 55.45.
The Commission shall also use the criteria in NUREG-1021 to evaluate the written examinations and operating tests prepared by power reactor facility licensees pursuant to paragraph (b) of this section.
3
Requirementspart 2 (b) Power reactor facility licensees may prepare, proctor, and grade the written examinations required by §§ 55.41 and 55.43 and may prepare the operating tests required by § 55.45, subject to the following conditions:
(1) Power reactor facility licensees shall prepare the required examinations and tests in accordance with the criteria in NUREG-1021 as described in paragraph (a) of this section; (2) Pursuant to § 55.49, power reactor facility licensees shall establish, implement, and maintain procedures to control examination security and integrity; (3) An authorized representative of the power reactor facility licensee shall approve the required examinations and tests before they are submitted to the Commission for review and approval; and (4) Power reactor facility licensees must receive Commission approval of their proposed written examinations and operating tests.
4
Requirementspart 3 (c) In lieu of paragraph (b) of this section and upon written request from a power reactor facility licensee pursuant to § 55.31(a)(3), the Commission shall, for that facility licensee, prepare, proctor, and grade, the written examinations required by §§ 55.41 and 55.43 and the operating tests required by § 55.45. In addition, the Commission may exercise its discretion and reject a power reactor facility licensee's determination to elect paragraph (b) of this section, in which case the Commission shall prepare, proctor, and grade the required written examinations and operating tests for that facility licensee.
5
Requirementspart 4 NUREG 1021 Requirement (ES-201): Each NRC regional office shall prepare at least one examination per calendar year to certify new examiners, if necessary, and to maintain examiner proficiency. This yearly examination shall minimally consist of the entire written examination and the outlines for the JPMs and scenarios (Appendix D ES-D-1 forms).
6
Requirementspart 5 For exams being written for new examiner qualification, the trainee will develop a portion of scenarios and JPMs to completion.
7
Communications Remember, everything goes through the Chief Examiner. Make sure he/she is informed on everything sent back and forth and make sure everything is password protected.
8
Communicationspart 2 Between licensee and Chief examiner:
Initial kickoff call - discuss things like
- 1. sim time,
- 2. Any contract work/repairs on the sim,
- 3. Any sim software uploads,
- 4. Procedure freezes (if done),
- 5. Exam security,
- 6. Dates for onsite for development of items, for onsite validation, and onsite for administration.
9
Communicationspart 3
- Establish with licensee what methodology will be used to ensure there is no overlap between NRC operating test and licensee audit exam.
- Establish with licensee early the expected division of resources, and target dates for NRC developed material completion and license-developed material completion.
10
Communicationspart 4
- Establish a schedule for development of written and operating test materials in parallel. One portion of the exam should not be prioritized over the other to avoid a resource crunch.
- Review the last 3 NRC exams and identify under sampled areas. NUREG 1021 Rev 11 requires that every scenario shall be new or significantly modified. A significant modification means that, for each scenario, at least two events have not been used on the previous two NRC initial licensing operating exams.
11
Format and Revision Control
- 1. Standard template for each product-why?
- 2. Major changes should get rev changes-why?
- 3. Hidden text is a nice feature in MSWord for the written exam
- 4. Personal tip: Merge the written exam into 4 sections once you get all the examiner questions done (ie first cut). This will help in sending back and forth to licensee due to firewall size restrictions.
12
Reference submittal
- 1. Need a lot more references if NRC authors the exam-why?
- 2. TCOA lists, IOAs lists, unit differences lists, simulator differences list, critical task list-All are important tools
- 3. Malfunction list from simulator-why?
- 4. Banks for everything (JPMs, sim scenarios, written questions)-why?
- 5. Operating experience is used-why?
13
Reference submittal part 2
- 1. Some stations are missing procedures (AOPs).
- 2. Some stations are not implementing industry OP Experience-not training or evaluating applicable OP Ex.
- 3. Some stations are moving items from the initial program out to GAP that is not allowed to be in GAP training (tagouts, print reading, refueling, etc) 14
Dates and Validations
- 1. Agree on dates early during kick-off call.
- 2. Need one onsite week for simulator development work (many weeks prior to actual validation week).
- 3. Stay ahead of schedule-always
- 4. Validate, make changes, validate again. Once you have the product where you think it needs to be, then take it through the formal validation process (Licensee validates it first and then the licensee validates it with the NRC present).
15
Validation TIPs
- 1. Use a good crew for validation of materials
- 2. Must have an Ops rep for verification of various items on the exam-a licensed SRO such as an AOM-check things like operational validity, procedure transitions on scenarios, TS calls are correct on scenarios, JPMs are reasonable (pay attention to alt path JPMs).
- 3. Remember that you are signing for the quality and content of the exam when you sign it, even if NRC wrote it. We all have to agree with signatures that it meets the standards and is ready to give.
16
Misc Observations
- Legacy Vision program, version 8.1.0.9 has security issues.
- New exam writers not going to workshops and sometimes have minimal experience in writing the NRC exam-then it sometimes turns out the RQ exams werent hitting the mark either. As a supervisor when you approve the exam (RQ or initial) you are stating that it meets the standards for measuring knowledge.
17
Conclusions It is still a very common misconception on how much time it takes to write the exam to meet all of the requirements and ALSO be a successful exam.
How do you measure success on an examination?
18
Questions
???
19