ML21274A022
| ML21274A022 | |
| Person / Time | |
|---|---|
| Site: | National Bureau of Standards Reactor |
| Issue date: | 09/20/2021 |
| From: | US Dept of Commerce, National Institute of Standards & Technology (NIST) |
| To: | Office of Nuclear Reactor Regulation |
| Shared Package | |
| ML21274A018 | List: |
| References | |
| Download: ML21274A022 (10) | |
Text
Root Cause Response Revision 1 September 20, 2021 Contents Introduction .................................................................................................................................................. 1 Corrective Actions ......................................................................................................................................... 2 Management Systems............................................................................................................................... 2 Qualification and Training ......................................................................................................................... 5 Procedures ................................................................................................................................................ 6 Equipment, tools and instruments ........................................................................................................... 7 Event Response ......................................................................................................................................... 9 Other corrective actions ......................................................................................................................... 10 References .............................................................................................................................................. 10 Introduction Two reports have been issued in response to the February 3, 2021, fuel failure event at the NCNR. The first was a root cause investigation by the NCNR Technical Working Group (TWG) [1], and more recently, the SEC subcommittee on Event Response and Corrective Action Subcommittee (ERCAS) issued an event response and root cause investigation and review of the TWG report [2]. In response to the TWG report, a total of nineteen teams within NCNR were formed to evaluate Corrective Action and Reactor Recovery Items (CARRI). The ERCAS report has been reviewed by the CARRI teams and is the subject of this response. CARRI teams input was used in formulating the corrective actions listed in this report.
NCNR management concurs with all corrective actions and suggested program improvements in the ERCAS report and is committed to implementing all of them.
Some longer-term items may require additional resources. NCNR management will work with NIST management to prioritize these recommendations and to develop a long-term budget to address these changes. Issues that directly impact the safe operation of the reactor and the safety of the public are prioritized and will be completed prior to reactor startup. Additional changes may require multiple years and thus will be incorporated into NCNR budget requests as necessary over the next several years.
Corrective actions (CAs) are intended to prevent recurrence of this type of event. Related observations and associated Suggested Program Improvements (SPIs) were provided in the ERCAS report for consideration by NCNR management and intended to strengthen NCNRs safety management system and associated processes. Responses to the ERCAS report are being provided to both CAs and SPIs, listed below by root cause as in the ERCAS report.
1
Corrective Actions Management Systems The ERCAS report recommended seven management corrective actions and seven management suggested program improvements. These are listed below, along with actions the NCNR is taking to address them.
MS-CA-1 Develop and implement a change management framework to evaluate sufficiency of existing change management processes, identify gaps and areas for improvement.
A change management evaluation process is being put into place to identify gaps and areas of improvements. Principal areas of the evaluation process include:
- Organization and staffing:
o Organizational realignment. See also MS-CA-2.
o Training to develop future consistency of proficiency among staff. See also QT-CA-1.
- Process:
o An enhanced Aging Reactor Management (ARM) program is being developed. See also MS-CA-4.
- Procedures:
o Complete overhaul of procedure process, including procedure use and adherence, writing, and consistency with industry best practices. See also PR CAs.
- Tools and Equipment:
o Managing the life cycle of tools and equipment through Engineering Change Notices, Trouble Tickets and the ARM program. See also IE CAs.
The change management process involves the synthesis of these processes and others and will be developed in conjunction with the development of the enhanced ARM program.
MS-CA-2 Develop system for knowledge and skills management in the presence of personnel attrition.
Several changes are being implemented to improve operator knowledge, training and retention. We are currently working to establish a fifth shift (there are currently four rotating shifts) for training and maintenance, among other necessary tasks to better manage knowledge in the presence of attrition.
We are also looking to provide retention incentives for reactor operators that will be put into place once staffing levels allow. In addition, we are in the process of filling and upgrading Operations and ARM management positions.
MS-SPI-1 Develop a process to manage CRO transitions based on specific requirements and duties of this position.
2
NCNR management is committed to appointing a permanent CRO and is working to do so prior to reactor startup. In addition, Operations management will be strengthened through additional hiring and promotions, including hiring a Deputy CRO. This will better assure succession planning.
MS-SPI-2 Define the career path to becoming a crew chief, including defining the skills required to fulfill role and responsibilities.
As part of the reevaluation of the training and management systems, the requirements and skills required to fulfill the crew chief are being more clearly defined. This will include a more defined pathway for personnel to progress towards this position if they desire. All reactor supervisors and supervisor candidates will also undergo leadership and development training.
MS-SPI-3 Review processes for modifications to procedures that involve use of engineered items and communicate changes to Reactor Engineering staff.
Procedure modifications and training on these modifications are currently underway involving all ROE and HP staff. Communications of changes, particularly involving engineered items, will be facilitated through a variety of methods, including the updated ARM program (see MS-CA-4) and System Review Teams (MS-SPI-4). We are continuing to develop more effective means for management and communication of NCNR procedures.
MS-CA-3 Assess efficacy of all tools [used in refueling] and determine necessary improvements.
The NCNR is in the process of fully evaluating the tools used in managing fuel in the reactor. This includes a more formal methodology for assessing changes in tool design. The NCNR is also assessing the precision of tools currently in use for fuel manipulation. This includes the index plate, pickup tools, and pickup extension tools.
Detailed dimensional measurements were made of reactor structures, including pickup tool assemblies.
The team performing those measurements made the following recommendation:
- Perform a batch replacement of the transfer arm pickup tools so that the flush-height check performed during the latch verification process matches the results of the standard transfer arms. Until the replacement is completed, provide clear administrative controls via procedure that do not permit confusion about the required vertical position of the clutch tool collar relative to the index plate for indication of a correctly latched fuel element.
Refueling procedures will be modified to provide updated administrative controls verifying latching (see PR and IE corrective actions). This will include discontinuation of the height check as a verification method (IE-CA-4). Replacement of the pickup tools will be done before or during the 2023 outage when the cold source will be replaced.
MS-CA-4 Prioritize and elevate the Aging Reactor Management program emphasizing oversight of communications between groups and ensuring that maintenance and other issues identified are resolved.
As of this writing, a personnel search for an ARM manager, reporting to the Chief of Reactor Operations and Engineering (ROE), is underway. This is a new reporting structure ensuring that the Chief of ROE has direct oversight of the ARM manager. A key part of the ARM manager position is to oversee and 3
enhance communication amongst groups. Once this position is filled, dedicated staff will be reassigned or hired to fill maintenance and other roles and to support this enhanced position.
MS-SPI-4 Develop a process for RO/RE to collaboratively manage engineered items over the full lifecycle, from development, acceptance testing, through use, routine PM and replacement that includes stakeholder engagement and feedback.
Collaboration between Operations and Engineering groups is essential in successful and reliable reactor operations. Several processes, including System Review Teams (SRTs) and the Trouble Ticket system, are being improved to ensure better and collaborative interactions between the groups, including involvement of at least two engineers in reactor SRTs. In addition, ROE managers will further identify programs to enhance collaboration and communications between the two groups.
MS-CA-5 Develop program for robust qualification of supervisors overseeing refueling operations.
Supervisor qualifications are being revised (along with operator qualifications in QT-CA-2) to demonstrate proficiency in refueling and oversight annually. In addition, supervisors must now undergo procedure oversight training (MS-CA-6) and training in supervisor responsibilities to assure that human performance tools (AR 1.1) are actively used. These requirements will be documented in the updated supervisor qualification program (QT-CA-3).
MS-CA-6 Require training for supervisors on oversight.
AR 5.0, Procedure Use and Adherence, has been modified to strengthen the oversight role that supervisors must play. AR 5.0 now requires the CRO, CRE, and CHP to ensure that all personnel have been trained on procedure use, adherence, and oversight. See also PR-CA-5.
MS-SPI-5 Increase management engagement with reactor operators, i.e., establish an ongoing dialog on practices, issues and planned changes.
NCNR management is committed to increasing and sustained engagement with reactor operators on a number of fronts to discuss current or upcoming issues. In addition to the current NCNR Management Observation Process, MP 1.2, Observation Program & Checklist has been drafted and will be implemented to provide a framework for management and peer observations of ongoing work.
MS-SPI-6 Provide oversight and mentoring for supervisors to ensure greater consistency in practices among crew chiefs.
In addition to engagement of operators, NCNR management is also committed to increase engagement of reactor supervisors and crew chiefs and will do so daily. This will include mentoring supervisors and providing guidance in consistent coaching and knowledge transfer for their trainees. The training program is also being upgraded to include establishment of consistent qualification and evaluation standards. See QT-CA-5.
MS-CA-7 Develop a plan for involving staff in continuous improvement of reactor operations, through participation in a preventive action program that encourages and rewards proactive efforts to improve quality, safety, and efficiency of operations.
4
NCNR management is requiring staff participation in all program improvements, beginning with the CARRI program, is planning on continued extensive staff participation and ownership in programs, and will require training on continuous improvement. Procedures and training programs are being redesigned and rewritten to emphasize and amplify robust safety practices (e.g., new AR 1.1 Human Performance Tools).
Continuous improvement and enhancement of safety culture is an overarching goal of the corrective actions presented here and includes a cultural change incorporating continuous improvement. This change will include an incentive program to recognize staff in making proactive improvements and enhanced staff communications and engagement.
MS-SPI-7 Develop a process to communicate SAC recommendations and NCNR actions taken to address recommendations; consider establishing an SEC subcommittee to track corrective and preventive actions implemented in response to recommendations from SAC and other external review committees.
NCNR management will work with the SEC Chair to establish a corrective and preventive action subcommittee to track and audit external recommendations.
Qualification and Training The ERCAS report suggested five training corrective actions and three suggested training program improvements. These are listed below, along with actions the NCNR is taking to address them. Those necessary to implement prior to reactor restart are so noted.
QT-CA-1 Require proficiency training for personnel prior to all refuelings, emphasizing the importance of latching and procedural compliance.
The following proficiency training is now required for all reactor operations personnel and will be completed prior to startup:
- Reactor Startup Operations o This training will standardize the way to operate the reactor, so it is uniform across crews.
- 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> of classroom training (not including other CARRI Team training):
o Procedural Compliance.
o Summary of Major Procedure Changes.
o Refueling.
o Reactor Startup.
o 20-MW Operations.
o Technical Specifications.
- Creation of a Continuous Learning Program.
o This training program will consist of 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> a year of training (training time may be adjusted) which will be modular, hands-on, and classroom style.
o These trainings will not take the place of the requalification plan.
o The writing of Admin Rule 19 will take place to formalize the training program as a whole and to make expectations clear.
5
QT-CA-2 Develop program for robust qualification of operators and candidates in moving fuel.
We are instituting a program, prior to startup, whereby 1) any person in training shall have completed proficiency qualifications, including use of the test stand, and 2) all personnel scheduled to be involved in fuel movements have demonstrated and documented proficiency in fuel movements, including latching, within the last 12 months.
QT-CA-3 Training materials, such as qual cards and experience with use of fuel handling stand, should reflect learning objectives.
The operator and supervisor training programs are being rewritten and redesigned so that knowledge transfer is adequately captured, qualification cards are used consistently, and fuel handling and latching qualifications are clearly spelled out. This will include use of the fuel training stand (to be improved --
See IE-CA-6).
QT-CA-4 Provide consistent and structured training and immediate and continual feedback to trainees during OTJ training to ensure comprehension of performance expectations.
QT-CA-5 Develop consistent standard by which all supervisors evaluate qualifications.
The improved operator training program will apply consistent standards in training operators and in supervisor evaluations and will have specific evaluation performance criteria which will be clearly communicated as part of the training program.
QT-SPI-1: Develop an understanding of the knowledge and skills for inclusion in the training program to meet the job performance expectations.
QT-SPI-2: Identify learning objectives for the training program to support successful job performance.
QT-SPI-3: Continuously evaluate and revise the training based on the performance of licensed SROs on the job.
NCNR management is committed to providing the resources to redesign the training program to provide a continual evaluation of needed knowledge and skills, learning objectives, and knowledge transfer, both from licensed operators and trainees. This will include conducting periodic management reviews of the training program to ensure that it continues to meet training goals.
Procedures The ERCAS report suggested six procedure corrective actions and one suggested procedure program improvement. These are listed below, along with actions the NCNR is taking to address them. All of these will be implemented prior to reactor restart.
PR-CA-1: Rewrite OI 6.1 and OI 6.2 to capture detail of fuel and latch movements to align with training.
OI 6.1 and 6.2 will be rewritten as a reader/worker procedure (conforming with the INPO standard 11-003 - see PR-CA-6) to capture details of fuel and latch movements to make sure all steps are followed.
PR-CA-2: Reinstitute requirement for latch checks prior to final pump restart and modify OI 2.1.
6
OI 2.1 and OI 1.1A CL will be modified to include a requirement that latch checks be done prior to the final pump restart. Any necessary rechecks will always include a final visual check.
PR-CA-3: Institute method of visual checks.
A video-based visual check system is in development. This system will be fully tested and operational with a visual check incorporated into OI 6.1 prior to reactor restart.
PR-CA-4: Institute a redundant rotation latch check, performed by a second individual.
OI 6.1 will be modified to require a second individual do an independent rotational latch check.
To prevent inadvertent unlatching, this second verification check will be made without additional tool movement.
PR-CA-5: Update procedures to require training for all personnel on procedure adherence.
Required training on procedure adherence has been provided for all ROE personnel. This multiple day training will be completed on September 23, 2021.
PR-CA-6: Revise procedures to be consistent with INPO 11-003.
All procedures will be revised to be consistent with INPO 11-003. Personnel revising and reviewing these updated procedures are required to take training on procedure protocols prior to being involved in this process. (note: all procedures determined necessary for reactor restart will be modified prior to restart. The revision of all 500+ reactor procedures will be a longer-term effort.)
PR-SPI-1: Consider providing this training on initial qualification so that operators understand better the importance of adherence to procedures.
All current reactor operator trainees are required to have procedure training. Procedure training will be part of the operator qualification process from now on.
Equipment, tools and instruments The ERCAS report suggested six equipment corrective actions and two suggested equipment program improvements. These are listed below, along with actions the NCNR is taking to address them. Those necessary to implement prior to reactor restart are so noted.
IE-CA-1 Institute a method of visual checks (ERCAS clarification: Specify use of a camera or video camera to provide indisputable proof that each element is fully latched).
As mentioned in PR-CA-3, A video system for latch checking is in development and will be completed prior to reactor startup IE-CA-2 Document that improved latching and latch check processes provide adequate defense against unlatching.
A safety analysis of improvements in latching and latch checks has been made. See Latch Improvement Safety Analysis.
7
IE-CA-3 Modify index plate so that it is consistently positioned in the same place and rotational fiduciary marks are clear.
Index plate metrology is currently being analyzed and we are exploring options in modifying or replacing the index plate, including prototyping a new sheet with improved fiduciary marks. As these measures are still under evaluation, the permanent solution for the index plate is still to be determined. However, corrections will be made to positioning and fiduciary marks prior to reactor startup.
IE-CA-4 Consider discontinuing use of height checks to verify latching Metrology studies have concluded that the current equipment does not support height checks with sufficient resolution (0.050" difference between partial latch in window and correctly latched).
Procedural improvements to latch checks combined with added fiduciary marks to the index plate and inclusion of visual inspection will make height checks obsolete. The height check does not rise to the level of meeting the TS requirement for latch verification and performing the measurement does not provide added value from a QA perspective. Therefore, height checks will no longer be used.
IE-CA-5 Put administrative controls in place (procedures) to assure no tool contact with fuel head following final visual latch verification prior to reactor startup.
OI 6.1 and OI 1.1A CL will be modified prior to reactor startup to assure no tool contact will be made with the fuel head after the final visual latch verification.
IE-CA-6 Increase access to the reactor top for training purposes or redesign/modify existing test stand to better simulate reactor top fuel loading/latching/latch checking experience.
Training, including the use of the test stand, has been examined, with the following recommendations:
- Add two element positions for simulating transfers.
o One of which is a transfer arm position (small transfer arm).
- Have a latch check tool available for the test stand for practicing latch checks.
- Create another T-handle tool for the test stand to practice transfers.
- Have a universal tool for test stand practice and use.
- Create a dropout chute tool simulator.
- Modify the current test stand so switching tools is more straigtforward.
- Create a refresher frequency for trainees and licensed operators to perform actions on the test stand to maintain proficiency.
As many of these require substantial engineering changes, the engineering group will be tasked with making these modifications as soon as practical. In the meantime, all personnel will be required to demonstrate proficiency prior to reactor startup, including using fuel movements on the reactor top as in QT-CA-2.
IE-SPI-1 Explore NI signal analysis tools capable of providing early detection/alarm of abnormal behavior.
Condition-based monitoring systems (CBMS) that could predict mechanical anomalies like a partially latched fuel element have been investigated. We have identified three technological routes capable of detecting an unlatched element: 1) noise gates on the NC channels, 2) software analyses of I&C signals, 8
and 3) acoustic monitoring of the vessel. Of the three, item 1 is the most straightforward to implement and we will install a prototype prior to or soon after reactor startup and evaluate its effectiveness.
Other options have the potential of detecting a wider variety of anomalies and will be explored as part of the ARM upgrade.
IE-SPI-2 Require tool manufacturer to provide accurate dimensional inspection reports for comparison of as-built condition to drawing specifications.
The ROE Engineering Group will henceforth require documented accurate dimensional inspections with the manufacturing of any replacement refueling tools.
Event Response Although the ERCAS report suggested no corrective actions in response to the actual event, there were six suggested program improvements. These are listed below, along with actions the NCNR is taking to address them. Evaluations have concluded that none of these are critical to implement prior to reactor restart.
ER-01 Develop a definitive checklist for use during an evacuation.
Emergency Instruction (EI) 0.4, Emergency Evacuation Checklist, and EI 0.5, Post Evacuation Checklist have been drafted. The checklists are currently being reviewed and will be incorporated into the EIs.
ER-02 Communicate and implement the identified corrective actions identified as part of a separate root cause evaluation of the incident (IRIS 21-IG-0017). (Detect or secure CO2 following an unplanned shutdown)
EI 3.9, Confinement Re-entry has been drafted. The EI includes procedures for evaluating potential hazards before and during re-entry into confinement after an off-normal condition. This will be incorporated into the EIs.
In addition, The possibility of securing CO2 during an unplanned reactor shutdown has been investigated with the conclusion that, as long as the thermal shield system were secured, there would be no unintended consequences for securing CO2 either locally or remotely prior to a building evacuation and thus will be an action incorporated into EI 0.4 (including the securing of the thermal shield system).
ER-03 Develop and implement a plan to improve monitoring and control capabilities to improve the utility and usability of the ECS; consider formalizing the use of the Reactor Data at Your Desktop system for monitoring and assessing plant conditions.
An ethernet-based data display will be implemented post-reactor startup. More significant upgrades of instrumentation, controls, and environment will require additional resources and are under evaluation.
NCNR management will further assess the requirements to more formally utilize the Reactor Data at Your Desktop system. More formal use of the system may require a complete 50.59 review.
ER-04 Develop guidelines that outline methods for making measurements, interpreting results, performing calculations, and making dose projections (e.g., dose projections that are used as basis for 9
radiological protective action recommendations and those used to upgrade and downgrade emergency classes).
These guidelines will be implemented in an update to NBSR Emergency Plan and Emergency Instructions. The NCNR Health Physics group is working with Reactor Operations to develop them, including calculation tools.
ER-05 Develop a process to communicate and track deficiencies [in emergency drills and exercises]
identified during follow-up critiques; ensure corrective and preventive actions are assigned appropriately and tracked for timely resolution.
See ER-06.
ER-06 Ensure emergency drills and exercises are rigorous, diverse and not predictable (i.e., the scenarios do NOT precondition responders to non-credible scenarios); scenarios are implemented in such a way that adequately challenge the emergency response organization (ERO) so as to identify and correct performance deficiencies and thereby enhance ERO performance during an actual emergency.
We will recommence weekly drills (ensuring each shift will see a drill every month) post-reactor startup and will include participation of ROE personnel outside the shift crew when possible. NCNR Management is committed to upgrading emergency exercises and drills to ensure that they are more rigorous and useful in providing training and identifying deficiencies. The drill implementation will include corrective actions assignment and tracking.
Other corrective actions Although not specifically mentioned in either of the root cause reports, there are also several active recovery items and long-term corrective actions being evaluated. See Memorandum on Recovery Items.
References
[1] September 13, 2021, Report to the NCNR Director, Root Cause Investigation of February 2021 Fuel Failure NCNR Technical Working Group, Revision 2.
[2] August 13, 2021, Report to the NCNR Director, SEC Subcommittee Report: Review of the NCNR Event Response and Technical Working Group Root Cause Analysis and Corrective Action Plan 10