IR 05000220/1992081
| ML17056B880 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 05/20/1992 |
| From: | Beall J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML17056B879 | List: |
| References | |
| 50-220-92-81, 50-410-92-80, NUDOCS 9206020144 | |
| Download: ML17056B880 (22) | |
Text
REPORT NOS.
U. S. NUCLEAR REGULATORY COMMISSION
REGION I
-22 2-
-41 2-DOCKET NOS.
LICENSE NOS.
g-220 2=4&1 QPR+$
NPF-LICENSEE:
FACILITYNAME'
MhwkP wr 3ILllifI d
se New York 1 212 in Mil Pin N 1 rPwr i
ni
2 tNS BCTIQ T: ~ik INSPECTION DATES:
F i
- March 4
2 n
INSPECTORS:
D. Caphton, Sr. Technical Reviewer, DRS J. Yerokun, Project Engineer, DRP H. Eichenholz, Sr. Resident Inspector, Vermont Yankee, DRP S. Hansell, Operations Engineer, DRS APPROVED BY:
J.
Beall, Team Leader, Engineering Branch, Division of Reactor Safety
<<h H
S Y.
Date 9206020T44 92052i PDR ADOCK 05000220
TABLE F
NTENT EXECUTIVESUMMARY........... ~..........................
1.0 INTRODUCTION......,
. ~....,.........................
2.0 IMMEDIATECORRECTIVE ACTIONS 2.1 Lessons Learned Training......
2.2 Review of Deviation/Event Reports
3
3.0 SHORT TERM CORRECTIVE ACTIONS 3.1 Deviation/Event Report Process
~
.
3.2 Oversight of Ongoing Work....
3.3 Self-Assessment Activities
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
5
4.0 CONCLUSIONS..... ~................... ~.............
5.0 EXIT MEETING.......................................
Appendix A - Persons Contacted Appendix B - Documents Reviewed Appendix C - Monitored Maintenance Activities
EXECUTIVESUMMARY On February 29, 1992, the NRC dispatched an inspection team to assess the adequacy of the Niagara Mohawk corrective actions following the inadvertent isolation of Nine Mile Point Nuclear Power Plant Unit 1 from the Lake Ontario, the unit's ultimate heat sink.
An augmented inspection team (AIT) was initiallydispatched immediately following the February 21, 1992 event; the AITcompleted its onsite review of the event on February 28, 1992.
The inspection team concluded that the immediate and the short term corrective actions taken by the licensee to address the causes of the event were acceptable.
The causes of the event had been identified by the AIT to be:
(1) Failure to follow the established work control process by various levels of personnel in multiple licensee groups; (2) Inadequate management overview to assure that the workers understood and followed established procedures; (3) Inadequate communications within and among organizations participating in work activities; and (4) Insensitivity to shutdown risk among multiple licensee organizations.
The licensee conducted training for all site supervisory personnel on the February 21, 1992 event.
In most cases, the training was given by the Unit 1 and the Unit 2 plant manager and detailed the mistakes made and the lessons to be learned from the event, including the needs to improve supervisor and management oversight and to understand and control shutdown risk.
The team determined that the training was effective, based on dire'ct observation and post-training interviews.
The team inspected work in progress during the Unit 2 refueling outage.
No significant deficiencies were identified.
Plant impact assessments were found to have good detail, significant supervisory field presence was noted, and control room supervision demonstrated good knowledge of significant work in progress.
The team noted a substantial increase during the inspection in the number of Deviation/Event Reports (DERs) open or awaiting disposition.
The increase was due to the Unit 2 refueling outage and licensee reviews and corrective actions associated with the Unit 1 loss of heat sink event and the Unit 2 loss of control room annunciators event.
The size of the backlog represents a challenge to the system for the prioritization and timely resolution of identified problems.
The licensee's self-assessment of the February 21, 1992 event was of high quality.
The report contained a good review of equipment impact and documented immediate actions taken.
The assessment was self-critical with corrective actions documented or proposed for the identified causal factors.
The assessment audits of the licensee's work control monitoring program also performed well in identifying problems and weaknesses.
The corrective actions initiated from the audits were generally of a long-term nature, such as rewriting existing procedures or drafting new ones.
The team noted that many of the corrective actions initiated from the above assessments were not complete at the close of the inspection such that the effectiveness of the initiatives in correcting the problems identified could not be assesse e
1.0 INTRODUCTION On February 21, 1992, at about 8:30 a.m., licensee personnel inadvertently isolated Nine Mile Point Nuclear Power Plant Unit 1 from Lake Ontario, the unit's ultimate heat sink, by closing all gates that let the water from the lake to the plant's service water bay.
An augmented inspection team (AIT) was dispatched by the NRC to determine the circumstances that led to this event, its causes, safety significance and generic implications, and the adequacy of the licensee's actions before, during, and after the event.
The AITconcluded that the causes for this event were:
(1) Failure to follow established work control process by various levels of personnel in multiple licensee groups; (2) Inadequate management over view to assure that the workers understood and followed established procedures; (3) Inadequate communications within and among organizations participating in work activities; and (4) Insensitivity to shutdown risk among multiple licensee organizations.
The AITcompleted its onsite reviews of the event on February 28, 1992.
For details of the AIT inspection, see NRC Inspection Report No. 50-220/92-80.
Upon completion of the AIT reviews, the NRC dispatched a followup inspection team to monitor and evaluate the effectiveness of the licensee's immediate and short term corrective actions.
The team conducted onsite reviews during the periods February 29 - March 4, 1992, and April 10-18, 1992.
The causes of the Unit 1 event were programmatic in nature, For that reason, the team reviewed the corrective actions taken on a site-wide basis, including inspections of ongoing maintenance activities during the Unit 2 refueling outage.
2.0 IMMEDIATECORRECTIVE ACTIONS The licensee's immediate corrective actions included developing and administering training on the lessons learned from the event, reviewing all outstanding DERs for design issues, and examining all planned work requests (WRs) for the planned Unit 2 refueling outage.
The team assessed the quality and effectiveness of the lessons learned training and conducted an independent review of a sample of outstanding DERs.
The team did not initially review the licensee efforts with respect to WRs because the program was still in the early stages.
2.1 Lessons Learned Training The licensee committed to provide training to site managers, supervisors and licensed operators on the lessons learned from the loss of heat sink event.
The outline for this training was part of the licensee's February 28, 1992 submittal on event root cause and corrective actions.
The training was generally conducted with small, multi-disciplinary groups by each unit's plant manager.
The team observed many of the training sessions and found them to be of good quality.
The sessions at both units communicated that work or other actions not in accordance with approved procedures could place the plant outside the design basis.
All training sessions emphasized that, ifa drawing discrepancy or other potential design issues were identified, the required actions were to stop work, initiate a DER, and wait for resolution.
The personnel at each meeting included all disciplines and levels of supervision, and all meetings resulted in active participation and questioning of senior management on the issues discusse The training at Unit 1 emphasized where the mistakes had been made during the Unit 1 event and the lessons to be learned.
The Unit 2 training was more comprehensive, containing examples of both good and poor practices at Unit 2 and covering the appropriate administrative procedures more thoroughly.
The team conducted several post-training interviews, some immediately following the meetings and others several weeks later.
Based on these interviews, the team determined that the training at both units had been effective in communicating management positions and expectations.
2.2 Review of Deviation/Event Reports The team reviewed the DER procedure ("Nuclear Division Interfacing Procedure NIP-ECA-01, Rev. 03, Deviation/Event Report" ), the list of outstanding DERs, and a sample of DERs in progress.
Interviews were conducted with operators, work planners, engineers, and members of management.
Initially, the interviews revealed some confusion in documentation requirements, for example, who could sign which blocks and when certain signature blocks could be marked "N/A." These weaknesses in understanding were generally resolved by the end of the inspection period.
The DER system was implemented in April 1991.
At the beginning of the inspection about 2000 DERs had been initiated with about 1200 still open.
Of the open DERs, about 200 had been assigned but the corrective actions were not yet complete.
The remainder of the open DERs were either in the initial stage or the assigned actions had been performed but closure and documentation were not yet complete.
After discussions with the team, the licensee reviewed all outstanding DERs and confirmed that no problems existed similar to the one involved in the Unit 1 isolation from heat sink event.
The team conducted an independent review of a sample of open DERs and identified no safety concerns.
3.0 SHORT TERM CORRECTIVE ACTIONS The team reviewed licensee corrective actions in the areas of management of the DER process, oversight of ongoing work, and self-assessment of the ultimate heat sink isolation event.
3.1 Deviation/Event Report Process A large increase in the number of DERs was noted during the six-week period covered by the inspection.
The total number increased from about 2000 to 3300, the number open went from about 1200 to 2000, and the number assigned but not completed went from about 200 to 750.
A review of the DER data indicated that many of the new DERs had resulted from the ongoing Unit 2 refueling outage work, and the licensee's actions in response to the Unit I loss of ultimate heat sink event and the Unit 2 loss of control room annunciators even e
The team reviewed a sample of in-process DERs and identified no significant deficiencies.
The team noted, however, that the large increase in the current number and initiation rate of DERs represented a challenge to the overall DER process.
Additionally, a revision to the DER procedure was approved during the inspection period, but had not been implemented so the effectiveness of the new DER procedure could not be assessed.
3.2 Oversight of Oiigoing %ork One root cause of the Unit 1 isolation of ultimate heat sink event involved problems in management oversight of work activities.
Problems in the understanding of and adherence to work procedure requirements also contributed.
Deficiencies in assessing the possible plant impact of work were a factor in both the Unit 1 isolation of heat sink event and the March 23, 1992, Unit 2 loss of control room annunciators event.
During this inspection, the team reviewed work packages for completed and in-progress jobs, interviewed workers and supervisors, and assessed management oversight of selected jobs and the maintenance program.
The team selected several maintenance activities (see Appendix C) during the Unit 2 refueling outage and monitored them to assess procedure adequacy, work package completeness, and supervisor oversight.
The team attended several pre-job briefings and found them to provide specific and detailed instructions for planned work.
Inspection of ongoing work found compliance with procedures and radiation protection requirements to be good.
One potential weakness was noted with respect to double verification of work steps.
The double verification by quality control (QC)
of two steps in Procedure N2-EMP-GEN-510 during work on valve 2RHS-MOV22A were waived by QC.
This waiver was allowed per Engineering Specification NMP-2-E0161A, Revision 15, "Electrical Installations Specification."
The second signatures waived by QC were not replaced.
As a result, the torque values for the valve actuator bolts were not verified by a second person.
The team discussed this potential weakness with licensee personnel.
The licensee stated that maintenance procedures were currently being revised to delete unnecessary QC signature requirements and to replace the QC signatures with double verification signatures by other personnel such as maintenance co-workers, The team had no further questions.
The team observed a significant level of supervisor field presence associated with maintenance activities.
Discussions with supervisors at several job locations showed them to have a good knowledge and involvement with the ongoing work. The maintenance manager had issued a memorandum on February 28, 1992, which set clear minimum standards for supervisor work observations.
The memorandum called for first line supervisors to make at least three work observations daily, and for second line supervisors to make one similar observation daily. Two of the three maintenance groups kept a log which documented the observations of the supervisors.
The team reviewed the logs and did not identify any significant deficiencie One contributor to the Unit 1 event was a misunderstanding by the control room supervisors of the work associated with the intake structure gate.
During this inspection, control room supervisors demonstrated, during interviews, good knowledge of plant conditions and the potential impact of significant maintenance and testing in progress.
The use of an onshift manager during the Unit 2 refueling outage also improved oversight of ongoing work.
The team reviewed the plant impact assessments contained in the work packages associated with each observed maintenance activity. The impact assessments contained good detail on the effects of the job.
Some of the jobs monitored utilized the same procedure (S-EPM-GEN-27070) as was in progress during the March 23, 1992 loss of control room annunciators.
The weaknesses in the plant impact assessment, which contributed to the March 23, 1992 event were corrected in the case of the work observed on relays associated with emergency diesel generator 2EGS*EG2.
3.3 Self-Assessment Activities The team reviewed three examples of licensee self-assessment activities:
the licensee assessment summary of the Unit 1, February 21, 1992 loss of heat sink event dated March 25, 1992, and the Unit 1 and the Unit 2 work control monitoring program status reports for the period of March 9 - April 6, 1992.
The licensee event assessment was found to be of high quality.
The report was self-critical with corrective actions documented or proposed for the identified causal factors.
The impact of the event on equipment was examined in detail in the report with acceptable engineering bases for the conclusions given.
The components potentially affected included the service water pumps, emergency service water pumps, fire protection pumps, and certain secondary plant pumps.
Additionally, the assessment specified confirmatory physical examinations and operational tests to be performed.
The team noted that these activities had been completed successfully prior to Unit 1 restart.
The team identified no significant deficiencies in the assessment of the event or in the area of short term corrective actions.
The team noted that several long term corrective actions such as procedure revisions and process reviews had not been completed or, in some cases, had not begun.
One specific example was a future self-assessment of corrective action effectiveness.
The team concluded that the licensee's internal commitment to these long term initiatives was a strength, but that the adequacy of implementation or effectiveness could not be assessed at this time.
The team reviewed the performance of the licensee's work control monitoring program.
This program was one of the corrective actions to the Unit 1 loss of heat sink event.
The program consists of one audit per week per unit of a selected portion of the work control program.
Each audit was conducted by a group of supervisory personnel and led by a senior manager or executive.
At the time of the inspection, four audit results were available from each unit for revie The initial audits of Unit 1 work emphasized performance-based observations while the Unit 2 audits were more directed at documentation issues.
This difference in approach was generally resolved by the end of the inspection with both unit programs using a combination or procedural compliance and performance based attributes.
The audits were found to be thorough within the scope selected for review.
Each audit initiated DERs to correct deficiencies or to address identified concerns.
The audits also identified other action items, each termed "Opportunity for Improvement."
Collectively, the scope of corrective actions required to address and resolve the concerns of the audits is significant.
The forty-one DERs and twenty-five Opportunities documented in the sampled audits included the development of formal guidance, the creation of new procedures, and revisions to existing procedures.
The team concluded that the work control monitoring program appeared to be an effective management tool for the identification of deficiencies and weaknesses in the areas selected for examination.
The type and volume of audit findings were consistent with the findings of the NRC AIT (Inspection Report 50-220/92-80) and the licensee's own event assessment.
These findings are indicative of the need for improvement in the work control program.
The DERs and Opportunities for Improvement initiated from the work control monitoring program generally had not been completed at the end of the inspection period, so the effectiveness of the program in correcting the problems identified could not be assessed at this time.
4.0 CONCLUSIONS Overall, the team concluded that the immediate and short-term corrective actions taken by Niagara Mohawk had been effective in addressing the root causes of the February 21, 1992 Unit 1 loss of ultimate heat sink event.
Specifically, training given to site supervisory and operating personnel had increased worker sensitivity to shutdown risk and supervisor oversight of ongoing maintenance.
The self assessments conducted by the licensee were effective in identifying additional needed corrective actions.
The team was unable to assess the total effectiveness of the self-assessment activities because many of the proposed corrective actions were of a long-term nature and were not complete.
5.0 EXITMEETING Exit meetings were held with the licensee at the conclusion of each period of onsite review, on March 4 and on April 1S, 199 APPENDIX A PER NTA TED h
kP r
i n
~Pjiiin R. Abbot C. Beckham J. Burton M. Conway R. Crandall K. Dahlberg A. DeGracia S. Doty J. Endries M. Eron J. Firlit D. Hosmer J. Kinsley M. McCormick, Jr.
J, Pavel L. Pisano J. Savoca R. Slade J. Spadafore R. Sylvia C. Terry T. Tomlinson S. Wilczek, Jr.
Manager, Engineering, Unit 2 Manager, QA Operations, Unit 2 Manager, QA Operations, Unit 1 Station Shift Supervisor, Unit 2 System Engineer Unit 1 Plant Manager Operations Manager, Unit 2 General Supervisor, Electrical Maintenance, Unit 2 President, Niagara Mohawk Corporation Assistant Station Shift Supervisor, Unit 2 Vice President - Nuclear Generation Unit 2 Manager Outage/Wk Control General Supervisor, Instrument and Controls, Unit 2 Unit 2 Plant Manager Site Licensing Engineer Unit 1 Outage Manager General Supervisor, Mechanical Maintenance, Unit 2 General Supervisor, Training Operations, Unit 2 Program Director, ISEG Executive Vice President - Nuclear Vice President - Nuclear Engineering Supervisor, Reactor Engineering, Unit 2 Vice President - Nuclear Support Asterisk denotes presence at the April 18, 1992 preliminary exi APPENDIX B D
ME RE IE D
1.
Unit 1 Chief Shift Operator and Station Shift Supervisor logs for February 28-March 4 2.
Unit 2 Chief Shift Operator and Station Shift Supervisor logs for April 10-16, 1992 3.
Unit 2 Work Instructions for April 9-17, 1992 4.
Administrative Procedure AP-5.2.3, Rev. 03, Preventive Maintenance Program 5.
Administrative Procedure AP-5.2.5, Rev. 01, Work in Progress (WIP)
6.
Administrative Procedure AP-5.4, Rev. 04, Conduct of Maintenance 7.
Administrative Procedure AP-5.4.2, Rev. 02, Troubleshooting 8.
Administrative Procedure AP-5.5, Rev. 02, Work Control 9.
Administrative Procedure AP-5.5.1, Rev. 06, Work Request 10.
Nuclear Division Interfacing Procedure NIP-ECA-01, Rev. 03, Deviation Event Report 11.
Generation Administrative Procedure GAP-OPS-01, Rev. 00, Administration of Operations 12.
Electrical Preventive Maintenance Procedure S-EPM-GEN-2Y070, Revision 1, dated 05/17/88; with Data Sheets, Test Results Information Sheets, and Work In Progress Data Sheets for the following components:
13.
Work Control Monitoring Program Plan for Nine Mile Point Unit One and Unit Two, Revision 0, March 17, 1992 14.
Nine Mile Point 1 Assessment Summary of the February 21, 1992 Loss of Heat Sink Event 15.
Unit 1 Deviation/Event Reports 1-91-Q-0685, 1532, 1560, 1616, 1762, 1767, and 1-92-Q-0190, 0265, 0314 16.
Unit 2 Deviation/Event Reports 1-92-Q-0101, 0111, 0112, 0115, 0136, 0167, 0214, 0235
PPENDIX RED MAINTENAN E A TIVITI 1.
Work Request (WR) 203884 on valve 2RHS-MOV22A 2.
WR 201613 on valve 2FWS-MOV21A 3.
WR 196692 on valve 2RDS-V17B 4.
WR 203806 on valve 2CSL-RV123 5.
WR 200679 on valve 2RHS-PV21A 6.
WR 190838 on penetration fire seals 7.
WR 194467 on RHR radiation monitors 8.
Preventive Maintenance ActivityS-EPM-GEN-2Y070 on 2EGS~EG2 relays Kl, K4,
, K6, K7, K14, K16 9.
Preventive Maintenance ActivityN2-EPM-GEN-R555 on GE Magne-Blast breakers and associated motors under activity 2NPS-SWG-003-14 10.
Preventive Maintenance ActivityN2-EPM-GEN-V712 for 600V Motor Control Center Bus Work on 2NHS-MCC009 Preventive Maintenance ActivityN2-EPM-GEN-10Y711 for 600V Load Center Bus Work on 2NJS-US6 12.
Preventive Maintenance ActivityN2-EPM-GEN-V712 for 600V Motor Control Center Bus Work on 2NHS-MCC012