IR 05000220/2002009
| ML022800456 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 10/07/2002 |
| From: | Linville J NRC/RGN-I/ORA/MID |
| To: | Conway J Nine Mile Point |
| References | |
| IR-02-009 | |
| Download: ML022800456 (23) | |
Text
ber 7, 2002
SUBJECT:
NINE MILE POINT NUCLEAR STATION - SUPPLEMENTAL INSPECTION FOR WHITE PERFORMANCE INDICATORS REPORT 50-220/02-009, 50-410/02-009
Dear Mr. Conway:
On August 23, 2002, the NRC completed a supplemental inspection at the Nine Mile Point Nuclear Station (NMPNS). The enclosed report documents the results of the inspection, which were discussed with you and other members of your staff on August 23, 2002.
The NRC performed this supplemental inspection to assess NMPNS evaluation of the white performance indicators associated with the unavailability of the Unit 1 High Pressure Coolant Injection System, and the Unit 2 Scrams. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspectors determined that NMPNS performed comprehensive evaluations of the causes of the issues. The NMPNS evaluation identified the primary root cause of the white performance indicators to be long-standing cultural issues relating to low standards and acceptance of equipment problems.
Given NMPNS acceptable performance in addressing the underlying causes of the unavailability of the Unit 1 high pressure coolant injection system, the white performance indicator associated with this issue will only be considered in assessing plant performance until the trend crosses below the threshold back into the green level, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program. Since the corrective actions have not yet been fully implemented, it is inappropriate for NMPNS to reset the fault exposure hours at this time.
Given NMPNS acceptable performance in addressing the underlying causes of the Unit 2 scrams, the white performance indicator associated with this issue will only be considered in assessing plant performance until the trend crosses the threshold back into the green level, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program.
Based on the results of this inspection, the NRC has determined that one Severity Level IV violation of NRC requirements occurred. The NRC has identified one additional issue that was evaluated under the risk significance determination process as having very low safety significance (Green). The NRC has also determined that a violation is associated with this issue. However, because of their very low safety significance and because they have been entered into your corrective action program, the NRC is treating these issues as Non-cited
Mr. John violations, in accordance with Section VI.A.1 of the NRCs Enforcement Policy. If you deny these Non-cited violations, you should provide a response with the basis for your denial, within 30 days of the date of this inspection report, to the Nuclear Regulatory Commission, ATTN:
Document Control Desk, Washington, DC 20555-001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the NMPNS facility.
In accordance with 10 CFR 2.790 of the NRCs Rules of Practice, a copy of this letter and its enclosure will be available electronically for public inspection at the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
James C. Linville, Chief Electrical Branch Division of Reactor Safety Docket Nos. 50-220, 50-410 License Nos. DPR-63, NPF-69 cc w/encl: M. J. Wallace, President, Nine Mile Point Nuclear Station, LLC R. L. Wenderlich, Senior Constellation Nuclear Officer Responsible for Nine Mile Point G. Wilson, Esquire M. Wetterhahn, Esquire, Winston and Strawn J. M. Petro, Jr., Esquire, Counsel, Constellation Power Source, Inc.
J. Rettberg, New York State Electric and Gas Corporation P. Eddy, Electric Division, NYS Department of Public Service C. Donaldson, Esquire, Assistant Attorney General, New York Department of Law J. Vinquist, MATS, Inc.
W. M. Flynn, President, New York State Energy Research and Development Authority J. Spath, Program Director, New York State Energy Research and Development Authority Supervisor, Town of Scriba C. Adrienne Rhodes, Chairman and Executive Director, State Consumer Protection Board T. Judson, Central NY Citizens Awareness Network
Mr. John Distribution w/encl:
Region I Docket Room (with concurrences)
G. Hunegs, SRI - NRC Resident Inspector H. Miller, RA J. Wiggins, DRA M. Evans, DRP H. Nieh, RI EDO Coordinator S. Richards, NRR (ridsnrrdlpmlpdi)
P. Tam, PM, NRR D. Skay, PM, NRR (Backup)
N. Perry, DRP P. Torres, DRP R. Junod, DRP W. Lanning, DRS R. Crlenjak, DRS DOCUMENT NAME: G:\ELECTRICAL\FUHRMEISTER\NMPNSPIS.WPD After declaring this document An Official Agency Record it will be released to the Public.
To receive a copy of this document, indicate in the box: "C" = Copy without attachment/enclosure "E" = Copy with attachment/enclosure "N" = No copy OFFICE RI:DRS N RI:DRS RI:DRP RI:DRS NAME RFuhrmeister ECobey MEvans JLinville DATE 09/26/02 10/02/02 10/03/02 10/07/02 OFFICIAL RECORD COPY
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
Docket Nos: 50-220, 50-410 License Nos: DPR-63, NPF-69 Report No: 50-220/02-009, 50-410/02-009 Licensee: Nine Mile Point Nuclear Station, LLC Facility: Nine Mile Point Units 1 and 2 Location: P.O. Box 63 Lycoming, New York 13093 Dates: August 19 - 23, 2002 Inspectors: R. Fuhrmeister, Sr. Reactor Inspector, DRS K. Mangan, Reactor Inspector, DRS D. Beaulieu, Sr. Resident inspector Approved by: James C. Linville, Chief Electrical Branch Division of Reactor Safety
SUMMARY OF FINDINGS
IR 05000220/02-009, IR 05000410/02-009, on 8/19-23/02, Nine Mile Point Units 1 & 2;
Supplemental Inspection. Inspection Procedure 95001, Inspection for One or Two White inputs in a Strategic Performance Area.
The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
Cornerstone: Initiating Events, Mitigating Systems
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess Nine Mile Point Nuclear Stations (NMPNS) evaluation of the white performance indicators associated with the unavailability of the Unit 1 High Pressure Coolant Injection System, and the Unit 2 Scrams. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspectors determined that NMPNS performed comprehensive evaluations of the causes of the issues. The NMPNS evaluation identified the primary root cause of the white performance indicators to be long-standing cultural issues relating to low standards and acceptance of equipment problems.
Given NMPNS acceptable performance in addressing the underlying causes of the unavailability of the Unit 1 high pressure injection system, the white performance indicator associated with this issue will only be considered in assessing plant performance until the trend crosses below the threshold back into the green level, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program. Since the corrective actions have not yet been fully implemented, it is inappropriate for NMPNS to reset the fault exposure hours at this time.
Given NMPNS acceptable performance in addressing the underlying causes of the Unit 2 scrams, the white performance indicator associated with this issue will only be considered in assessing plant performance until the trend crosses the threshold back into the green level, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program.
A Severity Level IV violation of 10 CFR Part 50.9(a) (Completeness and Accuracy of Information), dispositioned as a non-cited violation, was identified for failure to report the correct unavailability hours associated with the High Pressure Coolant Injection (HPCI) system. The information submitted to the NRC by the licensee in the first quarter of 2001, to fulfill requirements of the NRC Performance Indicator Program, was not correct. Specifically, while performing testing on the HPCI system on April 21, 2001, the systems controller was found inoperable. The licensee reported no unavailability hours related to this fault because the system was not required to be operable, due to the plant configuration, during the test period.
This is contrary to the requirements of the program in which the fault exposure time (the time between last successful operation of the system and determination of the fault) must be accounted for in the unavailability report.
This finding is more than minor because the actual unavailability hours of the HPCI system changed the NRC Performance Indicator color from Green to
- White.
As a result the licensees ii
mitigating systems cornerstone did not properly reflect its status as a degraded cornerstone for months. Additionally, the performance of a Supplemental Inspection by the NRC related to the controller failure was not conducted until 16 months after the occurrence. (Section 02.01.1.c)
GREEN: A violation of 10 CFR 50, Appendix B, Criterion XVI, (measures be established to assure that conditions adverse to quality are promptly identified and corrected) dispositioned as a non-cited violation was identified for failure to implement corrective actions for valve packing leaks inside the drywell which caused several plant shutdowns. Specifically, the failure to take effective corrective actions in a timely manner to address packing leakage from valve 2RCS*MOV18A that was observed in September 2000 and March 2001 led to the December 15, 2001, reactor scram which was a self-revealing event.
This finding is more than minor because it could reasonably be viewed as a precursor to a more significant event, which in this case was a 6 gpm reactor coolant system (RCS) leak that necessitated a manual reactor scram (an initiating event) on December 15, 2001. This issue was evaluated using Phase 1 of the reactor safety significance determination process (SDP)and determined to be of very low safety significance (green). Although the issue increased the transient initiator contributor, it did not contribute to the likelihood of a loss-of-coolant accident, it did not contribute to the likelihood mitigating equipment or functions would be unavailable and it did not increase the likelihood of an external event. (Section 02.01.2.c)iii
Report Details 01
INSPECTION SCOPE
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess NMPNS evaluation associated with the unavailability of the Unit 1 High Pressure Coolant Injection System and the Unit 2 Scram Performance. These performance issues were previously characterized as white in the quarterly performance indicators.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification
a. Determination of who identified the issue and under what conditions The inoperability of the HPCI System was identified as a result of the biennial surveillance test performed by NMPNS on April 22, 2001. The test found the controller output amplifier had failed and the cause was determined to be a capacitor in the amplifier circuit. In December 2001 NMPNS identified that the PI for HPCI unavailability resulting from the April failure was incorrectly reported to the NRC. On January 14, 2002, NMPNS identified a second failure of the controller during a newly instituted weekly operability test.
b. Determination of how long the issue existed, and prior opportunities for identification NMPNS first analysis of the issue failed to properly evaluate the unavailability and fault exposure associated with the controller failure and the unavailability was not accurately reported to the NRC. In December 2001 while reviewing an unrelated HPCI issue NMPNS realized that the controller failure had resulted in a significant fault exposure previously unidentified. NMPNS reviewed the operation of the HPCI system and determined that the last successful operation of the train was September 24, 2000. No other testing or monitoring of the system was conducted. This resulted in the No. 11 HPCI train being unavailable for 1619.3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />. The unavailability was reported to the NRC for PI - HPCI System Unavailability - in January 2002. The inspector agreed with the final NMPNS evaluation.
c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue NMPNS evaluated the effect of not having HPCI available by use of the PRA model.
The model was adjusted to account for the time that automatic actuation of HPCI was out of service, and for the manual actions which would have been required to activate and control the system. The NMPNS evaluation assigned a change in core damage frequency of 3.85X10(-7) to the two failures. The team reviewed the NMPNS evaluation and assumptions and confirmed their validity.
Introduction Severity Level IV. A non-cited violation of 10 CFR Part 50.9(a), (Completeness and Accuracy of Information), was identified for failure to accurately report the unavailability hours associated with the licensees HPCI system. This information is submitted to the NRC by the licensee as part of the NRC Performance Indicator Program.
Description The information submitted to the NRC by NMPNS in the first quarter of 2001, to satisfy the NRC Performance Indicator Program, was not correct. Specifically, on April 22, 2001, NMPNS performed biennial testing on the Unit 1- No. 11 HPCI train. While conducting the testing the systems controller was found inoperable due to a defective capacitor. The controller was repaired and retested. NMPNS did not record any unavailability hours because the technical specifications did not require the system to be operable in a shutdown plant condition. This is contrary to the requirements of the program. The reported unavailability hours must include any fault exposure time after system component failures are discovered. The unavailability time should have identified as fault exposure time based on half of the period between the last successful operation of the HPCI system, September 24, 2000, and the plant shutdown on March 18, 2001.
In December 2001 while discussing an unrelated HPCI issue NMPNS realized that the unavailability report was incorrect. NMPNS recalculated the unavailability hours and reported the correct unavailability hours to the NRC in January 2002. The team reviewed the corrected report and found no errors.
Analysis This issue is more than minor because the actual unavailability hours of the HPCI system changed the Performance Indicator color from Green to White. As a result the Unit 1 mitigating systems cornerstone did not properly reflect the status of the cornerstone as a degraded cornerstone for 9 months. Additionally, the performance of a Supplemental Inspection by the NRC related to the controller failure was delayed until 16 months after the occurrence.
Enforcement 10 CFR Part 50.9(a), (Completeness and Accuracy of Information) requires, in part, that information provided to the Commission by a licensee shall be complete and accurate in all material aspects. Contrary to this requirement, after the failure of the HPCI controller, NMPNS incorrectly reported the hours of unavailability of the system to the NRC for the PI indicator in April 2001. This issue is classified as a Severity Level IV violation in accordance with Supplement VII.D of the NRC Enforcement Policy.
However, because NMPNS corrected the error in January 2002 and because the issue was entered into the NMPNS corrective action program (DER 2001-6004), it is being treated as a Severity Level IV, Non-Cited violation, consistent with Section VI.A.1 of the NRC Enforcement Policy. (NCV 50- 220/02-09-01)
.2 Unit 2 Scrams
a. Determination of who identified the issue and under what conditions The white PI for unplanned scrams was self revealing through NMPNS collection of performance indicator data taken in support of the NRCs reactor oversight program (ROP). Each of the four reactor scrams that caused the PI to cross the white threshold was also self-revealing.
b. Determination of how long the issue existed, and prior opportunities for identification The Licensee Event Report (LER) and DER for each of the four reactor scrams appropriately documented how long the conditions that led to the reactor trips existed and prior opportunities to identify the conditions. To gain broader insights into the underlying causes of the scrams, NMPNS collective significance analysis evaluated all 14 scrams that occurred at Unit 2 from 1995 to 2001. NMPNS also retained a consultant to review Unit 1 scrams from 1995 to 2001 and Unit 2 scrams from 1989 to 1994.
c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue The LER and DER for each reactor scram appropriately documented the plant specific risk consequence. When assessed individually, each reactor scram was of very low risk significance (green.) The risk associated with the unplanned reactor scram PI is related to the frequency of this initiating event. Initiating events are those events that upset plant stability and challenge critical safety functions. If not properly mitigated, and if multiple barriers are breached, a reactor accident could result which may compromise the public health and safety.
The inspector determined that the LER and DER for the four Unit 2 reactor trips appropriately discuss compliance concerns. LER 50-410/2001-001-00 (EHC system relay failure) and LER 50-410/2001-004-00 (MSIV closure during main steam flow transmitter calibration) were closed and were dispositioned from an enforcement perspective in NRC Inspection Reports 50-410/2001-007 and 50-410/2001-008. The two remaining LERs associated with the Unit 2 white PI were reviewed as part of this supplemental inspection as discussed in section 04.1 of this report.
Introduction A violation of 10 CFR 50, Appendix B, Criterion XVI (measures be established to assure that conditions adverse to quality are promptly identified and corrected) dispositioned as a non-cited violation, was identified for failure to implement corrective actions for valve packing leaks inside the drywell which caused several plant shutdowns. Specifically, the failure to take effective corrective actions in a timely manner to address packing leakage from valve 2RCS*MOV18A that was observed in September 2000 and March 2001 led to the December 15, 2001, reactor scram which was a self-revealing event.
Description On December 15, 2001, packing leakage from valve 2RCS*MOV18A, the reactor coolant system (RCS) recirculation loop A discharge block valve, caused operators to commence a reactor shutdown and then to scram the reactor at 60 percent power when unidentified drywell leakage reached 4 gallons per minute. The associated LER indicates that the packing leakage had been noted previously during planned outages in September 2000 and March 2001 but the leakage was not adequately addressed. The corrective action was to manually back seat the valve, which did not change the leakage rate. In each case, NMPNS failed to generate a DER or work order to document the leaking packing. Later in July 2001, a work order was generated to repack the valve.
However, this work was added to the Spring 2002 work scope but not the forced outage work scope list resulting in missed opportunities to correct the leakage during subsequent shutdowns. Corrective actions included installing a different type of packing on valve 2RCS*MOV18A and three other similar RCS valves. Corrective actions to address the longstanding problem of packing leakage include developing a common site packing program procedure and assigning a component engineer within Mechanical Maintenance Department with site responsibility for valve packing technical requirements, field implementation, and oversight of the craft. Corrective actions also included ensuring that known leaks in the packing for valves that are critical to generation or reactor coolant system integrity are promptly scheduled for maintenance under the work prioritization and scheduling process.
Analysis This issue is more than minor because it could be reasonably be viewed as a precursor to a more significant event, which in this case was a 6 gpm RCS leak that necessitated a manual reactor trip (an initiating event) on December 15, 2001. This issue was evaluated using Phase 1 of the reactor safety significance determination process (SDP)and determined to be of very low safety significance (green). Although the issue increased the transient initiator contributor, it did not contribute to the likelihood of a loss-of-coolant accident, it did not contribute to the likelihood that mitigating equipment or functions would be unavailable and it did not increase the likelihood of an external event.
Enforcement 10 CFR 50, Appendix B, Criterion XVI, requires that measures be established to assure that conditions adverse to quality be promptly identified and corrected. The failure to take effective corrective actions in a timely manner to address packing leakage from valve 2RCS*MOV18A that was observed in September 2000 and March 2001 is a violation of this requirement. This Severity Level IV violation was identified by NMPNS during their analysis of the December 15, 2001, reactor scram which was a self-revealing event. This licensee-identified violation is more than minor because it could reasonably be viewed as a precursor to a more significant event, which in this case was a 6 gpm RCS leak that necessitated a manual reactor scram (an initiating event) on December 15, 2001. Because this was a licensee-identified violation, is of very low safety significance (green), and was entered into their corrective action program as DER NM-2002-1830, this violation is being treated as Non-Cited, consistent with Section VI.A.1 of the NRC Enforcement Policy. (NCV 50-410/02-09-02).
02.02 Root Cause and Extent of Condition Evaluation
a. Evaluation of method(s) used to identify root cause(s) and contributing cause(s).
The inspectors determined that the methods NMPNS used for their analysis provided for a thorough evaluation of the causes of the HPCI unavailability. NMPNS analyzed the causes of the capacitor and controller failures to determine root causes, and contributing causes. The dominant cause areas were then developed into an issue for further evaluation. Each issue was then subjected to a Why Staircase Analysis to identify underlying causes. To develop corrective actions, the causes were related to the corrective action type such as Culture, Process/Program, Procedural, Training, and Organization. Specific corrective actions tailored to the corrective action type were developed to address each cause.
b. Level of detail of the root cause evaluation The team determined that the NMPNS evaluation was thorough and identified several root causes related to the initial and subsequent events associated with the HPCI controller failures. A total of fifty-four
- (54) corrective actions were identified by NMPNS as a result of the four DERs written concerning the issue. The extent of condition included all safety significant systems and several issues related to problem identification and resolution. Additionally, NMPNS found that the root causes of the performance issues identified were not limited to the HPCI controller. NMPNS identified that poor test procedures, failure to proceduralize the NRC Performance Indicator Program, poorly implemented corrective action program, and no preventive maintenance program for the controller were all performance issues that contributed to the failures.
c. Consideration of prior occurrences of the problem and knowledge of prior operating experience The NMPNS initial evaluation included a review to determine if similar problems had previously been reported concerning the HPCI controller. NMPNS noted that this type of controller had failed in several other locations on site and there were similar reports throughout the industry of problems with this style controller. NMPNS, however, discounted this information since they had not previously had a problem with the HPCI controller. As a result only the capacitor was replaced. After the second failure of the controller, in January 2002, NMPNS thoroughly evaluated the previous controller failures and noted the failure to correctly evaluate the information related to this controller after the first failure was one of the root causes for the second failure.
d. Consideration of potential common cause(s) and extent of condition of the problem The NMPNS evaluation after the second failure did consider the potential common cause and extent of condition associated with the failed controller style for safety related components. However, NMPNS failed to consider additional risk significant components in their evaluation. During the inspection NMPNS agreed to reevaluate the extent of condition of this controller style to include risk significant systems based on the sites maintenance rule scoping documents. Additionally, NMPNS has begun a site wide program to perform preventive maintenance on all capacitors that could affect a systems operability.
.2 Unit 2 Scrams
a. Evaluation of method(s) used to identify root cause(s) and contributing cause(s).
The inspectors determined that methods NMPNS used for the collective significance analysis provided for a thorough evaluation of the common causes of the Unit 2 reactor scrams. The licensee analyzed the causes of the reactor scrams and a database was developed to capture specific information about each scram including the cause codes, system identification data, root causes, and contributing causes. The data was queried using different sorts to look for patterns and gain an understanding of the dominant contributors to the reactor scrams. The dominant cause areas were then developed into an issue for further evaluation. Each issue was then subjected to a Why Staircase Analysis to identify underlying causes. To develop corrective actions, the causes were related to the corrective action type such as Culture, Process/Program, Procedural, Training, and Organization. Specific corrective actions tailored to the corrective action type were developed to address each cause.
b. Level of detail of the root cause evaluation The inspectors determined that the root cause analyses documented in the DER for each individual reactor scram was sufficiently detailed to identify the root causes and contributing causes of each scram. In addition, the information gathered for each scram to support the collective significance analysis was sufficiently detailed to allow the licensee to develop meaningful insights into the underlying causes of the reactor scrams.
As described above, the licensee developed a database that included cause codes, system identification, root causes, and contributing causes for each reactor scram. The licensee appropriately recognized that while the database was a good tool in identifying broad areas of commonality, they also provided briefings to all team members about the details of each reactor scram to allow for a more thorough assessment of common causes.
c. Consideration of prior occurrences of the problem and knowledge of prior operating experience As discussed further in Section 02.02.2.a, their collective significance analysis provided a thorough evaluation of recurrent equipment problems and operating experience.
Industry or in-house experience was inadequate to prevent recurrence.
d. Consideration of potential common cause(s) and extent of condition of the problem Based on their evaluations, NMPNS determined that there were multiple causes for the scrams. The underlying causes were determined to be organizational in nature, and related to a long-standing culture at the station. The NMPNS causes are listed below:
Cause A1: There is little emphasis on driving to find true cause of equipment problems because budgets, policies and goals are not established with a focus on resolving long standing or recurring equipment problems. The current budgeting process often does not give priority to reducing reactor scrams and improvements that have a long-term reliability pay back. In addition, until recently, there has not been a long-term plan to improve overall plant reliability and this concept has not been fully internalized by all levels of station management. Solid links do not exist between the business plan and stations budgets and initiatives that result in a vertically aligned organization with intolerance to unanticipated equipment failures.
Cause A2: The corrective action program has not been effectively implemented when addressing previously identified problems because root cause evaluations were too narrow and station personnel have a tendency to rush through them to meet deadlines, which results in a short term operational focus. Management does not periodically review root cause progress, resolve teamwork and resource constraints, or hold personnel accountable for completing supporting activities for the resolution of long-term equipment problems Cause A3: The corrective action program has been ineffectively used in evaluating industry and in-house operating experience (OE). Station personnel tend to not value operating experience as a method to prevent recurrence of an event. As a result, recommendations are commonly not implemented. Policy guidance for effective use of the OE program is not well defined. Management expectations for the routine use of operating experience have not been established at the corporate, station management, and worker levels. The organization accepts risks without contingency or mitigating actions.
Cause B1: The site has not been able to adequately implement a risk averse environment in all phases of plant operation and maintenance. Because of the focus of keeping the unit running and meeting schedule, the site does not consistently evaluate the potential adverse consequences of an activity when making a decision.
Cause B2: Site management has not fostered or encouraged strong issue ownership and the use of collaborative teams to help solve longstanding and repeat equipment problems. Because of the focus on quickly getting the unit back on line after a performance upset, there is little emphasis placed on bringing the right resources together to find true cause and implement well balanced risk informed decisions to improve long term site performance.
Cause B3: Due to ineffective management of site resources, process controls and management oversight are sometimes inadequate when corrective actions warrant the development of programs for maintaining plant equipment. Examples include a feedwater pump monitoring program and a valve packing program.
Cause C1: Management actions encouraged quick solutions to problems rather than establishing expectations for fixes that focus on long-term performance improvement.
This focus on short-term solutions has inhibited coordination of activities between station organizations, especially in the area of coordinating the implementation of modifications.
Cause C2: The station has not established long-term performance improvement goals that include doing the right maintenance at the right time for important station equipment. The technical bases for preventive maintenance tasks and frequencies are not well defined, documented, or easily accessible. A well understood and thorough process is not used to review deferrals, changes, additions, and deletions of preventive maintenance tasks.
NMPNS collective significance analysis provided other insights into the causes of the scrams. From 1995 to 2000 only one scram was caused by direct active human error.
This scram occurred in 1995 when the incorrect circuit breaker was opened due to poor self-checking. Three reactor scrams in 2001 were caused by component failures due to existing latent conditions (EHC relay failure; RCS valve packing leakage, and feedwater pump motor failure). The findings and recommendations of the consultant that reviewed Unit 1 scrams from 1995 to 2001 and Unit 2 scrams from 1989 to 1994 were consistent with NMPNS findings and recommendations. The dominant systems causing the scrams were the main turbine and main feedwater systems.
Based on the above analysis, the team agreed with the licensee conclusion that site management has not translated strategies into focused initiatives that support the business plan. This situation has resulted in low standards and expectations and a short-term operational focus. Therefore, the site has not achieved and sustained high equipment reliability.
02.03 Corrective Actions
a. Appropriateness of corrective action(s)
The NMPNS immediate corrective actions to make the HPCI system operable were shortsighted and did not address the extent of condition of the issue. NMPNS replaced the failed capacitor but did not identify the age related problems associated with other components in the controller. The corrective actions as a result of the second failure, in which the controller was replaced with a refurbished controller in combination with the evaluation of the incorrect reporting of the HPCI unavailability has adequately addressed the issues with the equipment and associated programmatic failures.
NMPNS established a new test procedure to ensure the operability on a weekly basis.
This was done to limit the possible fault exposure time for a failed component.
Additionally, NMPNS intends to install test points to continuously monitor the HPCI control system via the plant computer. NMPNS also plans to replace all controllers of this style in risk significant systems with a new model.
NMPNS has also taken corrective actions to address the PI program failure to properly evaluate the fault exposure of the HPCI system. These include training on the PI for key personnel and creating a procedure that is used to evaluate systems in the NRC PI program.
Finally, NMPNS has taken appropriate steps to improve the corrective action program.
These include training key personnel in root cause analysis techniques, developing root cause analysis team leaders and revising the corrective action procedures.
b. Prioritization of corrective actions The NMPNS immediate corrective actions after the failure of the HPCI controller on both occasions restored the system to operability within the technical specification allowed outage time. Additionally NMPNS instituted a weekly operability test to limit the potential fault exposure of the system resulting from an undetected failure.
c. Establishment of a schedule for implementing and completing the corrective actions NMPNS has established a schedule to implement several long range plans. They include the Capacitor Replacement Program, Root Cause Analysis Training, Preventive Maintenance Optimization (PMO) Program and Controller Replacement. The team reviewed the NMPNS plans for accomplishing these activities and found the programs have a time line for implementation and, when completed, will address the contributing causes that had been established in the DERs.
d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence NMPNS has established a monthly test to verify the operability of the HPCI controller and the biennial test has been corrected to ensure that all components in the control system are tested. Additionally, the system has been placed in Maintenance Rule a(1)status. New train level unavailability and functional failure limits are being monitored.
.2 Unit 2 Scrams
a. Appropriateness of corrective action(s)
Corrective actions for each individual scram were appropriate for the specific component failure which caused the scram. Broader focused corrective actions to address the underlying cultural issues were developed as follows:
Action No. 1: Implement an initiative to raise site standards, expectations and behaviors for work groups and individuals with respect to equipment reliability and reducing frequency of scrams. The initiative should include: Establishing a comprehensive set of aggressive, measurable goals and strategies; assigning each goal an owner; establishing clearly defined actions plans to close performance gaps; creating a communications plan to stress the importance that unanticipated equipment failures are unacceptable and the importance of fixing problems right the first time to avoid future operational challenges. Due Date: June 30, 2003 Action No. 2: Improve the thoroughness and quality of all site root cause evaluations by ensuring root cause investigations are performed, or mentored, by a minimum number of highly trained root cause team leaders who are available on short notice to respond to site events. Due Date: December 20, 2002 Action No. 3: To help the site to focus on doing the right work at the right time, establish a Plant Health Committee. The purpose of the committee will be to help bring consistency to prioritizing work and help the site focus resources on systems needing attention. Due Date: October 18, 2002 Action No. 4: Incorporate revisions to the DER series procedure/Instructions to capture assessment of risk as discussed below. Add instructions to include a risk assessment for DER disposition proposed action due dates. Justification of dates should consider immediate actions completed along with monitoring capabilities. Due Date: November 15, 2002 Action No. 5: Identify the target population and conduct training to key station personnel on cause evaluation and troubleshooting skills. The training should include use of Kepner Tregoe Analytical Trouble Shooting and Decision Making skills as well as team building skills. The focus of the training will be to emphasize issue ownership, expand the use of good process skills to ensure causes are understood, and to ensure the extent of condition is properly evaluated. Due Date: November 30, 2002 Action No. 6: Implement a program and organization for maintaining the Preventive Maintenance Program after completion of PMO. Due Date: December 15, 2002 Action No. 7: Add a step to the action development section of procedure NIP-ECA-01 to direct the user to procedure NAI-PTM-01 "General Project and Task Development,"
when it has been determined that an action to develop a program or project is required. These types of actions will require a higher level of involvement from the organization including the assignment of a program/task manager, budget and other resources. Therefore, the reference and use of this instruction is warranted when developing corrective actions. Due Date: November 8, 2002 Action No. 8: In light of changes to the operating experience program, perform a self-assessment focusing on how well low level operating experience, both in-house and external is being used to improve station performance. Due Date: December 30, 2002 b. Prioritization of corrective actions The inspectors found that corrective actions were appropriately prioritized.
c. Establishment of a schedule for implementing and completing the corrective actions The inspectors found that corrective actions have been appropriately scheduled.
d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence Action No. 1 includes the development of measurable goals and strategies to provide an indication of the effectiveness of licensee efforts to improve equipment performance.
MANAGEMENT MEETINGS
.1 Exit Meeting Summary
The results of this inspection were discussed at a regulatory performance and exit meeting conducted on August 23, 2002, with Mr. J. Conway and other members of the NMPNS staff. At that time, the inspectors asked whether any of the information was considered to be proprietary. None of the information was identified as proprietary.
OTHER ACTIVITIES
.1 Licensee Event Reports
a. (Closed) LER 50-410/2001-006-00 & -01 Manual Scram Due to "A" Main Feedwater Pump Motor Failure The inspectors reviewed the NMPNS root cause evaluation and corrective actions associated with this LER and found them to be acceptable. Corrective actions included performing additional testing of the feedwater pump motors, including partial discharge testing and replacing motors with new design or rewind motors within four years. The main feedwater pumps are not classified as safety related. Accordingly, no violation of NRC requirements was identified. LER 50-410/2001-006-00 & -01 are closed. No findings of significance were identified.
b. (Closed) LER 50-410/2001-007-00 Manual Reactor Scram due to Unidentified Drywell Leakage This LER describes that on December 15, 2001, packing leakage from valve 2RCS*MOV18A, the reactor coolant system (RCS) recirculation loop A discharge block valve, which caused operators to commence a reactor shutdown and then to scram the reactor at 60 percent power when unidentified drywell leakage reached 4 gallons per minute. The LER describes that the packing leakage had been noted previously during planned outages in September 2000 and March 2001 but the leakage was not adequately addressed. The corrective action was to manually back seat the valve, which did not change the leakage rate. In each case, the licensee failed to generate a DER or work order to document the leaking packing. Later in July 2001, a work order was generated to repack the valve. However, this work was added to the Spring 2002 work scope but not the forced outage work scope list resulting in missed opportunities to correct the leakage during subsequent shutdowns. Corrective actions included installing a different type of packing on valve 2RCS*MOV18A and three other similar RCS valves.
Corrective actions to address the longstanding problem of packing leakage include developing a common site packing program procedure and assigning a component engineer within Mechanical Maintenance Department with site responsibility for valve packing technical requirements, field implementation, and oversight of the craft.
Corrective action also included ensuring that known leaks in the packing for valves that are critical to generation or reactor coolant system integrity are promptly scheduled for maintenance under the work prioritization and scheduling process.
NMPNS determined that the causes of the packing leakage from valve 2RCS*MOV18A were:
- (1) The decision to start up and run with a minor packing leak was made based upon an improper mindset involving the belief that valve packing leaks tend to seal up once the system reaches rated temperature and pressure;
- (2) Management monitoring of maintenance was not adequate to identify that the valve needed timely repair;
- (3) Licensee personnel did not write a DER or work order when packing leakage was identified in September 2000 and March 2001;
- (4) Regarding the longstanding problem of packing leakage, management direction was inadequate to ensure a sufficient commitment to implementation of the valve packing program. Consequently, the level of knowledge of maintenance and engineering personnel was deficient with respect to adequate packing gland nut torque, the selection of packing materials, appropriate packing configuration, and recognition of the impact of design on packing performance.
The following examples demonstrate that packing leakage has been a longstanding problem that NMPNS has failed to adequately address:
@ On December 9, 1994, Unit 2 was shut down and an Unusual Event was declared due to excessive packing leakage from valve 2CHS*HCV120. Similar to the December 2001 event, some packing leakage from this valve was known to exist at the time of startup, the valve was back seated rather than repaired, no DER was written to document the leakage, the valve was later taken off its backseat, and subsequent excessive packing leakage necessitated a reactor shutdown.
@ On December 13, 1994, Unit 2 was shut down due to excessive packing leakage from valve 2WCS*MOV200. Similar to the December 2001 event, some packing leakage from this valve was known to exist at the time of startup, management decided not to repair the valve based on the above described improper mindset, and subsequent excessive packing leakage necessitated a reactor shutdown.
@ On September 6, 1995, Unit 2 was shut down due to excessive packing leakage from valve 2CHS*AOV108. Similar to the December 2001 event, some packing leakage from this valve was known to exist at the time of startup, management decided not to repair the valve based on the above described improper mindset, and subsequent excessive packing leakage necessitated a reactor shutdown.
LER 50-410/2001-007-00 is closed.
Enforcement aspects of this issue are discussed in Section 02.01.2.c of this report.
ATTACHMENTS PARTIAL LIST OF PERSONS CONTACTED Nine Mile Point Nuclear Station J. Conway, Site Vice President J. Burton, Director, ISEG K. Engelmann, Assistant to site Vice President S. Doty, Acting Maintenance Manager R. Burtch, Director, Communications and Public Relations W. Connolly, Acting Manager, Quality Assurance J. Jones. Director, Emergency Planning P. Mazzaferro, Unit 1 Manager, Technical Support D. Bosnic, Unit 2 Manager, Technical Support B. Montgomery, Manager, Engineering Services S. Minnahan, Unit 2 Operations Manager L. Hopkins, Plant Manager M. Geckle, Director, Licensing G. Detter, Manager, Support Services R. Abbott, Executive Advisor G. Doyle, Director, Assessment and Corrective Actions R. Randall, Manager, Unit 1 Engineering R. Dean, Manager, Unit 2 Engineering S. Davis, Corrective Actions Advisor J. Robe, System Engineer Lead J. Vanvolkenburg, Assessment of Culture Coordinator J. Kinsely, Preventive Maintenance Optimization Program Coordinator C. Rowe, Corrective Actions W. Weaver, Procurement Manager B. Barrett, Procurement Engineer K. Chilton, Operator Events Coordinator L. Kassakatis, Capacitor Replacement Program Coordinator R. Glerum, System Engineer Nuclear Regulatory Commission G. Hunegs, Sr. Resident Inspector B. Fuller, Resident Inspector J. Linville, Chief, Electrical Engineering Branch LIST OF ITEMS OPENED AND CLOSED Opened and Closed 50-220/02-09-01 NCV Inaccurate reporting of Performance Indicator Data (Section 02.01.1.c)50-410/02-09-02 NCV Inadequate corrective actions to address RCS valve packing leakage leads to reactor scram (Section 02.01.2.c)
Closed 50-410/2001-006-00 & -01 LER Manual scram due to feedwater pump motor failure (Section 04.1.a)50-410/2001-007-00 LER Manual Reactor Scram due to Unidentified Drywell Leakage (Section 04.1.b)
LIST OF ACRONYMS AOV Air Operated Valve CFR Code of Federal Regulations CSH High Pressure Core Spray System DER Deficiency/Event Report EHC Electro - hydraulic Control System FWCI Feedwater Coolant Injection HPCI High Pressure Coolant Injection IMC Inspection Manual Chapter LER Licensee Event Report MOV Motor Operated Valve NCV Non-Cited Violation NMPNS Nine Mile Point Nuclear Station NRC Nuclear Regulatory Commission OE Operating Experience PI Performance Indicator PMO Preventive Maintenance Optimization PRA Probabilistic Risk Assessment RCS Reactor Coolant System ROP Reactor Oversight Program SDP Significance Determination Process WCS Reactor Water Cleanup System
DOCUMENTS REVIEWED
Procedures
N1-PM-029-013 Rev 00 - High Pressure Coolant Injection Controller Checks
NIP-ECA-01 Rev 25 - Deviation Event Report
N1-MRM-REL-0104 - Maintenance Rule Scope
NIP-ECA-01, Rev. 5, Root Cause Evaluations
NAI-ECA-10, Vol. 1, Rev. 01, Dispositioning Deviation/Event Reports
NAI-ECA-10, Vol. 2, Rev. 0, Cause Evaluations
Deficiency/Event Reports
NM-1999-1254 NM-1999-3504 NM-1999-4261 NM-2000-705
NM-2000-1592 NM-2000-2378 NM-2001-1773 NM-2001-2253
NM-2001-3665 NM-2001-4830 NM-2001-5609 NM-2001-5894
NM-2001-5931 NM-2001-6004 NM-2002-1241 NM-2002-0173
NM-2002-0224 NM-2002-169 NM-2002-3289 2-2001-5894
2-2001-5609 2-2001-4830 2-2001-2253 1-1997-0139
2-1996-0403
Miscellaneous Documents
NER-2E-011 Use of Aluminum Electrolytic Capacitors In Safety-Related and Q Power Supplies
Nine Mile Point Unit 1 Top Ten Issues Report
Feedwater System Health Report Second Quarter 2002
Weekly OE distribution Matrix for Week ending 4-30-02
MPR-2373 Rev 0 NMP1 HPCI Fault Exposure Review
Periodic Assessment of Maintenance Rule Program ( Mar 2000- December 2001)
Nine Mile Point Nuclear Station Phase 1 Improvement Plan
Nine Mile Point Nuclear Station Units 1&2, Phase II Improvement Plan
Nine Mile Point Nuclear Station, Equipment Reliability Survey Questionnaire, August 2002