IR 05000366/2008006
| ML081130045 | |
| Person / Time | |
|---|---|
| Site: | Hatch  |
| Issue date: | 04/18/2008 |
| From: | Scott Shaeffer NRC/RGN-II/DRP/RPB2 |
| To: | Madison D Southern Nuclear Operating Co |
| References | |
| IR-08-006 | |
| Download: ML081130045 (15) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION pril 18, 2008
SUBJECT:
EDWIN I. HATCH NUCLEAR PLANT - NRC SUPPLEMENTAL INSPECTION REPORT 05000366/2008006
Dear Mr. Madison:
On March 6, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection, in accordance with Inspection Procedure 95001, at your Edwin I. Hatch Nuclear Plant (HNP) Unit 2. The purpose of the inspection was to examine the causes for and actions taken related to the Mitigating Systems Performance Index, High Pressure Injection System Performance Indicator (PI) crossing the threshold from Green (within expected range) to White (low to moderate safety significance) for Unit 2 in the second quarter of 2007. This PI crossed the White threshold due to four functional failures of the High Pressure Coolant Injection (HPCI)
system. The enclosed inspection report documents the inspection findings, which were discussed on March 6, 2008, with you and other members of your staff.
The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.
Based on the results of this supplemental inspection, the inspector determined that, in general, the problem identification, root cause, and corrective actions were adequate. Several weaknesses were noted by the inspector relating to the evaluation of conditions adverse to quality. Additionally, one self-revealing finding of very low safety significance was identified which was determined to be a violation of NRC requirements. However, because the violation was of very low safety significance and has been entered into your corrective action program, the NRC is treating the violation as a Non-Cited Violation (NCV) in accordance with Section VI.A.1 of the NRCs Enforcement Policy. If you contest this NCV, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the United States Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington DC 20555-0001; with copies to the Regional Administrator, Region II; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at Hatch.
SNC 2 In accordance with the Code of Federal Regulations 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Docket Nos.: 50-366 License Nos.: NPF-5
Enclosure:
Inspection Report 05000366/2008006 w/Attachment: Supplemental Information
REGION II==
Docket No: 50-366 License No: NPF-5 Report No: 05000366/2008006 Licensee: Southern Nuclear Operating Company, Inc.
Facility: Edwin I. Hatch Nuclear Plant, Unit 2 Location: Baxley, GA 30830 Dates: March 3 - March 6, 2008 Inspector: C. Rapp, Senior Project Engineer Approved by: Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000366/2008-006; 03/03/08 - 03/06/08; Edwin I. Hatch Nuclear Plant, Unit 2;
Supplemental Inspection Procedure 95001 for a White Mitigating Systems Performance Indicator, High Pressure Injection System Performance Indicator This inspection was conducted by a Senior Project Engineer. One Green non-cited violation (NCV) was identified. The significance of most findings is identified by their color (Green, White,
Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,
Revision 4, dated December 2006.
This supplemental inspection was conducted in accordance with Inspection Procedure 95001,
Inspection for One or Two White Inputs in a Strategic Performance Area, in response to a White Performance Indicator (PI) associated with the High Pressure Coolant Injection (HPCI) system.
Based on the results of this inspection, the inspector determined that the cause evaluations were generally adequate. The inspector identified several weaknesses in the quality of the individual evaluations and associated corrective actions. The overall evaluation for the White PI identified these weaknesses. This evaluation identified comprehensive corrective actions which were properly prioritized.
Given the licensees acceptable performance in addressing this White PI, consistent with the guidance in IMC 0305, Operating Reactor Assessment Program, the PI will only be considered in assessing plant performance until it returns to a Green characterization. The implementation and effectiveness of the licensees corrective actions will be reviewed during future inspections.
NRC-Identified and Self-Revealing Findings
Cornerstone: Mitigating Systems
A Green NRC identified non-cited violation of 10 CFR 50 Appendix B, Criterion XVI,
Corrective Action, was identified when the licensee failed to thoroughly identify and correct water intrusion into the High Pressure Coolant Injection (HPCI) lubricating oil system (LOS) on two separate occasions. Residual water from these events was not fully removed which resulted in corrosion of the HPCI turbine controls. This violation was entered into the licensees corrective action program (CAP) as CR 2008100154.
The inspector determined the performance deficiency was the failure to remove all residual water from the HPCI LOS following water intrusion on two separate occasions.
The finding was more than minor because it was associated with the equipment performance attribute and adversely affected the objective of the Mitigating Systems cornerstone in that unplanned corrective maintenance of the HPCI pump rendered the system unavailable to respond to initiating events. This finding was determined to be of very low safety significance because the failure did not represent a loss of safety function of a single train. The inspector determined this finding was related to the thoroughness of evaluations aspect of the Problem Identification and Resolution cross cutting area. (P.1 (c)) (Section 03)
Licensee-Identified Violations
None.
REPORT DETAILS
INSPECTION SCOPE
The purpose of this supplemental inspection was to assess the licensees evaluation of a White PI for Unit 2 in the Mitigating Systems cornerstone. The licensee had four functional failures of the HPCI system that caused the Mitigating Systems Performance Index (MSPI),
High Pressure Injection System (HPIS) PI to cross the Green to White threshold during the second quarter of 2007. The inspector reviewed the licensees actions associated with the four functional failures listed below and conducted interviews with licensee personnel to ensure that the root cause and contributing causes of the events were identified and understood and that appropriate corrective actions to prevent recurrence were initiated.
- Failure 1: On March 15, 2007, during start-up follow a refueling outage, reactor pressure was held at 165 psig to perform the 165 psig HPCI pump operability test. When the system was started, the Turbine Control Valve, (TCV), failed to open and provide steam to the HPCI turbine. This issue was self-disclosing based on water and oil found leaking from the HPCI bearings. This failure was caused by water being introduced into the HPCI bearing LOS due to an improper tagout on February 16, 2007. The tagout created a flow-path from the Condensate Storage Tank (CST) to the Barometric Condenser. The tagout also prevented automatic draining of the Barometric Condenser on high water level. When the Barometric Condenser was filled, water overflowed into the LOS. This failure resulted in HPCI being declared inoperable. (CR 2007103319)
- Failure 2: On May 18, 2007, while investigating a reported oil leak, Maintenance found water in the Unit 2 HPCI turbine LOS. Water had migrated into the LOS from the main pump outboard mechanical seal bracket drain. Water entered the bearing cavity due to leakage past the HPCI pump outboard mechanical seal. Normally, the bracket drain will drain any mechanical seal leakage to the turbine skid. However, a valve in the drain piping became clogged allowing water to overflow into LOS reservoir through the bearing oil return lines. This failure resulted in HPCI being declared inoperable. (CR 2007105289)
- Failure 3: On May 25, 2007, during the performance of the HPCI system time response test, the HPCI turbine spuriously tripped after reaching rated speed, flow, and pressure.
The HPCI turbine subsequently automatically reset and restarted. The turbine trip was caused by the trip reset spring being out of adjustment. During a HPCI system outage, the trip tappet was actuated >40 times which resulted in accelerated relaxation on the spring. This failure resulted in HPCI being declared inoperable. (CR 2007105455)
- Failure 4: On January 6, 2008, in preparation for a HPCI system outage, an as found test was being performed. HPCI was required to reach a rated pump flow and pressure within 49 seconds. The time recorded during the performance of this test was 54.32 seconds. The cause of this degradation was a binding or sluggish Electronic Governor Remote (ERG) actuator. The binding EGR was caused by corrosion on EGR internals due to residual water from the two earlier water intrusion events. This condition resulted in HPCI being operable but degraded. (CR 2008100154)
The inspectors also reviewed the licensees overall evaluation for the PI crossing the Green to White threshold. This evaluation reviewed the evaluations for each failure to identify programmatic and organizational weaknesses that resulted in the White PI. This evaluation included the evaluation for Failure 4 and was documented in CR 2008100154.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a. Determination that the evaluation identifies who and under what conditions the issue was identified The inspector determined that the evaluation for each failure was sufficiently detailed to identify who and under what conditions the issue was identified.
b. Determination that the evaluation documents how long the issue existed and prior opportunities for identification The inspector determined that the evaluation for each failure documented how long the issue existed and prior opportunities for identification.
c. Determination that the evaluation documents the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues The inspector determined that the evaluations for Failures 1, 2, and 4 documented the risk consequences and compliance concerns. For Failure 3, an Apparent Cause was performed. This evaluation method did not include an assessment of the risk consequences. The licensee identified this weakness in their overall evaluation and adequately documented the risk consequences and compliance concerns.
02.02 Root Cause and Extent-of-Condition Evaluation a. Determination that the problem was evaluated using systematic methods to identify root causes and contributing causes The systematic method for each failure is listed below:
- Failure 1 - Events and Casual Factors Analysis
- Failure 2 - Barrier Analysis
- Failure 3 - Why Staircase Method (Apparent Cause)
- Failure 4 - Fault Tree Analysis, Events and Casual Factors Analysis, and Barrier Analysis The inspector determined that systematic methods used were adequate for Failures 1, 2, and 4. However, for Failures 1 and 2, the evaluations were narrowly focused. For Failure 1, the evaluation focused only on the tagout error. The identified casual factor of not inspecting the EGR for 27 days was not explored as a root or contributing cause.
For Failure 2, the evaluation focused on the bracket drain clogging. Mechanical seal leakage which allowed water into the bearing cavity was not identified as a root or contributing cause. The licensee identified these weaknesses in their overall evaluation.
For Failure 3, the use of an informal evaluation method resulted in not identifying a root cause of actuating the trip tappet over 40 times which resulted in loss of trip tappet spring tension. Further, contributing causes such as organization issues were not explored. The licensee identified this weakness in their overall evaluation. The inspector agreed the evaluation method used was not ideal for the situation.
b. Determination that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem For Failures 1, 2, and 4, the inspector determined the level of detail was adequate. For Failure 3, the level of detail was inadequate due to the use of an Apparent Cause evaluation instead of a formal Root Cause Analysis (RCA). The licensee identified this weakness in their overall evaluation.
c. Determination that the root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience The inspector determined that the evaluations for all four failures and the overall evaluation considered prior occurrences and operating experience.
d. Determination that the root cause evaluation addresses the extent of condition and the extent of cause of the problem The inspector determined the evaluations for Failures 1 and 2 considered extent of condition; however, the review was superficial and narrowly focused. For Failure 1, only a review of operating experience was conducted. For Failure 2, the review only considered the potential for drain clogging and did not include water intrusion sources.
For Failures 1, 2, and 3, no extent of cause review was conducted. The licensee identified these weaknesses in their overall evaluation. A comprehensive extent of condition and cause review was conducted as part of the overall evaluation.
e. Determination that the root cause evaluation, extent of condition, and extent of cause appropriately considered the safety culture components as described in IMC 0305 The inspector determined that the safety culture components were not considered or reviewed for Failures 1, 2, and 3. The overall evaluation included a comprehensive safety culture assessment for all four failures. Each safety culture component was related to the identified contributing factors. The CR for the improper tagout associated with Failure 1 and the CR for Failure 3 were both assigned severity level (SL) 3. The inspector noted that the SL 3 assignment appeared to be driven by the regulatory/
operational impact verses safety/risk impact. Consequently, these conditions were not rigorously evaluated as they would have been using a formal RCA method. This observation was discussed with licensee management.
02.03 Corrective Actions a. Determination that appropriate corrective action(s) are specified for each root/contributing cause or that there is an evaluation that no actions are necessary For Failures 1 and 2, the licensee drained the LOS reservoir, removed any residual water, and refilled the LOS with new oil. For Failure 2, the EGR was replaced as well.
The licensee did not check the remote servo unit or the interconnecting lines between the ERG and the remote servo unit for residual water. The inspector determined the corrective actions for Failures 1 and 2 were not adequate to remove residual water that remained in the LOS which resulted in Failure 4. The licensee identified this weakness in their overall evaluation. The regulatory aspects are discussed in Section 03.
For Failure 3, the licensee added steps to HPCI inspection procedure to adjust the trip tappet reset spring if not performed during the 24-month maintenance. The inspector determined the corrective action was appropriate for the identified apparent cause.
For Failure 4, the licensee drained the LOS reservoir, removed any residual water, refilled the LOS with new oil, replaced the EGR and the remote servo unit, and flushed the interconnecting lines between the EGR and the remote servo unit. The inspector determined the corrective actions were appropriate for the identified root and contributing causes.
For the overall evaluation, the licensee identified comprehensive corrective actions for the root and contributing causes. These corrective actions included the following:
- Require RCAs be performed by dedicated teams with a qualified root cause analyst assigned
- Require effectiveness review for root cause corrective actions
- Revise tagout development and review process
- Revise maintenance procedures for recovering from water intrusion
- Clarify requirements for operability determinations for degrading conditions
- Revised equipment monitoring process to detect degrading conditions
- Revised procedures to require identification and quantization of HPCI pump seal leaks The inspector determined the corrective actions were appropriate for the identified root and contributing causes.
b. Determination that the corrective actions have been prioritized with consideration of the risk significance and regulatory compliance The inspector determined that the corrective actions for the overall evaluation were appropriately prioritized.
c. Determination that a schedule has been established for implementing and completing the corrective actions The inspector determined that the corrective actions for the overall evaluation have been scheduled or completed.
d. Determination that quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence The inspector determined that corrective action effectiveness reviews using the evaluation criteria established in the CAP procedures were scheduled.
Regulatory Issues 03.01 Findings
Introduction:
A Green NRC identified NCV of 10 CFR 50 Appendix B, Criterion XVI, Corrective Action, was identified when the licensee failed to thoroughly identify and correct water intrusion into the HPCI LOS on two separate occasions. Residual water was not removed which resulted in corrosion of the HPCI turbine controls.
Description:
On January 6, 2008, the HPCI pump failed to reach rated pressure and flow within the 49 second response time required by the Technical Requirements Manual (TRM). The licensee removed the cover from the top of the EGR and found corrosion on the pilot valve that resulted in the pilot valve binding. The licensee attributed the corrosion to residual water in the LOS following water intrusion events on February 16, 2007, and on May 18, 2007. Following the February 16 intrusion, the licensee drained and flushed the LOS, cleaned the reservoir to remove any residual water, then refilled the LOS with fresh lubricating oil. The licensee reviewed this event with the vendor to determine if any additional actions were needed. Due to the auxiliary oil pump (AOP) not being run, no additional actions were recommended; e.g.;
replacement of the EGR. The licensee did not consider the potential for residual water in the remote servo unit and interconnecting lines.
Following the May 18, 2007, water intrusion, the licensee drained and flushed the LOS, cleaned the reservoir to remove any residual water, and refilled the LOS with fresh lubricating oil. However to obtain a sample of the lubricating oil, the AOP was started which resulted in circulating the lubricating oil/water mixture into the EGR. Therefore, the licensee also replaced the EGR as recommended by the consultant. Again, the licensee did not consider the potential for residual water in the remote servo unit and interconnecting lines.
Between May 25, 2007 and January 6, 2008, the licensee performed four other routine response time tests. The response times showed a gradual degradation from 19 to 33 seconds. Although the degrading response times were still within the TRM response time limits, the licensee did not adequately question the adverse trend. Slow response times are an indication of degraded EGR performance. The licensee also had previous opportunity to identify and correct the residual water in the LOS via oil sampling. The licensee had conducted routine sampling of the lubricating oil to identify any water in the reservoir. However, these samples were taken from the midline of the reservoir. The lubricating oil was designed to quickly separate from any water leaving the water on the bottom of the LOS reservoir. Therefore, sampling at the midline would not detect if water was in the LOS reservoir. The licensee eventually contacted the vendor for other areas that may contain residual water. The vendor indicated that potential areas were the remote servo unit and the interconnecting lines. The licensee sampled the lubricating oil these areas and found high percentages of water. The licensee determined that residual water from the remote servo and the interconnecting lines had entered the LOS and the EGR during the earlier HPCI response time tests which led to the EGR degradation in January 2008.
Analysis:
The inspector determined the performance deficiency was failure to remove all residual water from the HPCI LOS following water intrusion on two separate occasions.
The finding was more than minor because it was associated with the equipment performance reliability attribute and adversely the objective of the Mitigating Systems cornerstone in that residual water in the LOS resulted in the HPCI system being unavailable to respond to initiating events. This finding was determined to be of very low safety significance because the failure did not represent a loss of safety function of a single train. The inspector determined this finding was related to the thoroughness of evaluations aspect of the Problem Identification and Resolution cross cutting area.
(P.1 (c))
Enforcement:
10 CFR 50 Appendix B, Criterion XVI, Corrective Actions, requires in part that conditions adverse to quality be identified and corrected. Contrary to the above, the licensee failed to thoroughly evaluate water intrusion into the HPCI LOS on two separate occasions. Consequently, residual water was not removed from the LOS, which resulted in unplanned unavailability of the HCPI pump between January 7 and 12, 2008.
Because this finding is of very low safety significance and because it was entered into the licensees CAP as CR 2008100154, this violation is being treated as an NCV consistent with Section VI.A.1 of the NRC Enforcement Policy: NCV 05000366/2008006-01, Inadequate Response to Water Contaminated Oil Results in HPCI Unavailability.
04 MANAGEMENT MEETINGS
Exit Meeting Summary
The inspector presented the results of the supplemental inspection to Mr. D. Madison and other members of licensee management and staff on March 6, 2008. Specific details of the findings were also discussed with the licensee on April 18, 2008. The inspector confirmed that no proprietary information was provided or examined during the inspection.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- M. Ajluni, Engineering Director
R. Baker - Hatch Corporate Licensing Supervisor
- S. Bargeron, Plant Manager
V. Coleman - Chemistry Manager
J. Dixon - HP Manager
B. Goodwin - Engineering Design Manager
- J. Hammonds, Training Manager
G. Johnson - Operations Manager
J. Lewis - Hatch Plant Support Manager
- D. Madison, Vice President - Hatch
- L. Mikulecky, Performance Analysis - Sr. Engineer
- S. Soper, Engineering Support Manager
- S. Tipps, Principal Engineer- Licensing
D. Tootle - Fleet Oversight Supervisor
R. Varnadore. Maintenance Manager
M. Webb - Hatch Corporate Licensing Engineer
NRC personnel
- J. Hickey, Senior Resident - Hatch
- S. Shaeffer, Chief, Projects Branch 2
ITEMS OPENED AND CLOSED
Opened and Closed
05000366/2008006-01 NCV Inadequate Response to Water Contaminated Oil
Results in HPCI Unavailability (Section 03.01)