IR 05000250/2011013
| ML112580547 | |
| Person / Time | |
|---|---|
| Site: | Turkey Point  |
| Issue date: | 09/15/2011 |
| From: | Rich D NRC/RGN-II/DRP/RPB3 |
| To: | Nazar M Florida Power & Light Co |
| References | |
| IR-11-013 | |
| Download: ML112580547 (25) | |
Text
UNITED STATES ber 15, 2011
SUBJECT:
TURKEY POINT NUCLEAR PLANT - NRC SPECIAL INSPECTION REPORT 05000250/2011013
Dear Mr. Nazar:
On August 19, 2011, the U. S. Nuclear Regulatory Commission (NRC) completed a Special Inspection at your Turkey Point Unit 3. The enclosed inspection report documents the inspection results, which were discussed on August 19, 2011, with Mr. Kiley and other members of your staff.
The purpose of the Special Inspection was to review the circumstances associated with a loss of Intake Cooling Water System (ICW) function which occurred on August 11, 2011. The loss of function occurred when manual valve 3-50-406, ICW/Component Cooling Water (CCW) Heat Exchanger Outlet Bypass Valve, failed closed, which prevented ICW flow through the CCW heat exchangers. Operations department personnel monitored CCW parameters and restored ICW flow to the CCW heat exchangers by opening valve 3-50-407, restoring flow via an alternate flowpath. Unit 3 entered and exited Technical Specification 3.0.3 due to the loss of ICW function.
The event was evaluated by the NRC in accordance with Management Directive (MD) 8.3, ANRC Incident Investigation Program,@ and, due to the loss of safety function and resultant estimated increase in risk to the plant, the Special Inspection was initiated. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission=s rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, performed visual inspections, and interviewed personnel. The Special Inspection Charter is enclosed as an Attachment to the enclosed inspection report.
The Special Inspection focused on fact finding and thus did not complete a review of licensee failures to comply with regulatory requirements. Review and disposition of any such failures will be completed during staff review of the licensees root cause analysis of the event, and will be documented in the appropriate quarterly integrated inspection report after management review.
FPL 2 In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publically Available Records (PARS) component of NRC=s document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Daniel W. Rich, Chief Reactor Projects Branch 3 Division of Reactor Projects Docket No: 50-250 License No: DPR-31
Enclosure:
Inspection Report 05000250/2011013 w/Attachments Attachments: 1. Supplemental Information 2. Turkey Point Special Inspection Charter
REGION II==
Docket Nos.: 50-250 License Nos.: DPR-31 Report No: 05000250/2011013 Licensee: Florida Power & Light Company (FPL)
Facility: Turkey Point Nuclear Plant, Unit 3 Location: 9760 S. W. 344th Street Homestead, FL 33035 Dates: August 16 to August 19, 2011 Inspectors: G. Wilson, Senior Project Engineer A. Alen, Reactor Engineer R. Williams, Reactor Engineer Approved by: D. Rich, Branch, Chief Reactor Projects Branch 3 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000250/2011-003; 8/16/2011 -8/19/2011; Turkey Point Nuclear Power Plant, Unit 3;
Special Inspection This Special Inspection was conducted by a Senior Project Engineer, and two Reactor Engineers from the Region II office using Inspection Procedure 93812 to assess the circumstances surrounding the loss of intake cooling water function at Turkey Point Unit 3.
NRC-Identified
& Self-Revealing Findings None.
Licensee Identified Violations
None.
REPORT DETAILS
Event
Description:
On August 11, 2011, workers performing eddy current testing on the 3B CCW heat exchanger heard a noise and saw leakage from the 3A CCW heat exchanger ICW outlet channel head cover. Operators observed that ICW flow through the 3A and 3C CCW heat exchangers had stopped, and that valve 3-50-406 which is located in the common ICW discharge line to the discharge canal appeared to have failed closed. The closure of this valve prevented flow through both ICW headers to the CCW heat exchangers. The licensee entered Technical Specification (TS) 3.0.3 due to loss of function of the ICW system. Over the next 20 minutes, CCW temperature increased from 95 degrees Fahrenheit to 111 degrees Fahrenheit, and the Reactor Coolant Pump (RCP) Motor Bearing Cooling Water High Temperature annunciator alarmed. The licensee re-established flow through an alternate flowpath by opening normally closed ICW valve 3-50-407. ICW flow returned to the 3A and 3C heat exchangers at approximately 9,400 and 10,000 gpm respectively, the licensee exited TS 3.0.3, and CCW temperatures began to decrease and stabilized at 97 degrees Fahrenheit. When valve 3-50-406 failed closed, the ICW system back pressure increased causing leakage from the 3A CCW heat exchanger channel head flange due to gasket failure.
Special Inspection Team Charter Based on the criteria specified in MD 8.3, @NRC Incident Investigation Program,@ a Special Inspection was initiated in accordance with Inspection Procedure 93812, ASpecial Inspection.@
The objectives of the inspection, described in the charter, are listed below and are addressed in the identified sections:
- (1) Develop a timeline for the event and review the licensees response to the event, including reportability, event classification, and associated maintenance being conducted at that time.
(Section 4OA3.1)
- (2) Assess the ability of Turkey Point Unit 3 to meet its design basis functions and TS 3.7.3 requirements with the as-found condition. (Section 4OA3.2)
- (3) Review the adequacy of maintenance and testing that is routinely performed for the ICW/Component Cooling Water (CCW) Heat Exchanger Outlet Valves and assess any related effects on the material condition of the valves. (Section 4OA3.3)
- (4) Assess the licensees activities related to the problem investigation performed to date including root cause analysis, extent of condition, and risk assessment of the event.
(Section 4OA3.4)
- (5) Assess operational considerations and controls for affected and similar valves.
(Section 4OA3.5)
- (6) Assess the licensees activities related to identification of additional equipment failure mechanisms and damage to equipment due to the event. (Section 4OA3.6)
- (7) Assess the licensees efforts to identify other systems with functions important to safety which are vulnerable to similar single point failures. (Section 4OA3.7)
OTHER ACTIVITIES
4OA3 Event Follow-up - Special Inspection
.1 Develop a timeline for the event and review the licensees response to the event,
including reportability, event classification, and associated maintenance being conducted at that time.
a. Inspection Scope
The inspectors reviewed operating, maintenance and engineering logs, procedures and interviewed personnel to determine what actions the licensee took following the failure of valve 3-50-406, and to develop a timeline of events. The inspectors performed a walkdown of the CCW room where the valve is located and the control room.
b.
Observations On August 11, 2011, Unit 3 was operating at 100% power, eddy current testing was being performed on the 3B CCW heat exchanger, and fuel was being moved in the spent fuel pool. Events occurred as indicated by the following timeline.
1625: The Unit 3 reactor control operator (RCO) received a report from workers performing eddy current testing on the 3B CCW heat exchanger that a large amount of water was entering the Unit 3 CCW room from the outlet 3A CCW heat exchanger. The workers also reported hearing a loud bang in the Unit 3 CCW room.
The control room dispatched the Senior Nuclear Plant Operator (SNPO) to investigate.
1630: The Unit 3 RCO noted CCW head tank level rising and CCW outlet header temperatures rising.
1631: Shift manager declared Unit 3 to be in TS 3.0.3 based on reports that no Intake Cooling Water (ICW) headers were operable, as required by (TS 3.7.3).
1635: The SNPO reported that ICW/CCW basket strainer differential pressures were 0 psid and that flow through the 3A and 3C CCW heat exchangers were also 0 gpm.
The control room entered procedure 3-ONOP-019, Intake Cooling Water System Malfunction.
The control room reviewed fold out page of 3-ONOP-019 which contains trip guidance at 120 degrees Fahrenheit on the CCW system.
1637: The Shift Manager ordered the fuel movement in the Unit 3 spent fuel pool stopped.
The Senior Reactor Operator and additional operators arrived in the Unit 3 CCW room.
The SNPO reviewed the current status and indications with the SRO. The SNPO reported that the indications pointed to 3-50-406, CCW/ICW Outlet CV-2202 Bypass valve being closed and that they needed to open 3-50-407, CCW/ICW Outlet Spoolpiece Downstream Isolation valve.
1641: The control room received annunciator H-9/4, RCP Motor Bearing Cooling Water High Temperature alarm. The licensee reviewed annunciator response procedure, 3-ARP-CR.H and entered procedure 3-ONOP-041.1 RCP Off Normal.
1642: The SRO and SNPO reviewed the operating diagrams in the Unit 3 CCW room.
The SRO concurred with the SNPOs assessment of the ICWs system status and directed the SNPO to open valve 3-50-407.
The control room ordered the SNPO to close 3-50-371, ICW inlet to 3A CCW HX and 3-50-380, ICW outlet from the 3A CCW HX, to isolate ICW leakage.
1641: CCW temperatures noted to reach 111 degrees Fahrenheit.
1643: The SNPO began opening valve 3-50-407 by hand until it is off its closed seat and the DP across the valve is minimized. The SNPO then obtained an 18 inch (approximate) valve wrench to continue to open the valve.
The ICW system engineer arrived in the Unit 3 CCW room and assisted the SNPO in fully opening valve 3-50-407 using the valve wrench.
1646: The SNPO reported that valves 3-50-371 and 3-50-380 were not closed due to ICW flow being restored by opening valve 3-50-407.
1650: CCW temperature stabilized at 97 degrees Fahrenheit.
RCP Motor Bearing Cooling water High temperature annunciator cleared.
1651: The SNPO reported 9400 gpm ICW flow to 3A CCW heat exchanger, and 10,000 gpm to the 3C CCW heat exchanger.
Turkey Point Unit 3 exited TS 3.0.3 due to restoring ICW flow through the 3A and 3C CCW heat exchangers.
1652: 3-NOP-040.03, Fuel handling and Insert Shuffle in the spent fuel pool was stopped.
ICW leakage from the outlet of the 3A CCW heat exchangers was reported to have decreased to 5-10 gpm following opening of 3-50-407.
1655: The licensee concluded that 3A CCW was operable based on the reduced ICW leakage from the 3A CCW heat exchanger outlet and the stable system conditions including temperatures and head tank level.
1700 Due to elevated RCP temperatures, the licensee referred to 3-ONOP-030, Component Cooling water Malfunction and concluded that no additional actions were required by that procedure.
The licensee conducted a walkdown of the ICW system for possible degradation due to the closure of valve 3-50-406.
1705: The licensee reviewed 0-EPIP-20101, Duties of Emergency Coordinator, attachment 1 emergency action levels and concluded that no entry conditions were identified.
1708: The licensee exited procedure 3-ONOP-019, ICW Malfunction.
1710: The licensee reviewed procedure 0-ADM-115, notification of Plant Events, enclosure 1, NRC Notification Table and concluded that an 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> notification was required by 10 CFR 50.72(b)(3)(v).
1716: The licensee exited procedure 0-ONOP-041.1, RCP Off Normal.
1919: The licensee notified the Senior Resident Inspector of entry into TS 3.0.3.
2040: The licensee completed 3-ONOP-30 section 5.19, Restoring 3B CCW heat exchanger to service.
2125: NRC Operations Center was notified of the event.
2150: The licensee restored the 3B CCW heat exchanger from eddy current testing and declared it operable.
The inspectors determined that the licensee responded effectively in restoring ICW flow to the CCW heat exchangers, in that the licensee properly analyzed available indications, and followed procedures in identifying the failure of valve 3-50-406 and re-establishing ICW flow. The eddy current testing on the 3B CCW heat exchanger was the only maintenance activity identified as being in progress in the area of valve 3-50-406, and does not appear to have had any effect on its failure. Results of preliminary examinations indicated that components internal to the valve manual actuator appear to have failed from cyclic fatigue that could be attributed to fluttering of the valve. Fluttering of valve 3-50-406 had been identified since 2007, but corrective action had not been accomplished. Fluttering has been defined by the licensee as the slight movement of the valve disc and rocking back and forth motion based on a visual inspection of the position indicator. This motion causes stress on the actuator internal components.
The inspectors did not identify any emergency action levels that should have been declared, and determined that that licensee properly reported the event in accordance with 10 CFR 50.72.
.2 Assess the ability of Turkey Point Unit 3 to meet its design basis functions and TS
3.7.3 requirements with the as-found condition.
a. Inspection Scope
The inspectors evaluated the technical adequacy of the licensees basis for ICW system operability and verified compliance with TS 3.7.3. The inspectors reviewed the Updated Final Safety Analysis Report (UFSAR), design bases documents, and consulted with regional and headquarters staff to identify the design and licensing bases requirements of the system. The inspectors performed a detailed review of ICW system modifications specific to the systems discharge valve lineup configuration, from original design through the day of the event on August 11, 2011. The inspectors reviewed operator logs, reviewed the prompt operability determination, and interviewed reactor operators that were on shift the day of the event. The inspectors also performed a partial system walk-down of the ICW/CCW portion of the system to assess material condition of major system components including the systems butterfly discharge valves.
b. Observations The inspectors determined that the licensee met TS 3.7.3 requirements and adequately assessed the ICW systems capability to perform its design basis functions when ICW system flow was re-established through valve 3-50-407, in that valve 3-50-407 can accommodate normal and accident flow conditions. Additionally, from reviewing operations logs and operator interviews the inspectors determined the licensee made an adequate and timely entry and exit of TS LCO 3.0.3.
In accordance with the ICW system design bases, the design and operation of the system, assuming a single active failure, ensures cooling capacity is available to support the continued operation of safety-related equipment during normal and accident conditions. The failure of butterfly valve 3-50-406, on August 11, 2011, was considered a passive failure. At the time of the ICW design development this type of failure was not considered credible and was not part of the original ICW design criteria.
To meet the design criteria, including a single active failure, the ICW system has redundant pumps and supply headers, cross-tie valves, and isolation valves to allow system operation while a portion of the system is removed from service for maintenance.
Although the system has redundant supply headers, the original design had a single 30 inch discharge line to the canal equipped with a temperature control valve CV-3/4-2202 which controlled ICW flow through the CCW heat exchangers. CV-3/4-2202 was an air-operated valve designed to maintain a constant ICW outlet temperature by controlling system flow. To allow maintenance on this valve while maintaining system operation the control valve was provided with isolation valves 3/4-50-405 and 3/4-50-407 and 24 inch bypass valve 3/4-50-406.
In 1984, during an NRC design review, it was identified that the ICW system could not accommodate a single active failure of CV-3/4-2202. A failure of CV-3/4-2202 (in the closed position) would have caused isolation of ICW flow through the CCW heat exchangers. The licensee resolved this issue by permanently opening bypass valve 3/4-50-406. The inspector noted that for this modification, the licensee did not evaluate the long-term effects of flow-induced erosion on the 24-inch bypass line (vs. the original 30-inch line) nor the effects of flow inducted vibration on the 3/4-50-406 butterfly valve. The licensee plans to include evaluation of these issues in the root cause investigation.
Because CV-3/4-2202 was no longer required to perform an active function, in 1996, design change packages PC/M 94-096/131 (Unit 3 and 4 respectively), were approved to permanently remove the control valve from the system. This change maintained 3/4-50-405, 3/4-50-407 and 3/4-50-406 open, and thus provided redundant ICW/CCW discharge flow paths.
In 2004, the ICW system configuration was changed to require 3/4-50-407 to be normally closed. The purpose of this change was to force more ICW flow through the, non-safety related, turbine plant cooling water heat exchangers. The licensee concluded that this change was acceptable without any detrimental effects to the systems ability to perform its safety related function. The inspectors review of this modification identified that the licensees probabilistic risk analysis (PRA) model was not updated with the removal of the redundant discharge flow path. From 2004 to 2011, the licensees PRA model took credit for a redundant discharge path that was isolated with the closure of 3/4-50-407. This was entered in the licensees corrective action program (CAP) as action request (AR) 1686200.
.3 Review the adequacy of maintenance and testing that is routinely performed for the
ICW/Component Cooling Water (CCW) Heat Exchanger Outlet Valves and assess any related effects on the material condition of the valves.
a. Inspection Scope
The inspectors reviewed the valve and valve actuator vendor manual guidance specific to operation and maintenance and verified it was consistent with licensee maintenance practices and procedures. The inspectors interviewed maintenance personnel to assess the adequacy of the maintenance practices and knowledge level. The inspectors reviewed a sample of corrective and preventive maintenance work orders (WO)associated with ICW/CCW outlet valves to understand the types of deficiencies, if any, that were typically identified and the associated corrective actions. Additionally, the inspectors reviewed operating experience, generic communications, and American Society of Mechanical Engineers (ASME) in-service-testing (IST) requirements and consulted with regional staff to determine if there are any regulatory requirements to perform testing of the ICW/CCW heat exchangers outlet valves. The inspectors also held interviews with the system and component engineers to gather insights on the life-cycle and operational performance history of butterfly valves in the ICW system similar to 3/4-50-406.
b. Observations The ICW/CCW HXs outlet valves include:
- (1) CCW heat exchangers outlet isolation valves,
- (2) original discharge line valves 3/4-50-405 and 3/4-50-407, and
- (3) discharge line bypass valve 3/4-50-406. These valves are Henry Pratt butterfly valves with disc sizes ranging between 24-inches and 30-inches. The valves are provided with Henry Pratt MDT manual hand-wheel geared actuators, which are a compound lever and link traveling nut style operator designed to provide a 90 degree rotation of the valve and hold it in the desired position. All of these valves are normally in the locked-open position, except 3/4-50-407, which has been normally closed since 2004.
The subject valves do not have preventive maintenance (PM) or scheduled replacement except for periodic inspections. This is consistent with vendor guidance. The valve actuator does not have a scheduled replacement; however, it has a 36-month PM. The PM requires removal of the top cover of the actuator, visual inspection of the actuator internals, check of lubricant condition (replaces if necessary), and verification of proper clearances. If actuator or internals need replacement, the actuator is overhauled or replaced with a pre-assembled actuator.
The valve actuators are checked monthly for fluttering by inspection of the valve position indicator located on top of the actuator. Fluttering is an indication that the actuator internals are loose and need adjustment or repair or the valve sleeve bearings are degraded and the valve requires replacement. To address valve fluttering, the licensee performs an inspection of the actuator and takes corrective actions, as needed. If no problems are identified with the actuator and valve fluttering cant be reduced or eliminated (indicative of a problem with the valve) then the valve actuator is scheduled for replacement. Although there are no scheduled replacements for the ICW system valves or actuators, throughout the life of the plant, several of the system valves and actuators upstream of the CCW heat exchangers have been replaced. However, the valves downstream of the CCW heat exchangers have never been replaced due to difficulties in isolating the valves due to system configuration.
From a WO history review specific to valve 3-50-406 the inspectors identified the following:
- In 1994 flow induced-vibration of the valve was initially identified (WO 24003037).
The actuator was adjusted to reduce the vibrations. The WO completion notes documented that failure of the 3-50-406 valve actuator could cause the valve to close and result in a loss of heat sink.
- In December 2007, during the 36-month PM, considerable fluttering of the valve was identified, as indicated by the position indicator. The upper follower bushing was noted to be pitted from rust and water was found in the housing. The examiner recommended that the actuator be replaced. These results were entered in the CAP per AR 00510913 and work request (WR) 37015591. WO 37027428 was issued in December 2007 to replace the actuator; however, the inspectors found the WO was still in the planning stages due to the difficulty of isolation of this portion of the ICW system, and two refueling outages had passed without the accomplishment of any corrective maintenance.
- (1) the actuator continued to vibrate, and
- (2) the indicator was detached from the valve stem possibly due to the excessive vibration. This was entered in the CAP as AR01622098 and WR94017988. The licensee did not perform any additional work on the actuator and the valve subsequently failed closed on August 11, 2011, resulting in a loss of ICW.
From a WO history sample review for the other ICW/CCW outlet valves, the inspectors did not identify any similar indications as those exhibited by valve 3-50-406.
The inspectors determined the licensees overall maintenance strategy for the ICW/CCW heat exchangers outlet valves to be adequate and consistent with vendor guidance and recommendations. However, the inspectors identified that from 2007 until the failure in August 2011, the licensee failed to take corrective actions for deficiencies which were identified in valve 3-50-406 and its manual actuator. The licensee entered this issue into their CAP as AR 01680272.
The inspectors also noted that none of the subject valves were formally tested.
Depending upon the manual valve=s function, applicable regulations that ensure the manual valve can perform its safety function include:
- (1) 10 CFR 50 Appendix B Criteria III, Design Control, and Criterion XI, Testing,
- (2) 10 CFR 50.65, Maintenance Rule (MR),
- (3) License Conditions and UFSAR commitments,
- (4) the approved IST,
- (5) the Fire Protection Plan, and
- (6) the fire protection quality assurance program. Based on a review of generic communications GSI-127 and IN 2005-23 and the MR and IST regulatory requirements, the inspectors did not identify any specific regulatory requirements for testing of the locked-open valves. The ICW/CCW heat exchangers outlet valves, except 3/4-50-407, were locked-open, and because they were not required to change position to perform their safety function, they fell outside of the IST program scope. For 3/4-50-407, the inspectors determined that the valve became an Active, Category B, valve when closed in 2004 per Change Request Notice (CRN) M-11466 and would be required by ASME Operations and Maintenance Code requirements to be stroke tested. The inspectors identified this valve was not included in the scope of the IST program. However the valve is tested quarterly during ICW pump runs per 3/4-OSP-019.1. The licensee entered this issue into their CAP as AR 1680272.
including root cause analysis, extent of condition, and risk assessment of the event.
a. Inspection Scope
The team interviewed personnel and reviewed available station documents related to the licensees efforts to identify potential causes for the event. The inspectors also reviewed ARs and WOs to independently assess the conclusions from the licensees preliminary investigation.
b. Observations The inspectors noted that the licensee performed a review of engineering and licensing basis documents to understand the design basis of the ICW system regarding passive failures of any of its components. The documents included: engineering and design change reports; Stone & Webster documents used during original construction; the Updated Final Safety Analysis Report (UFSAR); and FPL correspondence with NRC.
The team concluded that except for instances specifically stated in the UFSAR, Turkey Point design criteria does not assume passive failures in fluid systems and therefore passive failures were not analyzed. Only single active failures were considered in the design of Turkey Point mechanical items or fluid systems.
A formal root cause evaluation was initiated by the licensee following restoration of the ICW system. The root cause evaluation will consider system design, operation, and maintenance issues associated with the failure, and will include programmatic and organizational issues for the failure of valve 3-50-406.
The team reviewed the results of the licensees post transient walkdown of all of similar types of valves on both Unit 3 and Unit 4 to identify any fluttering. This was conducted by visually inspecting the components and examinations using vibrometers. The results identified one valve actuator that was fluttering and the licensee scheduled appropriate corrective maintenance. The inspectors also reviewed the licensees prompt operability evaluation for the failure of valve 3-50-406 and the opening of valve 3-50-407. The inspectors also reviewed the modification to maintain valve 4-50-407 as a normally open valve on Unit 4. The team concluded that the alternate flow path on both units ICW system provided sufficient cooling capacity for the operation of safety-related equipment because using valves 3/4-50-405 and 3/4-50-407 can accommodate normal and accident flow conditions. The ICW system structural integrity was adequate such that the system could perform its function once valve 3-50-407 was opened.
.5 Assess operational considerations and controls for affected and similar valves.
a. Inspection Scope
The inspectors reviewed licensee documents and interviewed personnel to assess what controls were in place for the operation of the Unit 3 intake cooling water (ICW) valve 3-50-406 and other valves of similar design. The inspectors also compared the information contained within the valve vendor manual to licensee procedures, practices and training.
b. Observations Following the unexpected closure of valve 3-50-406 and subsequent loss of flow through the intake cooling water system (ICW), the licensee entered off-normal procedure 3-ONOP-019, Intake Cooling Water Malfunction, to restore ICW flow. This procedure contained steps for the manipulation of valves 3-50-406 and 3-50-407 to attempt restore the ICW system back to its normal operating status. The licensee took actions to open the normally closed valve 3-50-407 to restore flow ICW flow. Under the direction of a senior reactor operator (SRO) located at the event site, a plant operator began to open valve 3-50-407, first by hand to remove the disc from its closed seat, followed by use of an ~18 inch valve wrench to foster smoother rotation of the hand wheel. Shortly after the plant operator began opening the valve, an ICW system engineer arrived at the scene and assisted with the opening of the valve using the valve wrench. The inspectors reviewed the valve vendor manual and both licensee fleet and site procedures to assess the appropriateness of using the valve wrench to assist in the opening of valve 3-50-407. While the vendor manual was silent on use of a valve wrench to assist in the hand wheel operation, it did state that the valve operator was designed to accept an input torque of 450 ft-lbs, and recommended limiting the input torque to 300 ft-lbs. Based on the operator hand wheel radius of 12 inches, the 18 inch valve wrench length and an estimated 80 lbs of force that the plant operator applied to the wrench, the licensee calculated that approximately 200 ft-lbs of torque was applied to the valve operator. Both the licensees fleet and site procedures allowed for the use of valve wrenches on this type of gear operated valve, provided that direct SRO oversight was present. The inspectors noted that the fleet procedure was more restrictive than the site procedure regarding the use of valve wrenches. The fleet procedure required that a SRO be present during the use of a valve wrench at any time.
In contrast, the site procedure allowed the unsupervised use of valve wrenches provided both the wrench and the valve hand wheel met the criteria detailed in the procedure.
The licensee initiated AR 1679082 to document this issue.
failure mechanisms and damage to equipment due to the event.
a. Inspection Scope
The inspectors reviewed licensee documents, interviewed the licensees onsite metallurgist who performed the failure analysis for valve 3-50-406, and performed an independent visual assessment of the failed valve components to assess the licensees proposed failure mechanism for the valve. Additionally, the inspectors reviewed post-event walkdown findings and performed an independent walkdown of potentially affected intake cooling water (ICW) structures and components to assess any damage to equipment that occurred due to the event.
b. Observations The licensee performed an initial failure analysis on the failed actuator components of valve 3-50-406. The valve failure was attributed to a fractured lever component inside the valve actuator. The lever functioned to transfer the motion of the hand wheel to open or close the butterfly valve disc. The appearance of the fracture surfaces of the lever indicated the failure was brittle in nature with no signs of significant plastic deformation. The licensee considered several possible failure mechanisms including:
- (1) fast brittle fracture,
- (2) corrosion induced failure,
- (3) failure due to casting defects, and
- (4) fatigue. Due to the normally open position of the valve and upstream components such as heat exchanger tubes and baskets strainers that would limit debris of any significant size from impacting the valve disc, the licensee determined that fast brittle failure was not a viable failure mechanism. Likewise, the absence of extensive general corrosion and absence of casting defects on the fracture surface led the licensee to rule out items
- (2) and
- (3) as possible failure mechanisms. The licensees initial conclusion was that the probable failure mechanism of the actuator lever of valve 3-50-406 was low stress/high cycle fatigue, caused by flow induced vibration. The components were later sent out for a more detailed analysis employing scanning electron microscopy (SEM) and, as of this writing, the results were not yet available.
The licensee performed system walkdowns after the unexpected closure of valve 3-50-406. These walkdowns included both visual examinations of structures and components and examinations using vibrometers to detect for signs of potential weaknesses resulting from the pressure transient. The licensee found the 3C ICW pump discharge pressure gauge, PI-3-1452, damaged and pegged high at 46 psi (the gauge typically reads 20 psi under normal operating conditions) and the 3A CCW heat exchanger outlet channel head gasket to be leaking. The 3A CCW heat exchanger head gasket failed due to the pressure surge from valve 3-50-406 closing.
The inspectors also reviewed an assessment of the physical status of the ICW system performed by the licensee. Following repairs performed on the above mentioned damaged equipment, the licensee performed a pressure test on the ICW system to a pressure of 57 psig and did not observe any system leaks. After review of licensee documentation for post-event walkdowns and independent walkdowns performed by the inspectors, the inspectors found that the licensees assessment of equipment damaged by the event was adequate.
.7 Assess the licensees efforts to identify other systems with functions important to safety
which are vulnerable to similar single point failures.
a. Inspection Scope
The inspectors reviewed system drawings and other licensee documents as well as interviewed personnel to assess the licensees efforts to identify other systems with functions important to safety which are vulnerable to similar single point failures. This review focused on systems for which a single passive failure of a component would cause entry into a one hour or less Tech Spec (TS) action statement or entry into T.S.
3.0.3.
b. Observations Through reviews of system drawings and interviews with licensee personnel, the inspectors found that the licensee adequately identified systems susceptible to similar passive single point failures. The inspectors verified that the following systems were evaluated for passive single failure vulnerabilities: auxiliary feedwater, primary water, residual heat removal, high head safety injection, containment spray, emergency diesel generators, boration and chemical and volume control. Review of these systems revealed four components in the high head safety injection (HHSI) system that are subject to possible single point failure vulnerabilities: valves 3-867 and 4-867, the Units 3 and 4 cold leg safety injection boundary isolation valves; and valves 3-864C and 4-864C, the Units 3 and 4 refueling water storage tank to safety injection suction isolation valve. The identified failure mechanism for these four components was stem-to-disk separation, a failure mechanism different from the one provisionally assigned to valve 3-50-406. The inspectors noted that the identified possible single point failure vulnerabilities were included within the scope of the licensees ongoing root cause analysis of the event for evaluation.
4OA6 Meetings, including Exit
The inspectors presented the inspection results to Mr. Kiley and other members of licensee management on August 19, 2011. The inspectors asked the licensee whether any of the material examined during the inspection should be considered proprietary information. The licensee did not identify any proprietary information.
ATTACHMENT: SUPPPLEMENTAL INFORMATION
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee personnel
- N. Bach, Work Controls Manager
- M. Caseili, Engineering Manager
- C. Cashwell, Radiation Protection Manager
- R. Coffey, Maintenance Manager
- M. Crosby, Quality Manager
- B. Cross, Fleet Licensing Manager
- B. Carberry, Emergency Preparedness Manager
- G. Heisterman, Acting Maintenance Manager
- A. Julka, Reliability & Risk Manager
- M. Kiley, Site Vice-President
- E. McCartney, Plant General Manager
- S. Shafer, Assistant Operations Manager
- R. Tomonto, Site Licensing Manager
- R. Wright, Operations Manager
NRC personnel
- R. Croteau, Director, Division of Reactor Projects
- D. Rich, Chief, Reactor Projects Branch 3
LIST OF ITEMS
OPENED, CLOSED AND DISCUSSED
Opened
None
Opened and Closed
None
Closed
None