IR 05000400/2012008
| ML12194A281 | |
| Person / Time | |
|---|---|
| Site: | Harris  |
| Issue date: | 07/12/2012 |
| From: | Randy Musser NRC/RGN-II/DRP/RPB4 |
| To: | Burton C Carolina Power & Light Co |
| References | |
| FOIA/PA-2013-0250 IR-12-008 | |
| Download: ML12194A281 (29) | |
Text
UNITED STATES uly 12, 2012
SUBJECT:
SHEARON HARRIS NUCLEAR POWER PLANT - NRC SPECIAL INSPECTION REPORT 05000400/2012008
Dear Mr. Burton:
On May 30, 2012, the U.S. Nuclear Regulatory Commission (NRC) completed a reactive inspection pursuant to Inspection Procedure 93812, Special Inspection at your Shearon Harris reactor facility Unit 1. The enclosed inspection report documents the inspection results which were discussed on May 30, 2012, with you and other members of your staff.
The special inspection was commenced on May 7, 2012, in accordance with Management Directive 8.3, NRC Incident Investigation Program, and Inspection Manual 0309, Reactive Inspection Decision Basis for Reactors, based on the initial risk and deterministic criteria evaluation made by the NRC on April 24, 2012.
The special inspection reviewed the circumstances surrounding the failure of two safety-related main steam isolation valves (MSIVs) to close which occurred on April 21, 2012, and examined activities conducted under your license as they relate to safety and compliance with the Commissions rule and regulations and with the conditions of your license. The inspection started on May 7, 2012, and the preliminary inspection results were discussed with you and members of your staff on May 11, 2012. Subsequent onsite inspections were conducted May 16 - 18, 2012, to observe MSIV testing following your maintenance repairs, and further in-office reviews of post-maintenance testing results were conducted May 21 - 25, 2012.
No findings were identified during this inspection. In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs Agencywide Document Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Randall A. Musser, Chief Reactor Projects Branch 4 Division of Reactor Projects Docket Nos.: 50-400 License No.: NPF-63
Enclosure:
NRC Inspection Report 05000400/2012008 w/Attachment: Supplemental Information
REGION II==
Docket No. 50-400 License No. NPF-63 Report No. 05000400/2012008 Licensee: Carolina Power and Light Company Facility: Shearon Harris Nuclear Power Plant, Unit 1 Location: 5413 Shearon Harris Road New Hill, NC 27562 Dates: May 7-11, 2012, May 16-18, 2012, and May 21-25, 2012 Inspectors: J. Zeiler, Senior Resident Inspector, McGuire (Team Lead)
P. Lessard, Resident Inspector, Harris M. Farnan, Mechanical Engineer, NRR T. Steadham, Senior Construction Project Inspector, DCP Approved by: Randall A. Musser, Chief Reactor Projects Branch 4 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000400/2012008; 05/07/2012-05/25/2012; Shearon Harris Nuclear Power Plant, Unit 1;
Special Inspection.
This report documents special inspection activities conducted by a senior resident inspector, resident inspector, mechanical engineer, and senior construction project inspector to investigate the failure of two Main Steam Isolation Valves (MSIVs) to close during surveillance testing.
NRC-Identified and Self-Revealing Findings
None
Licensee-Identified Violations
None
REPORT DETAILS
Summary of Plant Events On April 12, 2012, the plant was in Mode 4, at zero percent power, in the process of a normal shutdown for a refueling outage. At approximately 4:45 am, the normally outage scheduled surveillance test to verify operability of the three main steam isolation valves (MSIVs) via stroke time testing was being performed in accordance with procedure OST-1046, MSIV Operability Test. At the start of this testing, all three MSIVs were in their normally open position from the previous 18-month operating cycle that had just completed. The A MSIV was tested first from the main control board (MCB) controls and stroked close in 4.51 seconds, which was within the required 5 second acceptance criterion. However, when the B and C MSIVs were given signals to close from the MCB controls, both valves failed to close. Actions were initiated to isolate the instrument air supply to the valve actuators which had no immediate effect.
Subsequently, without any other licensee actions, the B MSIV shut 37 minutes after instrument air was isolated and the C MSIV shut about 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> and 7 minutes after instrument air was isolated. This event was reported to the NRC as Event Notification (EN) 47857, dated April 21, and updated on April 24.
Between April 28 - 30, 2012, as part of maintenance troubleshooting on the MSIVs, the licensee conducted as-found air-operated valve (AOV) diagnostic testing on each MSIV. AOV testing was initially conducted with the actuator coupled to the valve, followed by AOV testing with the actuator uncoupled. Based on these test results, the licensee determined that the actuator performance for each valve was normal; however, there was high internal valve friction within the valve body of the B and C MSIV, as well as the A MSIV. On May 4, 2012, following the completion of as-found AOV diagnostic testing, the first valve (i.e., C MSIV) was disassembled and visually inspected. Excessive jacking force was needed to extract the piston-disc assembly from its valve body. Upon disassembly, unexpected corrosion was identified on the surfaces of the four disc piston rings, which are contained in pairs within two groves around the top of the upper structure of the valve disc. In addition, the piston rings were found locked up with no end gap clearance. The licensee suspected the degraded piston ring condition could have caused the high internal friction between the valve disc and body bore resulting in the failure of the MSIVs to stroke close. The licensee initiated actions to inspect the valve internal of the remaining two MSIVs.
Special Inspection Charter Inspection Objectives In accordance with Management Directive (MD) 8.3, NRC Incident Investigation Program, deterministic and conditional risk criteria were used to evaluate the level of NRC response for this operational event. Through review of the MD 8.3 deterministic criteria, the staff concluded that this event resulted in a significant loss of integrity of the primary containment boundary, a loss of safety function, and could possibly involve adverse generic implications. The staff noted this event may also have involved a repetitive failure in that a similar event occurred with the B MSIV on November 15, 2009. The conditional core damage probability (CCDP) for this event met the criterion for a Special Inspection. Region II determined that the appropriate level of NRC response was a Special Inspection.
The Special Inspection was chartered to identify the circumstances surrounding the failure of the B and C MSIVs to close from the main control board, review the licensees actions following discovery of the conditions, the condition of the A MSIV and the licensees actions to address the November 2009 MSIV failure to close. The inspection charter objectives were to perform data gathering and fact-finding in order to address the following:
1. Develop a sequence of events, including operator actions in response to the slow closure of
the B and C MSIVs.
2. Review and assess the use of applicable operating procedures during the event and the
applicable licensee (ERFIS) plant computer data.
3. Assess the available information on the main control board MSIV switches and their
maintenance / repair history.
to identify other potential vulnerabilities or maintenance practices which could have contributed to this condition.
6. Review the licensees corrective actions (CAs), causal analysis and extent of condition
associated with the MSIV failure to close issue. Considerations should include:
- Operational decision making
- Operational experience (internal and external)
- Vendor information on expected service life, recommended preventative maintenance, and if any bulletins or addendums were issued on similar valves or air actuators.
- November 2009 B MSIV failure to close
- Condition of the A MSIV
7. Collect data necessary to support completion of the significance determination process, if
applicable.
8. Identify any potential generic safety issues and make recommendations for appropriate
follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
The special inspection team conducted their reviews in accordance with NRC Inspection Procedure 93812, Special Inspection Procedure. The inspection team reviewed procedures, corrective action documents, and design and maintenance records for the equipment of concern. The inspection team interviewed key station personnel regarding the events, including operators, maintenance, engineering, and valve vendor personnel.
Background - MSIV Functional Design and Technical Information One MSIV is located in each main steam supply line downstream of the main steam safety valves and power-operated relief valve and are fully open during power operations and designed to close within 5 seconds to isolate steam flow from either direction when called upon to trip close during a design basis steam generator tube rupture event or steam-line break accident upstream or downstream of the MSIV. The MSIVs are held in the open position by instrument air which exerts pressure on the bottom of the piston actuator. Upon receipt of a closure signal, solenoid vent valves open to vent air from the bottom of the piston actuator, and the valve is closed by spring pressure on the piston actuator. The valve design also incorporates a means to perform a partial closure stroke (approximately 90 percent open movement) during normal operation, i.e., with steam flow present, or during plant shutdown, without steam flow present.
All three MSIVs are a 32 inch Y Pattern Globe valve model 1612GJMMTY manufactured by Rockwell International (now owned by Flowserve). The valve is a bi-directional balanced designthat results in low pressure drops. The major parts of the valve internals are the main disk, piston, piston rings, stem, stem pusher, check element, and back seat.
The valve is operated by an air-to-open, spring-to-close actuator model SA-A032 manufactured by R. A. Hiller (now owned by Rotork). The actuator is a pneumatic cylinder coupled in tandem with a hydraulic cylinder. The pneumatic cylinder provides the necessary force to compress internal springs and the springs provide a downward force in the absence of air. The hydraulic portion of the tandem cylinder provides a constant velocity to the downward force. The principal parts of the actuator are the springs, pneumatic cylinder, hydraulic cylinder, and actuator rod.
The actuator/valve assembly is operated by an electrical command system which is composed of electrical controls and a system of solenoid control and pilot operated air valves. Upon receipt of a valve open signal, air is directed via solenoid/air valve matrix to the actuator assembly to open the valve. Upon receipt of a valve close signal, the solenoid/air valves within the electric command system change state to exhaust the air in the actuator pneumatic cylinder.
Specific Valve Information: (See Figure 1)
Figure 1
- Valve Body - The basic valve is a Y pattern cast steel globe design where the bonnet of the valve is at 45 degrees. Internal to the valve are guide ribs of stellite #21 which provides a surface for the lower portion of the disk-piston assembly, which is also treated with stellite
- 21. The hard stellite #21 material resists corrosion and wear and reduces side loading.
- Main Valve Seat - The main valve seat area is an integral hardfaced seat of stellite #21.
The hard material resists erosion and wear under high velocity flow conditions.
- Back Seat - The back seat is also treated with stellite #21. It provides positive shut off between the bonnet and stem and allows packing to be replaced with the valve inservice and under pressure.
- Main Disk - The main disk provides the seal between the upstream and downstream sides of the valve. The disk has an internal check element assembly, the primary function of which is to act as a one way passage between the pressure under the disk and the piston balancing chamber. The check element is forced closed via internal conical spring when the pressures below and above the disk are equal. An imbalance of 2 psi will cause the check element to open. The check element is attached to the stem just below the pusher assembly. The check element disk and seat areas are also treated with stellite #21. Below the main disk check element area is a deflector plate assembly. The deflector plate assembly is designed to control excessive pull down forces generated by high velocity and mass flow rates with underseat flow.
- Valve Stem - The valve stem has a T-head design at the end of the shaft with a stem nut. A pusher assembly is screwed onto the end of the stem shaft and maintained with a lock pin.
The pusher assembly is the component that will lift and/or push the piston portion of the valve. The stem/pusher assembly is then attached to the check element assembly by first inserting the conical spring inside the check element, insert the T-head end of the stem shaft into the check element, and securing the two components with the stem nut. Once secure, the stem nut is locked into place with a locking pin.
- Piston - The top of the piston is hardfaced with stellite #21 to provide a wear resistant surface. In addition, there are two rows of piston rings located at the top of the piston which provides a guiding surface for the piston to move while providing a sealing action. They are comprised of an inner and outer ring for maximum sealing and made of a carbon impregnated cast iron which provides self lubrication. The piston is the last component assembled to complete the valve internal make up. The main disk has the deflector plate assembled onto its lower portion. Once fitted, the stem, stem pusher and check element assembly is lowered onto the upper portion of the main disk. The piston is then lowered over the stem and screws onto the main disk assembly. When the piston is tight, it is secured with a locking pin. Once assembled, the stem-piston-disk assembly is inserted into the main valve body bore.
Actuator Information:
The R.A. Hiller model SA-A032 is a pneumatic valve actuator and control system to be used on a large globe valve intended for nuclear power plant service. The model SA-A032 consists of a 38 inch bore pneumatic cylinder coupled in tandem with a 6 inch bore hydraulic cylinder. The actuator is an air-to-open, spring-to-close design. There are four guide tubes that support a total of six springs per tube. There are three inner springs and three outer springs per guided tube for a total of 24 springs. This design produces 63,988 pounds of force when the springs are compressed to the valve back seat area and 37,060 pounds of force when the valve is seated. The hydraulic cylinder consists of adjustable flow control valves which help to regulate and provide a constant velocity of the main valve disk regardless of various loads. The adjustable flow control valves will also affect the speed of the valve. This system is commonly referred to as the hydraulic speed control system.
Electrical Command System:
The electrical command system is composed of the electrical controls necessary to open, close, and partially exercise the MSIV. Opening the valve is accomplished by energizing two series solenoid control valves which provides control air to three pilot operated air valves. This directs the main air supply to the rod end of the pneumatic cylinder and starts retracting the actuator against the valve spring force. Closing the MSIV is accomplished by de-energizing the solenoid control valves which removes the air supply to the pilot operated air valves releasing air in the pneumatic cylinder via the exhaust ports on the pilot valves.
OTHER ACTIVITIES
4OA5 Other Activities - Special Inspection
.1 Develop a sequence of events, including operator actions in response to the slow
closure of the B and C MSIVs (Charter Item 1)
The inspection team interviewed operations and maintenance personnel who were directly involved with the event, reviewed pertinent plant computer data and control room logs, and reviewed written statements from other personnel that were involved with the MSIV testing on April 21, 2012. The inspection team used this information to develop a sequence of events and operator actions in response to the slow closure of the B and C MSIVs. The inspection team developed the following sequence of events ending at the start of the special inspection:
Date Time Narrative 04/21/12 3:17:00 AM The plant entered Mode 4, Hot Shutdown.
04/21/12 Main Control Room (MCR) operator informed an operator in the steam tunnel that all 3 MSIVs would be stroked. The MSIV system engineer was in the steam tunnel to observe valve closure.
04/21/12 MCR operator placed A MSIV hand switch in closed position and observed both dual and full closed indications within the expected timeframes. The operator in steam tunnel heard the air vent from the actuator with nothing noted as abnormal.
04/21/12 4:50:44 AM The A MSIV not full open annunciator was received in the MCR.
04/21/12 4:50:48 AM The A MSIV closed limit switch actuated approximately 4.51 seconds after the MSIV close signal was initiated by the operator.
04/21/12 MCR operator placed B MSIV hand switch in closed position and received dual indication within the expected time frame but did not receive the full closed indication. The operator in the steam tunnel heard the air vent from the actuator with nothing noted as abnormal.
04/21/12 4:53:36 AM The B MSIV not full open annunciator was received in the MCR.
04/21/12 The MCR operator contacted the operator in the steam tunnel to inspect B MSIV who informed the MCR operator that the B MSIV appeared to still be full open.
04/21/12 4:59:00 AM Control room operators contacted the operator in the steam tunnel to stand by the C MSIV while they attempted to close the valve.
04/21/12 Control room operators placed C MSIV hand switch in closed position. The control room operator received dual indication within the expected time frame but did not receive the full closed indication.
04/21/12 5:00:11 AM The C MSIV not full open annunciator was received in the MCR.
04/21/12 The operator standing by the C MSIV heard and felt the air being vented from the actuator and informed the MCR that C MSIV valve actuator vented but the valve did not move.
Local operator stated that the air venting from all three MSIVs was of similar duration.
04/21/12 5:00:45 AM Operators entered Technical Specification (TS) 3.0.3 based on declaration that B and C MSIV were inoperable.
04/21/12 The MSIV system engineer reported that he heard a similar air vent from all three actuators. The engineer stated that it lasted about 1 second, rapidly dropped off, with no additional air venting. The engineer observed the position of the B and C MSIVs and identified that each was just barely off the open limit switch and essentially full open.
04/21/12 The MCR contacted the outage control center to isolate air to the MSIVs due to their failure to close on demand.
04/21/12 An operator was dispatched to steam tunnel to isolate the air to each of the MSIV actuators. Using an off-normal tagging procedure, the operator closed the air isolation valve to the pressure regulator upstream of the A MSIV air solenoid valves. The operator opened the regulator drain petcock and observed approximately 3 seconds of air venting which he attributed to the air volume between the isolation valve and the actuator solenoid valves.
04/21/12 The operator isolated air to B MSIV in a similar manner and observed a similar amount of air from the drain but observed no valve movement.
04/21/12 The operator isolated air to C MSIV in a similar manner and observed a similar amount of air from the drain but observed no valve movement.
04/21/12 5:30:00 AM Instrument air was confirmed to be isolated to all MSIVs with no change in MSIV movement noted.
04/21/12 All personnel exited the steam tunnel.
04/21/12 6:00:00 AM Operations plan to reach Mode 5 is complete per Technical Specification 3.0.3 04/21/12 6:07:50 AM B MSIV closed 1 hr-14 min after close signal based on receipt of ERFIS computer position indication.
04/21/12 7:37:00 AM Troubleshooting plan approved by the operations shift manager.
04/21/12 Maintenance technicians verified that the C MSIV air isolation valve was closed and disconnected the tubing from the regulator. No air was vented. They then loosened tubing connections in 5 additional points and verified that there was no air to the 3 pilot operated air valves and observed that all 3 of the pilot valve exhaust ports were open. They visually verified that the C MSIV striker plate was contacting the upper limit switch but that the plate had moved enough such that the switch was not made up.
04/21/12 All personnel exited the steam tunnel.
04/21/12 9:38:00 AM The C MSIV closed 4 hr-37 minutes after close signal based on receipt of ERFIS computer position indication.
04/21/12 9:48:00 AM Personnel entered the steam tunnel to locally confirm that all MSIVs were closed.
04/21/12 11:27:00 AM Plant is in Mode 5. Because the MSIVs are no longer required to be operable in this mode, operators exited TS 3.0.3.
04/21/12 12:45:00 PM Operators completed an 8-hr event notification call per 10CFR50.72.
04/22/12 The licensee formed a root cause team to investigate the event.
04/27/12 Initial diagnostic testing on C MSIV was performed identifying high internal valve friction.
04/29/12 Initial diagnostic testing on B MSIV was performed identifying high internal valve friction.
04/30/12 Initial diagnostic testing on A MSIV was performed identifying high internal valve friction.
05/01/12 Maintenance uncoupled the C MSIV actuator from the valve and re-performed diagnostic testing which demonstrated proper actuator performance.
05/03/12 Maintenance removed and inspected the valve bonnet from the C MSIV.
05/04/12 Maintenance removed and inspected the C MSIV valve internals. Excessive jacking force was necessary to remove the valve disc from the body bore. The disc piston rings were found locked up with no end gap and unexpected corrosion observed on the piston ring surfaces.
05/05/12 Maintenance uncoupled the B MSIV actuator from the valve and re-performed diagnostic testing which demonstrated proper actuator performance.
05/07/12 Maintenance removed and inspected the B MSIV valve bonnet and valve internals. Excessive jacking force was necessary to remove the valve disc from the body bore.
05/07/12 2:00:00 PM The NRC special inspection team conducted an entrance meeting.
.2 Review and assess the use of applicable operating procedures during the event and the
applicable (ERFIS) plant computer data (Charter Item 2)
a. Inspection Scope
The inspection team interviewed operations and maintenance personnel who were directly involved with the event, reviewed pertinent plant computer data and control room logs, and reviewed written statements from other personnel that were involved with the MSIV testing on April 21, 2012. The inspection team reviewed copies of the procedures that were in use at the time of the event to determine if personnel were following approved procedures that were appropriate to the circumstances. The inspection team reviewed the operator implementation of applicable TS, operating procedures (such as standard operating procedures, abnormal operating procedures, emergency operating procedures, alarm response, plant status control, component operational guidance, and general conduct of operations) to determine if the operators response was in accordance with all applicable procedures and TS requirements.
The inspection team reviewed the licensees emergency plan to determine if the event met any of the emergency action level thresholds. The inspection team also reviewed the event notification to determine if it was reported within the time required by 10 CFR 50.72. Documents reviewed are listed in the attachment to this report.
b. Findings and Observations
No findings were identified. The inspection team determined that operators used approved procedures when they initially attempted to stroke the valves and the procedure steps were signed off as appropriate. The operators appropriately declared the B and C MSIVs inoperable based on the stroke test failures and entered the correct TS actions for the applicable plant conditions. During subsequent attempts to close the B and C MSIVs by isolating air to the actuators, the operators used an approved off-normal tagging procedure to perform the required instrument air alignments. The licensee properly classified this event in accordance with station procedures and NRC requirements.
.3 Assess the available information on the main control board MSIV switches and their
maintenance/repair history (Charter Item 3)
a. Inspection Scope
The inspection team walked down the A, B, and C MSIV switches and available valve indications in the main control room to determine if control room operators had adequate indications to determine the position of each MSIV. The inspection team also reviewed the maintenance and repair history of all main control room switches similar to (and including) the MSIV control switches to determine if their history indicated a potential issue that could have contributed to this event. Documents reviewed are listed in the attachment to this report.
b. Findings and Observations
No findings were identified. The inspection team did not identify any potential underlying issue with the MSIV main control board switches that could have contributed to this event. Available information from this event indicated that the solenoid valves for each MSIV de-energized and the air control valves changed state to exhaust the air from each actuator.
equipment to identify other potential vulnerabilities or maintenance practices which could have contributed to this condition (Charter Item 4)
a. Inspection Scope
The inspection team reviewed historical MSIV corrective maintenance repairs and preventive maintenance (PM), including maintenance activities associated with the valve internals, hydraulic / pneumatic actuators, and air-operated solenoid valves, to determine if potential vulnerabilities or maintenance practices could have contributed to the degraded MSIV conditions. The inspection team focused attention on those activities that might explain the high internal valve friction that was identified as a potential cause of the B and C MSIV failure to stroke close on April 21, 2012. In addition, the inspection team reviewed MSIV test program practices and TS required stroke test data for the previous five strokes of each MSIV to determine if test requirements were being properly conducted and if any degraded test result trends existed. Documents reviewed are listed in the attachment to this report.
b. Findings and Observations
Review of Corrective and Preventive Maintenance History The following is a summation of the maintenance history for the three MSIVs:
Year 1MS-80 A MSIV 1MS-82 B MSIV 1MS-84 C MSIV 1986 Replaced valve disk Replaced valve disk Replaced valve disk assembly assembly assembly 1992 Rebuilt actuator (PM) Rebuilt actuator (PM)1994 Rebuilt actuator (PM)2001 Repacked valve Actuator would not open valve without additional air volume 2003 Replace actuator with rebuilt unit (PM) and repacked valve 2004 Replace actuator with rebuilt unit (PM)2006 Replace actuator with rebuilt unit (PM) and repacked valve 2007 Failure to close within stroke time requirements of 3 - 5 seconds (5.17 seconds)2009 Actuator would not open Actuator would not open valve without additional valve due to regulator air volume mis-configured 2009 Failure to close because of problems with solenoid operated valves 2010 Actuator would not open valve without additional air volume The inspection team did not identify any past valve failures directly related to high internal valve friction from their review of the MSIV maintenance history. However, it was noted that in the last two refueling intervals, maintenance was making minor adjustments to the actuator hydraulic speed control system to decrease the time needed to shut the valves as a result of increasing stroke test closure time results. In retrospect, these minor hydraulic speed adjustments were possible indications that valve internal friction factors were increasing. Also, beginning in 2001, work deficiency documents were initiated due to the MSIVs experiencing difficulty in opening during refueling outage cycling. At the time, the licensee concluded that this phenomenon was due to the small diameter air supply tubing (i.e., 3/8 inch) and only took actions to add supplemental air supplies when the condition was encountered. The licensee did not justify why these issues might suddenly be occurring after years of successful opening without supplemental air. The inspection team suspected that this was another indicator that valve internal friction factors were increasing.
The inspection team noted from the maintenance work history review that the valve actuators and the air operated solenoid valves were on a periodic PM schedule. The actuators were being removed and rebuilt offsite by the associated vendor on a 7-year frequency and the air operated solenoid valves were being replaced every other refueling outage (3-year frequency). The inspection team determined that the frequencies of these PMs were consistent with the actuator vendor recommendations and acceptable industry practices for the air operated solenoid valves. The inspection team noted that the only record of any valve disassembly activities, whereby the condition of the valve internals including the disc piston rings would have been visually observed, was in 1986, prior to plant initial startup. At this time, the valve disc assembly (including the piston rings), for all three MSIVs, was replaced. Since that time, there had not been any corrective maintenance conducted requiring valve internal disassembly and the licensee had not developed any periodic PMs to visually inspect the condition of valve internals.
While the inspection team considered the lack of PMs for visually inspecting the valve internals to be unusual, based on review of the Rockwell International vendor manual, there was no vendor requirement or recommendation for such PMs. However, based on subsequent information from the valve vendor and conversations with several licensees with similar valves, it was found that some licensees conduct vendor refurbishments of the valve disc and check assembly, including replacement of the valve piston rings at periodic (e.g., 10-year) frequencies. In addition, the valve vendor indicated that sometime in the 1980's, a more corrosion resistant piston ring was developed and provided to users requesting replacement piston ring parts. However, there was no vendor requirement nor recommendation to replace the original rings with the newer rings since the vendor considered the newer piston ring material an upgrade/enhancement and the older rings were still considered acceptable.
Although the licensee experienced several air operated solenoid valve failures in the past, available information from the current event indicated that the solenoid valves for each MSIV operated as designed based on direct observations that air was exhausted from each MSIV actuator. The inspection team agreed with the licensees conclusion that the 2009 B MSIV failure to close event, which was a result of failure of two air operated solenoid valves, was not related to the current event.
Review of Previous MSIV Testing In accordance with HNP TS 4.7.1.5 Surveillance Requirements, each MSIV is required to be demonstrated operable by verifying full closure within 5 seconds when tested pursuant to the licensees Inservice Testing (IST) Program. The licensees IST program requires the MSIVs to be stroke tested every refueling outage (18-month frequency).
Based on review of completed stroke test procedures (i.e., OST-1046, Main Steam Isolation Valve Operability Test), the inspection team determined that the licensee had been conducting the tests at the frequency prescribed by their IST Program. The only recent valve stroke test failure was in 2007, when the B MSIV stroked closed in 5.17 seconds. This failure was attributed to an oversight from the previous 18 month stroke test (in 2006) that resulted in 4.92 seconds, which exceeded the administrative stroke time limit, but was not evaluated to adjust the actuator hydraulic speed control to decrease the closing speed.
The inspection team noted that the valve vendor manual recommended weekly valve partial exercising ten percent of its total stroke in order to assure that the actuator and valve was properly functioning. Prior to 2000, this partial exercising was being performed quarterly. In 2000, the licensee revised their IST program requirements to discontinue quarterly exercising in lieu of the 18-month cold shutdown TS stroke testing that was currently being conducted. During review of the licensees associated 10 CFR 50.59 evaluation that was conducted to support this change, the inspection team noted that it focused on the benefit in reducing the potential for plant trips/transients from test malfunctions, but did not address the impact that the change might have on valve reliability or that the change was contrary to the vendor recommendations for partial exercising. However, the inspection team acknowledged that other licensees have implemented similar IST program changes. The licensee initiated Nuclear Condition Report (NCR) 535314 to the address this issue.
The inspection team noted that the MSIVs had never been tested as part of the licensees AOV program prior to the current MSIV failures. The licensee indicated that the MSIVs were categorized as Category 2 valves, and as such, did not require periodic AOV diagnostic testing. However, based on the inspection teams review of the AOV program document, EGR-NGGC-0205, Air Operated Valve Reliability Program, Revision 7, the inspection team noted the valves did meet the definition for Category 1 valves which would have required periodic AOV diagnostic testing. The procedure defined a Category 1 AOV as those valves that perform an active safety-related Maintenance Rule function and have high safety significance. The MSIVs were safety-related active components and the licensees Maintenance Rule program classified the valves as having high safety significant function. The licensee initiated NCR 536078 to address the inspection teams concern over the AOV categorization of the MSIVs.
Conclusions (Opened) Unresolved Item (URI), B and C MSIVs Fail to Close During Surveillance Testing
Introduction:
The inspectors identified an URI associated with issues in the licensees MSIV maintenance and testing. These issues were potential contributing causes to the April 21, 2012, B and C MSIV failure to stroke close.
Description:
Several issues were identified regarding the licensees MSIV maintenance and testing. Some of the issues identified were:
- In the last two refueling intervals, maintenance was making minor adjustments to the actuator hydraulic speed control system to decrease the time needed to shut the valves as a result of increasing stroke test closure time results.
- Beginning in 2001, work deficiency documents were initiated due to the MSIVs experiencing difficulty in opening during refueling outage cycling.
- There had not been any corrective maintenance conducted requiring valve internal disassembly and the licensee had not developed any periodic PMs to visually inspect the condition of valve internals.
- The valve vendor manual recommended weekly valve partial exercising ten percent of its total stroke in order to assure that the actuator and valve was properly functioning. Prior to 2000, this partial exercising was being performed quarterly. In 2000, the licensee revised their IST program requirements to discontinue quarterly exercising in lieu of the 18-month cold shutdown TS stroke testing that was currently being conducted.
- Prior to the current MSIV failures; the MSIVs had never been tested as part of the licensees AOV program.
Summary: The licensees root cause investigation was not completed at the conclusion of the special inspection; the determination as to whether these issues represented performance deficiencies was not completed. Pending completion of the licensees root cause evaluation (RCE) and subsequent NRC review to determine if a performance deficiency exists, disposition of these issues will be tracked via Unresolved Item (URI)05000400/2012008-01, B and C MSIVs Fail to Close During Surveillance Testing.
(Charter Item 5)
And Review the licensees corrective actions, causal analysis and extent of condition associated with the MSIV failure to close issue (Charter Item 6)
a. Inspection Scope
The inspection team reviewed the licensees troubleshooting plans and activities, ongoing root cause analysis efforts, extent of condition reviews, and planned corrective actions to repair the MSIVs to an operable condition. The inspection team performed field walk downs of the MSIV components, both in the plant and in the maintenance shop, to observe the as-found condition of the valve components throughout the various phases of troubleshooting, valve disassembly and reassembly, and post-maintenance testing. The inspection team interviewed operations, maintenance, and engineering personnel who were directly involved with the troubleshooting and root cause analysis efforts. Documents reviewed are listed in the attachment to this report.
b. Findings and Observations
No findings were identified. The licensee promptly assembled a RCE team to address the failure of the B and C MSIVs. The initial troubleshooting actions were well planned and executed to ensure successful information gathering and follow-up assessment of the problems associated with the failures. Detailed troubleshooting plans were developed for each MSIV to evaluate potential failure modes associated with the following major component areas:
- Main Control Board Controls
- 3-Way Valve Pneumatic Components
- Hydraulic Components
- Actuator Pneumatic Circuit
- Valve Internals
- Valve Spring Assembly Thorough consideration for extent of condition was exhibited, in that, the A MSIV, which stroked within its allowed time, was included in the troubleshooting and root cause investigations. Using a support/refute failure mode methodology, the licensees RCE team identified thirty potential failure modes/causes based on the initial data collected surrounding the circumstances of the failures. The investigation included external visual inspections, AOV diagnostics, valve disassembly and inspections, and inspection and vendor forensic evaluation of the hydraulic / pneumatic actuator. The licensee utilized valve vendor (Flowserve) technical and design support throughout the initial valve troubleshooting and disassembly activities. The following is a brief summary of the major troubleshooting plan items developed and the associated results:
- Evaluate electrical command system to include main control room controls and the 3-way valve components: A systematic review of all three MSIV electrical command system components was performed. Included in the review were the main control board switches and controls, 3-way solenoid operated valves, air control valves, and all external filters and vents. No anomalies were found and it was concluded that the electrical command system performed as designed.
- Perform external visual inspection of the valves: An external visual inspection was performed. No anomalies were noted. During the initial operator actions to isolate air to the MSIV actuators, the operators noted that the valve stem moved 1 to 2 inches when air was vented. The licensee determined that this was most likely stem movement to fully engage the check element with the disc pusher and the disc pusher with the valve main piston. Therefore, this initial stem movement did not result in actual movement of the main disc.
- Perform as-found AOV diagnostic testing on valve and actuator assembly: As-found diagnostic testing was performed on all three valves. A special test rig had to be constructed due to the fact that diagnostics had never been performed on these valves. The diagnostic test consisted of slow stroking the valve by applying a gradual rising air pressure to the underside of the pneumatic piston cylinder on the actuator assembly while recording applied air pressure versus valve travel. By knowing the amount of air pressure delivered to the pneumatic piston cylinder area, the amount of force developed to overcome the spring tension can be calculated.
During the as-found diagnostic tests, all three valves indicated a higher than expected friction factor and non-uniform valve stroking. The wider the gap between the open stroke versus the close stroke indicates higher friction factors. Higher friction factors during valve stroke can be caused by stem packing, irregular stem, high friction between the piston rings and valve body, valve body anomalies, high friction between the lower disk body and the valve guide ribs, and higher friction factors in the actuator assembly. All three MSIVs exhibited high irregular friction factors with the A MSIV having the widest friction gap. After reviewing the results of the as-found diagnostic testing, the licensee determined that it was necessary to inspect/rework all three MSIVs and their associated actuators.
- Uncouple actuator from valve assembly and perform diagnostic testing on the actuator alone and evaluate: After de-coupling the actuator from the valve, another diagnostic test was performed on all three MSIV actuators. The diagnostic test on all three valve actuators yielded no anomalies. The actuators performed as expected.
- Disassemble valve and inspect: Excessive jacking force was necessary to remove the disc from all three MSIVs (i.e., about 29,000 pounds for A, 7,000 pounds for B, and 28,000 pounds for C). The disc nominally weighs 3,500 pounds. No significant damage was noted in any of the valve bodies and valve body guides.
Measured clearances of the valve bodies were acceptable. There was no significant scoring of valve or actuator stems in all three components. No significant damage or findings in the valve packing and backseat areas. No significant damage or findings on the valve disk and piston body. Measured clearances of the disk and piston assembly were acceptable. The valve disc piston rings on all three components were found locked up with no end gap clearance and were noticeably degraded.
The exterior of all three valve ring components had a corrosion type orange peel appearance instead of a smooth surface. The ring assemblies on all three valves had to be destructively removed from the piston due to binding. Measurements of the ring assemblies noted that the corrosion byproduct caused the overall dimensions to be swollen greater than the maximum ring diameter tolerances.
Samples of the removed rings were sent to metallurgy lab for examination and analysis. It was determined that the corrosion of the ring assemblies was the most likely cause of the B and C MSIV failures to close and the reason why all three MSIVs experienced high internal friction during as-found AOV diagnostic testing.
- Inspect actuator and hydraulic system: A diagnostic test was performed on all three actuators in the uncoupled configuration. No anomalies were noted and all three actuators performed as expected. A sample of the hydraulic oil was sent out for analysis and preliminary results noted no anomalies. Two of the actuators were sent out to the manufacturer for disassembly, inspection, and refurbishment. These actuators were re-installed on the B and C MSIVs. The A MSIV actuator was inspected and verified adequate to be replaced on the A MSIV.
- Evaluation of the 2009 failure to close event on 1MS-82 B MSIV: The licensee determined that the 2009 event was not the same type of failure. The root cause of the 2009 B MSIV failure to close was due to a problem within the electrical command system. Specifically, two solenoid operated valves failed to perform their safety function.
- Evaluate industry operational experience: The licensee conducted an extensive search and review of industry operational experience. The search yielded a utility with the same model valve. Feedback from that utility noted that they had a periodic valve refurbishment PM task which sends the valve internals out to the manufacturer for rework. The history has shown this PM to be effective. HNP inquired with the valve manufacturer if the removal of the piston rings was a possible option. The manufacturer responded that the rings play a role in the balancing of the piston/disk assembly during operation. With the rings installed, the valve balance efficiency is expected to be 95 percent. With the rings removed, the valve balance efficiency would drop off to around 80 percent. This would affect overall valve margin and place the piston/disk assembly into question on overall performance. The manufacturer does not recommend and/or endorse having the rings removed from the piston/disk assembly. Some utilities noted that they had their MSIV on a diagnostic monitoring program. HNP management has indicated that going forward the three MSIVs will have a periodic diagnostic test PM.
The inspection team determined that the licensee developed detailed troubleshooting and causal investigation plans for addressing the valve failures. Thorough consideration was given for all potential failure modes of the individual valve components and a meticulous plan was developed to gather the appropriate information to provide a basis for supporting or refuting the potential failure modes. The inspection team did not identify any potential failure modes that the licensee had not addressed.
At the conclusion of the special inspection, the licensees RCE and metallurgical examination of the degraded piston rings was still ongoing. The licensee preliminarily concluded that the cause of the MSIV failure to stroke was high internal valve friction caused by corrosion of the piston rings and corrosion bonding of the rings to the valve body. The corrosion caused swelling of the cast iron piston ring material (0.010 to 0.012 inch over the maximum diameter tolerance) resulting in the rings being locked-up (i.e.,
tight inside their respective mating groves on the valve disc), with no end gap clearance.
Measurements of new, unused piston rings were at least 0.003 inch below the maximum diameter tolerance. Internal inspections of the valve body noted discernable "rings" on the internal valve body bore at the location of the rings when the valve is in the open position. The licensee suspected that because the plant operated "breaker to breaker" over the past operating cycle, and the MSIVs had not been cycled since the last refueling outage (i.e., November 11, 2010), this allowed more time for buildup of the corrosive layer of material against the valve body bore and enlarged piston rings, preventing the valve spring force from initially overcoming the breakaway force.
The piston rings on each MSIV were original equipment and had not been replaced since plant initial commercial operation (approximately 26 years ago). In the 1980's, the vendor developed a more corrosion resistant piston ring, and now uses the new piston rings in new valves, as well as the replacement part if rings are ordered by its valve users. However, there was not any vendor requirement or recommendation to replace the original rings with the newer rings.
As part of the licensees immediate corrective actions, all three MSIVs were re-assembled using the newer, more corrosion resistant piston rings. On May 18, 2012, following licensee reassembly of the first MSIV, the inspection team witnessed AOV diagnostic testing that was conducted on the A MSIV. The valve cycled satisfactorily, with no indications of increased friction forces. Based on preliminary review of the test results against the valve vendor design performance calculations, the inspection team determined that the licensee had adequately restored the valve performance to its original design specifications. The reassembly and testing of the remaining two MSIVs, between May 18 and May 24, 2012, yielded similar results as the A MSIV. The inspection team determined that the licensee had adequately addressed actions necessary to restore the safety related function of the MSIVs.
.6 Collect data necessary to support completion of the significance determination process,
if applicable (Charter Item 7)
a. Inspection Scope
The inspectors reviewed licensee procedures, corrective action program documents, work orders, preliminary root cause investigation data, operability assessments, engineering evaluations, and operating experience information to gather data necessary to develop and assess the safety significance of any findings. Documents reviewed are listed in the attachment to this report.
b. Findings and Observations
At the time of the event, with plant in hot shutdown at zero percent power, the inspection team did not identify any other plant equipment failures or human performance related operational conditions that would have complicated the plant response. Based on licensee as-found AOV diagnostic testing that identified high internal friction in all three MSIVs, there was potentially greater safety significance if the valves had failed to close during design basis accident conditions from full power operation conditions. The degraded condition of the valves would have challenged their capability of performing their safety function to isolate the steam lines during a design basis steam generator tube rupture event or steam-line break accident.
At the conclusion of the special inspection, the licensee was in the process of evaluating the probabilistic risk significance of the event assuming all three MSIVs failed to close within 5 seconds as designed during design basis accident conditions. The safety significance evaluation of this event remains open pending follow-up of the URI discussed in Section 4OA5.4 of this inspection report.
.7 Identify any potential generic safety issues and make recommendations for appropriate
follow-up action (e.g., Information Notices, Generic Letters, and Bulletins) (Charter Item 8)
a. Inspection Scope
The inspection team reviewed the licensees preliminary root cause investigation results, internal operating experience database, corrective action program database, valve vendor information, and the NRC Operating Experience database to determine the potential for generic safety issues related to the problems with the MSIVs. Documents reviewed are listed in the attachment to this report.
b. Findings and Observations
The inspection team determined that this event identified several issues that may be of generic industry-wide interest, including the following:
- Identification of degraded MSIV piston rings due to corrosion induced swelling causing high internal valve friction
- Suspension of vendor recommended online periodic partial stroke testing of MSIVs in lieu of IST program cold shutdown full stroke testing
4OA6 Management Meetings
Exit Meeting Summary
On May 30, 2012, the inspection team presented the inspection results to Mr. Burton, and other members of the licensee staff via telecom. The inspection team confirmed that proprietary information was not provided or examined during the inspection period.
ATTACHMENTS:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee personnel
- C. Burton, Vice President Harris Plant
- D. Corlett, Supervisor, Licensing/Regulatory Programs
- J. Dufner, Director, Engineering
- D. Griffith, Training Manager
- K. Holbrook, Manager, Support Services
- E. Kapopoulos, Plant General Manager
- B. McCabe, Manager, Nuclear Oversight
- M. Parker, Superintendent, Radiation Control
- M. Robinson, Superintendent, Environmental and Chemistry
- T. Slake, Manager, Security
- J. Warner, Manager, Outage and Scheduling
- F. Womack, Manager, Operations
NRC personnel
- R. Musser, Chief, Reactor Projects Branch 4, Division of Reactor Projects, Region II
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
- 05000400/2012008-01 URI B and C MSIVs Fail to Close During Surveillance Testing (Section 4OA5.4)