IR 05000348/2006009

IR 05000348-06-009 and IR 05000364-06-009 on 04/25/2006 - 05/16/2006 for Joseph M, Farley, Unit 1, Special Inspection

ML061840350
Person / Time
Site:	Farley
Issue date:	06/30/2006
From:	Scott Shaeffer NRC/RGN-II/DRP/RPB2
To:	Sumner H Southern Nuclear Operating Co
References
IR-06-009
Download: ML061840350 (32)

Text

une 30, 2006

SUBJECT:

JOSEPH M. FARLEY NUCLEAR PLANT - NRC SPECIAL INSPECTION REPORT 05000348/2006009 AND 05000364/2006009

Dear Mr. Sumner:

On May 16, 2006, the US Nuclear Regulatory Commission (NRC) completed a special inspection at your Farley Nuclear Plant, Units 1 and 2. This inspection reviewed the circumstances surrounding the failure of all three Unit 1 downstream safety-related main steam isolation valves (MSIVs) to close on April 8, 2006. In accordance with Management Directive 8.3, a special inspection was warranted because the event involved multiple failures in a system used to mitigate an actual event, involved possible adverse generic implications, and involved repetitive failures of safety-related equipment.

The enclosed inspection report documents the inspection findings, which were discussed on May 16, 2006, with Mr. Randy Johnson and other members of your staff. The determination that the inspection would be conducted was made by the NRC on April 25, 2006, and the inspection started on April 25, 2006.

This inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel. Based on the results of this inspection, no findings of significance were identified.

In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosures, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS).

SNC 2 ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Docket Nos. 50-348, 50-364 License Nos. NPF-2, NPF-8

Enclosure:

Inspection Report 05000348/2006009 and 05000364/2006009 w/Attachment 1. Supplemental Information 2. Farley MSIV Event Timeline

REGION II==

Docket Nos.: 50-348, 50-364 License Nos.: NPF-2, NPF-8 Report Nos. 05000348/2006009 and 05000364/2006009 Licensee: Southern Nuclear Operating Company, Inc.

Facility: Joseph M. Farley Nuclear Plant Location: Columbia, AL 36319 Dates: April 25 - May 16, 2006 Inspectors: J. Baptist, Resident Inspector - Farley Nuclear Plant (Team Leader)

M. Scott, Senior Reactor Inspector Approved by: Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000348/2006-009 and 05000364/2006-009; 04/25/2006-05/16/2006; Joseph M. Farley

Nuclear Plant, Unit 1, Special Inspection.

The report documents special inspection activities conducted by a resident inspector and a senior reactor inspector to investigate the failure of three Unit 1 Main Steam Isolation Valves (MSIVs) to close.

NRC-Identified and Self-Revealing Findings

No findings of significance were identified.

Licensee-Identified Violations

None.

REPORT DETAILS

Farley MSIV Design Overview Each unit of the Farley plant has three lines of main steam flow in which each line has two MSIVs installed in series downstream from the safety relief valves outside containment. The upstream and downstream MSIVs, manufactured by Atwood & Morrill (now WEIR), are identical in construction and their bodies are separated by approximately six inches of piping. The MSIVs are 32-in., 600-lb disk, full-flow, swing-check, nonreturn-type valves with pneumatic actuators. During normal plant operation, the valves are kept open against a spring force by air pressure in the actuator. In the normal open position, the disc is held well out of the steam flow by the air operator. In case of high pressure in the containment, or high steam line flow in coincidence with low-low Tavg or low steam line pressure, the air pressure in the cylinder is relieved and the valve is closed within seven seconds by action of the spring to prevent the forward flow of steam through the valve. In the event of a steam line break, the MSIVs prevent continuous uncontrolled steam release from more than one steam generator, regardless of whether the break is inside or outside the containment, even when it is assumed that there is a failure of one of the isolation valves. The valve design also incorporates a means to perform an approximate 5% movement partial closure test. This test can be performed with or without steam flow present.

Event Description On April 8, 2006, Unit 1 was reducing power in preparation for a scheduled refueling outage. At 2:38 a.m., plant operators performed

, and observed that downstream MSIVs Q1N11HV3370B (3370B) and Q1N11HV3370C (3370C) did not move from of their open position while MSIV Q1N11HV3370A (3370A) closed approximately 70 percent. The MSIVs were tested at normal operating temperature and pressure (NOT/NOP) with no steam flow as was typical for this test.

Due to all three downstream MSIVs failing to reach the fully closed position within seven seconds, the licensee entered Technical Specification (TS) 3.7.2. Per the applicable Limiting Condition for Operation (LCO), the plant was required to restore the downstream MSIVs to operable status or verify at least one valve was closed in each line. Consistent with plant conditions to support a refueling outage, all upstream MSIVs were already closed which met the LCO. At 7:00 a.m., mechanics were requested to assist in closing the downstream MSIVs with each valve requiring varying degrees of assistance for closure. Condition report (CR)2006103043 was written to document the valve closure problem and the plant continued with the shutdown. On April 9, the licensee demonstrated the ability of the downstream MSIVs to close using the normal valve actuator.

On April 17, the downstream MSIVs were disassembled, but did not reveal an immediate root cause for the failures. The licensee formalized a root cause investigation team on April 21, and began investigating each MSIV failure. Following substantial root cause investigation, the downstream MSIVs were reassembled and successfully stroke tested on May 17. All MSIVs were opened on May 23, to facilitate plant startup and then fully stroke tested under normal operating conditions to verify proper reassembly and operation.

Following a period of operation at 100% power, on June 28, during partial stroke testing of the Unit 1 MSIVs, MSIV 3370B became stuck in the test position. The licensee locally verified the disk was at the 5% test position and entered a 72-hour LCO for an inoperable MSIV. The licensee attempted various methods including lubrication, packing adjustment, and mechanical assistance to free the disk, but there was no change in position. On June 30, the licensee decided to shutdown Unit 1 to perform troubleshooting and inspection of the MSIVs.

Special Inspection Charter Inspection Objectives Based on the event and the criteria specified in Management Directive (MD) 8.3, NRC Incident Investigation Program, a Special Inspection was initiated by the NRC in accordance with Inspection Procedure 93812, Special Inspection. The inspection charter objectives are listed below and addressed in the identified report sections.

1. Develop a sequence of events, including applicable management decision points prior to

the time the MSIVs failed through troubleshooting and repair activities. (4OA3.1)

2. Review licensee documents to assess if the licensee had previous failures of MSIVs to

close and also review current and previous maintenance practices for the MSIVs.

(4OA3.2)

3. Assess any corrective action the licensee took prior to the event to address MSIV

closure problems and determine if the actions were appropriate and timely. (4OA3.2 and 4OA3.3)

4. Assess operating procedures and operator training concerning this scenario and

determine if the procedures and training were adequate for operators to compensate for the lack of MSIV closure. (4OA3.4)

5. As requested by the Regional Senior Reactor Analyst, assist in the collection of data

necessary to support completion of the significance determination process. (4OA3.1)

7. Review this event for generic safety implications. (4OA3.6)

Summary of Licensee Root Causes Based on the information available at the end of the inspection period, the licensee identified the following could have contributed to the recent Unit 1 MSIV failures.

Maintenance Procedures and Practices C No preventative maintenance replacement schedule nor defined replacement criteria for MSIV bushings (shaft load bearing components).

C The orientation of the thrust bearing on the shaft was not specified nor was there an acceptance criteria for inspection of components subject to wear, notably shaft load bearing components.

C Techniques or tools used for removing and replacing packing.

C Key/keyway installation was skill-of-the-craft.

C The maintenance procedure lacks detail and depends on knowledgeable mechanics to implement successfully.

C Lubrication of shaft and packing conflict with vendor recommendations for packing installation and may have contributed to increased packing drag.

Design C The square keyway design creates a stress riser that makes the shafts susceptible to fatigue cracking under the service conditions of the downstream valves. This was exacerbated by use of keys that were less than the full length of the keyway.

C Severe service conditions in the downstream valves, due to the turbulence induced by the upstream MSIVs, was a known condition that affects fatigue and wear factors identified in the above causes. Substantial design changes could reduce or eliminate the severe service environment of the downstream valves.

Missed Opportunities C Investigations prior to 2002 typically stopped at the first plausible or most visible possible cause.

C In 2002, an investigation into partial stroke failure (CR 2002001026) may have missed the significance of sticking on the open stop and possibly misdiagnosed the cause of the failure.

C In 2000 (CR 2000005687) a similar failure occurred, and was corrected by lubricating the yoke bushing and shaft. The potential significance of this event was missed although its occurrence was referenced in the 2002 CR.

C In 2002, the partial stroke test was discontinued based on industry operating experience and further opportunity to identify degrading conditions in the shaft support components was lost until the hot as-found surveillance specification was added in 2004.

C The surveillance requirements for these valves consist of the ASME Code required Cold Shutdown/Refueling (CSD/RF) timed stroke test. Conditions under which this test should be performed were not clearly specified to match as closely as possible, the actual operating conditions under which the valve could be called upon. The test procedure did not establish consistent conditions for testing all valves, precluding good trending.

C Other than startup and shutdown, these valves are not normally operated. The previous partial stroke test requirement was deleted based on industry experience in that MSIV partial stroke testing posed a plant trip risk believed to outweigh the benefits of quarterly partial stroke testing. It is believed that partial stroke testing mitigated part of the drag increase, notably "sticking" packing that develops over time.

C Due to the uniqueness of the design, and perception that an aggressive PM program was in place to deal with the then known issues, these valves were not placed in the AOV program for "Flowscan" monitoring.

C As a consequence of the above lack of test specification, surveillances that may have provided detection of trends, degradation, or incipient failure were not in place.

C The surveillance test procedure does not contain instructions for equalizing pressure across a closed MSIV to allow reopening the valve which results in testing the upstream and downstream valves under different conditions.

4OA3 Event Followup

Unit 1 Downstream MSIV Failures

.1 Inspection Charter Objectives 1 and 5

a. Inspection Scope

The inspectors reviewed in detail CR 2006103043, documentation of the troubleshooting efforts, and operator logs covering relevant time periods. Cognizant licensee personnel were interviewed and the NRC Resident Inspectors, who had observed the sequence of events and the licensees initial response, were consulted. Additionally, debriefs were held with the licensee to aid in the understanding of key events leading up to the MSIV failures. Relevant risk information was also collected and provided to the NRC senior risk analyst to support an evaluation of the risk significance associated with the MSIV failures.

b. Observations MSIV closure which did not facilitate a methodical approach to understanding the MSIV failures. The inspectors concluded this resulted in a more challenging root cause determination process. For example, thermal expansion appeared to be a contributor as 3370A and 3370B were found in a more closed position on April 8, when the mechanics arrived to close the MSIVs. In hindsight, a quarantine of the failed MSIVs for careful disassembly, inspection, and review of valve ambient conditions could have provided additional opportunities to further develop the root cause of the MSIV failures. The inspectors also noted that a thorough evaluation of the MSIV issues had not commenced until April 21, when licensee management identified the CR describing the event had been assigned to the Maintenance department and dispositioned as work orders (WO). The CR was then reassigned to Engineering Support.

The root cause team proceeded with troubleshooting efforts and formulated the data into table format to assist in failure mechanism classification and corrective action completion. Table 1 identifies the MSIV failure mechanisms attributed to the Unit 1 MSIV failures. Based on the results of the root cause, the downstream MSIVs were reassembled on May 15 with numerous new components and cold stroked successfully in the required time. The downstream MSIVs were later hot-torqued and fully stroked at normal operating conditions during plant startup on May 23 with successful results.

Table 1: MSIV Failure Mechanisms Mechanism Valve(s) Action Bearing backwards B Procedure guidance Keys wrong size A, B, C New keys and controls Stress riser in keyways A, B, C New shafts with radius keyways Tight key on B shaft B Key/keyway control, new keys Packing rubs on shaft A, B, C New packing configuration to reduce friction; partial stroke testing Severe rubs from yoke A, B, C New bushings all valves bushing End plate clearance A, B, C (and Clearances opened per vendor upstream recommendations, opened on Unit valves) 2 Lubrication on packing A, B, C Revised procedure and briefed maintenance crew Gouges from foreign A Replaced shaft, revised procedure material entry into valve and briefed Rubs from spacers and A, B, C Replaced all spacers and bushings bushings Severe Service A, B, C Engineering review for potential future Conditions identified for modifications downstream valves The data in Table 1 demonstrates the complexity of the downstream MSIV failures and the corrective actions to prevent recurrence. The licensee provided timeline information which was used to develop Attachment 2 for a detailed sequence of events concerning discovery and correction of CR 2006103043. This information, along with licensee MSIV failure data provided by the licensee on June 19, 2006, was provided to the Regional Senior Reactor Analyst to assist in the completion of the significance determination process.

Based on observations of the post-failure component inspections, the inspectors concluded that the licensee collected detailed information to develop effective future corrective actions. Initial action to preserve valve failure information could have been improved.

.2 Inspection Charter Objectives 2 and 3

a. Inspection Scope

The inspectors performed reviews of selected components preventive-maintenance procedures, vendor documents, completed work orders, and CRs generated for non-conformances to verify that the MSIV maintenance was based on vendor recommendations and appropriate industry operating experience. During these reviews, the inspectors focused on potential common mode failure vulnerabilities that could be introduced by maintenance activities. The team reviewed procedures used to assemble and disassemble the MSIVs as well as procedures used to verify MSIV operability during normal and test conditions. A keyword search utilized industry operating experience records to look for cases similar to this event. The inspectors interviewed responsible engineers, supervisors, and other cognizant personnel. The inspectors reviewed maintenance training procedures pertaining to MSIV maintenance to verify that training was consistent with the procedures. Previous corrective actions taken were also evaluated.

b. Observations The licensee has a history of operational and maintenance issues regarding the downstream MSIVs. As early as 1980, the effects of a postulated severe environment in which the downstream MSIVs operate, appears to have resulted in increased internal component failures and repairs. The environment for the downstream MSIVs was believed to differ from that of the upstream MSIVs. The downstream MSIVs experience more turbulence because of their proximity to the upstream MSIV. The MSIVs were installed with six inches of straight piping between the upstream and downstream MSIV bodies and this arrangement does not allow steam flow to settle prior to entering the downstream MSIV. This added turbulence appears to have resulted in several failures primarily as disc-to-disc arm fastener failures, shaft cracking, shaft bending, and shaft binding precluding MSIV closure. Table 2 has been included as a brief history of similar mechanical issues concerning the downstream MSIVs. In contrast, the inspectors identified very few mechanical problems related ot the upstream MSIVs.

Table 2: Similar MSIV Failures at FNP Unit/Valve Failure Symptoms Cause As Stated In CR Date 2/ 3369B&C 09/83 Would not move Excessive Packing Friction.

from full open.

2/ 3370A 04/86 Would not move Excessive friction from shaft seal from full open.

O rings.

2/ 3370A 06/86 Would not move Packing Friction and Weak Test to test position.

Actuator.

2/ 3370A 07/86 Would not move Packing Friction and Weak Test from full open.

Actuator.

1/ 3370B 01/87 Closed 60% and Packing Friction and Zero Steam then bound.

Flow.

1/ 3370C 12/92 Would not move Improper Assembly of Valve and from full open.

Main Actuator.

2/ 3370A 10/95 Would not move Excessive Packing Friction.

from full open.

1/ 3370B 06/97 Would not go Packing and Yoke Bearing completely closed.

Friction.

2/ 3370A 10/99 Closed 80% and Worn Thrust Bearing and Hard then bound.

Packing. ( Listed as Not Uncommon)2/ 3370A 11/00 Would not move Friction From Dirty Main Actuator to test position.

Stem.

2/ 3370A 05/01 Would not fully Friction due to Misalignment of stroke.

Indicator Plate.

1/ 3370B&C 05/02 Would not move Valve Friction and Variances in to test position Air Actuator Pressure.

1/ 3369C 05/03 Would not go Improper Assembly of Valve and completely closed.

Main Actuator.

The inspectors noted that several downstream MSIV failures were attributed to packing friction. The licensee has modified the packing program several times to incorporate improved industry packing arrangements and methods. However, two key areas identified during the current MSIV root cause reviews were 1) the licensee historically was implementing generic MSIV and packing practices including lubricating the shaft and packing during MSIV assembly. The packing vendor, ARGO, identified that this practice would actually increase friction between the ARGO packing and the MSIV shaft; 2) the licensee did not coordinate between ARGO and the MSIV vendor, WEIR, such that packing specifications for this type of MSIV were not communicated and implemented. Since this recent event, the licensee has worked with the packing and MSIV vendors to develop a packing arrangement best suited for the MSIVs.

The root cause team also identified that the MSIVs have not been in a testing program that would identify failure types observed during this incident. The initial testing program to ensure operability was a quarterly test (partial) stroke of all MSIVs. This test verified that the MSIVs would break away from the open position when the main actuator force was removed. This test stroke was discontinued in 2002 to minimize the potential of a spurious reactor trip from accidental MSIV closure. Historically, the only other testing of the MSIVs was a full stroke test, in accordance with the Inservice Plan for Pump and Valve Testing, performed as each unit was starting up from an outage. The current failures were identified during the licensees first Unit 1 implementation of an as-found full stroke test performed at NOT/NOP with no steam flow as the unit was being shut down. This test had been suggested by site personnel to potentially reveal degraded as-found conditions of the MSIVs. This test was first performed on Unit 2 during unit shutdown for the October 2005 refueling outage. No Unit 2 MSIV failures were identified.

The root cause team also identified that maintenance procedure FNP-0-MP-39.0, Main Steam Isolation Valve Disassembly and Reassembly, had deficiencies. With respect to MSIV disassembly, there was no acceptance criteria for inspecting MSIV components as the MSIV was disassembled. This did not provide an avenue for data transmittal from mechanic to engineer and, therefore, did not facilitate a detailed evaluation of critical dimensions within the MSIV. Aged and worn components no longer maintaining their dimensional tolerances were identified to have been a portion of the frictional sum of forces which prevented the MSIVs from closing. Additionally, the reassembly instructions lacked sufficient detailed to ensure that the MSIV components were installed in the correct arrangement with proper tolerances. This had a direct effect on the current failure of MSIV 3370B, as the thrust bearing was installed backwards and was identified as the leading contributor to significant shaft cracking and the inability of the MSIV to close on demand. At the end of the inspection period, FNP-0-MP-39.0 was being modified to include specific assembly, disassembly, and inspection criteria and will focus on mechanical tolerances, cleanliness, and guidance when unexpected adverse conditions are encountered.

.3 Inspection Charter Objective 3

a. Inspection Scope

The inspectors reviewed the history of MSIV failures which exhibited similar symptoms to the failures on April 8. The review included pertinent corrective action program documents, maintenance work requests, and interviews with cognizant engineers.

b. Observations Review of the maintenance history for the MSIVs identified several performance problems with the downstream MSIVs. While the licensee did not identify a single predominate root cause for the failures on April 8, it appears that many factors have resulted in the repetitive MSIV issues. Specifically, the inspector identified that CRs 2000005687, 2001001155, and 2002001026 had overall inadequate problem identification and resolution. The licensees current Root Cause Team also identified weaknesses in the resolution of previous CRs. For example, the root cause for CR 2006103043 identified that Investigations prior to 2002 typically stopped at the first plausible or most visible possible cause. In 2002, an investigation into a partial stroke failure may have missed the significance of sticking on the open stop and possibly misdiagnosed the cause of the failure. The licensees current root cause evaluation team also noted that, due to extensive programmatic changes in the FNP Corrective Action Program since 2002, no programmatic corrective action was recommended for the resolution of issues prior to 2002.

Based on historical failures and the licensees initial root cause conclusions, the inspectors determined the licensee had experienced three MSIV failures since 2000 which exhibited symptoms similar to the failures in April 2006. The inspectors identified a number of missed opportunities the which licensee had to properly identify and correct the failure mechanisms which led to the most recent failures. Table 3 summarizes the similar MSIV failures experienced between 1999 and 2006.

Table 3: Recent Similar MSIV Failures at FNP 2/ 3370A 11/00 Would not move Friction From Dirty Main Actuator Stem.

to test position.

2/ 3370A 05/01 Would not fully Friction due to Misalignment of Indicator stroke.

Plate.

1/ 3370B&C 05/02 Would not move Valve Friction and Variances in Air to test position Actuator Pressure.

This failure data illustrates three notable examples of missed opportunities for the licensee to have identified and corrected the actual failure mechanism. The specifics of these examples are discussed below.

! On November 3, 2000, CR 2000005687 was written for Unit 2 due to a failure of MSIV 3370A to stroke during maintenance test FNP-1/2-STP-21.1. Initial indications of the MSIV failure were that the test actuator would not move the MSIV from the OPEN position to the TEST position and the main actuator was unable to return the MSIV to OPEN. Work Order (WO) 20009003 was written and the licensee suspected binding friction between the MSIV yoke bushing and shaft. The WO indicated that lubrication was applied between these two surfaces and the test stroke was re-performed. An evaluation performed on January 5, 2001, stated that the main actuator was stuck due to the accumulation of dirt on the actuator stem and the cleaning and lubricating of the stem removed the additional friction and allowed the MSIV to stroke. A second evaluation performed on January 19, 2001, supported the earlier evaluation by stating that the dirt on the stem at the main actuator bushings was the root cause. In the CR, the licensee stated that the main actuator could overcome the opposing forces of the MSIV weight, spring tension, and the added friction at the actuator bushing. It was also stated the MSIV would still perform its design function as the main actuator would be released and the MSIV weight and spring tension would be sufficient to overcome the friction from the dirty stem. These statements are inconsistent with the indications present at the time of the failure and no documentation could be identified that would indicate that a formal root cause was performed. While it is not possible to recreate the situations which existed at the time of the MSIV failure, the information identified in the root cause of CR 2006103043 indicated that there was sufficient margin to both open (main actuator) and close (disc weight and spring tension) the MSIV even in the presence of dirty actuator stems. The inspectors concluded that the November 2000 CR was a key opportunity to investigate and resolve the potential problem.

! On May 8, 2001, CR 2001001155 was written for Unit 2 due to a failure of MSIV 3370A to fully stroke during full stroke surveillance FNP-2-STP-45.7. The Maintenance department was requested to investigate and the licensee concluded that the MSIV indicator plate was providing additional friction due to thermal expansion and preventing the MSIV to fully close. This failure was listed as a maintenance rework item as the MSIV successfully closed during cold conditions.

WO 1003978 stated that the indicator plate was adjusted, packing was adjusted, and lubrication was applied to the yoke bushing and main actuator assemblies.

Subsequently, the test was performed satisfactorily and the plant continued with startup activities. CR 20001001155 was subsequently closed due to actions taken.

Discoveries from the current root cause of CR 2006103043 indicated that the friction available from the surface area that the MSIV shaft would come in contact with the MSIV indicator plate, would not be individually significant enough to prevent MSIV closure. Based on the above, the inspector concluded the original evaluation for CR 2001001155 was inadequate, in that, an unlikely root cause was determined and further investigations were not made. During the licensees current evaluation the indicator plate expansion theory was not included in the Table 1 corrective actions.

! On May 4, 2002, CR 2002001026 was written for Unit 1 due to the failure of MSIVs 3370B and 3370C to stroke during maintenance test FNP-1/2-STP-21.1. WOs 2002625 and 2002626 were implemented and, contrary to previously identified guidance, maintenance personnel lubricated MSIV stems using WD-40, which resulted in successful MSIV strokes. In 2000, the licensee had identified that WD-40 provides no lubrication per CR 2000005198 due to the evaporation of WD-40 at the temperatures of the stem in operation. Based upon the associated Action Item (AI)2000201180, the lube PM for the MSIV's that utilized WD-40 was deleted. It was also understood at that time, that any other lubricant would act as a dust and debris collector and would counteract the intended purpose of lubricating the stem. An apparent cause was performed for CR 2002001026 and it discounted the frictional forces existing from main actuator stem dust/debris as a credible cause preventing MSIV operation. CR 2002001026 did state that the apparent causes could be numerous and consist of actuator air pressure consistencies, friction, or lateral misalignments. However, the CR stated that most likely the cause of the event was the performance of the partial stroke itself. The licensee was concerned that the test was causing unnecessary stress and wear on MSIV internal components resulting in MSIV stroke failures.

Later in 2002, the conclusion of the apparent cause of CR 2002001026 appeared to shift focus from problem resolution and became supporting information needed to stop performing the FNP-1/2-STP-21.1 MSIV test procedure. The licensee incorporated guidance that the NRC recommends plants not perform a partial stroke test at power. Section 4.2.4 (Section on Supplemental Guidance on Inservice Testing of Valves, Power Operated Valves, Main Steam Isolation Valves) of NUREG-1482, published April 1995, stated NOTE: Related to MSIVs, a number of plants perform a partial-stroke exercise quarterly during power operations. The revised standard technical specifications bases for MSIV surveillance requirements stated that "MSIVs should not be tested at power, since even a part-stroke exercise increases the risk of a MSIV closure when the unit is generating power." Based on this information, MSIV test procedure FNP-1/2-STP-21.1 was subsequently suspended from use and resulted in the MSIVs only being stroked when FNP-1-STP-45.7 was required. The current Root Cause Team of CR 2006103043 stated, in part, that the MSIV test stroke procedure could have helped mitigate the drag increase from sticking packing which develops over time. The inspectors considered that the removal of this partial test stroke procedure may have had a detrimental effect on the reduction of cumulative friction forces which accumulate over time. The licensee has since temporarily reinstated this partial stroke on both operating units MSIVs at a quarterly periodicity to gather data and aid in determining long term operability throughout the current operating cycles.

By the end of the inspection, the licensees final root cause evaluation for CR 2006103043 was completed. Based on the root cause, the inspectors concluded that the licensees corrective actions focused on short term mechanical refurbishment of the failed MSIVs and establishment of baseline data for development of long term corrective actions. An accurate root cause determination for the failed downstream MSIVs is critical not only to fully evaluate the corrective actions for the failures but also to more accurately define potential common mode failure mechanisms for the upstream MSIVs. This can also greatly affect the risk determination for these failures. Therefore, pending final review of the root cause, contributing factors, long term enhancements for these failures, and review of the most recent June 28th MSIV failure, this issue will be identified as Unresolved Item (URI)05000348/2006009-001, Repetitive MSIV Closure Failures.

.4 Inspection Charter Objective 4

a. Inspection Scope

The inspectors reviewed the operating procedures and training provided by the licensee and also discussed the scenario with plant operators. The inspectors also evaluated whether the licensees use of night orders was sufficient to inform operators about abnormal MSIV behavior during post-accident conditions. Particular attention was focused on Unit 2 operations during the initial phases of the Unit 1 root cause evaluation.

b. Observations The inspectors determined that the licensed operators overall procedural knowledge of abnormal MSIV operations was sufficient. Key operators were aware of the history the FNP has had with MSIV problems and the actions required by them if the MSIVs did not operate properly. The training department had been responsive to past MSIV failures and had implemented four separate simulator scenarios in 2003 and 2005 requiring the operators to compensate for inoperable MSIVs. Following the current MSIV failures, the licensee issued a night order on April 22, which identified the MSIV issues on Unit 1 and provided guidance to operators should such an event occur on the operating unit. The night order referenced the proper emergency operating procedures, as well as management expectations, and was understood by the control room operators.

.5 Inspection Charter Objective 6

a. Inspection Scope

The inspectors reviewed the operability determination (OD) provided by the licensee to determine if the conditions which caused the Unit 1 MSIV failure were also present on Unit 2.

b. Observations The inspectors reviewed the OD to confirm that the licensee successfully performed the as found full stroke test prior to entering the latest Unit 2 refueling outage. During Unit 2 restart from the refueling outage, the licensee successfully performed full strokes of the MSIVs under Hot Standby conditions. When it was identified that the Unit 1 MSIVs had possible common mode failure mechanisms, based on the current failures, the licensee added Appendix 4 to procedure FNP-1-SOP-17.0, Main and Reheat Steam, to test for partial movement of the Unit 2 MSIVs. On April 24, the licensee performed procedure FNP-1-SOP-17.0, Appendix 4, to test (partial) stroke all the Unit 2 MSIVs.

The test stroke was observed by NRC inspectors. The test strokes were successful on all six Unit 2 MSIVs and an operability determination was completed based on these results and data collected from the previous Unit 2 refueling outage which ended in November 2005. The licensee concluded that the tests supported operability of the Unit 2 MSIVs based on the known conditions and limited root cause evaluations to date.

The licensee planned to perform quarterly test strokes of both units MSIVs until the licensee gains confidence in the corrective actions taken to date and further evaluation of long-term corrective actions to prevent recurrence. The licensee concluded that the Unit 2 MSIVs would remain operable until the next opportunity to replace key components identified for replacement by the root cause team. This information was supported by the most current inspection of Unit 2 MSIVs, the fact that Unit 2 components are less aged that Unit 1 components, and the incorporation of the quarterly partial test stroke.

.6 Inspection Charter Objective 7

a. Inspection Scope

The inspectors evaluated this event for potential industry-wide generic implications.

b. Observations One key attribute of the MSIV failure mechanism is that the two MSIVs are in series, oriented in the same direction, and in close proximity. This configuration results in turbulent flow through the downstream MSIVs. The turbulent flow induced severe vibration which may have resulted in the observed failure mechanisms. Therefore, the severity of service conditions of the downstream MSIVs is a condition which may be of generic industry-wide interest.

4OA6 Meetings, Including Exit

On May 16, 2006, the inspectors presented the inspection results to Mr. Randy Johnson and the other members of his staff who acknowledged the unresolved issues. The inspector confirmed that proprietary information was not provided or examined during the inspection.

Following the exit meeting, on June 28 during partial stroke testing of the Unit 1 MSIVs, MSIV 3370B became stuck in the test position. The licensee locally verified the disk was at the 5% test position and entered a 72-hour LCO for an inoperable MSIV. The licensee attempted various methods including lubrication, packing adjustment, and mechanical assistance to free the disk, but there was no change in position. On June 30, the licensee decided to shutdown Unit 1 to perform troubleshooting and inspection of the MSIVs.

ATTACHMENTS: 1.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

W. L. Bargeron, Assistant General Manager - Operations

W. R. Bayne, Performance Analysis Supervisor

S. H. Chestnut, Engineering Support Manager

P. Harlos, Health Physics Manager

L. Hogg, Security Manager

J. Horn, Training and Emergency Preparedness Manager

J. R. Johnson, Plant General Manager

T. Livingston, Chemistry Manager

B. L. Moore, Maintenance Manager

W. D. Oldfield, Quality Assurance Supervisor

J. Swartzwelder, Work Control Superintendent

R. J. Vanderbye, Emergency Preparedness Coordinator

R. Wells, Operations Manager

T. L. Youngblood, Assistant General Manager - Plant Support

NRC personnel

C. Patterson, Senior Resident Inspector

W. Rogers, Senior Reactor Analyst

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000348/2006009-01 URI Repetitive MSIV Closure Failures (Section 4OA3.3)