ML20045E743

From kanterella
Jump to navigation Jump to search
Insp Rept 50-458/93-18 on Stated Date.Five Apparent Violations Being Considered for Escalated Enforcement Action.Major Areas Inspected:Circumstances & Events Leading Up to Failure of MSIV 1B21*AOVF022B to Close
ML20045E743
Person / Time
Site: River Bend Entergy icon.png
Issue date: 06/29/1993
From: Gagliardo J
NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20045E741 List:
References
50-458-93-18, NUDOCS 9307060007
Download: ML20045E743 (32)
v • d • e


See also: IR 05000458/1993018

Text

{{#Wiki_filter:w APPENDIX .U.S. NUCLEAR REGULATORY COMMISSION- REGION IV j . Inspection Report: 50-458/93-18 Operating License: NPF-47 Licensee: Gulf States Utilities P.O. Box 220 St. Francisville, Louisiana 70775-0220 Facility Name: River Bend Station (RBS) l Inspection At: St. Francisville, Louisiana Inspection Conducted: April 22 through June 22, 1993 Inspectors: I; P. Loveless,-Resident Inspector W. i.. McNeill, Reactor Inspector

l

' Contributing Inspector: W. F. Smith, Senior Resident Inspector Approved: ' dl Q 3 o J N . Gagliardo, Chief, Pro ct Section C te ~ Inspection Summary Areas Inspected: Special, reactive, unannounced inspection-of the ' circumstances and events leading up to the. failure of 11ain Steam Isolation ~~ Valve (MSIV) IB21*A0VF022B to close. Specific reviews included the design of the valve, the f ailure mechanism, the previous repairs made to the valve, tha root cause of the failure, surveillance tests performed and the results, the' corrective actions taken by the licensee, and the safety significance'. Additional areas inspected included, generic implications as applied to the other MSIVs, local leak rate testing history of the MSIVs, and problems identified with excessive guide rib wear. Results: The inspectors concluded that MSIV IB21*A0VF0228 was stuck in the open

position and was, therefore, inoperable (Section 2.2). The cause of the failure of MSIV 1821*A0VF022B was determined by the

licensee to be insufficient clearance between the poppet and the guide ribs, combined with excessive guide rib wear. This cause was supported by the data, mapping, and measurements taken (Section 2.2.2). 9307060007 930629 -PDR ADOCK 05000458 PDR a - _ - _ _ _ _ - - _ - _ - _ - _ -

- . . . - - .-- -,. . '4' f -2- 'Following the identification of the failure, during Forced Outage 93-01,- e the licensee's corrective actions to address'the failure of. MSIV IB21*A0VF022B to close were excellent (Section 2.2.3). One apparent violation was identified for the failure of the operators- e to properly document and resolve problems identified with MSIV IB21*A0VF022B when it failed to meet-the acceptance criteria of- Procedure STP-051-0201 on February 27 and, again, on April 1, 1993' (Section 2.3). The licensee's determination of the cause for the failure of the e surveillance program to identify the problem with MSIV IB21*A0VF022B was 1 reasonable and was supported by the facts of the investigation- (Section 2.3.1). The failure of the surveillance program to identify that

MSIV IB21*A0VF022B had failed in the open position was considered safety significant (Section 2.4). On February 27 and again on April 1,1993, the licensee should have e known that MSIV IB21*A0VF022B was inoperable when the acceptance criteria of Procedure STP-051-0201 could not be met (Section 2.3). One apparent violation was identified for failure to meet the e requirements of Technical Specification 3.4.7 when MSIV IB21*A0VF0228 became inoperable (Section 2.3). The licensee determined that, if the outboard MSlV had also failed e during this time, a main steam line break accident scenario probably would have exceeded' design basis offsite dose limits '(Section 2.2.4).. . Although Procedure STP-051-0201 did. indicate the failure of MSIV

IB21*A0VF022B, the procedure would not, under all circumstances, have detected an immovable poppet (Section 2.3). ' One apparent violation was identified for the failure of

Procedure STP-051-0201 and the surveillance program to provide assurance that a failed MSIV would not go undetected as required by Quality Assurance Directive, QAD-ll, " Test Control" (Section 2.3). One apparent violation was identified for the failure of machinists or- , e quality control inspectors to verify the as-left dimensions for guide > rib bore as specified in Procedure 1821*A0VF022B following rework during- Refueling Outage 4 (Section 2.2.1). Two examples of an apparent violation pertained to the-failure of

mechanics to follow procedural controls during the rework performed on the MSIVs during Forced Outage 93-01 (Sections 2.2 and 3.2.2). .

. . , . , ... , -3- The licensee committed to continue monthly partial stroke testing and

perform quarterly full slow stroke testing while at power to. ensure that- - corrective actions have been-adequate. This testing will continue.until Refueling Outage 5 (Section 2.3.2). e The licensee's evaluation of the other seven MSIVs and conclusion that

the valves were not susceptible to failure was of high quality and was , well supported by inspection findings and the maintenance history ' (Section 3.1). - , The licensee's review and investigation to determine the generic aspects e of the failure of MSIV IB21*A0VF022B were excellent (Section 3.1). "

The offsite dose calculations for worst case MSIV leak rates were determined by the licensee to be bounded by the existing design basis analysis and/or regulatory limits (Section 3.2.3). Summary of Inspection Findings: Apparent Violation 458/93018-1 was opened (Sections 2.2 and 3.2.2). e

Apparent Violation 458/93018-2 was opened (Section 2.2.1). ? e Apparent Violation 458/93018-3 was opened (Section 2.3).

Apparent Violation 458/93018-4 was opened (Section 2.3).

Apparent Violation 458/93018-5 was opened (Section 2 3). Attachments: , Attachment 1 - Persons Contacted and Exit Meeting ' Attachment 2 - Main Steam Isolation Valve Drawings and List

I . -- . . . . .

. -. . - . . - - . . . . ~. -4- . ., DETAILS , 1 PLANT STATUS Throughout most of this inspection period,.the plant remained in a cold- , shutdown condition in Forced Outage 93-01. j On December 6, 1992, the reactor was shut down and taken to cold shutdown for Forced Outage 92-04. The reactor was restarted on December 14. The reactor. continued to run in Operational Condition 1 until it was shut down on April 17, 1993, for Forced Outage 93-01. At the end of this inspection period, the reactor was still shutdown because of continued leakage of the seal on reactor recirculating Pump B. 2 REVIEW 0F THE OPERABILITY OF MSIV 1821*A0VF022B (93702) 2.1 System Overview Each of the four main steam lines at RBS contains two MSIVs; one inboard valve

located inside the Drywell and one outboard valve located just outside the primary containment. - Each MSIV is a 24-inch diameter pneumatic-opening,. spring and pneumatic closing, internally-balanced poppet-type globe valve. The valves were. manufactured.by the Atwood and Morrill Co. Inc- Flow through each MSIV.is . rated at 3.11.x mlb/hr. Each is designed to close and remain closed for - 1 hour following an accident condition environment of 340of and~ design containment pressure. Each MSIV is designed to have a closurestime of greater than 3 but less than 5 seconds. The minimum closing time of 3 seconds is to prevent valve damage and to lessen the pressure-transient ~to.the reactor , pressure vessel caused by MSIV closure during operation. The maximum closure l time of 5 seconds is to limit the loss of coolant caused by a steam line ' . rupture and to provide for adequate containment of fission products in the event of a fuel element failure. . A detailed set of drawings of the valve is provided in Attachment 2 to this

report. , The valve body is welded with a 45 degree angle between the steam path and the ' centerline of poppet travel. This angle allows steam passages to be streamlined, which minimizes the pressure drop across the valve during normal ' operation. The inlet port on the poppet admits steam above the seat so that flow tends to shut the valve and the higher inlet pressure will tend to hold _ it shut. The bonnet is bolted to the body and forms part of the pressure boundary. The bonnet is penetrated by the valve stem through a stuffing box, which is sealed by two sets of replaceable stem packing with a lantern ring between each set. -. . . .

_ _ _ . _ _ _ - . . _ _ = . _ _ . . _

,,

, , . 1 N -5- ! - The lantern ring maintains physical separation between Jthe leak off: connection and the packing sets. Any steam or condensation at the leak off connection is ~ ' routed to the drywell equipment drains for the inboard valves and the. = containment equipment- drains for the outboard valves. j The main poppet is supported by three guide ribs located.120 degrees apart in

!

~ the lower' valve body. The lower guide rib is' located in the flow stream of the valve and connects to the lowest portion of the main seat.- The upper two . < 9uide ribs are integral parts of the valve body casting. All'three~ guide ribs are overl' aid with stellite on the face surfaces to minimize' wear from the poppet movement. A common problem with this valve design in the industry h'as - - been that the poppet vibrated and rotated in.the steam flow path,~ causing a unique. lower guide rib wear pattern'. '

- The base of the stem serves as an internal pilot valve, seating 'against' the - equalizing port at the bottom of the main disc (poppet). Of the 11-inch valve

stem travel, the first 10 percent (approximately 1 inch) opens the equalizing port so that the differential pressure is minimized across the poppet. The balancing orifice transmits the steam pressure through the equalizing port to" , the downstream side of the poppet. As it lifts, the poppet retracts into the .! upper portion of the valve body just below the_ bonnet. :This sufficiently _ removes the poppet from the steam flow path so that there is negligible steam . l ficw pressure drop across the valve, except that' caused by the valve body. It: 1 -is noted.that the pilot valve remains open throughout poppet travel,-including ~ j when the poppet is at rest in the fully retracted position. 1The valve has ., little effect on flow restriction during the first 75 percent of the closing stroke because of the venturi. shape of the valve body. Stem packing ~ leakage. i 1 is prevented by backseating a notch on the stem into the'bonnetfwhen the valve- . is fully _ opened. > During valve stem movement, the spring seat member activates the valve .' position limit switches at the 10 percent, 90 percent, and 92 percent open 1 - ! - positions. _ The 10 percent open' contacts were designed to send'" valve shut" signals to the MSIV positive leakage control system, and a " valve open" signal 1 ' to the red position indicating lights and the process computer' system. The 90 percent open contacts send " valve closed" signals to ' position indicating lights,' providing a green indication in.the main control room, and the - - 92 percent open limit switch contacts provide." valve open" signals to the reactor protection system. The valve bonnet supports the air operating cylinder by four actuator support' and. spring guide shafts. Helical springs around the guide shafts assist in ^ closing the valve during normal operation and can close the. valve during abnormal operation such as loss of closing air pressure. A hydraulic dash pot is provided.at the top of the valve stem to adjust the time required for full-stroke travel. i r i l - , - .. ,

. . . _. . t -6- 2.2 Failure of MSIV IB21*A0VF0228 On April 19, 1993, while performing Maintenance Work Order-(MWO) R174734 to repair a presumed faulty limit switch, the licensee determined that MSIV IB21*A0VF022B would not stroke closed past the 90 percent open limit switch. Initial response was to stroke the valve' approximately 10 times. During this process the stem would only travel approximately 1 inch. As a troubleshooting method, the licensee isolated the air supply to the actuator and bled down the accumulator pressure. The manifold containing the MSIV s solenoid operated valve and shuttle valves was removed. These actions should have relieved any air trapped on the bottom of the actuator and allowed the springs to close the valve. The valve remained open. A test manifold was then installed on the actuator, and the valve was exercised locally. The valve still would only stroke approximately 1 inch. The valve was finally uncoupled from the actuator. However, it remained in the open position. Based on the failure of these techniques to close the valve, the licensee concluded that mechanical binding was the probable cause of the valve's. failure to close. It was noted at the time that the valve did not show double indication (red and green lights) which would indicate that the stem was stroking less than 10 percent of full travel. On April. 22, 1993, the Plant Manager established an MSIV task force to determine the root cause of this failure and the required corrective actions. He requested that a full inquiry be made into the circumstances that led to this event. The task force consisted of representative from Design Engineering, System Engineering, Plant Maintenance, Licensing Oversight, Outage Management, Operations, and Projects. The team would be assisted by. General Electric and Atwood & Morrill representatives. On April 23 and 24, under MWO R164767, mechanics removed.the-valve operator and the bonnet. The operators found the poppet to be cocked and stuck in the fully open position. Outer depth measurements indicated that'one side of the poppet was lower in the upper bore by approximately .023 inches. Force was applied to the poppet, causing it to break free, and the poppet was - removed from the valve with little trouble. The initial visual inspection identified scoring on the poppet and grooves on the bottom guide rib. Mechanics stated that the wear indicated possible rotation of the poppet during power operation. The job plan for the disassembly called for the use of Corrective Maintenance Procedure (CMP) CMP-9141, " Main Steam Isolation Valves IB21*A0VF022A, B, C, D, and IB21*A0VF028A, B, C, D Disassembly, Inspection Rework and Reassembly," Revision 48. The inspectors noted that the procedure was written assuming that the valve was fully closed during the disassembly. As discussed, MSIV IB21*A0VF0228 was stuck in the fully open position. Based on this

_. , __ _ . . . _ . . _ _ _- _ _ ._ _> ,, . _. . . .- -7- , difference,'the mechanics were required to perform several actions not- , provided in the procedure. The lack of physical clearance in the. Drywell required-the mechanics to take the actuator ~and spring seat' assemblies apart. ' during disassembly vice the removal of the entire actuator in accordance with the procedure. Makeshift braces were installed and strapped to the yoke rod between .the valve cover and the bottom spring assembly to pr. event the relaxing of the actuator springs during disassembly. The. inspectors noted-that the hydraulic needle valves had to be removed in order to: disassemble the

actuator. This action was not identified in the procedure. The mechanical foreman tried to utilize a pair of channel locks to remove the smaller needle ' valve, without fully understanding how the valve was attached. This action - cat.;ed damage to the needle valve, which had to be replaced upon reassembly. In' addition to the procedure problems, 'the NWO Alara Plan A called' for the placing of a plastic catch containment under the valve to catch any dropped components. Not only was the catch not installed, but-the inspector did , observe a dropped component which required radiation protection support to retrieve. The inspectors noted that the workers and foreman appeared to understand that there were additional steps required that were not addressed by the job plan or the standard' corrective maintenance procedure. However, the problems were , ' not addressed prior to the-work being performed. The failure to follow the . job plan / procedure:or to correct the guidance is considered the first example ! of an apparent violation involving procedural compliance'(458/93018-1). y This example is considered a repeat of previous violations for, workers failing to obtain an upgrade of inadequate procedures prior to performing safety- ~ related work.(50-458/9305-02 and 50-458/9226-02). During disassembly of the valve, it'was found that the lower guide. rib was grooved to a maximum depth of .070 f..ches. This grooving appeared to be a direct result of poppet rotation and vibration impacting on the lower guide rib. The licensee had seen similar grooving of. lesser magnitudes in all MSIVs 2 ' in the past. The corrective maintenance has been to weld overlay at least the- lower guide rib and sometimes all three guide ribs, then 'remachine the guide- ribs to the required dimensions. In the past, dimensional analysis had shown ' as-found wear areas in the lower guide rib from .014 to .036 inches in depth after one cycle. ] Dimensional analysis of the stem established there was not'a problem with the- , adjustment of stem length. Excessive stem length would allow-the disk to sit y -further into the steam flow and, therefore, be subject to greater vibration ' and rotation. The tolerance used for guide rib machining in the past had'been for a bore having an inside diameter of 19.010 to 19.000 inches, according to vendor drawings. The valve disk had two " Stellite 21" bands that were, according to- vendor drawings, to be 18.975 to 18.970 inches in diameter. Hence, the gap ' between the valve disk and the guide ribs would be .025 to .040 inches. The vendor site representatives had previously suggested working within that

> - . - - -. - , - ~ . - .-:..-- .-v.-- --. - .,. -

._ _ . . .. _ __ _. . . -8- , tolerance by restricting it to .025 to .030 inches. A computerized coordinate mapping machine found the dimensions in Valve MSIV IB21*M VF022B to be 18.988 for the guide rib inside diameter and a caliber measurement of 18.978 for the- , poppet outside diameter. The go-gauge used in the past for verifying

sufficient guide rib clearance measured 18.974 for its outside diameter. Although the go-gauge would have fit in the lower bore with the as-found dimensions, it is apparent that the proper tolerances were not verified following the previous maintenance on the valve in Refueling Outage 4. 2.2.1 Previous Repairs to the MSIV During Refueling Outage 4, MSIV 1821*A0VF0228 and two other valves, 1821*A0VF022A and IB21*A0VF022C, failed the Technical Specification required local leak rate testing. Valve 1821*A0VF022A had its body seat ground and

its pilot seat lapped in accordance with MWO R152812. MSIVs IB21*A0VF022B and . 1821*A0VF022C also had the stellite removed and a weld overlay applied to , their lower guide ribs. The guide ribs were then machined in accordance with , MW0s R152815 and R152813, respectively. RBS personnel performed the repair of MSIV 1821*A0VF0228. Welding Services, Incorporated (a licensee contractor) performed the repair of MSIV IB21*A0VF022C. This same contractor provided i computerized coordinate measuring services (mapping of the guide ribs on , MSIV IB21*A0VF022C) at that time. Because of outage time constraints, this method was. not utilized on MSIV 1821*A0VF022B during Refueling Outage 4. On , August 31, 1992, MSIV IB21*A0VF0228 was successfully full stroke tested aceneding to Procedure STP-109-6302, "MSIV Full Stroke Operability Test," and' , declared operable. The method of measurement used on MSIV 1821*A0VF022B during Refueling ! Outage 4, and on all valves before the computerized coordinate measuring machine was available, required the use of a straight-edge and depth micrometers to determine guide rib groove depth. The outside diameter of the r valve disk was measured with caliper type micrometers. The inside bore diameter was purportedly calculated by measuring the three radii to the guide rails and averaging them to obtain a diameter. The radius measurement was performed with stick micrometers measuring f am the centering t'ar of the Climax Brand machining tool to the guide rit. In addition to measurement, the inside bore diameter was checked with a plug type go-gauge. - i An alternate measuring technique had also been utilized in the past. A section of the lower guide rib which had been assumed erroneously to have not been repaired or machined during the rework was used as a reference point to determine a final machining radius of the rib bore. This assumed that the as- found dimensions in this area were correct and could lead the machinist to conclude erroneously that the rib bore had been restored to acceptable dimensions. The machinist responsible for past measurements stated that he could not recall measuring the MSIV IB21*A0VF022B inside bore diameter during Refueling Outage 4. The licensee concluded that the methods utilized were insufficient

' - -. -

.. . - . . . . - .- - . . - - - - ~ - - - -.- - - . > .. , 9- t to ensure that the guide rib bore was adequately sized. Also, in certain cases the licensee determined that machinists had used assumptions, machining- practices, and gauging techniques, when determining the acccptability of the rib bare, in lieu of taking the. dimensions by a more direct method. , The inspectors also found there was no quality control inspection of these dimensions during Refueling Outage 4. Quality control involvement had , included hold and witness points on dye penetrant examination, reassembly, and , torquing of the valve, but the verification of valve-machining and final ' dimensions was not performed. Quality Assurance Directive, QAD-10, " Inspection Program," Paragraph 5.8 states that repairs and rework'shall be inspected in accordance with the original design and inspection requirements or a documented engineering approved alternative. The failure of the machinists or quality control inspectors to verify that the as-left dimensions of this valve met the original design specifications is an apparent violation (458/93018-2). This ' violation resulted in the loss of a quality barrier that could have prevented - the MSIV IB21*A0VF022B failure. ! Based on the above documented review, the licensee determined that the valve . had clearly failed and was not operable. This conclusion resulted in the i establishment of a task force to determine the root cause and develop + recommended corrective actions.

2.2.2 Root Cause of the MSIV Failure

The licensee's investigation and their review of the previous MSIV maintenance

history determined that the failure of MSIV 1B21*A0VF0228 was the result of insufficient clearance between the poppet and the guide.cibs, which caused I excessive wearing of the guide ribs in the location of the stellite bands on the main poppet. The licensee's calculation indicated a clearance of .010 inches. This was considerably less than the design limits of .025 to .040 inches designated by the vendor. In addition, the lower guide rib was about .010 inches above its specified location in the valve body bore circle. This difference in elevations of the body bore circle and the guide rib circle caused a mismatch between the center lines of the poppet and the body bore. The licensee , postulated that the smaller clearance and raised lower rib accelerated the wearing of the lower guide rib. The wear measured on the lower guide rib of MSIV IB21*A0VF022B was .070 inches. This wear pattern ~ existed after only 6 months of service. Historically, guide rib wear in the RBS MSIVs had been .035 inches or less after 18 months of service. . The licensee reviewed the history of MSIVs throughout the industry and determined that only once before had a valve failed in the full open position. This event occurred at Hamaoka, Unit 3, a Japanese boiling water reactor. .The licensee learned that Hamaoka had measured a poppet-to-guide-rib clearance of ' only .0152 inches and that excessive deformation of the valve cuide ribs had , t v ,~ + v s - . . --uu - -- - --

_. _ _ _ . . _ _ -. _ __ __ _ _ _ __ . _ . ._ - _ _ . . .. 4 a '7 ' - ., -10- been noted. These facts supported the licensee's evaluation of the cause of r ilure. the a During the licensee's review, it was determined that a previous method used by the machinists to conclude that the guide rib bore had been restored to acceptable dimensions had been inadequate. In the past, when repairs were performed to an MSIV guide rib, the-machinist would generally use a section of the guide rib which he assumed had not been repaired or machined as a reference point to determine the final machining radiusL of the rib bore. The assumption was erroneous in the case of MSlV 1B21*AV0F0228 because the rib had been reworked. Additionally, following machining,- a go-gauge with a diameter of 18.974 inches was inserted into the valve to ensure that the poppet would fit the bore without binding. Use of these two techniques led the machinist to conclude that the rib bore was adequately sized. This r.ethod would not ensure that the guide rib bore was between 19 and 19.010 inches as required. Thera was a failure to take hard measurements of the guide rib bora and comaare these dimensions with the as-left dimensions of the repaired poppet. In the past, the licensee relied extensively on vendor representatives to provide dimensions and tolerances necessary to achieve acceptable clearances. , Therefore, job plans and procedures had not, in all cases, contained adequate guidance or requirements for reworking of the valves, causing the machinist to rely on the factory representative for acceptability. Also, the licensee determined that personnel errors had occurred in that the machinists in rome cases used questionable assumptions, machining practices, and gauging techniques when determining the acceptability of the rib bare, in lieu of taking hard dimensions. The inspectors reviewed the data, mapping, and measurements taken by the licensee during the investigation. In addition, reviews of machining practices and interviews with site personnel were conducted. The inspectors . concluded that, with the exceptions noted, the licensee's cause determination was reasonable and supported by the facts. i 2.2.3 Corrective Actions for the MSIV Failure During Forced Outage 93-01, the licensee reworked MSIV 1B21*A0VF0228. The licensee machined off the previous weld overlay from all three guide ribs. The ribs were then rewelded with stellite and machined to within the recommended dimensions. To accelerate the outage schedule, the licensee had the poppet replaced with the one from MSIV IB21*A0VF022C (also being reworked at the time this poppet was repaired), had the poppet seat ground and polished, and had the pilot seat lapped to within manufacturers requirements. The old stem was replaced because of wear on the pilot outside diameter. The valve body seat was ground to proper dimensions and the valve stop plate was replaced. - The MSIV was reassembled and an informational local leak rate teat was performed. During the local leak rate test, the engineer noted increased

in-ltakage upon removal of air pressure from the actuator, it was suspected 1 l -- - _ - _ _ . , _ _ . _ . - _ . . . . i

- _ _ _ _ _ _ _ _ _ . . . -11- that the external valve springs were worn and contributed to the valve's inability to maintain proper seating. The old springs were replaced with spares, and the valve failed its local leak rate test again. . The MSIV was disassembled, the body seat ground,-and the poppet seat cleaned. Following the use of machinist bluing techniques to verify proper contact of 1 l the seating surfaces, the valve was reassembled. Again, the valve failed its local leak rate test. The valve was disassembled, and an analysis of the valve internals was performed. Mechanics identified that the body seat was not concentric with the guide rib bore and body upper bore. The body seat was ground to be concentric with both. The poppet seat was also remachined to match the body seat. The valve was reassembled and an acceptable local leak rate test was performed. This rework was identified as a result of lessons learned during the analysis of local leak rate test failures discussed later in Section 3.2 of tnis inspection report. In addition to the repair of MSIV IB21*A0VF022B, the licensee also evaluated the potential impacts of the cause of the failure on the other seven MSIVs. The reviews and findings of this evaluation are documented in Sections 3.1 and 3.2 of this inspection report. To prevent future valve failures in the open position, the licensee developed a corrective action program. Prior to completion of the task force activities and restart of the plant, the following actions were completed: 3 Gulf States Utilities obtained the MSIV component dimensions and e tolerances from the vendor and incorporated them into the Gulf States l Utilities document control system for future reference in procedure development as well as comparison with as-left dimensions. The standard MWO job plan for MSIV repair had been expanded to include l e ! data sheets for recording dimensions. This job plan was implemented for repairs performed on the MSIVs during outag2. All machinists have been counseled on boring machine setup and machining

techniques to assure accuracy. The requirements for the documentation of the measuring and test

equipment utilized during safety-related work was emphasized to those involved in the rework. The licensee also developed long-term actions to prevent future machining discrepancies: The CMP for the MSIVs and future job plans were to be revised to provide

further directions and data sheets for recording dimensions, specifying tolerances, and listing the measuring and test equipment to be used. _ _ _ _ - _ _ _ _ _ _ _ _ - _ - _ _ _ _ _ _ - _ _ _ _ _ _ _ - - _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ - _ - _ - _ _ _ _ - -

_ - _ j , .

.

-12- A set of instructions for boring machine set-up and machining techniques

were to be developed to ensure that consistent set-up, measurements, and machining techniques were used by all machinists. Gulf States Utilities would evaluate the need for identification of

additional critical variables that may require verification of dimensions and clearances for acceptable valve maintenance. The need for development of advanced training for use of the boring

machine was to be evaluated by the mechanical training advisory committee. The inspectors reviewed the licensee's corrective actions and determined that l the specific problems with MSIV IB21*A0VF0228 had been appropriately

corrected. The licensee personnel were meticulous in data taking and preserving as-found conditions during the investigation. Following the identification of the failure,-during the outage, the licensee's corrective actions to address the failure were excellrat. 2.2.4 Safety Significance of the Failure of MSIV to Clo.se The licensee reviewed the impact to the design basis accidents and an analysis of the safety significance of having MSIV IB21*A0VF022B stuck open. One event was found which could potentially exceed the existing design basis analysis and/or regulatory rw.irements. This event was the main steam line break outside contains 9t muming that- the second MSIV stuck open. This event could exceed 10 CFR Part 100 limits and had a probability c' ance which was 20 percent of the NRC safety goal for large releases. The licensee assumed that, following the main steam line break ou.. de , containment, the inboard MSIV failed to close. The assumption was ten made that a single failure of the associated outboard MSIV prevents its ,losure. l The main steam line could then only be isolated by Main Steam Shutoff ' Valve IB21*M0VF0988. This valve does not have an automatic closure signal; therefore, the licensee assumed that operators closed the valve 20 minutes into the accident. T'he excessive MSIV leakage will also cause the main steam positive leakage control system to shut down. This . system provides a positive seal eliminating i, out-leakage through the closed MSIVs directly to the environment. The system ' shuts down to prevent excessive in-leakage and resultant pressurization of primary containment. Primary containment would be protected by this. shutdown; however, the positive seal would be lost, opening the total MSIV leakage back up to the environment. A rigorous analysis was not performed for this event; however, the design engineering organization assumed that this analysis would probably exceed , design basis offsite dose limits. This review and assumption was mitigated by the fact that outboard MSIV IB21*A0VF028B in that line passed its local leak - _ _ , _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ . _ _ _ . . . .. . - . ..

_. _ _ . . . ._ , % .i

, -13- l rate test during Planned Outage 93-01. This indicated that the line would have been isolated and relatively leak tight had it been called upon to function during the time that MSIV IB21*A0VF0228 was stuck open. 2.3 Surveillance Test Discrepancies During normal plant evolutions the MSIVs were tested monthly according to Procedure STP-051-201, "RPS --Main Steam Line Isolation Valve - Closure

Monthly CHFUNCT," Revision 3. This test involved a partial stroke test for reactor protection system (RPS) verification and satisfying inservice testing _ requirements. Full stroke testing at outage times.was performed according to Procedure STP-109-6302, "MSIV Full Stroke Operability Test," Revision 5, for , containment isolation timing and inservice testing _ requirements. The RPS- closing time requirements were checked during outages according to -Procedure STP-051-4852, "RPS - Main Steam Line Isolation Valve Closure _18 - ^ Month Response Time (IB21*A0VF022B)," Revision 6, and the system logic was tested'according to Procedure STP-051-4262, "RPS - Main-Steam Line Isolation i Valve Closure 18 Month CHCAL, 18 Month LSFT," Revision 5. Finally, during outages, local leak rate testing was performed according to Procedure STP-208- . 3602, "_ Inboard and Outboard Main Steam Isolation Valves and Outboard Drain a Valve 'B' Steam Line Leak Rate Test," Revision OC with Change Notice 92- '+ 0756. - On February 27, 1993, operators had performed Surveillance Test Procedure STP-051-0201, "RPS-Main Steam Line Isolation _ Valve - Closure Monthly ~ ChFunct." The stated purpose of this test was to perform a channel functional x ' test of the RPS MSIV_ closure instrumentation and to verify valve. operability by partial stroke testing in accordance with Technical Specification 4.0.5 and _ q the licensee's pump and valve inservice testing program. Procedural Step 7.4.5 required the operator to " depress B21H-S3F MSL B INBD. 4 ' MSIV TEST push button until MSIV B21-F022B shows' double indication and then- immediately release." During the testing sequence, the operator depressed the , button and released it upon receiving an RPS half-scram. .During this__ ~ evol.ution, the expected double-position indication, indicating tha_t the ' 90 percent'open limit switch.had actuated,-did not light. The operators' response was in accordance with Precaution 5.1 which reads as follows:- l ! "5.1 While slow closing the MSIVs, use all available indications to determine that the MSIV is closing. Only allow' th'e' MSIV to close , to the point where: 5.1.1- The MSIV~ indicates mid-position, or , 5.1.2 RPS Half-scram is initiated, or -l 5.1.3 MSL flow changes are indicated."

It should be noted that Step 7.4.5 was annotated with a "TS" and an "I." Under Sectis., 3.0, " Definitions," Procedure STP-051-0201 states that steps

, t .. Y, e- , - - ,e n-- , ,, n .

. .- .. .- . - . . - - . -- - -. , . -14- , , marked with a TS are requred to be completed for Technical Specification acceptance criteria. Additionally, steps marked with a Capital I are required to be completed for inservice testing acceptance criteria. The operator annotated the step, referencing Note 1, in which he stated that "B21*F022B failed to make up close limit switch." The test was continued until completion. The operator presumed that the intermediate limit-switch (actuated at less than 90 percent open) had failed to function, and at the completion of the test the operator wrote MWO R174734 to repair the intermediate limit switch on the valve. The shift supervisor. documented the , test as'" Acceptable with Comments," and documented under Note 1 that, although the valve failed to indicate double indication, the test was judged satisfactory according to Section 5'.l. A test exception was not taken, and. the valve was declared operable by the shift supervisor. . Section 8.0, " Acceptance Criteria," of Procedure STP-051-0201 stated, in part, , ' that Step _7.4.5 must have been. completed satisfactorily in order.to meet the acceptance criteria. Administrative Procedure ADM-0015, " Station Surveillance Test Program," states in Section 4.6.4 that the shift supervisor / control operating foreman is responsible for signing surveillance procedures,. signifying completion of his review and that the acceptance criteria have been . . met. Section 8.3.2 states that if, during the conduct of a surveillance test , procedure, a component failed to meet the acceptance criteria but all.other tested components met the acceptance criteria, there were three options: l ! The whole procedure performance is declared unacceptable and so noted on e the Data Package Cover Sheet by checking the " Unacceptable" block, or The procedure performance may be declared to be a- partial performance . e and the unacceptable portions taken as an exception per Step 5.8. In this case, the partial test would be " Acceptable With Comments" and so noted on the Data Package Cover Sheet, or At the discretion of the SS/C0F (shift supervisor / control operating e foreman), the procedure performance may remain open or uncompleted until resolution and retest of the failed portions of the surveillance test procedure. The operator's failing to properly document and resolve the problems with

MSIV IB21*A0VF022B when it failed to meet the acceptance criteria i a failure to follow the surveillance test program documented in Procedure ADM-0015. Therefore, this is an apparent violation of -Technical Specification-6.8.1.d (458/93018-3). As discussed in Section 2.2 of this inspection report, MSIV.lB21*A0VF0228 was- ( found to be stuck open on April 23, 1993. lAt the time that Procedure STP-051-0201 was performed on February 27, 1993, MSIV IB21*A0VF0228 was, most likely, lodged in the _ open position. The test results of the STP ' indicated this failure and, if the appropriate actions had been taken by the I 1 1 .- - . . ._., - .. . j

- . . _ . - . - - - .- - . . . - , - . - l - l - . -15-

operators at that time, further testing of the valve would have confirmed that , the valve was inoperable. ' Technical Specification 3.4.7; states that two MSlVs per main steam line shall l be operable with closing times greater than or equal to 3 seconds and .less than or equal to 5 seconds. With one or more MSIVs inoperable; the action y - ! statement requires that the licensee is to isolate the'affected main steam line by use of a deactivated MSIV in the closed position. Otherwise, the , licensee is required to place the plant in at_least hot shutdown within the next 12 hours and in cold shutdown within the following 24 hours. ' On February 27, 1993, MSIV IB21*A0VF022B was apparently inoperable. From . " February 27 through April 17, the plant was continuously operated in the power' operating conditions with Main Steam Line B unisolated. This is an apparent

violation of Technical Specification 3.4.7 (458/93018-4). On April 1, 1993, STP-051-0201 was rep'eated at its normally scheduled interval. The RPS actuation and annunciation was received but,~again, the double valve position indication was not received. Based on the previous disposition, the operating crew assumed that the issue concerning the , acceptance criteria had been' resolved. Additionally, a deficiency tag had been hung on the control panel, indicating that the intermediate limit switch was not operating. The procedure was again signed off as acceptable with > comments. This is a second example of the apparent violation of Technical

Specification 6.8.1.d (458/93018-3). During the review of Procedure STP-051-0201, the inspectors determined that, although the procedure did indicate that the poppet in MSIV 1821*A0VF022B may

have been lodged in the full open position, this procedure would not have,

under all circumstances, detected an immovable poppet. Procedure STP-051-0201

was performed to meet the inservice testing requirements identified in .. 'i Technical Specification 3.4.8. Positive indication of valve disk movement was only to be verified if the 90 percent limit -switch was actuated. These, limit switches were subject to repositioning when the valve operator was disassembled and re-assembled. The 92 percent limit switches, used for input d to the reactor protection system, were' calibrated during surveillance testing to be set at 94 percent, +2 percent, -2 percent. These 92 percent limit

switches were mounted next to the 90 percent limit switches. ! i As documented in Section 2.1 of-this inspection _ report,-the valve stroke is

11 inches, including a 1-inch pilot valve stroke. By design, 90 percent.open ! would require a 1.1-inch stroke, thus -verifying that the poppet itself . moved. 1 However, given that the limit switches are only accurate to +/- 2 percent, the

setting could be as much as 0.22. inches off. Therefore, the worst case > scenario would be a seized poppet.with all limit switches'still indicating , poppet movement. Additionally, the- 92 percent limit switches were calibrated or verified as. , part of Procedure STP-051-4852 once every 18 months. However,'the 90 percent

limit switch was not calibrated or verified as part of the surveillance test j l T t- ! 2 -. -- . . - - - - .- -.- . . . .. --

. _ _ _ _ _ 4 . -16- , ! program. Testing records did not indicate that there was any regular verification of the setting of this switch, nor was there any quality control oversight of the positioning of this limit switch when the valve actuator was - disassembled and reassembled. Quality Assurance Directive, QAD-11, " Test Control," paragraph 5.3, states that surveillance testing shall be performed to provide assurance that failures do not remain undetected and that the required operability of safety- related equipment is maintained. There was no assurance that the main poppet had actually moved when the double indication was received with a limit. switch setting of 90 percent, nor was there assurance that the switch was even set to 90 percent based on the lack of calibration. The failure of Procedure STP-051-0201 and the surveillance program to provide such assurance- is an apparent violation (458/93018-5). 2.3.1 Cause of the Surveillance Test Procedure Performance Discrepancies The licensee reviewed these events and determined that the cause of the operator's failure to recognize the inoperability of the MSIV based on the test results was the senior reactor operator's lack of understanding of the requirements of Procedure ADM-0015 for surveillance test procedure acceptance criteria and use of " Acceptable with Comments." Had the crew observed the exact wording of Procedure STP-051-0201, Section 8.0, and applied it to the guidance of Procedure ADM-0015, they would have identified'that the MSIV did not meet the inservice testing requirements. The inspectors reviewed Procedure ADM-0015 and determined that following the administrative requirements should have resulted in a satisfactory review. Also, during an interview with the senior _ reactor operator, it was clear that the definition and usage of acceptance criteria were misunderstood. There- fore, the inspectors concluded that the licensee's root cause determination was reasonable and was supported by the facts of.the investigation. The licensee identified several factors contributing to the root cause:

Ambiguities in Procedure STP-051-0201 as written, e The operators' misconceptions.regarding MSIV design and operation, and e The shift supervisor's active involvement in the problem resolution rather than review and oversight of the final product. Procedure STP-051-0201.was identified by the task force to be ambiguous in three areas. First, the title identifies the procedure as only performing a . reactor protection system channel' functional test. -This was misleading " because the procedure also performed a partial stroke to verify valve operability in accordance with Technical Specification 4.0.5. ' . l. .

.- - - . . . . - - . - --- . . .. -17- l

Second, the acceptance criteria in Section ;8.0 were misleading. Section 8.1 > was titled " Channel Functional Test," and listed Step 7.4.5 as required ~to be , completed satisfactorily. This implied to the operators that successful completion of this step was required to satisfy the reactor protection system , channel functional test only. Section 8.2 was titled " Partial Stroke Test" and stated that-an MSIV was considered acceptable.if it completed one cycle of a partial stroke travel. This section did not list Step 7.4.5, nor did it. .. '! determine what constituted partial stroke travel. .This wording, combined with the operators' misconception of the operation of- the MSIV and limit switches, led to confusion of what was required to prove MSIV operability.

finally, the precaution given in Section 5.1. was misleading in that it implied - ' that the test need not be continued beyond the valve position at which a 1/2; scram was received. This led the operators to believe that the partial stroke

acceptance criteria would be met at this point. ' A contributing f actor to the.overall problem as determined by the task force - ! was the operators' misconception of the operation of the MSIV pilot valve, main poppet, valve stem, and limit switches. As described in Section 2.1 of- this inspection report, with the poppet stuck, the stem and pilot valve will- move approximately 1 inch. This movement is sufficient to actuate the RPS- . ' _ ' limit switches. The operators believed that this movement and subsequent RPS limit switch actuations indicated actual poppet movement when, in fact, it did not. Finally, the licensee stated that af ter Procedure STP-051-0201- was completed, the night shift supervisor performed a review of Technical . Specifications ~ and the procedure. He determined that the surveillance requirements.of Technical Specifications had been met, and he signed the procedure as acceptable with comments. Normally, the control operating foreman would have performed a i review of this nature and determined the acceptability of.the procedure. The- , shif t supervisor would then perform an independent leview. The inadequate surveillance testing, based on the position of the 90 percent. open limit switch appeared to be an industry standard. Most owners of Atwood and Morrill.MSIVs reported to the licensee that their respective limitLswitch settings were the same as those at RBS. The licensee issued a notice on Nuclear Network.to inform other licensee's that partial stroke testing, using > a. limit switch setting at 90 percent open, may not always' identify.an . immovable poppet. j 2.3.2 Corrective Actions'for the Surveillance' Test Procedure Discrepancies , The licensee's Operations Department convened.a special . investigation ' team to -

review these discrepancies and~ concerns raised by the inspectors and the MSIV task force. Recommended. corrective actions included revision'of

Procedure STP-051-0201, appropriate training of the operators, and a' review of j current surveillance test procedures to ensure that procedures orocessed as

" Acceptable with Comments" did not contain similar errors concerning the treatment of the acceptance criteria. t , , - - - . . m . . m. __ ,__- .

.- - . i . . . -18- . On May 21, 1993, the licensee issued a letter to the NRC documenting-the status.of corrective actions taken to improve the surveillance test program. . The licensee stated that they were confident that the plant staff's actions, would be successful 'in identifying other similar concerns if they existed.and ' would prevent similar occurrences in the future. , The licensee revised Procedure STP-051-0201 to more clearly. explain the limitation stated;in Section 5.1 of the procedure. Included in the procedure revision was a drawing'of the MSIV and the limit switches for increased . understanding of the valve s functioning. Additionally, the -limit switch ~ .' position and function was identified at the applicable steps in the procedure. .The inspectors reviewed Revision 4 to Procedure STP-051-0201 and Revision 6:to STP-109-6302, "MSIV Partial Stroke / Full Stroke Operability Tests," with an. approved' change notice and found them to be acceptable. The licensee held discussions with the shift supervisors involved to. clarify management's expectations with regard to surveillance test acceptance criteria. Additionally, a meeting with all shift supervisors, control operating foremen,- and shift technical advisors was_ conducted, to train them 3 on their respective roles as operations management and on the ADM-0015

requirements with regard to acceptance criteria. As a result of the licensee's review, a surveillance test procedure review group composed of individuals from operations, engineering, and quality assurance was developed to review the performance of record (last performance) for all surveillance test procedures. This review was designed to identify-

any tests which are completed and documented under a test' exception or closed as " Acceptable with' Comments." Those tests identified then underwent a , detailed review to determine that the acceptance criteria had 'actually. been met and that Technical Specification surveillance requirements were accomplished.

' During the review, 593 out of 1199' surveillance _ procedures were ~ identified that had'been completed as. acceptable with comments. Each of these items.were revi'ewed. in accordance with a checklist developed to ensure that the . ' surveillance procedure had been dispositioned-in accordance with plant administrative. procedures. Items.of comment ranged from instrument , deficiencies to abnormal system alignments. However, none of the items affected the Technical Specification requirements or the acceptance. criteria- ' of the procedure. Therefore, no errors concerning the treatmentlof acceptance- criteria similarJto those involving Procedure STP-051-0201- were-identified- during this. review. The inspectors ~ reviewed the licensee's checklist and determined that the appropriate questions had been asked.

' The licensee issued a design Modification Request 93-0021 to' change the position,of the " valve less than fully open" limit switch from'90 percentLto 85 percent. The MSIV task force stated that the potentially misleading. midposition indication would be corrected, prior to restart from Forced Outage 93-01, by changing.the limit switches on all valves from 90 to , 85 percent open position. The inspectors reviewed the modification request- , . .- e- ,

. ._ - - - ..- - . - . -. - . - -- . . - . ._ ..

. -19- 1 and determined that the design change should correct.the problem. The inspector noted that the modification package provided for.the. calibration of < these limit switches on a routine basis. The inspector ascertained that the limit. switch adjustments had been made prior to plant restart.

The licensee cammitted in Licensee Event Report 93-006 to continue monthly. partial stroke testing, and add quarterly full slow stroke testing while at power, to ensure that the corrective actions for the MSIV 1821*A0VF0228-were adequate. This testing will continue until Refueling Outage 5. The. licensee will reevaluate the need for continued increased testing at that time. The inspectors evaluated the licensee's corrective' actions andfdetermined that a repeat of the problems identified in this report should be prevented and , that all current surveillances should have been adequately dispositioned. The l'icensee's corrective actions in this area were_ considered to be good. , 2.4 Conclusions The inspectors determined that the failure of the RBS surveillance test- program to. identify that MSIV. lB21*A0Vf0228 had failed in the stuck open . ! position, .was a significant problem. Technical Specification 4.0.5 provides !1 surveillance requirements for the operators to. partial stroke test this-valve,- , on a quarterly basis, in accordance with the licensee's NRC-approved pump and valve inservice testing program. This requirement was written-to ensure that- j the valve was free to move and not failed open. The purpose of a surveillance , test program is to identify and correct potential' safety problems and

inoperable safety equipment in a' timely manner. MSIV IB21*A0VF0228 failed in the open position because insufficient clearance

between the poppet and the guide ribs, combined with excessive guide rib wear, 3 caused the main poppet to become lodged in the upper bore. . The inspectors ' determined that on February 27, 1993, the licensee should have.known that MSIV 1821*A0VF022B was inoperable. At this time, two licensed operators " , failed to properly document and resolve problems identified with the valve 1 when it-failed to meet the acceptance criteria of a surveillance test procedure. By failing to identify the valve inoperability, the licensee also . failed to shut down the reactor in accordance with Technical

Specification 3.4.7 when MSIV IB21*A0VF0228 became inoperable. . , The. licensee developed a task force to review the causes and to. recommend corrective actions for the failure of MSIV IB21*A0VF0228. The causes and ) ~ corrective actions'were reasonable and supported by the data, mapping,'and- , measurements taken. Once identified, the licensee's response and corrective , actions in response to the specific failure were excellent. The licensee l ' determined that the senior reactor operator lacked understanding of:the administrative requirements for surveillance testing acceptance. criteria. The ! corrective actions to address this lack of understanding were considered good. . n The licensee committed'to continue monthly partial stroke testing and to: perform quarterly full slow stroke testing while at power to ensure' that

, n n . . . - -- - .- .-- . - - - ..- - -. - - . - - .

- .. ._ - - - _ . . , ) . -20- b carrective actions have been adequate. This testing will continue until Refueling Outage 5. The inspectors determined that although Procedure STP-051-0201 did indicate - the failure of MSIV IB21*A0VF0228, the procedure would not have, under all circumstances, detected an immovable poppet because of limit switch inaccuracies and the lack of calibration of the limit switches. As a result, the licensee modified the limit switch position and provided for routine calibration of the limit switches. Repairmen performed work _during Forced Outage 93-01, that'was neither governed by nor in accordance with approved procedures. This appeared to be a repeat of previous violations for plant workers performing evolutions that-were not completely controlled. q 1 3 OPERABILITY OF SEVEN OTHER MSIVs As a result of the failure of MSIV 1821*A0VF022B, the inspectors questioned- the operability of the other seven MSIVs at RBS. This evaluation identified several areas of concern. These areas included: the potential for the other valves to fail in a similar manner as Valve IB21*A0VF0228; the historical problems with local leak rate test failures of MSIVs IB21*A0VF022A, 1821*A0VF022B, 1821*A0VF022C, and IB21*A0VF028B; and the design deficiencies which have caused guide rib wear to occur in these valves. 3.1 Potential for Other MSIVs to Fail in the Open Position As part of the review of the failure of MSIV IB21*A0VF0228, the licensee evaluated the potential that other MSIVs could exhibit similar characteristics and that the apparent causes for the failure of MSIV 1821*A0VF0228 could be generic to the other valves. The need to evaluate the inboard valves was mute. As documented in Section 3.2.2 of this inspection report, MSIVs IB21*A0VF022A,1B21*A0VF0220, and IB21*A0VF022D were disassembled and inspected to the same standards as MSIV 1821*A0VF0228- because of local leak rate test failures. By physical measurement, the licensee verified that these , three valves did have sufficient' poppet-to-lower-bore clearances. Therefore, these valves were not susceptible to failure in a similar manner as MSIV IB21*A0VF0228. The outboard valves all passed the local leak rate tests. Therefore, they , were not required to be disassembled for repairs. However, the need to disassemble.these valves to ensure that they were not candidates for failing in the open position was questioned by the inspectors and reviewed-in detail by the licensee. In evaluating the need to disassemble and inspect the outboard MSIVs, the licensee considered the apparent cause of the failure on MSIV'IB21*A0VF022B, The licensee concluded that the failure was effected, in part, by insufficient clearance between the poppet and guide ribs. Therefore, it was concluded by , , , ,- --: - e

.- . . __ . . . . - . . . ..- , - . -21- , the MSIV task force that any valve known to have sufficient clearance would not need to be disassembled. The MSIV Task Force conducted an MWO search to identify each time that an MSIV had failed a local leak rate test and was disassembled to perform repairs. In conducting this review, the licensee noted similarities in the test performance records between Inboard MSIV IB21*A0VF022D and Outboard > 'MSIV IB21*A0VF028B. Valve ID21*A0VF0288, the outboard MSIV on Steam Line B, failed its first three local leak rate te:ts. As corrective action to these. failures, the licensee- , built up the stellite overlay on the lower guide rib. This valve then passed its local leak rate tests during Refueling Outages 3 and 4 and during-Forced Outage 93-01. The task force evaluated the as-found condition of MSIV 1821*A0VF022D, 'the inboard valve on Main Steam Line D, and found minimal guide rib wear. The- average dimension of the guide rib diameter was found to be 19.010. inches. This valve had last been reworked during Refueling Outage 2. At.that time, the valve inner diameter at the guide ribs was documented to be 19.010. This- measurement was verified to be accurate by the dimensional analysis performed during Forced Outage 93-01. The licensee utilized this measurement to compare the similarities between MSIVs IB21*A0VF022D and IB21*A0VF0288. Both valves had shown very similar guide rib wear when inspected during Refueling 0utage 2. Both valves had then . ~ had all three guide ribs rewelded by Welding Services, Incorporated during l that outage. Both poppets were restellited and machined down to 18.976 ' inches, plus/minus .001 inches. Both poppet seats were ground and the pilot , seats were lapped. Additionally, both valves received new stems. The' licensee also determined that both valves had been machined by the same . two machinists using the same machining methods and the same type boring bar. The same individual performed the final machining on. both valves. Both valves had the go-gauge installed after machining was completed. Although MSIV.lB21*A0VF028B.did not have its guide rib diameter measurement documented- in the work package, the licensee believes that, since MSIV 1821*A0VF0220~was machined using the same individuals, the tools, and techniques, the valve was, considered to be within vendor tolerances. , 'The licensee concluded that, since both valves were worked within a few days of each other and machined'to dimensions specified by the vendor representative present-at that time, both valves had been stroke tested satisfactorily and had passed local leak rate testirequirements for the past- ' 21/2' cycles, and measured clearances .in MSIV IB21*A0VF0220 were acceptable, the internal inspection of MSIV 28B.was not necessary. ~The inspectors ' reviewed the documents which supported this conclusion and determined that the

< decision was reasonable. j .c h - - J -. ..i .m . . - - . . . _ . _ _ , _ . .

. - -_ - _ _ _ _ - - - . - ._ . - . - . ~_ - - - .-. . . -22- < Inboard MSIV 1821*A0VF022A had never been subjected to guide ~ rib repair or machining since its manufacturing. This valve was inspected during Forced Outage 93-01, and the licensee used its condition as a basis for determining the expected condition of the three outboard valves in question. Welding Services, Incorporated performed detailed mapping of the contour of

the guide ribs for MSIV 1821*A0VF022A. The average poppet-to-guide-rib ' clearance was found to be .027 inches, which was well within the vendor specified tolerance requirements. One exception was a small area of the bottom rib which had a .small raised area. This area caused that portion of . the guide ribs to be less than factory tolerances. The licensee contacted the vendor who concurrcd that this area was an insignificant variation in the factory machining process. When inspected, MSIV IB21*A0VF022A had .020 inches of lower guide rib wear. This wear had not substantially changed since it was first measured in Refueling Outage 3. This indicated to the licensee that, over a period of operation, the stellite surfaces may have become work hardened. The task force determined that MSIVs 1821*A0VF028C and lB21*A0VF028D have satisfactorily passed their respective local leak rate tests since plant startup in 1985. As a result, these valves have not been opened for inspection or rework since initial installation. MSIV 1821*A0VF028A has had ? the same leak test history, with the exception of one failure during Refueling i Outage 1 testing. At this time, no guide rib repair or machining was performed. Therefore, all three valves were assumed to remain within factory ! tolerances. The licensee concluded that all valves that have not had the guide ribs t repaired or machined since manufacture would not exhibit the excessively narrow tolerances required for a valve to fail in the_ open ' position.as discussed in Section 2.2.2 of this report. Therefore, the decision was made not to disassemble MSIVs 1821*A0VF028A, 1821*A0VF028C, and 1821*A0VF028D. 3.2 Local Leak Rate Test Failures As a result of the MWO search performed by the MSIV task force, the licensee identified that MSIV 1B21*A0VF022C had been repaired during Refueling Outage' 4- at the same time as MSIV lB21*A0VF0228. Therefore, the licensee decided to- -. disassemble and inspect MSIV 1821*A0VF022C to.look for similar machining deficiencies. Also, the licensee decided to perform an as-found local leak- rate-test as a baseline. The results of the-local leak rate test were unsatisfactory. Following the as-found test failure, the licensee removed the bonnet' and poppet from MSIV IB21*A0VF022C. The inspection and mapping of the lower guide rib indicated minimal wear. When extrapolated by the system engineer, the wear was consistent with the wear found following 18 months of service. .The . -task force 'oncluded that MSIV IB21*A0VF022C was not at risk of sticking by

c the same mechanism as MSIV 1821*A0VF022B. However, the licensee identified- t t .I ,, % ,m.5 -. , . m- , - . - , . . . m . . - , _ . - - - ., ,- -w

.- - - - . .- - . . . . -- - . . - . ._ - t .. -23- that MSIV IB21*A0VF022C had traditionally failed its local leak rate tests. The task force recommended a complete review of the local leak rate test failures of MSIV IB21*A0VF022C to identify the cause for its_ appropriate , corrective actions. Condition Report 93-0233 was written to document'this j review. The investigation of the leak rate failure of MSIV IB21*A0VF022B led the licensee to perform local leak rate tests of the remaining six MSIVs. All the outboard valves passed the tests. However, all the inboard valves exhibited excessive leakage rates. This condition was documented under Condition ' Report 93-0245. Therefore, MSIVs IB21*A0VF022A and IB21*A0VF022D were a disassembled and inspected. Neither valve indicated inadequate tolerances or excessive guide rib wear. 3.2.1 Cause of local Leak Rate Test Failures The licensee determined that the cause of the local. leak rate test failures ! was insufficient seating'of the main poppet into the body seat. During a normal closure of an MSIV, the poppet would be setting low in'the bore because gravity would tend to pull the poppet to the lower guide rib. This would > create a-slight lateral misalignment between the seat and the poppet because of the poppet-to-lower-bore clearan'ces. Upon contacting the body seat, the poppet slides up and into the body seat to complete concentric seating. The as-found measurements taken by a machinist bluing check showed .that all valves contacted only 270 to 330 degrees.

The licensee identified three contributing factors to this lack of appropriate seating. First, when the valves were disassembled, an oxide film was found on- the stellited surfaces. This film was normally found each time the MSIVs were inspected. .The nature of.the film is not currently known; however, the vendor. representative confirmed that the deposits were similar to those observed in

all other plants. This film has been removed each time a valve is inspected or lapped.

The~ vendor representative stated that the oxide film has a higher coefficient of friction than the stellite alone so that, when the poppet contacts the lower side of the body seat, this increased. friction prevents the sliding i -action required to take up the lateral displacement. The second contributing factor was identified as the inadequacy of valve t design that allows the accelerated spinning and vibration of the poppet. This mechanism is discussed in Section 3.3 of this inspection report. When . inspected the licensee identified wear and high spots on the bottom guide ' ribs. All four valves inspected 'also exhibited wear on poppet internals. This damage was determined to be caused by the poppet spinning and vibration. , Wear of these components caused additional friction during the closing and seating of the main poppet. The final contributing factor identified by the licensee was that the tests were performed during cold dry conditions. During hot conditions, the ribs i ,

.- - - . . - . . - - . . - - , ~- - - . . t =. . '( -24- l;

' and seats are normally wet and lubricated, thus providing less friction-for the poppet to slide up and into the body seat. The local leak rate. test history shows that all eight MSIVs displayed higher leak rates during Forced Outage 93-01, when the valves were closed under cold and dry conditions, than . ' during previous outages when the valves were closed while the system was still hot. 3.2.2 Corrective Actions for Leak Test Failures The licensee reworked and repaired these valves to return them to within-the manufacturer's tolerances. The rework included repairs- to the guide ribs, machining of the poppet seating surfaces, grinding of the valve body seat, and repair / replacement of poppet internals. This repair work included taking j measurements and observations similar to those taken for MSIV 1821*A0VF0228. , Following initial repair of the valves, MSIV 1B21*A0VF022A failed an informational leak rate test. The licensee determined that the actuator. stroke .had not been set properly and the valve was not closing completely. The inspector noted that.the job plan in MWO R159695,-Step'59 required the adjustment of the stroke following reassembly of -the . valve. However,'the mechanics did not properly measure and ' adjust the stroke at that time. Although postmaintenance testing did identify the deficiency, failure to follow the instructions for performing the corrective' maintenance of a safety- related component is considered the second example of an apparent violation- involving procedural compliance (Violation 458/93018-1). Licensee representatives reviewed this event and provided more explicit instructions to the workers in the licensee's standard job plan. Following i the repairs, MSIVs 1821*A0VF022A, 1821*A0VF022C, and 1B21*A0VF022D passed their respective local leak rate tests and stroke time tests. To prevent recurrence, the licensee was developing new machining procedures as documented in Section 2.2.3 of this inspection report. Additionally, the licensee will remove ~.the oxide film by polishing,the poppet seat to'a smoother finish in the future. This action will help reduce poppet: friction _and'the ' amount of oxide film buildup. Licensee representatives stated that full stroke testing at power will also help to minimize the oxide film buildup. ' The licensee stated that the most positive method to ensure full . seating of .the MSIVs would be the installation of a modification proposed by the vendor ) to install an-elongated anti-rotation poppet and a new design bonnet. The' , elongated poppet has a' nose piece which guides:the poppet to the. seat on closing. This should improve local leak rate test results'in the future. This upgraded poppet and bonnet combination has been installed in a number of- i i other plants in the nation. . Purportedly, the modification has achieved . excellent results. Gulf States Utilities has established a goal to install the antirotation kit in all eight MSIVs during Refueling Outage 5.

i a . 4 F , y a -I. - .w- e + ,

.. _. . _ _ _ . . _ - .- . . . _ _ . e

, ' -25-

3.2.3 Safety Significance of Failure of Inboard MSIVs to Pass Leak Tests . ; As part..of corrective action to the local leak rate test f ailures, the licensee performed an analysis of operating with an; inboard MSIV which did not? ! pass its leak test. .As discussed in Section 2.2.4 of this inspection report', , excessive MSIV leakage would cause .the main steam positive leakage control' ! ' system to shutdown. This would eliminate the. leakage control; system as a positive seal against radiological releases. - 1 For.these worst case local. leak rate test situations, offsite dose - consequences would be driven by MSIV leak rates. In each case, the licensee ' determined that, although increased, the offsite dose consequences of this- e condition were bounded by the existing design basis analysis and/or regulatory - limits. The licensee used Cycle 5 parameters for this analysis. l 3.3 Excessive Guide Rib Wear in MSIVs As stated by the licensee, part of the root cause for the failure of MSIV IB21*A0VF022B to close was the excessive guide rib wear caused by main poppet rotation and vibration. The inspectors, thereft,re, reviewed-the_ vendor recommendations with respect to poppet rotation. ' .! . On December 11, 1989, Atwood.& Morrill Co. Inc. had issued a recommendation to' . , i address disk rotation. The recommendation was to elongate the poppet, install a new cover for back seating, and_ pin the stem to the disk assembly and-other

associated changes. The system engineer in 1990 issued a request for > authorization to install this design change. There were no. records'of this . request. In 1991 and.on March 22, 1992, this same recommendation was re-addressed. These requests were identified with System Enhancement File - Numbers 91-9149 and 92-9008.- A source term reduction study issued. April 25, 1991, found the project acceptable. The Independent Safety Engineering Group- studied MSIV stem separation and recommended that this same modification be implemented (SCRB-16690, dated June 6, 1991). j _ t Until'1990, this modification was not accepted because purportedly'it was not ., a proven solution to the problem of rotation. Since-1990, other modifications '! appeared more relatively desirable. In March of 1991, the work scope l committee elected to implement this' modification in Refueling 0utage S. , However, a later review resulted in the selection of other modifications over- . the antirotation modification of the MSIVs. Presently, since'the failure, .. l this modification as documented under Modification Request 93-0027 was planned ' for installation during Refueling Outage 5. > The-licensee has, since 1991, had a System Enh'ancement File 'as part ofz 'j Procedure. TSP-0034, " Processing Modification Requests by System Engineering," , Revision 0. Modifications not processed but retained for plant bettermente .until-approved-by a work scope' committee'are in the System Enhancement File. "About 275 modifications have been processed in this manner. The- modification

" requerts'are' rated and reviewed by a committee. This modification was scored 327 and the highest score possible was 400.

q e r - . . - . = +m <=w- - t---- 5eEe

m- - -- . w m=,w .c

_ _ _ _ _ - _ _ _ _ _ . . -26- As a result of the findings of the MSIV task force Gulf States Utilities has, established a goal to install the antirotation kit in each MSIV during Refueling Outage 5. This goal is subject to existing. outage schedule constraints and favorable. industry operating experience with this modification. The inspectors reviewed the licensee's conclusion that the installation of the antirotation modification should minimize flow induced vibration of the poppet and the resultant guide rib wear. ! 3.4 Conclusions The licensee's' evaluation of the other seven MSIVs and conclusion that the valves were not susceptible to a failure similar to that of MSIV 1821*A0VF022B was of high quality and was well supported by inspection findings and .the maintenance history. The licensee determined that the cause of the local leak rate test failures was the lack of seating surface contact caused by several problems which increased the friction.of the poppet upon closing and contacting the body seat. Measures taken to reduce valve friction until the end of Cycle 4.were . considered adequate. Gulf States Utilities has established a goal to install the vendor supplied antirotation kit in each MSIV during Refueling Outage 5. This should provide acceptable long-term corrective action for the local leak rate test failures. Maintenance repairmen failed to properly set the actuator stroke on MSIV IB21*A0VF022A, causing it to fail to completely close upon -testing. The licensee implemented a new method of adjusting actuator stroke-during valve coupling to improve worker response in the future. ,' _ . _ . _ . _ _ _ . . _ _ _ _ . _ _ . _ _ _ _ . _ _ _ _ . _ . _ _ _ _ _ _

. ._ _ _ _ _ . .. . ATTACHMENT 1 , t 1 PERSONS CONTACTED 1.1 Licensee Personnel A. S. Abella, Maintenance Engineer '

  • R. J. Backen, Supervisor, Quality Assurance Systems
  • R. L. Biggs, Supervisor, Quality Control
  • J. B. Blakley, Assistant Plant Manager, System Engineering

~;

  1. J. E. Booker, Manager, Safety Assessment and Quality Verification

M. P. Brooks, Senior Technical Specialist R. E. Buell, Supervisor, Nuclear System Design ' W. L. Curran, Cajun Site Representative . . D. R. Derbonne, Assistant Plant Manager, Operations, Radwaste & Chemistry G. M. Dolney, Senior Technical Specialist

  1. P. E. Freehill, Assistant Plant Manager, Outage Management
  • K. D. Garner, Licensing Engineer
  1. K. J. Giadrosich, Director. Quality Assurance
  • P. D. Graham, Vice President (RBNG)
  • J. R. Hamilton, Manager-Engineering

R. P. Hebert, Senior Quality Assurance Engineer

  • D; E. Hoeppner, Supervisor, Valve Group

M. L.-Hora, Design Engineer G. R. Kimmell, General Maintenance Supervisor T. P. Lacy, Outage Director ,

    1. D. N. Lorfing, Supervisor, Nuclear Licensing

R. C. Lundholm, Supervisor, Process Systems - Maintenance

  • l. M. Malik, Supervisor, Operations Quality Assurance'

, G. D.-Mahan, Maintenance Engineer J. E. Martin, Maintenance Engineer R. H. Martin, Systems Engineer C. R. Maxson, ~ Senior. Compliance Analyst

  • J. S. Miller, Director, Nuclear Engineering

J. V. Normand, Senior Civil-Engineer

  • W. H. Odell, Manager, Director, Radiological Programs

_ > E. R. Purnell, Senior Quality Control Inspector

    1. S. R. Radebaugh, Assistant Plant Manager, Maintenance
  • J. P. Schippert, Plant Manager

B. R. Smith, Mechanical Maintenance Supervisor i

  • K. E. Suhrke, Manager, Site Support

W. J. Trudell, Assistant Operations Supervisor R. J. Vachon, Senior Compliance Analyst J. E. Venable, Operations Supervisor W. F. Wilson, Senior Technical Planning Specialist

  • L. W. Woods,.Jr., Supervisor, Operations Training

, 1.2. General Electric Personnel C. Nieh, Program Manager - ' L. Tabke, Site Service Manager ^ - - . . . . - . . . . ,

. . . -2- , 1.3 Atwood & Morrill Personnel

V. Bright, Site Field Representative

  • Denotes personnel that attended the exit meeting on June 1, 1993.
  1. Denotes personnel that attended the exit meeting on June 22, 1993.

In addition to the personnel listed above, the inspectors contacted other personnel during this inspection period. . 2 EXIT MEETING Exit meetings were conducted on June 1 and 22, 1993. During these meetings, the inspectors reviewed the scope and findings of the report. The licensee did not identify as proprietary any information provided to, or reviewed by, the inspectors. . B 5 1 i --e - +

.. . .. ., _ ._ . _ . . . _ _ _ i =: g

ATTACHNENT 2 .. c

  • b %.y

+h* ,6, .

. 4 s, . ./ [# , ta . g ' ' , I ' - -

- ,p / . ' @ - * ' e ' p/p* .( . ts . ^ . { , '. - . . i / 7.. , . e . . . - . O . s ;. - . . ,

. . .. - 9 . ,,% [ >

( -' g3g s 43 ' ,[ '( ;N . g \\ , \\*< - . y )Y .- \\ ,; . qw 4, , ~ g, f.f1 .~

. / 'g p l \\ s . ? . i - d \\7

  • $

_. ~ - {# > . x x 4 ]" di g Iig 3- ? i ' %.~, ,.- W b -i -. ; .i [ t Main Steam Isolation Valve l , . . . . . .

,- .. O nu I r - [7~ .- ._ _ , , . _ . . ' A .. -;f55i {., ,25-ff% ' , 8 -68 ' sW ,\\ 4 \\ '[" $._34, _ fa k'. / .. 'i, i.l , .: g?:_ .PEh *,( '". . .- i -c a .. .. , . . ,- ..~ ,/ . / {{ . 33 9' J $ y ' - . v. 4,3*c'.$7 d'; , ', ,'],+'2 a j' h ic gc 5

  1. L rg

s: -x , n x , ~ ' bb- - ,. % ,. ' ~: - . ' % ~ ~ _ .- ' -sh) Q & o - - - , !g . - 1j - -n- N ,, a .( - ut wo i , MM%) E-6 ttt *Att ! Cross Section of Valve

p _ o- , o > E MSIV LIST OF PARTS Part No. Name of Part 1 Body 2 Poppet- 3 Bottom Spring Seat Assy 4 Yoke Rod Guide 5-210 Cover , 6 Stop Plate 7-221 Valve Stem 7-222 Spring liolder 7-223 Pin 8 Spring Retaining Ring 9 Bushing 10 Lantern Gland 11 Gland 12 Gland Plate 13 Top Spring Seat 14 Spring Spacer 15 Spring (Outer) 16 Spring (Inner) 17 Bottom Spring Seat Cap 18 Valve Stem Plate- 19 Yoke 20 Yoke Rod- 21 Pull Down Bolt 22 Insert 23 Stanchion 24- Stud 25 Stud 26 Stud 27 Stud 28 Hex Nut .. w' w--w - i.- - - -2. . .--. --...u - .- . . 4 . - . - - - - --.-rw

' a . o MSIV LIST OF PARTS Part No. Name of Part E9 Hex Nut - 30 Hex Nut 31 Hex Nut 32 Hex Jam Nut 33 Locking Plate 34 Locking Plate 35 Gasket 36 Packing "QP" 37 Tandem Cylinder 38 Control Panel 39 Socket Headless Set Screw 40 Internal Spring , 41 Lockwasher 42 Name Plate 43 Machine Screw 44 Hex Cap Screw 45 Eye Bolt 46 Locking Plate 47 Switch Bracket 48 Limit Switch 49 Operating Lever

50 Machine Screw 51 Hex Nut 52 Lockwasher ! 53 Hex Cap Screw 54- Check Nut , 55 Lockwasher , 56 Switch Bracket 57 Limit Switch 58 Hex Nut " Flex Loc" , . }}

Retrieved from "https://kanterella.com/w/index.php?title=ML20045E743&oldid=4525640"