IR 05000220/1988002
| ML17055D663 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 03/04/1988 |
| From: | Cook W, Kaplan H, Persinko D NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML17055D662 | List: |
| References | |
| 50-220-88-02, 50-220-88-2, NUDOCS 8803150095 | |
| Download: ML17055D663 (56) | |
Text
U ~ S.
NUCLEAR REGULATORY COMMISSION
REGION I
Report No.
50-220/88-02
\\
Docket No.
50-220 License No.
,Licensee:
Nia ara Mohawk Power Cor oration 301 Plainfield Road S racuse New York 13212
.Facility Name:
Nine Mile Point
Inspection At:
Oswe o
.I i: J 1nspectors:
W. Cook, Senio Resident Inspector te H. Kaplan, Reactor Engineer, RI date P. Eselgroth, Team Leader, RI date Approved by:
W. V. Johnston, Acting Director, DRS date Ins ection Summar:
Ins ection on Januar 11-14 1988 Re ort No.
~/~AI: Wit t
0 t>>,
987f d t transient, the team performed system walkdowns; inspected available damaged components and in-progress repairs; interviewed maintenance and operations personnel; reivewed plant logs and equipment history records and preliminary event related pipe stress analysis work.
Results:
The team concluded that the licensee had established a comprehensive technical review effort for the event.
It was also apparent from records review that the feedwater system has a history of initiating unplanned challenges to the reactor protection system and thermal transients to the reactor and its components, There are no violations and no unresolved items.
There are three licensee commitments in Section 6.0 of the report.
8S03l50095 SS0307 PDR ADOCK 05000220
U.S.
NUCLEAR REGULATORY COMMISSION
REGION I
Report No.
50-220/88-02 Docket No.
50-220 License No.
DPR-63 Licensee:
Nia ara Mohawk Power Cor oration 301 Plainfield Road S racuse New York 13212 Facility Name:
Nine Mile Point
Inspection At:
Oswe o
New York Inspection Conducted:
Januar 11-14 1988 zjz~/ae Inspectors:
W. Cook, Senior Resident Inspector date H. Kaplan, Reactor Engineer, RI date P. Eselgroth, Team Leader, RI date Approved by:
W. V. Johnston, Acting Director, DRS date Ins ection Summar
Ins ection on Januar 11-14 1988 Re ort No.
~/A~: Wih>>
h
19.U87f d
transient, the team performed system walkdowns; inspected available damaged components and in-progress repairs; interviewed maintenance and operations personnel; reivewed plant logs and equipment history records and preliminary event related pipe stress analysis wor k.
Results:
The team concluded that the licensee had established a comprehensive technical review effort for the event.
It was also apparent from records review that the feedwater system has a history of initiating unplanned challenges to the reactor protection system and thermal transients to the reactor and its components.
There are no violations and no unresolved items.
There are three licensee commitments in Section 6.0 of the repor r~
U. S.
NUCLEAR REGULATORY COMMISSION
REGION I
Report No.
50-220/88-02
Docket No.
50-220 License No.
DPR-63 Licensee:
Nia ara Mohawk Power Cor oration 301 Plainfield Road S racuse New York 13212 Facility Name:
Nine Mile Point
Inspection At:
Oswe o
New York Inspection Conducted:
Januar 11-14 1988 Inspectors SIGNED COPY ON FILE W. Cook, Senior Resident Inspector 2/25/88 date
. Kaplan, Reactor Engineer, RI gate SIGNED COPY ON FILE D. Persinko, Reactor Engineer, NRR 2/25/88 date P. Eselgroth earn Leader, RI date Approved by:
W. V. Joh ton, Acting Director, DRS date Ins ection Summar
Ins ection on Januar 11-14 1988 Re ort No.
50-220/88-02 Areas Ins ected:
With respect to the December 19, 1987 feedwater system transient, the team performed system walkdowns; inspected available damaged components and in-progress repairs; interviewed maintenance and operations personnel; reviewed plant logs and equipment history records and preliminary event related pipe stress analysis work.
Results:
The team concluded that the licensee had established a comprehensive technical review effort for the event.
It was also apparent from records reviewed that the feedwater system has a history of initiating unplanned challenges to the reactor protection system and thermal transients to the reactor and its components.
There are no violations and no unresolved items.
There are three licensee commitments in Section 6.0 of the repor TABLE OF CONTENTS 1.0 Introduction and Pur ose....
~Pa e
2.0 Executive Summar
~
F 1 2. 1 Summary of the Event and Damage 2 '
Licensee Preliminary Identification of Cause...
2.3 Summary of NRC Inspection Results......
~
~
~
~
~
~
~
~ 2
.3
'3.0 Se uence of Events..
..3 3. 1 Initial Conditions
.
3.2 Time and Event 3.3 Preliminary Plant Inspection by Shift Operators
~
~
~
~
~
~
..5 4.0 Ins ection Results 4.1 Feedwater System Inspection
.
Feedwater Pumps Flow Control Valves (FCV)
Pipe Hangers...,
System Welds 4.1.1 4.1.2 4.1.3 4.1.4 4.2 History of Feedwater System FCV Event.....
4.3 History of Feedwater FCV Stem/Plug Configuration 4.4 Licensee Initiated Analyses
..5
..6
~
s 7
..8
.....10
4.4.1 4.4.2 4.4.3 4.4,4 Metallurgical Failure Analysis of Damaged Components Piping System Stress Analysis........
Missing Part Inspection and Analysis
.
Safety Assessment of the Event
.
.....13
~
~
~
~
~ 13
.5,.0 Observations and Conclusions
5. 1 Safety Significance..
5.2 Licensee Technical Review....
5.3 Assessment of Past Engineering Efforts..
~
~
~
~
~
~ 14
.15 6.0 Exit Meetin and Licensee Commitments
.
.15
~Aendices A.
B.
Feedwater System Description and Normal Operations List of Damaged Hangers Summary History of Flow Control Valve Configurations Feedwater System Event History Persons Contacted
1.0 Introduction and Pur ose The purpose of this 3', day team inspection was to inspect damaged portions of plant systems following the December 19, 1987 feed system event and make an assessment of the licensee's followup.
In particular, this was intended as an initial assessment of the adequacy of the licensee's evaluation program associated with their determination of the cause and appropriate corrective actions for the feedwater system vibration problems that led to a plant scram.
The inspection team performed system walkdowns, inspected available damaged components and in-progress repairs, interviewed maintenance and operations personnel, reviewed plant logs and equipment history records, and reviewed preliminary event related pipe stress analysis work.
Section 5 of the report contains team observations and conclusions and section 6 delineates specific licensee commitments to the NRC.
The Appendices contain some feedwater system descriptive information and a
partial history of plant scram events associated with the feedwater system.
Section 2 provides an Executive Summary.
2.0 Executive Summar 2. 1 Summar of the Event and Dama e
During the first half of swing shift on December 19, 1987 with Nine Nile Point One operating at approximately 98% of full power, the plant underwent a transient that was apparently initiated by the reactor feedwater system.
The transient spanned a time interval of about 3 to 4 minutes and was terminated by a reactor scram.
At approximately 1811 hours0.021 days <br />0.503 hours <br />0.00299 weeks <br />6.890855e-4 months <br />, with full feedwater flow being supplied by the combination of the unit high pressure turbine driven feed pump and one motor driven feed pump, various turbine driven feed pump water and lube oil alarms were received followed by feedwater heater level alarms.
An operator sent into the plant from the control room observed excessive vibration levels at the turbine pump feed control valve station and called the control room to recommend a reduction in reactor power and securing of the turbine driven feed pump.
At that point, control room operators took manual control of the main feed control valves and began reducing reactor power.
Shortly, thereafter, vibra-tions from the reactor feedwater system were felt in the control room and the reactor was scrammed at approximately 1814 hours0.021 days <br />0.504 hours <br />0.003 weeks <br />6.90227e-4 months <br />.
Plant response to the scram was norma k
Inspection of the feedwater system following the above event by the licensee and resident inspector revealed a number of damaged areas, as well and defects which may have existed prior to December 19, 1987.
The following is a
summary list of the post event system inspection observations:
One of the two feed control valves (¹13A) associated with the turbine driven feedwater pump (¹13)
had a valve actuator stem separated from its flow control plug.
The ¹13 turbine feedwater (FW) pump outboard bearing babbit material was wiped.
A triangular (1" x 2") piece was missing from the leading edge of one of the impeller vanes on the suction side.
One snubber near the ¹13 flow control valve station was found to have a significant (about 30') bend in the piston rod.
Sixteen other hanger defects involving bent and broken brackets, sheared bolgs etc.
were also observed and may have predated the subject event.
One of the motor driven FW pumps (¹ll) had through wall cracks on the suction side and indications on the discharge.
Three FW piping welds near the ¹13 feed control valve station and bent snubber had crack indications.
2.2 Licensee Preliminar Identification of Cause The licensee indicated to the team that their preliminary analysis suggests that the transient was initiated by a failure of the plug/stem assembly on flow control valve FCV-13A.
The licensee sent the ¹13A FCV stem and plug to Battelle Pacific North-west Laboratories for analysis to determine the failure mechanism.
During the inspection, the licensee informed the team that the pre-liminary feedback received from Battelle indicated that the stem/plug fracture was due to reverse bending fatique and that two points 180 apart on the fracture surface showed a "thumbnail" fatigue pattern.
The licensee also stated that Battelle had indicated to them that the failure did not show signs of the stem and plug being separated by a final jolt.
This would tend to support the theory of the stem/plug separation being the event initiator rather than, for example, a piece of the feed pump impeller breaking off and interfering with ¹13FCV operations.
A licensee consultant has identified vortex shedding around the flow control plug as the probable source of the fatique bending forces.
The licensee consider s that random motion of the severed valve plug inside of the valve body caused intermittent pressure/flow surges that could have resulted in the observed vibrations of the feedwater pipin.3 Summar of NRC Ins ection Results The inspection team's findings and conclusions are summarized as follows:
~
The licensee's approach to recovery.from the December 19, 1987 feedwater system transient and resultant reactor plant scram incorporates a comprehensive engineering review effort.
This effort includes detailed system inspections, the use of engineering failure analysis of failed components and a program for engineering assessment of cause and corrective action.
The licensee had committed to provide the NRC with a copy of their formal overall report on this event by Narch 1, 1988.
~
From the team's inspection of the plant, interviews of personnel and review of records there are no violations of requirements.
The team identified no basis for differing with the licensee's preliminary assessment of event cause.
The team notes that the history of failures of various plug/stem designs used in the 813 flow control valve, the known vibration characteristics of the feedwater system over the life of the plant and the December 19, 1987 event define an engineering problem that has been grappled with in the past but not as yet solved.
At'he conclusion of the exit meeting the team leader stated that the NRC will look to the final engineering analysis of the December 19, 1987 event and resultant corrective actions, for a technical basis for concluding that the level of confidence can be raised with respect to the avoidance of unplanned transients imposed on the reactor plant by the feedwater system.
3.0 Se uence of Events 3.1 Initial Conditions During the evening of December 19, 1987, the unit was operating at near full power (98%).
Four recirculation pumps were in operation with total recirculation flow equal to 67.4 x 10'bm/hr.
Reactor pressure and water level were nominal at 1020 psig and 74 inches, respectively.
No plant evolutions or surveillance testing were in progres.2 Time and Event 1811
¹13 feedwater (FW) pump seal water low differential pressure alarms in and out.
1812 FW Heater high level alarms and ¹13 FW pump auxiliaries alarm; fluctuating reactor water level observed.
Operator dispatched from control room to investigate the feedwater system problems'pon entering the turbine building the operator observed minor feedwater piping vibration of the piping in the overhead of elev.
277'he operator quickly ran to the ¹13 FW pump location and observed the pump vibrating.
The operator then proceeded to the ¹13 flow control valve station to see if their stems were cycling.
The operator could not determine if the stems were cycling because the valves and the associated piping were moving too much.
The operator contacted the control room and recommended a rapid reduction in power and tripping of the ¹13 FW pump.
1813 The Chief Shift Operator took manual control of feedwater control valves and the control room 'E'perator commenced decreasing recirculation flow to reduce reactor power.
1813.30 Fire alarms for the turbine oil reservoir room and the turbine stop valve area sounded.
Control room operators felt high vibration via the control room floor.
1813:56 Control room operators manually scrammed the reactor and disengaged the ¹13 feedwater pump clutch.
1814 Reactor water level dropped to approximately 53 inches (due to shrink in the annulus region) resulting in a reactor vessel low level scram signal and a shift of the
¹11 and ¹12 motor driven feedwater pump to the High Pressure Coolant Injection (HPCI) mode of operation.
The reactor scram caused an automatic turbine trip and the generator tripped on reverse power, as designed.
1816-Reactor water level was restored and both motor driven feedwater pumps tripped at a reactor vessel level of 95 inches, as designe Operators stabilized the unit in the hot shutdown condition without further incident.
1818
Operators reset the reactor scram.
1823 Plant personnel secured the fire alarms after investigation identified no fires in the turbine building.
The fire alarms were determined to have been caused by the asbestos dust triggering the fire detection systems (smoke detectors).
3.3 Preliminar Plant Ins ection B
Shift 0 erators Following the reactor scram, plant parameters quickly stabilized and operators conducted a preliminary inspection of the feedwater system.
The first visible sign of damage due to piping vibration was leakage from one of the feedwater flow venturi flanges
~
Some portions of the feedwater piping insulation had been shaken from the piping and lay on the turbine floor.
Clouds of asbestos dust prevented personnel entry into some areas for close inspection of piping, hqwever, no other visible indications of piping damage was immediately detected.
4.0 Ins ection Results 4. 1 Feedwater-S stem Ins ections 4.1. 1 Feedwater FW Pum s
¹13 FW Pump Following the event, the licensee disassembled and inspected the internals of the ¹13 FW pump and the visible portions of the suction and discharge piping internal surfaces.
The licensee found that a 1" by 2" triangular shaped piece was missing from the suction side leading edge of one of the pump impeller vanes.
The licensee also inspected the pump bearings and found that the
¹13 FW outboard bearing babbit material had been wiped on the lower half of the bearing.
The inspector examined the disassembled
¹13 FW pump and visually examined the exposed portions of the suction and discharge piping internal surfaces.
The break surface on the pump impeller suction vane where the 1" by 2" triangular piece had broken off was observed to have the same coloration as the surrounding impeller vane surfaces that are normally exposed.
It was therefore, difficult to ascertain by visual examination alone whether the impeller piece had broken off during the Oecember 19, 1987 event or at some other time.
An inspection of the pump impeller, barrel volute and piping internals did not reveal any conclusive indica-tions of how the impeller piece may have travelled after breaking off. It was noted that the pump impeller inlet (suction)
vanes
all showed signs of erosion on the front side of the vanes.
The licensee has indicated that this pump is susceptible to inlet eye recirculation at partial loading (possibly around 1/3 to 1/2 of full flow), and that this is one of the reasons for the
year replacement cycle for this pumps internals including the impeller.
The barrel volute also showed signs of high velocity cutting erosion at the outer edges of the discharge zone.
The extent of the above described instances of pump internals erosion did not appear to be significant considering the established 2 year replacement cycle for the pump internals.
¹11 and ¹12 FW Pumps Following the event, the licensee inspected the ¹11 and ¹12 FW pump bearings and found no problems with the bearings.
However, during the inspection of the ¹ll FW pump, the licensee found water weeping from the suction inlet line to the pump where this line meets the pump inlet nozzle.
By visual inspection, four indications were found on the east side of the pump casing to suction, line weld.
The licensee then performed dye penetrant inspections (PT).
Three of these indications were found to be through wall cracks.
Internal rejectable indications were also found on the west side and top of the pump casing to suction line weld.
The east side was excavated and repaired, as were the top and west areas.
Internal and external PTs were performed on the weld repairs and the pipe wall was ultrasonicly tested (UT) for thickness followed by radiographic inspection (RT) of the repair areas.
The licensee determined these repairs to be acceptable as these examinations revealed no rejectable indications.
Rejectable linear indications were also found on the ¹11 FW pump discharge at the pump casing to nozzle weld.
These indications were repaired by grind out and accepted.
Similar inspections and examinations were performed on the ¹12 FW pump and no problems were found.
The NRC inspector reviewed the records of the licensee's repair work, including the RTs and concurred with the licensee's conclusions.
4.1.2 Flow Control Valves
¹13 Flow Control Valves (FCV)
During the event, the flow control station consisting of the two FCVs (¹13A and ¹13B) was observed by an operator to be vibrating heavily, Disassembly and inspection of the two valves revealed that on ¹13A, the stem had separated just above the welded junction connecting the stem to the valve plug.
Visual inspection and nondestructive examination of the valve plug/stem assembly on FCV 13B revealed no sign of unusual wear or damage.
At the time of the NRC inspection, the ¹13A valve stem/plug assembly had been shipped off site for failure analysis.
It was noted in a review of pictures at the site of the failed assembly that the fillet weld between the stem and plug for the ¹13A valve was significantly smaller (about 1/8 inch or less)
than that observed
on the ¹13B valve (about 1/4 inch).
This assembly design consists of screwed attachment of the stem to the plug followed by the fillet weld between the stem and plug.
¹ll and ¹12 Flow Control Valves The licensee disassembled and inspected the ¹11 and ¹12 FCVs and found only indications of normal wear for valve internals.
The valves were reassembled as is.
H The licensee performed a detailed inspection of the hangers in the following system piping:
System ¹
Extraction steam to 4th FW heater 49 - Condenser to condensate pumps suction 51 Suction side condensate to FW pumps 29 -
FW pumps to 5th stage FW heater 30 -
FW heaters to contain. isol. valve
Containment isolation valve to Rx vessel
- Note only system ¹31 is safety related.
The licensee inspected 331 hangers, of which 264 were found to be satisfactory without the need for further action.
The remaining hangers were dispositioned as follows:
40 hangers were found acceptable with repairs such as tighten-ing bolts; replacing sheared bolts; tightening locknuts; and tightening clamps.
30 items required engineering disposition for things such as questionable load settings on snubbers; unusual hanger movement; bent or broken brackets; and bent snubber.
of these turned out to be respectable and required some form of rework.
Note:
The above numbers total 334 not 331, because 3 hangers had more than one item wrong.
Appendix B lists the specific hangers that were recorded as damaged.
In most cases the damage observed was minor and could have occurred at most any time.
The only damaged hanger on a safety related section of piping is 31-H5A feed piping in containment.
This involved a broken spring that showed signs of rust on the broken surfaces indicating a break at
some earlier time.
However, hangers 29-HS-11 and 29-HS-16 appeared to have damage that probably was caused during the December 19, 1987 event.
For example, the 29-HS-11 snubber rod had a significant bend in the approximately one inch diameter rod.
'l..l
~W14 As a part of their investigation of possible damage from the December 19, 1987 event, the licensee inspected twenty-one system welds and identified three with indications of cracking.
More specifically, the following welds were inspected and found to be acceptable:
FW1, SW44A, FW2, SW46C, SW46A, FW58, FW61, 3" Reheat drain tank connection, FW57, FW57B, FW9, FW10, SW ID-11B-I, SWID11B-O, SWID11A-I, SWID.11A-O, FW63, FW66.
The following welds were found to have indications which were dispositioned by grind out:
FW15('," linear (HT)),
FW16 (2 indications (NT)),
FW17 (4 indications (NT)).
All three of these welds are located in the vicinity of the ¹13FCVs and the 29-HS-ll bent snubber.
The team reviewed the licensee's records of this inspection and the three Nonconformance Report (NCR) disposition and had no additional questions.
The team reviewed the history of the weld repair of the leaking ¹ll (non-safety related)
feedwater pump suction line.
Leakage occurred in a butt weld, joining an 8" Sch 40 (.322" W) steel elbow to a
5% chrominum (Cr) cast alloy steel nozzle.
Leakage reportedly occurred from two 3/8" long linear indications oriented transverse to the weld.
The nozzle is integrally cast with the feed pump casing.
The feed pump was manufactured by Worthington Pump in 1968.
Welding of the suction and discharge piping was reportedly performed by Worthington in accordance with B31. 1.
Prior to initiating the repair the licensee consulted with Worthington Pump who indicated that to the best of their knowledge the subject weld had experienced extensive repairs in 1968 using type 309 stainless filler metal and recommended same for repair, No fabrication records were available for review.
On the basis of this information the licensee qualified welding procedure WPS-1-5-BA-102 and welders utilizing the Tungsten Inert Gas (TIG) process.
The test assembly consisted of SA 106 GR B and SA 335 GR P5 pipe welded with type 309 filler metal.
The welding procedure and welder qualification records were reviewed and found to comply with ASME Section IX requirements.
In addition to the area containing the leaks two other areas were repaired because of radiographic and penetrant indications.
Specific instructions (¹100228)
were issued to require removal of a boat sample to include the defects for metallurgical evaluation by
Battelle Northwest prior to repair.
To protect the 5% Cr nozzle from heat affected zone cracking and to avoid having to stress relieve, the instruction specified a preheat of 100'F 150'F, and a temper bead technique (removal of ~ of the first layer before depositing the next layer) when overlaying the casting side of the weld cavity.
The licensee reported that prior to filling up the cavity magnetic checks of the exposed walls revealed the presence of a non-magnetic material thus corroborating Worthington's statement that type 309 stainless had been used in welding the subject joint.
The presence of the existing stainless steel would act as a buffer layer and provide additional protection to the 5% Cr nozzle.
The completed weld was subjected to visual and liquid penetrant inspection.
Radiography was also performed for information purposes.
The inspector reviewed the radiographs and found no evidence of discernable indications.
The inspector visually inspected the subject weld and found the entire weld surface to be smooth ground.
Ultrasonic measurements of the com-pleted weld indicated some possible evidence of erosion/corrosion.
A minimum thickness of.22 on the elbow side of the weld was observed.
The specified minimum design wall thickness was
. 170" with an additional
.088" specified for corrosion allowance.
The licensee indicated that this area will be continuously monitored for wall thinning.
The results of the metallurgical investigation of the boat sample may provide evidence to determine the root cause of the leakage.
The licensee reported that inspection of feedwater pumps ¹12 and ¹13 did not disclose any condition requiring repair.
4.2 Histor of Feedwater S stem FCV Related Events The licensee had experienced several problems with the feedwater system since initial power operations.
Due to the apparent role of the ¹13 Flow Control Valve (FCV) in the December 19, 1987 event and the fact that the ¹11 and ¹12 FCVs for the motor driven FW pumps are the same basic design, the team was particularly interested in the history of these valves in connection with unplanned thermal transient and trips of the reactor plant.
The team assembled the following system history via discussion with licensee representatives and records review:
1969-70 - The licensee installed additional piping supports and restraints in the vicinity of the 13A and 13B feedwater control valves because of excessive flow-induced piping vibration.
In addition, vibration dampeners were installed on the stem of the control valves inside the valve actuator.
ll/75 Reactor scram due to loss of control air to the ¹11 and ¹12 feedwater control valves.
12/75 licensee replaced internals of the 13A and 13B feedwater (FW)
control valve /78 Reactor scram due to 13A FW control valve stem unscrewing from plug.
The stem and plug were pinned to prevent the stem from backing out of plugs 1/78 After subsequent startup, and shutdown, 13A FW control valve stem snubber pin was found broken and 13B FW control valve stem was found broken from plug, Feedwater control valves were repaired by welding the stem to the plug by recommendation of the vendor.
8/85 - Reactor scram (low reactor vessel level) due to failure of
¹11 FW control valve.
Problem identified as a broken range spring and lock screw in the valve positioner.
12/7/87 Reactor scram ( low reactor vessel level) due to malfunction of 13A and 13B FW control valves.
Operators attempted to manually override the 13B FW control valve due to excessive hunting of the valve.
The 13A FW control valve failed to respond to lt,vel demand after the 13B valve was overridden.
Licensee speculated the cause of the 13A FW control valve failure to respond to the position signal was because of dirt in the control air lines resulting in binding of the valve actuator.
12/19/87 Reactor scram (low reactor vessel level) apparently initiated by ¹13A FCV stem/plug failure.
NOTE:
The inspector determined that the licensee has been replacing the ¹13 FW pump impellars every refueling outage because of impellar blading cavitation and erosion.
Similarly, the licensee has replaced the 13A and 13B FW control valve stem and plug and cage assemblies every refueling outage because of normal operating wear during the cycle.
Appendix D to this report contains a partial history of plant scram events initiated by various parts of the feedwater system.
4.3 Histor of Feedwater Flow Control Valve Stem/Plu Confi urations
~
The original stem/plug design in the flow control valves consisted of a stem screwed into the plug.
In January 1978 while at full power, the 13A flow control valve stem unscrewed from the plug.
The reactor was shutdown for repairs (the licensee believes the stem was probably screwed back into the plug and pinned).
Following these repairs, startup commenced.
At approximately 200 NWe with the ll and 12 motor driven feedwater pumps on line, the 13 feedwater pump began to come on line also.
As flow increased from 2 to 2.25 million ¹/hr, vibration was experienced and the 13 feedwater pump was taken out of service.
Upon inspection, a broken pin was found in a stem motion snubber within FCV 13A; the pin was replace Upon completion of repairs, the ¹13 feedwater pump was again brought on line and again vibration was experienced as flow went from 2 to 2.25 million ¹/hr.
The ¹13 feedwater pump was shut down and inspections showed that the stem had sheared from the plug in the ¹13B flow control valve.
A new stem was fabricated at Nine Mile out of 304SS bar stock.
The licensee believes the design was the same as the original design.
By letter dated June 12, 1978, the valve manufacturer, Fisher, recommended that valve stems be welded to the plugs.
Fisher performed these modifications and the licensee believes these modified designs were installed during the first outage after June 1978.
This design was in effect during the feedwater transient an December 19, 1987.
Following startup from the present outage, the stem/plug will be replaced with a new design.
This new design utilizes a 1" shaft which tapers to a 2" screwed connection to the plug.
The stem is pinned to the plug to prevent unscrewing.
The stem material in the new design is 17-4 PH instead of 304SS.
Upon removal of the 13A and 13B flow control valves during this present outage, the inspector noted via first hand inspection and later via pictures that the fillet weld attach-ing the 13B FCV stem to plug was much larger than the fillet weld which sheared in the 13A FCV.
Since these fillet welds were performed by Fisher, quality control at Fisher during fabrication of these stem/plug components appears to have been inadequate.
The licensee could not supply the history of the 11FW and 12FW FCV designs to the team.
However, the licensee has stated that the stem/plug designs currently installed in the 11 and 12 feedwater flow control valves incorporate screwed and welded stem/plug configurations'hese stems/plugs were removed and inspected during the present outage and no cracks were found.
They were reinstalled.
On 12/7/87, reactor scrammed from 96% power on low level.
The cause was a malfunction of ¹13A 8 ¹13B FCVs.
The licensee attempted to manually override ¹13B FCV but could not prevent reactor trip because
¹13A FCV did not respond.
HPCI restored level.
The licensee concluded that dirt in the lines caused binding of the actuator.
Licensee does not believe this event is related to the present FCV failur.4 Licensee Initiated Anal ses Subsequent to the December 19, 1987 event, the licensee initiated the following analyses:
Preliminar Determination of Event Cause The licensee indicated to the team that their preliminary analysis suggests that the transient was initiated by a failure of the valve plug/stem assembly on FCV-13A.
The licensee informed the team that their consultant's preliminary review of normal flow control valve operation results in vibration of the valve plug along the same axis as the flow due to vortex shedding.
This axis which is at right angles to the movement of the plug/stem arrangement for flow control purposes.
The consultant informed the licensee that this vortex shedding induced vibration of the plug would result in a bending fatigue stressing of the plug/stem connection.
The licensee considers that random motion of the severed valve plug inside of the valve body caused intermittent pressure/flow surges that could have resulted in the observed vibrations of the feedwater piping.
The licensee also stated that a dynamic analysis by their engineering consultant firm has been initiated to determine the capability of this assumed event cause to develop the magnitude of piping system loads, necessary to produce the observed bend in the snubber (29-HS-11) rod.
4.4. 1 Metallur ical Failure Anal si s of Dama ed Com onents
".At the time of the team's inspection, the licensee had already sent the ¹13A FCV stem and plug to Battelle Pacific Northwest Laboratories for analysis to determine the failure mechanism.
During the inspection, the team examined photographs of the stem/plug failure.
The licensee informed the team that the preliminary feedback received from Battelle indicated that the stem/plug fracture was due to reverse bending and that two points 180 apart on the fracture surface showed a "thumbnail" fatigue patterns The licensee also stated that Battelle had indicated to them that the failure did not show signs of the stem and plug being separated by a final jolt.
This would tend to support the theory of the stem/plug separation as being the event initiato The licensee also sent a metallurgical
"boat" sample from the
¹11 FW pump cracked suction piping area to Battelle for analysis.
At the time of the inspection, the licensee planned to send the
¹13 FW pump impeller that has a piece missing from one of the vanes, to Battelle for analysis too.
4.4.2 Pi in S stem Stress Anal sis The team reviewed a preliminary piping system stress analysis being conducted by the licensee at their corporate engineering office.
The purpose of this stress analysis was to determine where the highest piping system loads and stresses may have occurred.
The licensee's analysi.s uses a sinusoidal forcing function (input at the elbow nearest to the flow control valves) with a frequency which matches the fundamental frequency of the piping system and with an amplitude which will cause the pipe hangers to be loaded to their full design capacity.
The team found an error in their analysis which may have resulted in too high damping factors (which will therefore underestimate the loads).
The licensee will re-run this analysis incorporating this correction.
4.4.3 Missin Part Ins ections and Anal sis As previously indicated, a triangular-shaped piece of metal about 2 square inches in area was identified as missing from one of the impeller vanes.
Inspections were made by the licensee in the pump casing, the accessible portions of the pump suction and discharge piping, and the valve bodies of FCV 13A and 13B in an attempt to locate the missing piece.
The internals of the flow check valve immediately upstream of the FCV's was also inspected.
No evidence of the missing piece was found.
A preliminary loose parts analysis concluded that it would not adversely effect any reactor structural components.
The licensee indicated that previous loose parts analysis performed for parts coming from upstream of the feedwater heaters demonstrated that no adverse effects would occur to any reactor components considering the diameter of the feedwater heater tubes.
However, the licensee did not have the missing parts analysis available for review at the time of the inspection.
4.4.4 Safet Assessment of the Event The licensee has indicated that their safety assessment of the December 19, 1987 feedwater system transient and subsequent plant shutdown is as follows:
~
There were no adverse safety consequences associated with this event nor was the reactor in an unsafe condition at any time.
The high vibration experienced in the feedwater system was terminated without adverse consequences to any system requi red to shutdown the reactor.
All operator-initiated and automatic safety systems operated as designe ~
A review of the potential safety consequences resulting from this event examines the case of the feedwater piping vibration continuing until the weakest point within the boundaries of the piping effected by the transient failed mechanically.
Preliminary analysis of a computer-generated piping model of the NNP1 feedwater system indicates that all high stress points experienced during the transient were located in the Turbine Building, outside of primary containment.
Field inspections of the feedwater system piping and supports revealed no evidence of pipe vibration inside of primary containment that could be determined to be a result of the December 19, 1987 feed system transient.
Therefore, the event can be evaluated from the perspective of a postulated failure of a high energy piping system outside of primary containment.
This event was analyzed in the NMP1 Final Safety Analysis Report (Updated),
Section XVI, 2.0, "Plant Design for protection Against Postulated Piping Failures in High Energy Lines."
Subsection 2.2 analyzed the fai lure of each high energy piping system at any point outside of primary containment and concluded that safe shutdown of the reactor can be accomplished and the unit could be maintained in the shutdown condition.
In addition, a more detailed analysis of a high pressure feedwater pipe rupture also concluded that a failure at any point outside of primary containment would not affect the safe shutdown capability of the reactor.
Therefore, the potential safety consequences of this feedwater transient would not have placed the reactor in a condition not previously analyzed.
The team found no basis for taking exception to these conclusions.
5.0 Observations and Conclusions The inspection team has the following observations and conclusions:
5. 1 Safet Si nificance With respect to plant safety, the December 19, 1987 event is considered to represent an undesirable challenge to the reactor protection system and thermal transient for reactor plant components.
5.2 Licensee Technical Review The licensee appears to have implemented an effective technical review effort associated with the feedwater system transient event to address the following areas:
Inspection, documentation and repair of damaged portions of the feedwater syste To have failure analyses conducted on appropriate portions of the feedwater system found damaged during the system inspections sub-sequent to the event.
Engineering analysis of the event cause and probable system loads imposed by the event to support the evaluation of long term adequacy of any design changes to be implemented as corrective actions.
5.3 Assessment of Post En ineerin Efforts With respect to the ¹13 Flow Control Valves (AKB), the team notes that there has been a history of instances where these valves have been involved in unplanned transients for the reactor plant.
The team sees the December 19, 1987 event as yet another data point in this history of occurrences and apparently the must significant to date.
Of par'ticular interest is the history of in-service difficulties experienced with the FLV plug and stem arrangement.
The team recognizes that the licensee has taken a number of steps in the past to eliminate the FCVs as a source of unplanned plant transients including replacement of valve internals on a two year cycle.
However, the events of December 9,
1987 demonstrate that the trouble free service life of the value parts is still shorter than the change out interval.
6.0 Exit Meetin and Licensee Commitments At the conclusion of the inspection the team met with the licensee repre-sentatives denoted in Appendix E.
The team leader reviewed the section
observations and conclusions with licensee management at that time.
At the exit meeting, the following licensee commitments were confirmed:
Before startup of NMP1 the NRC will be provided with a copy of the licensee's basis for concluding that the unretrieved 1"x2" triangular section of the ¹13FW pump impeller observed to be dislodged does not constitute a problem for safety related aspects of reactor operation.
P Before startup of NMPl, the results of licensee FW system inspection and repairs conducted since the December 19, 1987 event and the FW system status for startup will be reviewed and approved by the Station Operations Review Committee (SORC).
The resident inspector will be provided with a copy of this SORC package.
By March 1, 1988, the NRC will be provided with a copy of the licensee's formal engineering evaluation of the December 19, 1987 event cause, resultant system loadings from this assessment of cause and the corrective actions to be taken.
In particular, the NRC will be provided with the associated engineering analyse At the conclusion of the exit meeting the team leader stated that the NRC will look to the final engineering analysis of the December 19, 1987 event and resultant corrective actions, for a technical basis for concluding that the level of confidence can be raised with respect to the avoidance of unplanned transients imposed on the reactor plant by the feedwater system.
The team leader and the licensee discussed the contents of this inspection report to ascertain that it will not contain any proprietary information and therefore can be placed in the Public Document Room without prior licensee review for proprietary information (10 CFR 2.790).
No written material was provided to the licensee by the tea AppendixA Feedwater S stem Descri tion and Normal 0 eration The feedwater system consists of three motor driven, 50% capacity horizontal centrifugal feedwater booster pumps and three horizontal centrifugal reactor feedwater pumps.
The feedwater booster pumps are supplied by three 50% capacity condensate pumps and discharge to three parallel strings of low pressure feedwater heaters.
The discharge of the low pressure feedwater heaters flows to the reactor feedwater pumps.
One feedwater pump (¹13) is rated at 6,400,000 ibm/hr and is driven from the shaft of the units'igh pressure main turbine through a quick disconnect clutch and step-up gear.
The other two reactor feedwater pumps (¹11 and ¹12) are each rated at 1,900,000 lb/hr and are electric motor driven through step-up gears.
The feedwater pumps discharge through flow.control valves, one per each motor-driven pump and two parallel control valves for the turbine driven feedwater pump.
The ¹13 pump valves are a nominal 10" body, 8" trim with a 1" valve stem.
¹11 and ¹12 pump valves are an 8" body, 6" trim, also with a 1" valve stem.
The discharge of the reactor feedwater pumps flows through three parallel high pressure feedwater heaters to the reactor.
Typical low power operations involve use of the motor-driven feedwater pumps up to power levels of approximately 240 MItle (30-35%).
At this power level, the turbine driven feedwater pump is engaged and feed flow is shifted from the motor driven feedwater pumps.
The turbine driven feedwater pump carrys the entire feedwater flow requirements up to approximately 85% power.
Above approximately 85% power, a single motor-driven feedwater pump is run in parallel with the shaft-driven pump to'ull reactor power (610 MMe and 7,290,000 ibm/hr feedwater flow).
Typically the turbine driven feedwater pump control valves (13A and 13B)
are operated in automatic and the motor-driven pump control valve is in manua ~y0 p g;OVL
$0~~ -~m VIVI.VwrIILTILVI Il<<TLN.u. rlraellr VN III Otc tc
'I IIIhll>
WL4 N ~ II IIATC Vgh LIalV
+if'c@ C
~~CL ALL VN ~ 11 SV t
VN Sll VN Illsv
~ Ll tcv LLIL NCATCN CC1 STCAINCC d cia QVICC CLCN VNCSSLNCI TLNdsl LV
~ to II I
t ~
VIIIllSl fCV Ird I h Ltd INATCR tllIll CV tllIllCV
~N ~ 12 lv VCCCNATCC tWVS IN0ION 0CIVCNI LLL NCATCI lac NCATCA Lac NCATXN 112 Cllct CCt CE.
ll RIOT Lll eKPt ONS NTN tCV 11 ICIIST SCV
Lrrtcc IllIcr0t OXC. NTR tCV ll ICIIST SCV 00 TC tc tc 0)2 VLLcv ~ 11 NV FIQSIR I
CTACIQN FCLNNATtc STYLI Ol lll LNLIST SI2'0 TC Ill ttAtf LLrsl. Lrtls IL'i fcIXALATCNIrhbTLN VIV\\V
~
I<<A IN I ~ ~ >II.N I<<ANI ~ L~ N>AL DNAIN LLLNCN I'Irrr Lirrrlt.hvl'IIT!.TIN gePf
~Han er
~Dama e
~Aendix B
29-A1 29-HS
29-HS-16 29HS2 29HS3 29HS4 29HS5 29SC2 29MKY 29R4C 29SC100 29SCR3C 29SCR3D 31H5A 51MKA2 51SC101 51SC35 One bolt sheared off Bent rod on snubber; exam on lug welds unacceptable Bent I beam where snubber attaches; 2 cracked welds Bent/broken bracket Bent/broken bracket Bent/broken bracket Bent/broken bracket Base plate pulled from all ~"
Concrete broken/missing Broken rod at ceiling Broken angle iron Embedded plate pulled out '," from concrete Pulled from wall 2"; one T is bent; one T weld cracked Broken spring 1 bolt sheared off sheared bolt Both sway braces bent
~Aeedix C
Summar Histor of Flow Control Valve Confi urations Time Period Stem/Plu Confi uration Remarks 1969-1/78 1/78 1/78 Stem screwed into plug. Whether or not the stem and plug were also pinned during this entire period is unclear.
During this month, the ¹13FCVs were apparently operated with the stem and plug screwed together
& pinned.
Subsequent to above problem, a
new stem material was introduced (304 stainless steel)
1/78 ¹13A FCV stem unscrewed from plug at full power.
Stem and plug were apparently rescrewed and pinned.
¹13B FCV stem was found broken from plug after plant shutdown due to feed system vibrations.
Also, the ¹13A FCV internal snubber was found unpinned.
Fabrication work done at Nine Mile Point.
6/78
¹13FCV stems were welded to the plugs.
Manufacturer's recom-installed first outage after 6/78.
mendation.
Fabrication done at manufacturer.
12/87 Configuration in use was unchanged from 1978.
¹13A FCV stem found broken from plug. Stem/
plug weld on ¹13A noted to be 1/8" fillet or less compared to 4" fillet on ¹13B.
1/88 Licensee plans to install new design stem/plug screwed
& pinned arrangement.
New design recommended by manufacturer.
Stem diameter increased at plug connection point.
NOTE:
The ¹13 pump valves are equipped with a hydraulic snubber (directly in line between the valve stem and pneumatic operating cylinder).
¹11
& ¹12 valves are not equipped with this hydraulic snubber.
The hydraulic snubbers were added to the ¹13 valves back in 1971'as a
means of stabilizing valve plug travel due to flow induced vibration.
The addition of the snubber imposes additional stem loading.
The valve performance with the snubbers installed has been acceptable and the feedwater piping vibration was greatly diminished.
NOTE:
Discussion with the licensee indicated that the ¹11 and ¹12 FCV configurations have remained unchanged from plant startu AppendixD FEEDWATER SYSTEM EVENT HISTORY This Appendix contains a partial history of plant scrams caused by various subsystems of the feedwater system and related components.
Date CONDENSER VACUUM Event 6/15/73 Low vacuum scram caused by failure of relief valve in SJAE system Date DEMIN BLOCK VALVES Event 6/14/72 Reactor trip on low water level at about 25% power rolling turbine
.at 800 rpm.
Feedwater flow to reactor was lost due to closure of blocking valves on all condensate demineralizers..
FLOW CONTROL VALVES (¹13A & B)
1/20/78 Reactor scram due to ¹13 FW control valve stem unscrewing from plug.
Possible cause; dirt in control air lines.
12/19/87 Reactor scram apparently due to ¹13A FCV stem/plug failure.
Flow Control Valves
¹11 and ¹12 11/28/75 Scram due to loss of control air to the motor driven feed pump control valve 8/23/85 Reactor scram initiated due to reactor low water levels
¹11 Feedwater pump flow control valve closed.
The problem was found to be a detached range spring and lock screw in the valve positioner.
12/15/85 Scram due to ¹ll & ¹12 FCVs closing in the open position causing high Rx level.
¹13 compressor taken out of service causing loss of air to valve Hi h
Rx Level - Root Cause Unclear 1/12/75 Reactor scram on high reactor water level.
Average to flow summer failure.
Scram occurred on ascension.
Ramp slowed by ¹13 feedpump flow transmitter problems and low suction pressure to feedwater booster pumps due to deficiency in the condensate demineralizer s.
Low Rx Level Root Cause Unclear 7/5/81 Reactor scram on low water levels 3/4/85 False reactor low level signal caused scram.
3/4/72 Reactor trip on low water level when placing motor driven feedpump in service.
Mode switch misaligned causing operating error.
7/22/72 Scram occurred during transfer of feedwater level columns in preparation for surveillance testing.
Transfer to manual mode
&
disagreement between level columns contributed to low level condition and scram.
FEEDWATER CONTROL SYSTEM FW MASTER LEVEL CONTROLLER Date Event 2/28/72 Reactor trip on low level caused by feedwater control disturbance following loss of 162 MG set in continuous power supply.
Low rated fuse blown in D.C. control to MG set.
TURBINE DRIVEN FEED PUMP (¹13)
PUMP Date Event 12/21/74 A reactor scram occurred due to improper range switching of the IRMS.
Purpose of shutdown was to repair excessive drywall leakage and to repair ¹13 FW pump and FC valve vibration pump impeller was replaced due to excessive cavitation wear caused by low feedwater suction pressure which was caused by plugged condensate demin.
bottom lateral APPENDIX E PERSONS CONTACTED 1.
Nia ara Mohawk Power Com an
"T. J. Perkins, General Superintendent
"T. Roman, NMP1 Station Superintendent
- R. A. Cushman, Task Manager.,
FW System
- L. Wolf, Site Licensing Engineer
- L. A. Klosowski, Nuclear Design
"W. Drews, Technical Superintendent
"C.
G.
Beckham, Manager QA Operations
- T. Gurdziel, Incident Analysis Coordinator W. Connolly, QA Program Manager R. L. Tessier, Planning Coordinator F. A. Hawksley, ISI Superintendent, Nuclear R. Shelton, ISI Dept. Supervisor NMPl 2.
New York State
"P.
D. Eddy, Site Representative 3.
Nuclear Re ulator Commission
- P.
W. Eselgroth, Acting Tech nical Assistant, DRS, RI
- W. Cook, Sensor Resident Inspector, RI
- J. Johnson, Projects Section Chief, DRP, RI
"D. Per sinko, Operations Engineer, NRR
- W. L. Schmidt, Resident Ins'pector, RI
- denotes those in attendance a January 14, 1988 exit meetin '
POST INSPECTION SALP DATA SHEET 1.2111:15115115112.1:~255 3. Docket No./Report No.: 50-220/88-02 4. Inspection Dates:
1/11-14/88 5.2
15":~Ei
6.
5
55,2 2):
7.
Inspection Hours for this Functional Area:
for Team 8.
~SALP In ut The licensee's approach to recovery from the December 19, 1987 feedwater system transient and resultant reactor plant scram incorporated a
comprehensive engineering review effort.
This effort included detailed system inspections, the use of engineering failure analysis of failed components and a program for engineering assessment of cause and corrective action.
However, the history of failures of various plug/stem designs, the known vibration characteristics of the feedwater. system over the life of the plant and the December 19, 1987 event define an engineering problem that
'as been grappled with in the past but not as yet successfully solved.
9.
Note to SALP Coordinator:
At the time of this inspection and SALP Input, it was apparent that the licensee had an engineering effort in place that appears to be addressing the right issues relative to the December 19,,
1987 event.
However, the level of performance of their review effort remains to be seen.
Category:
4