IR 05000374/1986023
| ML20214P632 | |
| Person / Time | |
|---|---|
| Site: | LaSalle  |
| Issue date: | 09/17/1986 |
| From: | Norelius C NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML20214P629 | List: |
| References | |
| 50-374-86-23, NUDOCS 8609230326 | |
| Download: ML20214P632 (36) | |
Text
{{#Wiki_filter:U.S. NUCLEAR REGULATORY COMMISSION
REGION III
AUGMENTED INSPECTION TEAM Report No. 50-374/86023(DRP) Docket No. 50-374 License No. NPF-18 Licensee: Commonwealth Edison' Company Post Office Box 767 Chicago, IL 60690 Facility Name: LaSalle County Station, Unit 2 , Inspection At: LaSalle Site, Marseilles, IL Inspection Conducted: June 2-13, 1986 Team Members: G. Wright I. Villaiva R. Scholl R. Westburg J. Wechselberger A. Madison R. Kopriva J. Bjorgen M. Jordan //7l5[[o l - Approved By: Charles E. Norelius, Director Division of Reactor Projects Date ' Region III Inspection Summary Inspection from June 2-13, 1986 (Report No. 50-374/86023(DRP)) Areas Inspected: Augmented Inspection Team review of the events surrounding the June 1, 1986 feedwater transient, during which the LaSalle Unit 2 reactor failed to automatically shutdown when reactor water level decreased below the minimum technical specification value of 11.0 inches.
8609230326 860917 ' PDR ADOCK 05000374 G PDR , ,.,, - - -, . - - -. . -- -. . .- - .. .
. . _- -. - . _ _.. _ _ ., Augmented Investigation Team Report , 50-374/86023 ' ' Page No.
- I. INTRODUCTION
A.
>AIT Formation
' B.
_ Charter
, II. DESCRIPTION - EVENTS OF JUNE 1,1986
A'. Overview of Event ___
' B.
Sequence of Events" 6 -- ,- C.- Equipment Problems / Failures
' D.
Personnel Errors
III. INVESTIGATIVE EFFORTS
A.
Feedwater. Transient Details
B.
Anticipated Transient Without Scram (ATWS) Considerations
, C.
Subsequent Followup on June 1, 1986
D.
Operator Performance During and After the Transient.
E.
Special Tests '
, F.- Reactor Water Level Switches , !- (Static-0-Ring Differential Pressure Switches)
G._ History of Previous Events . 25.
H.
Generic Aspects of SOR Switch Problem
, IV. ROOT CAUSE: DISCUSSION
-V.
SUMMARY
> A.
Safety Significance
B.
Event Reporting
C.
Conclusions / Recommendations
VI. ADDENDUM TO AIT INSPECTION REPORT
- VII. ATTACHMENTS Attachment Description
, ' Number
Confirmatory Action Letter , '
AIT Charter
Alarm Printout
Hathaway Printout ,
Startrec Printout
Narrow Range Level Recorder
Transmitter vs. Switch Designation Level Drop Test Data
CECO Initial Test Program ' .
' , . . .. -... . . -. .. - - -. . . - - . .. -.
. . . . _ _ . Attachment Description Number
SOR Test Program
Historical Data Reviewed
LER 374/86-008-00
Information Notice 86-47
Level Switch Layout and SOR Model 103 Cut Away and Operation
Index of Pictures ' -15 IE Bulletin 86-02. Static-0-Ring Differential Pressure Switches
Restart Authorization Letter and SER ! l ' i ! s
a i & e i , d D
3 , 4. _ - -. - - _,.- ... _. _ _..,. - _. _, -. _ _ _ _ _, _ _ _ _. _ _ _ _... . _...... _ _ _.. _ _. _ _ _ -.. _,. _ _ _. _ - _.. _. _
. - _. . I.-, INTRODUCTION A.
AIT Formation On Sunday, June 1, 1986 the LaSalle County Station, Unit 2 experienced a feedwater transient which caused water level to drop below the technical specification minimum allowable scram point.
The unit did.not scram. On Monday, June 2, 1986 an Augmented Inspection Team (AIT) was formed consisting of: , Team Leader: G. C. Wright, Section Chief, DRP, Region III Team Members:- I. Villalva, SRI Lacrosse BWR, DRP, RIII R. Scholl, Jr., Senior Reactor Engineer, NRR R. Westburg, Reactor Inspector, DRP, RIII J. Wechselberger, Resident Inspector, Oyster Creek, Region I A. Madison, SRI, Quad Cities , Additional . . t Personnel: M. Jordan, Senior Resident Inspector, LaSalle Station J. Bjorgen, Resident Inspector, LaSalle Station R. Kopriva, Resident Inspector, LaSalle Station The AIT arrived onsite on June 2-3, 1986 and began gathering and evaluating the available data. The original team was augmented at the end of the first week by the addition of Mr. A.-Madison.
[ The purpose of the addition was to provide an operations perspective on the event in question and an event of May 9 which the' AIT had
identified as being a potential precursor to the June 1,1986 event.
In parallel with the AIT formation Region III issued a confirmatory action letter dated July 2,1986 (Attachment 1) which confirmed certain actions in support of the AIT and established conditions required to be met prior to plant restart.
B.
Charter A charter was formulated for the AIT and transmitted from G. C. Wright to Charles E. Norelius on June 10, 1986 with copies to NRR, IE, AE0D, . and selected Region III personnel.
(Attachment 2).
The AIT was-terminated on Friday, June 13, 1986 by the team leader.
i i ,
, ., - -, . ,,,. -,. - , _ _ _ -. _-,,.,_,.,_y._-,.._-.~,__.y..__.--_..y..- .,.,. _., _..,,., _ _..,. _ _ -.,,.,,, .,,. _.,,.,, . _ _.,,.,,.,.,,,.., _ - -
. .- - . .- . i ~ II. DESCRIPTION OF THE EVENTS OF JUNE 1, 1986 A.
Overview of the Event On ' June 1,1986 at approximately 4:21 a.m. (CDT) the LaSalle County, Unit 2 reactor experienced a feedwater transient from approximately 95%. power precipitated by a failure in the feedwater turbine. control circuitry.. The reactor vessel water level had gone both high and low and a half scram had been received.- Review of the event by the on-shift. personnel did not reveal any anomalies and therefore actions were taken to increase power starting at a post transient power of 440 Mwe. A routine shift turnover occurred after the unit
had reached approximately 660 Mwe. After the turnover, the oncoming shift again reviewed the recorder charts associated with the transient.
Between 7:00 and 7:30 a.m. the oncoming Shift Engineer (SE) and Shift Control Room Engineer (SCRE) determined that water level may ' have gone below the technical specification minimum allowable value of 11" without a full automatic occurring scram. An indicated reactor water level of 11" is measured from an artificial instrument "0" and is 14 feet above the top of the reactor core. Actions to increase power were curtailed, and a more extensive review of the transient and instrument behavior was initiated. The SE called in ' personnel to check the calibration of the four water level trip switches, the narrow range water level indication circuitry, and to interrogate a separate on-line data acquisition system (Startrec).
At approximately 9:30 a.m. the data was retrieved from Startrec and indicated that reactor level may have gone as low as 6&". By approximately 2:00-p.m. the licensee received the calibration data -
on the four water level switches at Level 3 (scram point) which
indicated that; while all four were out of calibration low (i.e., nonconservative) only one switch had a trip point below~the i technical specification-limit. The licensee decided that the conflicting information between indicated water level and indicated < level switch trip point warranted a unit shutdown.
In parallel with , conducting.the shutdown, a calibration check of the narrow range level monitors was performed and a subsequent one point check on the accuracy of the Startrec data was performed. The results of the additional checks were available around 6:00 p.m. and indicated that the narrow range monitor was accurate and that Startrec read i approximately two inches higher than the narrow range recorder.
" The licensee thus concluded that during the transient, level may.
' well have gone below the technical specification limit of 11" for approximately 1.5 seconds, and may(have been as low as 41" without the "A" reactor protection system RPS) channel tripping. The . licensee declared an Alert, per their General Station Emergency Plan L (GSEP) and placed the "A" RPS channel in the tripped condition. The unit was shutdown in a controlled manner reaching hot shutdown at 9:22 a.m. (CDT) on June 2, 1986 at which time the Alert was terminated and a detailed review and analysis of the event was commenced.
.
~. -. . - ,. - .- . -. -..... .. - - - - - . . - - .- -
B.
Sequence of Events The Augmented Inspection Team (AIT) and the licensee compiled sequence of event lists. The following is the sequence of events prepared by the licensee and reviewed and clarified by the AIT.
It is noted that detailed times between 0412 hours and 0435 hours are not provided because the information was being obtained from different sources (i.e., alarms, Hathaway printer and Startrec), using different timing devices. Copies of the alarm and Hathaway print outs and the Startrec trace are provided as Attachments 3, 4, and 5.
Initial conditions: 95% power; 1040 MWe; ramping up at 5 MWe/hr 0412 hours The Nuclear Station Operator (NS0) was conducting surveillance procedure LOS-B0-WI, " Balance of Plant Weekly Surveillance" on the "A" Turbine Driven Reactor Feedwater Pump (TDRFP), Step F.3.a, which tests low pressure steam stop valve.
Per the procedure the NS0 pushed the test pushbutton - and closed the Turbine Stop Valve (TSV) five to six percc.t and then released the pushbutton.
The NS0 noted that the Turbine Control Valve (TCV) - jumped in the open direction approximately five to ten percent and he immediately looked to see what effect this had on the "A" TDRFP flow.
The NSO observed that reactor water level was going - up fast.
The NS0 called for assistance. The Shift Control Room - Engineer (SCRE) came to assist immediately and the Center Desk NS0 came immediately afterwards from Unit 1.
- The high reactor water level alarm set at "40.5" was received.
- Both "A" and "B" TDRFP's locked up "A" was in manual control (46%) and "B" was in three-element control (44%). - The NS0 reset the lockout on the "A" TDRFP and started reducing its output manually to reduce reactor water level.
- Just before reactor water level reached its peak the SCRE reset the "B" TDRFP lockout.
- The "B" TDRFP went to O speed 0 flow.
-. - .. . . . . _ _ _ _ _. .-
As level started going below normal level the SCRE - had the Center Desk NS0 run back reactor recirculation flow to lower power level, and thus bolster level.
The NS0 started the motor driven reactor feedwater - pump (MDRFP), t A Channel B half scram on the D reactor water level - instrument (2B21-N024D) was received.
The NS0 and SCRE both in'icate that reactor water d - level reached the 121 inch trip point as indicated on the B narrow range (BNR) indicator. All three narrow range indicators were reading consistent with each other.
An ADS confirmatory level annunciator was received - and the reactor recirculation pumps downshifted which resulted in a reactor vessel water level swell.
- Level was at + 20" shortly afterward when SCRE looked at the water level indicator.
Efforts continued to return reactor water level to - a normal level of approximately 36".
- The MDRFP was placed in single element control but didn't respond well, so the "A" TDRFP was placed in single element control and the MDRFP was placed in manual control.
Considerable problems with feedwater heaters - were encountered during subsequent recovery operations which required significant operator attention.
Control Rods were inserted to eliminate the APRM High - alarms, which were the result of colder feedwater as the feedwater heaters were lost as indicated above.
0435 hours The SE called in the Nuclear Engineer to assist in upshifting the reactor recirc pumps and to help adjust rod pattern, 0457 hours The SE called the Unit 2 Operating Engineer (0E) to discuss the transient and problems with TDRFPs. They discussed the reactor recirculation pump downshift but do not recall any mention of half scram. Reactor water level was not discussed because the shift's review indicated that level had not gone below +12.5".
--- - - - - -. . -. - - - _ _.- - -. -
_ _ _ _ - _ _ _ _ _ 0518 hours.
The Nuclear Engineer arrived onsite.
0600 hours The reactor recirculation pumps were upshifted to fast speed. Reactor Power was now 600-650 MWe.
0615 hours The SE talked to the.IM' Foreman on the feed pump problem. Shift personnel were able to correct problem on the "B" TDRFP.
DAY SHIFT 0545 hours The oncoming SE arrived as the shift was preparing to upshift the reactor recirculation pumps.
- The oncoming SE observed the upshift - noted wide range reactor water level indicator had gotten down to "0" and that NR was unclear as to how low level had gotten.
During turnover (TO), the shift talked about the - problems with the feed pumps, the lockouts, and the half scram but no mention of a concern about low reactor water level.
- The SE and SCRE discussed transient.
The Unit 2 NSO stayed over so SE could discuss - transient with him, discussion focused on feed pump problems.
r - After T/0, the SE reviewed the control room panels and recorders again and started to develop a concern for a potential low level during the event.
- The SE, SCRE, and SF together looked at the reactor water level recorders - concern developed for actual level setpoint of level switches.
The day shift concluded that they had a concern which - should be addressed based on +5" indication of NR recorder; after very close examination including accounting for indentation in paper when pen wasn't inking.
' 0732 hours The SE told the SCRE to call out an instrument mechanic (IM) to check the calibration of the reactor water level scram switches.
The SE terminated the power increase.
-
.... .- . .. - -. . -. . --. - -
The-SE called the Unit 2 OE to discuss his concerns - about whether the Unit should have scrammed.
The OE reviewed, with the SE, the 5/9/86 scram and - level switch problems where B Channel had tripped with A Channel tripping some time later, but a calibration check had shown everything as being satisfactory.
A calibration of the B NR instrument was requested.
- - No action was to be taken until the calibration checks were completed.
0800 hours The IM Foreman contacted the Control System Technician (CST) and told him to report to site and perform LIS-NB-201. (calibration check on level switches) 0807 hours The SE called Unit 1 OE (Duty Officer) who concurred with the plan.
The Ceco Duty Officer suggested a check of response - time but this was never accomplished due to length of time required (2 days per CST).
0824 hours The Production Superintendent called the SE to check on plant status. They reviewed the transient; time spent on TDRFP and plant status - concurred with the plan.
0836 hours Technical Staff Engineer was called to retrieve data from Startrec.
0852 hours IM CST arrived on site.
- The CST talked to the Tech Staff Engineer about Startrec basics.
- The CST waited for the IM Foreman to arrive to see if additional calibrations are desired.
0857 hours Technical Support Assistant Superintendent arrived on site.
0907 hours A Technical Staff Engineer arrived on site to run Startrec.
0930 hours Startrec results received; print out indicated level reached a low of +6 1".
1008 hours An IM foreman arrived on site.
- - .. - - - -
1030 hours The Technical Support Assistant Superintendent noticed there had been a transient when reviewing the computer. He called the SE and discussed the transient and action plan.
1100 hours The SE authorized surveillance on reactor water level switches.
1121 hours The Unit 2 OE called the SE and discussed the response time suggestion.
1125 hours Calibration check of reactor water level switches begins.
1200 hours First reactor water level switch (A) completed - preliminary results indicate a. trip setpoint of +9.3".
(Nominal Trip pt. + 13 inches).
1330 hours The SE called the Technical Services Assistant Superintendent to discuss preliminary calibration results and his uneasy feeling about whether they should continue to operate.
, 1351 hours The Technical Services Assistant Superintendent, the SCRE, and the SE called the Production Superintendent to discuss calibrations checks.
Two switches completed - switches returned to +13.4" - trip setpoint after cycled; went even higher after repeated cycles.
Level switches calibrated just two weeks ago.
- Discussed power reduction at 50 MW/Hr.
- Production Superintendent said he would talk to - corporate duty officer.
1400 hours CECO Vice President called Load Dispatcher to discuss i system conditions and was told of problem at LaSalle.
Instructed station not to touch any more instruments in order to preserve the as found condition.
1400 hours Reactor water level switch calibrations completed.
1425 hours Production Superintendent discussed situation with Vice President - unit shutdown recommended.
- Production Superintendent issued order to SE to commence an orderly reactor shutdown at 100 MWe/Hr.
The operating crew was instructed to manually scram - the reactor if water level was even suspected of being near the +12.5" scram setpoint.
-.- -.- -
-. __ _ _ _ . . . . . The problem was considered in terms of an instrument- -: setpoint problem, not-an anticipated transient-without scram (ATWS).
. . 1428 hours-IM Foreman called the Master Instrument Mechanic to-discuss situation. Master IM indicated he would come to the site.
J Afternoon Shift , -1505 hours Master-IM reported to the site and discussed situation ' L with the IM Foreman. They decided calibration of the-B NR recorder should be checked.
1530 hours NRC Resident notified.
1630 hours Master.IM received permission to check calibration of B NR recorder.
1723 hours K. Grasser, Vice President, reports to site.
, 1750 hours Red Phone call to NRC on reactor shutdown (Courtesy call).
1830 hours Calibration completed on the B_NR recorder.
A GSEP ALERT was declared . - The NRC Resident was' notified The "A" RPS Channel was tripped (1/2 scram) - The licensee made a conscious decision not:to
- scram that reactor but to bring the' unit down in a controlled manner.
LStartrec was checked by stabilizing levelsat 36" as - ~ indicated on the B NR recorder and running Startrec to get a comparison.
1845 hours Red Phone call made to notify NRC of GSEP ALERT.
, 2030 hours D.,P. Galle returned from out of town and was notified.
< 2200 hours C. Reed located in New York and notified.
f June 2, 1986 0922. hours All rods fully inserted.
The GSEP ALERT was terminated.
-
11 + i .- - - - -,. - - - -.. .s.-w,w.,,,,,,.,w ,-,., e., , ,--,.+--a-*--=+- - ' ' = - = r "* r *w' t-f"Y-w-r - - - - ' " *<'r-w
C.
Equipment Problems An initial review of the event and equipment history revealed a number of apparent equipment failures or questionable operations.
1.
The "A" Turbine Driven Reactor Feedwater Pump governor valve unexpectedly opened during the stop valve test.
2.
Both Turbine Driven Feedwater pumps locked up.
3.
"B" TORFP ran back to zero flow.
4.
Computer printout only indicated that one Turbine feed pump was locked out.
5.
The "A" Turbine Driven Reactor Feedwater pump could not effectively be placed in 3-element feedwater control.
6.
Certain vendor recommended surveillances could not be performed on the TDRFPs due to control circuit problems.
7.
The Motor Driven Reactor Feedwater pump did not respond properly when the operator attempted to place its associated flow control valve in automatic.
8.
Three of four Static-0-Ring Model 103 environmentally qualified differential pressure switches, in a reactor vessel water level monitoring application, did not trip even though actual level was well below where the calibration and the calibration check indicated the trip point to be.
Items 1-4 were either new problems or specific to the June 1,1986 event.
Item 3 was caused by normal reaction of the automatic control system to a sustained water level greater than the setpoint. That the turbine did not subsequently respond to a decreasing level is not understood.
Items 1, 2, and 4 are not understood.
Items 5-6 were previously known to the licensee who was already working with the control vendor, Lovejoy, to correct the observed problems.
Item 7 was subsequently traced to the operator manipulating the wrong controller.
Item 8 is a new phenomenon for LaSalle County Station, Comonwealth Edison, and apparently the industry.
The discrepancy between where an instrument trips under actual installed conditions and where, using a basically standard industry procedure, the instrument was set to trip brings into question the suitability of application of the specific switches as well as the
_ _ - - . - . _.. .
operating characteristics of other differential pressure switches.
For a more detailed discussion of the Static-0-Ring differential pressure switch problem refer to Section III of this report.
D.
Personnel Errors A review of the June 1, 1986, event identified that three personnel errors were committed during the approximately two and a half minute event.
(Refer to Section III-D for more detail) 1.
Attempting to control the "A" TDRFP prior to resetting the control system lockout.
2.
Attempting to control the MDRFP with the wrong controller.
- 3.
Resetting the "B" TDRFP prior to ensuring that the " Demand" and " Existing" control system signals were in agreement.
The first two errors are of minimal safely significance. The third item was the cause of the decreasing reactor water level transient.
13 - - --_. - _ - . - - - - - _ - _ _ _, _ . -. .. -..- -. .. - _, - ........
III. INVESTIGATION EFFORTS A.
Feedwater Transient Details On June 1, 1986 at approximately 4:20 a.m. (CDT) the Unit 2 Nuclear Station Operator (NS0) began a surveillance on the low pressure stop valve for the 2A turbine driven reactor feedwater pump (TDRFP). The test involved depressing and releasing a test push button and observing movement of the stop valve. During the test evolution the NSO noticed the governor valve had opened an additional five to ten percent. The NSO immediately referred to the reactor vessel water level narrow range indicators to ascertain the effect on vessel level and noticed that reactor water level was increasing. Upon turning back to the feedwater control station the NSO noted that both turbine driven feedwater pumps had locked-up (control circuits " frozen" in an "as-is" state). The NSO reset the 2A turbine controls and started to decrease the turbine's output.
He also requested help to re The Shift Control Room Engineer (gainSCRE) control of the "B" TDRFP.
responded and reset the "B" feed pump control circuitry. When the circuit was reset, the turbine's output decreased to zero and stayed there. As level was decreasing rapidly, due to the loss of the B feed pump and manual runback of the A feed pump, the NS0 both increased demand on the A pump and started the motor driven feedwater pump (MDRFP). At about the same time the SCRE instructed the center desk NS0 to manually runback the Unit 2 reactor recirculation flow control valves. Shortly thereafter the unit received a half scram on the "B" reactor protection system (RPS) channel, a down shift to low speed of the reactor recirculation pumps, and an ADS confirmatory level (+12.5" nominal) annunciation. A later review of the Startrec trace indicated that level remained below the 12.5" nominal scram point for approximately one and a half seconds.
(Note that Startrec information is not available to control room personnel.)
Due to the rapid power reduction, i.e., approximately 500 MWe in less than two and a half minutes, a number of feedwater heaters had isolated thus reducing feedwater temperature and resulting in high alarms on the Average Power Range Monitors (APRM). To correct the situation the licensee inserted selected control rods to depress power.
After power had been reduced and reactor water level stabilized, the licensee reviewed the event via discussions with individuals and a review of retrievable control room indicators. The on-shift crew noted no abnormalities, and as such proceeded to return the unit to a normal condition, i.e. returning control rods to pre-transient positions, upshifting reactor recirculation pumps and re-establishing feedwater heating. By shift turnover the unit had returned to basically a pre-transient configuration with the unit at approximately 660 MWe.
... . . -. -- - . - -. - -. . . __. .. ... -.
.- __ 'B.
Anticipated Transient Without Scram (ATWS) Considerations 10CFR50.62(b) states: For purposes of this section, " Anticipated Transient Without Scram" (ATWS) means an anticipated operational ! occurrence as defined-in Appendix A of this part followed by the, failure of the reactor trip portion of the protection system specified in General Design Criterion 20 of Appendix A of this report. Based ' on discussions with the other NRC staff, it was concluded that this definition though, means that the condition requiring the trip is continuously present, and that the trip system does not immediately ' respond _and is incapable of responding. Therefore it was concluded that the June 1, 1986 event would not constitute an ATWS.
The AIT concluded that-the June 1, 1986 event did constitute an operational occurrence during which the required protective action of - the reactor protection system failed to function. Subsequent tests indicated.that in all likelihood the switches.would have functioned, although below-the' required setpoint; however, proof positive cannot be provided to support this contention.
The AIT obtained and reviewed the licensee ATWS procedures. 'The procedures were judged to be adequate and operations personnel had , been trained on them. Discussions with the crew on-shift at 4:20 a.m.-
on June _1,- 1986 did not reveal any consideration of an ATWS.
The condition required for entry into the ATWS Procedure did not i exist on subsequent shifts, i.e., the plant was in stable condition - with water level at a normal 36 inch level.
, l C.
Subsequent Followup on June 1, 1986 The oncoming crew discussed the event before and during the normal i shift turnover procedure. After the formal turnover of responsibi-lities, the new Shift Engineer (SE), SCRE, and Shift Foreman again reviewed the available transient information. After reviewing the narrow range level recorder trace (Attachment 5) the day shift individuals decided there was a question as to how low reactor vessel level had gone, based on what appeared to be a crease in-i the chart paper possibly caused by the recorder pen _ moving across, but not inking, the paper.
From this suspicion came three distinct actions. First, power 1_
escalation was terminated. Secondly, instrument mechanics were called in to perform a calibration check on the four RPS ,
reactor water level monitoring switches, (2B21-N024A-D) and the
"B" reactor vessel narrow range water level indication. Thirdly, a technical staff individual was called in to obtain and printout the ' information recorded by the Startrec data acquisition system.
(Note that Startrec is not safety-related nor is it on a routine surveillance or calibration frequency.
Its primary function was to l provide data acquisition during initial startup testing. CECO retained the system as an aid in diagnosing transients.) The
Startrec data was available at approximately 9:30 a.m. and indicated (
-._ _ ___ - - _ _ _ _ _ _, _. _. . - . ..- .. .
that reactor vessel water level had decreased to approximately six and a quarter inches. Calibration data for the 2B21-N024 switches was available around 2:00 p.m. and indicated the following trip points: N024A N024B N024C N024D Trip Point * Trip Point * Trip Point * Trip Point * 1st Cycle ** 65.5 64.8 64.1 64.6 2nd Cycle 64.6 63.8 63.1 63.9 3rd Cycle *** 64.2 63.6 62.8 63.9
- Trip Point expressed in inches of water column differential pressure.
Acceptable range per the procedure is 62.51 to 63.78 inches with a technical specification limit of 64.84 inches which is equivalent to 11" reactor vessel water level.
Note that decreasing water level results in increasing differential pressure.
- A cycle is one operation up to the trip point and back to the reset point.
- By procedure the third cycle was defined as-the as found condition.
A review of the first cycle data revealed that while all four switches exceed the administrative limit only one switch exceeded the technical specification limit (equivalent to 11 inches vessel level on narrow range level instruments). The third cycle data showed all trip points to be conservative with respect to the technical specification limit.
At this point the licensee had conflicting information to evaluate: (1) Indication from non-calibrated, non safety-related instruments that reactor vessel water level reached six and a quarter inches (four and three quarters inches below technical specification limit), (2) The non safety-related narrow range level recorder exhibited a crease in the paper, down to approximately five inches, (3) The safety-related level switches indicated, using first cycle data, that three of four switches would have tripped prior to reactor vessel level decreasing below 11 inches. The "as-found" data indicated all four switches would have tripped prior to reactor , vessel level reaching 11 inches.
In addition it was known that one switch did operate.
The decision was made to start shutting the unit down and to continue checking on the accuracy of the level transmitter / recorder circuitry and Startrec. At approximately 2:00 p.m. on June 1, 1986 a unit shutdown was commenced. One of the resident inspectors was notified around 3:30 p.m.
Calibration checks continued on the narrow range level transmitter (2B21-N004B) with the results, being received around 6:00 p.m. indicating that the instrument was accurate.
A final check on the accuracy of the Startrec system was performed and indicated that at a steady state reactor vessel level of 36 inches, Startrec indicated 38 inches.
4 _ - - . _ _ _ .. __
With the above information the licensee concluded that water level in the reactor vessel had decreased below the scram point without the "A" Reactor Protection System (RPS) channel tripping. The' licensee followed up on the previously made courtesy call, to the NRC Head-quarters duty officer on the unit shutdown, with a call declaring an Alert (per CECO General Station Emergency Plan (GSEP)) at 6:30 p.m.
(CDT).
In addition, the licensee manually tripped the "A" RPS channel.
The licensee continued to bring the unit down in a controlled fashion and maintained the Alert status until 9:22 a.m. (CDT) on June 2, 1986 when all rods were fully inserted and the unit was in hot shutdown.
D.
Operator Performance During and Following the June 1, 1896 Event Members of the AIT reviewed the actions of the operations personnel on-shift during the. event and those persons on the next shift.
The initial investigation into the June 1,1986 event did not identify any improper operator action. The NS0 appeared to have taken the appropriate actions, including returning the "A" TDRFP to automatic operation and starting the motor driven reactor feedwater pump. Actions were also taken to maintain level, by reducing reactor power via recirculation flow when the "B" TDRFP ran back to zero output. Following the event and prior to escalating power, the on-shift crew reviewed the information available to them. The crew concluded that while they had received a half scram, level had recovered prior to receiving the requisite one out of two twice logic for a full scram.
In addition the "B" TDRFP behavior was discussed with an instrument foreman who indicated that by removing and inserting a particular card in the control logic that the saturation condition would clear. The crew performed this evolution and regained control of the "B" TDRFP.
Subsequent discussions wnh the licensee indicate that there may have been three personnel errors. Two of the errors were relatively insignificant involving operating a controller prior to resetting a lockout and attempting to place the MDRFP control valve in automatic using the wrong controller. Neither of these errors had any significant impact on the transient.
The third error was committed when the lockout condition on the "B" TDRFP was reset. Apparently no attempt was made to balance the lockout control position and actual demand prior to resetting the lockout.
From the time the TDRFP's locked out until the "B" turbine was reset, reactor water level had remained above the 36 inch normal level. With level greater than the automatic controller setpoint the turbine controller ("B" turbine was in 3-element automatic) would attempt to " educe turbine speed, however, with the turbine control lockout in effect, the turbine could not respond to the requested speed reduction.
In the absence of any feedback in the form of decreasing speed, it appears that the "B" TDRFP's controller continued to integrate down to zero and then saturated. When the lockout was reset the turbine control system sensed the "0" speed demand and responded accordingly by ramping down to "0" output.
, _, _ _ _ ___ _ _ _ _ _ _. _ _ _ __ .
. . -.- _. - a'
A review of the Startrec traces (Attachment 5) indicates that the rapid reactor water level drop started at almost the same moment that the "B" turbine was reset.- While the initial reaction of the "B" turbine' appears to have been - appropriate, the. reason for the turbine not responding to the subsequent reactor water level decrease has not been completely explained.
The crew on-shift at the time of the transient, as mentioned. earlier, did not identify that reactor level had decreased below the minimum technical specification' limit. The oncoming shift noted an apparent , crease in the recorder paper which caused them to pursue the subject further. During the initial investigation, on day shift, the unit was held at a steady state pcwer level. When the apparent discrepancy between indicated water level and scram switch setting , was identified around 2 p.m., the licensee commenced an orderly shutdown and notified the resident inspector of possible problems with the feedwater system or level indicating system (which also has an input into the.feedwater control system). When information became available around 6:00-6:30 p.m. on June 1, 1986 that the level indicating system, including Startrec, was properly calibrated, the licensee declared an Alert condition. As the licensee suspected only RPS Channel "A", a manual trip was inserted into Channel "A", and the. orderly shutdown was continued, with a hot shutdown condition being reached at 9:22 a.m. on June 2, 1986.
The AIT evaluated the response of the individuals versus the information they had available and have the following coment: 1.
With the exception of the' personnel errors (see Section III.F) the AIT has concluded that the personnel on-shift at the time ' of the event, took prompt and proper corrective action based on what they perceived was the problem. The AIT finds no fault with the on-shift personnel for not identifying that reactor vessel water level had decreased below the technical ' r_ specification limiting condition for operation (LCO) level.
2.
The AIT has concluded that the oncoming shift's recognition and persistence in following up on the perceived water level problem was exceptional. There is some philosophical question
i as to when the Shift Engineer should have called the NRC Headquarters Duty Officer (see Section V.B).
However, this t
does not detract from his overall excellent performance in ~ identifying, evaluating, and following up on the event.
E.
Special Testing of Static-0-Ring, Inc Differential Pressure Switches Based on the calibrations and checks described in Section III.B, the licensee proceeded to devise a test program to physically i vary reactor vessel water level and to measure the point at which ' each Level 3 reactor water level (RPS scram level) switch tripped.
18 . - gy,, ,-,- w--e-ev.,3 er r ,,,r,,,,r...,w,3<w,- . -. -,.,-~-, -, - .e, - y .r.
- - ,-.--,,-m-m,w .,-,,,,w,,-,,-,,,ver-c-,-, -,e- , - -,,,,, -,,, -, .---ww- ---. --,-,w
The test involved lowering reactor vessel level via the reactor water cleanup system reject'line. The condition of all four differential pressure switches-(instrument numbers 2B21-N024A, B, C, and D) was monitored, as well as the output of the level transmitter associated with each switch. -(It is noted that a transmitter had to be installed on the instrument rack holdin the 2821-N024D switch as there was no installed transmitter.)g Further, it is noted that the condensing chambers for the two reference legs of the level monitoring columns were not at the same elevation and, the individual transmitters although properly calibrated, were not cross calibrated with each other.
. Testing was conducted twice at reactor (Rx) vessel pressures of approximately 950, 500, and 0 psig. The results of the tests are tabulated below.
It must be noted that the trip levels indicated below were obtained from each switch's associated level transmitter.
(A tabulation of what each transmitter read when a specific switch tripped is included as Attachment 7).
SUMMARY OF REACTOR LEVEL DROP TEST DATA A.
Measured Trip Setpoints in inches of Rx. Vessels Level *** Test Rx. Pres.
Trip Setpoint (Rx. Level Inches) No.
Date Time (psig) N024A N024B N024C N024D
6-2-86 1500 950 9.0 3.9 6.9 10.2
6-2-86 2100 950 10.8 5.1 8.9 8.4
6-3-86 1900 500 9.6 6.0 8.4 ---
6-3-86 2100 500 12.3 11.0 11.4 10.5
6-4-86 2100
10.7 11.7 11.4 11.6
6-5-86 1400
11.1 11.1 12.2 13.5 B.
Corrected Trip Setpoints in inches of Rx. Vessels Level * Test Rx. Pres.
Trip Setpoint (Rx. Level Inches) No.
Date Time (psig) N024A N024B N024C N024D
6-2-86 1500 950 8.6 3.5 6.5 9.7
6-2-86 2100 950 10.3 4.6 8.4 7.9
6-3-86 1900 500 9.3 5.8 8.2 11.2**
6-3-86 2100 500 12.0 10.7 11.2 10.3
6-4-86 2100
10.7 11.7 11.4 11.6
6-5-86 1400
11.1 11.1 12.2 13.5 C.
Corrected Trip Setpoints in Differential Pressure *;*** Test Rx. Pres.
Trip Setpoint ("WC at Switch) No.
Date Time (psig) N024A N024B N024C N024D
6-2-86 1500 950 66.1 69.7 67.6 65.3
6-2-86 2100 950 64.9 68.9 66.2 66.6
6-3-86 1900 500 65.6 68.1 66.2 64.3**
6-3-86 2100 500 63.7 64.6 64.3 64.9
6-4-86 2100 000 64.6 63.9 64.2 64.1
6-5-86 1400 000 64.4 64.4 63.6 62.7
- Corrected for the Rosemount static pressure shift.
- Estimated value based on the level readings at the other racks and the elevations of the condensing chambers.
- Switches had been calibrated on 6/1/86 to trip at 63.1, 63.1, 63.3, and 63.4 inches water column for switches N024A-D respectively.
After the test at 950 psig the licensee checked and readjusted the switch setpoints. The first cycle and "as-found" data is listed below. For comparison, the actual trip points from the vessel drop testing at 950 psig are also listed.
N024A N0248 N024C N024D 1st Cycle Trip Point: 65.0 64.2 63.9 64.6 "As-Found" Trip Point: 63.1 61.6 62.9 63.7 950 psig drop test Trip Point: 66.1 69.7 67.6 65.3 During and following the level drop testing the licensee compiled a list of where S0R, Inc. differential pressure switches were used at the LaSalle County Station. A review of the list revealed that in addition to the level indication application, the switches were used in flow applications on ECCS systems to control the position of the minimum flow valves and the main steam lines to detect high steam flow.
With the non-conservative results obtained on the level monitoring applications, the licensee performed a series of tests on selected minimum flow valve switches to determine their operating characteristics.
The testing involved calibrating the switch, starting the system, varying system flow and recording the flow at which the minimum flow valve closed and opened. The only automatic function associated with the Technical Specifications (T3) is valve opening for pump protection from overheating. The test was performed on the Unit 2 High Pressure Core Spray (HPCS) system, the Unit 2 Residual Heat Removal (RHR) A and C subsystems, and the Unit 1 HPCS system. The results were as follows:
. . - _ _ _ -. .. ._ __ _. . . _ - -
-. ... . - . ' f.
, ,, / 1.
Unit 2'HPCS Normal Calibration Set)oint 1000 gpm TS _imit 900 gpm~ . ' Flow for valve opening a.. First System Run After Calibration 565 gpm 565 gpm b.
Second System Run - After "Overrange"* 692 '< Calibration 692
- "0verrange": The licensee exposed instrument to maximum rated differential pressure and then performed normal calibration.
-s c.
Third System Run - After Flow Transmitter ~632 { Calibration ' 2.
Unit 2 RHR A ' ' a.
Overrange Calibration Set)oint 1000 gpm TS .imit 550 gpm Flow for valve opening b.
First System Run --After Overrange 750 gpm Calibration 790 gpm 612 gpm 3.
Unit 2 RHR C Nominal Setpoints Set)oint 1000 gpm TS _imit 550 gpm ! Flow for valve opening a.
First Run - As Found Data 750 gpm 559 gpm 774 gpm , ' b.
Second Run - After Overrange 500 gpm Calibration 353 gpm - 250 gpm
c.
Third Run - After Normal Calibration 0* gpm 0 gpm 0 gpm
a
,
. .,, ---.--.c,,,s .e,-,.,,.--, . - - - -. - --,-,.,,,-.,-----.,--.--,,.,..--c., . -,,... - -,, - -,,--,,...,,.v.,
. . .. ._- _ __ h
- Valve did open until pump flow was 0 gpm lg 4.-
-Unit 1 HPCS Set)oint 1000 gpm-
TS _imit 900 gg Flow at-valve opening First Run - As Found Data 395 433 353 It was noted that the flow values recorded above were obtained from flow instruments which read from 0 gpm to over 8000 gpm and, as such, some inaccuracies in reading, as well as instrument accuracy, can be. expected. However, the general conclusion. drawn was that the erratic setpoint behavior was not limited to a level- ' application but appears to be generic to, at a minimum, SOR. INC differential pressure switches.
Based on the results of the above tests the licensee embarked on a more extensive testing program which, at the termination of the AIT, , was not complete. Copies of the initial testing programs for Commonwealth Edison and SOR, INC are attached (Attachments 8, and 9).
F.
Review of SOR, INC Differential Pressure Switch a Behavior / Operation at LaSalle County 1.
Differential Pressure (DP) Switch-Anomalies The inspectors reviewed data listed in Attachment 10 relative ' to the behavior of the 50R, Incorporated differential pressure switches at LaSalle. The review included computer printouts, recorder traces, procedures,-calibration sheets, discrepancy , reports, deviation records, environmental qualification (EQ) reports, Sargent and Lundy (S&L) specifications, interviews with key personnel, and discussions with the resident inspector from a plant with similar problems, a.
Historical Perspective s Previous to the June 1, 1986 and similar May 9, 1986 - (Section III G) Feedwater transients, LaSalle had observed erratic behavior of S012 pressure switches. These switches are made of the same materials as the DP Mi switches; however, they operate-on a different principle.
~ Pressure switches use a piston-actuated force balance to , . actuate the switching element directly, while DP switches use a piston-actuated force balance with different pressures on either side of the piston to actuate the switching element through a cross shaft and lever
_ - - _ _ _.., -.... _. ._ _ , _. _. _. -. - -. _. _ _ _. _ _. _. _. _ _
- _ _ _ _ -- - - - - -- - - - . , ,m-
t \\ y ~ ' '- () )J% g.
' ? ,o y y i g-
arrangement.,Because of tlie~ pressure switch problem, . LaSallehao'beenincontact]withtheDystedCreek, Plant ie personnel whcJprovided inft.nnitfon relativi to their DP .t . switch problem. As a resu R of'this exchahge of infoiga, ~D-tion, LaSalle. calibrated t)Tair DP switches fin March, Unit t ahead of their requireo, frequency.; In accordance with the . calibration procedure, the DP switches ~were cycled prior to the calibration. However, bedaus the pressure switch anomalies (see u,e oW the,NRC review of spection Reports q No.:50-373/86013;No.50-374/86013(DRS))ahd,the ! licensee's feeling that-cycling the switciicould mask erratic behavior, the' calibration data:were gecorded for< cycles one and two even though the thifd'cyc?e was-officially recorded as the."as found" value. ':The calibra-tion indicated that the N024B and N024D switches were out of tolerance with the administrative setpoint limit of ' 62.51 to 63.78 inches,of water. Following the May 9 w event, the N0248 switch was out of tolerance and following ' , the June 1 event, the N024A and the N024D switches were
. out of tolerance. With a data base of six cilibrations, ~ ' the following was observed.
j . , , (1) Switch N024A was out of tolerance three times (50%). It exceeded'the administrative limit twice, both ' non-conservative..It exceeded ths LCO limit of 64.84 inches:oncea ' '
- a
- s (2) Switch N0248 was out of toihrance four times (66%). It exceeded the administrative limit three times, all s . conservative.
It exceeded,the LC0 limit ence.
l (3) Switch N024C was out of'ticlerance three times (50%). It exceeded the administrative limit twice conserva- ' . tivelyandoncenon-conservptively.
(4) Switch N024D was out of tolerance four times (66%). It exceeded the administrative limit once conserva-tively and three times non-conservatively.
, , Discussions with the instrument technicians, indicated that ' the DP switches have had a drift / erratic setpoint problem
since they were installed. This point is also. illustrated ? x by the three sets of calibration data prior to the events.
N . The N024A switch was out cf tolerance two out of three fc times, N024B two out three times, N024C two out three times,' and N0240 three out of three times. Discrepancy Records Nos. 86-007, 86-167, and 86-231 documented the out of tolerance conditions during the January-3, March 31, and May 10, 1986 calibrations. In each case, the action to prevent recurrence was "cu.rrently being' reviewed per trend analysis program." The CECO, trend analycis program is based on a six month interval and, therefore, ttte apparent . erratic behavior had not come to management attention as- (, of June 1, 1986.
I ' s . \\
s .f .-
._ y ,
i.
,. '
N ,t i , b.
Calibration Procedure Review The inspector's review of the calibration procedures for the Unit 2 Low Water Level Scram produced several concerns.
First, the practice of cycling the DP switches prior to . calibration erases the "as found" condition of the switch and makes the calibration data "look" better. Second, the as found condition required by procedure No. LIS-NB-201 is taken on the third cycle which is not the true as found value. Third, concerns with DP switch cycling caused the inspectors to review the EQ package for the S0R switches.
' The review of this package showed that the calibration of i the DP switches between EQ tests also cycled the switches prior to calibration. This puts in question the qualifica-
,
tion of these switches since the actual field conditions at , g o LaSalle have caused the switches to fail to perform as 4~ expected far short of their qualified life.
Finally, both ,'J ' ' the EQ test and the LaSalle calibration at 0 psig static , system pressure alone do not appear to be adequate, based on the results of the post event vessel level drop tests at various pressures.
. c.
Reactor Vessel Water Level Drop Test y , -,,x During thr June 1, 1986 feedwater transient, only one out of four DF switches for the Level 3 Scram initiated a , y scram signal. Subsequent calibration of the switches indicated that all were out of tolerance; however, only . Switch N024A exceeded the LC0 limit of 11 inches reactor ' '
, vessel level. The licensee then proposed a reactor level drop test to test the operation of the level switches at !;) various static pressures. The inspectors reviewed the , reactor level drop test procedure, No. LST-86-096, prior to the test and witnessed the June 2, 1986 test at 950 psig of reactor pressure. Six level drop tests were performed and the data shows an actuation point increase (i.e. non-conservative setpoint shift) with increase in static pressure.* Previous calibration data and observa-tions.from the IMs also indicate possible cycling and hysteresis effects. The cycling effect is such that with a setpoint starting at a higher or lower level than the ,' calibrated setpoint, on successive actuations the setpoint tends to move towards the original calibrated setpoint.
(, The hysteresis effect is such that the setpoint first
actuation is either higher or lower than the desired , ' setpoint depending on whether the switch was actuated from N ,, ' a O psig start point of after overranging the differential , pressure, c f , T6 sting for June 13, 1986 revealed that setpoint shift due to static pressure
g alone was actually conservative.
. Y .-
. - .. -. - - _ - -. - - . - - -. . - -
... . . -. . - . . . - -. - - _ r
, i
' d.
SOR DP Operability Evaluation Test The licensee has started a testing program to detennine if ! the magnitude and type of observed behavior of the SOR DP switch actuations is switch-unique or generic to that
model of switch. The inspectors reviewed the' test plan, the test procedures, and observed portions of the static DP switch testing in the plant and found them acceptable from a technical standpoint.
G.
History of Previous Events . A review'of recent events at LaSalle Unit 2 revealed _a May-9, 1986-event which appeared to have been a precursor of the June 1,'1986 event. The AIT initiated a review of the May' 9 event both from the ' eperation's personnel performance standpoint and subsequent management review of the event.
a . On May 9,.1986 a worker in the plant bumped an instrument rack and tripped one turbine driven reactor feed pump. The resultant i.
feedwater transient took reactor water level dorn to approximately , eight inches as indicated on the narrow range meters before the reactor automatically scrammed. The event is documented in LER 374/86-008-00 (Attachment 11).
The licensee's followup of the event indicated that the "B" and "C" narrow range indicators were reading approximately 1.5.to 2 inches-
low. The "as found" scram switch setpoints.were between 12.6 and 14.42 inches of reactor vessel water level. The "A" narrow range i reactor vessel water level indication was lost'due to the initiating
event.
(Startree traces were not available for the event) The licensee's review concluded that the unit had automatically tripped.
E at the 12.5 inch scram point as the scram level switches j (2B21-N024A-D) were found to be set at or above that point.
The ' discrepancy between the indicated level and trip point was, in part, attributed to instrument calibration and, in part, to operator error in reading the instrumentation.
' The AIT's review generated a number of concerns in both the performance and review areas of the May 9,1986 event as detailed t
below: , . c 1.
The Shift Engineer (SE) (SRO) did not position himself to
adequately control all actions.
Immediately upon entering the Unit 2 control area, the SE noted ' that the "A" narrow range (NR) level instrument was downscale and
that both high and low level alarms were actuated. He proceeded i-to the wide range (WR) level * recorder to verify adequate water l
- (Note: The WR level instrument is not accurate at this time because it is calibrated with no reactor recirculation However, an j
' approximate indication can be achieved.) pumps running.
'=
- -. -. .. . ... .. . . - . . ..
. - .. . . .. . . - . -. . level. There he found'the "A" channel WR also-deenergized and the "B" indicating adequate level which verified that "B" NR was indicating accurately. When water level reached , approximately 12.5 inches on the narrow range recorder, the L NSO requested permission to scram the unit. Apparently the SE ' did not respond, but continued to monitor reactor water level.
.' It is the AIT's position that once the SE had verified adequate level that he should have placed himself in a better position
to properly observe and control unit evolutions and.any new
- -
circumstances.
2.
A manual scram was not initiated when vessel level dropped b'elow technical specification action levels as required.
i While the SE was monitoring the WR level recorder, the unit operator (NS0) was monitoring the NR level instrumentation.. A half scram was received when reactor vessel water level reached +13.0 and +14.0 inches. At +12.5 inches the NSO noted that he had not received a full scram. He reported the level to the SE and requested permission to manually scram the , reactor. -The NS0 armed the manual scram push buttons and
. started announcing water level decreases every inch. Also, -during this time, the Automatic Depressurization System (ADS)-
permissive alarm was announced (this is set at the same vessel
water level as the low level scram setpoint), thus confirming that vessel water level was at or below the scram setpoint. At
approximately +8.0 inches the other reactor protective system i (RPS) channel tripped on low' water level resulting in a full scram. The NSO had requested permission to insert a manual ' scram an additional time just prior to the automatic scram.
< It is the AIT's position that the reactor should have been '. manually scrammed when vessel level fell and continued to fall.
. (as indicated on three independent level monitoring instruments) below the scram setpoint, or at least when vessel level: fell below the +11 inch LCO point in the technical specifications.
The AIT's position is based on the premise that in commercial nuclear power plants experiencing upset conditions that the operators are to act as the backup to the automatic systems.
To carry out this responsibility, the individual must be ready to
initiate the proper action with or without the permission of his supervisor.
i 3.
Licensee management did not adequately document the post trip review.
Sufficient information was available from operations personnel to make the determinations noted in Items 1 and 2 above and to at least question the operability of tne scram level switches. Also, two senior IMs, who were in the control room and had been trained on plant operations, provided additional information to the ' Resident Inspectors. The inspectors in turn requested that the ' licensee resolve the question of why the scram did not occur at [ the required setpoint.
i
26 , r-- -e - yt-- %y*ve-r--.-'t4t----- F*-**-*w**-'-'-*v- 'M-v r e- -wr v---'t-->t va-
'-n--=--
-N-n
- --'eur*-'-**s-w-
=*r' v=*---*w-m m -*mM***+er*"+=--e' - m-**--
. _ The licensee's review determined that the non safety-related NR instruments'were found to read 1.52 inches low and the safety-related trip switches (2B21-N024A-D) were found to trip between 12.6 and 14.2 inches. Any other discrepancy was attributed-to the NS0 misreading his instrumentation. Therefore, the post - trip review cormnittee did not believe a scram below the required level had occurred nor did they find fault with the operators involved, and thus made no mention of the IMs or the NRC questions to the Plant Manager as part of their recommendations to startup. The AIT is concerned that much of the review committee's rational for deciding there were no problems was not documented. ~It is further the AIT's position that on May 9, 1986 sufficient physical information did not exist to allow for the identification of the SOR differential pressure switch problem.
4.
It appears, from discussions with the day shift SE of June 1, 1986, that his knowledge of the May 9, 1986 event spurred his-action since he had a clue that_ reactor water. level may have gone below the 11 inch minimum setpoint.
The personnel on-shift at the time of the June 1,'1986 event . were not aware of the May 9,1986 event. However, as they were ' convinced that level had not gone below the 12 inch mark, the ! lack of knowledge of the May 9,1986 event in all likelihood ! would not have altered their decision.
The AIT concludes that although the May 9, 1986 event was a precursor to the' June 1, 1986 failure to scram, insufficient , ! information existed for the licensee to have concluded or further j questioned the operability of the reactor water level trip switches.
H.
Generic Aspects of SOR Switch Problem ! ' Shortly after the problem was isolated to the behavior of the SOR, Inc differential pressure switches both the licensee and the AIT set about generating a -list of switches by location, application and model. Two basic applications were identified; (1) reactor water level, and (2) system flows. The following is a list of switches by designation, description, range, setpoint, and model.
Approxi-Approxi-mate mate Switch No.
Description Range Set Pt.* Model E31-N007AA,AB RHR/RCIC Steam 20-200" +123"(AA) 103-B203 High Flow-123"(AB) E31-N007BA, BB RHR/RCIC Steam 20-200" +87" (BA) 103-B203 High Flow-87" (BB) E31-N008A,B,C,D MS Hi Flow A 100-500 111 PSID 102-B305 PSID
_ ________ E31-N009A,B,C,D MS Hi Flow B 100-500 111 PSID 102-B305 PSID . E31-N010A,B,C,D MS Hi Flow C 100-500 111 PSID 102-B305 PSID
- Differential pressure in WC unless otherwise ir.dicated.
E31-N011A,B,C,0 MS Hi Flow D 100-500 111 PSID 102-B305 PSID E31-N012AA,BA RHR Wtr Hi Flow 20-200" 121.5" 103-B203 E31-N012AB,BB RHR Wtr Hi Flow 20-200" -121.5" 103-8203-B21-N024A,B,C,D RPV Level 3 7-100" 63.78" 103-B212 B21-N031A,B,C,D RPV Level 2 20-200" 143.63" 103-B203 B21-N037AA,BA RPV Level 1 40-300" 202.16" 103-BB205 B21-N037AB,BB RPV Level 2 20-200" 145.63" 103-B203 B21-N0376A,DA RPV Level 1 40-300" 202.16" 103-BB205 B21-N037CB,DB RPV Level 2 20-200" 145.5" 103-8203 B21-N038A,B-RPV Level 3 Inter 1.
7-100" 63.78" 103-B212 E12-N010AA,BA RHR Flow DP 5-35" 5.9" 103-8202 E12-N010AB,BB RHR Flow DP 5-35" 15.3" 103-8202 E12-N010CA RHR Flow DP 5-35" 5.9" 103-B202 E12-N010CB RHR Flow DP 5-35" 15.3" 103-B202 E21-N004 LPCS Flow 5-35" 8.6" 103-B202 E22-N006 HPCS Flow 5-35" 18.1" 103-BB202 B21-N026,AB,BB RPV Level 2 30-200" 145.63" 103-BB203 B21-N026CB,DB RPV Level 2 30-200" 145.63" 103-88203 ' E31-N013AA,BA RCIC Stm Flow 20-200" +130" 103-B203 E31-N013AB,BB RCIC Stm Flow 20-200" -170" 103-8203 B21-N101B RPV Level 8 7-100" 33.27" 103-8212
-. . - _ - - _ _- - _ - -. _. . ._, _-
. The~following information was provided by SOR, INC. to the NRC for distribution in Information Notice 86-47 and is repeated here for completeness.
Plants with Similar Differential Switches
Plant Model Number Penn. Pwr. & Light /Susquehanna 103/8202 So. Cal. Edison / San'0nofre 103/B903 TVA/ Brown's Ferry 103/B212 TVA/Sequoyah 103/BB212 103/BB203 103/BB203 WPPS/WNP-2 103/BB203 GPU/0yster Creek 103/B905 103/BB212
103/B212 103/B202 ' N.E. Nuc./ Millstone 103/B903 South Texas Projects 103/BB212 103/BB003 Commonwealth Edison /LaSalle 103/B202 103/B212 103/B203 103/BB203 103/BB212 103/BB205 103/BB202 . I
. _ -. _ _ . . -- .- . - - . _. . .... --_.. - - -. _
_. _. _ _. _ _ _ _ _ _ _ . _.
_ _ ... _ . . 'IV.
ROOT CAUSE' DETERMINATION' For the purposes of. the following discussion, the event of June 1,1986 . ' will be broken down into two parts, the first part being the feedwater-transient, and the.second part being the failure of the reactor to scram.
' The. initial feedwater transient was initiated by a failure (s)'in the feedwater control system which caused then"A" TDRFP flow to increase, . resulting in an increasing reactor. vessel water level-and both turbine driven reactor feedwater pumps (TDRFP) locking up..A second feedwater transient was; initiated when the "B" TDRFP lockout was improperly reset.
When the "B" TDRFP lockout was reset, the turbine control system ramped the turbine speed (and therefore water flow) to zero.. Subsequent reviews indicate that due to a personnel error, no at. tempt was made to balance o the existing turbine speed to the controller's output prior to resetting L the lockout. Because the reactor vessel level had been in excess of the 36 inch nominal level:for some period of time, the feedwater control.. i
circuit (although unable to physically alter the turbine's speed, due to i' the' lockout) electronically lowered the speed demand signal to the turbine.
' ~Thus once the lockout was reset and the automatic system was again able ~ to physically alter turbine speed, the control _ system ramped the turbine-down to zero speed. Although the turbine and control system properly ! responded initially, there remains a question as to why the "B" TDRFP.
control system did not respond to the subsequent decrease in water level.
' The on-shift personnel regained control of the "B" TDRFP.by-pulling and . reinserting a circuit board in the feedwater control system, thereby
[ clearing the ' apparent saturation in the control circuitry.
The failure of the reactor to scram, even though reactor water level
-decreased below the prescribed set point, is attributed to the failure of ' the Static-0-Ring (SOR) differential pressure ~ switches to function. The licensee qualified the " improper" functioning in a series of' tests during l which-reactor vessel level was actually lowered and switch actuation points monitored. The initial test was conducted on June ~2,1986 with
the reactor at approximately 950'psig. The results of the test revealed-
- ! that even though the switches were calibrated to trip at approximately 13.4 inches vessel level, the actual trip points varied from 3.9 inches 1-to 10.2. inches. Although the reactor vessel Level 3 scram switches
exhibited depressed setpoints, none failed to actuate during any of the j-reactor vessel level drop tests.
' ' Testing of Static-0-Ring differential pressure switches in flow applications revealed similar results, i.e., the as-calibrated trip points bore little resemblance to the physical actuation points.
Further testing by the licensee-after the AIT was terminated, appears to - indicate while the switches do not behave as originally anticipated, they - ' do not behave in a totally erratic manner. The switches appear to undergo - a setpoint shift when exposed to static pressure, greater than that at , which they were calibrated, for some time (24 hours or less).
It also i appears that the setpoint shift has a " limit," i.e. does not continue to shift, and thereafter is reasonably predictable.
' . J
- ---..- - -. - - - - . - -. -. ... - -. . -.. - - -
,. - - . - - - . . - - - - s . - s . - V.. SUMARY t A.
Safety Significance of Event =The AIT has determined that the June 1, 1986 event was safety-significant although it presented no undue risk to the health and safety of'the public. The safety' significance can be dissected into
-three: areas: (1) As'a result of failure of the level switches to trip at the required level, the margin of. safety was reduced.' i Analysis by the General Electric Company, the nuclear steam supply vendor,-indicated that for LaSalle the peak clad temperature following an accident is relatively insensitive to the reactor level as long as the switches actuated before the water level drops below ~ L the variable leg tap; however, the susceptibility of other switch - applications including ECCS initiation and ECCS pump minimum flow valve control, to the set point shift phenomenon results in an overall reduction in the margin of safety.
(2) The chain of events
was precipitated in part by simultaneous failures of several diverse " systems;" e.g., Reactor Protection, feedwater controls, and personnel; (3) The failure of the licensee to more expeditiously verify that the level in the reactor had gone below the low level i scram setpoint without the unit scranning.
j B.
Reportability of Event Approximately 14 hours had elapsed before the-" Alert" was declared, ! per Item 13 on Table LA5-1 of the Commonwealth Edison Company
General Station Emergency Plan. - This raises the question of report timeliness; specifically why a call was not made to the Headquarters Duty Officer in the 2:00-2:30 time frame.
' - The LaSalle General Station Emergency Plan (GSEP) defines an Unusual Event as: Events in progress or have occurred which indicated a potential l degradation of the level of safety of the plant.
To assess the'"...have occurred" portion of the definition an
assessment of the data available to the shift was initiated. The ! information available to the on-shift crew at 2:00 p.m. on June 1, 1986 was as follows: , i !- 1.
Impression on narrow range recorder paper below the 11.0 inch j level.
' 2.
Startrec data indicated level had gone as low as +6.25 inches.
' 3.
Scram level switch setting per normal calibration procedure.
a.
2821-N024A - 11.89 inches vessel level equivalent ! t
1 ' . - -..-,...,. _. - - -. -. _ - _., , _. - - _,, _ - _. _., _ _..., _. - - _.. - - - - -.. -. - -,.. -,. _. -. -.. - - _. -
- -. - .. . . _ _ _ c . b.
2821-N0248 -'12.74 inches vessel level equivalent .c.
2B21-N024C - 13.86' inches vessel level equivalent d.
2821-N0240 - 12.31 inches vessel level equivalent ,1 4.
Operators on-shift at 4:30 a.m.. indicated level did not go ' below 11.5-12 inch mark on the narrow range meters.
.
Interviews with day shift personnel revealed that, around 2:00 p.m., l they.placed little reliability in the narrow range level indication and felt they couldn't rely on the scram switch settings.
- The entry in the Shift Engineer's log book, Book 106, Page 75, at 1430 hours on June 1, 1986 states: Commenced normal unit shutdown ' on U-2 per LGP-2-1. The reason for: S/D is a lack of confidence on - ' i: the reliability of the Low level scram' switches. All.2B21-N024A, B, C, and D are in CAL. A was found. out...but placed in CAL during the LIS. Commenced decreasing power with'RR at 100 MWe/hr.
- While the data available at 2:30 p.m. gave conflicting information
it.is apparent that at least the Shift Engineer and Shift. Control Room Engineer considered the reactor protection system reactor vessel level switches to be questionable.
' i i Condition 21 on GSEP Table LAS-1 states: Any other conditions of . - equivalent ma'gnitude to the criteria used to define the accident !. . category as determined by Station Director." The Unusual Event for ! Condition 21 states: " warrants increased awareness on the part of the
state and/or local offsite officials."
. After a review of the information the AIT has concluded that in hindsight a " Condition 21" Unusual Event declaration would have , ! been the prudent action for the licensee to take. However, the guidance given to the Shift Engineer, the latitude granted by Condition 21, and the lack of consistent data makes it impossible + to state unequivocally that such a declaration was required.
' Regardless of the written requirements on reportability, the AIT
, believes that the spirit of the early notification system make a ' notification the reasonable course of action when the condition of the reactor or protective systems is unknown or questionable. As such the AIT feels that a courtesy notification should have been made at approximately 2:00 p.m. on June 1, 1986 when the results of the scram , level switch calibrations and resultant discrepancy between indicated j level and switch actuation was identified.
C.
Conclusions and Recommendations '
1.
LaSalle Specific
As in Section IV, this section will deal with two separate issues: The feedwater control system and the failure of the , reactor to scram.
32 , . ... . . . . - - -
- - . . -- .- a.- Feedwater Control System: The feedwater transients were ' caused by a combination of equipment failure and personnel error. The equipment failure aspect to date is unexplained , -although it is known that the facility has experienced various problems with the system from the time it was installed.
, ' (1) EquipmentFailure(s) It is the AIT's reconnendation that the licensee ' evaluate its priority for resolving the long-standing problems associated with the Lovejoy controllers-with special attention being paid to electrical saturation - . phenomena.
(2) Personnel ~ Errors Knowledge, although effective at the time training is i given, has the potential to be lost over time.
In the i case at hand, the major failing was resetting the TDRFP lockout prior to ensuring that the controller was in concert with present operating conditions.
o While this error and the other errors discussed Section III.F may be isolated instances, they point out > the person / machine interface difficulties under adverse i conditions. Possible solutions include more on-the-job training to reenforce proper actions or modification of . the systems to make them more resistant to people induced
errors.
b.
LaSalle Unit 2 Failure to Scram
p The failure of the reactor to scram as designed can be ' attributed to the' failure of the SOR, Inc. differential pressure switches to function in a predictable manner, j as defined by the normal calibration procedure.
The AIT has concluded that the normal calibration techniques < used at LaSalle County, i.e., calibration at 0 psig static
pressure, do not verify or ensure the operability of the ! SOR differential pressure switches.
From this conclusion ' it is obvious that if the licensee plans, as they do at this time, to continue to use the switches, that a program equivalent to a new product qualification program must be , completed.
> The program as a minimum would have to address / include ' the following:
' (1) Quantification of switch behavior due to elevated static pressure.
i $ j
-, .... _. _ _.. - -.,.... _ _, _ _ - _ _, _ _. _. -., _ -,., _.,, _ _.. - - - _., -, _. _, . _ _ _ _ - _ _.,
, .. j -
(2) Quantification of switch behavior due to time at-elevated static pressure.
t (3) -Correlation between behavior during atmospheric calibration and behavior at normal operating ' conditions.
> (4) Method for detecting switch failure via normal calibration procedure.
(5) Define.and justify surveillance frequency for calibration of the switches.
(6) Evaluation of philosophy of cycling switches prior ' to taking "as found" readings.
(7) Impact on present environmental qualification of . the finding that calibration technique may not be , appropriate to define switch operability.
(8) Determination of changes in switch behavior in the long term.
' 2.
Generic Applications j On June 1,1986 three of four level sensory switches at the LaSalle County Unit 2 facility fail to trip. Subsequent tests ' , revealed that the problem was not component failure, but rather ' an operational characteristic F the switches, heretofore unknown to the. licensee, which resulted in a non-conservative set point shift.
- The level switches were a differential pressure switch in a level sensing application. 'leview of other applications revealed that the same basic switch was also used in flow , applications. Testing of the switches in the flow application _ revealed that they also exhibited the set point shift ' . characteristic when exposed to operating conditions different than those experienced during calibration.- . .. As the problems were not limited to a single switch or a single l application the AIT recommended, on Thursday, June 12 and again on Friday, June 13, 1986, through Region III to NRC ' Headquarters that plants using S0R, Inc differential pressure . switches be shutdown (The AIT did note that a number of the ' plants identified as having SOR differential pressure switches were already in a shutdown condition) pending verification that
the switches at their facility did not exhibit similar behavior , !. to those at LaSalle. An Information Notice 86-47 (Attachment 12) was issued on June 10, 1986 describing the LaSalle findings to date.
Included with the information notice was a listing, l provided by S0R, INC, of other facilities which had SOR, INC.
differential pressure switches.
i _. -.....,_.
..... - _. - _.. _ -.. _ _. _ _,. _ _ _. _, _,. _. _ _,. _. _. _.., _.. _......, _..... _,. _.,. _,... _,..,...,
Work is continuing on the issuance of a bulletin requiring specific actions to be taken by those licensee's using 50R differential pressure switches.* The AIT offers the following reconnendations based on their review of the SOR DP switch anomalies: a.
Stop cycling DP and straight pressure switches prior to calibration.
b.
Use the values recorded on the first cycle as the'"as found" conditions, c.
Perform the switch calibrations at a static pressure as close to their normal operating pressure as possible.
Alternatively, develop and verify a correlation between the set point at calibration pressure and the set point at normal operating pressure.
- IE Bulletin 86-02'" Static "0" Ring Differential Pressure Switches" was issued on July 18, 1986.
' __ ____ _ _ _ _, _ _ -. _ - --- - - - - - - -- ~-
VI Addendum To Augmented Inspection Team
Inspection Report
On August 8, 1986 LaSalle Unit 2 started up from the outage beginning on June 1,1986 due to problems with the Static-0-Ring differential switches. Between the end of the Augmented Investigation Team (AIT) Report and August 8, 1986 the licensee completed their defined testing program and implemented actions to assure that the differential pressure switches would function as required, and Regional inspectors witnessed selected portions of the testing program. In the interim, numerous meetings were held between the NRC and Commonwealth Edison o discuss testing philosophy, approach, schedule, and results. The last meeting was held in Washington D.C. on July 18, 1986. Based on the July 18, 1986 meeting and additional clarifying information provided by CECO, the Office of Nuclear Reactor Regulation provided Region III with a Safety Evaluation Report (SER) on the switches. The SER. concluded that the licensee's action plan as supplemented by specific items provided an adequate basis for the restart and short term operation of the LaSalle County Station Unit 2 until the next refueling outage. In a separate action Region III met with the licensee on July 9, 1986 to discuss actions taken by CECO in response to the June 2, 1986 Confirmatory ActionLetter(CAL). It was concluded that all items listed in the CAL had been satisfactorily addressed and completed. Based on the SER, the acceptable response to the CAL, and the recomendation of the Staff, Mr. James G. Keppler authorized restart of LaSalle County Unit 2 on August 7, 1986 (Attachment 16). ..
- - -. -.- .__ _- -.. -. - - - . _ _. .- - -- }}