05000390/LER-2003-002, Failure in Power Distribution Panel in Reactor Protection Panel 1-R-9
| ML032050283 | |
| Person / Time | |
|---|---|
| Site: | Watts Bar  |
| Issue date: | 07/18/2003 |
| From: | Lagergren W Tennessee Valley Authority |
| To: | Document Control Desk, Office of Nuclear Reactor Regulation |
| References | |
| LER 03-002-00 | |
| Download: ML032050283 (11) | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications |
| 3902003002R00 - NRC Website | |
text
Tennessee Valley Authority, Post Office Box 2000, Spring City, Tennessee 37381-2000 William R. Lagergren, Jr.
Site Vice President, Watts Bar Nuclear Plant JUL 1 8 2003 10 CFR 50.73 U.S. Nuclear Regulatory Commission ATTN:
Document Control Desk Washington, D. C. 20555 Gentlemen:
In the Matter of
)
Docket No.50-390 Tennessee Valley Authority WATTS BAR NUCLEAR PLANT (WBN) UNIT 1 -
DOCKET NO. 50-390 -
FACILITY OPERATING LICENSE PF LICENSEE EVENT REPORT (LER) 50-390/2003-002 This submittal provides Licensee Event Report 390/2003-002.
This LER addresses an event that occurred on May 21-22, 2003, which resulted in one channel of Reactor Protection System being inoperable for a period of time longer than allowed by the plant Technical Specifications. This event is being reported under 10 CFR 50.73(a)(2)(i)(B).
There are no regulatory commitments in this submittal.
If you have any questions about this change, please contact P. L.
Pace at (423) 365-1824.
Sincerel W. R. Lagerg Enclosure cc:
See page 2 PNmd on enyced pa
U.S. Nuclear Regulatory Commission Page 2 JUL 18 2D3 cc (Enclosure):
NRC Resident Inspector Watts Bar Nuclear Plant 1260 Nuclear Plant Road Spring City, Tennessee 37381 Mr. K. N. Jabbour, Senior Project Manager U.S. Nuclear Regulatory Commission MS 08G9 One White Flint North 11555 Rockville Pike Rockville, Maryland 20852-2738 U.S. Nuclear Regulatory Commission Region II Sam Nunn Atlanta Federal Center 61 Forsyth St., SW, Suite 23T85 Atlanta, Georgia 30303
Abstract
On May 21, 2003, WBN Unit 1 was operating at 100 percent reactor power. At approximately 2153 EDT, the WBN operators were alerted to potential instrument channel lil trouble when control room alarms were received. This equipment provides channel IlIl process instrumentation of the engineered safety features actuation subsystem portion of the Reactor Protection System. The operators were able to clear the alarms and plant continued operation. However, throughout the early morning of May 22, 2003, alarms indicating the same trouble continued.
Maintenance personnel performed troubleshooting and Identified that a faulty power distribution panel located within Reactor Protection Set Channel Ill, Panel 1-R-9 was the source of the problem. The faulty power distribution panel was replaced and Panel 1 -R-9 was returned to service.
Throughout this event, the operators did not identify Panel 1 -R-9 as inoperable since after clearing the alarms, plant parameters were within range (with an exception of entry Into LCO 3.4.1 RCS Pressure, Temperature, and Flow DNB Urits due to reactor coolant pressure less than 2214 for a short period) and plant operation continued. However, it was concluded after the event that the panel should have been declared inoperable and Technical Specifications actions completed within the 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> time requirements. This event is being reported under 10 CFR 50.73 (a)(2)(i)(B) as operation prohibited plant technical specifications.
The cause for this event was that there was no specific guidance to assist the operators in diagnosing a panel malfunction of the sort that would also affect operability of the Reactor Protection System panel. Corrective actions include the development of an Abnormal Operating Procedure to address panel malfunctions. Additionally, the initial maintenance troubleshooting activities identified the problem to be In the data link handler portion of the rack which would not affect operability.
flfl T..W*
'
I' SW Ii r-n rus
- - o t -.
,lU.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACIUTY NAME (11l DOCKET LER NUMBER (6)
PAGE (31 05000 YEAR SEQUENTIAL REVISION NUMBER 2
of 9
Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT (f more space is required, use addtional copies of NRC Form 366A) (117)
I. PLANT CONDITION(S)
On May 21, 2003, at approximately 2153 Eastern Daylight Savings Time (EDT), WBN Unit 1 was operating at 100 percent reactor power. The Reactor Coolant System (RCS) (Energy Industry Identification System (EIIS)
Code AB) pressure was approximately 2235 psig and RCS Tavg was approximately 587 degrees F.
II. DESCRIPTION OF EVENT
A. Event At approximately 2153 EDT on May 21, 2003, the WBN operators were alerted to potential instrument channel IlIl trouble when a number of alarms (EIIS Code ALM) were received in the main control room. Indications observed by the operators were RCS Loops 1, 2, 3, and 4 flow low and pressurizer level low, pressurizer pressure low, pressurizer pressure low Si, pressurizer level high, Refueling Water Storage Tank (RWST) Level Lo & Lo-Lo which were associated with Instrument Panel 1 -R-9. Additionally, letdown was determined to be isolated. The operators entered procedure, AO1-20, Malfunction of Pressurizer Level Control System." The operators responded by dispatching an auxiliary unit operator and a senior reactor operator to the auxiliary Instrument room, where they were met by maintenance personnel who had also responded due to their monitoring of the operators radio frequency. Upon observing that letdown had Isolated, the operators Isolated charging by closing valves (EIIS Code FCV) 1-FCV-62-90 and 1 -FCV-62-91 in accordance with 92C of Annunciator Response Instruction (ARI)-88-94, Reactor Coolant System," The operator then, in accordance with AOI-20, reestablished charging and letdown.
At 2203, when it was observed that RCS pressure had decreased below the 2214 psig value, the operators entered Technical Specification 3.4.1, RCS Pressure, Temperature, and Flow DNB Limits.0 At 2210, control board indications appeared to be normal. The alarms had been cleared except for Protection Set IlIl channel failure." At 2218, LCO 3.4.1 was exited since RCS pressure was now observed to be greater than 2214 psig.
At 2220, procedure AOI-20 was exited since plant conditions had become stable. At 2244, a priority work order, was initiated to address the equipment problem. At 0106 on May 22,2003, another burst of alarms were received as before. However, normal plant parameters remained stable. At 0307, maintenance personnel were able to begin troubleshooting which indicated that trouble appeared to be with I of 2 data link handler cards located in 1 -R-9 panel. Previous experience of problems with these cards had shown that this type problem did not affect racks' functionality. It was determined that the work order was to be planned to replace the cards and to reset the local trouble lights on the following day shift. At this point, the operators had no evidence that any process indications had failed. Based on the information that the failure was in data link handler, the operators concluded that no operability Issues existed.
At 0514, another burst of alarms were again received. The operators cleared the alarms as before. Plant parameters again appeared normal.U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME 11 DOCKET LER NUMBER 6)
PAGE 131 05000 YEAR SEQUENTIAL REVISION l
NUMBER l
3 of 9
Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT (If more space is required, use addtional copies of NRC Form 366A) 1171 At 0701, another burst of alarms were received. Plant was still stable, no controlling channels were affected.
Again, at 0708, another burst of alarms as before was received. With these alarms and with the continued repetitive nature of these alarms taken into consideration, the operators declared Panel 1 -R-9 as inoperable and entered the following Limiting Condition for Operation (LCOs):
LCO No.
3.3.1 3.3.2 3.3.3 3.6.6 Condition/Description N (RCS Lo Flow)
W (OT delta T & low press)
X (Pressurizer Level Hi)
D (Lo Pressurizer Press)
L (one P-1I Interlock Inoperable)
A (Wide Range Pressure and pressurizer recirculation, train A, fails open)
A (Containment Spray recirculation, train A fails open)
Action and Completion Time N.l Place channel In Trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> W.1 Place channel in Trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> X.1 Place channel in Trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> D.1 Place channel in Trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> Li1 Verify interlock Is in required state for exiting conditions within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.
A.1 Restore within 30 days A.1 Restore with 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> At 0730, the operators completed action L.1, of LCO 3.3.2 by verifying that P-11, "Pressurizer Interlock, was in Its required state. Between 0730 and 1000, three additional bursts of alarms as before were received. At 0934, plant maintenance determined, through troubleshooting, that the problem was in 1 -R-9 power distribution panel (EIIS Code PL) and not the data link handler boards. At 1000, Panel 1-R-9 was downpowered for replacement of power distribution panel within panel 1 -R-9. The operators verified that the loops and instruments for this panel were in their proper condition as required by technical specifications. At 1040, Plant Maintenance completed the required replacements. At 1223, the post maintenance test was completed satisfactorily. The operators then exited the technical specifications discussed and listed above. Panel 1 -R-9 was then returned to service.U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME 11)
DOCKET LER NUMBER 16)
PAGE 3) 05000 YEAR SEQUENTIAL REVISION NUMBER l
4 of 9 Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT If more space is required, use additional copies of NRC Form 366A (1171 B. Inoperable Structures, Components, or Systems that Contributed to the Event:
There were no inoperable systems that contributed to this event.
C. Dates and Approximate Times of Major Occurrences
Time Event 05/21/03 21:53 21:53 21:54 21:59 22:03 22:10 22:18 22:20 22:44 05/22/03 01:06 01:09 Received burst of alarms for Protection Set IlIl trouble. These indications were associated with Instrument Panel 1-R-9. Entered AO1-20.
Dispatched an AUO and a SRO to auxiliary instrument room, maintenance personnel monitoring frequency 1 radio overheard dispatches responded to the auxiliary instrument room.
Due to letdown isolating, isolated charging by closing 1 -FCV-62-90, and -91 per ARI-92C.
Per AOI-20, selected away from 1 -LT-68-320, pressurizer controlling level change, reestablished charging, and letdown.
Entered 3.4.1 DNB limits action A restore in 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> due to RCS pressure being <2214 during channel IlIl problems with letdown isolating and heaters off. No loss of safety function exists for the entry into LCO 3.4.1, ODNB."
Control board indications appear to be normal, all alarms except for Protection Set IlIl channel failure have cleared for pressurizer controlling level channel now selected for 1 -LT-68-339.
Exited LCO 3.4.1 RCS pressure >2214.
Held crew brief on AOI-20 entry. Plant conditions are stable exiting AO1-20.
Priority 2 Work Order 03-10878-000 written to repair 1-R-9 alarms. The following alarms in 1-R-9 are channel set failure and LPS communication failure.
Received another burst of alarms they were same alarms on Channel IlIl as before.
Alarms came in again and cleared several times finally cleared out except Protection Set IlIl Channel Failure. Normal plant parameters remained stable.U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME 111 DOCKET LER NUMBER 161 PAGE 31 05000 YEAR SEQUENTIAL I REVISION NUMBER 5
of 9 Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT (If more space is required, use add donal copies of NRC Form 366A) (171 Time 03:07 Event Held pre-job brief for 1-R-9 Channel IlIl troubleshooting.
04:04 05:14 07:01 07:08 07:30 08:15 08:45 09:26 09:27 09:34 Troubleshooting data from 1 -R-9 rack problem indicates we have trouble with 1 of 2 DLH (Data Link Handler) cards. Package will be planned to replace them and to reset the local trouble lights on day shift. Indications are that no process Indication have failed.
Received Channel IlIl burst of alarms again, clear plant parameters appear normal.
Received burst of alarms related to the 1 -R-9 rack failure. Plant is still stable, no controlling channels were affected.
Received burst of alarms related to the 1-R-9 rack failure. Plant is still stable, no controlling channels were affected. Due to repetitive alarms associated with rack 9, the rack has been declared inoperable. Entered LCO 3.3.1 condition N (RCS Lo flow):
place channel in trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />; condition X (Pressurizer level Hi): Place channel in trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />; condition W (OT delta T & low press): place channel in trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />; Entered 3.3.2 condition D (Lo Pressurizer Press): place channel In trip within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />; condition I (one P-11 interlock Inoperable): Verify interlock is in required state for exiting conditions within 1 hr. Also entered LCO 3.3.3 action A (Wide range pressure and pressurizer level): restore within 30 days and Entered LCO 3.6.6 action A (containment spray recirculation, train A, fails open): restore within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />.
Completed LCO 3.3.2 action : verified P-1I in required state.
Received burst of alarms related to the 1-R-9 rack failure. Plant is still stable, no controlling channels were affected.
Held pre-evolution briefing with Instrument Mechanics for troubleshooting on rack 9.
Received burst of alarms related to the 1 -R-9 rack failure. Plant is still stable, no controlling channels were affected Received burst of alarms related to the 1 -R-9 rack failure. Plant is still stable, no controlling channels were affected.
Instrument Mechanic reports that 1 -R-9 troubleshooting determines the failure is in the power distribution panel, not the data link handler cards.
NBC FORM 368A U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME 111 DOCKET LER NUMBER (63 PAGE 131 05000 YEAR SEQUENTIAL I REVISION l
NUMBER l
6 of 9
Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT (If more space is required, use addotional copies of NRC Form 366A) 1171 Time Event 05/22/03 09:57 Completed prejob briefing on performance of IMI-99.009 to allow downpowering 1-R-9 in accordance with IMI-99.009. This downpowering action placed affected channel III bistables in trip or bypass position.
10:00 10:40 11:00 11:05 12:23 1 -R-9 is downpowered. All 1 -R-9 loops and instruments are in the proper condition required by technical specifications.
Instrument Mechanic reports new power distribution panel is installed and is to come to main control room for a prejob briefing for powerup and Post Maintenance Test performance.
Completed prejob briefing for powering up power distribution panel. Precautions and Technical Specifications discussed. Covered post maintenance test items and actions required If post maintenance test fails.
Entered LCO 3.0.5 to perform Post Maintenance Testing on 1 -R-9.
Post Maintenance Tests completed satisfactorily following the replacement of the power supply distribution panel on rack 1-R-9. Exited LCO's 3.05,3.3.1, 3.3.2, 3.3.3 and 3.6.6 associated with Work Order 03-10878-000. 1 -R-9 is returned to operable status.
D. Other Systems or Secondary Functions Affected
The loop and equipment associated with Channel IlIl from Panel 1-R-9 was affected by this equipment failure.
E. Method of Discovery
The equipment failure described by this LER was self revealing by the number of alarms received by the main control room.
F. Operator Actions
See Section II.A, 'Description of Event,' above for operator actions during the event. However, based on information discovered during troubleshooting, the equipment that failed in 1-R-9 affected operability of the panel. Therefore, panel 1-R-9 should have been declared inoperable earlier and actions required by the Technical Specifications taken and completed for the affected channels within the 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> completion times. However, this did not occur and the actions were not completed within the required time frames. There, this condition is being reported under 10 CFR 50.73 (a)(2)(i)(B) as operation prohibited by Technical Specifications.
-U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME II DOCKET LER NUMBER 16)
J PAGE (3) 05000 YEAR SEQUENTIAL REVISION I
Watts Bar Nuclear Plant 06000 - 390 2003 002 00 TEXT (if more space is required, use addtional copies of NRC Form 366A) 117)
G. Safety System Responses:
The systems and the affected channel loops responded as would be expected given the failure identified in Panel 1-R-9.
Ill. CAUSE OF THE EVENT A. Immediate Cause:
The immediate cause of the event is the failure of the power distribution panels located within Panel -R-9.
B. Cause
The cause for failing to recognize the potential noperability of the Panel 1 -R-9 was that there was no specific guidance to assist the operators in diagnosing Eagle 21 malfunction of the sort that would also affected operability of the Reactor Protection System (RPS) (EIIS Code JC) rack. During the investigation for this event, it was found that WBN's sister plant, Sequoyah Nuclear Plant, had developed an Abnormal Operating Procedure (AOP) to address Eagle 21 malfunctions.
C. Contributing Factors There were no contributing factors identified in this event.
IV. ANALYSIS OF THE EVENT
The purpose of the RPS is to provide automatic protection against unsafe and Improper reactor operation during steady-state and transient power operations and to provide initiating signals to mitigate the consequences of faulted conditions. The RPS provides protection during all reactor modes of operation except the refueling mode.
The RPS has no defined operating modes although the system's logic considers the reactor power level in determining whether a faulted condition exists or not.
The RPS provides redundant (one out of two, two out of three or two out of four) instrumentation channels for each protective function and one out of two logic train circuits These redundant channels and trains are electrically isolated and physically separated from each other. Any single failure within a channel or train does not prevent protective system action when required. Loss of input power, the most likely mode of failure, to a channel or logic train will result in a signal calling for a trip, with the exceptions of containment spray and switchover from injection mode to recirculation mode following a safety injection. The channels for these functions are energized to trip to avoid spurious actuations.
The RPS is composed of two subsystems, the Reactor Trip subsystem (EIIS Code JC) and the Engineered Safety Features Actuation Subsystem (ESFAS) (EIIS Code JE). The reactor trip subsystem automatically keeps the reactor operating within a safe region by shutting down the reactor whenever the limits of the region are approached. The engineered safety features actuation subsystem uses selected plant parameters, determines whether or not predetermined safety limits are being exceeded and, they are, combines the signals into logic matrices sensitive to combinations Indicative of primary or secondary system boundary ruptures. Once theU.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME ElI DOCKET LER NUMBER 16)
PAGE 13) 05000 YEAR SEQUENTIAL REVISION NUMBER 8
of 9
Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT (If more space is required, use additional copies of NRC Form 366A) 117) required logic combination is completed, the system sends actuation signals to the appropriate ESF components.
The reactor trip portion of the RPS contains circuitry consisting of two to four redundant channels which monitor various plant parameters, and logic circuitry consisting of two redundant logic trains, which shall receive inputs from the process protection and NIS channels to complete the logic necessary to automatically open the reactor trip breakers.
The ESFAS portion of the RPS consists of two discrete portions of circuitry: (1) the process instrumentation portion consisting of three to four redundant channels per parameter or variable to monitor various plant parameters such as the Reactor Coolant System and steam system pressure and temperatures and containment pressures; and (2) a logic portion consisting of two redundant logic trains which receive inputs from the process protection channels and perform the logic needed to actuate the ESF. Each train is capable of actuating the required ESF equipment. The intent is that any single failure within the ESFAS shall not prevent system action when required.
The equipment failure for this event involved Reactor Protection Set Channel IlIl (Panel 1-R-9) which houses a portion of the Channel IlIl process instrumentation of the Reactor Trip and ESFAS portion of the RPS.
Specifically, the equipment failure involved the power distribution panel which supplies, in part, the Loop Calculation Processor (LCP), which performs the effected channel calculations. The LCP set the comparator outputs which generate the appropriate trip outputs. When the power to the LCP cycled off and on the LCP would sometimes reset during a reboot which caused anomalous indications in the main control room for a short time. However, once the panel completed its reboot sequence and the alarms cleared, plant parameter indication would again be normal and the plant continued to operate within required parameters. The remaining channels of the RPS remained unaffected by this condition.
V. ASSESSMENT OF SAFETY CONSEQUENCES
Based on the discussion In Section IV above, there was no safety significance to this event.
VI. CORRECTIVE ACTIONS
A Immediate Corrective Actions:
Refer to Section II, Description of Event* for discussion of the actions taken. The failed power distribution panel was replaced and the equipment associated with 1 -R-9 was returned to service.
B. Corrective Actions to Prevent Recurrence:
The following actions are tracked under TVA's Corrective action program and therefore not consider to be regulatory commitments:
TVA will evaluate the SQN's AOP for applicability to WBN and, based upon this evaluation, develop a WBN procedure to address Eagle 21 malfunctions.
NRC FORM 356A U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT TEXT CONTINUATION FACILITY NAME iI DOCKET LER NUMBER 16l PAGE 13) 05000 YEAR SEQUENTIAL REVISION NUMBER l
9 of 9
Watts Bar Nuclear Plant 05000 - 390 2003 002 00 TEXT (if more space is required, use adctional copies of NRC Form 366A) 1171
VII. ADDITIONAL INFORMATION
A Failed Components:
The failed component was standard Eagle-21 Power Distribution Panel II Assembly, Group I, Drawing 2005E38 series, Schematic Drawing 3D21738, Rev 2.
B. Previous LERs on Similar Events:
A search of the previous WBN LERs was performed. There has been no previous LER concerning the failure of the power distribution panels in the Reactor Protection Sets.
C. Additional Information
None.
D. Safety System Functional Failure Consideration:
This event is not considered a safety system functional failure in accordance with NEI 99-02. The functional capability of the overall system was not jeopardized. In addition, the operators concluded that no loss of safety function existed for the entry Into LCO 3.4.1 based on a lack of an active failure since there was no indication of failure of the instrument loops but on an undetermined failure within the protection set 1 -R-9.
E. Loss Of Normal Heat Removal Consideration:
This event is not considered a scram with loss of normal heat removal.
Vil. COMMITMENTS None.