05000529/LER-1998-003, :on 980113,discovered That One Channel of RWT Level Sys Had Failed High.Caused by Water Intrusion Into Electrical Termination Pull Box.Weep Holes Were Drilled Into Bottoms of Pull Boxes Nearest Level Transmitters
| ML17313A576 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 09/08/1998 |
| From: | Marks D ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR |
| To: | |
| Shared Package | |
| ML17313A575 | List:
|
| References | |
| LER-98-003, LER-98-3, NUDOCS 9809140262 | |
| Download: ML17313A576 (12) | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i) 10 CFR 50.73(s)(2)(ii) 10 CFR 50.73(a)(2) |
| 5291998003R00 - NRC Website | |
text
LICENSEE EVENT REPORT (LE FACIUTYNAME(1)
DOCKET NUMBER (2)
PAGE (3)
Palo Verde Unit 2 TLE (4)
Refueling water tank level channel failure due to water intrusion 0
5 0
0 0
5 2
9 1oFO EVENTDATE 5 LERNUMBER S)
REPORT DATE 7 OTHER FACIUTIES INVOLVED 3 MONTH DAY YEAR YEAR SEQUENTIAL REVISION NUMBER NUMBER M
H T NUMBERS 0
5 0
0 0
0 1
1 3
OP RATING 0
0 3
0 1
9 8
9 8
IS REPORT IS SUBMiTTEO PURSUANT TO 0
8 9
8 0
9 N/A THE REQUIREMENTs oF 10 cFR 4: (crick one or mors of Ihe folkNrng) (11) 0 5
0 0
0 MODE (9)
POWER LEVEL(10)1 P
P NAME 20.402(b) 20.405(a)(1)(i) 20.405(a)(1)(i) 20.405(a)(1)(ii) 20.405(s)(1)(iv) 20.405(s)(1)(v) 20.405(c) 50.35(c)(1) 50.35(c)(2) 50.73(a)(2)(i) 50.73(aX2Xi) 50.73(s)(2)(ii)
LICENSEE CONTACT FOR THIS LER (12) 5O.73(s)(2)(N) 50.73(sX2Xv) 50.73(a)(2Xvi) 50.73(a)(2)(vga(A) 50.73(a)(2)(YE)(B) 50.73(a)(2Xx)
ELEPHONE NUMBER 73.71(b) 73.71(c)
OTHER (Specify in Abstract bekxv and)0 Texl, NRC Form Daniel G. Marks, Section Leader, Nuclear Regulatory Affairs EA CODE 6
0 2 3,9 3
6 4 92 COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBEO IN THIS REPORT (13)
CAUSE
SYSTEM COMPONENT MANUFAC-TURER REPORTABLE TO NPRDS
CAUSE
SYSTEM COMPONENT MANUFAG TURER REPORTABLE TO NPRDS B
B Q L T R
3 6
9 SUPPLEMENTAL. REPORT EXPECTED (14)
X NO YES (Ifyes, complete EXPECTED SUBMISSION DATE)
EXPECTED SUBMISSION DATE (15)
MONTH DAY YEAR BSTRACT (Limrtto 1400 spaces, Le.. spproematety fdteen angle-space typewntten Ines) (16)
On January 13r 1998, at approximately 1945 MST, Palo Verde Unit 2 was in Mode 1
(POWER OPERATION), operating at approximately 100 percent power when control room personnel discovered that one channel of the refueling water tank level system had failed high.
A revi,ew of historical computer data for this channel determined the failure had actually occurred at 1219 MST on the same day.
The other three channels were OPERABLE and would have initiated any required recirculation actuation siqnal since the syst: em uses a
2 out'of 4 logic.
Technical Specifications required the failed channel be placed in bypass with
'n one hour however, it was not recognized that the channel had failed until 1945 MST.
The failed channel was bypassed at 1950 MST.
The cause of the channel failure was determined to be water intrusion into an electrical termination pull box.
As corrective action weep holes were drilled into the bottoms of the pull boxes nearest the level transmitters in all four channels in all three units at Palo Verde. In addition, the flexible conduits were sealed between the pull boxes and the transmitters.
The delay in submitting this LER was due to not recognizing, at the time of discovery, that firm evidence existed to determine the channel had failed prior to discovery and that the TS allowed outage time had been exceeded.
A previous similar event was reported in LER 528/98-001-00.
9809140262 17)80908 PDR ADOCK 05000529 S
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITYNAME Palo Verde Unit 2 DOOKETNUMBER LER NUMBER SEQUENTIAL REVISION NUMBER NUMBER PAGE EXT 0
5 0
0 0
5 2
9 9
8 0
0 3
0 1
0 2 of 0 6
REPORTING REQUIREMENT:
This LER (529/98-003-00) is being written to report an event that resulted in Palo Verde Unit 2 operating in a condition prohibited by the plant's Technical Specifications, as specified in 10CFR50.73(a)
(2) (i)(B).
Specifically, on January 13, 1998, at approximately 1945
- MST, Palo Verde Unit 2 was in Mode 1
(POWER OPERATION), operating at approximately 100 percent power when control room personnel (utility-licensed operator) determined that one channel of the refueling water tank (RWT)
(BQ) level system bad failed high.
A review of historical computer data for this channel resulted in firm evidence that the failure occurred at 1219 on the same day.
Technical Specification 3.3.2 Action 13 required the INOPERABLE channel be placed in a tripped or bypassed condition within 1-hour.
- However, the failed channel was not discovered until 1945 the same day.
This failure would have prevented the channel from performing its safety function of automatic initiation of a recirculation actuation signal when the refueling water tank decreased to a preset level during accident conditions.
The other three channels were OPERABLE and would have initiated the required'ctuation signal since the system uses a
2 out of 4
"logic. Control room personnel placed the failed channel into bypass at 1950 on January 13, 1998 to comply with technical 'specifications.
2.
EVENT DESCRIPTION
On January 13, 1998,at 1945 following shift turnover, a Reactor Operator (utility-licensed operator) noted that the RWT level indication on the main control board for channel C was failed high to 100%.
The other three channels were indicating approximately 88% level.
A review of historical computer data indicated the instrument had actually failed during the previous shift at 1219.
The Plant Protection System (PPS)
(JC) trip for the failed channel, of the Recirculation Actuation Signal (RAS),
was bypassed at 1950 to comply with Technical Specification (TS) 3.3.2 Action Statement 13.
This Action Statement required the INOPERABLE channel be bypassed or tripped within one hour.
TS Surveillance Requirement 4.3.2.1 required a channel check of the RAS instrumentation be completed once each shift. This check is completed between the hours of 0800-1100 for day shift and 2000-2300 for night shift with an acceptance criteria of less than or equal to 3% deviation between OPERABLE channels.
The day shift had already completed the channel check when the channel failed.
The night shift identified the failure during routine control board monitoring prior to performing the next scheduled
\\
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION ACIMTYNAME Palo Verde Unit 2 DOCKET NUMBER LER NUMBER SEOUENTIAL YEAR REVISION NUMBER NUMBER PAGE 0
5 0
0 0
5 2
9 9
8 0
0 3
0 1
0 3 of 0 6
EXT shift channel check.
There were computer-generated alarms associated with the RWT level indication circuit starting at 1121 on January 13, 1998
- however, the operators did not recognize that a potential problem existed.
It should be noted that for computer generated
- alarms, the computer prints the alarm on the control room alarm typer and displays the alarm on a control room computer screen.
- However, no audible signal is given that an alarm has been generated and no operator action is required to acknowledge the alarm.
The source of the failure was determined to be water intrusion into the conduit and pull box for the level transmitter, presumably rainwater from recent storms.
A significant amount of water was found within the conduit and pull boxes associated with transmitter 2J-CHC-LT-203C.
The water within the transmitter field termination side caused the loop current to fail to the equivalent of a high level signal (i.e. greater than 20 mA) and the display for the loop failed to 100%.
This is a non-conservative failure outcome since the high level signal in this channel would have prevented an automatic RAS initiation in channel C should actual RWT level have reached the low level setpoint during a design basis
'event.
This failure would not have affected the other three channels or the capability of the automatic RAS actuation at the system level. This was a single failure within a four-channel logic system, therefore all safety functions would have been satisfied.
This failure would not have affected the manual actuation capability of the RAS logic in channel C.
When this condition was evaluated for transportability, the inspection of the other RWT transmitters in all three units determined that a
significant amount of water was also found in the pull box associated with channel A for Unit 2.
The transmitter for channel A did not have any standing water within it due to the physical differences in routing of the flexible conduit for the channel A transmitter.
No water was found in the remaining transmitters.
It was not recognized by Regulatory Affairs personnel (utility-non-licensed), at'he time of discovery, that firm evidence existed to determine the channel had failed prior to discovery and that the TS allowed outage time had been exceeded.
The event was recognized as being reportable when Regulatory Affairs personnel were reviewing the investigation report.
The delay in submitting this LER was discussed with the senior NRC Resident Inspector who contacted the Acting Branch Chief at the Region IV NRC office per NUREG-1022, Rev.
1 paragraph 5.1.1.
There were no safety system actuations and none were required.
l
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION AGILITYNAME Palo Verde Unit 2 DOCKETNUMBER YEAR LER NUMBER SEQUENTIAL REVISION NUMBER NUMBER PAGE EXT 0
5 0
0 0
5 2
9 9
8 0
0 3
0 1 04of06 ASSESSMENT OF THE SAFETY CONSEQUENCES AND IMPLICATIONS OF THIS EVENT:
The design of the RAS initiation requires that a single failure will not
cause
nor prevent a necessary actuation.
Since there are four channels of RWT level instrumentation and the logic for initiation uses a
2 out of 4
- scheme, the failure of channel C would not have caused or prevented a
RAS.
In addition, Technical Specification 3.3.2 Action 13 allows continued operation with one INOPERABLE RAS channel until the next COLD SHUTDOWN.
Although the failed level channel was discovered during routine control board monitoring, the condition would have been identified during the next performance of the RWT level channel check.
The acceptance criterion used for this check is less than or equal to 3% deviation between OPERABLE channels.
The failed channel was indicating approximately 12% deviation from the other channels and therefore would have been discovered.
The event did not result in any challenges to the fission product barriers or result in any release of radioactive materials.
Therefore, there were no adverse
safety consequences
or implications as a result of this event.
This event did not adversely affect the safe operation of the plant or health and safety of the public.
4.
CAUSE OF THE EVENT
The root cause 'of the level instrument failure was a non-conformance created from original construction (cause code:
B).
The RWT level transmitter conduit and pull boxes were not installed in accordance with design drawings.
The conduit and tray standard design anticipates water in-leakage may be possible and therefore required sealant be applied to the flexible conduit leaving the pull boxes.
In addition, the standard design for any electrical pull box, whether indoors or outdoors with very few exceptions, requires weep holes/drain holes be drilled into the bottom of the box closest to the end device, in this case the level transmitters.
A root cause investigation was conducted and determined that the cause of the delay in placing channel C
RWT level in bypass was cognitive personnel error (SALP Cause Code:A) in that control room personnel did not maintain an adequate degree of attention to the control room alarm screen and alarm typer.
There was sufficient information available in the control room to determine the transmitter had failed if control room personnel had noted the alarm on the alarm screen or alarm typer and investigated the cause of the alarm.
There were no unusual characteristics of the work location (e.g.
- heat, noise) that directly contributed to the error.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITYNAME Palo Verde Unit 2 DOCKET NUMBER LER NUMBER SEQUENTIAL REVISION NUMBER NUMBER PAGE EXT 0
5 0
0 0
5
- 2. 9 9
8 "0
0 3
0 1 05of06 5.
STRUCTURES,
- SYSTEMS, OR COMPONENTS INFORMATION:
The RWT level transmitter is a Rosemount, model 1153DB5PA.
The cause of failure of the channel C level instrument was determined to be water intrusion into the conduit and pull box for the level transmitter, presumably rainwater from recent storms.
A significant amount of water was found within the conduit and pull boxes associated with transmitter 2J-CHC-LT-203C.
The water within the transmitter field termination side caused the loop current to fail to the equivalent of a high level signal (i.e. greater than 20 mA) and the display for the loop
'failed to 100%.
This is a non-conservative failure outcome since the high level signal in this channel would have prevented an automatic RAS initiation in channel "C" should actual RWT level have reached the low level setpoint during a design basis event.
This failure would not have affected the other three channels or the capability of the automatic RAS actuation at the system level. This was a single failure within a four-channel logic system, therefore all safety functions would have been satisfied.
This failure would not have affected the manual actuation capability of the RAS logic in channel C.
The channel C level instrument was determined to have failed high at 1219 on January 13, 1998 and was discovered at 1945 on the same day whe'n control room operators observed a level deviation with other similar channels.
The instrument channel was placed in bypass at 1950 the same day, repaired, and was returned to OPERABLE status at 2141 on January 15, 1998.
There are no indications that any structures,
- systems, or components were inoperable at the start of the event that contributed to 'this event.
No failures that rendered a train of a safety system inoperable were involved.
No failures of components with multiple functions were involved.
6.
CORRECTIVE ACTIONS TO PREVENT RECURRENCE:
A root cause determination investigation of the level instrument failure was conducted in accordance with the APS Corrective Action Program.
Actions to prevent recurrence were completed 'and consisted of drilling weep holes into the bottoms of the pull boxes nearest the RWT level transmitters for all four channels in all three uni.ts at Palo Verde.
In
- addition, the flexible conduits were sealed between the pull boxes and the transmi'tters.
This work was completed in accordance with design.
No further corrective actions are planned.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITYNAME Palo Verde Unit 2 DOCKET NUMBER LER NUMBER SEQUENTIAL REVISION NUMBER NUMBER PAGE 0
5 0
0 0
5 2
9 9
8 0
0 3
0 1
0 6 of 0 6
EXT As corrective action for the failure to promptly recognize the condition, the applicable Control Room personnel were coached.
Cognitive personnel errors that are the result of mental lapses are not normally correctable with revised procedures or additional training.
No further action is required.
7.
PREVIOUS SIMILAR EVENTS
LER 528/98-001-00 reported a similar condition in which the 1-hour TS action requirement to bypass an INOPERABLE instrument channel was not complied with. Corrective action for that event would not have prevented the condition being reported in this LER since the steam generator low setpoint condition occurred on January 30, 1998, after the RWT level channel failure had occurred.
A similar condition of water intrusion occurred in Unit 3 in 1993 on the channel C
RWT level instrument.
- However, the condition in that channel only was corrected, resulting in a missed opportunity to identify and correct similar problems in all RWT level transmitter junction'boxes.
No LER was submitted and none was required.
8.
ADDITIONAL INFORMATION
None