IR 05000341/1989036
| ML20011E418 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 02/01/1990 |
| From: | Axelson W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML20011E415 | List: |
| References | |
| 50-341-89-36, NUDOCS 9002130381 | |
| Download: ML20011E418 (11) | |
Text
.c
_.-
'
..
....
ff
' -
.,. a n_-
u
.
,
.
.
-
U.S. NUCLEAR REGULATORY COMMISSION REGION !!!
l i
Report No. 50341/89036(DRP)
j l
Docket No. 50-341 License No. NPF-43 Licensee: The Detroit Edison Company I
6400 North Dixie Highway i
Newport, MI 48166 j
Facility Name: Fenni 2 Nuclear Power Station
'
Inspection At: Fermi 2 Site, tjewport, Michigan Inspection Conducted: December 13 through December 28. 1989 In0pectors:
W. G. Rogers
M. J. Farber
RWden Approved W.L.Axelson,Ibief J/n//b Reactor Projects Branch 2 Date Inspection Summary Inspection on December 13 through December 28, 1989 (Report No. 50-341/89036(DRP))
!
Areas Inspected: 5pecfal unannounced inspection by the resident and a regfonal Inspector frto.the circumstances surrounding the improper installation of wide
'
range reactor water level transmitter B21N0810. No Safety Issues Management
. System (SIMS) items were closed.
-
I, Results:
In.the area inspected, violations were identified relating to fnadequate installation, inspection, and calibration procch res; failure to i
complete required LCO actions in Modes 2 and 5; and failu N to complete a required LCO action upon discovery of the inoperable transmitter.
The root causes of this incident were multiple and involve several areas including; inadequate instructions to the Instrument and Control (!&C)
repairmenandQualityControl(QC) inspector,failuretofollowfundamental
-
skill.of the craft on the part of these workers, lack of attention to detail, poor human factors engineering relevant to the original design of non-standard tubing configuration on the same instrument rack with standard
'
6I.
.
Cl
. -
-
.. -.. - - -.. - -
.
-
. -
.
_
gy
-
,,
.
..
.
,
't
'
-.
,
tubing configuration, poor lighting and labeling on the five-valve manifold
,
and its associated piping, radioactive contamination of the workplace, and
,,
incorrect instructions to the post-modification inspector.
The inspection disclosed a weakness in the development of specific maintenance instructions, and quality control inspection plans, i-r -.
?
!
.
'I
.
4'
,
I
. -
s
r
..
,
.
.
s j
0.[ TAILS
'
1.
Persons Contacted The Detroit Edison Company
-). Anthony, Licensing
- R. Matthews, Asst. Supervisor, Maintenance and Modifications E. Nicolite, General Supervisor, I&C Maintenance
- J. Walker, General Supervisor, Nuclear Engineering
- J. Wald, Supervisor, Quality Engineering t
- R. McKeon, Operations Superintendent
- K. Sessions, Quality Assurance
'
'H. LeCompte Quality Assurance
- D. Gipson, Plant Manager
- W. Orser, Vice President, Nuclear Operations
,
- W. Colonnello, HPES Coordinator
- S. Catola, Vice President, Nuclear Engineering and Services
>
- L. Goodman, Director, Licensing
- R. Haupt Nuclear Engineering
- A. Settles, Superintendent, Technical Engineering Nuclear Regulatery Commission (NRC)
- W. Rogers, Senior Resident Inspector S. Stasek, Resident Inspector
- Denotes those present at the Management Interview on December 28, 1989.
.
The inspectors also interviewed other members of the licensee's staff during this inspection.
,
2.
Follow-up of Events (93702)
'
a.
Event Summary On December 11, 1989, the licensee discovered that wide range i
reactor water level transmitter B21N081C was improperly installed and would not have detected a low water level in the reactor vessels
The discovery was made as a result of an operator noting that level indication from the transmitter in question had remained up-scale while the other three channels indicated normal vessel level. The transmitter and associated circuitry were subsequently declared inoperable. Troubleshooting revealed that the transmitter, which
,
had been replaced during the recent outage, had not been installed in the inverted position needed to accommodate the non-standard tubing configuration of the instrument rack. The licensee removed, inverted, and reinstalled the transmitter. A post-maintenance surveillance test was satisfactorily completed and the instrument declared operable on December 12, 1989.
.
-
r
-
'3:
{
-
.
.
,
t b.
-E_ vent Chronology February. 7.1989 - Rosemount Corporation issues a 10 CFR Part 21
.
notification discussing loss of iill fluid from Model 1153 differential pressure transmitters.
Nine of the serial numbers listed are identified as being at Fermi; eight are installed in the plant and one is in sp&res.
The licensee subsequently implements a
monitoring plan for these transmitters to effect replacement should
!
they show signs of failure.
!
September 8_, 1989 - Transmitter 221N001C, Wide Range Reactor Water r
Level Chcnnel C, on rack PD05 in the reactor building, (one of the
,
eight installed suspect transmitters) falls out of tolerance during perfcrmance of 18 month calibration surveillance, 44.020.009, step
6.3.3.
The plant is in Cold Shutdown (mode 4) during its first
'
refueling outage.
In accordance with the monitoring plan the transmitter is subsequently scheduled for replacement and the
-
necessary design packages and work requests are developed.
l
Date Unknown - Replacement transmitter B21W081C it cemoved from a new instrument rack which is under construction in the warehouse.
This replacement rack is part of Engineering Design Package (CDP)
6740 and was to be installed during RF01 but coordination problems led the licensee to request deferral of its installation until RF02.
October 25,1959 - 821H081C replacement transmitter is installed under MaintenTnce Work Request (MWR) No. 0040890909 by two I&C repairmen.
A QC inspecter witnesses the installation process.
October 26. 1989 - Calibration surveillance test 44.020.009 was performed as a post-modtfication test (PMT) for MWR No. 004C890909.
The technician performing the test noted nothing abncrmal and the transmitter was satisfactortly tested and declared operable.
December 3,1989 - C611bration surveillance test 44.020,009 is reperformed on B21N081C to recover from a test equipment problem identified during periodic testing in the Measuring and Test Equipment (M&TE) shot. Once again, nothing abnormal is noted and the surveillance is successfully completed.
December 6,1989 - The plant entered Mode 2 (Startup) at 1740 EST.
D_ecember 10. 1989 - At approximately 1700 EST an operator performing shiftly indicator checks noted that water level indication on Master Trip Unit (MTU) B21N6 SIC had not come on scale as expected and was out of tolerance high at greater than 220 inches indicated. The associated channels (A B, and D) were all indicating normal reactor water level.
The Nuclear Shift Supervisor (NSS) directs I&C Maintenance to begin a troubleshooting effort.
The indicator and its associated transmitter 821N081C were subsequently declared
c
- - - - - -. ---
- - -
-
g.;
-
--
.-
..
.
-,
.
'
.
x inoperable and the appropriate Limiting Condition for Operation (LCO) action statement was entered.
In accordance with Instrument Trip Sheet, NPP-23.601, Rev. 4, Page 10, the fuse for MTV B21681C was pulled, however, the fuse for the associated slave trip unit B21N684C was not removed as required.
-December 11,1989 - At 0420, during troubleshooting I&C personnel discover.that the tubing on rack P005 is non-standard and that the transmitter is installed in standard fashion. They recognize that this puts the high side of the process (level) to the low side of the transmitter and vice versa. The NSS is informed and directs 1&C personnel to inspect all other transmitters changed out during the outage for proper installation... no other problems are found.
The troubleshooting effort, by causing a half-isolation of the Main Steam Isolation Valve closure logic, revealed the failure to pull the fuses for the slave trip unit (B21N684C). The NSS directs that the fuses be pulled. MWR No. 002C891211 was written to correct the improper installation of B21N0810.
December 12, 1989 - The transmitter was reoriented, calibration procedure 44.020.009 was satisfactorily completed, and the transmitter was declared operable.
c.
Design Background B21N081C is one of four channels of wide range reactor vessel water level associated with the Isolation Actuation Instrumentation under Technical Specification (TS) 3.3.2.
This transmitter is not part of the Reactor Protection System or Emergency Core Cooling System actuation logic. The transmitter provides input signals to MTV B21N681C and slave trip unit B21N684C which provide indication and actuations in division II of the actuation logic.
Figure 1 is an instrument loop diagram showing the transmitter, trip units, wiring connections, and loop functions.
The original transmitter was a Rosemount Model 1153DB5 differential pressure transmitter commonly used throughout the industry for both level and flow measuring applications.
It provides a 4 to 20 milliamp (ma) direct current signal, proportional to reactor water level, to MTV B21N681C and slave trip unit B21N684C, MTU B21N681C is a Rosemount electronics component which receives the 4 - 20 ma signal and provides level indication and alarm and actuation of selected systems, when reactor water level falls below level 2 (110.8 inches above top of active fuel). The syst' ems actuated by this MTV on level 2 include primary containment isolation groups (PCIS) 2, 12, 14, 16, 17, and 18; reactor water cleanup system (RWCU) isolation; standby gas treatment system (SGTS), control center heating, ventilation, and air conditioning
C
-.
-
.
>
~
-
.
-
c
,
'
.
(CCHVAC); reactor building HVAC (RBHVAC), non-interruptable air system (NIAS); main steam line outboard drain isolation valves; and
'5 a variety of valves and dampers associated with secondary containment isolation. The logic for Division II of these systems
[
is arranged such that both channels "C" and "D" must trip to provide
,
the required operations: Division I is similarly arranged with
-
channels "A" and "B"..
The improper installation of transmitter B21NOSIC resulted in all of these Division 11 systems being
incapable of responding to an actual low water level in the reactor.
All of these systems, with the exception of RWCU which has redundant actuations on flow differential, room temperature, temperature
,
differential, and standby liquid control initiation, have a
redundant actuation signal from high drywell pressure.
SGTS has
.
additional redundant actuation signals from the process radiation
'
monitoring system.
l Slave trip unit B21N684C is a similar Rosemount electronics component which provides actuation for inboard and outboard Main
,
Steam Isolation Valves when reactor water level falls below level 1 (31.8 inches above top of active fuel).
The logic is this case is arranged such that channels "A" or "C" and "B" or "D" tripped will close all eight MSIVs. The improper installation of transmitter B21N081C resulted in a degradation of the MSIV closure logic, however, this logic has redundant actuations from drywell pressure, main steam line radiation, flow, and pressure, main steam line tunnel temperature, condenser pressure, and turbine building area temperature, d.
Event Evaluation i
'
(1) The inspectors determined that the improper installation of B21N081C could have been detected at any one of three critical
points: at completion of the installation, at the QC
-
inspection, and during the post-modification testing.
The inspectors. concluded that the improper installation of B21N081C was due to failure to perform a fundamental attribute of instrument installation due to lack of attention to detail
'
on the part of the repairmen compounded by an inadequate installation procedure. The high and low chambers of the transmitter from the new rack (in the warehouse) were exactly opposite of the orientation on the existing P005 rack in the reactor building due to the non-standard tubing configuration on the existing rack. The transmitter can be installed either way by the installer depending on the desired orientation. The only way to.tell which side the high and low chambers are on is by an "H" and "L" embossed nexi, to where the tubing is
connected. The two I&C repairmen never thought about the i
orientation of the chambers and just installed the transmitter as it had been on the warehouse rack. An inadequate
,
.
- -
j
-
,.
.
.
.
.
t
.
<
.
.
installation procedure which relied heavily on skill of the i
craft was a contributor to the error.
dictates that on completion of an installation such as the one
'
in question the installers should have checked the tubing all the way back to the rack valves.
The installation procedure
. should have addressed the existing non-standard hookup of
tubing on B21N0810 an.d should have referenced a drawing which showed the existing tubing arrangement.
Human factors deficiencies contributed to the error. The less than obvious non-standard tubing arrangement itself compounded the problem; the " crossed" tubing occurred between the rack valves and the (
five-valve manifold and none of that tubing was touched...
only the tubing from the manifold to the transmitter. The use
of the non-standard tubing configuration on the same. rack as instruments with standard tubing, the lack of adequate lighting, the lack of labeling of the manifold valves, and the existence of contamination on the rack all may have contributed to the situation.
!
The inspectors concluded that the failure of the QC inspector
-
to identify the error was also lack of attention to detail due to inadequate inspection techniques and an inadequate inspection plan furnished by the engineering authority who developed the design package for the transmitter replacement.
Checking the orientation of the chambers and the tubing arrangement was within the craft capability of the inspector and should have been considered part of the installation verification. Since the tubing from the manifold was untouched it never occurred to the QC inspector to examine it.
The engineering authority-responsible for the development of the inspection plan should
-
have included inspection of the entire tubing arrangement using the drawing mentioned above, instead the only drawing included in the inspection plan emphasized proper torquing of the-mounting bolts.
The inspectors reviewed the post-modification testing conducted to verify the installation and determined that the procedure, NPP 44.020.009, Rev. 23, "NSSS - Reactor Vessel Low Water Level (Levels 1 and 2), Division II, Channel C Calibration / Functional" contained a technical error. The procedure contained detailed instructions for valve manipulations to isolate the transmitter and a line sketch of the valves on the manifold and transmitter to direct I&C technicians as to where to connect test equipment for the calibration. On the sketch the valves are labeled but they are not labeled in the plant... in addition the sketch in the procedure indicated high on the left and low on the right.
Correct labeling of the valves on the manifold in the plant would have differed with the sketch, the technician would have questioned his procedure, and the incident might have been averted.
The technician followed his procedure exactly and l
i
'
o-
,, -,
,-.
,.
- - -
- - -
C
-
.,.-
i
..
-
I i
-
connected his test equipment high to left and low to right, as
'l the transmitter was installed. The technician did not check i
the tubing from the manifcid to the rack valves.
It s.hould be understood that the technician connected his equipment properly
according to his procedure and the calibration was successful
... if he had traced back to the rack valves and discovered that the tubing was non-standard he would have connected his equipment high to right and low to left, the calibration would not have worked, and the installation problem would have been identified.
The surveillance procedure should have contained
notes or warnings identifying the existing non-standard tubing arrangement and the sketch should have shown the correct j
(inverted) orientation of the transmitter. Given the detailed
instructions and the sketch contained in the procedure it is
)
not reasonable to expect the technician to trace the tubing
)
back to the rack valves. The failure of the post-modification testing to identify the improper transmitter orientation
results from an incorrect procedure in that the procedure and its included drawing matched the improper orientation rather
'
than calling out the non-standard tubing arrangement.
l 10 CFR 50, Appendix B, Criterion V and TS 6.8 require the use of procedures and instructions appropriate to the circumstances
,
'
to ensure the proper completion of activities. While lack of attention to detail on the part of the I&C repairmen and the QC
inspector is the root cause for this incident, inadequate installation instructions, an inadequate QC inspection plan,
_
and an incorrect surveillance procedure were all key l
contributors in that each could have independently averted the problem.
The use of inadequate and incorrect instructions and procedures in the installation and testing of transmitter
.
B21N081C is an apparent violation (50-341/89036-01(DRP)) of 10
CFR 50, Appendix B, Criterion V and TS 6.8.
(2) The inspectors reviewed operability requirements for the improperly installed transmitter and the systems which it supplies actuation signals.
Technical Specification (TS) table 3.3.2-1 provides the
'
isolation actuation instrumentation requirements for minimum operable channels, valve groups operated, applicable
,
operational condition (mode) and action required for failure to meet any appropriate action statement.
In Modes 1, 2, and 3, and when handling irradiated fuel in the secondary containment, during core alterations, or during operations with the potential for draining the reactor vessel, two channels, B21N081C and B21N081D are required to be operable for the division II trip system.
The following functions are affected:
<
Primary Containment isolation; Reactor Water Cleanup (RWCV)
system isolation; Secondary Containment isolation; Control
.
.
.
.
.-.
_
r
.u
.
,
)
.
,
e
,.
-
'
i
.-
l
'
'
CenterHeating, Ventilation,andAirConditioning)(CCHVAC) rec i
Gas Treatment (SGTS) actuation. With channel "C" inoperable, I
l action statement 3.3.2.b requires that it be placed in the
'
" tripped" condition within one hour.
,
L i
On October 30, 1989, with the reactor in Mode 5 with core l
alterations in progress, division I of CCHVAC was removed from i
service for approximately 62 hours7.175926e-4 days <br />0.0172 hours <br />1.025132e-4 weeks <br />2.3591e-5 months <br />. During this time both
.
B21N081C and B21N081D were required to be operable to provide a division II CCHVAC recirculation shift in response to a low I
reactor vessel water level. With channel
"C" inoperable, and not tripped as required this shift would not have occurred.
This t
L is an apparent violation (50-341/89036-02a(DRP)) of TS 3.3.2.b.
On October 31, with the reactor in Mode 5 with-core alterations in progress, channel "A" was removed from service for approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />.
In this condition neither division I nor
division 11 trip system was capable of providing required actuations in the event of a low reactor vessel water level.
<
l TS allow placing a channel in an inoperable status for up to 2
-
hours without placing the channel in the tripped condition providing the parameter is monitored by another operable
channel; since channel "B" was operable during this time period
'
division I trip system condition was acceptable. Division I of CCHVAC was still out of service as noted above and the entire channel "A" outage was contained within the 62 hours7.175926e-4 days <br />0.0172 hours <br />1.025132e-4 weeks <br />2.3591e-5 months <br />.
Between November 8 and November 27, 1989, there were several
'
occasions where division I of CCHVAC or a channel of the
division I trip system was removed from service while the
'
reactor was in Mode 4 or in Mode 5 with no irradiated fuel
.
handling in the secondary containment, core alterations or operations with the potential for draining the reactor vessel in progress.
In accordance with TS there are no operability
requirements for the Isolation Actuation Instrumentation under these circumstances.
On entry into Mode 2 on December 6, 1989, at 5:40 p.m., TS
.
requirement 3.3.2.b. again became effective and was not recognized nor acted on until December 10, 1989, at 7:08 p.m., a period of approximately 98 hours0.00113 days <br />0.0272 hours <br />1.62037e-4 weeks <br />3.7289e-5 months <br />.
This is an apparent violation (50-341/89036-02b(DRP)) of TS 3.3.2.b.
,
On December 8,1989, with the reactor in Mode 2, channel "A" was removed from service for 24 minutes.
TS 3.3.2.c requires that when both trip systems contain less than the minimum required number of channels (2), at least one trip system is to be placed in the tripped condition and actions taken in accordance with table 3.3.2-1.
None of the action statements
'
.
.,
,.
Y associated with any of the affected systems required action sooner than one hour and the provisions allowing 2-hour inoperability were applicable.
i (3) Failure to pull the slave fuses - This is attributed to personnel error on the part of the NSS. His original perception of the problem was that it was confined to the master trip unit which supplied the level indication signal.
When he was subsequently informed that it was a transmitter problem it was during the press and confusion of turnover and he did not recognize that it was necessary to pull the fuses for the slave unit. The on-coming NSS failed to double check that the appropriate action was taken, i
i TS 3.3.2.b requires that with less than the minimum number of required channels operable for a trip system the affected
'
channel is to be placed in the tripped condition within one hour. When channel "C" was declared inoperable on December 10
{
at 6:51 p.m., only the master trip unit, B21N681C was tripped.
'
The channel was not completely tripped until December.11, at 6:55 a.m., a period of 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> This is an apparent violation i
(50-341/89036-02c(0RP) of TS 3.3.2.b.
e.
Safety significance i'
With B21N081C inoperable, one division of Standby Gas Tre'atment, Non-interruptable Air System, Control Center HVAC, Non-interruptable Air System, Primary Containment Isolation would not have responded
<
to a low reactor vessel water level.
As noted earlier in the Design Background section, all of these
,
systems have redundant actuation signals and, unless removed from t
service as discussed above, were capable of proper response and were considered operable during the 47 day period when B21N081C was inoperable.
t During the period in Mode 5 when core alterations were in progress, division II of CCHVAC and NIAS would have actuated on high radiation levels or could have been manually initiated. Although the reactor
water low level actuation is required by Technical Specifications, a licensee review has determined that no credit is taken for this
'
signal in the accident analysis of events occurring in Modes 4 and 5.
During the period in Mode 2, reactor power never exceeded 4 percent.
The licensee's accident analysis for a steam line break outside primary containment at full power indicates that 10 CFR 100 limits will not be exceeded. The release path for this accident is considered to be direct unfiltered to the environment.
The radiological consequences of a failure of both divisions of
,
l-
e q., -
..
.
a..
i
,
isolation / actuation during an accident occurring at 4 percent power is considerably less than that at full power.
The Reactor Protection System (RPS) and the Emergency Core Cooling
.
System (ECCS) were unaffected by this event and were available, with
'
<
L the exception of the High Pressure Core In ection system for a period of time while reactor pressure was less than 165 psig, to mitigate the consequences of an accident ir mode 2.
'
The safety significance of this event is trinimal based on the
.
availability of redundant actuation signa',s for isolation systems, the operability of the RPS and ECCS during the event, the lack of any significant power history, and that no low reactor vessel water level events requiring initiation occurred during the period in question.
3.
Corrective Actions
,
.
The following corrective actions were taken by the licensee in response
.
'
to this event:
Walkdowns were performed on other transmitters installed during the outage to verify proper installation. No other discrepancies were identified.
,
Training on the event was conducted for I&C personnel
'
A critique of operator performance is being written and will be included in the operator's January required reading.
,
Quality Assurance is developing a " lessons learned" based on the
!
event and training will be conducted for inspectors during the month of January.
- A Human Performance Evaluation System (HPES) review is being performed to identify causes and develop preventive measures to orevent recurrence.
The report is plar.ned to be completed in January.
^
4.
Management Interview (30703)
The inspectors met with licensee representatives (denoted in Paragraph 1)
on December 28, 1989 to discuss the scope and findings of the inspection.
In addition, the inspectors also discussed the likely informational content of the inspection report with regard to documents or processes reviewed by the inspectors during the inspection, the licensee did not identify any such documents or processes as proprietary.
e 11