NRC-90-0029, Responds to Violations Noted in Insp Rept 50-341/89-36. Corrective Action:All Personnel in Operations & I&C Viewed Professionalism Video Which Included Presentation of Facts Concerning Event

Responds to Violations Noted in Insp Rept 50-341/89-36. Corrective Action:All Personnel in Operations & I&C Viewed Professionalism Video Which Included Presentation of Facts Concerning Event

ML20012B746
Person / Time
Site:	Fermi
Issue date:	03/05/1990
From:	Sylvia B DETROIT EDISON CO.
To:	NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
References
CON-NRC-90-0029, CON-NRC-90-29 NUDOCS 9003160214
Download: ML20012B746 (9)

Text

, 4 k  :

s. Ralph syM2 4

Senio# vice Pess ovnt 4

I; u oo wo e o u mow..,

l Newport. M*Chagen 46166 asian s864150 March 5, 1990 NRC-90-0029

U. S.; Nuclear Regulatory Commission Attention Document Control Desk Washington, D.C. 20555 q

References:

^(1) Feral 2 :

. NRC Docket No. 50-341 NRC License No. WPF-43 (2) NRC Inspection' Report No. 50-341/89036

Subject:

Response to a Notice of Violation Attached is the response to~ the Notice of Violation for the improper installation of a wide range reactor water level transmitter.

Included in the response is a discussion oft (1) corrective action taken and the results achieved; (2) corrective action to be taken to avoid further violations; and (3) the date when full compliance will-be achieved.

. .Our response also addresses the root causes contributing to these violations. Your cover letter to the Notice of Violation requests

Detroit Edison address root causes contributing to the following (our response'is provided with each root cause):

(1) The maintenance personnel improperly installing the transait'ter.

This is specifically addressed under iten B.1 of the Enclosure.

The root cause of this event was'that the !&C technicians did not properly trace down the sensing lines. A contributing factor was-

+ that an independent verification of the sensing line was not

performed nor.was it required by procedure. Also contributing to this event was the lack of labeling of the sensing-lines for B21-N081C since this transmitter is different from other

' transmitters in the immediate vicinity. '

(2) The. technical error inLthe post modification (surveillance) t:

procedure.

This is specifically addressed under iten B.3 of the Enclosure.

The work package.which implemented the Engineering Design Package

-(EDP) instructed the post modification testing be completed using a surveillance procedure which contained a figure that did not reflect the appropriate configuration. However, this surveilla.nce was performed correctly'but could not have caught

.this~ problem'since it was testing the function of the transmitter I only. This. problem could only have been identified by 0(

9003160214 900305 N

, PDR- ADOCK 05000341

-Q PDC- p 1 [\\\

?

L - - -

b

~

t i

L U. S. Nuclear Regulatory Commission l March 5, 1990 -

NRC-90-0029 I Page 2 1

confiraation of the installation or by personnel when, at normal i reactor water level, reactor coolant temperature approached  ;'

normal operating temperature and other wide range level indicators began to indicate true water level, but this one did i not. ,

In conclusion, all the appropriate elements of a thorough Post i Modification Testing (PMT) program were evident in this case. i There was installation testing, a functional test and then an operational / surveillance test. In the last phase of PMT, i.e.,

the " operational" test, the problem was discovered.

I (3) The Quality Control inspection activities not disclosing the .

problems.  !

j This is specifically addressed under item B.2 of the Enclosure.

The root.cause of this event was that the Quality Assurance l

' inspector (contractor) did not adequately check the transmitter .

installation when signing off the hold point during installation  !

of the EDP. The inspector is no longer working at Fermi 2.

(4) The operations staff not properly placing the instrument channel  ;

. in the trip condition after discovery of the inoperability of the  ;

I instrument.

l- This is specifically addressed under our response to Viciation 89-036-02 of the Enclosure. Three factors which contributed to this inappropriate action were: poor communication during shift  :

turnover, insufficient information transmitted by I&C to l Operations during turnover of the job responsibility and lack of direction in procedure 23 601, " Instrument Trip Sheets."  ;

l I Detroit Edison does not agree that the post modification testing for ,

~

the involved design change was inadequate. The root cause of these violations is personnel error during construction installation and verification. The Enclosure to this letter addresses the corrective actions taken and results achieved for each of the root causes discussed above. .

If there are any questions relating to this response, please contact Joseph Pendergast at (313) 586-1682. ,

Sincerely cc: A. B. Davis h tff /

R. C. Knop W. G. Rogers J. F. Stang-Region III

Enc 1;. cura to NRC-90-0029 Pa'ge 1 S1ATEMENT OF VIOLATION 89036-02a,b and er A. Technical Specification Limiting Condition for Operation 3 3 2 requires 2 channels per division of Reactor Vessel Water Level Isolation Actuation Instrumentation to be operable in Operating Conditions 1, 2, 3, and when handling irradiated fuel in the secondary containment, during core alterations, or during operations with a potential for draining the reactor vessel. Action statement 3 3 2.b requires that with the number of operable channels less than the minimum required, the inoperable channel and/or that trip system shall be placed in the tripped condition within one hour.

Contrary to the above:

On October 30, 1989, with the reactor in Mode 5 with core alterations in progress, division I of CCHVAC was removed from service for approximately 62 hours7.175926e-4 days <br />0.0172 hours <br />1.025132e-4 weeks <br />2.3591e-5 months <br />. During this time both Reactor Vessel Water Level Isolation Actuation Instrumentation channels "C" and "D" were required to be operable to provide a division II CCHVAC recirculation shift in response to a low reactor vessel water level. Channel "C" had been inoperable since October 26, 1989 as a result of improper installation of differential pressure transmitter B21N0810. During this 62 hour7.175926e-4 days <br />0.0172 hours <br />1.025132e-4 weeks <br />2.3591e-5 months <br /> period, action was not taken to place channel "C" in the tripped condition.

On entry into Mode 2 on December 6,1989, at S:40 p.m., both Reactor Vessel Water Level Isolation Actuation Instrismentation Channels "C" and "D" were again required to be operable.

Channels "C" was still inoperable as a result of the improoer L

installation of differential pressure transmitter B21NO810 and l

action was not taken within one hour to place the inoperable I channel in the tripped condition as required. This requirement I

was not recognized nor acted on until December 10, 1989, at 7:08 p.m., a period of approximately 98 hours0.00113 days <br />0.0272 hours <br />1.62037e-4 weeks <br />3.7289e-5 months <br />.

When transmitter B21WO81C was declared inoperable on December 10 at 6:51 p.m., only the associated master trip unit, B21N6810 was

! tripped by removing the fuses. The channel was not completely tripped until December 11, at 0655 hours0.00758 days <br />0.182 hours <br />0.00108 weeks <br />2.492275e-4 months <br />, when the fuses for the slave unit (B21N6480) were removed, 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> later.

Corrective Actions Taken And The Results Achieved:

As requested by the NRC in the cover letter to the Notice of Violation, DECO's response should address the root causes contributing to the operations staff not properly placing the instrument channel in the tripped condition after discovery of the inoperability of the instrument.

F Enclecuco to NRC-90-0029 Pa'ge 2 The Technical Specification 3 3 2.b Action Statement violations occurred since it was unknown that this problem existed until the Surveillance was performed. Plant conditions were not conducive to discovery of the problem until the Reactor Vessel Water Level came on scale with these instruments. Once the condition was discovered, prompt action was taken by the operations personnel.

The root cause of the third example cited in the violation was personnel error. Once the transmitter was found inoperable by I&C, Operations was required to place all trip units which received input from B21-N081C in the tripped condition within one hour. 'lhree factors which contributed to this inappropriate action weret poor communication during shift turnover, insufficient information transmitted by I&C to Operations during turnover of the job responsibility and lack of direction in procedure 23 601, " Instrument Trip Sheets."

An accountability meeting was held between the personnel involved and plant management. Additionally, a fact finding meeting was held; based upon review of the facts, discipline was administered in accordance with company policy.

A critique of this_ problem has been written and was included in the January required reading for Operations personnel. This critique outlined the poor communications during shift turnover between the Operating personnel and by I&C who reported the inoperable transmitter.

Procedure 23 601 was revised to provide the user with more conservative directions. In this particular event the only indication available to the Operating staff was from the master trip unit, B21-N681C. Based on this indication, the Operating staff placed the master trip unit in the tripped condition once it failed the channel check. Because the slave trip unit B21-N6840 received the same input as B21-N681C it too should have been considered inoperable. The procedure now requires both the master and the slave trip units to be placed in the tripped condition.

Corrective Action To Be Taken To Avoid Further Violations:

All personnel in Operations and I&C have viewed the Fermi 2 professionalism video presented by management which included a presentation of the facts concerning this event. This video emphasized the level of professionalism that is needed from nuclear personnel.

The potential generic ramifications of this event were considered in the accountability action plan developed by Detroit Edison as described in Detroit Edison letter NRC-89-0300. Included in this action plan is verification that there are no other problems with the return to service of safety-related equipment from the outage or startup following the outage. This plan has focused on

i.

. '41 Enclosure to-NitC-90-0029 Psge 3 professionalism, accountability, followup and simplification of work activities. It will serve to improve performance of personnel in all activities throughout the plant.

?

The Date When Full Compliance Will Be Achievedt t-Detroit Edison is in full compliance.

l l

En31osura to

, NBC-90-0029 Page 4 l 1

i STATEMENT OF VIOLATION 89036-01: j i

B. 10 CFR 50, Appendix B, criterion V and Technical  !

Specification 6.8 require the adherence to procedures and instructions appropriate to the circumstances to ensure the proper completion of activities, j Contrary to the above j 1

1. On October 25, 1989, differential pressure transmitter B21N08IC was improperly installed using procedures inappropriate to the circumstances in that the procedure did l not address the non-standard sensing line configuration for i this instrument or reference appropriate drawings describing q the configuration.

2. On October 25, 1989, Quality control inspectors continued  !

the installation of pressure transmitter B21N081C without appropriate inspection procedures resulting in the failure to identify improper installation of the transmitter. -

3 On October 26, 1989, post-modification testing of the ,

installation of transmitters B21N081C was performed with a  ;

procedure inappropriate to the circumstances in that the procedure was not. capable of identifying the improper -

transmitter installation.

Corrective Actions Taken And The Results Achieved:

B1, As requested by the NRC in the cover letter to the Notice of Violation, Deco's response should address the root causes  ;

contributing to the maintenance personnel improperly installing the transmitter. ,

The root cause of this event was that the I&C technicians did not properly trace down the sensing lines. A contributing factor was '

that an independent verification of the sensing line was not performed nor was it required by procedure. Also contributing to this event was the lack of labeling of the sensing lines for B21-N0810 since this-transmitter is different from other transmitters in the immediate vicinity.

A planned work request was used to change out the transmitter.

It did not address the "non-standard" configuration because we do not have a " standard" configuration, however, most other

• differential pressure transmitters on this instrument rack are oriented differently. The drawing showing configuration for this  ;

transmitter was listed on the work documents list included in the package. It has been determined that the root cause of the inappropriate installation was lack of attention to detail on the part of the craftsmen and supervision involved.

. - . . . .~

t a .

Encir uro L3  ;

. NRC-90-0029 Page 5 {

B2. As requested by the NRC in the cover letter to the Notice of

• Violation, DECO's response should address the root causes contributing to the Quality Control inspection activities  ;

not disclosing the problems. l The root cause of this event was that the Quality Assurance inspector (contractor) did not adequately check the transmitter .

installation when signing off the hold point during installation of the Engineering Design Package (EDP). The inspector is no  ;

longer working at Fermi 2.

B3 As requested by the NRC in the cover letter to the Notice of t Violation, DECO's response should address the root causes  :

contributing to the technical error in the post modification  :

(surveillance) procedure. [

The work package which implemented the EDP instructed the post modification testing be completed using a surveillance procedure which contained a figure that did not reflect the appropriate configuration. However, this surveillance was performed correctly but could not have caught this problem since it was testing the function of the transmitter only. This problem could only have been identified by confirmation of the installation or by-personnel when, at normal reacter water level, reactor coolant temperature approached normal operating temperature and other ,

wide range level indicators began to indicate true water level, j but this one did not. t The PMT requirements are considered adequate if PHT can verify i that the objectives of the design modification have been mot and  ;

critical design parameters have been satisfied. In the case of EDP 10757, Rev 0, (like-for-like replacement of transmitter B21N0810), these criteria were met by performing transmitter ,

calibration / functional test procedure NPP 44.020.009 Rev 23 and response time test procedure NPP.44.020.013, Rev 20.

On the other hand, had the PHT specified that a channel check be performed, the instrument channel check would not have satisfied PMT criteria, since during cold shutdown the reactor water level j is maintained above the calibrated range of level transmitter B21N0810. Therefore, the associated trip unit meter would always give an offscale indication. The instrument channel check for  ;

the purpose of the qualitative assessment of channel behavior during operation is performed by operators in Modes 1, 2, and 3 and when handling irradiated fuel in the secondary containment on e a shift basis (every 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />) per Technical Specification (TS)

Table 4 3 2.1-1. With all channels reading upscale, the improper transmitter installation could not have been detected by a  ;

channel check. The first opportunity to perform a channel check would be in Modes 3 or 2, which would be done anyway to satisfy Technical Specifications. Nothing would have been gained by specifying that a channel check be part of the PMT in Modes 4 or i 5.

j' Enclosura to

., NBC-90-0029 Page 6 The installation error of course, was detected by the channel check performed routinely in Mode 2. The way to detect construction errors is for the installation personnel to assure correct orientation and for Quality Assurance (QA) to verify that installation was done in a manner conforming to the design, i.e.,

according to rack H21-P005 General Arrangement Drawing 61721-2281-05 Rev H. _ Additionally, EDP 10757 did specifically require the QA inspection of the installation.

In conclusion, all the appropriate elements of a thorough PMT program were evident in this case. There was installation testing, a functional test and then an operational / surveillance test. In the last phase of PMT, i.e. , the " operational" test, the problem was discovered.

Based on the above, DECO does not agree that the PMT was inadequate.

Other Actions Taken by Detroit Edison Upon discovery that the transmitter had been improperly installed, other transmitters installed under the same EDP and those installed on racks H11-P009 and H11-P010 were verified to be properly installed. In addition, all other transmitters installed during the First Refuel Outage were verified to be properly installed. No other discrepancies were found.

An accountability aceting was held between the personnel involved and plant management. Additionally, a fact finding meeting was held. Based upon review of the facts, discipline was administered in accordance with company policy.

Administrative Procedure NPP-PS1-01, " Planning Of Maintenance Activities", has been revised. It now requires that if the work involves replacing or installing a differential pressure type transmitter, ensure that an independent verification is performed of the high and low sides of the transmitter to ensure they are orientated correctly.

The sensing lines have been labeled high and low side at the transmitter for B21-N0810. This should assist personnel in determining proper installation orientation in the future.

Surveillance procedures 44.020.007, 44.020.008, 44.020.009, and 44.020.10 that govern NSSSS-Reactor Vessel low water level (levels 1 and 2) calibration / functional tests have been revised.

Figures 1 and 2 have been updated to correctly reflect actual plant configuration. This will assure that the I&C technician has a diagram to work from that reflects the actual configuration.

1

-. .- 4 Encircura to  ;

, ,NRC-90-0029 Page 7 A critique of this problem has been written and was included in ,

'the required reading program for I&C personnel. This critique i provided an explanation of the sequence of events, departments -

involved, consequences of the event, conclusions, lessons  :

i learned, and recomesndations which increased the I&C personnel understanding of the details of this event.

i Informal training on this event was given to I&C shop personnel  !

following the event. This training included a discussion with I&C personnel to inform them of the importance of the hi6h and low markings on Rosemount Transmitters and of the importance of ,

transmitter orientation.

PQA has developed training based upon this event. The " lessons i learned" from this event were discussed during the training l sessions. Other events which were identified in the management accountability meeting minutes were also discussed. All PQA i personnel are aware of the contents and actions taken to prevent future personnel errors as established in the management  !

Accountability Action Plan. Training for the Production Quality Assurance inspectors was completed in January of 1990.

The PQA inspector (a contractor) was terminated from work at Fermi 2. In addition all inspections conducted by this inspector  ;

have been rechecked. EBASCO Services Incorporated has issued a

" Lessons Learned" and incorporated it into their QC personnel training curriculum on this event.

. Corrective Action To Be Taken To Avoid Further Violationst L All personnel in I&C and PQA have viewed the Fermi 2  :

professionalism video presented by management which included a presentation of the facts concerning the incorrectly installed

Reactor Water Level Wide Range Transmitter.

J- The potential generic ramifications of this event were considered l in the accountability action plan developed by Detroit Edison as ,

described in Detroit Edison letter NRC-89-0300. Included in this '

action plan is verification that there are no other problems with ,

the return to service of safety-related equipment from the outage

- or startup following the outage. This plan has focused on professionalism, accountability, followup and simplification. It will serve to improve performance of personnel in all activities throughout the plant.

The Date When Full Compliance Will Be Achieved:  ;

t Detroit Edison is in full compliance.

J

_, .- , - .,, , - - - -. .# , --