ML18092A951
| ML18092A951 | |
| Person / Time | |
|---|---|
| Site: | Peach Bottom, Salem, 05000000 |
| Issue date: | 12/18/1985 |
| From: | Rameysmith A Office of Nuclear Reactor Regulation |
| To: | Black K, Booher H, Rowsome F NRC OFFICE FOR ANALYSIS & EVALUATION OF OPERATIONAL DATA (AEOD), Office of Nuclear Reactor Regulation |
| Shared Package | |
| ML18092A952 | List: |
| References | |
| NUDOCS 8512270304 | |
| Download: ML18092A951 (15) | |
Text
- ,_
MEMORANDUM FOR:
FROM:
SUBJECT:
UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D. C. 20555 December 18, 1985 Harold R. Booher, Chief Maintenance and Training Branch Division of Human Factors Technology, NRR Frank H. Rowsome, Chief Human Factors Issues Branch Division of Human Factors Technology, NRR Kathleen M. Black, Chief Nonreactor Assessment Staff Office for Analysis and Evaluation of Operational Data James E. Lyons, Chief Technical and Operations Support Branch Planning and Program Analysis Staff, NRR Ann Ramey-Smith, Engineering Psychologist Human Factors Issues Branch Division of Human Factors Technology, NRR ENCLOSURE TRIP REPORT FOR SALEM AND PEACH BOTTOM SITE VISITS REGARDING WRONG UNIT/WRONG TRAIN EVENTS This memorandum documents the activities and findings of an NRC staff visit to the Salem and Peach Bottom sites on July 23-26, 1985.
Members of the NRC team for this visit included A. Ramey-Smith (DHFT), E. Trager (AEOD), and
~
D. Persinko (DHFT).
The site visit was conducted as part of the short-term effort to determine whether simple~ low cost improvements can be identified and implemented to reduce the frequency of wrong unit/wrong train events occurring at nuclear power reactor facilities.
Upon completion of all site
- visitsg the factors contributing to the events will be evaluated and a report issued which discusses causes and recommendations.
Long term assessment will be addressed as part of the Maintenance and Surveillance Program Plan being conducted by DHFT.
General Information The Salem site is located 20 miles south of Wilmington, Delaware.
There are two reactors, Salem 1 and Salem 2, located at the site with the Hope Creek facility adjacent to the Salem site. Salem 1 has a maximum dependable capacity (net) of 1079 M~le and was placed into commercial operation on June 30, 1977.
Salem 2 has a maximum dependable capacity (net) of 1106 MWe and was placed into commercial operation on October 13, 1981.
B9th units are Westinghouse PWRs and the architect/engineer for both units was the licensee, Public Service Electric and Gas.
Multiple Addressees The Peach Bottom site is located 19 miles south of Lancaster, -Pennsylvania, and has two reactors, Peach Bottom 2 and 3.
Both units are General Electric BWRs and the architect/engineer for both units was Bechtel.
Peach Bottom 2 has a maximum dependable capacity (net) of 1051 MWe and was placed into commercial operation on July 5, 1974.* Peach Bottom 3 has a maximum dependable capacity (net) of 1035 MWe and went into commercial operation on December 23, 1974.
The licensee is Philadelphia Electric.
Site Visit Agenda The discussions and in-plant observations centered around two wrong unit/wrong train events that occurred at Salem between 1982 and 1983, and two that occurred at Peach Bottom between 1981 and 1985.
The LER numbers for these events at Salem are 272-82-003 and 83-024, and at Peach Bottom are 278-81-008 and 85-008.
During both site visits, the NRC team inspected the locations of the reported wrong unit/wrong train events to the extent possible, and discussed the events with plant management as well as many of the individuals directly involved with the event.
Enclosures 1 and 2 provic:tf!
the sequence of events resulting in the LERs at Salem and Peach Bottom, respectively, the licensee's conclusions regarding the event, and NRC staff observations.
During the visits licensee staff wPre asked to provide any available information on events that were not reportable but that involved the wrong unit or the wrong train. This information is also discussed in the enclosures.
General Observations at Salem Ta in Re uest and In uir S stem The Tagging Request and Inquiry SystP.m TRIS is in p ace at Sa em as well as Hope Creek}. This computerized system features label~ unique to each plant component and can ensure that the operator's worksheet and equipment tags correspond to the equipment labels.
- Some standard tagging requests have been programmed into the computer.
Poth the worksheet and equipment tags are printed out by the computer including, for example, desired component switch location.
TRIS keeps track of normal and current switch or valve positions as well as inoperable equipment.
At the end of every shiftp the operators request the printout of inoperable equipment to permit them to monitor system status. TRIS relies on operator input to keep the system accurate.
Figure 1 below pictures an example of a TRIS label. The first digit of the numerical indicator represents the unit designation, the second is the train or volta9e designation, the third may be the elevation (as appropriate), and the fourth digit (or set of digits) is the componPnt identifier. Currently at Salem, there are about 30,000 items in TRIS, with labels on all major equipment.
About 90 percent of the breakers are labelled; control panels will be among the last to be incorporated into TRIS.
Multiple Addressees FIGURE 1.
TRIS LABEL AT SALEM
Multiple Addressees
.4 -
Maintenance Tracking System Salem i~ working to upgrade its work order system through improving interdepartmental coordination.
Each department uses a computer system for tracking work order requirements and status, e.g.,
retest requirementst work in-progress status, and work completed.-
The licensee (PSE&G) indicated that this system contributed toward significantly reducing the work order backlog from unmanage~ble numbers to a workab~e number of approximately 400.
PSE&G is planning to implement a computer system called the Managed Maintenance Information System (MMIS) developed following the ATWS event of February 1983 to improve the management of maintenance activities. This system will store work order information on any particular component in a system since about 1983 or 1984.
Some concern was expressed over the component identification. scheme that may be employed.
Unique numerical identifiers will be provided for each motor, valve, bearing, and so on, within a system.
If the components' functional names are dropped from the labels and computer system listings, and identificati~n numbers are used instead (rather than in addition), user acceptance of the system may suffer.
considerably.
Color Coding Color coding on signs and the floor is used to differentiate between Units 1 and 2, with Unit 1 blue and Unit 2 yellow.
Corresponding operations procedures and tagging request forms unique to each Unit are likewise color coded, with procedures common to both units being white.
(In the event colored paper is not available, white paper is used.)
However, the licensee's staff mentioned that the color of tags was the same for both units (i.e., red) and that a color-coded bonder around the tags would be useful in clarifying the unit identity of equipment.
I&C procedures are not color coded.
Although it was stated by a licensee representative that I&C errors contribute significantly to the number of reactor trips, it is unknown whether the nature of these I&C errors is such that they involve misidentification of equipment. Color coding is also used to differentiate between the four reactor protection system* channels (red, white, blue, and yellow for channels I, II, III, and IV).
Incident Investigations Fact finding meetings are held after any significant or unusual event.
At the discretion of the Operations Manager, Operations Engineering is instructed to conduct an investigation and prepare a written statement. This investigation process has been in effect for about two years.
As far as taking advantage of operating experience at other utilities is concerned, the licensee's staff indicated that there was not much direct communication wi"th other utilities; rather, INPO and NRC documents and notices are used.
Another information source mentioned is the Nuclear Operations and Maintenance Information System.which provides information on equipment.
Information is provided to the operations staff on significant and unusual events.
One means is the operating daily newsletter callee "The Rising Sun" which is required reading for operators.
Multiple Adcressees Labeling The TRIS labels contain n~un names and unique numeri~ designations for components.
The.first digit is the unit designator, the second is the train or voltage code, the third is the elevation (as appropriate) and the fourth digit is the component designator. Other labels are also currently installed that provide the functional name of the component, but the numerical designator may not be unique. These labels, are stainless steel embossed tags (see Figure 2).
The NRC staff noted that these labels have poor readability, particularly under low illumination, due to poor contrast between the letters and the label background.. The licensee's staff indicated that there have been durability problems with the more-readable plastic labels, but that they are looking into improved labeling methods (for example, engraved tags).
The licensee uses small symbol labels to help personnel differentiate between units (a number 11 1 11 within a triangle for Uni_t 1 and a number 112 11 within a octagon for Unit 2).
NRC staff members felt these unit designation labels were not striking.
The licensee has a "QA Valve Survei 11 ance Program" underway at this time for valves.
The program is intended to ensure valves are labeled anc that the labeling is not defective.
Labels must sometimes be removed to perform maintenance.
However, there is
.no explicit requirement that procedures require the replacement of labels.
There is no administrative procedure that covers labeling and identification.
General Observations At Peach Bottom Critical Equipment Monitoring System (CEMS)
Discussions with the licensee's staff revealed that the Peach Bottom facility is in the process of implementing the CEMS to address the needs of operations personnel.
The CEMS labels use a unique code description for each component and a computer-readable bar code, To this point, implementation of the system has been limited to the assignment and installation of component labels, primarily on valves.
Of approximately 40,000 manual valves, 13,000 have been labeled.
When fully implemented, the CE~S will include breakers using the existing breaker numbers but adding unit and system designations. A plant operator will be able to verify that he has located the proper piece of equipment in the plant by utilizing a hand-held computer terminal capable of reading bar codes.
The CEMS will be used during surveillance and permits and blocking activities. It will also be utilized in monitoring systP.m status because the hand-held terminal will have the capability of receiving input on component positions (e.g., valve open or breaker closed) from the plant operator as he varies component status in the plant. The status of automatic systems will have to be manually fed into the CEMS upon actuation.
The CEMS will also eventually be used to generate permits for blocking tasks.
The licensee's staff indicated that the implementation of the CEMS is v£>ry involved and time consuming because the assignment of unique identifiers to components requires that associated plant documentation be modified as well.
Multiple Addresses 6 -
FIGURE 2 STAINLESS STEEL EMBOSSED LABELS AT SALEM
I-Multiple Addressees Such documentation includes procedures, check-off lists; prints and drawings, and training materials. The plant drawings and prints present a special problem because changes on one print may affect as many as 400 associated documents.
The licensee's staff indicated that two solutions seem feasible for addressing this problem:
(1) develop a cross-reference of numbering systems, and/or (~) develop separate CEMS drawings for use by operators.
Another difficulty impeding full implementation of the CEMS is the computer hardware and software currently being used at Peach Bottom.
The computer hardware and software being used with the CEMS is a few years old and is relatively slow in processing data.
The licensee's staff indicated that the time lag between the data being sent via the hand-held terminal and receiving a response from the computer is as much as 15 to 30 seconds.
The staff went on to say that this delay would increase the time to complete tagging and surveillance activities by two to three times.
The hand-held terminal is bulky and because conmunications between the computer and hand-held terminal utilize radio transmissions, "dead spots in the plant and interference from other transmissions are worrysome.
The licensee's staff indicated that operator satisfaction with the system could be expected to be poor unless these problems are ameliorated.
Work is underway to improve the hand-held terminal and to acquire a more "state-of-the-art data management system.
When these concerns are corrected~ the CEMS could provide an efficient operations and management tool.
Incident Investigations According to information provided by the licensee's personnel, no investigation into root cause is conducted beyond identification that a personnel error was made.
The licensee's personnel
- indicated that more thorough investigations are conducted during industrial -
accident reviews.
Color Coding Peach Bottom Units 2 and 3 are not physically differentiated by a color code.
However, the licensee's staff indicated that som~ procedures are color coded to differentiate between the Units, with Unit 2 procedures being yellow and Unit 3 being green.
In addition, locked valve keys are different for each Unit. Color coding is also applied to valve stems on safety systems.
The sterns are painted green if their normal position is locked closed and red if normally locked open.
Such painting makes surveillance activities easier.
Labeling Component labeling at Peach Bottom currently employes plastic-type engraved labels, hung from a metal chain for valves, that provide unit, system~ and item designations.
The licensee's staff indicated that these labels do tend to get brittle and dirty (see Figure 3).
Peach Bottom is currently in the process of installing CEMS labels* (discussed above) which are expected to rectify these concerns (see Figure 4).
However, the currently used CEMS labels are susceptible to high temperature degradation, a problem the licensee is working to address. }he NRC staff's limited tour of
Multiple Addresses FIGURE 3 ENGRAVED LABELS AT PEACH BOTTOM FIGURE 4 CEMS LABELS AT PEACH BOTTOM
Multiple Addre~sees the Peach Bottom facility found most valves and other components labeled with the exception of a group of valves (see Figure 3).
Exit Meetings Prior to leaving each site, the NRC team expressed its appreciation to the Salem and Peach Bottom staffs for their cooperation in planning the visit, coordinating the tour and discussions, and providing available information.
The management at Salem cooperated with the NRC staff in providing requested information and arranging interviews with cognizant individuals.
The Peach Bottom management did not appear to have prepared for the staff's visit (ref.
memo from Persinko to Gears, August 13, 1985).
Fortunately, on the secorid day of the visit, we were able to obtain useful information from two training coordinators.
The meeting with the training coordinators had not been planned by the licensee prior to our arrival; consequently, there were nPcessarily gaps in the information provided.
Enclosures:
As stated cc:
J. Funches T. Ippolito G. Cwalina D. Fischer G. Gears E. Trager D. Persinko
~~
Ann Ramey-Smith, Engi eering Psychologist Human Factors Issues Rranch Division of Human Factors Technology
ENCLOSURE 1 WRONG UNIT/WRONG TRAIN EVENTS AT SALEM
- 1.
LER-272-82-003 - Diesel (Wrong Train)
- 2.
The following event information was provided by the licensee in the LER:
"On January 6, 1982, while making No. JC Diesel ready for service, and tagging out No. lB Diesel for planned maintenance, the Operator racked in the No. lC 4KV Breaker, and then racked out the No. 18 4KV Breaker at 1021 hours0.0118 days <br />0.284 hours <br />0.00169 weeks <br />3.884905e-4 months <br /> without returning the No. lC Diesel Lockout Switch to the normal position.
The problem was that he misinterpreted the tags on the Diesel Lockout Switch.
"ON" means the lockout switch is lock-out, not that the diesel is on.
When the error was brought to his attention, at 1032 hours0.0119 days <br />0.287 hours <br />0.00171 weeks <br />3.92676e-4 months <br />, he returned the No. lC Diesel Lockout Switch to the normal position. Therefore, from 1021 until 1032 hours0.0119 days <br />0.287 hours <br />0.00171 weeks <br />3.92676e-4 months <br />, both lB and lC Diesels were inoperable and Action Statement 3.8.1.2 was entered.
No operations involving core alterations or positive reactivity changes were in progress at the time... The cause was misinterpretation of the tags Oil.
the lockout switch.
The tags on the Diesel Lockout Switches were changed from ON and OFF to LOCKOUT and DIESEL IN SERVICE to prevent further confusion."
In discussions during the site visit, the licensee's staff indicated that the labels in place at the time of the event were indeed misleading.
However, they pointed _out that there is a local annunciator panel located directly over the switches the operator was manipulating (See Figure 5). This panel contains a tile engraved to read "Diesel~Generator Locked Out."
~lthough the nomenclature on the switch differed from that on the annunciator tile, the operator could be, but was not, alerted to the status of the diesel-generator.
The revised position labeling on the switch more cl~arly describes the status of the diesel and is consistent with the tenninology used on the annunciator tile (see Figure 6).
LER-272-83-024 - Vital Instrument Inverter (Wrong Train)
The following event information was provided by the licensee in the LER:
"At 1422 hours0.0165 days <br />0.395 hours <br />0.00235 weeks <br />5.41071e-4 months <br />, May 25, 1983, during routine power operation, an operator inadvertently opened the supply breaker to ~o. IC Vital Instrument Inverter.
The operator was implementing a Tagging Request to de-energize the No. 2C Vital Instrument Inverter and erroneously performed the evolution on the Unit 1 device.
An automatic transfer to the alternate power supply occurred satisfactorily and there was no loss of equipment or indication. With the inverter out of service, the No. lC Vital Instrument Bus and its associated A.C bus train were inoperable, and Technical Specification Action Statement 3.8.2.1 applied.
The operator immediately realized his mistake, n6tified the Control Room, and restored the inverter to service.
t
- f.
J,.
t f..
~
l~ i> r*1,Q, lj ~ rl FIGURE 5.
LOCAL DIESEL GENERATOR ANNUNCIATOP PftNEL
- NS*
- '§i5ii'ili¥§W"f5d$
IEjrt FIGURE 6.
LOCAL DIESEL CONTROL SWITCH 11 In an interview following the incident, the operator involved stated that he had erroneously entered Salem Unit I Relay Room and had mistakenly operated the br~aker on No. IC Vital Inverter.
The operator overlooked the accepted operating practice of verifying proper equipment by identification prior to operation of the equipment.
Although errors due to inadvertent operation of similar equipment in the opposite unit had been observed in the past, they were low frequency in nature.
The incident was therefore assumed to involve an isolated personnel error.
"As noted, the operator immediately restored the No. IC Vital Inverter to operation.
The related A.C. source w~s declared operable, and at I427 hours, May 25, I983.
Action Statement 3.8.2.1 was terminated.
The operator involved in the incident was reprimanded.
Finally, a study of Salem tagging operations was recently completed and considers the potential for operator confusion resulting from dual unit operation.
Results of the study are presently being evaluated by the station; recommendations will be implemented as appropriate."
In conversations with the licensP.e's staff, some found the error hard ~
understand because when exiting the common hallway between the two control rooms you turn right to the elevator that goes only to Unit 1 equipment (rather than left to the stairs that must be used to get to the Unit 2 equipment).
However, there were several factors that may have contributed to the operator's error. First, the operator was relatively inexperienced in the field (approximately six months experience),
Second, at the time of the event the nuclear control_
operator (NCO) was working as a relief NCO for both Units I and 2.
Previously in the shift, the NCO had done tagging on the Unit 2 inverter.
He was then assigned to the Unit 1 control room, where he physically was when given the assignment to open breakers on Unit 2.
The NCO was convinced that he had a mental set regarding the unit he wa~
~
working on.
The operator felt the unit color coding was good but felt*
.the paper tags would also have to be color coded to be a reminder (perhaps just the border of the tags).
In addition, it was suggested that it is important for licensed operators to explain what is to be accomplished and to get feedback that they are understood.. As a result of the recognized potential of this sort of error, a surveillance and tagging operator is now assigned to each unit rather than one operator consistently working on both units during a shift.
- 3.
Incident Report 83-030 - Reactor Coolant Pump (Wrong Unit)
The licensee's staff described an incident that occurred but which was not reportable as an LER.
The sequence of events was described as follows.
The equipment operator (EO) was supposed to tag out 22 RCP (Unit 2) but tagged out 12 RCP (Unit 1) instead. The Unit 1 RCP was running.
The EO opened the I2 RCP breaker rather than the 22 RCP.
breaker.
He realized he had made a mistake so he then closed the breaker.
In the meantime, the control room operator (CRo)*noticed trouble with the pump and tripped the pu~p. The EO saw the breaker open
_. 4 -
and again closed the breaker~ At that time the equipment protective circuits opened the breaker, but the pump motor h~d been overheated.
Repairs to the pump significantly extended the Unit 2 outage.
The EO involved was a relatively new EO but had worked in the plant for f0ur to five years.
He had gone through the Memphi~ State training program and had received three months classroom and three months on the job EO training.
The licensee's staff indicated that they were uncertain as to whether this training included being instructed to never open or close breakers locally (bypassing protective circuits) on a running RCP.
The only known potential ctintributor to the error was that the EO had been doing overtime work (he had worked five or six 12-hour days).
- 4.
Incident Report ~ Wrong Transformer The final incident discussed with the Salem staff involved -Operation of the wrong transformer in the switchyard.
The EO misheard the control room operator's instructions and opened the 1T60 discdnnect rather tha~
the. 2T60.
The error was considered to have been the result of poor communications.
The licensee's staff indicated that the issue of improving communications is being evaluated at this time.
ENCLOSURE 2 WRONG UNIT/WRONG TRAIN EVENTS AT PEACH BOTTOM
- 1.
LER-278-81-008 -- CAD Line (Wrong Train)
The following event information was provided by the licensee in the LER:
"During the installation of a new penetration test connection to a 1 11 containment atmospheric dilution line (CAD), primary containment integrity was breached for a, short time.
With the unit at full load, a safety block was applied to a portion of.the CAD system to install a new test connection. A section of the 'A' loop CAD system piping to be modified should have been isolated by closing the manual valve between containment and the location where the pipe was to be cut.
Due to an incorrect location identification for the manual valve on a system check-off list, the manual isolation valve was closed on the 18 1 CAD loop instead of the isolation valve on the 'A' loop.. The construction work force then proceeded to cut the 'A' loop piping and to install a new welded tee connection.
The slight differential pressure which existed between primary containment and secondary contrainment alerted the craftsmen to the problem...
"The occurrence was due to improper designation of valve location on a system procedure which resulted in an operator being directed to close a manual isolation valve on the incorrect loop.
The isolation valves in the location did not have identification tags...,
"Operability of the 1 B 1 CAD system was re-established within one hour.
The equipment locations on the incorrect system check~off list have been corrected and identification tags have been installed on the manual isolation valves."
The licensee's staff that were interviewed indicated that they were unaware of any investigation, additional to the LER process, conducted to identify the possible contributors to the event.
The involved personnel were not available for an interview.
However, the licensee 1s staff indicated that possible contributors to this event include the lack of labels on valves, an incorrect system check-off list providing wrong location identification for the manual valve, and a confusing CAD system design.
On Unit 3 the 118 11 CAD system injects into the 11A 11 loop of the RHR system, which enters containment through the 118 11 penetration.
The same logic applies to the 11A 11 CAD system.
However, on Unit 2 the 11A 11 CAD system injects into the 11A 11 loop of the RHR system, which er.ters containment through the 118 11 penetration (Unit 3 was the 11A 11 penetration).
When working on Unit 3, the operator closed the manual isolation valve on the "B" loop instead of the isolation valve on the "A" loop.
- 2.
LER-278-85-008 -- Torus Test Bypass Valve (Wrong Component)
The following event information was prov'ided by the licensee in the LER:
.. a.
uon March 13, 1985~ at 9:45 a.m., with Unit 3 operating at 90%
power, System Procedure S.3.3.L., "High Pressure Coolant Injection (HPCI) Turbine Test Slow Start 11
, was being used to operate the HPCI turbine.
The procedure requires that the HPCI pump take suction from the Condensate Storage Tank (CST) and discharge back to the CST.
During the valve lineup, the control room operator mistakenly opened the torus test bypass valve (M0-3-23-31) rather than the CST test return valve. This resulted in approximately 37,000 gallons of water from the CST being pumped into.the torus. The torus water level increased from 14.8 feet to 15.3 feet, 0.4 feet above the technical specification limit.
When the valving error was distovered, the operator closed the discharge to the torus and shut off the HPCI turbine. The torus filter water pump was used to return the suppression pool water inventory to the proper level within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />...
11This event was caused by failure to follow procedures.
The procedure (S.3.3.L) requires the CST test bypass valve M0-3-23-24 to be opened, but the licensed operator mistakenly opened the torus test bypass valve M0-3-23-31...
"The HPCI turbine was shut down and the torus filter water pump was used to reduce the torus level to a value within the technical specifications limit within six hours.
The operator received prompt disciplinary action following the event.
11 The licensee's staff that were interviewed indicated that they were unaware of any investigation, additional to the LER process, conducted to identify the possible contributors to the event.
However, the licensee's staff indicated that the individual involved was a relatively new operator with less than six-months experience at the time, and that his inexperience may have contributed to the error.