IR 05000317/1987023
| ML20237A420 | |
| Person / Time | |
|---|---|
| Site: | Calvert Cliffs  |
| Issue date: | 12/07/1987 |
| From: | Lester Tripp NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20237A414 | List: |
| References | |
| 50-317-87-23, 50-318-87-25, NUDOCS 8712140468 | |
| Download: ML20237A420 (28) | |
Text
-
I
.
U. S. NUCLEAR REGULATORY COMMISSION
REGION I
Docket / Report:
50-317/87-23 Licer.se:
DPR-53 50-318/87-25 DPR-69 Licensee:
Baltimore Gas and Electric Company Facility:
Calvert Cliffs Nuclear Power Plant, Units 1 and 2 Ir spection At:
Lusby, Maryland
)
Dates:
September 1 - November 20, 1987 l
Inspectors:
T. Foley, Senior Resident Inspector
D. Trimble, Resident Inspector L.
rivi y, R actor Engineer Approved By:
/k7/[7 U6well E. Tripp," Chief, Reactor Projects Date Section 3A Summary:
September 1 - November 20, 1987:
Combined Inspection Report Nos.
50-317/87-23, 50-318/87-25
-
Areas Inspected:
(1).acility activities, (2) routine inspections coverage, (3) operational events, (4) ASCO solenoid valve and diesel generators, (5) Plant Operations Review Safety Committee, (6) radiological controls, (7) physical security, (8) Licensee Event Reports, and (9) reports to the NRC.
Inspection hours totalled 396.
Results:
l One violation was identified regarding inadequate corrective action (Detail 4.a); four instances are noted where components were declared operable and subsequent failures occurred due to the same root cause (see also Detail 3).
Inspection findings also raise concerns that (1) there is a lack of documenta-tion of useful data on Surveillance Test Procedures (STPs) which hampers trend-ing of component inadequacies and failures, (2) root causes are not pursued in a thorough or timely manner, (3) the Plant Operations Safety Review Committee is willing to L;1erate unidentified failure initiators, and (4) an apparent willingness exists to declare equipment operable before operability is thoroughly demonstrated.
871214046B 07120g 7 PDR ADOCK 0500 t
-
.
e DETAILS Within this report period, interviews and discussions were conducted with various licensee personnel, including reactor operators, maintenance and surveillance technicians and the licensee's management staff. Night shift
inspections were conducted on October 28 and 29, 1987.
Weekend inspec-tions were performed on September 19, November 7 and 8, 1987.
1.
Summary of Facility Activities Unit 1 commenced this period at 100% power and ran continuously until a September 11 trip at 5:15 a.m. due to a failed Reactor' Coolant Pump Surge Capacitor.
The unit returned to power September 12, and except for power reductions for cleaning-of marine growth from condensers, continued 100%
operation until November 5.
The unit then underwent a controlled shut-down in order to facilitate testing of degraded Auxiliary Feed pumps.
Repairs were completed by November 9 and power operation resumed on
'l November 10. The next day the unit tripped due to an electrical fault on i
one of the two unit main transformers. The unit was returned to service de-rated to 66% on November 13. The unit shutdown the following day to facilitate partial repairs to the transformer.
It was restarted on November 15 and operated at about 66% power until shutdown on November 20 for reconnecting the transformer.
Unit 2 entered the period at 100% and remained there until September 7
-
when the turbine / reactor tripped due to a turbine electro-hydraulic con-trol system oil leak on the pressure transmitter for an EHC pump. This was repaired and the unit returned to service later that day.
On October 28, the unit reduced power to 40% to repair an unisolable steam leak on
- 22 steam generator blow down line.
The licensee experienced dif ficulty j
with this repair and commenced a shutdown to cold shutdown on October 30.
The unit returned to service on November 1 and remained in service through
{
the rest of the period.
I A significant portion of NRC inspector and licensee's time was spent
involving AFW pump problem identification and discussions regarding oper-l ability, i
I 2.
Review of Plant Operation - Routine Inspections l
.
a.
Daily Inspection During routine facility tours, the following were checked: manning,
)
access control, adherence to procedures and LCO's, instrumentation, recarder traces, protective systems, control rod positions, contain-ment temperature and pressure, control room annunciators, radiation monitors, effluent monitoring, emergency power source operability, control room logs, shift supervisor logs, tag out logs, and opera-ting orders.
No unacceptable conditions were noted.
! - -
-
. _ _ _ _. _ _ _ _. _
l
_ _ _ _ - _
. _ _ - _ _ _ - _ _ _ - - - _ - _ - _ _ _ _
.
.
b.
System Alignment Inspection Operating confirmation was made of selected piping system trains.
Accessible valve positions and status were examined.
Power supply and breaker alignment was checked.
Visual inspection of major cun-ponents was performed.
Operability of instruments essential to
. system performance was assessed. The following systems were checked:
--
Unit 1 Auxiliary Feed Water System checked October 23 and 30, 1987 l
--
Unit 1 Emergency Core Cooling System Pump Room Air Cooling
,
checked September 21, 1987
'
Unit
Emergency diesel Generator Service Water checked
--
September 29, 1987.
The Calvert Cliffs interim Reliability Evaluation Program (IREP)
emphasizes the importance of the ECCS (Emergency Core Cooling System)
Pump Room Air Cooling System for ensuring the operability of the ECCS pumps. System fans and the salt water valves providing cooling water to the air coolers are not actuated t,y a Safety Injection Actuation Signal (SIAS).
Instead, they actuate automatically when room tem-perature reaches 104 degrees F (+0,
-5 degrees) and can also be manually actuated from handswitches in the Control Room. The inspec-tor confirmed that components associated with the automatic tempera-ture actuating circuit were on the Q-list and included in the EQ (Equipment Qualification) program.
A test of the automatic start functions is performed once per refueling cycle (once per 18 months to 2 years), and the valves are cycled and timed once per quarter.
Because of the importance of the ECCS pump room cooling system, the inspector asked.the licensee to consider testing the fans on a fre-quency more commensurate with the testing of the pumps they protect, i.e., more of ten than the current frequency of once per refueling cycle.
The licensee agreed that more frequent testing appeared appropriate and stated they would, in the future, test the fans on the same frequency as the valves (quarterly).
Emergency Operating Procedure E0P 5, Loss of Coolant Accic'ent, had been temporarily changed (CCOM Change Report 87-173) to manually i
start the fans and open salt water valves following an earlier licen-see discovery that EQ program upgrades were needed for components associated with actuation of the salt water valves.
Those upgrades have been completed, and it was the licensee's intent to remove the requirement for manual start of the cooling system from the E0P.
q r
_ _ _ _ _ _ _ _. _ _ _
. _ _ _ - - _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _
,
.
i.
In recognition of the impor tance of the room cooling function. the licensee stated they will'now retain the requirement for manual start of the system in the E0P as a backup for the automatic function.
No unacceptable conditions were noted, c.
Biweekly and Other I':spections During plant tours, the inspector observed shift turnovers; boric acid tank samples and tank levels were compared to the Technical Specifications; and the use of radiation work permits and Health Physics procedures were reviewed. Area radiation and air monitor use and operational status was reviewed.
Plant housekeeping and clean-liness were evaluated.
Verification of several tag outs indicated the action was properly conducted.
--
In Service Testing (IST) Inspection The inspector conducted a review of the licensee's implementa-tion of its IST program with particular emphasis on pump test-irg. Only minor inspection effort was devoted to valve testing.
7he inspector also conducted an indepth review of the licensee's recent activities to address anomalies encountered during the testing of salt water pumps.
The basic requirements for defining the IST program are included in the documents listed below.
The licensee implements these requirements primarily by the execution of its IST program which was submitted to NRR for approval on February 26, 1987.
The inspector reviewed the licensee's procedures, recent test re-i sults and other activities as listed below. Also the inspector l
conducted discussions with various licensee personnel. Based on
)
these reviews and discussions, the inspector verified that:
)
The licensee has assigned responsibilities to persons and
-
organizations for:
a.
preparation, review, and approval of IST procedures; b.
scheduling of IST for normal and increased frequency testing; c.
performance of testing per approved procedures; d.
proper certification and calibration of IST instru-ments; and e.
training for those personnel responsible for IST implementing procedures.
_ _ - _ - - _ _ - _ _ _ _ _ _ - _ _ _ _ _
_-_
. _ - -
.
The test procedures used are the latest ones approved and
-
that test acceptance criteria used are valid 'for the com-ponent being tested.
The licensee performs IST per an approved scheduled within
-
. the. limitations described in the IST prograin, including increased frequency testing.
IST results are recorded per the approved procedures and
-
that data are evaluated within the time constraints deline-sted in the ASME Code,Section XI.
The IST procedures and data reflect all the requirements.
-
of the appropriate edition of the ASME Code Section XI including:
-
a.
evaluation of imposing and removing increased fre-quency testing requirements; b.
evaluation and justification of changes to test acceptance criteria; c.
pump vibration test data analysis and acceptance cri-teria justification, including location of vibration measurement; d.
requirements that pump tests be conducted at reference conditions; and e.
compliance of test instruments to 10 CFR 50 and ASME code requirements.
The IST data are evaluated per the requirements of ASME
-
Code Section XI and that appropriate follow up actions are taken.
IST records are mainta.ined in accordance with ASME code
-
requirements. However, this area needs better organization consistent with good engineering practice and the ASME code record keeping requirements as noted below.
Section 3 of this report describes a weakness in documentation of defic-iencies discovered during inservice testing which are sig-
!
nificant but not directly related to Section XI acceptance
'
criteria.
"
References and requirements eviewed were:
-
10 CFR 50.55a(g), In Service Inspeccion Requirements i
ASME Code Section XI, Subsections IMP and IWV, 1983
-
Edition, including Summer Addenda
____-____-_____-_-_____N
-_- - _ _ _ _
_-. _ _
._
.
Calvert Cliffs FSAR and Technical Specifications
-
-
Regulatory Guide 1.33, Rev.
2,
"QA Program Requirements (Operations)".
Documents and activities reviewed were:
-
Calvert Cliffs Nuclear Power Plant IST Program for Safety Related Pumps and Valves, Rev.
O, 12/31/86, submitted on 2/26/87 Surveillance Test Procedure (STP) No. 0-73-2, "ESF Equip-
-
ment Performance", Quarterly Test, approved 9/30/87
-
QA Audit of Surveillance Test, Audit 87-03 with Audit Report dated 3/27/87
-
Calvert Cliffs Instruction (CCI) 104H, Appendix 104.60,."In Service Testing of Pumps and Valves".
In a submittal to NRR on February 26, 1987, the licensee pro-vided its IST program for safety related pumps and valves for the second 10 year interval.
The licensee indicated that this IST program including all relief requests would take effect on April 1,1987, for both Units 1 and 2.
The Operations Surveil-
=
lance Coordinator (OSC) is responsible for implementing the IST program through the execution of CCI-104H, Appendix 104.60.
This document defines the administrative controls that are necessary to properly implement the IST program. This includes actions such as:
The OSC is responsible to ensure pump retesting occurs at
-
30-day intervals (15-day intervals for Auxiliary Feed Pumps
.
(AFPs) if during in service pump testing a measured value for any parameter falls within the " Alert Range".
j The OSC will maintain a Supplemental Test Sheet Log depict-
-
ing the current status of all pumps and valves which have j
failed to pass their respective inservice tests.
)
The OSC develops a daily schedule each month for conoucting all STPs and provides this schedule to Operations and Performance Engineering personnel who perform the actual testing. All pumps except the AFPs are tested quarterly and are inc'Med in STP N0.
0-73.
The AFPs are testing monthly in STP No. 0-d.
If problems
]
are encountered, the OSC is instrumental in obtaining the re-quired engineering assistance from the plant and project engi-neering and design engineering groups.
The inspector was ad-vised that the OSC was the only individual involved in IST on a full time basis.
From a QA standpoint, inspectors in the QC
,
-
- _ _ - - _
. _ _ _ _ -_
__ ___
___ _ _ -
.
operations unit perform surveillance on selected STPs on an ongoing basis.
Also, the Quality Audits unit conducts annual audits of all surveillance testing which includes IST.
The inspector made several observations as discussed below while reviewing the results of the June and September 1987 in service pump testing and the licensee's response to QA Audit 87-103.
While pump testing was being conducted to comply with ASME Code Section XI, the IST program was not yet functioning smoothly for several reasons:
New baseline data (reference values) was required for each
-
parameter to be measured due to the licensee's commitment to the 1983 ASME Code Section XI.
While obtaining new reference values for certain pumps, system related ques-tions and problems arose which had to be addressed.
For example, the #22 High Pressure Safety Injection pump recir-culation line was determined to be plugged.
Evidence of low flow in this line necessitated x-rays of a flow-restricting orifice which showed metal chips in the ort-fice. Also, the performance of the #11 ar.d #13 salt water q
pumps was called into question when September testing showed the #11 salt water pump to be in the " Low Alert" range and the #13 salt water pump to be in the " Low Action" range for the f' low parameter.
The IST program requires firm reference values for pump parameters and prompt cor-rective action when act iptable parameter values are not obtained. While the licensee was taking appropriate cor-rective action to address these system problems, the licensee admittedly was struggling with the implementation f
of its IST program with only one full time IST individual.
-
Performance of STP No. 0-73 involves the testing of twenty-one (21) safety related pumps which takes 1-2 weeks to com-plete for each unit and possibly can lead to confusion and error, especially at this transition stage of the IST pro-gram. In this regard, the inspector noted that #21 and #22 Low Pressure Safety Injection pumps had unacceptable high flows in the September testing due to different ini tial conditions between Ju,e and September testing.
Also recording of test results and follow up actions for #21 pump on one STP cover sheet was difficult to review due to the questionable results on several pumps. These procedure adequacy concerns have the potential to hinder proper IST program implementation.
- _ _ _ _ _ _ _
_ _ _ _ _ _
._-_-
l
<
.
,
I The inspector's review of QA Audit 87-03 indicated a lack of
-
aggressiveness by plant and project engineering management to respond to a QA finding concerning the fact that system engi-neers reviewing STPs are not properly trained and certified.
As noted in other inspections, system engineer training is im-portant for the proper support of overall plant activities and not just limited to the IST program. The inspector noted that Audit 87-03 was conducted in January 1987, with an original j
response due in May 1987.
Plant and project engineering re-a ceived QA management approval for several extensions of the response due date.
Final response is now scheduled for October 31, 1987.
During the course of the inspection, the inspector had the fol-
'
lowing additional observations:
There was evidence of constructive involvement from engi-
]
-
neering personnel in the resolution of the salt water pump anomalies. Major contributors were more experienced per-sonnel in the design engineering group while the less i
experienced (1986 graduate) salt water system engineer contributed to a lesser extent.
The inspector noted positive QC involvement with mainte-
-
rance persnnnel in conducting repairs to #13 salt water
.
pump.
-
The inspector noted a positive attitude in the performance engineering personnel who support IST in providing flow and vibration data.
These individal were well acquainted with their specific duties as evidenced by an in depth demon-stration of their flow and vibration equipment.
The OSC had a thorough understanding of the requirements of
-
the IST program.
Although the Manager of Engineering Services recognized
-
that the system engineering training 3rogram had gotten off to a slow start, the licensee was comnitted to complete an initial system engineer training eff ort in October 1987.
In response to the concerns identified by the inspector in the IST program and STP No. 0-73, at the exit meeting, the Manager of Nuclear Operations committea to the following:
,
_- _
__
_ - _ _ _ _ _ - _ _ _ _ - _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ _ - _ _ _ - _
_ _ - - - -
_ - _ _ _ -
.
.
-
Another individual wocid be assigned full time to the IST program to assist the OSC.
Both individuals woulo work to improve the IST program organization by clarifying and properly documenting equipment baseline data.
The objec-tive would be to upgrade the IST program organization and facilitate future inspections of the IST program.
-
The procedure group would revise STP No. 0-73 to eliminate possible confusion and thereby avoid errors which poten-tially exist in the current procedure format.
The inspector acknowledged these commitments.
No unacceptable conditions were ncted.
l 3.
Operational Events l
a.
Auxiliary Feed Water (AFW) Pump Problems (1) System Description The Final Safety Analysis Report states that three auxiliary feed water pumps are iiistalled consisting of one motor driven
and two non-condensing steam turbine driven pumps. For a shut-l down, only one pump is required to be operating, the others are in standby.
Upon automatic initiation of auxiliary feed water, one motor driven and one turbine driven pump automatically start.
The turbine driver is supplied with steam from the steam generator as long as the pressure is above 50 psig.
Each tur-bine has a manually set governor for controlling turbine speed.
Once set for a certain speed, the governor is designed to main-tain approximately constant speed with a minimum of 50 psig steam pressu-e.
The motor driven pump is supplied from an electrical bus which can be powered by a site emergency diesel generator. The steam driven pumps have a capacity of 700 gpm at 2490 feet of head.
The motor driven pump has a capacity of 450 gpm at 2800 feet of head. A feed rate of 300 gpm to the steam generator (s) is necessary to remove decay heat and reduce the RCS temperature to 300 degrees F.
These pumps take suction from a 350,000 gallon condensate storage tank which is protected against tornadoes. The turbine driven pumps operate reliably as long as there is steam pressure in excess of 50 psig in one of the steam generators.
If necessary at this point, with site power available, the steam supply can be switched to the auxil-iary boiler steam system.
In addition, in an emergency the steam driven train can operate independent of off-site power and the diesels for up to two hours.
The AFW air accumulators pro-vide a sufficient control air source until operators could
..
- _ _ _.
i i
"
I manually regulate the system.
The steam generator's auxiliary I
feed water system is initiated by remote manual control or auto-
matically on low level in either steam generator. Each flow leg has a flow control valve.
The valve can be set for automatic operation or placed in remote manual control from the main con-trol room or auxiliary shutdown panel.
Each auxiliary feed water discharge flow leg is supplied with two block valves in series.
During a main stea,n line break the block valves in the flow legs to the ruptured steam generator will automatically shut.
A motor driven pump discharge header cross connect is j
available to provide flow, if necessary, to the other unit if either No. 23 or 23 pump is inoperable.
The turbine is a Terry Steam Turbine GS-2 with a rated speec of 3990 RPM and a set overspeed trip of 5250 RPM (Nameplate) (5200
)
+/-250) which was checked during the previous refueling outage.
]
The governor is a Woodward PL-PG governor set to limit speea to
3990 RPM.-
(2) Problems Prior to Inspection Period During the past sew ral months, inspectors became aware of possible inadequacies within the surveillance test program.
Operators would not always record on surveillance test proced-ures (STPs) tripping or start f ailures of the AFW pumpt since these were not parameters trended or used for acceptance cri-teria.
The very limited scope of the STP acceptance criteria
,
would not pr9hibit a situation in which a pump could pass its I
surveillance even if it tripped during testing.
Further inves-tigation determined that auxiliary feed pumps had failed to start the first time on several occasions.
Because of the l
repeat nature of the problem, it appeared that the root cause was not being identified and corrected.
On May 26, 1937, the inspector witnessed the testing of AFW pump
Unit 1 was in Mode During the test the tur,bine
'
tripped the first time and ope ators re-latched the trip device
.,
and attempted the initiation again.
The turbine tripped a
!
second time. The operators sune oned a maintenance technician to adjust the overspeed trip mechanism. The maintenance technfcian observed the mechanism and the pump was restarted. The turbins tripped a third time. The technician indicated that the gover-nor might have to be replaced.
The inspector left to obsedve other startup testing.
Subsequent review of surveillance records revealed that on May 24-27, 1987, no indication of #11 i
i
_ _ _ - -
-
_
f
.
AFW pump tripping was documented on surveillance test prScedures STP-0-5 or 0-9A (STPs for #11 AFW test performed on May 24-27 were not found).
However, the shift supervisor's log of May 26, 1987 states at 1420 hours0.0164 days <br />0.394 hours <br />0.00235 weeks <br />5.4031e-4 months <br /> "in Action Statement 3.7.1.2
- 11 AFW out of service during performance of STP-0-9A-1" and, at 1540 hour: the " steam driven AFW pump No.11 trips on overspeed each time steam is brought into it normally". On May 26, Main-tenance Order M0207-146-844A was performed to replace #11 AFW pump governor. The MO states that "during STP old governor was allowing turbine to trip on overspeed". STP-0-9A-1 and STP-0-5-1 were performed as the post maintenance and surveillance tests and were completed on May 28, 1987.
The pump was considered j
operable at that time.
During the previous months, resident inspectors had witnessed other AFW pump failures to initially start, with subsequent successful starts. Two of those failures occurred due to water hammer while testing on auxilia ry steam (normally the pumps would use main steam) and were not therefore totally representa-tive of performance on main steam.
However, they did indicate possible oversensitivity in the trip latch mechanism.
Discus-sions with operators indicated that the pumps failed to start occasionally but usually the operator would be successful in operating the pump during the second try.
This apparently was not perceived as a major concern because of redundant pumps (the units' 300 percent installed AFW capacity including 2 steam driven pumps and 1 motor driven pump), with the motor driven pump having high reliability.
It was also stated by licensee representatives that the steam driven pumps' reliability was a known weakness and that one of many reasons the motor driven pumps were installed was to compensate for this reduced reliability.
In the shift supervisors log :n June 30, 1987, the supervisor noted at 1625 hours0.0188 days <br />0.451 hours <br />0.00269 weeks <br />6.183125e-4 months <br /> that during the performance of STP-0-5 and 0-9A-2 on Unit 2 "each AFW PP tripped on its first start, due to water hammer" Those pump trips were not documented in the STPs.
Pump operability was declared at 1940 hours0.0225 days <br />0.539 hours <br />0.00321 weeks <br />7.3817e-4 months <br /> after a second test.
Work to correct, investigate, or ether demon-strate that the suspected root cause was cor,,
ed was not immediately initiated.
However, monthly tests were begun of a more challenging nature and which were more representative of actual conditions under which the pump would have to start.
On July 23, during a total loss of off site power, although not stated in the Licensee Event Report or the shift supervisor's log, the in-service steam driven pump failed to start the first time.
.
_
._
_ _.
__
.
.
As noted above, start failures were not always identifieu on the surveillance tests. One explanation provided by' operators per-forming such tests was that starting the first time was not part of the. acceptance criteria and therefore initial trips were acceptable.
The procedures, however, do provide a space for comments about unusual operational cha racte ri stics, but were generally left blank.
Subsequently, the inspector di scussed the above with the Managers of Engineering and Operations.
Additionally, the inspectors cited the following either active cr previously addressed problems with the AFW systems:
turbine bearing overheating
-
-
bearing oil contamination with water
-
excessive pump packing leakage ineffective turbine drain design
-
steam inlet valve stroke times are often slow
-
-
solenoid valves which previously were not qualified for the existing instrument air pressure sporatic tripping of AFW pumps
-
-
spurious AFAS actuation inability to test steam line check valves
-
-
critical oil levels in governor causing turbine hunting or trips
-
excessive moisture in the steam lines.
The managers acknowledged many of the problems but questioned the concern about the ability of the AFW pumps to start on the initial attempt.
The reliability of the AFW pumps was pre-viously thought by management to be adequate. Subsequently, the plant manager, as noted above, in order to demonstrate either the pump reliability or its' inadequacies, directed that the AFW pumps be tested on a more frequent basis (monthly) with the more comprehensive (STP-0-9A) refueling surveillance and emphasized that all fai' lures and inadequacies will be documented on the test procedure.
This was re-emphasized in the General Supervisor-Operations Night Orders.
(3) Surveillance 'last Failures and Troubleshooting Efforts On October 23 at 7:33 a.m. during the performance of STP-0-9-1 ( AFAS Monthly Logic Test), #12 Auxiliary Feed Water pumo speed oscillated due to governor hunting.
The turbine then tripped app.arently on overspeed. The pump was declared inoperable and a Technical Specification seven day Limiting Condition for Opera-
,
tion (LCO) Action Statement began. The licensee performed main-
'
tenance en the pump including adjustment of the linkage and replacement of the overspeed trip lever.
The test was per-formed again at 5:00 p.m.
and the pump did not trip but the governor valve oscillated severely (hunted).
_ _ _ _ _ _ - - _ _ - _ _ _ _ _ _
.
.
The STP was re performed on October 24 at 0500 hours0.00579 days <br />0.139 hours <br />8.267196e-4 weeks <br />1.9025e-4 months <br /> after a 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> cool down period and the pump tripped. Oil was added to the governor and the test performed ag&in satisfactorily except hunting was noted. The pump was declared operable.
On October 25, the test was re performed and the pump again tripped seemingly on overspeed.
The licensee then declared it inoperable and began an investigation and testing program to attempt to identify and correct the problem.
During the succeeding six days the pump was tested daily one or more times.
The licensee sought information from the industry on AFW problems and brought the vendor representative on site.
The investigative team focused on water in the steam lines, worn control linkage, governor replacements and oil flushes.
The pump was tested successfully five times and seven times unsu::-
cessfully between October 24-29.
Dr. October 28 and 29, the pump was tested several consecutive times without tripping after a governor replacement and imple-mentation of a drainage plan to remove water upstream of the trip valve.
The licensee representatives stated that a test would be performed (after 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> for cooling) to declare it operable if the pump did not trip in spite of hunting.
On October 30 at 1:05 a.m., a test of #12 AFW pump was performea af ter. draining all the associated steam pipe and pump casing drains five hours prior to the test.
The pump was previously aligned for normal automatic initiation and directed to feed the steam generators (vice aligned for recirculation back to the condensate water storage tank (CST)).
The system was actuated from the Control Room.
The turbine / pump smoothly came up.to 4700 RPM and remained there for approximately 20 seconds; water was emitted from governor valve bushing area, turbine seals, and the turbine casing relief.
Then, without apparent oscillation
or other significant parameter changes the turbine tripped.
j Control Room parameters indicated an initial flow of 450 GPM.
Then flow automatically regulated back to 150 GPM as designed.
The pump remained inoperable.
The licensee decided to windmill the turbine at about 500 RPM for three hours anri attempted to run the pump on recirculation back to the CST.
The purpose of this was to warm the steam lines and purge any moisture out of the system.
.
_ _ _ _ _ _ _ _ _ - - - - - - - - - -
_____
_ _ _ _ _ - _ _ _ _ - _ __
_ _ _ _ _ _ _ _ _ _ _ _ _ _
.
At 4: 00 a.m., the turbiae/ pump was initiated from the Control Room.
The pump smoothly came up to 4000 RPM, ran smoothly for 10-20 seconds, then began oscillations of 200 RPM (pump speed 4000 +/- 200 RPM). The most significant effect of this was that the governor valve and linkage were operating in a very rapid manner from seemingly full open to full shut at a frequency of one cycle every 2-4 seconds, about 20 times per minute.
The licensee secured the pump after 2-4 minutes of operation.
Apparently, significant damage to the pump speed control linkage could occur had the pump not been secured.
The licensee adjusted the windmilling to about 1000 RPM in order to force the bearing oil slinger rings to move with the shaft rotation, and again commenced purging the steam lines.
At a 6:00 a.m.
test, the pump came up to speed and, approxi-mate j five seconds after speed stabilized, it then began hunt-ing. Manual throttling down on the trip throttle valve enabled i
the governor to stablize control.
Smooth operation continued through normal cycling of the remote governor controller even after the trip throttle valve was fully re-opened. Additional attempts to create upset conditions by cl.anging pump speed while in recirculation caused no oscillations.
At 6:30 a.m., No. 12 AFW pump was placed in a windmilling condi-tion (at about 900 to 1000 PRM) as the on-line pump, with #11 AFW pump in standby.
At 0800, #12 AFW pump was automatically started successfully. TFe pump smoothly came up to speed on the governor and exhibited no oscillation.
The licensee determined that the condensate and moisture had finally been minimized after approximately six hours of warming and continuous steam flow.
The #12 AFW pump vas then declared operable.
Later on October 30, NRC management conveyed via a telephone conversation with licensee managers concerns about the declara-tion of operability of the #12 AFW pump af ter only one success-ful test.
The licensee believed that their troubles'ooting n
efforts had characterized the root causes, i.e., excessive mois-ture in the steam lines and worn control linkages. Immediately following that phone call the inspector performed a partial walkdown of the AFW steam piping drain system and found that the two primary upstream drains were isolated and rendered ineffec-tive by a improperly closed valve (1-MS-306).
The valve was immediately opened.
During the telephone call and later on October 30, additional successful starts were done on #12 AFW pump. On November 2 the licensee submitted a letter justifying the basis for determination of operability of #12 AFW pump and the planned action to be taken to improve the reliability of tne pump.
The licensee stated the cause te be:
_ _ _ - _ _ _ _ - - _ _ -
f i
.
.
l
"We believe the two underlying problems which caused the observed performance are water conderisation in the steam lines. to the turbine driven AFW pumps and the degraded clearances on the governor valve-linkage due to wear.
A Field Representative from Dresstr-Rand has been assisting
.in troubleshooting and repair activities to date.
His analysis of the situation concurs with the above conclus-ions, and is supported by data available in the Nuclear Plant Reliability Data System, and discussions with both MPR Associates and Bechtel representatives."
The licensee indicated that the following actions had.been taken i
in the interim:
'
-
daily starts of the pump to simulate an automatic demand, from a wind nillir.g condition, without operator action;
-
a dedicated operator additional to the minimum staffing requirements;
-
procedure revisions to direct proper operation of the AFW system from a windmilling conditions and to dampen out oscillations.
The licensee also stated:
"We are not satisfied with the current performance of No.
12 AFW pump, althoegh we feel it is still OPERABLE, and will aggressively pursue resolution.
If the AFW pump reliability has not been sufficiently improved by Friday, November 6,1987, at 0733, we plan to initiate an orderly reactor SHUTDOWN of Unit 1 in accordance with' Technical Specification 3.7.1.2 ACTION Statement a.2 (b).
We will remain SHUTOOWN until the situation has been improved to our satisfaction."
On November 2, NRC Region I sent to Baltimore Gas and Electric
]
Company Confirmatory Action Letter 87-15 to reiterate NRC con-
cerns regarding the numerous turbine trips, the unidentified and uncorrected root cause of the trips and the licensee's deter-mination regarding pump operability.
The letter confirmed the licensee's actions regarding the interim controls and the licensee's commitment to place the unit in a safe condition by November 6, should reliability not be sufficiently improved.
_
___ _A
_ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ - _
t
.
.
Subsequently, un November 3, the redundant pump (#11 AFW pump)
tripped during surveil' lance testing.
The pump was declared inoperable and retested six additional times unsuccessfully.
Each test displayed characteristics of hunting similar to that of #12 AFW before the continuous windmilling, One' test was i
completed without tripping.
This would place the licensee in a
72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement if both pumps were considered inoper-able.
However, as described above, the licensee had already committed to a shutdown on November 6.
The licensee removed Unit 1 from service on November 5 to facilitate troubleshooting and re,nairs for the degraded / inoperable AFW pumps.
.
Because NRC's review of BG&E's basis for the determination of operability of #12 AFW was predicated on the existing conditions at the time, i.e. that #11 AFW pump was operable, NRC sent, on November 6, a supplemental Confirmatory Action Letter (87-15) to assure that #12 AFW pump remained in its existing condition which provided the most reliability for the system while repair i
I activities continued for #11 AFWP and to modify the previous letter to permit testing of the system under hot standby conditions.
l On November 4-6, the licensee changed governors, manufacture new linkage, overhauled the throttle valve and fixed air leaks
'
in the instrument air lines between the I/P converter to the governor on #11 AFWP.
This was followed by providing various instruments for on-line trending and recording of pump para-meters.
Subsequently, two tests failed and two tests resulted in hunting without trips.
Orie test performed utilizing auxil-iary steam ran smoothly with oscillations.
On November 6, the licensee ordered and received r.s buffer springs for the governor (a vendor recommendation).
The licen-see was unaware that three different strength springs were sailable for use with each governor.
The vendor recommended the strongest spring available (F type) for the application of the steam driven pumps.
It was later determined that all four pur ps had the weaker, either (E) or (G) springs, in the governors.
The licensee replaced these springs in #11 AFW pump governor and performed four successful tests of the pump without oscilla-tions. Another test was conducted six hours later, and then #11 pump was aligned for automatic initiation. The Plant Operations Review Committee considered #11 pump operable for the purpose of working on #12 AFW. Then en November 7, after waiting 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />,
- 11 AFW pump was again tested and declared fully operationa _ _ _ - _ _ _ _ _ _ _. _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _
.
.
Subsequently, on November 8, #12 AFW pump had its springs re-placed and governor valve and control link 9ge refurbished.
It was tested several times without hunting or tripping under var-iou: speed and flow path conditions. A similar test program to that of #11 AFW pump took place, (two pump starts at four hour intervals and one after 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />).
On November 9, #12 AFW was declared operable.
On November 8, the buffer springs on Unit 2's #21 and 22 AFW pumps were checked, found to be G-type, and replaced with the stiffer F-type springs.
Both pumps were then tested.
During the #22 pump's test, significant governor oscillation was noted.
The oscillations continued for about 3 minutes and then stopped i
with no operator action.
The test was repeated on #22 pump.
Performance was satis f acto ry, and the pump was declared oper-able. Following the report period, on Noven'ber 25, #22 pump was again tested, and significant guvernor hunting occurred.
De-
)
tails of this problem will be addressed in Inspection Report 317/87-27;318/87-28.
This again illustrated a situation in which the licensee declared operability based on a single successful test and thereby did not identify an inherent equip-ment problem.
On November 9, #11 AFW pump was again declared inoperable to
!
perform maintenance due to a significant quantity of we+er being noted in the pump bearing oil sight glass.
The ; smp was returned to service that day.
In spite of the #22 AFW pump problem, replacing the governor buffer springs appeared to dramatically improve pump performance
$
of all AFW pumps and was a principle solution to the sporadic f
overspeed tripping and hunting problems.
The licensee has,
.
however, still noticed some increased sensitivity of pumps in the standby alignment to oscillations.
I'ne licensee attributes I
this to the fact that the 4-hourly drains of the standby pump turbine and steam line are not as effective as those performed I
on the on-line pump.
Therefore increased amounts of water accumulate in the standby pump steam line.
l One weakness was noted by the inspector in the licensees testing procedures in that only one of the main steam admission valves was being opened (CV-4070 or CV-4071).
During actual plant i
transients calling for AFW actuation, both valves will open and
!
will thereby more severaly challenge AFW turbine controls.
The tests should be as representative as possible of actual condi-tions.
The licensee incorporated this change into test procedures.
)
l j
l
-
- - _ _ _ _ _ _ _ _ _
.
j
.
(4) NRC/ Licensee Management Meeting
,
On November 10, a NRC/ licensee management reeting was held at the Region I office to discuss the AFW problems.
In this meet-ing the licensee pointed out they had been taking steps to study and improve AFW system reliability prior to the recent events.
There included such items as a system reliability study and modifications of the main steam admission valves. A number of propesed improvements had been under evaluation, j
They described the evolution of their current surveillance test-ing program and pointed out that this testing has gradually j
progressed toward the point of matching actual conditions under j
which the pumps are designed to perform.
Therefore the tests i
are becoming more challenging to the control systems and more clearly identifying system weaknesses.
The NRC noted that it was apparent that the controls for the pumps had been allowed to
'
excessively degrade over a long period of time making the pumps more susceptible to these types of problems. The licensee de-scribed longer term upgrades they plan to make to increase sys-
)
tem reliability including improvements in steam line drains, j
governor linkages, and main steam admission valves, j
Many improvements to the AFW system have resulted from these events. Hawever, many problems continue to exist (as previoJsly identified in the beginning of this section).
For example, as recorted in Licensee Event Report 87-07 on October 12, an un-
planned actuation of the Auxiliary Feed Water Actuation System occurred during a maintenance activity.
The actuation occurred I
while de powering the cabinet. The root uuse of this actuation
]
remains unknown.
(5) Summary The above problems were caused by the licensee's failure to (1)
adequately document, trend and evaluate equipment performance and correct inadequacies; and (2) a lack of detailed knowledge of plant equipment by both maintenance and engineering.
The licensee did not utilize an aggressive, knowledgeable test group tc diagnose these types of problems and thoroughly pursue root cause in a timely fashion.
It appears that the licensee does not always pursue a problem to positively identify the root cause in a timely manner, unless power operation is jeopardized.
This was addressed in the 1986
Systematic Assessment of Licensee Performance (SALP) report.
Tne AFW system degradation exemplifies another situation in which NRC involvement was necessary to obtain the root cause identification of problems with components which the licensee
,
- - __--- _
_ _
_ _ _. - - _ -
_.
!
ig..
.
1
+
had previously declared operable when, in fact, component sur-veillance testing demonstrated obvious degradation in equipment i
performance or reliability.
Plant Operation and Safety Review Committee members based their determination 'of operability on l
whether the component will fulfill its intended function.
They
'
do not evidently cons; der all the design bases or FSAR require-ments. The committee is not tasked with determining whether the root cause is identified or what effect it may have on opera-bility. These issues were discussed with the Vice President of Nuclear Energy.
These examples will be utilized in another section of this report.
!
'
b.
Emergency Diesel Generator (EDG) Speed Sw' itch On November 4, during the performance of STP-0-7-2 Engineering Safety Features Logic Test for Unit 2,
an " Auto Start Block Alarm" was received on the control panel for #11 EDG.
Subsequently Maiaenance Request 11327 was issued which determined that the 810 rpm switch
,
contact was not closing on the diesel speed nitch.
The switch carc
tains two contacts which at 250 rpm secures starting air and at 810 rpm opens service water valves for cooling plus other functions.
The reason for switch failure could net be determined. A new switch was obtained, tested and installed.
Tt e #11 EDG was subsequently tested with STP-0-7 and returned to service.
Subsequently, on November 10,
- 11 salt water header was removed from service for c1 caning.
This necessitated that #11 EDG be declared inoperable as well due to a lack of cooling. On hovember 12 at 0056 hours6.481481e-4 days <br />0.0156 hours <br />9.259259e-5 weeks <br />2.1308e-5 months <br /> during the performance of STP-08C-0 on 11 EDG, a " failed to start" alarm was received. Oper-ators assumed that the cause was failure to obtain adequate fuel supply to the injectors because of previous maintenance on the fuel injectors. At 2:16 a.m. during the performance of STP-0-7 for both
At 4:33 the maintenance department completed troubleshooting and deterrnined the speed switch was faulty.
Subsequently, MR 26773 was issued stating that "11 EDG i
will not stay running when started resulting in a start failure" Electricians removed and inspected the speed switch again. Internal parts were damaged. Two other previously broken speed switches were disassembled to use parts to refurbish the original Switch.
The switch was berch tested and installed. The diesel was tested and the start failure recurred.
The electricians dismantled the switch
!
installed earlier and found a spring had fallen off the centrifugal
- counterweights.
The internals were knurled and non-functional.
A used switch previously removed from #12 EDG and calibrated was ob-tained.
The inspector witr.essed shop.esthg of this switch.
The l
switch was then installed in #11 EDG and tested per STP-0-8.
j l
l l
l i
l l
__
.
_ _ _ _ _ - _ _ _ _ _ _ _ _
. _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _._ _
_
. _ _.
-.
_ - _____.
_ _ _ _. - _ _ _ _ _ _ _ _ _
.
At this time the licensee was making preparations for returning the I
unit to power operation.
The unit had been shutdown due to a trans-
)
former bushing failure.
After commencement of the test and once the rated speed and voltage l
were obtained as required by the STP, #11 EDG was declared operable
!
at 1520 hours0.0176 days <br />0.422 hours <br />0.00251 weeks <br />5.7836e-4 months <br /> on November 12. At 1555 hours0.018 days <br />0.432 hours <br />0.00257 weeks <br />5.916775e-4 months <br /> the licensee commenced
!-
reactor start up by pulling the shutdown control rods out. However, at 1615 hours0.0187 days <br />0.449 hours <br />0.00267 weeks <br />6.145075e-4 months <br /> at the completion of STP-0-8, the auto sttrt. block alarm recurred.
The EDG was again declared inoperable and reactor start up secured and rods driven back in.
The switch was again re-moved and dismantled.
The technician replaced the springs and ad-justed the RPM setting of the 250 rpm switch. The 250 rpm switch was found in the closed position thereby continuously preventing starting air flow starting the EDG.
The switch was calibrated and installed again.
The licensee tested the EDG again with STP-0-8 successfully and declared it operable at 2210 on November 12, and then immediately commenced reactor start up. At 2215 the unit changed to Mode 2 and was critical by 2240.
The inspectors are co;.cerned about the licensee's inability to thoroughly test components and the apparent eagerness to declare com-ponents operable before they are adequately tested and demonstrated to be so.
This is an additional example of the aforementioned fail-ure to promptly identify and correct conditions adverse to quality.
j
!
The need for comprehensive post maintainence testing and the philos-ophy of declaring components operable after a single test warrants management review. Had this particular failure not been self reveal-ing via an alarm the diesel may have remained inoperable until called upon by a test or operational event.
>
No violations were identified.
,
4.
ASCO Solenoid Valve and Diessi Generator Problems During the period, four problems associated with the emergency diesel generators (DG's) were idertified.
One (speed switch failure) is de-scribed in section 3 of this report.
The remaining three problems con-cerned:
(1) the occurrence of an intermittent failure of DG12 (due to leaking solenoid valve) for which the licensee did not take adequate cor-l recthe action, (2) 6 problem involving substitution of alternate model
.
solerecid valves on all three DG Salt Water (SRW) valves without requisite i
engineering review as a possible design change, and (3) a separate defici-
'ency involving the use of a wrong design solenoid valve on all three DG
SRW supply valves.
The circumstances permitting the solenoid substitu-l tien problem above to have occurred also could have resulted in improperly
!
eva;uated substitutions to have been made on other safety related systems.
-
_ _ _
__ - - _ _ _ _ _
l l
'
.
a.
Intermittent +. uiesel Failure During DG surveillance testing (STP 0-7 and STP 0-8) at 11:26 a.m.
y on September 8,1987, the #12 diesel tripped on high Jacket cooling j
water temperature.
Operations personnel unsuccessfully tried to
'
,
I identi fy a root cause.
The problem, however, was intermittent in l
nature and did not recur during a subsequent ' retest.
Based on the l
one retest and the fact that limited troubleshooting could identify l
no reason for a trip, licensee management concluded that the diesel was operable.
A loose fuse and holder assembly associated with a handswitch (for selecting either the Unit 1 or Unit 2 service water
,
system for supplying cooling water to the diesel) was thought to be'
'
a possible cause in that, coincidental with the trip, normal position indication of service water valve position (caused by the loose as-sembly) was not available. However, prior to the diesel r test, a check was performed that indicated that'the loose fuse holder was not the cause of the diesel trip. Although it was not on the agenda, at the urging of the NRC inspector, on September 9 the Plant Operations and Safety Review Committee (POSRC) discussed the trip of DG12 and determined that the diesel should be considered administratively inoperable and that further troubleshooting should be done. At this time, DG21 was out of service for maintenance.
Over the next 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, troubleshooting efforts (coordinated by systems engineering group personnel) were performed on DG12. However, these efforts were unsuccessful in identifying a root cause.
On September 10 the POSRC was briefed on the troubleshooting efforts.
Because the problem had not recurred during subsequent testing and no root cause could be identified, at 7:30 p.m. DG12 was declared operable. It was decided that more frequent testing was not necessary.
In a subsequent dis-cussion between the NRC inspector and licensee management representa-tives, the licensee indicated that the diesel trip was viewed as a random occurrence.
They stated, however, that they would require additional troubleshooting to be performed on DG12.
They did not specify to the inspector a time frame in which that troubleshooting would be performed.
To the knowledge of the responsible systems engineer, the troubleshooting accomplished between September 10 and September 28, consisted of in-of fice review of applicable circuit logics / configuration for potential sources of the overheating symptoms.
On September 28, DG12 was removed from service for maintenance not related to the above problem (repair fuel oil leak).
Following maintenance completion, post maintenance testing was performed.
Dur-ing the testing ( at 4:45 p.m. on 9/28/87), the diesel had to be un-loaded after 9 minutes of operation and shutdown due to low raw water
!
(service water) pressure because the SRW supply valve (1-SRW-1588)
l failed to open in automatic.
A second test was run and again the diesel had to be unloaded af ter 6 minutes of operation and shutdown
-_-
_
_ - _ - _ _ __
_
_ _ _
._
_ _ _ _
_ _ _
_. __
_
. _ _ _ _
_ _ _ _
_ _ _ - _ _ _ _
.
..
because the SRW valve was not functioning properly. Troubleshooting revealed that the ASCO solenoid valve (1-SV-1588) associated with the SRW supply valve was leaking (seat leakage) when in the de-energized position (its position when the DG is operating) allowing sufficient passage of instrument air to cause closure of the SRW valve.
By design, when the solenoid valve de-energizes, it (1) isolates instru-ment air from the SRW valve operating diaphragm, (2) vents residual air off the top of the diaphragm back to a controller of a constant air bleed design (allowing the valve to open), and (3) through the same path, allows an output air pressure signal from the controller to regulate the SRW valve position.
The controller then regulates SRW valve position, as necessary, to maintain a specified differen-tial pressure across the DG heat exchangers.
A new solenoid valve (described below) was installed, and the problem was corrected. The solenoid valve air leakage problem was the apparent root cause of the intermittent September 8 DG failure.
In retrospect, a very undesirable condition existed during the period l
of September 8-28, in that the reliability of DG12 was significantly l
reduced by the presence of an unidentified failure initiator. Addi-tionally, on at least one occasion (September 9) during that period, a second DG (only 3 DG's on site) was taken out of service for main-tenance, even further reducing emergency power system reliability.
The September 8 decision, immediately following the initial failure, to consider DG12 operable after very limited troubleshooting and only one successful test particularly reflects a lack of conservatism in approach to correcting significant equipment deficiencies.
The event, as a whole, is an example of licensee failure to take adequate corrective action for a condition adverse to quali fy and is an apparent violation (317/87-23-01; 318/87-25-01).
A similar ext.mple j
is described in section 3 of this report.
b.
Improper Substitution of Solenoid Valves Following the discovery of the leaking solenoid valve (1-SV-1588)
associated with the SRW supply valve for DG12 (details in subpara-graph a above), the technician experienced difficulty in locating a replacement solenoid in the licensee's supply system. In that system spare parts are assigned and stocked by a " mechanism" or " mech."
number. A computer generated cross reference system (material cata-log) is provided to obtain the mech, number using a short description of the component to be replaced as an entering argument. That de-scription often includes the vendor model number as well as physical description.
A apparent weakness in the system is that the mech, numbers are not cross referenced to the plant's component numbers, e.g.
1-SV-1588.
Since the repair was occurring after normal working hours, personnel more familiar with the spare parts system were not available to help the technician.
_ _ _ _ _ _ _ _ _.
I-
.
4 The codel number of the failed component, ASCO HPX '8320A26, was no longer carried in stock by the company.
The technician utilized a handwritten, uncontrolled document labelled " safety related book".
This document was originally generated by Instrument and Controls shop personnel to help in cross referencing and identifying mech, numbers for components. It was apparently created because the formal cross referencing systems available at the time were not " user friendly", e.g. components not conveniently grouped by vendor.
This uncontrolled document listed the failed HPX 8320A26 model.
Next to that entry a mech. number and model number NP 8320A189E were listed.
The technician interpreted this to mean that the NP 8320A189E was an approved replacement for the failed component.
He then obtained and installed a NP 8320A189E model.
The following day a QC supervisor noted that the model numbers of the failed and the replacement sole-noids did not agree.
Additionally, QC found that the SRW supply valve solenoids for all three DGs did not have model numbers wh;ch agreed with the plant instrument index.
This indicated that model substitutions had been made without requisite engineering review per the facility design change procedure. A Nonconformance Report (NCR 7240) was initiated.
Inspector discussions with Instrument and Control' Design Engineering personnel and a QC supervisor concerning how these unauthorized sub-stitutions could have been made revealed that QC and Instrument and Controls personr.el have, on several occasions, found mismatched in model numbers between solenoids being replaced and the new parts.
This was happening on a frequent enough basis that a standing Facility Change Request (FCR) was created, FCR 84-1074, in 1984 just to process these problems as they arose. The problem was attributed to a January 14, 1981 memo from the engineering procurement coordin-ator which authorized purchasing to substitute 10 new ASCO model num-bers under previously existing mech. numbers for ASCO valves.
Al-though the author of the memo probably believed the new models were j
equ valent replacements for the previous models, no engineering d
analysis was performed to ensure that in all applications where old models were installed the new models were acceptable. The uncontrol-led document noted above would have directed technicians to pre-vlously used mech. numbers.
Technicians could have gone to pre-viously used mech. numbers expecting the older models and then re-ceive new models.
Apparently the model number mismatches were.nct always noted, possibly explaining the problems on the three DG SRW valves.
The controlled cross reference system should have prevented this type of problem. in that the short stock description should have been changed suf ficiently (at the tice the new models were authorized for purchase) that technic {ans would not be able to find a mech. number for discontinued models.
In retrospect the fact that technicians were continuing to draw models out of stock not matching installed
_ _ _ - _ _ _ _ _ _ - -.
_ _ _..
_ _ _ _
'
,.
.
.
components.should-have been an indicator that something was wrong.
j 1he principal problem was orobably that the uncontrolled document was l
being utilized by technicians instead of the controlled cross' refer-l ence system.
Perhaps because the full implications of this problem I
were not recognized by engineering in 1984, no effort was undertaken
!
to perform field verifications to ensure unauthorized substitutions i
had not been made.
Instead a decision was made.to depend on QC and craft personnel to identify model number mismatches. Later, a verif-ication was done for the subset of equipment included in.the EQ program.
l Based upon the mismatches tioted on the DG valves, it appears a field verification effort is needed for remaining safety related solenoid i
.
valves.
The NRC will monitor licensee response to the NCR 7240.
This item is unresolved pending determination. of the ade m cy.o f licensee corrective action and assessment of whether tnis meets the criteria for consideration as a
licensee identified violation (317/87-23-02).
c.
Use of Wrong Design Solenoid Valves on DG s
\\
The inspector, believing the Instrument ar.d Control Design Engineer-ing group had been asked by QC to respond to the DG solenoid substi-l tut 1on concerns, discussed these concerns with the engineers. They I
were, as yet, unaware of the problem.
They promptly looked into the issue and discovered that the solenoid valves (1-SV-1587,1-SV-1588, and 2-SV-2 587) used on all three DG's apparently since plant con-struction were of an imoroper design.
The installed valves were cf a "normally closed" design which, when de-energized, allows air flow in only one direction (from the SRW valve diaphragm toward the valve l
controller). In fact, the valves should have been of the " universal" type which allow air. flow to and from the SRW valve diaphragm to per-mit proper valve modulation. The licensee stated they would initiate a Facility Change Request (FCR) to correct the problem. FCR issuance was expected during the week of November 18, 1987.
Since the SRW valves had apparently functioned in an adequate manner (although not optimal manner) for a number of years, the immediacy of the problem was reduced.
5.
Plant Operations Safety Review Committee During the period PCSRC meetings were held frequently.
The inspector
attended portions of several meetings.
All technical specification re-quirements were me a
-
!
..
.
In responsc to 'the the 1986 SALP, the licensee had considered changes in the POSRC structure in order to improve objectivity and effectiveness. On November 4, 1987 a new structure was introduced without changing the
organization. The new POSRC now requires (1) an agenda presented to POSRC l
members.before the meeting, (2) a facilitatory who directs the meeting, (3)
that notes and consensus are clearly stated, and (4) that the facilitatory make a separate recommendation to the chairman who is not the facilitatory.
The initiatives have improved the organization.
F1,ever, the inspector noted, after attendance at meetings on September 10, October 30, and November.4, 1987 that POSRC members appeared willing to tolerate known Unidentified Failure Initiators (UFI,) within safety related components and
!
consicer these components operable. This was demonstrated on September 10 when the #12 Emergency Diesel Generator was declared operable after it had tripped on high cooling water temperature.
The cause was not found until three weeks later when..it became inoperable due to high temperatures again. Most POSRC members recognized that the UFI had not been corrected.
Some members insisted that the prcblem was an isolated ever.t..
Details of this event are discussed in section 4.
Similar decisions were made with regard to declaring #12 AFW pump operable knowing the UFI/ root cause had not been corrected (details in section 3).
6.
Radiological Controls Radiological controls were observed on a routine basis during the report-ing period.
Standard industry radiological work practices, conformance to radiological control procedures and 10 CFR Part 20 requirements were observed. Independent surveys of radiological boundaries and random sur-veys of non-radiological points throughout the facility were taken by the inspector.
No unacceptable.- conditions were identifiers.
7.
Observation of Physical Security Checks were made to determine whether security conditions met regulatory requirements, the physical security plan, and approved procedures. Those checks included security staffing, protected and vital area barriers, vehicle searches ard personnel identification, access control, badging, and compensatory measures when required.
--
On Octooer 6,1987, Calvert Cliffs Security Force conducted a joint exercise with the state police.
The purpose of the exercise was to evaluate the effectiveness of the Calvert Cliffs Security Operation coordination with local law enforcement during a contingency event.
Participants included the Maryland State Police SWAT Team, State Hostage Negotiators, State Fire Marshal and the K-9 Unit.
- _ -
_
- - _ _ - _ - _ _ _ _ - - -.
,
.
The. exercise also satisfied a semi-annual shift drill requirement which is mandated by the Security Training and Qualification Plan.
The exercise commenced at 4:30 p.m. on October 6, and lasted until 10:00 p.m.
The exercise involved a hostage situation and discovery of an explosive device in the Protected Area.
'The inspector witnessed the exercise and noted that the coordination was very well performed.
The Maryland State Policy Special Tactical Assult Team Element S.T. A.T.E. performed very professionally. Secur-ity access requirements were within NRC guidelines.
No unacceptable conditions were noted.
i 8.
Review of Licensee Event Reports (LERs)
LERs submitted to NRC:RI were reviewed to verify that the details were clearly reported, including accuracy of the description of cause and ade-quacy of corrective action.
The inspector determined whether further information was required from the licensee, whether generic implications were indicated, and whether the event warranted on site follow up.
The following LER's were reviewed:
LER No.
Event Date Report Date Subject Unit 1 87-07 04/02/87 09/29/87 Rev. 3, EQ Discrepancies Requiring Shutdown 87-13 09/11/87 09/11/87 Reactor Trip caused by Reactor Pump Surge Capacitor Failure 87-14 10/16/87 11/06/87 Containment Personnel Air Lock Door Gasket Surveillance Not Per-formed Unit 2 87-06 09/07/87 10/01/87 Turbine EHC Line Rupture forces Manual Trip of Reactor 87-07 10/12/87 11/11/87 Unplanned Actaation of Auxiliary Feed Water
_
_
_ _ _ _
__
.
/
..
~
p s.
0 Turbine EHC Line Rupture
--
At 3:13 p.m. on September 7,1987, Unit 2 was manually tripped from 100"; power in response to a rupture in the main turbine's Electro-Hydraulic Control (EHC) system fluid tubing.
Turbine trip and a subsequent automatic reactor trip were imminent.
Plant safety sys-tems performed as designed.
Subsequent investigation pinpointed the location of the fluid rupture to be in 1/2" stainless steel tubing leading to a pressure trans-mitter off of #21 EHC pump discharge line. Corrective action con-sisted of replacing the ruptured tubi-ng and of installing additional j
tubing clamps to better support this tubing.
Fracture mechanics analysis done on the affected section of tubing revealed the cause of failure to b from vibration induced metal fatigue.
--
Reactor Trip Caused By Failed Surge Capacitor At 5:15 a.m.
on September 11,1987, Unit 1 tripped from 100% power due to low reactor coolant flow following the tripping of the #12A Reactor Coolant Pump (RCP) power breaker.
The breaker trip was due to a failed surge capacitor.
Surge capacitor failures have previ-ously caused a number of plant trips, and the licensee plans to implerent a design change (at the next refueling outage on each unit)
that will eliminate the surge capacitors and ' install inductors to fulfill the protective function currently performed by the capac-itors.
Missed Surveillance Test
--
Following a routine entry into the Unit.1 Containment on October 9, 1987, a Containment air lock door seal functional test (required by Technical Specification 4.6.1.3 to be performed within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />), was not performed due to administrative oversight.
The licensee dis-covered the oversight on October 16 and then a functional test was performed with satisfactory results..
This is a licensee identified violation meeting the requirements of Section V, Appendix C,10 CFR 2.
Therefore, a notice of violation will not be issued (317/87-23-03).
i No unacceptable conditions were noted.
i 9.
Review of Periodic and Special Reports j
l Periodic and special reports submitted to the NRC pursuant to Technical Specification 6.9.1 and 6.9.2 were reviewed.
The review ascertained:
inclusion of information required by the NRC; test results and/or support-ing information; consistency with design predictions and performance spec-ifications; adequacy of planned corrective action for resolution of prob-lems; determination whether any information should be classified as an abnormal occurrence, and validity of reported information. The folluwing periodic reports were reviewed:
i
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
l
- - _ _ _ - _ _
.__
>
-
L
'
August ar,d September Operations Status Reports for Calvert Clif fs No.
--
I Unit and Calvert Cliffs No. 2 Unit, dated September 22 and October j
19, 1937, respectively.
No unacceptable conditions were identified.
10.
Unresolved Items Unresolved items require mere information to determine their acceptability and one such item is discussed in Detail 4.b.
11.
Exit Interview Meetings were periodically held with senior facility management to discuss
.
the inspection scope and findings. A summary of findings was presented to l
the licensee at the end of the inspection.
I 9