IR 05000250/1987028
| ML17342A808 | |
| Person / Time | |
|---|---|
| Site: | Turkey Point  |
| Issue date: | 07/17/1987 |
| From: | Shymlock M, Stadler S NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| To: | |
| Shared Package | |
| ML17342A806 | List: |
| References | |
| TASK-1.A.2.3, TASK-TM 50-250-87-28, 50-251-87-28, NUDOCS 8707290253 | |
| Download: ML17342A808 (33) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION
REGION II
101 MARIETTASTREET, N.W., SUITE 2900 ATLANTA,GEORGIA 30323 Report Nos.:
50-250/87-28 and 50-251/87-28 Licensee:
Florida Power and Light Company 9250 West Flagler Street Miami, FL 33102 Docket Nos.:
50-250 and 50-251 Facility Name:
Turkey Point 3 and
Inspection Conducted:.
June 15-19, 1987 Inspectors:
S,
. Stad er License Nos.:
Accompanying Personnel:.
Approved by:
M. B.
Shym Division of T. J. O'onnor M. J.
DeGraff J. Arildsen
~>r I k, Section Chi Reactor Safety SUMMARY ate Signed Scope:
This special, announced reactive inspection was conducted in the areas of a series of loss of boric acid flowpath events that occurred between May 28 and June 3, 1987, and the status of licensed operator training and operator requalification and instructor qualifications.
On May 28, 1987, the failure of a mechanical seal on the 4b boric acid pump resulted in continuous nitrogen intrusion into the boric acid and chemical volume control systems.
This nitrogen intrusion resulted in gas binding of the boric acid and charging pump and loss of normal and emergency boration flowpaths to Units 4 and 3 on several occasions between May 28 and June 3,
1987.
This inspection consisted of a
review of the circumstances surrounding these events, reviews of associated procedures, logs, charts, and plant change modifications and interview with operators, technical staff, and'aintenance personnel.
The review of licensed operator training, upgrade operator requalification training, and instructor qualifications was also part of an ongoing NRC overview of the licensee corrective actions to previously identified deficien-cies in these areas.
This part of the inspection included a review of training records and procedures, observation of training classes, and interviews with operators and training staff personnel.
Results:
Two violations were identified.
(Failure to maintain design and document controls; see paragraph 5.a and failure to establish and implement procedures; see paragraph 5.b).
8707290253 87072i PDR ADQCK 05000250
REPORT DETAILS Licensee Employees Contacted
- B. A. Abrishami, Acting Technical Department Superintendent
- J. Arias, Supervisor, Regulatory Compliance
- C. Baker, Plant Manager
- W. Bladow, Quality Assurance (QA) Superintendent
- D. Chancy, Engineering Manager B. Crittendon, Site Engineering Organization
- J. Donis, Site Engineering Supervisor R. Farach, QA Performance Monitoring
- S.
D. Ferrell, Licensing Engineer
- J. A. Finn, Training Superintendent
~J. Franklin, Engineering Manager - JPE
- D. Grandage, Operations Superintendent
- D.
W. Haase, Chairman, Safety Engineering Group P. Higgins, Technical Licensing - Lead Technical Engineer
- G. Holling, Licensed Operator Training Coordinator
- J. Isakson, Technician F. Krenke, QA Performance Monitoring Section
- D. Osborn, Site Engineer
- L. F. Pabst, Manager - JPE.
- W. Pike, Safety Engineering Group
- G. Salamon, Compliance Engineer
- F. H. Souttiworth, Maintenance Superintendent
- J. C. Strong, Jr., Electrical Maintenance
- D. Tseng, Safety Engineer
- J. B. Webb, Nuclear Operations
- C. Wethy, Site Vice President Other licensee employees contacted included engineers, technicians, operators, mechanics, and training staff personnel.
NRC Resident Inspectors
- R. Brewer, Senior Resident Inspector
- J. MacDonald, Resident Inspector
- K. Van Dyne, Resident Inspector
- Attended exit interview 2. 'xit Interview The inspection scope and findings were summarized on June 19, 1987, with those persons indicated in paragraph I above.
The inspector described the areas inspected and discussed in detail the inspection findings.
No dissenting comments were received from the licensee.
The licensee did not identify as proprietary any of the materials provided to or reviewed by the inspector during this inspectio Licensee Action on Previous Enforcement Matters This subject was not addressed in the inspection.
4.
Unresolved Items Unresolved items were not identified during this inspection.
5.
Loss of Boric Acid Flow Path Event This inspection reviewed the circumstances surrounding the recurrent losses of boric acid injection capability between May 28 and June 3, 1987.
A mechanical seal failed on 4b boric acid transfer pump allowing large quantities of nitrogen into the boric acid system which prevented adequate pump operation.
The inspection centered on design control of the system and how operations personnel responded to the events.
a.
Design Plant Change/Modification (PC/M)
80-83 Boric Acid Transfer Pump Replacement was implemented to correct recurring problems associated with the then current pump installation, a
GE-20K Chempump manufac-tured by the Crane Company.
A replacement pump was chosen to rectify problems associated with net positive suction head, spare parts availability, tolerance to the elevated environmental temperatures, and tolerance to the boric acid solution.
The major aspect of the PC/M deals with the selection of a Gould Pump model 3196
"ST" as the replacement pump.
Other aspects of the PC/M pertained to the installation of associated hardware.
Change Request (CR)
no.
implemented the design for the transfer pump's seal cooling system, which consisted of a natural convection cooling system with cover gas pressurization to preclude boric acid leakage.
This information is contained in Appendices J
and K of PC/M 80-83.
The safety evaluation in Appendix J states that the safety evaluation for PC/M 80-83 Section IY applies.
The safety evaluation contained in PC/M 80-83,Section IV states that the change does not involve an unreviewed safety question.
The Implementation Guidelines contained in Appendix J list the following specific instructions for pressurization of the Seal Water Tank:
To pressurize the tank, all fittings should be secured, the plug removed from the vent line, a pressure source connected, and the vent valve gradually opened to allow the system to pressurize to 40 psig.
Upon reaching the required pressure, the vent valve is to be closed, the pressure source removed, and the vent plug reinstalle Based on the above information it'ppears that the seal cooling system.was intended to be operated as a closed or isolated system, with any adjustments for pressure or water level to be made in a
batch type manner.
This method of operation, when properly moni-tored, would provide the operators with a very effective method of monitoring the seal for impendi'ng failure.
This method of operation would be consistent with the aspects of the safety analysis which depicts this modification's design, construction, and testing as not increasing the probability of an accident analyzed in the FSAR.
CR 10 revised Appendix J and added appendices L and M to incorporate a permanently connected Nitrogen (N
) supply to the seal water tanks and its associated piping.
The recon for these changes, as docu-mented in CR 10, was to eliminate the need for periodic pressuriza-tions to account for gradual gas leakage through fittings, gauges, etc.
The safety evaluation for Appendix L references the safety evaluation of PC/M 80-83 Section IV as also being applicable.
This evaluation does not evaluate the operation of the seal water tank with a constant supply of N
.
The revision to Appendix J incorporated by CR 10 does not reflect the connection to the plant's N
system.
The revised pressurization instructions for the seal wafer tank are as follows:
To pressurize tank, ensure that drain valve, plug and all fittings are properly secured.
Remove plug from vent line, connect pressure source (instrument
'air and/or N ),
and gradually open valve in supply line to allow system to pressurize.
Regulator shall be set at 40 psig.
The Design Analysis section of Appendix L
delineates the pressurization method as follows:
Pressurization of any of the tanks can be made by opening the isolation valve on the inlet to the tank connecting the N
supply for fill.
The new pressure regulator on the N>
s)pply line is to be set at 40 psig.
Based on CR 10, the information presented depicts a system intended to be operated as a sealed system, with any adjustments for pressure to be made in a batch type manner.
This method of operation would be consistent with the aspects of the safety analysis.
The drawings contained in PC/M 80-83 depict the piping and valve placement for the seal water tank, the mechanical seals, and the N>
system.
PC/M drawings do not indicate the normal operating valve positions.
Plant Maintenance and Operating Drawings (PMOD) are developed from the information contained in PC/Ms, which are then
utilized to revise Operating Diagrams.
PMOD EPP-80-83-015, Rev.
was developed from the information contained in PC/M 80-83.
PMOD EPP-80-83-015, Rev.
2 depicts the vent, drain and N
isolation valves as open.
Operating Diagram 5610-T-E-4505, Rev. 22, sheet 2 of 3, depicts a configuration in which vent and drain valves are closed and the N
isolation valves are open.
The plant engineering staff was unable to provide documentation to support reasons for the revised valve positions indicated in Operating Diagram 5610-T-E-4505.
Yalve positions for the N
system were established by procedure O-OP-065.3, Nitrogen Gas Supply System.
Adequate document reviews would have discovered the discrepancies between the evolution of operation drawings and the operational descriptions contained in various appendices contained in PC/M 80-83.
CFR 50, Appendix B,Section III Design Control states that measures shall be established to assure that the design bases, are correctly translated into specifications, drawings, procedures, and instructions.
Design control measures shall provide for verifying or checking the adequacy of design, such as by the performance of design reviews.
The verifying or checking process shall be performed by individuals or groups other than those who performed the original design.
CFR 50.59 requires the licensee to review changes to the plant for unreviewed safety questions.
An unreviewed safety question is considered to exist:
i) if the probability of occurrence or the consequences of an accident or malfunction of equipment important to safety previously evaluated in the safety analysis report may be increased; or ii) if a possibility,for an accident or malfunction of a different type than any evaluated previously in the safety analysis report may be created; or iii) if the margin of safety as defined in the basis for any technical specification is reduced.
CFR 50, Appendix B,Section VI, Document Control states that measures shall be established to control the issuance of documents, including changes thereto, which prescribe all activities affecting quality.
These measures shall assure that documents, including changes, are reviewed for adequacy and approved for release by authorized personnel.
Contrary to the above, the licensee failed to maintain, adequate design control over a modification to the boric acid system including the associated safety evaluations, reviews, and the correct transla-tion of the changes and assumptions into drawings and procedures.
Additionally, the licensee did not perform an analysis to determine the effects of a continuous N
supply via a failed mechanical seal on
the emergency boration system or for the potential for common mode failure of both trains of the boric acid transfer system and for the systems of both units 3 and 4.
The licensee also failed to analyze the condition for a failed N
pressure regulator which would allow full header pressure of 110 p(ig to pressurize the seal water tank.
The-safety evaluation contained in PC/M 80-83 Section IV analyzed the change only from the aspect of a
pump failure, and its ensuing inability to pump boric acid.
FPEL letter JPE-PTPO-84-1602 entitled Boric Acid Transfer Pump Replacement summarizes the basis for the classification of the boric acid pump seal cooling system as Non-Nuclear Safety-Related QA/QC as follows:
Double mech'anical seals do not typically fail catastrophically; i.e.,
a small leak begins, evidenced by cooling medium loss or by product 1'eakage externally from the outboard seal.
Such leakage has ample time to be evidenced by operations.
Failure of the seal will not detrimentally affect operation of the pump:
if cooling medium leaks (demineralized water),
negligible dilution of product occurs; if product leaks out, flow rate would not be significant enough to affect pump performance; as the seals are nor 'relied on for their load carrying capability, their failure would not impact shaft support/alignment.
The above mentioned scenarios do not address the potential for the continuous in-leakage of N
into the emergency boration system.
An adequate review of the Biange Request would have noted an inten-tion to operate the system with the N
system continually valved in and that such operation would be contrary to original design intent.
The licensee also failed to adequately control the implementation of PC/M 80-83 into procedure O-OP-065.3, Nitrogen Gas Supply System.
This procedure dictates the valve alignment for the N
system to the boric acid transfer pump seal water tank.
An adequate review of the procedure would have referred to the original PC/M for the designed parameters under which the N
supply would be used.
This review would have noted the inconsf'stencies in the the pressurization operations; thereby, prompting the procedure writer/reviewer to question the author of the design change as to the correct valve position.
Failure to maintain design control and document control is an apparent violation (250, 251/87-28-01).
An additional area of concern pertains to the licensee's response to IE Information Notice No. 82-19:
Loss of. High Head Safety Injection Emergency Boration and Reactor Coolant Makeup Capability.
This Notice pertains to Duke Power Company's McGuire Unit I losing its
i
emergency boration and makeup flow paths.
The Notice documents not only the loss of flow paths but also the need to be cognizant of all possible sources of gas intrusion into the Reactor Coolant System (RCS)
and Chemical and Volume Control System.
Although the system configuration represented in PC/M 80-83 differs from the subject fai lure at McGuire, the Notice specifically states that there is a
concern that the potential of a single component malfunction, person-nel error, or maintenance error in the safety or non-safety-related portion, of the system could lead to the same consequences.
Conversa-tions with the licensing staff indicated that the IE Notice was reviewed and deemed not applicable because the Turkey Point facility did not have suction dampers nor a similar H2 flow path.
Conversa-tions with operations personnel conveyed a
concern that the N
intrusion via the seal water cooling tank may have been the sour'f high N
concentrations in the Volume Control Tank, low Hydrogen concento"ations in the RCS, and the source of gas which caused the charging pumps to be vented on a more frequent basis.
The design engineer for PC/M 80-83 was not aware of IE Notice 82-19.
Operations Response to the Event The boric acid transfer system consists of two Unit 3 pumps; 3a and 3b; two Unit 4 pumps; 4a and 4b; and three Boric Acid Storage Tanks (BASTs)
A, B
and C.
There are numerous possibi'1ities for cross'-
connecting the pumps and tanks to each unit, however, 3a and 3b pumps and
"A" BAST are normally lined up by procedure to Unit 3, while 4a and 4b pumps and
"B" and
"C" tanks are lined up to Unit 4.
On May 28, 1987, at approximately 9:00 p.m., operators were unable to borate Unit 4 through any available flowpath including:
boric acid storage tank (BAST) "C" through the blending station;
"C" BAST through the emergency boration valve MOV 350; or from the refueling water storage tank (RWST) through the "B" charging pump.
Off-normal Operating Procedure 4-ONOP-046. 1, Emergency Boration, did not address the actions to be taken by operators if all boration paths to a
unit are lost.
It was also noted that the 4b boric acid (BA) pump overheated, requiring the use of a fire extinguisher and that the thermal overloads tripped at least twice requiring reset.
Unit 4 was in cold shutdown at the time of this event.
Technical Specification 3.6.a requires. at least one boric acid flowpath to be available under these conditions.
Tfie Plant Supervisor
- Nuclear (PS-N) discussed the reportability of this loss of all unit boric acid flowpaths with a
member of Plant Licensing, and it was decided that the event was not reportable under
CFR 50.72.
The licensee indicated to the NRC Senior Resident Inspector, who was in the control room, that Operating Procedures OP-046, Chemical and Volume Control System (CVCS) Boron Concentration Control provided for an alternate flowpath
from the 3b BA pump to the Unit 4 Reactor Coolant System (RCS).
Since it was considered that a flowpath was available for boration of Unit 4, the event was not considered by the licensee to be reportable under 50.72.
This reportability question will be turned over to the resident inspectors for follow-up.
The PS-N, who believed the cause of the loss of BA flow to be attrib-utable to inadequate heat tracing and BA plugging, established the flowpath from the 3b BA pump discharge to Unit 4 RCS.
This action established flow to Unit 4 from the 3b BA pump, normally assigned to Unit 3 and the
"B" BAST.
Review of OP-046 by the inspectors, however, determined that this flowpath from the discharge of the 3b pump to Unit 4 RCS was not included in this procedure.
The only evolution addressed in this procedure related to Unit 4 and Unit 4 BA pumps, was the utilization of the 4a pump in lieu of the 3b pump to provide recirculation flow from the Unit 3 to the batch tank, not the RCS.
Since the PS-N did not utilize the alternatives provided by OP-046, a Plant Work Order (PWO) -or Temporary Procedure to control this evolution, the flowpath established was unauthorized.
Because there were ongoing efforts in place at this time to reestablish BA flow to Unit 4 by venting of the charging pump and the unit was in cold shutdown, the urgency did not appear to exist to require unauthorized valving of BA flowpaths.
Ouring this same timeframe, a system engineer had been called in to troubleshoot the BA system.
The system engineer and a non-licensed operator were given instructions to troubleshoot the system through checking temperatures and heat tracing and by overseeing evolutions in progress.
The system engineer and non-licensed operator subse-quently valved in the Unit 4 BA system into the suction of the 3b BA pump through MOV 390.
The system engineers and non-licensed operator were not authorized by plant procedures to perform valving, and their actions exceeded the bounds of their direction.
In addition," no procedure was utilized to perform and document this evolution, and the PS.-N was not aware of these actions.
Valving in the Unit 4 BA system to the suction of the 3b BA pump, the only operable Unit 3 BA pump, allowed nitrogen intrusion to occur into the Unit 3 BA system from the failed 4b BA pump mechanical seal.
After venting the "8" charging pump three times and removing large quantities of what was thought to be air, the operators were able to reestablish BA flow to Unit 4 from the RWST.
On Nay 29, 1987, at approximately 1:45 a.m.,
the operators were unable to establish boric acid flow to Unit 3 with the 3b BA pump.
Unit 3 was in refueling at this time.
The log entries indicated that the Reactor Control Operator (RCO)
"discovered" that the system engineer had realigned the 3b BA pump to the
"C" BAST instead of the normal alignment to the
"B" BAST.
Since Operations personnel believed the
"C" BAST was the source of flow blockage, the operator aligned the 3b BA pump to the
"B"
BAST.
Flow could still not be established with this flowpath, and the clearance was lifted on the 3a BA pump as an additional attempt to establish boric acid flow to Unit 3.
Boric acid flow could not be established to Unit 3 from any BAST through any BA pump.
Since the Unit 3 RWST was drained because the unit was in refueling, the charging pump flowpath could not be utilized to establish boric acid flow.
With no boric acid flowpath available to Unit 3, the operator suspended fuel handling and isolated all potential dilution paths to meet Technical Specification requirements.
The PS-N recommended to the Duty Call Supervisor (DCS)
that the Event Response Team (ERT)
be activated to investigate the source of the boric acid problems.
The responsibility for activating the ERT is delegated to the PS-N by Administrative Procedure, O-ADM-011, Short Notice Outage Work (SNOW)
Response Organiza-tion.
The DCS asked the PS-N to delay the ERT activation until the start of normal work hours, and the PS-N concurred.
This decision was apparently based on the early morning hour and the potential impact on the individuals assigned to the ERT.
At 4:30 a.m.,
the operators valved out the 4b BA pump for Mainte-nance to determine the.source of the blockage.
Maintenance was directed to support operations in unplugging the inlet side of the pump.
The pump was drained and it was subsequently discovered that no line blockage existed, as noted on the PWO.
The pump was reassembled and returned to Operations.
At this point, operators still believed that the source of blockage was the
"C" BAST, even though flow had been unsuccessfully attempted from the "B" BAST through both Unit 3 BA pumps.
At approximately 8:00 a.m.,
on May 29, representatives-of the licensee's ERT consulted with Operations personnel and decided against activating the ERT.
The decision was apparently based on the restoration of a flowpath to Unit 4 from the RWST and charging pump and to Unit 3 through 3b BA pumps.
It also appears that the ERT representatives and Operations were in agreement that the cause of the loss of boric acid flow was attributable to BA plugging, probably in the
"C" BAST.
The PS-N who had earlier requested activation of the ERT did not, however, concur in this decision and had assumed the ERT would be activated at the start of normal working 'ours on May 29, 1987.
One critical piece of information that was not utilized in this decision not to activate the ERT, was the loss of water level in the seal water tank.,
For nitrogen intrusions into the BA system to occur, the water level in the seal water tank had to be gone.
The failure of the seal and the water level had to have occurred before I:45 a.m.,
on May 29, and may have occurred hours or 'even days earlier.
Despite this fact, the non-licensed operators'og sheets indicated greater than 50%
seal water level at 9:00 p.m.
on May 28, and 1:00 a.m.
and 5:00 a.m.
on May 29, as well as the previous shifts on May 28 and before.
As
a result, this piece of information which helped the ERT to make the correct diagnosis five days later, was not available to them on the morning of May 29, 1987.
At 8: 18 a.m.,
on May 29, the licensee also notified the NRC of the loss of all boric acid flowpaths to Unit 3 under the requirements of
CFR 50.72.
Because the loss of BA flow occurred at 1:45 a.m., this report should have been made within four hours.
As with the reporta-bility issue on May 28, this late report under 50.72 will be turned over to the resident inspectors for follow-up.
The 4b BA pump was valved back in on May 31, 1987, at 2:30 p.m.
thereby again allowing N
into the, BA system.
Valving in the 4b pump once again introdu@d nitrogen into the boric acid system associated with Unit 4.
The Maintenance efforts conducted on May 29 and 30, had apparently not identified the failed mechan-ical seal and missing seal water level.
Once again, the non-
')icensed operator logs indicated a
normal seal level at 5:00 a.m.,
on May 31.
On June 1,
1987, at 3:30 a.m.,
the operators were unable to obtain boric acid flow to Unit 4 through either the 4a or 4b pumps and the normal or emergency flowpaths.
Flow from the RWST through the charging pump was verified.
On this shift, the non-licensed operator reported'hat the water level in the mechanical seal water tank of the 4b BA pump was low and wrote a plant work order
{PWO) to have it refilled.
This refilling operation occurred on June 2, 1987.
While attempting to refill -the seal tank, Maintenance noted that as soon as the tank was pressurized with nitrogen, the seal tank water level would drop.
Further inspection by.the mechanic revealed a
'leaking seal on the pump..
Operations decided to leave the tank unpressurized.
Low suction temperatures were noted on the suction of the 4a BA pumps, 120'F, the 4b BA pump, 140'F, and heat tracing circuit 136'F.
These low temperatures were attributed by Operations to the use of a portable pyrometer and the exposed BA piping sections.
Since Operations was still convinced that the lack of flow was due to plugging, the addi-tional symptoms of the loss of level in the mechanical seal water tank and the low boric acid temperatures were not analyzed or the ERT activated.
On June 2 at 1:30 a.m.,
the operators were again unable to borate Unit 4 through the use of BA pumps 4a or 4b and the
"C" BAST.
Despite the additional symptoms of low seal tank water level and
"cold spots" in the BA piping, Operations continued to attribute this lack of flow to blockage and the ERT was not activated or Onsite Engineering assistance sought.
At 10:00 p.m.,
on June 2, operators and maintenance personnel noted that the 4b BA pump mechanical seal kept draining upon pressurization with nitrogen, but the nitrogen intrusion problem was still not diagnosed or the ERT activate On June
at 1:58 a.m.,
the operators were again unable to effect boric acid flow to Unit 4 with the 4b BA pump through the normal or emergency boration paths.
At 2: 15 a.m.,
an attempt was made to establish boric acid flow with the 4a BA pump from the "B'-'AST, but this attempt was also unsuccessful.
Additional attempts to provide Unit 4 boric acid flow using the 4a BA pump from the
"B" BAST were also ineffective.
At 5:05 a.m.,
the operator attempted to establish boric acid flow to Unit 3 via the normal path,
.the blending station, or the emergency boration flowpath, MOY-350.
At this point, the normal and emergency boration flowpaths to both Unit 3 and Unit 4 had been lost and the only means of boration was the RWST and the charging pumps.
The licensee notified the NRC of the loss of the emergency flowpaths to Unit 3 and 4 at 5:51 a.m.
At 8:00 a.m.,
on June 3, 1987, the ERT finally assembled to address the recurring boric acid flowpath failures that occurred between May 28 and June 3, 1987.
The ERT reviewed the following symptoms:
Loss of boric acid flowpath to Unit 3 Low temperatures on 4b, 4a, and 3b boric acid suction lines Low or zero boric acid pump discharge pressure Inability to maintain seal level in the 4b BA pump mechanical seal The team quickly concluded that a loss of seal water due to a
damaged inboard seal could allow nitrogen intrution into the 4b BA pump casing and the BA system.
With a "constant" nitrogen supply, the suction lines and boric acid pump could become filled with nitrogen.
Based upon this hypothesis, the ERT directed Operations to take the following actions:
(a)
Isolate the Unit 3 BA pumps from Unit 4 by closing valve 390 and vent the 4b BA pump to establish a flowpath.
(b)
Completely isolate 'the 4b BA pump and vent the 4a flowpath to. Unit 4 RCS.
The ERT also recommended that operators check the seal water sight glass on all BA pumps daily and report a
loss of seal level immediately.
Isolating the 4b BA pump stopped the constant supply of nitrogen from the 3000 gallon liquid nitrogen storage tank, and the licensee was able to restore the boric acid flowpath to both Unit 3 and Unit 4.
Additional corrective action included isolating the nitrogen supply valves to the boric acid pump accumulators as detailed in paragraph 5.C.'
Although the'RT diagnosed the seal failures and nitrogen intrusion promptly when finally activated, the circumstances surrounding the multiple loss of the boric acid system that occurred during the 7-day period are cause for concern.
These concerns are mitigated only by the fact that the units were both
'hutdown at the time of the events.
The major area of concern is the apparent willingness of Operations personnel to depart from approved procedures, and the loss of configuration control over a safety-related system.
Both a failure to implement procedures and a
lack of safety-system configuration control have been repetitive occurrences in the past and were believed -to be improving under various performance improvement programs and increased management emphasis on procedural compliance.
Due to a failure to utilize procedures, unauthorized valve line-ups, undocumented trouble-shooting, and a
lack of independent verification, the licensee appears to have lost configuration control over the Boric Acid and Emergency Boration system on May 28 and for the six days that followed.
Other major areas of co'ncern include the lack of prompt management involvement including a
reluctance to activate the ERT or to provide engineering support, incorrect seal level readings by operators, and apparent deficiencies in newly
'revised procedures.
The licensee elected not to activate the ERT following the failures of the Unit 4 Boric Acid system on May 28 and Unit 3 failure on May 29 despite a request from the PS-N.
The licensee indicated that the decision not to activate the ERT was based on several factors including that flow had been reestablished to both units, two other ERTs involving previous events were ongoing reducing the number of available team personnel, and that Operations personnel agreed it was not necessary.
Not all of the boric acid flowpaths had been established at 8:00 a.m.,
on the morning of May 29.
The only flowpath that had been established to Unit 4 was the RNST through the charging pumps, not the normal or emergency flowpaths from the BA pumps and BASTs.
Flow to Unit 3 had been reestablished from the 3b BA pump and
"B" BAST, but only because the 4b BA pump had been isolated for maintenance, thus stopping the nitrogen intrusion to Unit 3.
Operations personnel apparently focused in on BA plugging as the root cause.
They were unaware of the potential for mechanical seal failure and they were convinced the problems were caused by boric acid plugging from a lack of adequate heat tracing.
This had been a recurring problem in the past, particularly with the original type of heat tracing before it-was replaced.
The operators also appear to have been convinced that the source of plugging was the
"C" BAST, even though they were unable to obtain flow from the
"B" BAST and 3b BA pumps on the morning of May 2 Activation of the ERT and/or Onsite Engineering Support at this time could have provided an independent analysis and possible prompt resolution of the problem.
Most of the symptoms that the ERT would utilize to resolve the problem in minutes on June
were also available on May 29.
There had been a loss of boric acid flowpaths to both Unit 3 and 4, the charging pump supplying Unit 4 had to be vented three times to start, flow could not be obtained from the
"C" or
"B" BASTs, and there were low temperatures on the boric acid suction lines.
One critical piece of information that was available on June
and should have been available on May 29 also, was the observed loss of seal water level in the 4b BA pump mechanical seal.
Since engineers on the ERT were, aware of the seal failure mode of nitrogen intrusion, this information could have possibly helped resolve the boric acid problems five days earlier.
Since the non-licensed operators logs continued to record these seal levels as normal on May 28 and 29, this information was not, available.
The circumstances surrounding these incorrect readings, as well as similar incorrect readings on May 31, are a
area of concern.
It also appears that the ERT and licensee management were aware on the morning of May 29 of the unauthorized boric acid valve line-ups that had occurred on the two previous shifts, but did not take corrective actions to ensure restoration to normal alignment including independent verifications or to ensure that unauthorized and undocumented valving did not continue as part of the trouble-shooting process.
There is also a concern that the established procedures may not have been adequate to support valve alignments, trouble-shooting, and emergency boration abnormal operations.
At least two of these procedures, 0-0P-046, Boron Concentration Control, and ONOP-046. 1, Emergency Boration, were recently revised as part of the Procedures Upgrade Program (PUP).
OP-046 did not address the flowpath from the Unit 3 BA pump to the Unit 4 RCS,, and ONOP-046. 1 did not provide direction for a
loss of all boration paths.
In addition, the operators'ogs indicated that refueling was discontinued, and the sources of dilution to the RCS were tagged when the boric acid flowpath to Unit 3 was lost.
The inspector reviewed Operating Procedure 16002.6, Preparations and Precautions for Refueling Shuffle, dated May 21, 1987.
Prerequisite check list no.
1 to this procedure requires that the primary water dilution supply valve, 396, be closed and tagged during refueling operations.
The licensee's interim upgrade Technical Specifications 3/4.4.9.1.3 requires that the primary water supply to the boric acid blender shall be verified at least once per
days closed and secured in position by mechanical stops or by removal of air or electrical power.
The Technical Specification bases indicate that this locking closed
of the va1 ves precl udes the possibi 1 ity of uncontrol led boron dilution of the filled portion of the RCS.
Procedure 16002.6, in requiring only tagging of the dilution source, is less restrictive than this new Technical Specification requirement for locked valves.
Although the upgraded Technical Specifica-tions are presently under trial use and not binding at this time, the licensee indicated that they are attempting to comply on a daily basis.
This procedure should be revised to require locked closed valves and 31-day surveillance prior to implemen-tation of the revised Technical Specifications.
Revisions of this procedure, OP 16002.6, to require locking of primary water dilution valves during refueling will be tracked by the resident inspectors.
10 CFR, Part 50, Appendix B, Criterion V, requires that activi-ties affecting quality shall be prescribed by documented instructions, procedures, or drawings of a
type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings.
Technical Specifications 6.8.1 requires that written procedures shall be established and implemented for activities recomnended in Appendix A of Regulatory Guide 1.33.
Appendix A recommends, in part, that procedures for the operation of safety-related systems should be established.
NUREG-0737, Item I.C.6, Independent Verification, requires the implementation of procedures to verify the correct performance of operating activities.
,This item was implemented by an Order dated July 10, 1981.
Contrary to the above, the licensee did not establish or implement adequate procedures to assure configuration control over emergency boration, a safety-related system, between Nay 28 and June 3, 1987.
Examples include the following:
The boration flowpath established on July 28 from the discharge of the 3b BA pump to the Unit 4 RCS was not authorized by established procedures, and the administra-tively allowable alternatives of a
Plant Work Order or an approved temporary procedure were not utilized.
The boration flowpath established from Unit 4 Boric Acid system to the suction of the 3b BA pump resulted in nitrogen intrusion from the Unit 4 BA system to the Unit 3 BA system and a
loss of all boric acid flowpaths.
This evolution was performed by non-licensed personnel without SRO direction.
This evolution was also performed without an approved procedur Independent verification was not implemented in accordance with administrative procedure O-ADM-31, Independent Verification, and NUREG 0737, Item I.C.6 to ensure, valving alignment documentation and restoration from the unauthorized valve line-up.
Low boric acid pump mechanical seal water levels were not reported as low on May 28 and
in'ccordance with procedure O-OSP-201.2.
Off-Normal Operating Procedure ONOP-046.1, Emergency Boration, did not provide directions to operators for a
loss of all boration flowpaths, including flow from the RMST.
Additional troubleshooting valving efforts between May 30 and June 3, 1987, apparently allowed additional nitrogen intrusion from the failed seal in the 4b BA pump into Unit 4 and an additional loss of the 3b BA pump.
These evolutions were also not conducted utilizing approved procedures, were not documented, and independent verifica-tion was not performed or documented to assure proper realignment, The 4b BA pump overheated on May 28 requiring the use of a fire extinguisher and resulted in tripping the thermal overloads.
The licensee failed to write an NCR per Administrative Procedure 0190.12, or to perform inspection and testing per procedure 0190. 19 to ensure the pump and motor were not damaged.
These examples of a failure to establish or implement ade-quate procedures to control the configuration and operation of a
safety-related system are an apparent violation (250,251/87-28-02).
Corrective Actions As previously mentioned, losing the boric acid transfer pumps due to nitrogen intrusion resulted in a loss of the boric acid addition path to the reactor, coolant system.
To preclude the probability of nitrogen intrusion from occurring again, the licensee has initiated changes to various procedures governing the operation and control over certain systems.
The procedures changed are noted below:
O-OP-065.3, Nitrogen Gas Supply System (delineates the valve configuration for the nitrogen supply system line-up).
Nitrogen inlet isolation valves (847 D, E, F, G, gas supply to the boric acid pump seal tanks)
which were previously maintained in the open position have had their operational line-up changed to locked closed.
>lith these valves open, nitrogen could flow
freely into the system if a failure of the mechanical seals on the boric acid transfer pump were to occur.
By locking these valves in the closed position, the continuous supply of nitrogen is isolated from the boric acid system.
Additionally, the outlet isolation valves (847 L, M,
N, P)
downstream of the pressure control valves, have also had their operational line-up changed from open to closed.
The inspector reviewed the change to this procedure as well as performing a
system walkdown to verify actual position of the nitrogen isolation valves.
O-ADM-205, Administrative Control of Valves, Locks, and Switches.
The above mentioned nitrogen inlet isolation valves (847 D,
E, F,
G) to the pump seal tanks were added to this Administrative Procedure.
This procedure indicates these valves are now maintained in a closed and locked condition.
This procedure also establishes periodic verification of valve status.
ONOP 2608.2, CVCS Malfunction of Boron Concentration Control System.
The changes to the procedure were made to the following sections.
Under the symptoms section 4.0, step 4.10 was added; boric acid seal water tank pressure is less than 35 psig.
Under section 5.0,'ubsequent operator actions were added such that if boric acid pump seal tank pressure is less than 35 psig with level less than 50Ã in seal tank, notify operations and, if directed, remove the pump from service, isolate and submit a
plant work order.
There is a note which states that low seal water tank level is indicative of seal leakage.
If water level in the seal tank is greater than 505, then open the isolation valves and fill the tank with nitrogen until the pressure is between
and 40 psig.
A caution statement was also added to.indicate that low seal tank pressure concurrent with low seal tank level could result in nitrogen binding of the boric acid pumps.
O-OSP-201.2, S.N.P.O Daily Logs.
The change implemented reiterated that the normal range of nitrogen pressure in the boric acid pump seal tank is between 35 and 40 psi.
Additional direction was added to inform the operator if pump seal tank pressure is less than 35 psi to perform the actions required by ONOP 2608.2, CVCS Malfunction of Boron Concentration Control System.
The above mentioned pro'cedure changes implemented 'y the licensee appear adequate to prevent recur rence of a similar event.
The-licensee has taken substantial control over isolation of the 3000 gallon nitrogen supply from the Boric Acid system by closing one set of valves while locking and closing a
second set.
This limits the nitrogen'ntrusion into the Boric Acid system to the amount maintained in the boric acid pump
e
seal water tank.
The licensee has also given the Operations Department more guidance in the operation of the system from taking daily logs to being able to better identify a faulted condition.
Additionally, Operations has clearer guidance on actions to be completed once a faulted condition is detected, through Training Brief No. 201, "Loss of Boric Acid Flow Path to Both Units," dated June 11, 1987.
6.
Licensed Operator Requalification Training and Instructor gualifications Previous major deficiencies identified in the area of training have included an inadequate training staff and an unsatisfactory Licensed Operator Requalification program.
Specific continued concerns in this area have included:
(1)
Unsatisfactory results in Licensed Operator Requalification and Upgrade Requalification Examinations; (2)
Inadequate number of qualified Licensed Operator Requalification instructors; (3 )
Utilization of instructors who have failed NRC-administered examinations; (4)
Use of uncertified contract instructors to teach systems and integrated plant response, and lack of site-specific training for contract instructors; (5)
Inadequate current instructor development program and related documentation; (6)
In-plant overtime for operators attending the requalification program
. and excessive overtime for instructor staff; and (7)
Lack of an effective tickler system to provide for tracking of emergent training briefs to ensure timely incorporation of appropriate briefing material into permanent lesson plans.
The inspector observed 1987/1988 Licensed Operator Requalification classroom training (Rod Control System, SD005-Rev.
2-1; and Emergency Diesel Generator and Loading, SD 8137).
The training observed was found adequate in scope, relevancy, and presentation.
Additionally, the inspector conducted interviews with three Licensed Operator Requalifi-cation students, two Hot License students, and two Licensed Operator instructors.
All personnel interviewed perceived an improving trend in the quality of the Licensed Operator training, and all the requalification students commented positively on the incorporation of various licensed operator identified areas of particular need into the requalification training progra Cl
The Licensed Operator Requalification and Upgrade Requalification training program appears to show improvement, but final resolution of this concern is pending review of the next Licensed Operator and Requalification examination results.
In addition to addressing the poor performance results in the Upgrade Accelerated Requalification Training program, Inspection Report 87-09,
'nresolved Item 250,251/87-09-02 identified a shortage of qualified SRO licensed or certified instructors to teach systems and integrated plant response training.
The inspector noted
new contract 'instructors holding SRO licenses being hired to alleviate the instructor shortage.
Twelve of those contract instructors had already reported and were enrolled in a
5-week training program which includes plant-specific systems training and in-plant training.
FPRL's proposal of this 5-week training program to the NRC on Nay 14, 1987, stated that the training program would be taught by FPSL SRO instructors.
Three courses in the program, Emergency Diesel Generator (SD gl37),
Reactor Protection (SD 5'63),
and Steam Dumps (SD 8105)
have been taught by an SRO qualified contract instructor.
The inspector noted that the contract instructor did receive plant-specific training in these courses prior to his presenta-tions.
During NRC inspection 87-09 in February 1987, two contract instructors who failed to meet the requirements of NUREG 0737, Item I.A.2.3, were specifically identified to the training staff as having conducted applicable Licensed Operator training.
The week following the inspection, one of the aforementioned contract instructors taught reactor coolant pumps to a
Hot License class.
The inspector noted that the contract instructor did receive three days of plant-specific training, evaluation, and certification for teaching that course prior to his presentation.
Furthermore, two FPSL instructors, one having failed his last NRC requal-ification examination, and the other having failed his last two NRC requalification examinations, taught Licensed Operator Requalification classes in June 1987.
Having been restricted from instructing on specific topics due to the examination failures, they were, nonetheless, certified to teach courses whose contents included the restricted topics.
The first instructor taught Post Accideht Sampling System and Post Accident Hydrogen Monitoring, while his restrictions from teaching included Instrumentation, Dose Limits, and Reactor Coolant System Activity topics.
The second instructor taught various SOERs, LERs.,
IE Inspection Reports and Notices, and INPO OSMRs which included:
SOER-81-15 SOER-82-9 LE R-86-17 IE Information Notice No. 86-38 Partial Loss of DC Power Turbine Generator Exciter Explosion Auto Actuation of Auxiliary Feedwater Due to Lo-Lo S/G Level Deficient Operator Actions Following Dual Function Valve Failures
i
INPO 0&MR-277 Spurious Reactor Trip Due to Radio Transmission The second instructor's restrictions included the following topics:
ONOPs EOPs Technical Specifications Administrative Procedures Turbine Control Auxiliary Feedwater Reactor Protection System The Training Staff was aware of these specific instances and had evaluated use of these instructors prior to the classes.
This practice appears to vary from FP&L intentions presented to the NRC on May 14, 1987.
Through student and instructor interviews, the inspectors noted that student and instructor overtime was still conducted.
FP&L has adequately addressed this concern in making an effort to limit the overtime by requiring senior management approval for extraordinary additional hours.
This employee overtime policy was promulgated by the Site Vice President on June 1, 1987.
The inspector reviewed the implementation progress of the Training Infor-mation Management System (TRIMS).
TRIMS is the means by which FP&L intends to provide:
configuration control of training materials; manage-ment of personnel and program training records; maintenance of examination questions and relative statistical data; maintenance of class data and training program schedules; and maintain the tracking, documenting, and updating of training commitments to include training materials
'and implementation.
Although over three months behind the project milestone schedule for implementation of the Records Management module, the TRIMS system development is steadily progressing.
On June 17, 1987, specific prograoming was initiated to provide a biweekly report to each of the training supervisors detailing the status of incorporation of emergent briefing and training materials into established lesson plans.
The inspectors reviewed two of the initial reports and noted that, properly distributed, the reports appear to provide an adequate means of ensuring the timely handling of that training material.
Inspector review of the Instructor Development program records revealed very little documentation being maintained on contract instructors.
Specifically, there is no record maintained to evidence supervised (not
,self-study) in-plant contract instructor training.
In most cases, only a resume and a letter of certification to teach for FP&L is filed.
This sharply contrasts with the personnel training records of the FP&L training staff.
Additionally, the inspector.noted several minor training record entry errors and brought them to the attention of the training staf,
In conclusion, the licensee has continued to make limited improvements in the area of training.
The licensee has addressed the unsatisfactory Licensed Operator Requalification Program and related areas of concern through more qualified instructors, the hiring of qualified contractor instructors, enhanced instructor classroom training, scheduling and holding instructor in-plant and plant-specific training, and the imple-mentation of an effective tickler system to ensure the incorporation of emergent training and briefing material into permanent lesson plans.
Items of continuing concern include:
the use of recently hired contract instructors to conduct the plant-specific systems training for the contract instructors training program; the continued use of FP8L instruc-tors who have failed NRC requalification examinations to teach Licensed Operator classes in the areas in which they have displayed weakness; and the lack of documentation of supervised in-plant contract instructor training in the Instructor Development program record 'I
'