IR 05000410/1991081
| ML17056B488 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 11/01/1991 |
| From: | Cowgill C NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML17056B487 | List: |
| References | |
| 50-410-91-81, NUDOCS 9111130119 | |
| Download: ML17056B488 (48) | |
Text
U.S. NUCLEAR REGULATORY COMMISSION
, REGION I, Report No.:
Docket No.:
License No.:
50-410/91-81
'0-410 NPF-054 Licensee:
Niagara Mohawk Power Company 301 Plainfield Road Syracuse, New York 13212 Facility Name:
Inspection at:
Dates:
Nine Mile Point, Unit 2 Oswego, New York September 3 - 16, 1991 Inspectors:
J. E. Beall, Senior Resident Inspector, Beaver Valley
'Team Leader)
J. E. Menning, Project Manager, NRR J. T. Yerokun, Project Engineer, DRP R. S. Bhatia, Reactor Engineer, DRS R. R. Temps, Resident Inspector, DRP Approved By:
C. Cowgi, hief Reactor Projects Bran o.
Date
~Summer:
A special team inspection was conducted to review licensee short term corrective actions following the August 13, 1991, reactor trip and Site Area Emergency and to assess the readiness of Nine Mile Point Unit 2 for restart.
Results:
See attached Executive Summary.
911113011~7 911105 PDR
'ADOCK 05000220 Q
PDfk
TABLEOF CONTENTS EXECUTIVE SUMMARY ~......................................
ii 1.0 BACKGROUND
~
~
~
~
~
~
~
~
~
~
~
~
~
~
2.0 PERSONNEL PERFORMANCE 2..1 Event Critique 2.2 Control of Jumpers Directed by Emergency Operating Procedures 2;3 Control of Components Affected by Emergency Operating Procedures
~
2.4 Delayed Surveillance Test on the Suppression Pool to the Drywell Vacuum Breaker...... ~....................
~ ~.....
2.5 Procedures and Training for Loss of UPS
2
3
3 3.0 MAINTENANCEPROGRAM EFFECTIVENESS 3,1 Uninterruptible Power Supply............. ~... ~.......
3.2 Other Equipment Problems, Root Causes and Corrective Action 3.3 Preparation of Replacement Main Transformer for Service..
~
.
~..
, 3.4 Review of Vendor Manual Program....................
~
3.5 Maintenance ActivityTracking
~
~
~
4
~
~
~
13
16 4.0 ORGANIZATIONALEFFECTIVENESS 4.1 Site Area Emergency Declaration and Emergency Plan Implementation 4.2 Resolution of Previously Identified Problems 4.3 Emergency Response Critique.....................
~
.
~..
17
18 5.0 CONCLUSION..
EXECUTIVE ARY Personnel Performance The Readiness Restart Assessment Team discussed licensee performance issues with other NRC personnel involved in the immediate event response as well as the subsequent incident investigation activities.
Based on these discussions, the team reviewed portions of the licensee's self critique; operator actions taken subsequent to the loss of annunciator event to restore equipment to a readiness state; and, corrective action implementation.
On an overall level, the licensee's event critique was found to be generally good with a high level of senior management participation.
Configuration controls for re-establishing. normal system alignment after use of the emergency operating procedures (EOPs) were generally good; however, the licensee identified three sets of switches that could be repositioned by the use of'certain EOPs and that were'not in the licensee's procedure used to track configuration control. Procedure changes were made to include these switches.
The NRC determined that this change would enhance the existing configuration control program.
Operators did not conduct a required surveillance on the suppression pool to drywell vacuum breakers within the required two hour time period following the liftof a Safety Relief Valve during the transient.
The licensee has revised the appropriate plant procedures to'ensure that this surveillance activity is completed within the required time interval. The NRC assessed that this should preclude recurrence.
The inspe'ction team found that the licensee's training program for operation of the Uninterruptible Power Supplies (UPSs) was good.
However, the program did not formally provide information on restart of a faulted or failed Uninterruptible Power Supply.
Although operators were successful during the event in regaining UPS power via the alternate supply (maintenance supply), they were only partially successful at restoring the UPSs to a normal configuration.
It was determined that these activities were outside the scope of the original training program.
Subsequently, the licensee has modified the training, incorporating lessons learned from this event.
The inspection team determined that the enhanced training on UPS operations was good.
Maintenance Pro ram Effectiveness The Restart Readiness Assessment Team reviewed the licensee's corrective actions for several equipment performance problems that were identified during the post event investigations by the NRC and the licensee.
Except for the failure analysis on the "B" phase Main Power Transformer and certain long-term analyses for breaker and logic card failures associated with the UPSs, all associated equipment problem root cause determinations have been completed.
Short-term corrective actions, including both hardware and procedure changes also have been complete Executive Summary (Continued)
The team verified that the licensee completed and successfully tested the modification to the non-safety UPSs.
= The team also verified that the UPS control logic batteries were replaced with fully charged batteries; As a related issue, the team reviewed the licensee's control and use of vendor supplied information, such as recommended preventive maintenance activities. The team reviewed the vendor manual program to determine if recommended preventive maintenance activities were being appropriately incorporated into station procedures.
The team conducted a detailed comparison of the vendor manuals with the maintenance program for a sample of important safety and non-safety components.
A high degree of conformance was noted with
'variances justified by acceptable, documented engineering evaluations.
The program procedures were found to be adequate and interviews with a sample of the responsible personnel found them to be cognizant of their duties.
The team determined that the program was being adequately controlled, but identified some weakness in the timeliness of licensee review of certain vendor manual revisions.
The licensee had an initiative in progress which resolved this concern; licensee management had required that each safety related item in the vendor manual backlog be assessed individually prior to startup.
During the event, there were indications of a potential water hammer in the Residual Heat
'emoval'System.
The licensee conducted a visual examination of the affected equipment and provided an evaluation concluding that any possible stresses experienced during the water hammer were within the design of the piping and supports.
The team evaluated the licensee's review of this area including procedure changes to minimize the potential in the future.
The corrective actions taken by the licensee regarding operator procedure use when warming the residual heat removal system for shutdown cooling mode of operation were appropriate.
This included re-emphasizing to all operators the actions to take when steps in a procedure cannot be followed. The team also reviewed and found acceptable licensee actions regarding the oscillation and check valve indication difficulties experienced on the reactor core isolation cooling system and the steps taken to prepare the new "8" phase Main Power Transformer for service.
Finally, the team reviewed the status of the overall maintenance program, the tracking of outstanding items, and the level of management oversight.
The licensee had made substantial progress in reducing the corrective and preventive maintenance (PM) backlogs.
Licensee management was found to be actively involved in tracking and prioritizing maintenance activities.
In addition, all outstanding safety related corrective maintenance activities are being assessed by the licensee'or potential operability impact prior to restart.
r anizational Effectivene.
The Emergency Response Organization (ERO) performance was assessed to be good.
Operator recognition and assessment of the event was good. The licensee was found to have appropriately classified the event-in a timely manner.
Also, direction and coordination of the ERO within each emergency response facility was effective.
Three problems w'ere identified during the
Executive Siimmary (Continued)
licensee's critique of the event that warranted corrective action prior to restart.
During the initial stage of the event the licensee communicator made some required notifications of offsite organizations in the wrong order.
Secondly, some personnel responding to the event had difficulty in crossing offsite boundaries to gain site access because they did not have their emergency responder
"green card" in their possession.
Third, there was a delay in implementing the personnel accountability procedure.
The team reviewed the licensee's corrective actions for the three problems and found them to be acceptable.
Details of inspection activities in the area of emergency response are contained in NRC Report Nos. 50-220/91-19 and 50-410/91-1 DETAIL/
1.0 BACKGROUND On August 13, 1991, the Nine Mile Point Unit 2 (NMP-2) main transformer experienced a
failure which resulted in a'reactor trip from full power.
The electric fault, which.was on the B phase of the transformer, led to the loss of five non-safety Uninterruptible Power Supplies (UPSs). The affected UPSs powered control room non-safety displays, annunciators and alarms.
Loss of these control room indications concurrent with the transient associated with the reactor trip required declaration of a Site Area Emergency (SAE) in accordance with approved procedures.
Licensee operators successfully restored the UPSs and, upon entry into a Cold Shutdown condition, the SAE was terminated.
The NRC dispatched an Augmented Inspection Team (AIT)
to the site on the day of the event.
The NRC investigation of the event was later upgraded to an Incident Investigation Team (IIT)..For details of the August 13, 1991 event, see Inspection Report 50-410/91-17 and the IIT report, NUREG-1455, "Transformer Failure and Common Mode Loss of Instrument Power at Nine Mile Point,.Unit 2 on August 13, 1991.
The circumstances of this event are also discussed in NRC Information Notice 91-64:
Site Area Emergency Resulting from a Loss of Non-Class 1E Uninterruptible Power Supplies.
Niagara Mohawk Power Corporation (NMPC) agreed not to restart NMP-2 until the NRC had completed investigation of the incident.
These commitments were documented in Confirmatory Action Letters 91-12 and 91-13.
Based on information from the IIT, the Office of NRR reviewed the licensee's root cause analysis of the UPSs failure and the corrective actions taken or planned.
The completion of the NRR review was documented in a September 16, 1991, Safety Evaluation Report (SER). The SER concluded that licensee corrective actions associated with the UPS failures were appropriate for restart of NMP-2.
A special Restart Readiness Assessment Team (RRAT) conducted an onsite review of short term corrective actions and licensee readiness to restart NMP-2.
The RRAT Inspection Plan (Attachment 1) was developed to address issues identified by the IITas requiring on-site follow up for resolution.
The RRAT inspection was conducted September 3 - 16, 1991, with the exit meeting held as part of a public meeting on the evening of September 16, 1991, in the Lanigan Building on the Oswego campus of the State University of New York. A list of the key licensee personnel contacted during this inspection is provided in Attachment 2.
On September 18, 1991, the NRC released Niagara Mohawk Power Corporation from the Confirmatory Action Letter restart hold.
The restart authorization was documented by a September 18, 1991, letter from Mr. Thomas T.
Martin, Region I Administrator, to Mr. B. Ralph Sylvia, Executive vice President - Nuclear, Niagara Mohawk Power Corporatio.0 PERSONNEL PERFORMANCE 2.1 E~C'
Overall; NMPC management of the restart effort was good.
The team attended portions of the licensee's internal event critique. The critique was found to be generally good with a high level
"
of senior management participation.
Numerous Work Requests, Plant Change Requests, Engineering Evaluation memos and other 'documents were initiated as part of the licensee's investigatio'n and troubleshooting activities (i.e., after the event).
The team reviewed a sample of the licensee documents and identified no notable deficiencies.
2.2 ontrol of Jum ers Directed Emer enc
ratin Procedures The team reviewed the licensee's controls and documentation as'sociated with temporary circuit modifications (jumpers) directed by the Emergency Operating Procedures (EOPs).
The EOPs require that jumpers be installed or switches positioned to bypass certain signals or automatic actuations.
Specifically, on August 13, 1991, operators were required to install jumpers to bypass scram signals to allow the reactor scram to be reset.
Following the reactor scram there was no indication of control rod positi'on since the full core display, the rod control system and rod worth minimizer and the CRD reed switches were deenergized.
Once power was restored, there was indication that six rods were not fully inserted.
Operators p'roceeded to install the jumper and reset the scram.
Once the scram was r'eset, the rods settled to the OO position, indicating that the rods had fully inserted to the over-travel position.
The RRAT reviewed the control ofjumper installation and removal and identified no deficiencies.
The team determined that the appropriate procedure (N2-EOP-6, EOP Support Procedure)
was referenced by the EOPs and directed the following actions:
pulling relays, liftingleads, pulling fuses or installing jumpers.
For each of these actions called out in the EOPs, N2-EOP-6 contained an attachment (Attachment 29, Circuit Modification Tracking Sheet)
to control initiation and restoration of the temporary alteration.
The installation and removal steps in each supporting attachment required that the action step-be checked and that the Senior SRO sign an attachment, recognizing modification of the circuit.
Although normal temporary modification (TM) controls required signatures for these actions, Administrative Procedure 6.1 (step 1.2,10) specifically excluded normal-TM controls for temporary alterations directed by the EOPs, The tracking sheet also provided a column for signing that tlie system had been returned to normal.
After the event, N2-EOP-6 was revised to provide an additional column on the attachment for a verification signature that the system has been returned to normal.
This made the restoration section consistent with normal TM controls, although as noted previously, these controls were exempted by Administrative Procedure The team identified no deficiencies, concluded that the control of the installation and removal ofjumpers was found to be acceptable, and the post-event revision was determined to represent a procedure enhancement.
2.3 ontrol of om onent Affected Emer enc 0 eratin Procedures The licensee conducted a post-event review of the EOPs to determine ifthere were any instances of equipment (other than those discussed in Section 2.2) taken out of the normal standby lineup.
No additional temporary alterations requiring control were identified; however, the repositioning ofthree sets of switches used to defeat or inhibit functions were identified. These switches were not used during the August 13, 1991 event, but could be repositioned during other, different events.
The switches were added to the Circuit Modification Tracking Sheet N2-EOP-6 to provide tracking of their repositioning and return to normal.
Also, N2-OP-101C, Plant Shutdown, was revised to add a step to require the operators to ensure that any switches repositioned during EOP actions be restored to normal.
The team reviewed the licensee's actions and concluded that temporary alterations called for by the EOPs were properly controlled and in accordance with approved procedures.
The team found that the procedure revisions represented-enhancements and constituted a noteworthy initiative.
2.4
'ela ed urveillance Test on the u
ression Pool to the D well Vacuum Breaker The operators were delayed in identifying that two of the Safety Relief Valves (SRVs) had lifted during the scram.
Consequently, the cycling of the suppression pool to drywell vacuum breakers, required by the Technical Specifications (TS) to be performed within two hours after SRVs lift, was not performed within the TS time-limit. The applicable procedure, (N2-OP-101C, Plant Shutdown)
was revised after the event to add a step requiring that the TS surveillance be performed within two hours after a scram.
Additionally, the procedure now requires the operators to verifyifany SRV has or has not lifted and to use reactor pressure trend information off of the Post Accident Monitor (PAM) recorders as well as the SRV tailpipe temperature recorder to make this determination.
The team concluded that the revisions to the procedure adequately addressed the delay in completing the surveillance test and should prevent recurrence.
2.5 Procedures and Trainino for ss f P,
As discussed in Section 1.0, the event involved the loss of five UPSs.
Although operators were able to restore power in about 30 minutes, there appeared not to.have been specific training on the loss of annunciators such as occurred during the event.
Also, no list of the specific loads powered from each UPS was present in the control roo The team reviewed the UPS lesson plan dated October 26, 1989, and found it to be well written.
The lesson plan provided information*on the various types of transfers that can be performed on the UPSs, power up of a shutdown UPS, and general loads offof the various UPSs.
The lesson plan did not provide information on restart of a faulted UPS; however, the team concluded that the information was outside the scope of a licensed operator class.
The in-plant Job Performance Measure (JPM) was also determined to be comprehensive and well written.
Subsequent to the event, the licensee revised procedure N2-OP-71, 13.8 KV/4160 V/600 V AC Power Distribution, to provide instructions on the restart of a faulted UPS.
The procedure-revision addressed the restoration of critical loads, via the maintenance supply, offof a faulted UPS. This revision was incorporated into a lesson plan and the team confirmed that all licensed and non-licensed operators had received training on it.
Additionally, training in the field, through use of a JPM, was performed to further acquaint operators with the operation of the UPSs.
The team concluded that the training on operation of the UPSs, the incorporation of lessons learned from the event, and the enhancements in operator training were acceptable.
3.0 MAINTE<NANCEPROGRAM EFFE<CTIVENESS 3.1 ninterru tible Power Su l
3.1.1 Root Cause of ninterru tible Power Su lies Failure During the August 13,1991 event, five non-safety related Exide Uninterruptible Power Supply (UPS) units tripped because of a ground fault on the B phase of the main step-up transformer.
The main step-up transformer fault caused a voltage drop on the maintenance supply to all of the five UPS units. As designed, the UPS system control logic is powered by DC power supply from the maintenance power supplied.
The inventer output power supply acts as a alternate power supply to the UPS system control logic. The logic power has a relay K5 in relay panel A27 which is normally energized from the maintenance power supply and has two sets of contacts for the selection of power source supplies (preferred or alternate).
In normal configuration, the control logic power supply is supplied through a set of contacts held closed when the K5 relay coil energized from the maintenance supply.
Upon a loss of the maintenance power supply the K5 relay coil is deenergized, thereby causing the switching of the UPS control logic power to the alternate sourc The IITconcluded that the voltage degraded on the maintenance supply simultaneously in all of the five UPS units.
The degraded voltage condition on the maintenance supply caused the voltage on the control logic power supply to decrease below its trip setpoint value and as a result
'all the five UPS units, 2VVBB-UPSIA,B,C,Dand G, tripped. Therefore all these five units shut down (all units output and incoming power source breakers to the units opened).
Automatic load transfer to the maintenance supply was prevented by design due to the degraded voltage condition on the maintenance supply.
It was also discovered that, under degraded voltage conditions, the UPS logic power switching circuit did not actuate ifthe logic tripped due to low voltage condition (i.e., below its setpoint limit).
The licensee documented the results'of their root cause analysis in submittals to the NRC dated September 10 and 11, 1991. The NMPC analysis and corrective actions were reviewed by NRR'nd found to be adequate for restart of NMP-2 as documented in the NRR Safety Evaluation Report, "Nine Mile Point Restart Issues," dated September 16, 1991.
The RRAT conducted onsite review, based on the IITconclusions regarding the root cause of the UPS failures, ofUPS modifications (Section 3.1.2), post-modification testing (Section 3.1.3),
maintenance program (Section 3.1.4), and planned PM upgrades (Section 3.1.5).
,3.1.2 M diIi The licensee's corrective actions included modifying the UPS control logic'power supply by changing from the existing maintenance power supply source to a more reliable UPS inventer output power supply. During this inspection, it was verified that all UPS control logic batteries had been replaced.
These batteries could hold the UPS unit through a brief power degradation or interruption.
The team reviewed the post event modifications for accuracy and functionality.
Engineering design change packages 2EDC No. 2E10463 through 2E10467 were reviewed along with their associated Work Requests (WRs).
These design modifications changed the UPS control circuit logic power supply source from the existing arrangement of maintenance power supply as the preferred source to the inventer output power supply source.
The maintenance power supply was changed to an alternate power supply for the UPS control logic circuit. Relay coil (K5) was rewired to be energized from the UPS output power, supply instead of the maintenance power supply.
The two sets of contacts of K5 relay were arranged (swapped) to ensure that the UPS output power supply was available to the UPS control logic in a normal configuratio The team reviewed the WRs implementing the UPS modifications and satisfactorily verified the conformance of the mqdified wiring (point to point) for two UPS units (2VVBB-UPSlAand G).
The remaining units'ocumentation and test results indicated that the changes were done adequately.
The UPS control logic batteries were verified to have been replaced under UPS trouble shooting and testing work orders.
The team also reviewed the'licensee's safety evaluation for these modification and verified that NMPC had adequately determined that the subject modification did not involve an unreviewed safety question.
No unacceptable conditions were identified during this review.
3.1.3 Post-Modification Testin The licensee performed post modification testing to demonstrate that the UPS control logic power was supplied from a more reliable (UPS output) source and that ifa transient occurred on the input AC source to UPS units, it would not affect the UPS control logic power" supply.
The team reviewed the licensee's test procedures and results to determine whether the results adequately demonstrated the acceptability of the above modifications.
The team found that the licensee had performed several tests on these units.
A fast transient interruption of approximately 100-150 milliseconds duration was simulated on input AC power source of the UPS control logic to prove that the unit would not trip. Also, the function of the K5 relay and associated contacts was verified by transferring the UPS unit from the preferred AC power source to the alternate power source.
The test demonstrated that the UPS control logic control power was connected as per the design modification, that the UPS unit could withstand an abnormal condition transient; and that the unit could carry the assigned loads without interruption.
The team concluded that the units had been adequately tested.
3.1..4 Review of UPS Maintenance Pr ram The team reviewed NMPC's preventive maintenance program (PM) for the plant UPSs.
This involved reviewing various vendor manual recommendations of all UPS units and plant preventive maintenance procedures.
The team also held discussions with plant staff and the UPS system engineer.
The UPS system was divided into the emergency or safety related UPS system and the normal or non-safety related UPS system.
The plant normal UPS system provides power to the plant non-safety instrumentation and control loads, annunciators, communication systems, fire protection panels, RPS system, essential lighting and computers.
The plant safety-related UPS system provides power to plant safety-related instrumentation and control system Based on the review of the UPS vendor manuals, the team found that the significant PM recommendations were that the UPS units be checked frequently for cleanliness and the physical condition of the air filter, capacitors, batteries, and fan units conditions.
In addition, on an annual basis, the power connections should be checked for tightness and early recognition of deteriorating components due to overheating.
Also, on a regular basis the key parameters of UPS units, such as input voltage, unit output voltage, frequency, and current performance should be monitored.
The team reviewed several licensee's PM procedures established for safety-related and non-safety UPS units.
The safety-related PM procedures appeared -to be more comprehensive and in accordance with the vendor manual recommendations.
Safety-related PM procedure N2-EPM-GEN-665 covered general inspection and monitoring of key parameters of the UPS units on a weekly basis.
The same procedure was also utilized to check for any sign of overheating, fan operability and filter replacement concerns on a weekly basis as compared to a monthly basis inspection recommended by the vendor manual. Refueling procedure No. N2-EPM-GEN-RF635 was used for the overall inspection of the UPS units as well as the inspection of the internal wiring and other components for degradation or defects.
The team found the safety-related PM program adequate based on the procedures established by the licensee's and the team's verification of samples of PM procedures completed before August 13, 1991.
The team found that the non-safety UPS PM program was not as comprehensive as that of the safety related.
Plant procedure N2-EPM-GEN-Q575 covered'the inspection and replacement of filters on a quarterly (3 months) basis During this time all key parameters were monitored.
Any adjustment and abnormalities were Corrected under separate corrective action procedures.
The team reviewed tlie past six months maintenance history of the affected UPS units and found no significant deficiencies or major concerns.
In addition to the maintenance PM program, the Operations department also monitored the unit output voltages and any abnormal condition on these units per procedure No. N2-OP-71.
During discussions with plant maintenance staff and the system engineer, the team found that the licensee had recognized the weaknesses in these UPS PM program and had requested additional procedures to enhance the overall program.
At the time of the inspection, the PM program for the non-safety UPS units contained no provision to inspect the units'nternal wiring connections for tightness and component degradation (e.g.,
due to overheating).
However, the licensee had previously recognized these concerns and had initiated several deviation/event reports requesting the development ofadditional PM procedures to cover vendor recommendations outlined in the vendor manuals.
The team assessed the safety-related PM program to be adequate; however, enhancements to the non-safety PM program, such as additional periodic inspections, were appropriat.1.5 PM Pro ram U rades As discussed above, NMPC had planned to upgrade the existing PM program of safety and,non-safety UPS before the event. The team found that several Deviation/Event Reports (DERs), had been initiated by the system engineer to upgrade the existing PM program between April2, 1991 and July 16,,1991.
The team concluded that the new PM procedures, when issued; should address the differences between vendor manual recommendations for the UPS units and the licensee's program.
The licensee stated that the target date to issue the new/revised procedures was December 31, 1991.
Based upon the review of the licensee's PM upgrade plan and the modifications completed on the affected non-safety UPS units the team concluded that NMPC's actions were acceptable to support a safe return to power operations.
3.2 ther ui ment Problems Root auses and orrec ive Action 3.2.1 Feedwater S stem Valves MOV-84 A B and During the events of August 13, 1991, operators were not able to open feedwater pump suction valves MOV-84A,B and C under system pressure generated by the condensate booster pumps.
This was not a major problem since the operators'ere able to use the startup flow control valve (which bypasses the affected valves) to supply feedwater to the reactor vessels.
The differential pressure across the suction valves was about 500 to 600 psig.
Subsequent evaluations by the licensee indicated the following potential cause's:
The actuator to valve mounting bolts were loose, the torque switch bypass (open) was set too low, the torque switch was unbalanced, the valves had been operated incorrectly.
Upon inspection, the licensee found that the mounting bolts were loose, thereby allowing slight actuator rotation during valve stroke.
Diagnostic analysis indicated that the torque switch was balanced (same opening and closing set torque values), a'nd that the torque switch bypass setting (6% of travel) was too low.
The operators varied from procedure by not opening the valves'ypass lines prior to attempting to open the valves.
The operators stated that this option was selected because they did not want to send personnel to locally open the manual bypass valve due to the fact that radiation monitors were alarming in the area (possible radiation exposure).
The licensee tightened the loose bolts on the actuator to a torque value agreed upon with the vendor. The team reviewed the applicable vendor manuals and found that there were no specific torque requirements for the size and materials of the actuator bolts.
The licensee initiated a revision to the feedwater valves'endor manual to specify the new torque values for the actuator bolts.
The licensee also inspected all safety-related MOVs for loose bolts.
None were foun,1
Safety-related MOVs normally require the actuator bolts to be tightened to some specified torque value.
The bolts on all other MOVs were torqued or lock wired. The team concluded that the feedwater valves'oose bolts had no implications on safety-related systems.
The licensee stated
'hat for added assurance, procedure N2-MMP-GEN-316, "Preventive Maintenance for Safety Related Valves," would be revised to use lock wires on the bolts in addition to tightening to specified torque values.
The licensee changed the torque switch bypass setpoint from 6% to about 12% open. Diagnostic tests had indicated that the torque switch was not bypassed long enough to allow pressure across the valve seat to be equalized.
The team concluded that there were no similar problems with safety-related valves since safety-related MOVs had torque switch bypass settings of 90% to 95% open.
During the review, the team noted that the valve manufacturer had sized, the valves based on an 800 psi differential pressure requirement.
The estimated differential pressure seen by the valves during the event was lower than this design pressure.
The as-found torque switch settings were such that enough torque was available for the valves to have opened under the estimated differential pressure conditions. The licensee developed a special test procedure (STP-27) to test the valves under similar differential pressure conditions.
When this test was found not.to be feasible, NMPC revised the system's operating procedure (OP-3, "Operating Procedure for Condensate and Feedwater,") such that the feedwater suction valves would not be opened under differential pressure conditions. The team reviewed the revised procedure and had no concerns.
The team reviewed the NMP's maintenance activities, diagnostic test results, the post maintenance tests performed, and the maintenance history for valves MOV 84A,B and C. The team also reviewed the Operation and Maintenance Manual for the involved valves to determine ifany safety implications and/or discrepancies existed that might indicate potential programmatic problems with vendor manuals.
None were found.
Licensee actions associated with the feedwater valves (MOV 84A,B, and C), including troubleshooting and corrective actions, were found to be acceptable.
3.2.2 Reactor Core Isolatiori oolin S stem Oscillation The Reactor Core Isolation Cooling (RCIC) system exhibited flow oscillations during one part of the June 27, 1991, quarterly surveillance test (N2-OP-ICS-Q002).
Similar but greater ( 100 gpm) RCIC oscillations led the control room operators to take manual control of the system during the event to control reactor vessel level. The team reviewed licensee actions associated with evaluating the earlier RCIC surveillance test result The RCIC quarterly surveillance test involves hn automatic start of the system by a test signal and the monitoring of system performance with the flow path back to the source (instead of the
.
reactor vessel as during normal operation).
System performance was satisfactory, including an'nduced flow change (600 gpm to 450 gpm) to check system stability. After completion of tliis part of the test, operators began to set up for the Inservice Testing (IST) portion of the test for pump performance tracking. It was at this transition point of the test, while setting up for IST data, that RCIC oscillations were observed.
Oscillations were estimated at 50 gpm by the test engineer at the time, but licensee later concluded that the oscillations were approximately 30 gpm (20 gpm is acceptable, the "green band").
The oscillations could be initiated or stopped by small 'movements of the test return valve (used to simulate backpressure).
The behavior was reviewed by the senior onshift SRO (SSS) and assessed as not impacting operability.
Licensee personnel felt that the oscillations were a test effect due to system dynamics.
The oscillations were not logged in the control room shift supervisor's log or reported to Operations management.
An engineer initiated a WR to tune the controller at the next outage.
The team concluded that RCIC test performance had received an adequate level of review.
However, absence of a'Control Room log entry concerning the oscillations prevented licensee operations management from having the opportunity for assessment and oversight.
That oscillations could be induced by variance in discharge piping pressure was an indicator that RCIC might not be stable during an event.
Loop calibration during the post-event outage identified some
"as found" values out of tolerance.
Also, air and "crud" were found during venting.
The team noted some weaknesses in the development of the initial I Ec C troubleshooting procedure, but found the final testing and troubleshooting activities to be acceptable and in accordance with generic industry guidance.
The licensee initiated a change to the surveillance test to include observing RCIC performance at different backpressures and to add criteria to assess stability.
Also, during the planned startup, the licensee stated that RCIC would be tested at 500 psig as well as the normal pressures (150 psig and fullpressure).
The team reviewed the licensee's planned RCIC startup testing and found it acceptable.
3.2.3 Reactor
. re I olation oolin Valves A V-1S6 and 1S7 During the event, the control room operators observed dual indication of the position of testable check valve AOV-156 in the control room.
Immediately following the event, AOV-157 also failed the position indication test.
Subsequent troubleshooting efforts indicated that the valves had been in the actual required positions but problems with the limit switches had caused erroneous valve positions to be indicated in the control room.
For the troubleshooting efforts, work request (WR) ¹193344 was initiated for the AOV-157, while WR ¹193343 and WR
¹194584 were initiated for the AOV-15 Following the troubleshooting efforts, slight adjustments were made to the valves'imit switches.
AOV-157 later failed its position test again.
Work request 1190265 was then initiated to perform physical repairs on the valve as required.
The valve was disassembled for internal inspection and found stuck closed from the last troubleshooting efforts.
The problem was attributed to vacuum that had developed between the two valves.
The cam plate for the position, indicator was found broken and was repaired.
The valve was reassembled and tested satisfactorily.
Post modification testing also included a hydrostatic test.
The hydrostatic test was performed satisfactorily at 1,045 psig using procedure N2-OSP-ICS-R0001, "RCIC Pressure Isolation Valve Leakage Test."
The team reviewed the maintenance history of both AOV-156 and 157, and found that a high number of limit switch (position indication) problems had been experienced since, initial plant startup (about 5 years).
The licensee stated that this type of problem was prevalent in the industry with the testable check valves and that resolution efforts were ongoing. Water slugging against the valves'isk caused damage to the position switch cams which.then resulted in the position indication problems.
The licensee attempted to prevent future water slugging damage by modifying applicable procedures (N2-OSP-ICS-CS001, Cold Shutdown IST Surveillance and OP-35, RCIC Operating Procedure)
such that the leg between the valves would be filled with water prior to system operation.
The team c'oncluded that while the position indication problems had existed for a period of 5 years, the ability of the'estable check valves to perform their safety function had not been degraded.
Also, the troubleshooting efforts, subsequent repairs and post maintenance testing (including the hydrostatic test) had been performed adequately.
3.2.4 Water Hammer in the Residual Heat Removal S stem While preparing to start up the R'esidual Heat Removal (RHR) System in shutdown cooling, there were reports of noise (potential water hammer) in the system.
Also, there'ere problems with the system drain valve operation from the control room.
The team reviewed the procedure that was in use during the potential water hammer event, (Procedure N2-OP-31,
"Residual Heat Removal System,"
Section F.6.20) and noted that it provided instructions on system piping warmup.
After interviewing the operators, the team concluded that the procedure (with one exception) had been properly followed. The team found that the procedure had been used previously without any indication of a water hammer.
N Engineering personnel walked down the portions of the RHR system which would have likely.
experienced water hammer.
No indications of damage to piping, valves or pipe supports and hangers were identified.
The team performed an independent walkdown of the affected piping and did not identify any damag.12 The licensee conducted an engineering analysis which concluded that the stresses encountered during the startup of the system were bounded by the system design bases.
The procedure was revised to allow slower introduction of flow. The team reviewed the engineering analysis and the revised procedure and identified no deficiencies.
While attempting to control reactor vessel water level during shutdown cooling, control room operators attempted to open the RHR discharge to radwaste valve (2RHS*V142).
The valve could not be opened from panel P601. Operators were dispatched to the area of the valve and were able to effectively throttle the valve manually. Work Request (WR) ¹193350 was initiated to troubleshoot and fix the problem.
Dirty contacts (39C-39) were discovered in the remote shutdown panel.
The contacts were cleaned and the valve was tested satisfactorily.
The team reviewed the maintenance work package and found no discrepancies.
A review of the valve's'aintenance history showed no trend of similar or repetitive problems.
3.2.5 Water Hammer in Reactor Water leanu stem Operators reported hearing water hammer during -restoration of the Reactor Water Cleanup System (WCS).
Also, the WCS experienced an automatic isolation on high differential flow during restoration.
The team reviewed the licensee's root cause determination, potential damage assessment, and corrective actions.
Operators manually tripped the one operating WCS pump shortly after the scram, in accordance with the Plant Shutdown procedure (N2-OP-101C, step H.1.11); an option to transfer WCS in the full reject mode to the main condenser was not exercised.
Several hours later, directions were given to place the WCS back in service per the WCS operating procedure (N2-OP-37, section E.4). The team determined that this was the first time that this section of the procedure had been performed under the particular conditions present at the time:
WCS pump secured, portions of the WCS greater than 220 degrees Fahrenheit, and reject flow path to the main condenser still at vacuum.
The operator tasked with restoring WCS to operation followed the procedure verbatim; however, the procedure was deficient in that it contained several steps which were not sequenced properly and subsequently led to the water hammer and WCS isolation events.
In the performance of section E.4 of N2-OP-37, as it existed on August 13, the isolation valve on the discharge side of WCS pump 1B was shut preparatory to venting portions of the system, since it had been out of service for greater than 30 minutes.
Closing this isolation valve caused the downstream portion of the WCS system to be isolated from primary plant pressure.
The procedure then aligned the return flowpathway, referred to as reject fiow, to the main condenser which was at vacuum.
Then, section E.4 directed the 1B WCS pump to be started (its discharge valve was still shut) and incorrectly called for valve 2WCS-FV135, which is used to throttle and control the rate of the reject flow, to be opened 30 percent.
Since the reject path was selected to the condenser wliich was at vacuum, opening 2WCS-FV135 allowed a substantial portion of the WCS upstream of the valve to become depressurized.
Portions of those piping sections
which contained hot (>220 F) water then flashed to steam, forming voids in the system.
When the WCS pump discharge valve was later opened to reinitiate flow, the system repressurized and the voids collapsed resulting in the water hammer event.
Following the event, engineering personnel performed a walkdown of the system which revealed no abnormal conditions with the piping, equipment or supports associated with 2WCS-FV135.
Additionally, piping upstream and downstream of the valve were inspected and no concerns identified.
The team performed an independent walkdown of the affected piping and did not identify any evidence of excessive valve and pipe movement or any apparent damage to the system or its supports.
The flashing and subsequent voiding in the WCS piping also affected the flow detector used to provide information on the reject flow rate to the differential timer circuitry.
When flow through the WCS was reinitiated, the differential flow timer circuit sensed a mismatch between inlet flow, which was indicating properly, and the reject flow rate which was improperly indicating a low flow condition. The false flow indication was the result'of the flow transmitter being affected by the flashing and void formation in the discharge piping.
Due to the sensed flow mismatch, the flow timer circuit initiated a WCS isolation (as designed) which caused the WCS containment isolation valves to shut.
The team reviewed the licensee's investigation of the WCS automatic isolation and identified no deficiencies.
The team reviewed the revision to section E.4 of N2-OP-37 issued on August 27, 1991.
This revision to the procedure ensured that the WCS upstream of2WCS-FV135 remained pressurized before the throttle valve is opened.
The team concluded that the revision should prevent voiding and subsequent water hammer in a significant portion of the WCS upstream of 2WCS-FV135.
The water in the rest of the potentially affected piping would be much cooler, about 120 degrees F. The N2-OP-37 revision also required that the throttle valve be opened slowly and that flow through it be gradually increased so as to minimize or eliminate flashing downstream of the valve. Also, flashing downstream of the throttle valve is further prevented or minimized by the presence of a restricting orifice designed and installed for that purpose.
The licensee also completed an engineering evaluation which determined that the forces experienced by the WCS during the event were within system design bases.
The team reviewed the engineering evaluation and identified no deficiencies.
3.3 Pre aration of Re lacement Main Tran f rmer for ervice The team reviewed licensee actions to prepare transformer 2MTX-XM1Dfor service as the "8" main transformer.
This transformer was prepared for service following the failure of transformer 2MTX-XM18 on August 13, 1991.
(The licensee planned to ship transformer 2MTX-XM18offsite for failure analysis.)
Transformer 2MTX-XM1Dwas installed spare and did not have to be moved to function as the "8" main transformer.
The replacement unit is a Type GSU transformer manufactured by McGraw-Edison with Serial No. C-06607-5-The replacement transformer was inspected and tested prior to energization in accordance with Section VIII'flicensee procedure EOP 401A, "Inspection, Installation, Testing and Storage Procedure for New and Rebuilt Transformers."
Initial energization via a backfeed commenced on August 26, 1991. Following the completion of a 84-hour "soak" period, the transformer was deenergized and inspected and tested in accordance with Section IX of procedure EOP 401A.
The team reviewed procedure EOP 401A and documentation of the completed work and
, discussed preparation for ser vice activities with the cognizant electrical maintenance supervisor.
The team also reviewed vendor recommendations for energization of the transformer and determined that licensee actions were consistent with vendor recommendations.
The licensee analyzed gas in transformer oil samples prior to and during the 84-hour "soak" period.
These samples were taken in accordance with procedure N2-EPM-GEN-0692,
"Obtaining Oil Samples from Outdoor Transformers."
The team reviewed the results of these analyses and noted that the ethane concentrations of the three samples taken during the 84-hour period were 102, 108, and 108 ppm.
Procedure N2-EPM-GEN-0692 provides a limit of 115 ppm for ethane.
Discussions with the cognizant electrical maintenance supervisor revealed that the licensee was aware of the elevated ethane lev'els in the transformer oil. The licensee stated that initiallythe transformer oil would be sampled daily after the transformer was re-energized.
The frequency would then be adjusted based on evaluation of the initial results.
The team concluded that transformer 2MTX-XM1Dhad been prepared for service in accordance with NMPC procedures and applicable vendor recommendations.
3.4 Review of-V'endor Manual Pro ram 3.4.1 Vendor Manual Revision Procedure and Trackin The team reviewed the program procedure, NEL-426, "Vendor Technical Manuals-Review and Processing."
The procedure adequately specified the type and nature ofreview required for new and revised vendor manuals.
Based on interviews, the team found licensee personnel to be cognizant of their duties as specified by NEL-426.
The team noted that NEL-426 did not contain any time guidelines or limits for each review.
The licensee was found to be knowledgeable of the size of the review backlog (about 360), but not the overall age of the items.
Upon further review, it was determined that there were many items which were very old (about 45 were over 3 years).
The team noted that the licensee had an initiativealready in progress to assess the impact of each item in the backlog.
The licensee had an internal commitment from senior management to complete this assessment prior to startup.
The team concluded that the lack of time limits or controls on vendor manual reviews constituted a program weakness.
The absence of program tracking by management was also found to be a weakness.
The licensee concurred that additional measures were needed to assure timely review of vendor manual change 'I
3.4.2 Im lementation of Vendor Manual Recommendati n
- The team reviewed the PM for the RCIC Terry Turbine, main turbine EHC system, RPS MG sets, KSV emergency diesel generators, and 4,160 V Magne-Blast breakers.
The team initially reviewed related vendor manuals to identify recommended PM. The vendor recommendations were then compared with the plant PM program through the review of PM, surveillance, and operator rounds procedures.
The intent of this effort was to determine if the PM program adequately addressed vendor recommendations.
Vendor recommendations for the RCIC Terry Turbine that were reviewed by the team were listed on page 5-23 of the vendor manual identified by File No. N20645, "Instruction Manual for Terry Turbine - RCIC System."
Recommendations for weekly, monthly, quarterly, annual, and five-year maintenance are provided on this page.
The plant PM program was found to be consistent with the vendor recommendations with two exceptions.
The vendor recommended that the operability of the turbine assembly and supporting pressure regulation equipment be checked on a monthly basis.
The team noted that these activities were being performed on a quarterly basis in accordance with the TS and procedure N2-OSP-ICS-002, "RCIC Pump and Valve Operability Test and System Integrity Test." Additionally, the vendor recommended that all linkages be cleaned and lubricated on a quarterly basis.
This work was found to be performed on a refueling outage basis in accordance with procedure N2-MPM-ICS-V452,
"Reactor Core Isolation Cooling Turbine and Accessories."
In folio ving up on these two
'eviations from vendor recommendations, the team reviewed Equipment Qualification Maintenance Program Data Sheet P800ADC for the RCIC Turbine and determined that the deviations had been evaluated and adequately justified by the licensee.
Review of the main turbine EHC system focused on the recommendations listed in Section XII of vendor manual GEK-46355E, "Hydraulic Power Unit." Section XIIlists recommendations for weekly, monthly, quarterly, semiannual, annual, and four-year testing and maintenance.
The seven EHC-related PM procedures reviewed by the team in this inspection effort were found to be particularly well written and provided a high level of detail.
Review of the RPS MG sets focused on the maintenance recommendations listed on page 3 of vendor manual GEH-2301,
"Horizontal, Ball-Bearing, Polyphase Induction Motors," and on pages 5 and 6 ofvendor manual GEI-65501, "Tri-Clad Brushless Synchronous Generator."
Specific intervals for PM activities were generally not provided for the RPS MG sets.
Review of the KSV emergency. diesel generators utilized the recommended annual maintenance items listed on pages 15-2 and 3 of the vendor manual identified by File No. N20395, "KSV Diesel Generator Instruction Manual."
Review of4,160V Magne-Blast breakers utilized the annual maintenance recommendations listed on page 33 of the vendor manual identified by File No. N21135, "GE Metal Clad Switchgear Type M26 and M36, GEH-1802."
The team concluded that the licensee's PM program for safety-related and important to safety equipment adequately addressed vendor recommendations.
Although two deviations involving frequency of maintenance were identified, the related activities were included in the PM program and were being performed.
Additionally, the deviations had previously been evaluated and adequately justified by the licensee.
3.4.3 Vendor Manual Pr ram Review umma The team found the Vendor Manual program to be adequately proceduralized.
Program weaknesses were identified in review timeliness and management tracking.
The quality of the vendor manual reviews sampled was very good.
Implementation of completed vendor manual changes was also good, including adequate justification for identified variances from vendor manual recommendations.
The team found the licensee initiative (already in progress) to assess, prio'r to startup, each outstanding vendor manual revision to be adequate short term corrective action.
Long term corrective actions to address the identified program weaknesses will be reviewed as part of the routine NRC inspection program.
3.5 Maintenance Ac ivit Tr ckin A Maintenance Performance Assessment Team (MPAT) inspection was conducted at the Nine
- Mile Point facility in October, 1990 (Inspection Report 50-220/90-81; 50-410/90-80).
The MPAT found performance in the maintenance area to be acceptable.
The MPAT concluded that the maintenance personnel were knowledgeable and experienced, and that work was performed deliberately in accordance with procedures.
During this inspection, the RRAT reviewed outstanding maintenance activities with respect to tracking and prioritization.
The team found the tracking mechanisms available for management oversight to be of good quality.
The reports provided diverse profiles of the data associated with outstanding maintenance activities.
For example, Preventive Maintenance (PM) items were broken out by discipline, categorized as safety or non-safety, and the backlog trended.
The team found good evidence that licensee management was effectively tracking and prioritizing outstanding maintenance items.
The backlog of corrective maintenance items and late PMs showed consistent improvement since the beginning of 1991.
All tracked items had been reduced including corrective maintenance items (900 to 600), late non-safety PMs (5600 to 900),
and late safety PMs (100 to 30).
The team reviewed each late safety-related PM and found that the licensee had adequately assessed each item for potential impact on system op'erability.
Each request for deferral of a safety related PM was found to require written justification with the deferral document approved by the Maintenance Manager.
One item associated with the Redundant Reactivity Control (RRC)
system was examined in detail including comparison with vendor manual recommendations.
The deferred PM (91-13-006) was found to be adequately justified based upon previous site specific information and industry experience.
4.0 ORGANIZATIONALEFFECTIVENESS 4.1 Site Area Emer enc Declaration and Emer enc Plan Im lementation Nl NRC's inspection activities of licensee's actions associated with the declaration of a Site Area Emergency (SAE) and implementation of the Niagara Mohawk Emergency Plan are documented in report Nos. 50-220/91-19 and 50-410/91-19.
Overall, the licensee's actions were found to be in accordance with the approved Emergency Action Limits (EALs).
4.2 Resolution of Previou I
Identified Problems Three examples of component problems occurred during the event which had previously been identified by the licensee but not corrected.
These problems involved the absence of UPS internal battery replacement, the oscillation of the RCIC system and a circuit design deficiency in the Essential Lighting System.
The RRAT conducted an onsite review of the licensee's efforts to resolve these items for adequacy in identification, prioritization and tracking.
The actions associated with the batteries for the non-safety UPS batteries were found to be adequate (see Section 3.1). The actions associated with the observed RCIC oscillation were also found to be adequate, although the team noted that the abserice of control room documentation of the oscillation was a weakness and prevented oversight by Operations management (see Section 3.2.2).
Essential lighting is a non-safety system.
Engineers documented in a November 1988 licensee problem report (PR 08220) that there was no lighting in some stairwells during a maintenance outage of the UPS.
It was noted that the backup (local battery) lights were powered from a different source than the UPS power.
Therefore, loss of UPS deenergized the lights but not the battery packs.
Since the battery packs did not lose external power, they did not illuminate their lanterns.
Similarly, loss of battery pack power would cause their lanterns to come on even though stairwell lighting was not los The licensee initiated the preparation of a modification to address the lighting design question in early.1989 (Mod ¹89-042 and 89-043). At the time of the August 13, 1991 event, the circuit change had become part of a larger modification involving UPS load reallocation.
This larger scope modification was planned for the refueling outage scheduled in early 1992.
The RRAT found that the impact of the lighting problems was minimal during the event.
Based on this minimal impact, the team 'found licensee prioritization ofthe modification to be adequate.
4.3 Emer enc Res nse Criti ue The NRC reviewed the licensee's post event investigations, as detailed in Report Nos. 50-220/91-19 and 50-410/91-19, and found them to be generally good.
Problems identified were corrected prior to restart.
Following the loss of annunciators and the reactor trip, the licensee'properly classified the event as a Site Area Emergency in accordance with NMPC emergency procedures.
Although not a corrective action for this event or required for restart, the licensee determined that a change to that classification was warranted, given the continuous availability of all safety grade control room instrumentation throughout the event.
Specifically, the licensee concluded that ifreactor pressure, vessel level, reactor power and containment pressure were available, then an Alert was the appropriate classification.
The licensee requested the change to their emergency procedures on September 11, 1991. The NRC reviewed the change and approved it on September 12, 1991.
5.0 CONCLUSION The team concluded that the corrective actions taken by the licensee for the problems identified as a result of the August 13, 1991 event were adequate.
Specifically, these actions included
'nhanced training on UPS operations; modification of the UPS control logic power supply; and, revision of procedures.
The team also noted that licensee management was actively involved
"in the overall maintenance program.
At the exit on September 16, 1991, the team stated their intention to recommend to NRC management that the licensee was ready to safely. restart Unit ATTACHMENT1 Restart Readiness Assessment Team Ins ection Plan