IR 05000277/2003007
| ML030720052 | |
| Person / Time | |
|---|---|
| Site: | Peach Bottom  |
| Issue date: | 03/13/2003 |
| From: | Blough A Division Reactor Projects I |
| To: | Skolds J Exelon Generation Co, Exelon Nuclear |
| References | |
| IR-03-007 | |
| Download: ML030720052 (26) | |
Text
rch 13, 2003
SUBJECT:
PEACH BOTTOM ATOMIC POWER STATION - NRC SPECIAL INSPECTION REPORT 50-277/03-07
Dear Mr. Skolds:
On January 30, 2003, the NRC completed a special inspection at the Peach Bottom Atomic Power Station to evaluate the circumstances related to the December 21, 2002, Unit 2 automatic reactor scram with loss of the normal heat removal path. The results of the NRC teams inspection were discussed on January 30, 2003, with Mr. Rusty West and other members of your staff. The enclosed report presents the results of the inspection.
The NRC team examined activities related to reactor safety and compliance with the Commissions rules and regulations, and with the conditions of your operating license. The inspectors reviewed selected procedures, examined representative records and equipment, interviewed personnel, and observed activities per the NRC teams charter (Attachment 3).
Based on the results of this inspection, the inspectors identified one issue of very low safety significance (Green). This issue was determined to involve a violation of NRC requirements.
However, because of its very low safety significance and because it has been entered into your corrective actions program, the NRC is treating this issue as a non-cited violation, in accordance with Section VI.A.1 of the NRCs Enforcement Policy. A violation of very low safety significance identified by Exelon is also listed in Section 4OA7 of this report. If you deny the non-cited violations noted in this report, you should provide a response with the basis for your denial, within 30 days of the date of this inspection report, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Peach Bottom facility.
In addition, this report discusses the NRC teams assessment of your staffs evaluation of the causal factors, corrective actions, equipment operability, human performance and associated risk significance of the Unit 2 automatic reactor scram. This assessment included reviews of the numerous equipment problems that complicated the reactor operators response to and recovery from the event. In these areas, the team concluded that your staffs immediate actions to identify the root causes of the reactor scram and equipment problems were generally acceptable.
Mr. John After performing an initial review of this automatic reactor scram with loss of normal heat removal, including degraded and unavailable equipment, we determined that the increased risk from this event was approximately 1.0E-6 (in terms of conditional core damage probability (CCDP)). After conducting more detailed analysis, we subsequently concluded that the CCDP for this event remained at the same order of magnitude at approximately 4.9E-6. This indicates that the risk associated with this event was low.
In accordance with 10 CFR 2.790 of the NRCs "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
A. Randolph Blough, Director Division of Reactor Projects Docket Nos.: 50-277 License Nos.: DPR-44
Enclosures:
1) NRC Inspection Report No. 50-277/03-07 2) Attachment (1) Supplemental Information 3) Attachment (2) Chronology of Events 4) Attachment (3) Peach Bottom Special Inspection Charter
Mr. John
REGION I==
Docket Nos: 50-277 License Nos: DPR-44 Report Nos: 50-277/03-07 Licensee: Exelon Generation Company, LLC Correspondence Control Desk 200 Exelon Way, KSA 1-N-1 Kennett Square, PA 19348 Facility: Peach Bottom Atomic Power Station Unit 2 Location: 1848 Lay Road Delta, Pennsylvania Inspection Period: January 13, 2003 through January 30, 2003 Inspectors: A. McMurtray, Senior Resident Inspector S. Hansell, Senior Resident Inspector-Susquehanna F. Arner, Senior Project Engineer E. Cobey, Senior Risk Analyst (in-office)
Approved by: Mohamed M. Shanbaky, Chief Projects Branch 4 Division of Reactor Projects
SUMMARY OF FINDINGS
IR 05000277-03-07; Exelon Generation Company; on 01/13-01/30/2003; Peach Bottom Atomic
Power Station; Unit 2. Special inspection of the December 21, 2002, Unit 2 automatic reactor scram with loss of the normal heat removal path due to a failed electro-hydraulic control (EHC)system card. Event Follow-up.
This inspection was conducted by two senior resident inspectors and a senior project engineer with support from a regional senior risk analyst. One finding of very low safety significance was identified during the inspection. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
A. Inspector Identified Findings
Cornerstone: Mitigating Systems
- Green.
The inspectors identified a non-cited violation (NCV) of very low safety significance (Green). The non-cited violation of Technical Specification (TS)3.5.3 is due to the inoperability of the Unit 2 reactor core isolation cooling (RCIC)pump in the automatic flow control mode since March 1994. In 1994, a modification to the RCIC pump flow controller was performed involving replacement of the controller and subsequent increase in the controller gain setting. This gain-set adjustment rendered the RCIC pump incapable, in automatic flow control, of delivering 600 gpm at reactor pressure, as required by TS 3.5.3.
This NCV was determined to be of very low safety significance. The flow rate for Unit 2 RCIC pump in the automatic mode, although degraded, was sufficient to meet the reactor decay heat requirements and provide make-up water to the reactor vessel during transient events. Additionally, the RCIC pump met design and licensing flow requirements with the pump flow controller in manual.
(Section 4OA3)
Licensee Identified Violations
A non-cited violation of very low significance, identified by Exelon, has been reviewed by the inspectors. Corrective actions, taken or planned by Exelon, have been entered into Exelons corrective action program. This NCV and corrective action tracking number are described in Section 4OA7.
ii
Report Details SUMMARY OF
PLANT STATUS
At approximately 8:35 p.m. on December 21, 2002, Unit 2 reactor automatically scrammed after all of the main steam isolation valves (MSIVs) closed due to a Group I primary containment isolation system (PCIS) actuation. The Group I PCIS occurred when reactor pressure decreased below 850 psig with the Mode switch in RUN, after the four main turbine control valves went full open and several main turbine bypass valves unexpectedly opened. The licensee determined that the cause of the event was a failure of a steam line resonance compensator (SLRC) circuit board in the electro-hydraulic control (EHC) system, causing the steam valves to open allowing the excessive steam flow and the consequent drop of reactor pressure to Group I PCIS set point.
Conditions Prior to the Event Unit 2 was in Mode 1 (Power Operation) and operating at 100 percent rated thermal power when the event occurred. There were no activities in progress related to main turbine bypass valves or EHC system. All plant parameters were normal and plant systems were operating as expected at the time of the event.
Event Summary Following the automatic reactor scram and Group I isolation, PCIS Group II and III isolations were received, as expected, when the low reactor vessel level (Level 3) (173" above the top of active fuel) setpoint was reached. The high pressure coolant injection (HPCI) and reactor core isolation cooling (RCIC) pumps automatically started and injected water into the vessel when the lo-lo reactor vessel level (Level 2) (124" above the top of active fuel) setpoint was reached.
Additionally, the alternate rod insertion system initiated on Level 2 and both reactor recirculation pumps tripped as designed. The main turbine tripped due to a main generator lock-out from a reverse power trip.
The HPCI and RCIC pumps restored reactor water level and the pumps automatically shut down, as designed, when level reached the high reactor vessel level (Level 8) setpoint. The RCIC pump was subsequently restarted to maintain reactor vessel level. During this restart, erratic operation of the RCIC pump flow was observed by operations personnel while operating in the automatic flow control mode. Due to this erratic operation, the flow controller was placed in manual. The pump operated properly in the manual mode. The RCIC pump was used for reactor vessel level control until reactor pressure was reduced below the condensate pump discharge pressure. Then, operations personnel used the C condensate pump to maintain vessel level. The operators had to use the 18 inch, main feedwater discharge line for supplying water to the vessel since both the 2C reactor feedwater pump, air-operated, discharge bypass level control valve, AO-2-06C-8091, and the 2C reactor feedwater pump discharge isolation, motor-operated valve for reactor start-up, MO-2-06C-8090, failed to open. The 10 inch, 2C reactor feedwater pump discharge bypass line containing AO-2-06C-8091 and MO-2-06C-8090 provides better flow control than the 18 inch main feedwater discharge line. The 2'C condensate pump remained in service until the unit was placed on shutdown cooling using the residual heat removal (RHR) system.
Reactor pressure was controlled using both manual actuation of the main steam relief valves and circulating the HPCI pump flow from the condensate storage tank (CST) to CST.
The operation crew response to the initial EHC failure was appropriate and performed as required by procedures. The control room operators had approximately 10 to 15 seconds to respond to the initial overhead annunciator alarms prior to the MSIVs closing and the automatic reactor scram. After the reactor scram, the operators entered the applicable emergency operating procedures (EOPs) to control reactor power, level, and pressure. The primary containment EOP was also entered due to the safety relief valves (SRVs), HPCI, and RCIC steam addition to the torus water. The transition to RHR shutdown cooling mode was performed without complications. The plant operating procedures provided clear direction for the operators in response to the observed plant conditions and equipment malfunctions.
The crew initially missed the occurrence of an excessive reactor vessel metal temperature cooldown rate. The reactor coolant system (RCS) cooldown rate is required to be less than 100 degrees Fahrenheit per hour (°F/hr) per Technical Specifications (TS). The reactor operator documenting the cooldown data did not evaluate one of the five parameters in the surveillance test procedure and therefore did not inform the shift manager that a metal temperature reading exceeded the TS allowed value. Later in the shift, the operators recognized the error. The greatest reactor vessel bottom head metal temperature cooldown rate was 120 °F/hr. This licensee identified violation is described in Section 4OA7.
Several hours after the automatic reactor scram, the reactor water clean-up (RWCU) system received an automatic isolation due to a non-regenerative heat exchanger outlet high temperature condition. At the time of this isolation, the RWCU system was removing water from the reactor vessel and discharging to the main condenser. The isolation was due to the failure of CV-2-12-4157 to maintain RWCU flow below the system cooling limit of 180 gallons per minute (gpm). Flow oscillations were in excess of 200 gpm. The isolation of the RWCU system limited the operators ability to reduce reactor vessel water inventory to maintain reactor vessel water level in the proper band. The RWCU temperatures were also needed to verify that the reactor recirculation pump parameters were within required TS limits prior to re-start. The system isolation occurred prior to the re-start of the reactor recirculation pumps and delayed the restoration of forced water flow in the reactor vessel.
The operators follow-up actions after the reactor scram were good, especially considering the higher than normal number of equipment deficiencies that complicated their ability to stabilize the plant in a shutdown condition. The most notable equipment complications were:
- the RCIC flow oscillations when the system was controlled in automatic, which resulted in a delay in re-setting the reactor scram due to difficulty in maintaining reactor water level stable
- the inability to re-open the MSIVs due to three main turbine bypass valves that failed open, which resulted in the inability to use the condenser as a heat sink
- 2'C reactor feedwater pump discharge bypass valves, AO-2-06C-8091 and MO-2-06C-8090, failed to open, which required the use of the 2'C main feedwater discharge line while supplying water to the reactor vessel with the 2'C condensate pump. This condition prevented better flow control of the 2C condensate pump and challenged the reactor operators as they maintained the reactor vessel at the required water level.
- an RWCU system isolation on high temperature due to a known degraded dump control valve, which limited the ability of the operators to reduce reactor water level to within the proper band and delayed restarting the reactor recirculation pumps.
In addition to the failure of the A SLRC card that caused the event and the equipment challenges noted above, the following degraded equipment and equipment failures complicated the plant operators response and recovery from this event:
1. Reactor building ventilation exhaust damper, AO-2-40B-20463, failed to automatically
stroke closed on the Group 3 PCIS signal. Operations personnel were able to close the damper using the specific damper control switch in the control room. This is a secondary containment isolation valve. Ventilation damper problems, including this damper, were documented in NRC Inspection Report 50-277/02-06, 50-278/02-06.
These ventilation damper problems were due to the inadequacies in preventive maintenance activities and procedures on safety-related ventilation dampers and resulted in a Green finding. Exelon entered this issue into their corrective action program as Condition Report (CR) # 137759.
2. Three of the nine turbine bypass valves failed to close after the 2'A EHC pump was shut
down. After the scram, all of the turbine bypass valves opened. Operations personnel shut off the EHC pumps to attempt to close the bypass valves so that the MSIVs could be re-opened to re-establish the normal heat removal flow path (i.e. condenser).
3. A degraded dump control valve, CV-2-12-4157, in the RWCU system caused an
isolation of the RWCU system.
4. Safety relief valve, (SRV) 71F, was declared inoperable due to the SRV tailpipe vacuum
relief valve, VRV-2-02-8096F, indicating full open. The vacuum relief valves prevent the condensation of steam in the tailpiece from drawing a vacuum and drawing water from the torus up into the tailpiece following cycling of the SRV. If a vacuum relief valve is not closed when the SRV opens, steam will bypass the torus where the steam is normally quenched and will discharge directly into the drywell causing an increase in drywell temperature and pressure.
5. Wide range neutron monitoring C and G channels were declared inoperable due to a
failed K5A relay. The wide range neutron monitoring system provides operations personnel with information relative to the neutron flux level at very low reactor power levels in the core. All other channels were operable during the event.
6. The 2'A EHC pump was secured due to EHC fluid leaking from a crack at a welded
fitting in a piece of tubing that runs from the pump compensator to the pump.
7. The 2'B EHC pump tripped on thermal overload during pump start. Operations
personnel reset the thermal overload and successfully restarted the pump.
8. The SRV acoustic position indicators, SRV POS-071C and 071H, were declared
inoperable due to cycling of the indications at reactor pressure below 450 psig. Each SRV has both an acoustic indicator and a thermocouple that indicate whether or not the relief valve is open. The thermocouples for both SRV 71C and SRV 71H remained operable throughout this event. Engineering personnel determined that the SRV acoustic position indicators should remain operable down to a reactor pressure of 150 psig.
Exelon made an associated four-hour 10 CFR 50.72 notification (EN No. 39466) for this event, to the NRC Operations Center, early Sunday morning, December 22, 2002. A complete chronology of the event is provided in Attachment 2.
Subsequent to the scram, Exelon developed a troubleshooting plan to determine the cause of the EHC malfunction which had resulted in the spurious opening of the main turbine control valves and several of the bypass valves. As-found voltages measured on the A SLRC circuit card indicated that the card was not operating properly. The card was replaced and tested in order to return the EHC system to service.
OTHER ACTIVITIES
[OA]
4OA3 Event Follow-up
.1 Review of Equipment and Human Performance Issues Associated with this Event
a. Inspection Scope
This inspection was conducted in accordance with NRC IP 93812, Special Inspection, to assess Exelons actions associated with the equipment and human performance issues that occurred during the December 21, 2002, Unit 2 automatic reactor scram with loss of the normal heat removal path. Exelon conducted an initial prompt investigation of this event. Exelon also performed a root cause analysis to determine the cause and contributing causes of the automatic reactor scram, equipment challenges and operator performance deficiencies that occurred during this event. General Electric personnel were consulted for determining the cause of the scram and some of the subsequent equipment problems. The inspection team reviewed the associated design basis documents, Technical Specifications, Technical Requirements Manual, the Updated Final Safety Analysis Report, General Electric evaluations and analysis, test procedures and data, vendor manuals, engineering evaluations and Condition Reports (CRs). The team also reviewed operator logs, primary plant computer data, operations procedures, including Off Normal and Emergency Operating Procedures and selected maintenance procedures, work requests and other associated documents, including Action Requests (A/Rs). A list of the documents reviewed by the team is provided in Attachment 1.
The team interviewed engineers, maintenance technicians, members of the root cause investigation and operations personnel, including members of the crew that were on shift during this event. The team observed the initial plant response at the Peach Bottom control room simulator. The team also inspected several of the components that contributed to the equipment challenges during this event.
The team reviewed computer data associated with main turbine control valve and bypass valve positions, EHC system pressure, and reactor pressure to analyze the response of the system during the transient. Additionally, interviews were held with the associated personnel involved in the EHC troubleshooting subsequent to the event. The team reviewed the EHC system design to ensure that the backup, B pressure control loop functioned in accordance with the design and would not have prevented the transient for this particular failure mechanism.
b. Findings
Inoperability of the Unit 2 Reactor Core Isolation Cooling (RCIC) Pump in the Automatic Flow Control Mode Introduction The inspectors identified a non-cited violation of very low safety significance (Green).
The non-cited violation of Technical Specification (TS) 3.5.3 is due to the inoperability of the Unit 2 reactor core isolation cooling (RCIC) pump in the automatic flow control mode since March 1994. The Unit 2 RCIC pump would not deliver the Technical Specification required flow rate (i.e. a minimum of 600 gpm) into the reactor vessel, at normal reactor pressure (approximately 1030 psig), while operating in the automatic flow control mode.
Description In March 1994, station personnel increased the gain setting on the Unit 2 RCIC pump flow controller during a Modification Acceptance Test (MAT) following replacement of the controller. This adjustment was made to improve stability of pump flow while testing the pump with flow from the CST to the CST. The pump was never tested with flow from the CST to the vessel, nor was an adequate engineering assessment performed to ensure that the pump delivered the required flow to the reactor vessel, in automatic, after the gain-set adjustment.
During the scram on December 21, 2002, the RCIC pump had flow rate swings between approximately 200 gpm and 700 gpm, with a nominal flow rate of approximately 500 gpm, while the controller was operating in the automatic mode. The normal flow rate from this pump is 600 gpm at normal reactor operating pressure. The operators placed the controller in manual immediately after the flow swings were observed and continued to feed in manual, to maintain reactor vessel level, until the unit was placed in cold shutdown. The RCIC pump did automatically start and inject initially following reactor vessel level reaching the lo-lo setpoint.
Analysis The inspectors determined that the Unit 2 RCIC pump was inoperable since March 1994, because the pump was incapable, in the automatic flow control mode, of delivering 600 gpm at reactor pressure, as required by TSs. The failure to adequately verify or check the RCIC system response, using calculational methods or suitable testing, after the flow controller gain-set was increased in 1994, is a performance deficiency. The gain-set adjustment rendered the pump inoperable in the automatic flow control mode during reactor vessel injection. Traditional enforcement does not apply for this issue because it did not have any actual safety consequences or the potential for impacting the NRCs regulatory function and was not the result of any willful violations of NRC requirements.
This finding was considered more than minor because it was associated with the design control attribute of the Mitigating Systems cornerstone to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences. This finding was determined to be of very low safety significance (Green) using Phase 1 of the Significance Determination Process (SDP) for Reactor Inspection Findings for At-Power Situations. This issue was of very low safety significance because Unit 2 RCIC pump flow was high enough (i.e. a nominal flow rate of approximately 500 gpm), in the automatic flow control mode, even with the swings in flow rate, to maintain reactor vessel water level. Additionally, the RCIC pump met the design basis flow with the RCIC flow controller in manual. Exelon entered this issue into their corrective action program as Condition Report (CR) # 137771.
Enforcement Technical Specification 3.5.3, RCIC System, requires that the RCIC system shall be operable or the RCIC system must be restored to operable within 14 days or the unit shall be placed in Hot Shutdown (Mode 3) within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. Contrary to this requirement, the RCIC system was inoperable since March 1994 because the RCIC pump could not meet required design flow (i.e. a minimum of 600 gpm) into the reactor vessel, at normal reactor pressure, with the flow controller in automatic. This violation of Technical Specification 3.5.3 is being treated as a non-cited violation consistent with Section VI.A.1 of the NRC Enforcement Policy. (NCV 50-277/03-07-01)
.2 Event Causal Factors, Root Causes and Corrective Actions
a. Inspection Scope
The team assessed the adequacy of Exelons activities to determine the root cause of the Unit 2 automatic reactor scram. The team also independently assessed the causal factors for the event, equipment challenges and operator performance deficiencies, and the appropriateness of Exelons initial corrective actions, including extent of condition reviews for the equipment and operator performance issues. The team evaluated whether or not there were any common causes for the reactor scram or subsequent equipment performance challenges and assessed Exelons overall response to the event. The team reviewed data, control room instrument recorder charts, procedures, corrective action documents and work requests. The team conducted plant tours and interviewed site personnel, including station management. The team also observed the initial plant response at the Peach Bottom control room simulator.
b. Findings
No findings of significance were identified. However, the team identified several observations, which are discussed in this report because these observations relate to the quality of Exelons root cause investigation and associated corrective actions.
The team determined that Exelon appropriately investigated and determined the cause of the Unit 2 automatic reactor scram. Exelon formed a large event response team composed of technically diverse individuals, and provided the team with significant management involvement/oversight. The team also noted that the post scram response to equipment and human performance issues and corrective actions, including extent of condition reviews were generally thorough and insightful; however, it was too early to determine the effectiveness of the corrective actions.
During their root cause review of the event, Exelon determined that the A SLRC card had an operational amplifier with an unknown manufacturing defect which caused the EHC failure and plant transient. The A SLRC card was installed during the Unit 2, September 2002, refueling outage and placed in-service in the EHC system approximately three months prior to the failure. Exelon determined that the component which had failed had similar characteristics to several previous failures of SLRC cards identified in a recent vendor analyses study of industry data. However, previous failures had been identified during calibration of the cards prior to placing them in service, suggesting they were all infant mortality failures.
The team determined that the preliminary root cause analysis associated with the EHC card failure was thorough and investigated the underlying reasons surrounding the installation of the A SLRC card with a defective component. Additionally, Exelons root cause examined why a condition report had not been generated at the time of failure of a similar component associated with the B SLRC card that was also installed and replaced during the recent Unit 2 refueling outage.
Although the team concluded that the failure to initiate a condition report did not directly result in this plant event, the team noted this failure resulted in potential missed opportunities to formally evaluate failure analyses results and ensure appropriate individuals were informed and involved in determining corrective actions.
Overall, the team concluded that Exelons Event and Root Cause Analysis Review was an acceptable effort that provided proper focus and detail on investigation details. The team found that Exelon identified the appropriate root and contributing causes, and implemented appropriate immediate corrective actions. The investigation also assessed the extent of condition for the equipment problems at both Units 2 and 3.
Notwithstanding, the team identified some weaknesses associated with Exelons efforts:
- In addition to not writing a CR for the failure of the B steam line resonance compensation card in the EHC system during the 2R14 outage, the inspectors identified the following examples of not initiating CRs for degraded equipment or equipment failures:
Trip of the 2'B EHC pump on thermal overload during pump start.
(Occurred during the event)
Failure of the K5A relay for the wide range neutron monitoring C and G channels. (Occurred during the event)
Leakage and erratic operation of the RWCU dump valve to the, CV-2-12-4157 coming out of the 2R14 refueling outage. (October 2002)
Inoperability of the 3'B EHC pump due to EHC fluid leaking from a crack at a welded fitting in a piece of tubing that runs from the pump compensator to the pump. (May 2002)
SRV vacuum relief valves position switch replacements (three: VRV-3-02-9096C, L and J) due to failures identified during the 3R13 refueling outage. (September 2001)
Although not initiating a CR for each of these items, as required by LS-AA-125, represented performance deficiencies; none of these issues were greater than minor since they were not a precursor to a significant event, would not become a more significant safety concern if left uncorrected. Exelon initiated several CRs documenting the failure of station personnel to write CRs as required by LS-AA-125.
- Additionally, the team identified the following issues which although minor were considered weaknesses by the team. These issues had not been identified by station personnel during the determination of causes and development of the corrective actions for equipment problems associated with this event:
CR # 137789 did not identify or evaluate that two of the three main turbine bypass valves that failed to close, had actuator degradation noted and accepted as is, during valve maintenance activities in November 1998. Also, the CR did not identify or evaluate why action was not taken to address the request for preventive maintenance on the main turbine bypass valve actuators noted in the October 2000, A/R # A1282275.
2. CR # 137757 did not identify or evaluate that the current to pneumatic
transducer, I/P-8091, to the controller for 2'C reactor feedwater pump discharge bypass level control valve, AO-2-06C-8091, was noted to be grossly out-of- calibration during maintenance activities. Transducer, I/P-8091, was found to be out-of-calibration after the scram. Station personnel concluded that this out-of-calibration transducer contributed to the failure of AO-2-06C-8091 to open.
CR # 137744 did not identify or evaluate that maintenance personnel did not order a replacement position indicator switch for vacuum relief valve, VRV-2-02-8096E, in a timely manner, after this switch was identified as degraded in October 2002. If maintenance personnel had ordered this switch earlier, it would likely have been available to support repair on one of the two degraded switches for valves, VRV-2-02-8096E or VRV-2-02-8096F, identified by operations personnel during the scram.
CR # 137762 did not identify or evaluate why additional instructions in vendor manual E-113-2, for the NDT acoustic monitoring system, were not performed while checking the calibration of the two acoustic position indicators, SRV POS-071C and 071H. These additional checks included checking the setting on the delay pot for this system and performing a check of the sensitivity of the detectors. These position indicators were initially declared inoperable due to cycling of the indications at reactor pressure below 450 psig. After maintenance personnel performed calibrations on these indicators per SI2M-2-71-ALC2, the indicators were declared operable. These calibrations did not include the additional checks noted above. After the unit had restarted and reached full power, operations and engineering personnel declared SRV POS-071C and 071H inoperable until the additional checks noted above were performed.
The corrective actions for CR # 137738 required engineering personnel to evaluate the population of non-safety related, 480 volt motor control units for motor operated valves with approximately 1015, potentially defective, thermal overload relays. Engineering personnel were to look at valves that may impact power operation or the ability of operations personnel to respond to or recover from transient conditions if the valve failed to operate. The inspectors noted that engineering personnel did not include risk significance when they prioritized thermal overload relay replacement. For example the corrective action did not include using a list of risk reduction worth (RRW) or risk achievement worth (RAW)components and any other risk insights. The station has been replacing older thermal overload relays during preventive maintenance activities on safety-related motor operated valves since May 1996 after an engineering evaluation, ECR PB 96-04398-000, was issued. This engineering evaluation noted that Cutler-Hammer thermal overload relays, that were manufactured prior to certain dates, were defective.
The team identified a few equipment performance issues that challenged the operators response and recovery during this event that were related to the effectiveness and completeness of maintenance activities during the recent Unit 2 refueling outage.
However, the inspectors were more concerned that maintenance personnel were generally not writing CRs, when required, for components that were found degraded or to have failed during maintenance activities. This prevented the station from identifying trends with equipment performance and was a contributing factor to the degradation of several of the pieces of equipment identified above.
The team presented these observations to Peach Bottom management and staff at the exit meeting on January 30, 2003. Peach Bottom management noted that their Event and Root Cause Analysis Review had not yet been finalized, and that the observations presented would be factored into the stations continuing evaluation and associated report.
.3 Risk Significance of the Event
a. Inspection Scope
The team conducted an initiating event assessment and concluded that the risk of this event was low. This risk evaluation was based upon the following assumptions:
- The NRCs standardized plant analysis risk (SPAR) model for the Peach Bottom facility was used for this analysis.
- The SPAR model was revised to account for the maintenance configuration of plant equipment. At the time of the event, all mitigating equipment, except the 2D high pressure service water pump, was available.
- The SPAR model adequately accounted for the equipment problems that complicated the operators response to the event.
- Recovery of the main condenser late in the event was credited.
b. Findings
The dominant accident sequence for this event involves the failure of the high pressure coolant injection system, failure of the reactor core isolation cooling system, and failure of the operators to manually depressurize the reactor to allow injection using available low pressure sources. The team concluded that the conditional core damage probability (CCDP) for this event was approximately 4.9E-6. This indicates that the risk associated with this event was low.
4OA6 Meetings
.1 Exit Meeting Summary
The inspectors presented the results of this special inspection to Mr. Rusty West and members of Exelons management on January 30, 2003. Exelon management acknowledged the findings presented. No proprietary information was identified.
4OA7 Licensee Identified Non-Compliance
The following violation of very low safety significance (Green) was identified by Exelon and is a violation of NRC requirements which met the criteria of Section VI of the NRC Enforcement Policy, NUREG-1600, for being dispositioned as a Non-Cited Violation.
.1 Technical Specification Reactor Vessel Metal Temperature Cooldown Rate Exceeded
Technical Specification (TS) 3.4.9 requires that the reactor coolant system (RCS)cooldown rate shall be less than 100 degrees Fahrenheit per hour (°F/hr). Exelon surveillance test, ST-O-080-500-2, Recording and Monitoring Reactor Vessel Temperatures and Pressures, provides procedure direction to ensure compliance with the TS limit. Contrary to TS 3.4.9 requirements, on December 21, 2002, plant operators determined that the reactor vessel bottom head metal temperature cooldown rate was 120 °F/hr. Even though the TS cooldown limit was exceeded, the magnitude of the cooldown rate did not result in exceeding the TS reactor vessel brittle fracture temperature and pressure limits, and therefore, did not result in an increased probability that a loss of coolant accident would occur. This violation is considered to have very low safety significance, and is being treated as a non-cited violation. This issue was addressed by various corrective actions and was entered into Exelons corrective action process (CR # 137136). (See Section 4OA3)
ATTACHMENT 1
SUPPLEMENTAL INFORMATION
a. Key Points of Contact
Exelon Generation Company
- R. West, Vice-President
- G. Johnson, Plant Manager
- B. Hanson, Operations Director
- P. Davison, Maintenance Director
- E. Eilola, Acting Site Engineering Director
- M. Anthony, Work Management Director
- C. Behrend, Senior Manager Plant Engineering
- S. Beck, Acting Regulatory Assurance Manager
- D. Falcone, Acting Shift Operations Superintendent
- R. Truax, Operations Services Manager
- D. Foss, Senior Regulatory Engineer
- J. Felice, Senior Corrective Action Coordinator
b. List of Items Opened, Closed, and Discussed
Closed
None
Opened/Closed
50-277/03-07-01 NCV Reactor Core Isolation Cooling Pump Inoperable in
the Automatic Flow Control Mode Since 1994
c. List of Documents Reviewed
Engineering Documents
ECR # PB 92-00208-000 Acoustic monitoring system for SRVs position indication
ECR # PB 93-03983-001 Replace RCIC flow controller, FC-2-13-091
ECR # PB 96-04398-000 Thermal overload relays date codes
Design Basis Document: P-S-39, Rev. 13, Reactor Core Isolation Cooling (RCIC)
Design Basis Document: P-S-45, Rev. 15, Main Steam, Turbine and Extraction
Steam Systems
Design Input Document: MOD P00239, Rev. 0, HPCI / RCIC flow controllers
modification
(contd) 13
Drawings
280-M-308, Rev. 54 Unit 2 Piping and Instrumentation Diagram, Feedwater and
Feed Pumps
280-M-338, Rev. 14 Unit 2 Piping and Instrumentation Diagram, Main Turbine
EHC System
280-M-351, Rev. 71 Unit 2 Piping and Instrumentation Diagram, Nuclear Boiler
280-M-391, Rev. 32 Unit 2 Piping and Instrumentation Diagram, Primary and
Secondary Containment Isolation Control Diagram
M-1-S-25, Rev. 57 Electrical Schematic Diagram, Unit 2 Feedwater Control
System
Updated Final Safety Analysis Report Sections
Section 4.4 Nuclear System Pressure Relief System
Section 4.7 RCIC System
Section 5.3 Secondary Containment System
Section 7.10 Feedwater Control System
Section 11.5 Turbine Bypass System
Exelon Procedures
ARC-206 20C208R A-3 Maximum Combined Flow Limit in Control
ARC-206 20C208R A-4 Main Steam Line Bypass Valve Open
MA-AA-716-013, Rev. 0 Rework Reduction
M-C-701-005, Rev. 9 Main Turbine Bypass Valve Inspection and Maintenance
IC-11-00497-2, Rev. 5 Alignment Procedure for the Unit 2 EHC System of the
General Electric Turbine Generator
GP-18 COL, Rev. 35 Scram Review Procedure Check List
LS-AA-125, Rev. 4 Corrective Action Program (CAP) Procedure
ON-139-001, Rev. 10 Turbine EHC System Malfunction
OP-AA-102-103, Rev. 0 Operator Work-Around Program
OT-110, Rev. 6 Reactor High Level - Procedure
OT-111, Rev. 2 Reactor Low Pressure - Procedure and Bases
SI2M-2-71-ALC2, Rev. 5 Calibration Check of Unit 2 Main Steam Relief Valve
Position Switches POS 2-02-071A-L, 2-02-070A-B
SO-12.1.A-2, Rev. 30 Reactor Water Cleanup System Startup for Normal
Operations or Reactor Vessel Level Control
SO-12.2.A-2, Rev. 14 Reactor Water Cleanup System Shutdown
ST-O-080-500-2, Rev. 10 Recording and Monitoring Reactor Vessel Temperatures
and Pressure
ST-O-013-301-2, Rev. 23 RCIC Pump, Valve, Flow and Unit Cooler Functional and
In-Service Test
ST-O-013-200-2, Rev.14 RCIC Flow Rate at < 175 psig
T-101, Rev. 17 RPV Control - Procedure/Bases
(contd) 14
Vendor Manuals
280-E-113-22, Rev. 1 Operation and Technical Manual for the NDT International
Inc. Fluid Flow Detection System
PB SDOC M-2-371 General Electric Turbine-Generator Service Manual,
Maintenance Manual, GEK 5595, Volume IIA
Work Orders
R0029220 Perform Preventive Maintenance on Motor Control Unit for
Reactor Feedwater Pump C Discharge Start-up Valve,
MO8090
R0271051 Perform Preventive Maintenance on Motor Control Unit for
Reactor Feedwater Pump C Discharge Start-up Valve,
MO8090
R0490878 Inspect Steam Side #2 Bypass Valve
R0492550 Disassemble / Inspect Steam Side #8 Bypass Valve
R0607484 #6 Bypass Valve Disassemble and Inspect
R0844795 Calibrate LIC-8091 and Loop to AO-8091
R0849882 EHC Alignment/Filter Replacement
R0884786 Main Steam Relief Valve Tailpiece Vacuum Relief Valve
Position Indicator In-service Test
C0143877 Replace FC-2-13-091
C0198227 Repair Leaking Bonnet Joint on AO-2-06C-8091, 2'C
Reactor Feedwater Pump Discharge Control Valve for
Reactor Start-up
C0201516 Replace the Tube Sub-assembly on the 3'B EHC Pump
C0203492 Adjust or Replace the Position Switch, as required, for the
Vacuum Relief Valve, VRV-2-02-8096E, for Main Steam
Relief Valve E
C0203568 Replace Tube Assembly on the 2'A EHC Pump
C0203605 Adjust or Replace the Position Switch, as required, for the
Vacuum Relief Valve, VRV-2-02-8096F, for Main Steam
Relief Valve F
Action Requests (A/Rs)
A0005618 10CFR21 report concerning defects and noncompliances
involving Eaton Corporation / Cutler Hammer motor
starters and overload relays
A0798125 Failure of Cutler Hammer overload relays during
preventive maintenance tests
A1282275 Request for PMs on turbine bypass valve actuators
A1363898 2R14 EHC critical card replacement per Plant Material
Condition Excellence Initiative (PMCEI) report
(contd) 15
A1388965 2'E main steam relief valve discharge line vacuum relief
valve, VRV-2-02-8096E, open annunciator alarm coming
in without indication of valve open and operability
determination for VRV-2-02-8096E(F)
A1389415 RWCU dump valve, CV-2-12-4157, not properly controlling
flow to the main condenser
A1397816 Unit 2 RCIC flow controller will not operate in automatic
A1397819 C reactor feedwater pump discharge bypass level
controller, LIC-8091, failed to open following the Unit 2
reactor scram
A1397822 Operability evaluation - Unit 2 reactor vessel cooldown rate
exceeded
A1397818 Unit 2 main steam line H relief valve acoustic position
indicator, POS-2-02-071H, cycled during blowdown
A1397820 Unit 2 main steam line C relief valve acoustic position
indicator, POS-2-02-071C, cycled during blowdown
A1397821 2'C reactor feedwater pump discharge isolation valve for
reactor start-up, MO-2-06C-8090, tripped on thermal
overload following the Unit 2 scram
A1397829 Unit 2 RWCU temperature switch, TS-2-12-099, possibly
out-of-calibration
A1397889 Vacuum relief valve, VRV-2-02-8096F, indicating full open
A1397893 Unit 2 wide range neutron monitor - channels C and G
inoperable due to a failed K5A relay
A1397901 2'A EHC pump had to be secured due to an EHC line
breaking at a fitting at the pump
Condition Reports
00137110 Turbine bypass valves open due to failed EHC card
causing reactor scram
00137136 Reactor cooldown rate exceeded 100 F/hr. post reactor
00137149 Unexpected 2'A EHC pump tubing equipment failure
00137621 Reactor water cleanup isolation
00137734 T-103 Entry, MS tunnel hi temp alarm
00137738 Unit 2 scram: Feedwater MO-8090 tripped on thermal
overload after scram
00137744 Unit 2 scram: POS-8096E vacuum relief valve open alarm
without indication
00137757 Unit 2 scram: AO-8091corrective maintenance unexpected
failure causes OT-110 < 600 psig
00137759 Unit 2 scram: SV-20463 slow to operate and unplanned
TS action entry
00137762 Unit 2 scram: SRV POS-071'C and H acoustic position
indicator problems
00137771 Unit 2 scram: RCIC pump flow oscillations in auto control
00137789 Unit 2 scram: #2, 6 and 8 turbine bypass valves failed to
close
(contd) 16
00140311 CR not initiated for defective EHC circuit board
Other
Exelon Turbine Controls Six Sigma Pilot Project - Turbine Control System Reliability
Improvement, December, 2001
Unit 2 Archive Data - H054, H068 RCIC Flow-Dated December 21, 2002
Modification Acceptance Test: P00239A, Rev. 0, Unit 2 RCIC Flow Controller
Replacement, FC-2-13-091"
Degraded Equipment Log 1/16/2003
Licensee Event Report (LER) # 2-89-015, "Malfunctioning EHC System Component
Causes Reactor Scram When Removed From Service"
Control Room Operator Log from 12/21/2002 thru 12/22/2002
Plant Operations Review Committee Meeting Minutes No. 02-38, 12/23/02
d. List of Acronyms
CCDP conditional core damage probability
CR condition report
CST condensate storage tank
ECR engineering change request
EHC electro-hydraulic control
GPM gallons per minute
HPCI high pressure coolant injection
MSIV main steam isolation valve
NCV non-cited violation
PCIS primary containment isolation system
RCIC reactor core isolation cooling
RWCU reactor water clean-up
SDP Significance Determination Process
SLRC steam line resonance compensator
SPAR simplified plant analysis risk
TCV turbine control valve
TS Technical Specification
VRV vacuum relief valve
CHRONOLOGY OF THE EVENT
Saturday, December 21, 2002:
~2035: - Main steam line low pressure alarm
- Emergency operating procedure (EOP) T-101 entered for reactor pressure
vessel (RPV) water level and pressure control
- EOP T-102 entered for primary containment control
~20:35:16 Total main turbine control valve (TCV) position 100%
~20:35:33 Total main turbine bypass valve position 55% (limited by EHC "Max Combined
Flow)
~20:35:39 Main steam line low pressure MSIV closure signal (Group 1 isolation) due to
main steam line pressure less than 850 psig. with Mode switch in RUN
~20:35:40 Reactor automatic scram due to MSIV closure, all control rods inserted
~20:35:43 Main steam isolation valves stroke full closed
~20:35:43.5 Reactor mode switch in Shutdown position (operator manual reactor scram
signal)
~20:35:44 Reactor water cleanup (RWCU), residual heat removal (RHR), feedwater and
other miscellaneous systems isolate on Low (Level 3) reactor water level (Group
signal)
~20:35:44 Valves and dampers associated with ventilation systems isolate and the standby
gas treatment (SBGT) system starts on Low (Level 3) reactor water level (Group
signal)
~20:35:44 Reactor water level Lo-Lo (Level 2) signal (high pressure coolant injection
(HPCI) and reactor core isolation cooling (RCIC) initiation signal). HPCI and
RCIC automatically started and injected water into the reactor pressure vessel
~20:35:45 "A" and "B" reactor recirculation pumps trip due to alternate rod insertion (ARI)
Lo-Lo (Level 2) reactor water level signal
~20:35:52 Total main turbine control valve position 0%
~20:36 Reactor building ventilation exhaust damper, AO-2-40B-20463, failed to
automatically stroke closed on the Group 3 isolation signal. This is a secondary
containment isolation valve. Valve was taken to closed position manually using
the damper control switch in the control room.
~20:37:08 HPCI and RCIC isolated on high reactor water level (Level 8)
~20:45 RCIC controller placed in manual due to flow oscillations in the automatic mode
of operation.
~21:00 HPCI placed in the reactor pressure control mode of operation
~21:05 Commenced normal reactor cooldown to achieve cold shutdown (less than 200
degrees F)
(contd) 18
~21:10 Entered EOP T-103 due to main steam line tunnel high temperature
~21:48 Primary containment isolation system (PCIS) Group 1, 2, & 3 isolations were
reset. However, main steam isolation valves were not reopened due to three
main turbine bypass valves failing to shut after the 2'A electro-hydraulic (EHC)
pump was secured.
~21:55 Reactor scram signal was reset
~22:05 Exceeded Technical Specification 100 degree F/hour cooldown rate limit on the
reactor vessel bottom head metal temperature
~22:34 RPV pressure reached condensate discharge pressure, HPCI and RCIC
secured, and subsequent pressure reduction conducted with main steam safety
relief valves (SRVs). RPV water level controlled with condensate.
~22:45 2'Creactor feedwater pump discharge bypass level control valve, AO-2-06C-
8091, failed to open
~23:22 2'A and 2'B condensate pumps shut down
~23:45 SRV 71H acoustic position indicator lights cycling on and off
Sunday, December 22, 2002:
~00:18 NRC prompt notifications completed for the emergency core cooling system
(ECCS) discharge into the reactor vessel, reactor protection system (RPS)
actuation, and PCIS Group 1, 2, & 3 isolations
~01:00 After shutting the 2'C reactor feedwater pump discharge isolation valve for
reactor start-up, MO-2-06C-8090, to facilitate troubleshooting on AO-2-06C-
8091, the motor operator for MO-2-06C-8090 tripped on thermal overload while
attempting to reopen the valve
~01:15 SRV 71C acoustic position indicator lights cycling on and off
~01:33 Verified that the generator lockout was from the directional power device
(reverse power trip). Directional power device was reset
~02:10 Reactor water cleanup (RWCU) system placed in service to gravity drain water
from the reactor vessel to the main
~02:20 RWCU system isolated automatically due to non-regenerative heat exchanger
high temperature (temperature reached 250 degrees F). This resulted from a
leaking RWCU dump valve to the condensor (CV-2-12-4157). This valve was
identified as leaking in October 2002 following the 2R14 refueling outage.
~03:07 Main turbine shut down
(contd) 19
~04:00 Placed reactor building ventilation on SGBT to allow repair of damper, AO-2-
~05:00 Residual heat removal system placed in shutdown cooling mode of operation
~06:05 Unit 2 reactor reached cold shutdown (mode 4)
~07:00 Safety relief valve, SRV 71F, declared inoperable, due to the SRV tailpipe
vacuum relief valve, VRV-2-02-8096F, indicating full open
~10:12 Shutdown SGBT system and place reactor building ventilation in service
~14:39 RWCU system placed in service to letdown reactor water level to the condensate
storage tank. The RWCU control valve to the main (CV-4157) was stuck closed
(AR #1389415)
~16:15 Wide range neutron monitoring C and G channels declared inoperable due to a
failed K5A relay.
~19:14 Failed K5A relay replaced. Wide range neutron monitoring C and G channels
declared operable
~21:10 2'A EHC pump started to support troubleshooting on the two remaining open
main turbine bypass valves. EHC fluid is leaking from a crack at a welded fitting
in a piece of tubing that goes from the pump compensator to the pump. (AR
- A1397901) The 2'A EHC pump is immediately secured. The two bypass
valves closed with EHC pressure
~22:43 Attempted to start the 2'B EHC pump. Pump tripped on thermal overload during
start. Shift management directed that the thermal overload be reset and an amp
meter installed at the pump breaker. Attempted to restart the 2'B pump. Pump
started. Amp meter pegged initially, then settled at 50 amps. Pump ran
satisfactorily.
Peach Bottom Special Inspection Charter
Peach Bottom Atomic Power Station Unit No. 2
Automatic Reactor Scram with Loss of Normal Heat Sink - With Equipment and Potential
Human
Performance Problems
The objectives of the inspection are to determine the facts surrounding the automatic reactor scram with loss of normal heat sink that occurred at Peach Bottom Atomic Power Station Unit 2
on December 21, 2002. Specifically the inspection should:
b. Independently evaluate the equipment and human performance issues to assess the
adequacy of the scope of Exelons investigation.
c. Independently evaluate the quality and implementation of Off Normal Procedures and
Emergency Operating Procedures.
d. Independently evaluate the risk significance of the event.
e. Assess the adequacy of Exelons investigation and root cause evaluation of the
circumstances surrounding the cause of the automatic reactor scram.
f. Assess the adequacy of Exelons investigation and root cause evaluation regarding post
scram response from the perspective of the equipment and human performance.
g. Assess the adequacy of Exelons plans for corrective actions and extent of condition
review for the equipment and human performance issues.
h. Assess whether there was a common cause (i.e., maintenance activities during the
refueling outage) of the scram or post scram equipment performance issues.
i. Document the inspection findings and conclusions in a special inspection report in
accordance with Inspection Procedure 93812 within 45 days of the exit meeting for the
inspection.