IR 05000387/2013007
| ML13080A158 | |
| Person / Time | |
|---|---|
| Site: | Susquehanna  |
| Issue date: | 03/21/2013 |
| From: | Christopher Miller Division of Reactor Safety I |
| To: | Rausch T S Susquehanna |
| References | |
| IR-13-007 | |
| Download: ML13080A158 (43) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION REGION I 2100 RENAISSANCE BOULEVARD, SUITE 100 KING OF PRUSSIA, PENNSYLVANIA 19406-2713 March 21, 2013 Mr. TimothySenior Vice President and Chief Nuclear Officer PPL Susquehanna, LLC 769 Salem Boulevard, NUCSB3 Berwick, PA 18603
SUBJECT: SUSQUEHANNA STEAM ELECTRIC STATION UNIT 2 - NRC SPECIAL INSPECTION REPORT 05000388/2013007
Dear Mr. Rausch:
On February 7, 2013, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection at your Susquehanna Steam Electric Station. The inspection was conducted in response to the repetitive nature of operational problems related to reactor vessel level control, which contributed to five valid Reactor Protection System actuations between November 9 and December 19, 2012, at Susquehanna Unit 2, repetitive issues with the operation of the Integrated Control System (ICS), and specific questions related to operational decision-making that led to the December 19, 2012 unplanned reactor shutdown. The NRC's initial evaluation of this event satisfied the criteria in NRC Inspection Manual Chapter (IMC) 0309, "Reactive Inspection Decision Basis for Reactors," for conducting a Special Inspection. The Special Inspection Team (SIT) Charter (Attachment 2 of the enclosed report) provides the basis and additional details concerning the scope of the inspection. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on February 7, 2013, with you and other members of your staff. The inspection team examined activities conducted under your license as they relate to safety and compliance with Commission rules and regulations and with the conditions of your license.
The inspection team interviewed various plant personnel including operations personnel involved in the December 19, 2012, automatic plant shutdown. The inspectors reviewed plant logs, selected procedures and records, event evaluations, causal investigations, relevant performance history, and corrective actions planned and completed to assess the significance and potential consequences of issues related to the December 19, 2012, event.
The inspection team concluded that Susquehanna Unit 2 operated within acceptable limits, and no equipment malfunctioned during the automatic reactor shutdown. Nonetheless, the inspection team identified several issues related to human performance and compliance with procedures that contributed to the event. The enclosed chronology (Attachment 3 of the enclosed report) provides additional details regarding the timeline and sequence of events.This report documents two non-cited violations (NCVs) and one finding (FIN) of very low safety significance (Green). However, because of the very low safety significance of the violations and because they were entered into your correction action program, the NRC is treating them as non-cited violations (NCV) consistent with Section 2.32 of the NRC Enforcement Policy. If you contest any violations documented in the enclosed report, you should provide a response within 30 days of the date of the inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator Region I; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Senior Resident Inspector at Susquehanna Steam Electric Station. In addition, if you disagree with the cross cutting aspect assigned to any finding in this report, you should provide a response within 30 days of the date of the inspection report, with the basis for your disagreement, to the Regional Administrator, Region I, and the NRC Senior Resident Inspector at Susquehanna Steam Electric Station. In accordance with Title 10 of the Code of Federal Regulation (10 CFR) Part 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room and from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,/RA/ Christopher G. Miller, Director Division of Reactor Safety
Docket No. 50-388 License No. NPF-22
Enclosure:
Inspection Report 05000388/2013007
w/Attachments:
Supplemental Information (Attachment 1)
Special Inspection Team Charter (Attachment 2) Detailed Sequence of Events (Attachment 3) Simplified System Drawings (Attachment 4)
cc w/encl: Distribution via ListServ
SUMMARY OF FINDINGS
IR 05000388/2013007; 01/14/2013 - 02/7/2013; Susquehanna Steam Electric Station, Unit 2; Inspection Procedure 93812, Special Inspection. A three-person NRC team, comprised of two regional inspectors and one resident inspector, conducted this Special Inspection. Inspectors identified two non-cited violations (NCVs) and one finding (FIN) of very low safety significance (Green). The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter (IMC) 0609, "Significance Determination Process" (SDP), dated June 2, 2011. Cross-cutting aspects are determined using IMC 0310, "Components Within Cross-Cutting Areas," dated October 28, 2011. All violations of NRC requirements are dispositioned in accordance with the NRC's
Enforcement Policy, dated June 7, 2012. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight
Process," Revision 4.
Cornerstone: Initiating Events
- Green.
Inspectors identified a Green NCV of Technical Specification (TS) 5.4.1, Procedures, related to the requirement to operate the feedwater system in accordance with procedures and implement the procedure change process. The PPL procedures implementing these requirements state that if an approved document that addresses the circumstances does not exist, then create a procedure or perform the task using another approved method (i.e., troubleshooting plan or work order). Contrary to this requirement, on December 19, 2012, Pennsylvania Power and Light (PPL's) operators opened the breaker to the 'A' Reactor Feed Pump (RFP) discharge isolation valve (3A)valve motor operator (i.e., when the 3A valve failed to open as expected) without establishing or implementing procedural guidance or implementing another process such as a troubleshooting plan or work order. This action resulted in the feedwater control system logic causing closure of other feedwater valves, isolating all normal feedwater flow to the Reactor Pressure Vessel (RPV), and a subsequent automatic reactor shutdown (scram) on low water level. The PPL staff entered this issue into their corrective action program (CAP) as Condition Report (CR) 1668242, and conducted site-wide training on procedural use and adherence standards.
The inspectors identified a performance deficiency because on December 19, 2012, PPL did not implement an approved procedure to open the breaker to the 3A valve motor operator, which resulted in a subsequent unplanned reactor scram. This finding is more than minor because it is associated with the human performance attribute of the Initiating Events Cornerstone and adversely impacted the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Additionally, this finding was similar to example
4.b of IMC 0612, Appendix E, "Examples of Minor Issues." The finding was evaluated using NRC IMC 0609 Appendix A, "User Guidance for Significance Determination for At-Power Situations," and the Station Standardized Plant Analysis Risk (SPAR) Model for a detailed risk assessment. Based upon the detailed risk assessment, the change in core damage frequency associated with this performance deficiency was in the low E-7 range, or of very low safety significance (Green). The inspectors determined that this finding had a cross-cutting aspect in the area of Human Performance, Work Control, ii Enclosure because PPL operators did not appropriately plan work activities associated with opening the 3A valve manually by incorporating the need for planned contingencies, compensatory actions and abort criteria consistent with nuclear safety. H.3(a) (Section 3)
Cornerstone: Mitigating Systems
- Green.
A self-revealing Green NCV of TS 5.4.1, "Procedures," was identified involving the failure to incorporate the results of a Failure Modes and Effects Analysis (FMEA) completed in January 2010 into applicable operating procedures. The FMEA identified a vulnerability involving operator response to a loss of power to the RFP discharge isolation valves 3A (B, C) during the transfer from Discharge Pressure Mode (DPM) to Flow Control Mode (FCM). Specifically, PPL's FW operating procedures were not maintained to ensure operators could adequately recover RPV water level control when challenged with a system failure such as the condition that resulted in the Unit 2 scram on December 19, 2012. The PPL staff entered this issue into the CAP as AR-OPG-1654037, CR 1666244, and CR 1666253.
The finding was more than minor because it was associated with the procedure quality attribute of the Mitigating Systems Cornerstone and adversely impacted the objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). The finding was evaluated using NRC IMC 0609, Attachment 0609.04, "Initial Characterization of
Findings," and Appendix A, "User Guidance for Significance Determination for At-Power
Situations," and screened as very low safety significance (Green) per Exhibit 2. The inspectors determined that this finding had a cross-cutting aspect in the area of Human
Performance, Resources, because PPL staff did not ensure that procedures were complete, accurate and up-to-date to assure nuclear safety. Specifically, PPL's engineering modification procedures and checklists did not ensure that known single point design vulnerabilities were adequately addressed in FW procedures to ensure operators could adequately recover reactor water level prior to the Unit 2 reactor scram on December 19, 2012. H.2(c) (Section 3)
- Green.
A self-revealing finding (FIN) of very low safety significance (Green) was identified for PPL staff's failure to follow their CAP procedure, NDAP-QA-0702, "Action Request and Condition Report Process," in response to an identified issue with the FW system. Specifically, on August, 23, 2011, PPL's staff did not initiate an action request (AR) or condition report (CR) after determining that ICS digital FW valve control needed to be placed in Manual Valve Control mode prior to de-energizing the 3A motor operated valve (MOV) in order to prevent a loss of all FW flow. This issue went unaddressed and subsequently on December 19, 2012, Unit 2 scrammed on low RPV water level when operators, while attempting to open the stuck 3A valve, opened the 3A valve power supply breaker with the 'A' RFP FW valve controls in automatic causing a loss of all normal FW. The PPL staff entered this issue into the CAP as CR 1653480.
The finding was more than minor because it was associated with the human performance attribute of the Mitigating Systems Cornerstone and adversely impacted the objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). The finding iii Enclosure was evaluated using NRC IMC 0609, Attachment 0609.04, "Initial Characterization of Findings," and Appendix A, "User Guidance for Significance Determination for At-Power
Situations," and screened as very low safety significance (Green) per Exhibit 2. The inspectors determined that this finding had a cross-cutting aspect in the area of Problem Identification and Resolution, CAP, because PPL's staff did not implement the CAP with a low threshold for identifying issues completely, accurately, and in a timely manner commensurate with their safety significance. Specifically, PPL's staff did not identify non-conforming issues for FW valve control, design and operation that eventually led to a loss of normal FW and scram of Unit 2 on December 19, 2012. P.1(a) (Section 5)
REPORT DETAILS
1. Background and Description of Event In accordance with the Special Inspection Team (SIT) Charter (Attachment 2), the inspection team conducted a detailed review of the December 19, 2012, unplanned reactor shutdown event at Susquehanna Steam Electric Station, Unit 2 including: a review of prior related Feedwater (FW) Integrated Control System (ICS) equipment problems on November 9 and December 16, 2012; operator training and Just-In-Time Training (JITT); pre-job briefs, operator procedures; use of human performance error prevention techniques; licensee management oversight; and the effectiveness of the CAP and operator work around (OWA) program with respect to ICS equipment problems after the ICS modification was installed in 2010 (Unit 1) and 2011 (Unit 2).
The inspection team gathered information from the NRC resident inspectors, the plant process computer (PPC) alarm printouts and parameter trends, interviewed station personnel, observed simulator training and fidelity for ICS, reviewed procedures, logs, and various technical documents to develop a detailed sequence of events (Attachment 3). On December 19, 2012, at 1731, Unit 2 reactor shut down (scrammed) from 18 percent power with the main generator on the grid. Operators were swapping the 'A' RFP from Discharge Pressure Mode (DPM) (i.e., single element digital FW control) to Flow Control Mode (FCM) (i.e., three element digital FW control) with the 'B' RFP in standby. At 1709, Operators placed the 'A' RFP in FCM. During this transition, at 1711, operators observed that ICS did not reposition the 3A valve to open as expected.
At 1715, the Shift Manager (SM) and the operators discussed opening the 3A valve motor operator power supply breaker and manually opening the valve off its closed seat.
The SM then called the Outage Control Center (OCC) at 1720, and the OCC confirmed the 3A valve not opening was a repeat issue from the operator challenge list. At 1730, the SM contacted the OCC to inform them of the decision to manually open 3A valve. Operators were sent to the 3A valve and its motor operator power supply breaker with the intent of removing power from the motor operator and then manually opening the 3A valve off its closed seat using the clutch and local hand wheel. This decision was based, in part, on previous history of the 3A valve failing to open on August 23, 2011.
When the operator opened the 3A power supply breaker at 1730, the 3A valve position indicating lights were deenergized, and the ICS system logic sensed the 3A valve in its open position (i.e., when power was lost to the valve's green indicating light, the ICS logic input was zero volts, which the logic used as input for a full open valve position). As a result the Startup Level Control Valve (41) and the Startup Isolation Valve (51A), automatically closed as designed, resulting in loss of all normal FW to the Unit 2 RPV.
Operators attempted to take manual control of the 'B' RFP discharge isolation valve (51B) and place the 'B' RFP in service in DPM. The 51B valve indicated manual control but did not respond to operator demands due to FW system constraints concerning the first RFP being selected to FCM. During this short time period (less than 2 minutes), RPV water level rapidly lowered; and before operators could take the mode switch to shutdown, an automatic scram occurred at 1731 due to low RPV water level of 13 inches.
The 3A valve power supply breaker was reclosed, and operators were able to place the 'A' RFP back in DPM to restore RPV level control during post scram actions. 2. Integrated Control System (ICS) Design and Modification
a. Inspection Scope
The team determined that ICS is a non-safety related balance-of-plant process control system, which PPL staff had designated as important to safety. The Unit 2 ICS was installed in 2011. The team reviewed ICS as-built design drawings, and procedures, including normal and abnormal operating procedures to determine whether the original (i.e., pre-ICS modification) design basis, licensing basis, or performance capability of the FW flow and RPV water level control system capabilities had been degraded by the ICS modification. The team assessed PPL staff's technical evaluations and design details, and interviewed licensed operators, licensed operator simulator instructors, and engineering personnel to determine whether the ICS would function in accordance with the modification's assumptions, and with design and licensing requirements. Drawings and procedures were reviewed to verify whether they were properly updated to reflect the post-modification design and operation. Selected post modification test (PMT) results were reviewed to verify whether the acceptance criteria had been met.
Findings/Observations No findings of significance were identified. The inspection team identified an ICS design weakness in that the ICS design did not incorporate good engineering practice. Specifically, the ICS did not validate the 3A valve position input as either valid or invalid data for automatic control functions but did validate the position input for indication on the Human-Machine Interface (HMI) display screen (e.g., yellow/red for valid, or gray with cyan boarder for invalid). As a result, when an operator opened the 3A power supply breaker, the 3A valve position circuit was deenergized and the ICS system logic sensed the 3A valve in its open position. As a result the 41 and 51A valves automatically closed as designed. At the same time the 3A valve position indication changed to gray with a cyan border on the HMI display screen.
This design allowed ICS to perform automatic control functions using an invalid valve position, which directly led to a significant operator challenge and subsequent plant scram due to a loss of normal FW flow to the reactor. The inspectors determined this issue of concern was not a performance deficiency because there was no standard identified that had not been met.
After the December 19, 2012 event, PPL's staff identified that the ICS modification had changed the control logic for FW MOVs controlled by ICS from momentary contacts (original hand switch) to maintained contacts (ICS software switch). The effect of the maintained contact was to bypass the MOV contactor seal-in circuit and allow the contactor to re-energize if the torque switch were to relax and re-close. The inspection team determined that this design change was not evaluated during the modification process, as required by PPL's design verification process, and was contrary to PPL's design requirements and specifications for Limitorque MOVs. The PPL staff determined this issue did not directly contribute to the event because the likely cause for the 3A valve failing to open (e.g., stuck on its closed seat) was not related to potential MOV motor hammering. The inspection team determined that this was a minor design deficiency because PPL's staff subsequently determined that the effect of potential motor hammering did not exceed the design capability of the Limitorque operator. The PPL staff entered this issue into their CAP as CR 1654543. The inspection team also identified ICS HMI display human factors issues of concern as follows:
When the operator selects an ICS controller pop-up screen, the controller screen covers up a significant portion of the system screen and the associated process variables that the operator might need in order to adjust the controller settings. The PPL staff entered this issue into their CAP program as CR 1661485.
Invalid MOV position indication may not be sufficiently prominent to alert an operator during periods of high stress, such as during transient response. Specifically, an invalid indication changes the color of a valve icon bow-tie from yellow/red to gray with a small cyan border, instead of a bright backlit color that could stand out at a distance. As an example, when 3A valve power supply breaker was opened on December 19, 2012, the HMI indication changed from yellow (valve closed) to gray with cyan border (invalid/unknown position). In addition, the invalid indication of a gray "bow-tie" was backlit in a gray box. The PPL staff entered this issue into their CAP program as CR 1661485. The inspectors determined these issues of concern involved insufficient consideration of human factors. There were no performance deficiencies identified because the inspectors did not identify a standard that had not been met.
3. Procedures
a. Inspection Scope
As part of the SIT Charter, dated January 7, 2012, the inspectors were tasked with evaluating the effect of operating procedures and previous procedure changes associated with ICS on the December 19, 2012 event. The inspection team interviewed the control room operators, as well as engineering and management personnel that were directly involved with the recent unplanned shutdowns and reactor scrams involving ICS on Unit 2. The inspectors reviewed PPL's procedures (i.e., standard operating, off-normal, transient and scram procedures that involved FW and ICS control), operating logs, licensee event reports, root cause evaluations, and recent AR/CRs generated for ICS issues.
b. Findings
/Observations (1) Failure to Maintain Adequate Feedwater Procedures
Introduction.
A self-revealing Green NCV of TS 5.4.1, "Procedures," was identified involving the failure to incorporate the results of a Failure Modes and Effects Analysis (FMEA) completed in January 2010 into station operating procedures. The FMEA identified a vulnerability involving operator response to a loss of power to the 3A (B, C)valve during the transfer from DPM to FCM.
Description.
On January 22, 2010, the FMEA of the ICS modification package identified a single point vulnerability regarding the 3A valve closed status. During the transition from DPM (i.e., single element startup level control) to FCM (i.e., three element FW level control), if power is interrupted to the 3A valve motor operator, the ICS system senses the 3A valve is in the open position and the FW control valves (i.e., 41 and 51A), will automatically close resulting in loss of all normal FW flow to the Unit 2 RPV. The ICS digital feedwater control system was designed to use the voltage applied to a valve position indicating light circuit (i.e., red light, green light) as the logic input for valve position. The ICS was designed to use "zero volts" on the valve's green light as the valve full open position indication for ICS logic operation (i.e., the green light is lit when the valve is closed or intermediate, and not lit when full open). This ICS design vulnerability allowed a zero voltage input, due to the loss of control voltage to the MOV (when the breaker was opened) to be interpreted as a valve full open signal, which in turn allowed the ICS logic to automatically close the 41 and 51 valves, as designed after the 3A valve "indicated" that it had gone full open.
This system vulnerability allowed ICS to perform automatic control functions using an invalid valve position, and had the potential for a significant operator challenge to control RPV water level and possibly cause a reactor scram on loss of normal FW flow. The FMEA stated that the mitigation action for this design vulnerability was to have the operators maintain the system in manual control during certain modes of operation. The inspectors determined that this known system vulnerability was reviewed by PPL's staff during the modification installation process, but not specifically addressed in FW procedures.
On December 19, 2012, Unit 2 scrammed at 18 percent power with the main generator connected to the grid. At the time of the scram, operators were swapping the 'A' RFP from DPM to FCM with the 'B' RFP in standby. The Unit 2 'A' RFP 3A valve failed to open during this transition. Operators were sent to the 3A valve and opened the power supply breaker with the intent of manually opening the valve off its closed seat locally and then closing the breaker. This decision making was based, in part, on the August 23, 2011 event (discussed in Section 5). When the operator opened the 3A power supply breaker, the 3A valve position circuit was deenergized and the ICS system sensed the 3A valve in its default position (open), as a result the 41 and 51A valves, automatically closed as designed, resulting in a loss of all normal FW. Operators attempted to take manual control of the 'B' RFP and place it in service in DPM per OP-245-001, "RFP and RFP Lube Oil System," but RPV water level rapidly lowered and a scram occurred on low RPV water level. The inspectors noted that at the time of the December 19, 2012 event, procedure OP-245-001 did not contain guidance to the operators regarding a loss of power to the 3A valve during the DPM to FCM transfer. The PPL staff revised the procedure after the event to add a caution and procedural steps to address this issue. The inspectors determined that the revised procedure did not contain guidance on how to respond to a closure of the 51A valve if closure of the 41 valve is not recognized by the operator within the initial 10 seconds. In response to the inspectors' observation, PPL's staff wrote CR 1663285 to address this procedural issue.
Analysis.
The performance deficiency associated with this issue is that PPL's staff did not maintain adequate FW operating procedures that ensured operators could recover RPV water level control when challenged with the single point vulnerability prior to the Unit 2 reactor scram on December 19, 2012. The performance deficiency was evaluated in accordance with IMC 0612, Appendix B, "Issue Screening," and determined to be more than minor because it was associated with the procedure quality attribute of the Mitigating Systems Cornerstone and adversely impacted the objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). The finding was evaluated using Section A of Exhibit 2 of NRC IMC 0609, Attachment 0609.04, "Initial Characterization of Findings," and Appendix A, "User Guidance for Significance Determination for At-Power Situations," and screened as very low safety significance (Green), because the performance deficiency did not result in a loss of safety function or represent an actual loss of function of one or more non-Technical Specification trains of equipment designated as high safety-significant in accordance with PPL's maintenance rule program for greater than 24 hrs.
The inspectors determined that this finding had a cross-cutting aspect in the area of Human Performance, Resources, because PPL's staff did not ensure that procedures were complete, accurate and up-to-date to assure nuclear safety. Specifically, PPL's engineering modification procedures and checklists did not ensure that known single point design vulnerabilities were adequately addressed in FW procedures to ensure operators could adequately recover reactor water level prior to the Unit 2 reactor scram on December 19, 2012. H.2(c)
Enforcement.
TS 5.4.1.a, "Procedures," requires, in part, that written procedures shall be established, implemented, and maintained covering the applicable procedures recommended in Regulatory Guide (RG) 1.33, Revision 2, Appendix A. RG 1.33, Appendix A lists activities that should be covered by written procedures. Section 4 identifies procedures for startup, operation, and shutdown which indicate 4.o "FW system (FW pumps to reactor vessel)." Contrary to the above, in January 2010, PPL's staff failed to maintain adequate FW operating procedures that ensured operators could adequately recover RPV water level control when challenged with the single point vulnerability described in the FMEA. This resulted in the Unit 2 scram on December 19, 2012. Because this finding is of very low safety significance and was entered into PPL's CAP as AR-OPG-1654037, CRA 1666244, and CRA 1666253, it is being treated as an NCV in accordance with the NRC's Enforcement Policy. (NCV 05000388/2013007-01, Failure to Maintain Adequate Feedwater Procedures)
During the inspectors' review of the December 19 scram and previous ICS events, the inspectors noted additional examples of PPL procedural issues. Specifically: Inspector interviews conducted after the December 19 event indicated the operator incorrectly used the wrong side of the Hard Card (operator aid) for loss of a feed pump when in either DPM or FCM. The operator placed the standby 'B' RFP in-service in DPM after he had previously entered ICS FCM for the 'A' RFP rather than placing the 'B' RFP in-service in FCM using the back side of the Hard Card. The inspectors determined the Hard Card provided weak procedural guidance and should be revised to clarify when each side of the card should be utilized. (CR 166244) On August 23, 2011, the 3A valve would not move from its closed position during a reactor startup. The PPL staff classified the issue as an "Operator Challenge" in the Operator Burdens list. The inspectors determined that no workaround instructions or compensatory actions were developed and given to the operators as required by the Operator Burden procedure. The Operator Burdens process had been previously identified by the resident inspectors (March 2011) and Operations Nuclear Oversight (December 2011) as being ineffective in informing operator decision making and inadequate in identifying, tracking, evaluating and resolving Operator Burdens. On January 11, 2013, PPL staff issued 13-02 Hot Box (Operations Read and Sign),
"Operator Aggregate Index", after determining the Operator Aggregate Index and Operator Burdens procedure were ineffective. Prior to the December 19, 2012, Unit 2 reactor scram, PPL's operator aggregate index had identified 5 Operator Work Arounds (OWAs), 4 operator challenges, and 8 control room deficiencies for both Units. The Hot Box investigation on January 11, 2013, yielded a significant increase of 11 OWAs, 21 operator challenges, and 19 control room deficiencies. The inspectors determined while this performance deficiency contributed to valve 3A not being addressed, the most significant cause of this loss of FW event was inadequate CAP implementation, which is addressed in the finding documented in section 5.b.(1).
For the November 9, 2012, Unit 2 scram, the inspectors determined that procedures ON-200-101 and OP-AD-001, provided insufficiently clear procedural guidance for post scram RPV water level band operation and level control when ICS is unavailable. The inspectors determined that these procedures should be revised to enhance the guidance provided to the operators. (CR 1652942) For the December 16, 2012, Unit 2 scram, the inspectors determined that ON-200-101 provided insufficiently clear procedural guidance for post scram RPV level band control to the operators. The PPL staff revised this procedure to include guidance within the procedural steps on ensuring operators prioritize raising RPV water level post scram to ensure a second scram signal is not received. The inspectors determined that this procedure should be revised to enhance the guidance provided to the operators. (CR 1652942) (2) Failure to Establish and Implement Written Procedures Prior to Operating Plant Equipment
Introduction.
The inspectors identified a Green NCV of TS 5.4.1, "Procedures," for PPL staff's failure to establish and implement a written procedure on December 19, 2012, while operators took actions not described in procedures associated with a previously identified equipment issue with the 3A valve during a Unit 2 reactor startup and power ascension.
Description.
On December 19, 2012, operators were in the process of swapping the 'A' RFP from DPM to FCM with the 'B' RFP in standby. The 2 'A' RFP 3A valve failed to open as expected during this transition. When the 3A valve failed to open, Shift Supervision directed operators to the 3A valve and its power supply breaker with the intent of de-energizing the breaker locally and then manually opening the 3A valve off its closed seat. This decision making was based, in part, on the August 23, 2011 event (discussed in Section 5). However, the operating crew overlooked the fact that on August 23, 2011 the ICS was in manual operation, vice automatic.
When the operators opened the 3A valve power supply breaker, the ICS system sensed the 3A valve was in the open position and the FW control valves, 41 and 51A, automatically closed as designed, resulting in loss of all normal FW flow to Unit 2.
Operators attempted to take manual control of the 'B' RFP FW 51B valve and to place the 'B' RFP in service in DPM. The 51B valve indicated manual control, but did not respond to operator demands due to unrecognized FW system constraints when the first RFP was selected to FCM. During this time period, RPV water level rapidly lowered, and before operators could take the mode switch to shutdown, an automatic scram occurred on low RPV water level. After the scram, the 3A valve power supply breaker was reclosed, and operators were able to place the 'A' RFP back in DPM to restore RPV level control.
The inspectors determined through operator interviews that PPL's staff had a history of not adequately addressing non-safety related MOV issues, such as suspected valve binding or failure of the valve to open, by de-energizing the valve power supply breaker and then manually opening the valve off its closed seat. Operators stated that PPL's staff routinely conducted these actions on stuck closed MOVs, without an approved or written procedure. During the review of PPL procedures, the inspectors noted the following PPL procedural requirements:
NDAP-QA-0029, "Procedure and Work Instruction Use and Adherence," Section 5.1.3, states in part, if an approved document that addresses the circumstances does not exist, then create a procedure per the Procedure Change Process, NDAP-QA-0004, or perform the task using another approved method (i.e. troubleshooting plan, work order). NDAP-QA-0510, "Troubleshooting Plant Equipment, Section 5.11, states in part, Troubleshooting Activities - Troubleshooting activities include but are not limited to the following evolutions-5.11.13 Repositioning valves or breakers and manual operation of Motor Operated Valves-"
OP-AD-002, "Standards for Shift Operations," Section 2.4, states in part, when conducting tasks during all plant operational modes and during transients, operating crews are expected to precisely control plant evolutions in accordance with approved procedures and control plant evolutions effectively by using procedures.
NDAP-QA-1902, "Integrated Risk Management," Att. B, Reactivity Manipulations, indicates that work on the FW system and ICS presents a potential reactivity risk because it could possibly affect reactivity. Att. C states to conduct the risk activity according to applicable procedure or instruction.
The inspectors determined that the PPL's staff did not follow these procedural requirements related to the 3A valve manipulations in accordance with approved procedures.
Analysis.
The performance deficiency associated with this issue is that PPL's staff did not follow TS required procedures on December 19, 2012, while attempting to resolve a problem with the operation of the 3A FW valve during Unit 2 reactor startup and power ascension. The issue was evaluated in accordance with IMC 0612 and determined to be more than minor because it was associated with the human performance attribute of the Initiating Events cornerstone and adversely impacted the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Additionally, this finding was similar to example 4.b of IMC 0612, Appendix E, "Examples of Minor Issues."
The finding was evaluated using Section B of Exhibit 1 of NRC IMC 0609 Appendix A, "User Guidance for Significance Determination for At-Power Situations." Since the performance deficiency caused a reactor shutdown (scram) and the loss of mitigation equipment relied upon to transition the plant from the onset of the trip to a stable shutdown condition (e.g. loss of condenser, loss of feedwater) a detailed risk evaluation was required. This evaluation used the Susquehanna Station Unit 2 SPAR Model, Revision 8.21. Based upon initial review of the performance deficiency using
0609.04, "Initial Characterization for Findings," the Senior Reactor Analyst (SRA) implemented the SPAR Model to conduct a detailed risk assessment. The SRA modeled the performance deficiency using the event assessment methodology and set the Loss of Feed Water Initiating Event to 1.0 and the remaining initiating events to zero. Based upon discussions with the site PRA staff, control room operators do not inhibit the automatic depressurization system on a plant transient, except for anticipated transients without scram (ATWS) events. Accordingly, the basic event for operator manual depressurization (ADS-XHE-XM-MDEPR) was set to FALSE (never fails). Consequently, the risk associated with this finding was determined to be in the low E-7 range or of very low safety significance (Green). The dominant core damage sequences involve a loss of feedwater with subsequent failure of suppression pool cooling and containment venting, followed by loss of feedwater events with failures to scram (ATWS) related core damage sequences. External event contributions were not significant for this performance deficiency and the overall estimated change in core damage frequency was conservatively high due to the plant transient being initiated at a low reactor power level (18% vice 100% power) following a plant shutdown.
The inspectors determined that this finding had a cross-cutting aspect in the area of Human Performance, Work Control, because PPL's staff did not appropriately plan work activities associated with opening the 3A valve manually by incorporating the need for planned contingencies, compensatory actions and abort criteria consistent with nuclear safety. H.3(a)
Enforcement.
TS 5.4.1.a, "Procedures," requires, in part, that written procedures shall be established, implemented, and maintained covering the applicable procedures recommended in RG 1.33, Revision 2, Appendix A. RG 1.33, Appendix A lists activities that should be covered by written procedures. Section 1 identifies Administrative procedures, among which is 1.d "Procedure Adherence and Temporary Change Method." Specifically, PPL procedure NDAP-QA-0029, "Procedure and Work Instruction Use and Adherence," Section 5.1.3, states in part, if an approved document that addresses the circumstances does not exist, then create a procedure per the Procedure Change Process, NDAP-QA-0004, or perform the task using another approved method (i.e. troubleshooting plan, work order). Contrary to this, PPL's staff did not establish and implement an approved document or the procedure change process when operators opened the 3A valve power supply breaker. Consequently, when this breaker was opened, the FW control valves automatically closed, resulting in loss of all normal FW and an automatic reactor scram on low RPV water level. Because this finding is of very low safety significance and was entered into PPL's CAP as CR 1668242, it is being treated as an NCV in accordance with the NRC's Enforcement Policy. (NCV 05000388/2013007-02, Failure to Establish and Implement Written Procedures for Operating Plant Equipment)
4. Training and Operator Knowledge
a. Inspection Scope
The inspection team interviewed personnel, reviewed simulator modeling and operator performance, reviewed crew training material and Just-In-Time Training (JITT) material for the initial and subsequent reactor startups, remedial training for the operators involved with the event, and training plans developed after the event on December 19, 2012. The inspection team also reviewed previous ICS related events from November 9, 2012, and December 16, 2012.
b. Findings
/Observations No findings were identified. The inspection team identified weaknesses in PPL's operator training program, including:
Training lesson plans did not address the logic inputs from the RFP discharge isolation valves to the ICS. PPL's ICS classroom and simulator training for licensed operators and technical plant staff was limited in scope regarding ICS malfunctions and transients (e.g., RFP trip, loss of a single reactor level indicator, loss of a steam instrument). The training did not provide ICS FW startup level control training opportunities with malfunctions such as loss of power to an MOV (e.g., single point vulnerability identified in the ICS design which credited operator manual actions).
Therefore, the operators were not aware through training that a loss of control power to these valves during the transition from DPM to FCM would result in a loss of normal FW (CR 1665479) Licensed Operator Requalification (LOR) Exam Bank contained limited ICS and FW Malfunctions. For example, the inspectors reviewed the six Job Performance Measures (JPMs) in the LOR Exam bank and noted that the tasks examined included mostly routine FW operations with only one alternate path JPM that tested operator response to an ICS/FW malfunction. The LOR exam simulator scenarios were also reviewed and found to contain limited ICS malfunctions (i.e., a total of three FW/ICS malfunctions). (CR1661759) JITT conducted for the reactor startup prior to the December 19, 2012 event was not administered to all operators on shift at the time of the event. The inspectors found that only three of the eight control room operators attended the training [the Reactor Operator (RO), the Unit Supervisor (US), the Reactivity Senior Reactor Operator (SRO)]. The inspection team also found that the optional operator training prior to the event did not include any simulator exercises on RFP startup, placing FW in 3 element control and transitioning from DPM to FCM, and FW ICS transients. (CR 1661470 and 1661762)
The inspection team identified potential operator knowledge weaknesses, including:
Operators controlling FW during the December 19, 2012 event did not use the Hard Card (operator aid) properly (see Section 3); and did not attempt to take RFP valve control in manual to reopen both the 41 and 51A valves. (CR 1666244)
During the November 9, 2012 event, the operators did not fully understand: 1) what narrow range level indications were available; and 2) that the narrow/wide range level divergence would increase with time as reactor pressure decreased during plant cool down. (CR1659749) During the December 16, 2012 event, the operators failed to prioritize the number one method of preventing RPV stratification ahead of the other 5 methods listed in ON-200-101. Also, the operator did not recognize that the Reset Setpoint Setdown button needed to be pressed prior to raising level set-point per OP-245-001.
(CR1659749) The inspectors concluded that these potential ICS licensed operator training and knowledge weaknesses point to the need for PPL's staff to conduct more in-depth ICS training and testing for both licensed operators and technical plant staff. (CR 1665479)5. Corrective Action Program (CAP) and Operator Workaround Process Effectiveness
a. Inspection Scope
As part of the SIT Charter, dated January 7, 2012, inspectors were tasked with determining the effectiveness of the CAP with respect to ICS equipment problems since installation of the ICS modification in 2010 on Unit 1 and 2011 on Unit 2. The inspection team interviewed personnel, reviewed various CAP and Operator Burdens procedures, and PPL's staff generated CRs/ARs related to ICS and FW.
b. Findings
/Observations Failure to Implement the CAP
Introduction.
A self-revealing finding (FIN) of very low safety significance (Green) was identified for PPL staff's failure to implement CAP procedure, NDAP-QA-0702, "Action Request and Condition Report Process," in response to an identified issue with the FW system.
Description.
On August 23, 2011, PPL operators were in the process of starting up Unit 2. After reaching approximately 18 percent power, operators attempted to place the 'A' RFP in FCM from DPM per procedure OP-245-001, "RFP and RFP Lube Oil System." During this transition to FCM, the 3A valve did not open as expected. After resetting the 3A valve thermal overload relay that was found tripped, operators placed the ICS FW control system in the Manual Valve Control mode. A second attempt to open the 3A valve was also unsuccessful. The Operations staff consulted with the ICS FW Subject Matter Expert (SME) and opened the 3A valve power supply breaker, then manually manipulated the 3A valve off its closed seat. At this point, the 3A valve thermal overload relay was again reset, the breaker was closed and the valve was opened. After operators placed the ICS FW valve controls in the automatic mode, the SME recognized that because of the system design, the FW valve controls needed to be in the Manual Valve Control mode prior to opening the 3A valve power supply breaker in order to prevent ICS from falsely recognizing the 3A valve as being fully opened which would in turn result in a loss of normal FW flow (i.e., automatically closing of the 41 and 51A valves). Although the SME recognized this issue and communicated it to other operators via internal PPL emails, no action request (AR) or condition report (CR) was generated to capture the issue into the CAP. As a result no corrective actions were developed or implemented to ensure the vulnerability was addressed in written procedures. The inspectors determined that PPL's "Action Request and Condition Report Process" procedure, NDAP-QA-0702, Section 6.1 states, in part, "-all PPL personnel have the obligation to initiate an AR for identified or perceived problems, issues, concerns and non-conformances." Based on this procedural requirement, the inspectors determined that contrary to PPL procedure, the system design vulnerability identified on August 23, 2011, concerning the FW valve controls should have been documented in the CAP when it was identified by the SME and operations personnel. This issue went unaddressed until subsequently on December 19, 2012, Unit 2 scrammed on low RPV water level when operators opened the 3A valve power supply breaker with the 'A' RFP FW valve controls in automatic, causing a loss of all normal FW.
Analysis.
The performance deficiency associated is that PPL's staff did not implement their CAP process by initiating a CR in response to identified FW system issues. The issue was evaluated in accordance with IMC 0612 Appendix B, "Issue Screening," and determined to be more than minor because it was associated with the human performance attribute of the Mitigating Systems Cornerstone and adversely impacted the objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). The finding was evaluated using Section B of Exhibit 1 of NRC IMC 0609, Attachment 0609.04,
"Initial Characterization of Findings," and Appendix A, "User Guidance for Significance Determination for At-Power Situations," and screened as very low safety significance (Green) because the performance deficiency did not result in a loss of safety function or represent an actual loss of function of one or more non-Technical Specification trains of equipment designated as high safety-significant in accordance with the licensee's maintenance rule program for greater than 24 hrs.
The inspectors determined that this finding had a cross-cutting aspect in the area of Problem Identification and Resolution, CAP, because PPL's staff did not implement the CAP with a low threshold for identifying issues completely, accurately, and in a timely manner commensurate with their safety significance. Specifically, PPL's staff did not identify non-conforming issues for FW valve control, design and operation that eventually led to a loss of FW and scram of Unit 2 on December 19, 2012. P.1(a)
Enforcement.
This finding does not involve enforcement action since no violation of regulatory requirements was identified. This was because PPL's procedure, NDAP-QA-0702, "Action Request and Condition Report Process," is not required to be implemented as part of Susquehanna's 10 CFR 50, Appendix B, Quality Assurance Program for systems such as Feedwater, which are not safety-related. The PPL staff has entered the issue into the CAP as CR 1653480. Because this finding does not involve a violation, it is identified as an FIN. (FIN 05000388/2013007-03, Failure to Implement the Corrective Action Process)
The inspectors noted two similar, but minor examples of PPL's staff failing to implement the CAP. Specifically: On August 25, 2011, two days after the 3A valve did not open upon demand for the first time, Operations generated CR 1455447 and PCWO 1456387 to address the 3A valve issue during the next scheduled or forced outage. The inspectors reviewed the Unit 2 operating history and identified that PPL's staff had multiple opportunities to address the 3A valve issue during outages in June and November of 2012, but did not. On October 18, 2012, operations staff generated CR 1632449 to add the 3A valve work order to the November turbine outage. The PPL staff closed this CR and disapproved the work for the outage because the 3A valve had not exhibited any issues during plant startups earlier in 2012.
The inspectors concluded that the PPL staff did not have a sufficient technical basis for concluding the valve was reliable without taking any corrective actions. (CR 1653480) Following the December 19, 2012 event when Unit 2 scrammed on low RPV reactor water level when operators opened the 3A valve power supply breaker with the 'A' RFP FW valve controls in automatic causing a loss of all FW, PPL's staff conducted troubleshooting activities to determine the cause for an issue with the FW 51B valve failing to open. The PPL staff observed locally that the 51B MOV motor hammering was occurring for 1 to 2 minutes following valve closure. An engineering evaluation of this issue identified that the differential pressure across the closed 51B valve was significantly greater than the design capability of the Limitorque operator for the valve. The evaluation also determined that the Limitorque operator had a self-locking worm gear design that should not have been susceptible to torque switch relaxation and subsequent motor hammering.
Inspectors determined that PPL's staff failed to identify that the 51B MOV motor hammering was a non-conforming condition with the valve's design. The inspectors reviewed PPL staff's actions and found that no MOV diagnostics were performed or scheduled, no grease samples were taken or evaluated, and no specific follow-up actions for the 51B valve were planned. The PPL staff initiated a CR to evaluate the MOV logic change, but did not enter the 51B valve motor hammering issue into the CAP for further evaluation or correction. Because of this, the inspectors determined that this was contrary to NDAP-QA-0702, Section 6.1 states, in part, that all PPL's personnel have the obligation to initiate a CR for identified or perceived problems, issues, concerns and non-conformances', and Section 8.1.1 that states, in part, that if during the course of an evaluation, or within the course of answering a corrective action a new potential issue is identified, a new CR should be implemented. The inspectors determined the failure to initiate a CR for a 51B non-conforming condition was a minor performance deficiency because the valve functions as an isolation valve to the Start-up Level Control Valve and is normally positioned fully open or fully closed. The failure of this valve to open would have been self revealing, and only one train of FW (e.g., only one of the three 51 valves) is required to open to support start-up level control. (CR 1654543) 6. Organizational Response 6.1 Crew Roles and Responsibilities
a. Inspection Scope
The inspection team interviewed personnel, reviewed various procedures and records to assess PPL's personnel crew roles and responsibilities before, during and after the December 19, 2012 event.
b. Findings
/Observations No findings were identified. During operator interviews, the inspection team identified that: It was not clear which SRO had given the order to take manual actions to open the 3A valve. The inspectors found that the action to open the 3A valve was discussed between the SROs (i.e., Shift Manager, Unit Supervisor, Reactivity SRO, and General Operating SRO) and ROs assigned to the shift and a consensus was reached that was endorsed by the SM.
The General Operating (GO) SRO position is not currently defined in a PPL procedure. Roles and responsibilities for licensed operators are defined in Conduct of Operations Procedures. The PPL staff has entered this issue into the CAP as CR 1661456.
6.2 Immediate Response and Restart Readiness Assessment
a. Inspection Scope
The inspection team interviewed personnel, reviewed various procedures and records to assess PPL staff's immediate response and restart readiness after the December 19, 2012 event.
b. Findings
/Observations No findings were identified.
The inspection team determined: PPL staff's evaluation of potential motor hammering during trouble shooting activities to determine the cause for the 51B valve failing to open and the extent of condition for other ICS controlled FW MOVs was inadequate (See Section 5 for the 51B valve inadequate evaluation). Following PPL staff's observation of motor hammering on 51B, PPL staff investigated whether the 3A and HV-20616A1-C1 valves were also subject to motor hammering. With the MOVs in a static condition (e.g., valves had not been recently stroked), local visual observations and motor thermography were performed to determine whether motor hammering was occurring. EWR 1654453 was presented at the Startup Plant Operation Review Committee (PORC) meeting and documented that these valves were found to be "satisfactory with no cycling at the time of investigation." However, the inspectors determined that based on the short term nature of the hammering condition immediately following valve closure, as observed with the 51B, the field investigation results were inconclusive as to whether the other MOVs were also subject to on-going motor hammering issues. (CR 1654543) During PPL staff's trouble shooting of the 3A valve, the 3A MOV was re-lubricated prior to performing the valve diagnostic testing. The inspectors determined that such pre-conditioning could mask degraded as-found high friction issues which could have contributed to the failure of the 3A valve to open on December 19, 2012. The PPL staff has entered the issue into the CAP as CR 1654543.
7. Post-Event Root Cause Evaluation and Actions
a. Inspection Scope
The inspection team reviewed PPL staff's Root Cause Analysis (RCA) report for the event to determine whether the causes and associated human performance issues were properly identified. Additionally, the inspection team assessed whether interim and planned long-term corrective actions were appropriate to address the cause(s).
b. Findings
/Observations No findings were identified. The PPL staff identified two root causes and one causal factor in their RCA: Root Cause-1 (RC1): The decision to open the HV20603A valve breaker was made without a formal evaluation of impacts (knowledge based decision) that reflected a conditioned operator response and inadequate risk evaluation of activities. Root Cause 2 (RC2): Opportunities were missed to identify and provide compensation for the design of the ICS logic interface when opening the valve breaker. Causal Factor 1 (CF1): The Operations Burden program (OI-AD-096) did not provide evaluated compensatory actions or drive the correction of the identified issue with the HV20603A valve prior to Dec 2012 event.
The inspectors determined that the RCA was generally thorough and considered RC2, and CF1 reasonable. In addition, the inspectors determined that the associated proposed corrective actions appeared to adequately address the underlying causal factors with the exception of Procedure Use and Adherence (PU&A).
The inspectors concluded that the RC1 was too narrowly focused and did not identify PU&A as a root cause. The RCA concluded the operators were in process (i.e.
complying with procedures). The RCA focused on OP-AD-001, "Operations Standards for System and Equipment Operation", which states in step 10.2, "Operators shall take manual actions to operate equipment when the auto functions fail." This is mentioned in a number of places in the RCA.
There are two other salient points that were not mentioned in the RCA discussion of PU&A: 1) OP-AD-001, step 10.2 is preceded by a caution which states, "Personnel injury or further degradation to plant equipment may result from operating equipment in manual that is normally operated in automatic"; and 2) OP-AD-001, also states in Section 7.4, "Overriding system/equipment controls or interlocks shall not be allowed unless specifically authorized by, or in accordance with, approved procedures." It provides an example as "opening circuit breakers". Since the plant had been in a stable condition for almost a half hour on December 19, 2012, prior to the crew opening the breaker for the 3A valve, the inspectors determined there was no urgency for the crew in proceeding in the face of uncertainty. In contrast during a plant transient (i.e., in a situation where it is necessary to implement Off-Normal or Emergency Operating Procedures), it is appropriate for the crew to take timely actions to stabilize the plant (e.g. taking manual control of a malfunctioning automatic system). NDAP-QA-0029, step 5.14.2, states that if a document cannot be performed as written or there is an unexpected result to "place the equipment-in a safe, stable condition" and "if actions required are outside of the procedure or work instruction, then use approved site processes to maintain configuration control." The inspectors determined that the plant was in a stable condition and therefore it was inappropriate to take manual action to open the breaker without written direction. The RCA states that due to the significant number of stuck valve issues present at the plant, the operators were somewhat preconditioned to respond to stuck valves in this manner as the normal and accepted approach (i.e., culture at Susquehanna). NDAP-QA-0029 allows shift supervision to provide direction on how to resolve the issue and return to the procedure. However, it does not specify how and with what process this should be done (i.e. switching orders, procedure change, or troubleshooting activity, etc.). As discussed above in section 3.b(2), NRC finding "Failure to Establish and Implement Written Procedures Prior to Operating Plant Equipment", there are a number of established station procedures and processes that should have stopped the crew from initiating troubleshooting activities without written guidance.
The inspectors concluded that the staff generating the RCA missed an opportunity to address and reinforce operation staff's PU&A standards. The inspectors noted that PPL's staff issued the "Operations Intervention Plan" (AR 1665479) on January 28, 2013, which is an effort aimed at strengthening procedure use and adherence standards and changing the cultural environment at the station. In response to the inspectors' concern, PPL's staff initiated CR 1668242.
4OA6 Meetings, Including Exit
Exit Meeting Summary
On February 7, 2013, the inspection team discussed the inspection results with Mr.
Timothy S. Rausch, Senior Vice President and Chief Nuclear Officer, PPL Susquehanna and members of his staff. The inspection team confirmed that proprietary information reviewed during the inspection period was returned to PPL.
A-1-1
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- T. Rausch, Senior Vice President and Chief Nuclear Officer
- P. Bone, Reactor Operator
- L. Crawford, Assistant Operations Manager-Shift
- R. Cover - Senior Assessor, Nuclear Oversight
- M. Crowthers, Corporate Project Manager
- M. Hanchuruck, Licensed Operator Requalification Program Supervisor
- J. Hartzell, Supervisor Plant Analysis Program
- T. Iliadis, General Manager of Operations
- M. Jacopetti, Unit Supervisor, Operations
- A. Jardine, Operations Manager
- J. Jennings, Supervisor Nuclear Regulatory Affairs
- M. Lichtner, Shift Manager, Operations
- J. Petilla, Supervisor Nuclear Regulatory Affairs
- D. Przjemski, Senior Design Engineer
- P. Scanlon, Manager Engineering Programs
- J. Schleicher, Supervisor Design Engineering
- S. Skoras, Senior Risk Assessment Engineer
- H. Strahley, Assistant Operations Manager-Training
- R. Streeper, Operations Training Manager
- J. Tripoli, Manager Nuclear Regulatory Affairs
- J. Willis - Unit Supervisor, Operations
- B. Yu, Senior Electrical Design Engineer
NRC Personnel
- P. Wilson, Deputy Director, Division of Reactor Projects
- D. Jackson, Chief Operations Branch, Division of Reactor Safety
- M. Gray, Chief Branch 4, Division of Reactor Projects
- P. Finney, Senior Resident Inspector - Susquehanna
- J. Grieves, Resident Inspector - Susquehanna
- A. Rosebrook, Senior Project Engineer, Division of Reactor Projects
A-1-2SUPPLEMENTAL INFORMATION Attachment 1
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened/Closed:
- 05000388/2013007-01 NCV Failure to Maintain Adequate Feedwater Procedures (Section 3.b.(1))
- 05000388/2013007-02 NCV Failure to Establish and Implement Written Procedures for Operating Plant Equipment
(Section 3.b.(2))
- 05000388/2013007-03 FIN Failure to Implement the Corrective Action Process (Section 5.b.(1))
Closed: None
A-1-3SUPPLEMENTAL INFORMATION Attachment 1
LIST OF DOCUMENTS REVIEWED
Drawings
- D107278 Sht. 11, RFP Suction Valves
- HV-206-16A1 Schematic, Revision 6 D107278 Sht. 9, RFP Discharge & Bypass Valves
- HV-206-03 &
- HV-206-51 Schematic, Revision 9
- E-129 Sheet 12, RFP Suction Bypass Valves
- HV-206-16A2 Schematic, Revision 6
- FF62201 Sheet 160, RFP 2A Isolation Valves Logic Functional Diagram, Revision 0 & IDCN 1
- FF62201 Sheet 161, RFP 2A Isolation Valves Logic Functional Diagram, Revision 1
- FF62201 Sheet 162, RFP 2B Isolation Valves Logic Functional Diagram, Revision 0 & IDCN 1 FF62201 Sheet 163, RFP 2B Isolation Valves Logic Functional Diagram, Revision 1 FF62201 Sheet 164, RFP 2C Isolation Valves Logic Functional Diagram, Revision 0 & IDCN 1
- FF62201 Sheet 165, RFP 2C Isolation Valves Logic Functional Diagram, Revision 1
- FF62201 Sheet 561, Start-Up Level Control Logic Functional Diagram, Revision 1 & IDCN 1
- FF62201 Sheet 562, FW Bypass & Low Load Valve Sequence Logic Functional Diagram, Rev. 0 FF62201 Sheet 590, RFP 2A Isolation Valves Logic Functional Diagram, Revision 1
- FF62201 Sheet 591, RFP 2B Isolation Valves Logic Functional Diagram, Revision 1
- FF62201 Sheet 592, RFP 2C Isolation Valves Logic Functional Diagram, Revision 1
- M-2106 Sheet 2, Feedwater P&ID, Revision 25 M-2106 Sheet 3, Feedwater P&ID, Revision 21 M-2106 Sheet 4, Feedwater P&ID, Revision 21
- Engineering Calculations, Analysis, and Evaluations Safety Evaluation
- SE-013, Modifications
- EC 910695, Integrated Control System and Reactor Feed Pump Turbine Speed Control System, Revision 0
- ICS-044, Integrated Control System Failure Modes and Effects Analysis, Revision 0
- AR 1656675, Risk Assessment for Unit 2 Shutdown on 12/19/12, Revision 0
Condition Reports
(CRs) (* denotes NRC identified during this inspection)
- 1201240
- 282140
- 1336306
- 1356772
- 1455447
- 1488273
- 1500217
- 1535819
- 1549033
- 1563230
- 1563746
- 1565001
- 1617070
- 1632449
- 1633295
- 1638643*
- 1640628
- 1640693
- 1640845
- 1641009
- 1641797
- 1641818
- 1652338
- 1652377
- 1652426
- 1652814
- 1652942
- 1653480
- 1653679
- 1653762
- 1654037
- 1654430
- 1654453
- 1654543
- 1654546
- 1654551
- 1654555
- 1654635
- 1655159
- 1655479
- 1655717
- 1656675
- 1657662
- 1658124
- 1660805*
- 1661027*
- 1661130
- 1661348*
- 1661438*
- 1661456*
- 1661470*
- 1661485*
- 1661513*
- 1661759*
- 1661762*
- 1663285*
- 1665479
- 1666244
- 1666253
- 1668242*
- 1670036*
- 1670042*
- A-1-2SUPPLEMENTAL INFORMATION Attachment 1
- Operating Training Procedures
- 45.OP.4705.101, Manual Transfer of First RFP to Flow Control Mode, Revision 0
- 45.OP.4706.151, Manually Placing Second RFP In-Service, Revision 0 45.OP.4819.151, Transferring RFPs during Low Load Valve Operations, Revision 1
- TM-OP-045I-ST, Reactor Feedwater Level Control System (ICS/DCS), Revision 3
Procedures
- DCAS-81, Failure Modes and Effects Analysis, Revision 1
- GDG-17, Failure Modes and Effects Analysis Design Guide, Revision 1
- GO-200-005, Plant Shutdown to Hot/Cold Shutdown, Revision 54
- MFP-QA-1220, Engineering Change Process Handbook, Revisions 12 & 14
- NDAP-QA-0002, Procedure Program, Revision 33
- NDAP-QA-0029, Procedure and Work Instructions Use and Adherence, Revision 20
- NDAP-QA-0032, Human Performance- Standards for Error and Event Prevention, Revision 14
- NDAP-QA-0300, Conduct of Operations, Revision 32
- NDAP-QA-0320, Special, Infrequent or Complex Test/Evolution, Revision 16
- NDAP-QA-0510, Troubleshooting Plant Equipment, Revision 6
- NDAP-QA-0702, Action Request and Condition Report Process, Revision 38
- NDAP-QA-0726, 10
- CFR 50.59 and 10
- CFR 72.48 Implementation, Revision 13
- NDAP-QA-1220, Engineering Change Process, Revision 8
- NDAP-QA-1902, Integrated Risk Management, Revision 10
- OI-AD-085, Operations Switching Orders, Revision 2
- OI-AD-096, Operator Burdens, Revision 8 (5,6,7)
- OI-TA-001, Event Report Data Collection and Retention, Revision 7
- ON-245(145)-004, RPV Water Level Anomaly, Revision 20
- OP-249-002, RHR Shutdown Cooling, Revision 52
- OP-AD-001, Operations Standards for System and Equipment Operation, Revision 49
- OP-AD-002, Standards for Shift Operations, Revision 41
- OP-AD-003, Shift Surveillance Scheduling, Log Sheets, Turnover Sheets and Rounds, Revision 45
- OP-AD-055, Operations Procedure Program, , Attachment F, Revision 14
- OP-AD-327, Post Reactor Transient/Scram/Shutdown Evaluation, Revision 26
- SO-293-001, Quarterly Turbine Valve Cycling, Revision 37
- TM-OP-045I-ST, Reactor Feedwater Level Control System (ICS/DCS), Revision 03
- TQ-210-3208, Just-In-TimeTraining (JIIT), Revision 0
Miscellaneous
- 50.59 Evaluation, Integrated Control System and RFPT Speed Control Unit 1 EC940986, Unit 2 EC864462 and EC910695
- A-1-3SUPPLEMENTAL INFORMATION Attachment 1
- Assistant Operations Manager (AOM) Directive 12-07 Revisions 1and 2 ERPM M1337-82, HV20603A MOV Inspection, Grease Sampling, and Stem Lube, due 7/09/13
- ERPM M1337-83, HV20651B MOV Inspection, Grease Sampling, and Stem Lube, due 11/2315
- Hot Box 12-48, Operations Phase I Package (Standards and Behaviors), 12/23/12
- Hot Box 12-49, Loss of Power to RFP Discharge Isolation Valves, 12/27/12 Hot Box 13-02, Operator Aggregate Index, 01/11/13
- HV-206-03A Quiklook Test Results File 12357D02, performed 12/22/12
- ICS Log for 12/19/12
- JIIT
- U2 CV Testing SCRAM Startup, Training Roster, dated 12/17/12
- LER 05-388/2012-002-00, Unit 2 Manual Scram Due to Loss of the Integrated Control System NRC Resident documentation of Interviews of Shift on December 21, 2012 Involved in the December 19, 2012, Scram Operations Logs for 08/23/11, 06/14/12, 11/09/12,12/16/12 and 12/19/12
- Operator Challenges NIMS Report for 12/20/12
- Operator Remedial Training, 12/19/12 Operator Training Needs Analysis, performed 12/21/12 Overview of ICS Training and Evaluation for Licensed Operators as of 4/7/10
- PPL Plant Computer Logs (PPC2A) for 12/16/12, 12/19/12
- RTPM V0005-03, HV20603A MOV Overhaul based on Grease Sample, due 12/31/99
- SC 02-12-01, Post Event Review Report, 11/09/12
- SC 12-12-02, Post Event Review Report, 12/16/12 Susquehanna Error Prevention Team Assessment Report (SEPTA -
- CR 1654158) 12/19/12 U1 and U2 Operator Aggregate Index for December 2012
- U1 and U2 Operator Aggregate Index for November 2012
- U2 Startup PORC for 11/16/12
- Investigations and Assessments
- CR 1640540, 11/09/12 IRT Investigation Results - Unit 2 Unplanned Shutdown Due to Loss of Integrated Control System
- CR 1656675, Risk Assessment for Susquehanna Unit 2 December 19, 2012 Shutdown Completed Surveillance and Modification Acceptance Tests
- CSD-FSIM-009a, WSC Unit 1 Simulator ICS Test, completed 5/2911
Work Orders
- 1640632
- 1470771
- 1456387
- 293989
- 880148
- A-1-4SUPPLEMENTAL INFORMATION Attachment 1
LIST OF ACRONYMS
- AT [[]]
WS anticipated transients without scram
CR Condition Report DPM Discharge Pressure Mode
EC engineering change
EWR engineering work request
FMEA failure modes and effects analysis FW feedwater
HMI human-machine interface
- JI [[]]
TT just in time training
NCV non-cited violation NRC Nuclear Regulatory Commission
OCC outage control center
- PO [[]]
RC plant operations review committee
PPC plant process computer
PP&L Pennsylvania Power and Light
- UFS [[]]
AR Updated Final Safety Analysis Report
- SP [[]]
AR standardization plant analysis risk
SME subject matter expert
US unit supervisor
A-2-2
- UNITED [[]]
- STATES [[]]
- COMMIS [[]]
- SION [[]]
- BOULEV [[]]
- SUI [[]]
- KING [[]]
- OF [[]]
- MEMORA [[]]
NDUM TO: Peter Wilson, Senior Sponsor Special Inspection Team Donald Jackson, Manager Special Inspection Team Peter Presby, Leader
Special Inspection Team
- INSPEC [[]]
- TION [[]]
- TEAM [[]]
- TO [[]]
- SUSQUE [[]]
- HANNA [[]]
- STEAM [[]]
- UNPLAN [[]]
- NED [[]]
- ON [[]]
DECEMBER 19, 2012 In accordance with Inspection Manual Chapter (IMC) 0309, "Reactive Inspection Decision Basis
for Reactors," a Special Inspection Team (SIT) is being chartered to evaluate operator
performance and decision making regarding the Susquehanna Steam Electric Station Unit 2
unplanned reactor shutdown on December 19, 2012. The SIT will expand on the event follow-up inspection activities started by the resident and region based inspectors.
The decision to conduct this special inspection was based upon meeting two deterministic
criteria of IMC 0309 related to repetitive operational performance concerns. A conditional risk
assessment of the event determined plant risk was low within the overlap region where either baseline inspection or a Special Inspection shall be considered. A Special Inspection was recommended due to the repetitive nature of operational problems related to reactor vessel
level control, which have contributed to three valid Reactor Protection System actuations
between November 9 and December 19, 2012, repetitive issues with the operation of the
Integrated Control System (ICS), and specific questions related to conservative operational
decision making that led to the December 19, 2012 unplanned reactor shutdown. While these conditions did not affect safety related equipment, they have resulted in plant transients which unnecessarily challenged safety related systems. The recent unplanned plant shutdowns raise
A-2-2 questions regarding conservative decision making, operator knowledge, training and procedures for the ICS system and equipment reliability. A special inspection is warranted to understand
the scope of the issues and the licensee's actions to address these problems. The inspection will be conducted in accordance with the guidance of NRC Inspection Procedure 93812, "Special Inspection," and the inspection report will be issued within 45 days
following the final exit meeting with the licensee.
The special inspection will commence on January 14, 2013. The following personnel have been assigned to this effort:
Senior Sponsor: Peter Wilson, Deputy Director Division of Reactor Projects Region I Manager: Donald Jackson, Chief Operations Branch, Division of Reactor Safety
Region I Team Leader: Peter Presby, Senior Operations Engineer
Division of Reactor Safety
Region I
Full Time Members: John Richmond, Senior Reactor Inspector Engineering Support Branch 3 Division of Reactor Safety
Region I
Justin Hawkins, Resident Inspector (Limerick) Division of Reactor Projects Region I
Enclosure: Special Inspection Team Charter
A-2-3SPECIAL
- INSPEC [[]]
- TION [[]]
- TEAM [[]]
CHARTER Attachment 2 Special Inspection Team Charter to Evaluate Susquehanna Steam Electric Station Unit 2 Unplanned Reactor Shutdown on December 19, 2012 A. Background On December 19, 2012 at 1731, the Susquehanna Steam Electric Station Unit Two reactor automatically shut down from 18% power on low reactor pressure vessel level.
The plant response was uncomplicated in that all safety systems responded as
designed. At the time of the scram, the 'A' reactor feed pump (RFP) was in service.
Operators were transitioning the 'A' RFP from discharge pressure mode of feedwater
control to flow control mode. During this transition, operators observed that feedwater valve 3A did not automatically open as expected. Operators proceeded to open valve 3A's power supply breaker to the valve motor operator and intended to subsequently
manually cycle the valve off its seat using the handwheel. When the power supply
breaker was opened, feedwater valves 41 and 51A unexpectedly closed, resulting in a
loss of all feedwater and an automatic reactor shutdown.
SIT: The IMC 0309 review concluded the following two deterministic criteria were met. The
event involved repetitive failures or deficiencies in operations and involved questions pertaining to licensee operational performance. The conditional risk assessment determined risk was at 2.2 E-6 using the Susquehanna Unit
IMC 0309, this places the event in the overlap region
where either baseline inspection or a Special Inspection shall be considered.
A Special Inspection was recommended due to the repetitive nature of operational problems related to reactor vessel level control, which have contributed to three valid Reactor Protection System actuations between November 9 and December 19, 2012,
repetitive issues with the operation of the Integrated Control System (ICS), and specific questions related to conservative operational decision making that led to the December
19, 2012 unplanned reactor shutdown. Although the ICS system is a non-safety system,
improper operation of this system can be a transient initiator and complicate the plant response to a transient.
NRC inspectors completed interviews with the operators involved in the December 19, 2012 unplanned reactor shutdown and observed the operator training sessions
conducted on December 22 and 23, 2012 prior to plant restart. The interviews confirmed that on December 19, 2012, when the 3A valve did not automatically open as expected, operators were not exercising this valve in accordance with the system operating procedure, or using any formal written process (such as a troubleshooting
plan) when they opened the valve 3A power supply breaker. The interviews also
indicated that the operators did not discuss whether it was appropriate to attempt to manually operate the valve with the control system and valve in automatic at the time.
A-2-4SPECIAL
- INSPEC [[]]
- TION [[]]
- TEAM [[]]
CHARTER Attachment 2 C. Scope
The scope of special inspection is as follows:
Develop a complete sequence of events which resulted in the December 19, 2012 unplanned reactor shutdown, including, but not limited, prior related equipment problems, operator just-in-time training, pre-job briefs, use of human performance error prevention techniques, and licensee management oversight. Evaluate operator performance and management oversight regarding operator crew roles and responsibilities and conservative decision making when an unexpected system response occurred on December 19, 2012. Evaluate the effect on this event of operator training, procedures, and simulator fidelity associated with the
- ICS. Evaluate just-in-time operator training and procedures changes to prepare operators to respond to
ICS issues that occurred
during the November 9, December 16, and December 19, 2012 unplanned reactor
shutdowns. Determine the effectiveness of the corrective action program and operator work around program with respect to
ICS modification was installed in 2010 (Unit 1) and 2011 (Unit 2). Review a sample of motor operated valve problems, similar to those encountered on December 19, 2012, in other plant
systems which could result in plant transients. Include insights from the March 2012
NRC Inspection Procedure 95002 Supplemental Inspection. Identify any potential generic safety issues and make recommendations for appropriate follow up action (e.g. Information Notices, Generic Letters, and Bulletins). D. Guidance Inspection Procedure 93812, "Special Inspection," provides guidance to be implemented
by the team. The inspection should emphasize fact-finding in its review of the circumstances surrounding the event. Safety concerns identified that are not directly related to the event should be reported to the Region I office for appropriate action.
The team will report to the site, conduct an entrance meeting, and begin inspection no
later than January 14, 2013. While onsite, the Team Leader will provide daily briefings
to Region I management, who will coordinate with the Office of Nuclear Reactor Regulation to ensure that all other pertinent parties are kept informed. The Team Manager shall provide a recommendation as to whether the SIT should be upgraded to
an Augmented Inspection Team in accordance with IMC 0309. A report documenting
the results of the inspection should be issued within 45 days of the completion of the
inspection. This Charter may be modified should the team develop significant new information that
warrants review. Should you have any questions concerning this Charter, contact Don Jackson at (610) 337-5306.
A-3-1DETAILED
- SEQUEN [[]]
- CE [[]]
- OF [[]]
- EVENTS [[]]
- AND [[]]
- TIMELI [[]]
NE Attachment 3 The following is a detailed sequence of events and integrated timeline developed and verified by the inspectors. The timeline includes other related Integrated Control System (ICS) milestones,
equipment issues, other documented NRC inspection conclusions and recent transients on Unit
up to and including the unplanned reactor scram on December 19, 2012. Timeline: 01/22/10 The
FMEA) identified numerous single point vulnerabilities. Specifically, the reactor feed pump (RFP) discharge
isolation valve position input into ICS was identified as a single point
vulnerability wherein a loss of position status could result in a loss of startup level control and a subsequent loss of feedwater (FW) flow. (Missed Barrier - System Design Weakness) The
- FM [[]]
EA stated that this vulnerability would be mitigated because an operator
could maintain system operation in manual control. (Missed Barrier - No operator guidance / procedure revision)04/01/10 Unit 1 (U1)
PPL determined that the root cause for the April 22 and May 14, 2010, RPV
water level transients identified less than adequate engineering rigor and
incorrect vendor input into the simulator model for
- RPV water level transients than the plant therefore modeling a more stable plant response in the simulator. (See
NRC Inspection Reports (IRs) 2010004 and 2012 95002) 03/31/11 NRC resident inspectors documented that changes to the Operator Burdens program had not been effectively managed and that identification of Operator
Burdens was not complying with the associated procedure (OI-AD-096). (See
- NRC [[]]
- IR [[]]
ICS installed. 08/19/11 U2 scrammed while performing quarterly functional surveillance testing on Reactor Water High Level trip channels. PPL determined that an improperly
terminated trip channel initiated a Reactor Protection System (RPS) actuation.
Inspectors determined that inadequate post modification testing procedures, including no control scheme testing and inadequate functional testing, led to the
U2 scram. (See
- NRC [[]]
IR 2011005) 08/23/11 U2 startup: 0844 U2 transitioned to Mode 1; Mode switch placed in Run.
A-3-2DETAILED
- SEQUEN [[]]
- CE [[]]
- OF [[]]
- EVENTS [[]]
- AND [[]]
- TIMELI [[]]
- HV -206-03A (3A) valve did not open. The 3A motor operated valve (MOV) thermal overload relay were found tripped and reset by operators. Operators then placed the 'A'
FW) valve
controls in Manual. 1206 Operators attempted to open the 3A valve using ICS in manual control mode and thermals tripped again. The operator opened the 3A valve power supply
breaker and used the local MOV handwheel to crack the 3A valve off its closed
seat. Thermals were then reset and the breaker was closed. The 3A valve then travelled to the open position. 1216 Operators closed the
- LV -206-41 (41) and placed it in Auto with setpoint of 18 inches. Manual valve control was used to close the
HV-206-51A (51A). Operators awaited
engineering verification from the Operations System Matter Expert (SME), a
licensed reactor operator, to return the 'A'
- RFP [[]]
FW valve control to Auto. 1331 Operators restored the 'A' RFP valve controls to Auto.
Note: After reviewing the 3A valve issue, the Operations
ICS valve controls needed to be selected to Manual prior to opening the 3A valve power supply breaker to prevent ICS from incorrectly interpreting a loss of
power at the 3A MOV as a valve full open indication which would in turn result in
a loss of
FW valves 41 and 51A). No condition report was
generated to address this issue. (Missed Barrier - Failure to use the
- PCWO 1456387 was created for the 3A valve binding issue experienced on 08/23/11. Operations added this issue to the Operator Challenge as part of the Operator Burdens Procedure,
OI-AD-096.
Note: No workaround instructions or compensatory actions were given to the operators concerning this operator challenge until after the event on 12/19/12
which was contrary to the guidance provided in the Operator Burdens Procedure. (Missed Barrier - Failure to Follow Procedure)11/30/11
QA identifies and generates a CR (1488273) to address gaps in the Operator Burdens procedure, specifically related to identifying and addressing
potential burdens. 12/09/11 The Work Control Center documented in
- PCWO 1456387 that the 3A valve cannot be investigated or manipulated until a forced or scheduled outage. 10/18/12
PCWO 1456387 to the scheduled U2 turbine outage in November 2012. The work was not performed during the
November outage. (Missed Barrier - Ineffective use of
- CAP ) 11/09/12 Summary: The U2 scram at 0117 on November 9, 2012, involved a latent deficiency related to
- ICS lockup causing the operators to insert a manual reactor scram. Following the scram, operators established
NR) RPV water level indication locked up at a pre-scram level of 33 inches. The operator assigned to level control, monitored
A-3-3DETAILED
- SEQUEN [[]]
- CE [[]]
- OF [[]]
- EVENTS [[]]
- AND [[]]
- TIMELI [[]]
- SPDS [[]]
- NR [[]]
- NR [[]]
- RPV water level indications believed to be available to the operator at this time were located on the
- WR /NR divergence at normal operating pressure post-scram, but did not fully understand that the divergence would increase with time as reactor pressure decreased during plant cool down (Knowledge Gap #2). Per the scram off normal procedure, operators changed the
- RPV water level control band to 13 - 30 inches, which did not allow the operator controlling level enough margin to maintain
- RPV water level. (Missed Barriers - Operator Knowledge Gaps)0117 U2 scram @ 90 percent power due to an
- RPV water level of 15 inches. 11/13/12 Prior to a scheduled U2 turbine outage, operations generated a
- CR 1632449 with an action to not work the 3A valve based on the 3A valve properly working in multiple startups since the initial binding issue on 08/23/11. (Missed Barrier - Ineffective
RFP trip. 12/05/12 The Operations Aggregate Assessment Committee held a quarterly meeting to assess the Operator Burdens program. No formal actions were documented in
the CAP process to changes that needed to be made in the Operator Burdens
process. (Missed Barrier - Ineffective Process)12/16/12 Summary: The U2 scram at 0155 on December 16, 2012, occurred during testing of the control valve (CV) 2. During the testing of
- FW system to initiate level setpoint setdown to control level at approximately 18 inches through the 41 valve. The scram procedure,
- ON [[-200-101, has immediate actions for the operators following the scram. Section 5 of this procedure details subsequent operator actions to recover from the scram. These actions may be performed in any order based on the nature of the event. In completing these subsequent steps to the procedure, operators performed steps to prevent]]
- ON -200-101, the order of preference for performing actions to prevent stratifications lists the number one method as raising
RPV water level to 45 - 54 inches. Operators did not prioritize the
A-3-4DETAILED
- SEQUEN [[]]
- CE [[]]
- OF [[]]
- EVENTS [[]]
- AND [[]]
- TIMELI [[]]
- NE Attachment 3 number one method ahead of the other 5 methods, which included resetting the scram, minimizing
- CRD flow, and restarting a recirculation pump. (Knowledge Gap #1) This lack of performing the methods in the preferred manner delayed the operators in resetting setpoint setdown and raising
- RPV water level. The operators also did not recognize that the Reset Setpoint Setdown button needed to be pressed prior to raising level setpoint per
- OP -245-001, Att. A, Placing Feedwater in Startup Level Control Following a Scram. (Knowledge Gap #2) At 0208, U2 received a second scram signal on low
- CC is scheduled to be completed in February 2013. (Missed Barriers - Operator Knowledge Gaps)0155 U2 scram during testing of
- JITT held for the 12/19/12 startup. Only 3 of 8 operators on shift at the time of the 12/19/12 loss of
JITT operators received on 12/17/12. (FW training
listed as optional.) (Missed Barrier - Ineffective
- JITT )12/18/12 The Operator Aggregate Index for both Units contained a total of 5 operator workarounds (
OWAs), 4 operator challenges, and 8 control room deficiencies.
This list included the 3A valve but no procedural or compensatory guidance was developed to deal with this challenge as required by the "Operator Burden" procedure. (Missed Barrier - Failure to Implement the Procedure)12/19/12 Summary: At 1731 on December 19, 2012, U2 scrammed at 18 percent power with the main generator on the grid. Operators were swapping the 'A'
- RFP [[3A valve failed to open during this transition. Operators were sent to the 3A valve and its power supply breaker (2B142 Cub 074) with the intent of de-energizing the breaker and then manually opening the 3A valve off its closed seat. This decision making was based, in part, on the previous 3A valve issue with getting this valve to open on August 23, 2011. (Note: The Shift Manager (]]
- SM ) for the 08/23/11, 3A valve binding issue was also the Shift Manager for this event and the owner of the Operator Burdens program.) When the operator opened the 3A power supply breaker, the 3A valve remained closed and the
- RFP [[]]
- RPV water level rapidly lowered and before operators could take the mode switch to shutdown, a scram occurred on low
- RPV water level of 13 inches. The 3A valve power supply breaker was reclosed and operators were able to place the 'A'
RFP back in DPM to restore level
A-3-5DETAILED
- SEQUEN [[]]
- CE [[]]
- OF [[]]
- EVENTS [[]]
- AND [[]]
- TIMELI [[]]
FCM. 1711 Operators noticed 3A valve failed to open. 1715 Operators discussed opening the 3A valve breaker and cracking the valve off its closed seat manually. The SM made the decision to open the breaker. No written procedure or process is entered at this time to conduct this operation
with the 3A valve. (Missed Barrier - Failure to use a written process/procedure)1720 The
- OCC ) which confirmed the 3A valve not opening was a repeat issue from the operator challenge list. 1730 The
- OCC to inform them of the decision to manually open 3A valve by opening breaker. The 3A valve power supply breaker is opened by operators with the
- RPV [[]]
- WATER [[]]
- LEV [[]]
EL level to lower. The field operator recloses the 3A valve breaker. The control room
operator attempts to place the 'B' RFP in-service. The operator is unable to
open the 51B valve due to system design also the operator incorrectly used the
Hard Card,
- DPM. ) (Missed Barrier - Inadequate Procedure)(Missed Barrier - Operator Knowledge Gap)1731:32 U2 scram on
- RPV low level. (Maximum cool down rate of 90 deg F/hr from 17:31 to 18:31.) 1731:35 Operators place the mode switch in Shutdown 1731:58
- RPS [[]]
- RPS [[]]
- WR [[]]
RPV level - 28.7" (minimum level reached). 12/20/12 Assistant Operations Manager issued Directive 12-07 Rev 1 - 'During normal operations plant components shall be operated only with written guidance. During transient conditions, components may be operated without written
guidance on direction of the control room.'
The action to open the breaker and manually open the valve on the only in-
service pump was not within procedural guidance for placing the system into
ICS modification may have changed its design.
A-3-6DETAILED
- SEQUEN [[]]
- CE [[]]
- OF [[]]
- EVENTS [[]]
- AND [[]]
- TIMELI [[]]
- PORC. [[The results for the diagnostic testing on 3A valve determined the disc pullout was not smooth (spikes in torque, current and thrust were observed). The 3A valve stem was lubed during troubleshooting but condition remained. Actions for startup included revising]]
FW operating
procedures to include directions for the in-service FW pump discharge isolation
valve issues (AR 1654543; 1654418; 1654551). 12/26/12 Operations revised
RFP and RFP Lube Oil, to contain guidance to operators regarding how to respond to a loss of power to the 3A valve when
transitioning to
- FCM. 01/11/13 Operations Hot Box 13-02, Operator Aggregate Index, issued. The Operator Aggregate Index is expected to be red for both Units for January 2013 due to additional
PPL extent of condition investigation as a result of the 12/19/12 U2 RX scram.
The investigation yielded a significant increase to the 12/18/12 Operate
Aggregate Index numbers (11 undocumented
- OWA s, 21 undocumented operator challenges, and 19 undocumented control room deficiencies). 01/23/12 The inspectors determined that the revised procedural guidance in
OP-245-001 from 12/26/12 did not contain guidance on how to respond to a closure of the 51A valve if closure of the 41 valve is not recognized by
the operator within 10 seconds. In response to the inspectors'
observation,
CR 1663285 to address this procedural
issue. (Missed Barrier - Inadequate Procedure)
A-4-1SIMPLIFIED
- SYSTEM [[]]
DRAWINGSAttachment 4
A-4-2SIMPLIFIED
- SYSTEM [[]]
DRAWINGSAttachment 4
A-4-3SIMPLIFIED
- SYSTEM [[]]