IR 05000387/2014008
ML14255A250 | |
Person / Time | |
---|---|
Site: | Susquehanna |
Issue date: | 09/12/2014 |
From: | Bower F L Reactor Projects Region 1 Branch 4 |
To: | Rausch T S Susquehanna |
Bower F L | |
References | |
EA 11-221 IR 2014008 | |
Download: ML14255A250 (47) | |
Text
September 12, 2014
EA 11-221 Mr. TimothySenior Vice President and Chief Nuclear Officer PPL Susquehanna, LLC 769 Salem Boulevard, NUCSB3 Berwick, PA 18603
SUBJECT: SUSQUEHANNA STEAM ELECTRIC STATION - NRC 95002 SUPPLEMENTAL INSPECTION REPORT 05000388/2014008 AND FOLLOW-UP ASSESSMENT LETTER
Dear Mr. Rausch:
The U.S. Nuclear Regulatory Commission (NRC) staff conducted a supplemental inspection using Inspection Procedure (IP) 95002, "Inspection for One Degraded Cornerstone or Any Three White Inputs in a Strategic Performance Area," at your Susquehanna Steam Electric Station Unit 2 (Unit 2) from July 14, 2014 through July 31, 2014. This letter summarizes the results of that inspection and describes our previous and ongoing assessment of Unit 2. Based on our ongoing review of performance indicators and inspection results, the NRC is also updating its assessment of Unit 2 and providing you this follow-up assessment letter that informs you of the NRC's assessment of your facility and supplements, but does not supersede, the mid-cycle letter (ML14239A210) issued on September 2, 2014.
On March 20, 2013, your staff reported that the Unplanned Scrams with Complications performance indicator crossed a threshold from Green to White for your Unit 2. Based on your report, the NRC staff assigned a White performance indicator Action Matrix input to the Initiating Events cornerstone in the fourth quarter of 2012. In response to this Reactor Oversight Program Action Matrix input, the NRC informed you by letter (ML13092A011) that Unit 2 entered the regulatory response column of the Action Matrix and that a supplemental inspection using IP 95001, "Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area," would be required. The results of the 95001 inspection were documented in Inspection Report 05000388/2013011 (ML13322B321).
On October 24, 2013, your staff reported that the Unplanned Scrams per 7000 Critical Hours performance indicator for Unit 2 crossed the threshold from Green to White. Based on your report, the NRC assigned a second White performance indicator Action Matrix input to the Initiating Events cornerstone in the third quarter of 2013. In response to these two Reactor Oversight Program Action Matrix inputs, the NRC informed you by letter (ML11310A301) that Unit 2 entered the degraded cornerstone column of the Action Matrix and that a supplemental inspection using IP 95002 would be required.On May 29, 2014, you informed the NRC that Unit 2 was ready for the supplemental inspection.
On July 31, 2014, the NRC completed the 95002 supplemental inspection and the NRC inspection team discussed the results of this inspection with Mr. J. Franke, Site Vice President, and other members of your staff. The inspection team documented the results of this inspection in the enclosed inspection report. On September 3, 2014, the NRC conducted a public exit meeting for the 95002 inspection and conducted a public regulatory performance meeting.
During this meeting, the NRC discussed the implementation of the corrective actions that were developed in response to PPL's White performance indicators and subsequent entry into the degraded cornerstone column of the Reactor Oversight Program Action Matrix with Mr. R. Frannsen, General Manager of Operations, and other members of your staff.
The NRC performed this 95002 supplemental inspection to determine if: (1) the root and contributing causes for the significant issues were understood, (2) the extent of condition and extent of cause for the identified issues were understood, and (3) your completed or planned corrective actions were sufficient to address and prevent repetition of the root and contributing causes. The NRC also conducted an independent review of the extent of condition and extent of cause for the White performance indicators and an assessment of whether any safety culture component caused or significantly contributed to the performance issue. The NRC determined that your staff's evaluation identified the primary root causes of the performance issues to be: Less than adequate decision making related to Integrated Control System design scope and operating experience (OE) reviews resulted in Unit 2 scrams on November 9, 2012; December 16, 2012; December 19, 2012; and September 14, 2013; and Less than adequate management reinforcement and personal accountability for information-use procedure use and adherence in Operations and Engineering resulted in Unit 2 scrams on November 9, 2012; December 16, 2012; December 19, 2012; and September 14, 2013. The inspectors determined that your staff identified numerous corrective actions to address the root causes of the performance issues that caused the two White performance indicators.
These corrective actions included:
Performing a review of previously reviewed OE to ensure the quality of the OE evaluations; Establishment of a monitoring program to track timeliness and quality of OE reviews; Implementing industry best practices regarding formatting of procedures; Implementing Operations Department supervisory observations of operating crews regarding procedure use and adherence; Development of the "Accountability with Respect" model to improve accountability within the SSES organization; and Development and implementation of internal performance indicators to track progress of corrective actions.
The inspectors determined that the root cause evaluations and the common cause assessment were conducted to a level of detail commensurate with the significance of the performance issues that led to entry into the degraded cornerstone column of the Reactor Oversight Program Action Matrix. The inspectors determined that the cause evaluations associated with thisinspection appropriately considered safety culture components as they related to the various root and significant contributing causes. In addition to satisfactorily completing the supplemental inspection and public performance assessment meeting, we note that the Unplanned Scrams with Complications and the Unplanned Scrams per 7000 Critical Hours performance indicators returned to Green in the fourth quarter of 2013. Therefore, all the findings and performance indicators that input into the Action Matrix are Green. Accordingly, per the guidance in Inspection Manual Chapter (IMC)
0305, "Operating Reactor Assessment Program," Unit 2 will transition to the Licensee Response Column (Column 1) of the NRC's Action Matrix as of the date of this assessment follow-up letter.
During the 95002 supplemental inspection, the NRC inspectors documented one finding of very low safety significance (Green) in this report that involved a violation of NRC requirements.
Because of the very low safety significance, and because it is entered into your corrective action program, the NRC is treating this finding as a non-cited violation, consistent with Section 2.3.2.a of the NRC Enforcement Policy. If you contest the violation or significance of the non-cited violation, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington DC 20555-0001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the Susquehanna Steam Electric Station. If you disagree with the cross-cutting aspect assignment in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region I, and the NRC Resident Inspector at the Susquehanna Steam Electric Station In accordance with Title 10 of the Code of Federal Regulations 2.390, "Public Inspections, Exemptions, Requests for Withholding," of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in theNRC's Public Document Room or from the Publicly Available Records (PARS) component of the NRC's Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,/RA/ Fred L. Bower III, Chief Reactor Projects Branch 4 Division of Reactor Projects Docket No. 50-388 License No. NPF-22
Enclosure:
Inspection Report No. 05000388/2014008 w/
Attachment:
Supplemental Information cc w/encl: Distribution via ListServ
SUMMARY
Inspection Report (IR) 05000388/2014008; 07/14/2014 - 07/31/2014; Susquehanna Steam Electric Station, Unit 2; Supplemental Inspection - Inspection Procedure (IP) 95002.
A senior resident inspector, two resident inspectors, one senior reactor inspector, one senior project engineer qualified as a human factors engineer, two project engineers, and one reactor inspector performed this inspection. The inspectors identified one finding of very low safety significance (Green), which was determined to be a non-cited violation (NCV). The significance of most findings is indicated by their color (i.e., greater than Green, or Green, White, Yellow, Red) and determined using Inspection Manual Chapter (IMC) 0609, "Significance Determination Process (SDP)," dated June 2, 2011. Cross-cutting aspects are determined using IMC 0310, "Aspects Within the Cross-Cutting Areas," dated December 19, 2013. All violations of NRC requirements are dispositioned in accordance with the NRC's Enforcement Policy, dated July 9, 2013. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 5, February 2014.
Cornerstone: Initiating Events
The NRC staff performed this supplemental inspection in accordance with IP 95002, "Supplemental Inspection for One Degraded Cornerstone or Any Three White Inputs in a
Strategic Performance Area," dated February 9, 2011, to assess PPL's root and common cause evaluations associated with Susquehanna Unit 2's (Unit 2) entry into the Degraded Cornerstone
Column of the Reactor Oversight Program (ROP) Action matrix due to two White performance indicators (PI) in the Initiating Events cornerstone. Specifically, the Unplanned Scrams with Complications PI crossed the Green/White threshold during the 4th quarter of 2012 and the Unplanned Scrams per 7000 Critical Hours crossed the Green/White threshold during the 3rd quarter of 2013. In the fourth quarter of 2013, both PIs returned from White back to
- Green.
This supplemental inspection focused on six individual scrams which occurred in four separate events. The first two scram events consisted of a scram while the reactor was critical and a second scram while the reactor was shutdown and resulted in the Unplanned Scrams with
Complications PI exceeding the Green/White threshold during the 4th quarter of 2012. It is noted that the second scrams while shutdown did not count towards the PI calculations. The second two scram events occurred while Unit 2 was critical and, in conjunction with the previous two scram with complications events, resulted in PPL crossing the Green/White threshold for the Unplanned Scrams per 7000 Critical Hours PI in the 3rd quarter of 2013. PPL was informed of Unit 2's entry into Column 3 of the ROP action matrix and the NRC's intention to perform a 95002 Supplemental Inspection in a follow-up assessment letter dated November 5, 2013 (ML11310A301)1. PPL conducted a collective review of all the unplanned scrams and identified two root causes for the risk significant performance issues. The root causes were: Less than adequate decision making related to Integrated Control System (ICS) design scope and operating experience (OE) reviews resulted in Unit 2 scrams on November 9, 2012; December 16, 2012; December 19, 2012; and September 14, 2013.
Designation refers to an ADAMS accession number. Documents referenced are publicly available using the accession number in ADAMS
Enclosure Less than adequate management reinforcement and personal accountability for information-use procedure use and adherence in Operations and Engineering resulted in Unit 2 scrams on November 9, 2012; December 16, 2012; December 19, 2012; and
September 14, 2013.
PPL has taken significant and numerous corrective actions to address the risk significant performance deficiencies. The inspectors concluded that, based on these corrective actions and other actions taken to date, PPL has fully addressed the causes and conditions that led to the two Unit 2 White PIs and have provided the assurance level required to meet the inspection objectives defined in IP 95002.
- Green.
A self-revealing Green NCV of Technical Specification (TS) 5.4.1, "Procedures," was identified because PPL did not adequately maintain operating procedures for plant shutdown to hot standby. Specifically, the general operating procedure for plant shutdown to minimum power and the reactor feed pump (RFP) operating procedure were revised, and the technical reviews did not adequately verify the functional and technical adequacy of the procedures. The technical reviews did not identify a valve lineup conflict existed between the two procedures. The conflict resulted in an improper feed lineup to the reactor pressure vessel (RPV) causing two level transients and corresponding power transients of approximately five percent on March 20, 2014. PPL's corrective actions included mitigating the pressure vessel level transients, collecting personnel statements, revising the general operating procedure to remove the valve conflict, initiating an apparent cause evaluation to determine the cause of the level transients, resetting the Operations Department human performance clock due to operator performance issues during the event, reviewing the event with every Operations Department shift crew, performing a standdown for the Operations Department to compare operator performance issues between this event and the December 19, 2012 scram event, as well as entering the events of March 1, 2014, and March 20, 2014, into the corrective action program as condition report (CR)-2014-08941 and CR-2014-10388. The inspectors determined that PPL's inadequate maintenance of procedures for plant shutdown to hot standby was more than minor, because it is associated with the procedure quality attribute of the Initiating Events cornerstone and affected the cornerstone objective of limiting the likelihood of those events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the inadequate technical reviews associated with the revision of procedures for placing standby RFPs into service in startup level control and valve control (manual), and procedures for placing standby RFPs into service in the flow control mode (FCM) and valve control (manual) mode, resulted in two reactor power transients up to five percent and two significant reactor vessel water level transients which challenged the stability of the plant. Additionally, this issue is similar to Example 4b described in IMC 0612, Appendix E, "Examples of Minor Issues,"
which states that issues are not minor if procedure issues cause a reactor trip or other transient. The inspectors evaluated the finding using Attachment 0609.04, "Initial
Characterization of Findings," worksheet to IMC 0609, "Significance Determination Process," issued June 2, 2011. The attachment instructs the inspectors to utilize IMC 0609, Appendix A, "Significance Determination Process for Findings At-Power," issued June 19, 2012. The inspectors determined this finding did not cause a reactor trip and the loss of mitigation equipment relied upon to transition the plant from the onset of the trip to a stable shutdown condition (e.g. loss of condenser, loss of feedwater) and is therefore of very low safety significance (Green). A cross-cutting aspect was assigned in the area of Human
Enclosure Performance, Change Management, because leaders did not use a systematic process for evaluating and implementing change so that nuclear safety remained the overriding priority. Specifically, PPL did not maintain a clear focus on nuclear safety when implementing changes to the general operating procedure for shutdown to minimum power and this resulted in an unintended procedure discrepancy [H.3]. (Section 4OA4.02.03.f.1)
Enclosure
REPORT DETAILS
4. OTHER ACTIVITES
4OA3 Follow-up of Events and Notices of Enforcement Discretion
.1 (Closed) Licensee Event Report (LER) 05000388/2012-003-01: Unit 2 Automatic Scram While Performing Turbine Control Valve Surveillance Testing
a. Inspection Scope
On December 16, 2012, at 1:56 a.m., the Unit 2 reactor automatically scrammed during the performance of quarterly channel functional test of the turbine control valve fast closure channels of the Reactor Protection System (RPS). Both the high pressure coolant injection (HPCI) and the reactor core isolation cooling (RCIC) systems automatically initiated. At approximately 2:10 a.m. on December 16, 2012, a second reactor scram signal was received due to reactor water level lowering to +13 inches when Operations attempted to raise the Feedwater ICS reactor water level setpoint setdown value from +18 inches to +35 inches. The scram signals and other associated system actuations were reported in accordance with Title 10 of the Code of Federal Regulations (10 CFR) 50.72(b)(2)(iv)(A), 10 CFR 50.72(b)(2)(iv)(B), and 10 CFR 50.72(b)(3)(iv)(A) in event notice 48598. These events are also reportable as a LER in accordance with 10 CFR 50.73(a)(2)(iv)(A).
The LER was reviewed for accuracy, the appropriateness of corrective actions, historical equipment OE, violations of requirements, and generic issues. Based on laboratory results from the disassembly and inspection of the #1 turbine control valve, no direct cause of the unexpected RPS scram signal was identified. The root cause of the initial scram, as reported in Revision 0 of the LER, was that PPL failed to incorporate industry best practices for reducing the number of partial scrams received during testing, as discussed in the Boiling Water Reactor Owners Group (BWROG) scram reduction effort. Corrective actions identified within the LER consisted of replacing the fast acting solenoid valve and shutoff valve for the #1 turbine control valve and reviewing the extent of condition need for replacement on other valves. Additionally, PPL revised quarterly turbine valve cycling surveillance procedures to require the use of an RPS test box to limit the likelihood of receiving a scram during future tests. The inspectors noted that receiving a partial scram signal during testing was within the licensing basis of the plant and therefore viewed this action as an enhancement to improve plant reliability. The two root causes of the second reactor low water level scram, as reported in Revision 1 of the LER, were: (1) that plant operators did not perform Step 10 of OP-245-001, "Reactor Feed Pump (RFP) and RFP Lube Oil System", and (2) that the ICS design control value of +18 inches for setpoint setdown did not provide adequate margin to prevent operational overlap with the RPS low level scram setpoint of +15 inches. Corrective actions identified within the LER consisted of revising station procedures and hard cards to specify water level bands that would not create a reactor low water level scram and to provide direction for resetting the ICS setpoint setdown, performing training on the event for all licensed operators, and implementation of changes to the Unit 1 and Unit 2 ICS setpoint setdown value to raise the setpoint from +18 inches to +22 inches.
7 Enclosure The inspectors reviewed Revision 0 to the LER in IR 05000388/2013011 (ML13322B321). No findings were identified. PPL supplemented the LER because the final causal analysis of the event was not complete at the time of the original LER submittal. The supplement to the original LER provided revised information regarding the causes and completed corrective actions as well as additional clarifying information.
The inspectors reviewed this revision to the LER and did not identify any new performance deficiencies or violations. This LER is closed.
b. Findings
No findings were identified.
4OA4 Supplemental Inspection
01 Inspection Scope The NRC staff performed this supplemental inspection in accordance with IP 95002 to assess PPL's evaluation associated with two white PIs which affected the initiating events cornerstone in the reactor safety strategic performance area. The objectives of the 95002 inspection were to: Provide assurance that root and contributing causes of individual and collective risk-significant performance issues were understood; Provide assurance that the corrective actions for risk-significant performance issues are sufficient to address the root and contributing causes and prevent recurrence; Independently assess and provide assurance that extent of condition and the extent of cause of individual and collective risk significant performance issues are understood; and Independently determine if safety culture components caused or significantly contributed to the performance issues. Unit 2 entered the Regulatory Response Column of the NRC's ROP Action Matrix because one PI was of low to moderate safety significance (White) in the Initiating Events cornerstone. Specifically, the "Unplanned Scrams with Complications" PI crossed the Green to White threshold value in the fourth quarter of 2012. The NRC conducted a supplemental inspection, IP 95001, "Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area," from September 13, 2013 through September 29, 2013, in response to the White Unplanned Scrams with Complications PI. The results of the 95001 inspection were documented in Susquehanna Unit 2 Inspection Report 05000388/2013011 (ML13322B321).
Although successfully completing a supplemental inspection under IP 95001 typically would have allowed Unit 2 to return to the Licensee Response Column from the Regulatory Response Column of the ROP Action Matrix when the associated PI returned to Green, Unit 2 experienced additional unplanned scrams on December 19, 2012, and September 14, 2013. These two scrams caused the "Unplanned Scrams per 7000 Critical Hours" PI to cross the Green to White threshold value. Therefore on October 24, 2013, based on information provided by PPL, the NRC published a White PI for the
"Unplanned Scrams per 7000 Critical Hours" PI for the third quarter of 2013. This PI result, in conjunction with the earlier White "Unplanned Scrams with Complications" PI 8 Enclosure required the issuance of an assessment follow-up letter. Unit 2 entered the Degraded Cornerstone Column of the NRC's ROP Action Matrix in the third quarter of 2013 as a result of the two White inputs in the Initiating Events cornerstone. PPL was informed of Unit 2's entry into the Degraded Cornerstone Column and the NRC's intention to perform a 95002 Supplemental Inspection in a letter dated November 5, 2013 (ML11310A301).
PPL informed the NRC staff on May 29, 2014, of their readiness for the 95002 supplemental inspection.
In preparation for this inspection, PPL performed a common cause analysis (CCA) for entry into the Degraded Cornerstone Column to identify weaknesses that existed in the site organization. Additionally, the CCA was conducted to identify weaknesses which led to the degraded cornerstone, and to document a cumulative causal evaluation. PPL also performed a root cause evaluation (RCE) for each of the unplanned scram events of December 19, 2012, and September 14, 2013.
PPL had previously performed a CCA for low reactor water level related scrams, separate RCEs for the two unplanned scrams with complications and a RCE for the White Unplanned Scrams with Complications PI in preparation for the 95001 supplemental inspection completed in September 2013. These products were inspected by the 95001 inspection (ML13322B321) and were not reinspected in detail by this 95002 inspection.
PPL documented their assessments and evaluations associated with the events that are the focus of this inspection in the following CRs:
CR 1640540 - RCE for November 9, 2012 Complicated Scram CR 1652377 - RCE for December 16, 2012 Complicated Scram CR 1653480 - RCE for December 19, 2012 Scram CR 1746168 - RCE for September 14, 2013 Scram CR 1676146 - RCE for Unit 2 White PI for Complicated Scrams CR 1659749 - CCA for Three Unit 2 Low Reactor Water Level Scrams CR 2013-01143 - CCA for Unit 2 Degraded Cornerstone For clarity and ease of reading, the individual CR numbers will not be repeated in the body of this IR. The report is broken up into sections for each event. The reference to RCE or CCA in each section will be specific to the applicable CR above for the event being discussed. The inspectors reviewed PPL's causal evaluation for each issue, in addition to other evaluations conducted in support of and as a result of the identified causes. The inspectors reviewed the corrective actions taken or planned to address the identified causes. The inspectors conducted interviews with PPL personnel to ensure that the root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and prevent recurrence. The inspectors independently assessed the extent of condition and extent of cause of the identified issues. Additionally, the inspectors performed an assessment of whether any safety culture components caused or significantly contributed to the issues. Finally, the inspectors reviewed two apparent cause evaluations (CR-2014-08941 and CR-2014-10388) performed in response to reactor water level transients which occurred in March 2014 and did not cause a reactor 9 Enclosure scram to assess the effectiveness of corrective actions implemented due to the RCEs and CCAs which are the focus of this inspection.
02 Evaluation of the Inspection Requirements 02.01 Problem Identification.
a. IP 95002 requires that the inspection staff determine that the licensee's evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions under which the issue was identified.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
On December 19, 2012, Unit 2 experienced an automatic scram from 18 percent power during restart following an automatic scram on December 16, 2012. This was a self-revealing event, which was caused by a decision to open the 2A RFP discharge valve breaker without a formal evaluation of impacts or a controlling document. Additionally, not identifying single point vulnerabilities in the design and not documenting lessons learned in the corrective action program contributed to the event. The potential for the event existed since August of 2011 when the 2A RFP discharge valve stuck closed during an attempt to transfer the 2A RFP to the FCM.
The inspectors determined that PPL's RCE appropriately documented the identification of the issues and the conditions under which they were identified.
.3 September 14, 2013 Scram
On September 14, 2013, during a planned shutdown, operators inserted a manual scram of Unit 2 at approximately 14.9 percent power after reactor vessel water level reached the high level turbine trip setpoint and all RFP turbines had automatically tripped. This was a self-revealing event, which was caused by incorrectly configured critical software in the Unit 2 ICS computer. The overall process for implementation of ICS software logic changes lacked the appropriate level of rigor and formality. The potential for the event existed since May 11, 2013, when ICS software modifications were installed on the Unit 2 ICS. The ICS automatically controls the feedwater system during startup and power operations.
The inspectors determined that PPL's RCE appropriately documented the identification of the issues and the conditions under which they were identified.
10 Enclosure b. IP 95002 requires that the inspection staff determine that the licensee's evaluation of the issue documents how long the issue existed and prior opportunities for identification.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
The RCE for this event noted that a similar event occurred on August 23, 2011, when the 2A RFP discharge valve did not open automatically while placing the 2A RFP in service. Specifically, after the 2A RFP discharge valve did not automatically open during start-up operations on August 23, 2011, steps were taken to place the ICS valve control in manual prior to opening the breaker for the discharge valve and manually cracking the valve off its shut seat, which precluded a feedwater control transient. However, these steps were not captured by PPL in either the corrective action process or the operator burden program, and therefore could not prevent the scram on December 19, 2012. The RCE also noted that the same organizational weaknesses that existed in August 2011 were present and contributed to the December 19, 2012 scram. These organizational weakness included "group think" where the operators accepted the Shift Manager's solution of opening the valve breaker to allow manual opening of the valve without challenging or understand in the effects on the plant and "can do attitude" where operators manually operated the valve without adequately evaluating the risk of performing that action. The inspectors determined that PPL's RCE was adequate with respect to identifying how long the issue existed and prior opportunities for identification.
.3 September 14, 2013 Scram
The RCE for this event determined that a prior opportunity existed to identify the software error which caused the September 14, 2013 scram. CR 1735339 identified an error in the Unit 2 ICS software on August 12, 2013. A Susquehanna Error Prevention Team Assessment (SEPTA) was performed on August 29, 2013, in response to CR 1735339 and concluded that following a review of all ICS software changes performed for Unit 1 and Unit 2, all the required software changes were installed as intended. The RCE identified that a less than adequate extent of condition review was performed during the SEPTA and did not identify the errant line of code that resulted in the manual scram on September 14, 2013.
The inspectors determined that PPL's RCE was adequate with respect to identifying how long the issue existed and prior opportunities for identification.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
The CCA for the six scram events occurring between November 9, 2012 and September 14, 2013 (four while on-line and two while shutdown) identified two root causes and three common causes for the risk significant performance issues:
Less than adequate decision making related to ICS design scope and OE reviews resulted in Unit 2 scrams on November 9, December 16, December 19, 2012, and September 14, 2013. (Root Cause) Less than adequate management reinforcement and personal accountability for information-use procedure use and adherence in Operations and Engineering resulted in Unit 2 scrams on November 9, December 16, December 19, and September 14, 2013. (Root Cause) Less than adequate design and implementation of the ICS has resulted in reactor scrams. (Common Cause) Less than adequate review, evaluation, and implementation of OE has resulted in reactor scrams. (Common Cause) Less than adequate procedure content and procedure use and adherence has resulted in reactor scrams. (Common Cause) The CCA concluded that, after review of the RCEs associated with the individual events as well as similar events which occurred in Unit 1, that the identified causes were recurring causes since 2010 The inspectors determined that PPL's CCA was adequate with respect to identifying how long the issues existed and prior opportunities for identification and correction.
c. IP 95002 requires that the inspection staff determine that the licensee's evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issues both individually and collectively.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL's RCE reviewed and documented the consequences of the scram. The event was caused by operators opening the breaker for the 2A RFP discharge valve with the valve closed and with valve control for the 2A RFP in automatic resulted in ICS isolating feedwater flow to the reactor vessel. Reactor vessel water level lowered to -29 inches before the ICS recovered it automatically. RCIC and HPCI were not required to start and remained in standby. All isolations and initiations occurred as expected, and all safety systems operated as expected. The safety consequences of the event were bounded by the Unit 2 design accident analysis. PPL's risk significance determination for the scram resulted in a value of 1E-06 for Core Damage Probability (CDP) and 1E-07 for Large Early Release Probability (LERP) and concluded that the event was of very low safety significance.
The inspectors determined that PPL appropriately documented the plant-specific risk consequences and compliance concerns for this event.
12 Enclosure
.3 September 14, 2013 Scram
PPL's RCE reviewed and documented the consequences of the scram. The actual consequence of this event was a reactor scram from 14.9 percent power and was caused by an ICS software error. PPL's risk significance determination for the scram resulted in a value of 1E-06 for CDP and 1E-07 for LERP and concluded that the event was of very low safety significance.
The inspectors determined that PPL appropriately documented the plant-specific risk consequences and compliance concerns for this event.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL's CCA evaluated risk significance and determined that the four events did not overlap in time, so the determination of cumulative risk through the addition of each of the event's CDP would be very conservative. Thus, the overall impact of the four Unit 2 scrams was determined by individually evaluating the four events for risk. Individual results for all four events were all less than 1E-06 for CDP, and less than 1E-07 for LERP. These individual results represent a Green significance level and are of very low safety significance.
The inspectors determined that PPL appropriately documented the risk consequences and compliance concerns associated with the collective events.
d. Findings
No findings were identified. 02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation.
a. IP 95002 requires that the inspection staff determine that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL performed a RCE in accordance with PPL procedure NDAP-00-0752, "Cause
Analysis.
" PPL used the following systematic methods to complete the RCE: Data gathering through interviews and document review; Events & Causal Factors charting; Cause and effect (why-charting); Hazard-barrier-target analysis; and TapRoot.
13 Enclosure The inspectors determined that PPL evaluated the issues using a systematic method to identify root and common causes.
.3 September 14, 2013 Scram
PPL performed a RCE in accordance with PPL procedure NDAP-00-0752, "Cause
Analysis.
" PPL used the following systematic methods to complete the RCE:
Data gathering through interviews and document review; Events & Causal Factors charting ; Cause and effect (why-charting); Hazard-barrier-target analysis; and TapRoot.
The inspectors determined that PPL evaluated the issues using a systematic method to identify root and common causes.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL performed a CCA in accordance with PPL procedure NDAP-00-0753, "Common Cause
Analysis.
" PPL used the following systematic methods to complete the CCA:
Cognitive trending of the previously identified causes; Data binning; and Cause and Effect (Why Chart) analysis. The inspectors determined that PPL evaluated the issues using a systematic method to identify root and common causes.
b. IP 95002 requires that the inspection staff determine that the licensee's RCE was conducted to a level of detail commensurate with the significance of the issue.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL assembled a multi-discipline team to perform the RCE for this issue. The RCE identified three root causes and one causal factor. PPL determined that the root causes were:
The decision to open the 2A RFP discharge valve breaker was made without a formal evaluation of impacts that reflected a conditioned operator response and inadequate risk evaluation of activities. As a result, ICS design issues concerning the interface between the motor operated valve power source, the valve limit switch, and ICS logic were not recognized.
Opportunities were missed to identify and provide compensation for the design of the ICS logic interface when opening the valve breaker. Specifically, no CR or procedure change was initiated to document specific lessons learned from a similar event that occurred on August 23, 2011, such as opening the 2A RFP discharge valve breaker with the RFP valve controls in manual. Additionally, no specific instructions or directions were provided to operating and off normal procedure writers on how to address single point vulnerabilities. Operations management did not recognize and correct a pattern of operator procedure use behaviors associated with the manual operation of non-responsive motor operated valves. Operators manually operated valves without meeting the intent of NDAP-QA-0029, "Procedure and Work Instruction Use and Adherence," to operate the power plant with certainty of the consequence of the action. Specifically, on six prior occasions plant operators performed manual actions to operate motor operated valves without following the procedural requirements to revise the applicable procedures prior to taking the manual actions.
PPL determined that the causal factor was: The operator burden program did not provide evaluated compensatory actions or drive the correction of the identified issue with the 2A RFP discharge valve prior to the scram on December 19, 2012. Specifically, the operator burden program did not provide direction to the operating crews on what to do about challenges or work-arounds.
Based on the extensive work performed for this RCE, the inspectors concluded that the RCE was conducted to a level of detail commensurate with the significance of the problem.
.3 September 14, 2013 Scram
PPL assembled a multi-discipline team to perform the RCE for this issue. The RCE identified one root cause and four causal factors. PPL determined the root cause was:
The overall process for implementation of ICS logic block code changes lacked the appropriate level of rigor and formality for maintaining digital control systems with the potential to affect reactivity. PPL Identified the causal factors were:
Less than adequate extent of condition evaluation associated with the SEPTA for CR 1735339 because it did not identify the ICS software logic coding error that ultimately resulted in the scram on September 14, 2013; Less than adequate procedure use and adherence of information use procedures for planning work orders resulted in less than adequate work instructions; Less than adequate management and supervisory oversight of work planning, work order reviews, and work associated with ICS logic code changes; and Previously identified extent of cause actions for CR 1348940, "Unit 1 BTV [bleeder trip valve] Steam Leak," to ensure work packages meet management's standards, were focused on the central planning group's three disciplines (mechanical, electrical, and instrumentation and controls) and did not include computer engineering. Based on the extensive work performed for this RCE, the inspectors concluded that the RCE was conducted to a level of detail commensurate with the significance of the problem.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL performed a CCA in response to Unit 2's entry into the degraded cornerstone column of the ROP Action Matrix. PPL's CCA was performed by a multi-disciplined team consisting of personnel from the Maintenance, Regulatory Affairs, Emergency Planning, and Corrective Action and Assessment departments, as well as three specialist consultants. The team used multiple analysis methods and identified two root causes and three common causes. PPL determined that the root causes were:
Less than adequate decision making related to ICS design scope and OE reviews resulted in Unit 2 scrams on November 9, December 16, December 19, 2012, and September 14; 2013, and Less than adequate management reinforcement and personal accountability for information-use procedure use and adherence in Operations and Engineering resulted in Unit 2 scrams on November 9, December 16, December 19, and September 14, 2013. PPL determined that the causal factors were: Less than adequate design and implementation of the ICS has resulted in reactor scrams.
Less than adequate review, evaluation, and implementation of OE has resulted in reactor scrams. Less than adequate procedure content and procedure use and adherence has resulted in reactor scrams. Based on the extensive work performed for this CCA, the inspectors concluded that the collective evaluation was conducted to a level of detail commensurate with the significance of the problem.
c. IP 95002 requires that the inspection staff determine that the licensee's RCE included a consideration of prior occurrences of the issue and knowledge of OE.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
16 Enclosure
.2 December 19, 2012 Scram
PPL's RCE contained a review of both internal and external OE. PPL also reviewed operator training to identify any gaps in the training that would influence the operator's decision to open the 2A RFP breaker.
PPL reviewed industry best practices for excellence in decision making, for opportunities to improve operator decisions regarding emergent work. PPL performed benchmarking of procedures for troubleshooting and risk assessment, comparing PPL procedures to procedures used by other licensees. A focused self-assessment was performed on the station's digital modification process and available industry operating experience concerning implementation of digital system projects. The self-assessment identified gaps in the current engineering processes for digital projects and engineering changes.
Finally, a review was completed of the previous ICS scrams in 2010, 2011, and 2012.
PPL determined that latent organization issues associated with operator knowledge existed. PPL also determined that the previous assessments had not resulted in design changes to the ICS logic associated with the 2A RFP discharge valve.
The RCE noted that the corrective actions for a previous RCE (CR 1517527), regarding shortfalls in operations supervision oversight and command and control, had corrective actions designed to improve crew performance, which had not been completed at the time this RCE was being performed.
Based on PPL's detailed evaluation and conclusions, the inspectors determined that the RCE included a consideration of prior occurrences of the issues and knowledge of prior OE.
.3 September 14, 2013 Scram
PPL's RCE contained a review of both internal and external OE. PPL's review included a search of their corrective action program and industry OE websites for related issues.
PPL's review concluded that:
The OE review identified that none of the internal or external OE reviewed identified the same failures or the same causes as identified in the RCE for the September 14, 2013 scram. The OE review identified several gaps and enhancement opportunities regarding the digital modification and engineering change projects that were entered into the corrective action program for further consideration.
Based on PPL's detailed evaluation and conclusions, the inspectors determined that the RCE included a consideration of prior occurrences of the issues and knowledge of prior OE.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL's CCA included an evaluation of internal and external OE as well as a review of the effectiveness of the OE process. PPL's CCA team identified less than adequate OE evaluation and usage as a common cause amongst the four plant scrams. Specifically, 17 Enclosure PPL determined that poor decision-making during OE evaluations resulted in multiple latent problems. The CCA team identified corrective actions to:
Improve the program to help ensure quality OE reviews in the future; Identify significant OE since January 1, 2003, and ensure that the station's review was adequate; Re-review OE that was similar to that which was not adequately reviewed and contributed to the December 16, 2012, turbine control valve testing scram; and Re-review the applicable 2005 BWROG scram frequency reduction effort recommendations. Based on PPL's detailed evaluation and conclusions, the inspectors determined that the CCA included a consideration of prior occurrences of the issues and knowledge of prior OE. d. IP 95002 requires that the inspection staff determine that the licensee's RCE addresses the extent of condition and extent of cause of the issues.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL's RCE evaluated the extent of condition for the scram and focused on three aspects of the event, including: Opening the RFP discharge valve breaker without explicit guidance and without having a full understanding of the potential risk; Plant component manipulations performed without specific guidance and without an understanding of the potential risk; and Reactor vessel level transients that result in automatic plant scrams.
The review validated that operations personnel would normally open breakers to manually adjust motor operated valves that were stuck. Interviews and a review of the corrective action program did not reveal other plant manipulations, outside of manual breaker operation, that were completed out of process or without written procedures. The review of reactor vessel level transients resulting in automatic plant scrams evaluated three recent scrams that were initiated by improper reactor vessel levels. PPL's RCE evaluated the extent of cause for the scram on December 19, 2012. The RCE stated that the first root cause was the decision to open the 2A RFP discharge valve breaker was made without a formal evaluation of impacts that reflected a conditioned operator response and inadequate risk evaluation of activities. PPL determined that the extent of cause for the first root cause would include any knowledge based decisions. PPL bounded the extent of cause to emergent activities with a less than adequate risk perception and less than adequate procedure use and adherence. A trend code search of CRs from January 1, 2007, to December 31, 2012, was completed for each of the root cause trend codes and did not identify a specific trend but noted that 18 Enclosure identified gaps in command and control were being addressed by corrective actions from a previous RCE (CR 1517522).
The RCE stated that the second root cause was that opportunities were missed to identify and provide compensation for the design of the ICS logic interface when opening the valve breaker. PPL determined that the extent of cause for the second root cause would include other plant events where OE and knowledge is not captured. PPL bounded the extent of cause by significant events that have an element of the corrective action process not being used. A trend code search of CRs from January 1, 2010, to December 31, 2012 was completed for each of the root cause trend codes and determined that this was an area for improvement. The RCE further noted that this area for improvement had been previously identified in CRs 1499040 and 1564051 and that corrective actions had been initiated.
The RCE stated that the third root cause was that operations management did not recognize and correct a pattern of operator procedure use behaviors associated with the manual operation of non-responsive motor operated valves. Operators manually operated the valves without meeting the intent of the general adherence expectation to operate the power plant with certainty of the consequence of action. PPL determined that the extent of cause for the third root cause would include all oversight of procedure use and adherence requirements as well as any procedure non-compliance issues. PPL bounded the extent of cause by activities with less than adequate procedure use and adherence during emergent conditions. A trend code search of CRs from January 1, 2010, to December 31, 2012, was completed for each of the root cause trend codes.
The RCE noted that CR 1673628 previously identified an increase in issues related to procedure compliance in the fourth quarter of 2012. Additionally, in response to the operator human performance issues during the scram on December 19, 2012, operations implemented a phased intervention plan that includes classroom training on operator fundamentals and focused on conservative decision making. The inspectors concluded that PPL's RCE addressed the extent of condition and extent of cause of the issues.
.3 September 14, 2013 Scram
PPL's RCE reviewed the extent of condition for the scram and determined that the extent of condition included: software changes made to the ICS field control processors in Units 1 and 2 since the implementation of the ICS modifications, software changes made to other digital control systems, and the installation of digital electro-hydraulic control system controls. A review of the software changes made to the ICS field control processors prior to August 2011 was completed. A review of software modifications made to other systems with digital control systems in Units 1 and 2 was completed. Also, an action (ACT-03-CR-1746169) was created to ensure lessons learned from this event are incorporated into change packages for the digital electro-hydraulic control pressure control and logic system. PPL's RCE reviewed the extent of cause for the scram and determined that the extent of cause of the root cause, which identified that the ICS software logic change lacked process formality and rigor, included the procedure that controls rewiring in conventional control systems. A review was performed on procedure MT-GE-010, "Control, 19 Enclosure Instrumentation Cable, Lower Range Power Circuit Wire Terminations and Cable Jacket Repair," and determined that the procedure contained sufficient rigor and formality.
The inspectors concluded that PPL's RCE addressed the extent of condition and extent of cause of the issues.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
The purpose of PPL's CCA was to identify any causes common to the six unplanned scrams (four while the reactor was critical, two while the reactor was subcritical) that have occurred at Unit 2 between November 9, 2012 and September 14, 2013, and to validate that corrective actions were adequate to ensure that the extent of those causes were addressed. The CCA team performed an extent of cause analysis and identified the following two extent of causes: Less than adequate decision-making related to digital design scope encompassing any digital modification to the plant that have the potential to initiate plant events; and Less than adequate accountability to procedure use and adherence of information-use procedures is applicable to procedure use and adherence of information-use procedures by any department PPL included corrective actions to independently review the station OE program, re-evaluate the changes to the OE program implemented by the root cause for the December 16, 2012 scram, re-review completed OE evaluations, review department management meeting packages to ensure observations are meeting requirements, and revise management observation requirements.
PPL's CCA also conducted an extent of condition analysis. PPL used events occurring between April 22, 2010 and September 14, 2013, for both Unit 1 and Unit 2 as the extent of condition bounding dates. Twelve reactor scrams and single point vulnerabilities that can cause a reactor scram when greater than 20 percent reactor power were identified in the extent of condition analysis. PPL implemented two interim compensatory actions as a result of the extent of condition analysis:
Initiate procedure changes to place the standby RFP in service prior to removing the in service RFP from feeding in Startup Level Control until permanent system design changes are implemented; and Initiate procedure changes to place the standby RFP in service prior to removing the in service RFP from feeding in FCM until permanent system design changes are implemented.
The inspectors concluded that the extent of condition and extent of cause identified in PPL's CCA was adequate.
e. Findings and Observations
No findings were identified.
The inspectors noted that PPL's CCA identified four areas of weakness in their design control process, related to ICS design changes (i.e., digital control system design):
Single point vulnerability mitigation; Use of plant simulator to validate design adequacy; Software installation, validation, and verification; and Post installation testing of design changes. The inspectors assessed PPL's revised engineering design procedures in each of the above areas and concluded that the changes provided sufficient administrative controls and guidance, if properly implemented, to ensure that the identified individual design problems would not reoccur. However, the inspectors identified an additional weakness in PPL's design control process in the area of operating and design margin identification and evaluation.
Specifically, Engineering Change (EC) 1694052, "ICS Changes to Address 2012 Scram Items," in part, revised the setpoint value for feedwater level control setpoint setdown from +18 to +22 inches. The basis for the change was to provide additional post-scram operating margin between the lowered water level control point (i.e., setpoint setdown) and the nominal low water level scram setpoint, in order to avoid a second scram after the reactor was shut down. The inspectors identified that the Design Considerations Applicability Sheet 78, "Operating and Design Margin," for EC-1694052 did not identify that a margin evaluation was required and procedure MFP-QA-1220, "Design and Operating Margin Evaluation," was not performed. The inspectors determined that raising the setpoint setdown value 4 inches reduced the margin between the expected post-trip water level swell and the high-high water level trip at +54 inches. The reduced margin increases the likelihood that the high-high water level trip would isolate main steam and result in a loss of reactor heat sink. The inspectors concluded that a margin review was required, but had not been performed as required. In immediate response to this issue, PPL re-reviewed the design change package and determined that calculation EC-045-1046, "Feedwater Level Control ICS Setpoints," had, in fact, performed an adequate margin review. PPL concluded that their design control process had failed to identify the need for an operating margin evaluation, but the margin review had been adequately performed by an associated calculation. The inspectors assessed calculation EC-045-1046 and concluded that the potential impacts to operating and design margins, in fact, had been adequately evaluated, but not properly documented. Therefore, the inspectors concluded that not identifying and documenting the need for an operating and design margin review during the engineering change process was a weakness. The inspectors determined this weakness was a minor issue because PPL's review showed that the issue concerning the setpoint setdown was bounded by an existing calculation and was administrative in nature. PPL entered this issue into their corrective action program as CR-2014-24402 and CR-2014-24160.
21 Enclosure 02.03 Corrective Actions.
a. IP 95002 requires that the inspection staff determine that (1) the licensee specified appropriate corrective actions for each root and/or contributing cause, or (2) an evaluation that states no actions are necessary is adequate.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL's RCE identified corrective actions addressing each of the root and contributing causes as well as any additional identified weaknesses. Higher tiered corrective actions to prevent recurrence (CAPR) were identified for each of the root causes. The inspectors reviewed all of the corrective actions to ensure that they addressed the identified causes. The CAPRs included:
Revisions to Unit 1 and Unit 2 operating procedures for placing RFPs into FCM; Caution signs were hung on the RFP discharge and startup isolation valve breakers indicating that opening the breakers impacts ICS logic; Revisions to OP-AD-002, "Standards for Shift Operations," to define intrusive actions during normal plant operations and positive control; Revisions to OP-AD-002, "Standards for Shift Operations," to require positive control and/or risk based decisions in accordance with NDAP-QA-1902, "Integrated Risk Management;" Clarifying management expectations related to actions required following failure or inability to complete a procedure; Revisions to NDAP-QA-0029, "Procedure and Work Instruction Use and Adherence," to ensure that any plant manipulations have controlling documents and have been risk assessed; and Revision to OP-AD-002, "Standards for Shift Operations" to define operator specific skill of the craft work activity actions.
The corrective action to address the contributing cause was that PPL revised the operator burden program procedure OI-AD-096, "Operator Burdens." With the exception of the findings and observations documented below in section 4OA4.02.03.f, the inspectors determined that the proposed corrective actions were appropriate and addressed each root and contributing cause.
.3 September 14, 2013 Scram
PPL's RCE identified corrective actions addressing each of the root and contributing causes as well as any additional identified weaknesses. Higher tiered CAPRs were identified for each of the root causes. The inspectors reviewed all of the corrective actions to ensure that they addressed the identified causes. The CAPRs included:
Revisions to procedure guidance for post maintenance testing of ICS software logic changes; Development of procedural guidance for installing changes to ICS software logic changes, including appropriate verification requirements; and, Revisions to procedure NSEI-AD-506, "Computer System Problem Reporting," guidance to ensure work planning and work on digital control systems incorporate adequate verification requirements, detailed procedural steps, use of human performance tools, requirements for pre-job briefs, and risk reviews.
To address the contributing causes, in addition to the CAPRs listed above, PPL identified the following corrective actions:
Revised several work planning procedures to clarify who performs specific steps in the work planning process to improve accountability; Required supervisory personnel to conduct detailed work package assessments of planned and/or completed work packages per month to verify that work package planning and documentation standards are adequate; and Conduct a stand down with the Computer System Engineering personnel to reinforce the lessons learned from this event.
The inspectors determined that the proposed corrective actions were appropriate and addressed each root and contributing cause.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL's CCA identified causes common to the six unplanned scrams and listed CAPR for each of the common causes. The inspectors reviewed all of the corrective actions to ensure that they addressed the identified causes.
PPL identified that less than adequate decision-making related to ICS design scope (acceptance of single point vulnerabilities) and OE reviews was a common root cause to each of the Unit 2 scrams. PPL reviewed their OE program to both improve OE evaluations. A review was conducted to identify and re-review previous less than adequate evaluations. This review included an independent evaluation of the OE program by the station's Performance Improvement Process Leader. As a result of the review, the Performance Review group has established a monitoring program to track timeliness and quality of OE reviews. PPL has also added OE to the station's Performance Improvement excellence plan.
PPL identified less than adequate management reinforcement and less than adequate personal accountability for information-use procedures use and adherence in Operations and Engineering Departments, as a common cause to each of the Unit 2 scrams. PPL assessed whether previous corrective actions taken to address the common cause (including implementation of industry best practices regarding procedure format, development and use of procedural use expectations, implementation of procedure use and adherence supervisory observations, and use of performance indicators to track progress) have continued to be successful. PPL determined that the station was off-track to meet the expected outcome of the corrective actions as defined by the effectiveness review of an individual RCE associated with a previous specific event (CR 1389530). In response, PPL implemented multiple actions including procedure 23 Enclosure enhancements, increased management reviews, and program changes to ensure that the actions would be effective.
With the exception of the findings and observations documented below in section 4OA4.02.03.f, the inspectors determined that the proposed corrective actions were appropriate and addressed each contributing cause.
b. IP 95002 requires that the inspection staff determine that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL prioritized corrective actions with consideration of risk significance and regulatory compliance. CAPRs and corrective actions associated with the three identified root causes and the causal factor were given adequate due dates and were completed commensurate with their risk significance. At the time of the inspection, all CAPRs and corrective actions were completed.
.3 September 14, 2013 Scram
PPL prioritized corrective actions with consideration of risk significance and regulatory compliance. CAPRs and corrective actions associated with identified root cause and the four causal factors were given adequate due dates and were completed commensurate with their risk significance. At the time of the inspection, all CAPRs were completed.
The causal factor corrective actions that were not already completed had reasonable due dates assigned and had been prioritized in accordance with site procedures and based on consideration of risk significance and regulatory compliance.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
Following the completion of the CCA, PPL prioritized the corrective actions with consideration of risk significance and regulatory compliance. PPL's CA team compiled the actions from each of the associated RCEs to create an integrated list of actions. The compiled list also included new actions assigned as a result of the CCA. PPL assigned a due date and classification of "Preventative", "Corrective", 'Admin", or "Enhance" to each of the actions.
The inspectors reviewed the prioritization of the corrective actions associated with the CCA, as well as the procedural requirements found in site procedure NDAP-QA-0701,
"Action Request and Condition Report Process." The inspectors verified that the prioritization was in accordance with site procedures and based on consideration of risk significance and regulatory compliance.
c. IP 95002 requires that the inspection staff determine that the licensee established a schedule for implementing and completing the corrective actions.
24 Enclosure
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
.2 December 19, 2012 Scram
PPL established a schedule for completing corrective actions associated with this event and assigned corrective actions to the appropriate individuals or organizations to ensure that the actions were planned or taken in a timely manner. CAPRs and causal factor corrective actions associated with the three identified root causes and the causal factor were adequately scheduled and completed commensurate with their risk significance.
At the time of the inspection, all CAPRs and causal factor corrective actions were completed.
.3 September 14, 2013 Scram
PPL established a schedule for completing corrective actions associated with this event and assigned corrective actions to the appropriate individuals or organizations to ensure that the actions were planned or taken in a timely manner. CAPRs and causal factor corrective actions associated with the three identified root causes and the causal factor were adequately scheduled and completed commensurate with their risk significance.
At the time of this inspection, all CAPRs were completed and the causal factor corrective actions that were not already completed had reasonably scheduled due dates.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL's corrective actions and proposed corrective action plan provided dates for completion of actions as described in the common cause assessment. At the time of the inspection, 33 of the 42 corrective actions had been completed. Completed actions include: procedure revisions to enhance human performance tools, improvements to station OE processes, implementation of a new station ownership model, and revision to the station procedure for failure modes and effects analysis to provide compensation for identified single point vulnerabilities.
The inspectors reviewed the remaining open corrective actions and determined that the schedule for implementation and completion was reasonable and timely.
d. IP 95002 requires that the inspection staff determine that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition.
.1 November 9, 2012 and December 16, 2012, Complicated Scrams
This inspection requirement was accomplished during a 95001 inspection as documented in IR 05000388/2013011 (ML13322B321).
25 Enclosure
.2 December 19, 2012 Scram
PPL has planned several effectiveness reviews and self-assessments for each root and common cause corrective action to ensure that these actions prevent recurrence and are complete and appropriate. An interim effectiveness review was completed March 6, 2014, and the final effectiveness review is planned to be performed 6 to 12 months after the completion of all CAPRs and causal factor corrective actions. At the time of the inspection, the final effectiveness review action was due October 31, 2014. Each root cause has quantitative or qualitative criteria assigned in the effectiveness review plan, which has been drafted in accordance with PPL's RCE procedure. Notwithstanding, revisions to the final effectiveness review plan associated with the RCE were implemented during the course of the NRC 95002 inspection.
The inspectors determined that PPL had established adequate measures for determining the effectiveness of the corrective actions.
.3 September 14, 2013 Scram
PPL has planned several effectiveness reviews and self-assessments for each root and common cause corrective action to ensure that these actions prevent recurrence and are complete and appropriate. An interim effectiveness review is scheduled to be completed by October 2014, and the final effectiveness review will be performed 6 to 12 months after all CAPRs and causal factor corrective actions are complete. Each root cause has quantitative or qualitative criteria assigned in the effectiveness review plan, which has been drafted in accordance with PPL's RCE procedure.
The inspectors determined that PPL had established adequate measures for determining the effectiveness of the corrective actions.
.4 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
PPL has planned several effectiveness reviews and self-assessments for each root and common cause corrective action to ensure that these actions prevent recurrence and are complete and appropriate. The common cause analysis scheduled two interim and one final effectiveness review to evaluate the corrective actions associated with the analysis.
Each root and common cause has quantitative and qualitative criteria assigned in the effectiveness review plan, which has been drafted in accordance with PPL procedure NDAP-QA-0701, "Action Request and Condition Report Process."
The inspectors determined that PPL has successfully developed and implemented an effectiveness review plan for the corrective actions associated with the common cause analysis.
e. IP 95002 requires that the inspection staff determine that the licensee's planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable. The NRC staff did not issue an NOV to the licensee; therefore, this inspection requirement was not applicable.
26 Enclosure
f. Findings and Observations
Findings
.1 Inadequate Maintenance of Procedures
Introduction.
A self-revealing Green NCV of TS 5.4.1, "Procedures," was identified, because PPL did not adequately maintain operating procedures for plant shutdown to hot standby. Specifically, the general operating procedure for plant shutdown to minimum power and the RFP operating procedure were revised, and the technical reviews did not adequately verify the functional and technical adequacy of the procedures. The technical reviews did not identify a valve lineup conflict existed between the two procedures.
The conflict resulted in an improper feed lineup to the RPV causing two level transients and corresponding power transients of approximately 5 percent on March 20, 2014.
Description.
PPL performed a CCA in response to Unit 2's entry into the degraded cornerstone column of the ROP Action Matrix and documented the results in CR-2013-01143. Two of the common causes identified by PPL had related interim compensatory corrective actions to revise procedures to allow plant operations while permanent design changes were designed and implemented. Specifically, OP-245-005, "Infrequent Manual RFP System Operations," and GO-200-004, "Plant Shutdown to Minimum Power," were revised in regards to the method of placing standby RFPs into service and removing operating RFPs from service. The revisions were completed on February 21, 2014. The review process did not identify that a valve position conflict existed between the two procedures which resulted in an unintended and undesired parallel feed path to the RPV.
On March 20, 2014, Unit 2 operators commenced a shutdown to repair the 2A RFP discharge valve. A pre-job brief for the shutdown was performed but did not establish reactor water level control bands in accordance with OP-AD-004, "Operations Standards for Error and Event Prevention." OP-AD-004 discusses parameters to consider when establishing scram criteria, which may be "adjusted based on the rate of change of the parameter." OP-AD-004 establishes a scram criteria guideline of greater than +50 inches RPV water level and "not in control."
At the beginning of the event, the reactor was at 14 percent power with 2B RFP feeding the reactor in the FCM, 2C RFP was in standby mode and 2A RFP was offline.
Operators were in the process of transferring 2C RFP from the standby to the discharge pressure mode and taking 2B RFP offline, in accordance with GO-200-004, "Plant Shutdown to Minimum Power," and OP-245-005, "Infrequent Manual RFP System Operations."
GO-200-004 directed warming the startup feed line using the 2B RFP (operating pump)by opening the 2B RFP startup feed line isolation valve but did not contain instructions to shut the feed line isolation valve when feed line warmup was complete. OP-245-005 contained a step to warm the startup feed line using the 2C RFP (oncoming pump) by opening the 2C RFP startup feed line isolation valve. The result was an unintended feed path to the reactor from the 2C RFP, through the startup feed lines for both pumps and into the reactor through the 2B RFP discharge valve. The effect on the plant was that the additional feedwater provided by the 2C RFP caused the reactor water level to rise.
27 Enclosure The operators did not recognize the discrepancy between the two procedures and did not recognize the valve alignment issue.
The reactor vessel water level increased approximately 16 inches in two minutes (a rate of 1-inch per 7.5 seconds) and reached approximately +52 inches before operators took action to manually lower 2C RFP speed to reduce its output and reactor vessel water level lowered to its normal level of +35 inches. The additional feedwater caused reactor power to increase by approximately 5 percent. The Feedwater ICS will trip all operating steam turbines, including the feed pump turbines, at +54 inches to prevent damage to the turbine blading, causing a loss of feed condition which would have required a manual scram shortly thereafter.
Operators stabilized the plant at the pre-event conditions and discussed the potential issues which may have caused the level transient. The operators decided that the plant was not in a stable condition in the current configuration and decided to perform the procedure again. A second RPV level transient occurred with reactor water level reaching +48 inches before the 2B RFP was taken out of service, which terminated the second feedwater flow path. The additional feedwater caused reactor power to increase by approximately 5 percent. The inspectors noted that if an adequate pre-job brief was performed and proper feedwater control bands were established, the operators would have been required to scram the reactor during the first transient.
Following the reactor water level transients, corrective actions were taken to collect personnel statements, complete a SEPTA report, correct the procedure discrepancies, perform an operations stand-down, and enter the events of March 1, 2014, and March 20, 2014, into the corrective action program as CR-2014-08941 and CR-2014-10388.
Analysis.
The inspectors determined that PPL's inadequate maintenance of procedures for plant shutdown to hot standby was a performance deficiency within PPL's ability to foresee and correct and should have been prevented. Specifically, the technical reviews associated with changes to the general operating procedure for plant shutdown to minimum power and the RFP operating procedure did not adequately verify the functional and technical adequacy of the procedures. This finding is more than minor because it is associated with the procedure quality attribute of the Initiating Events cornerstone and affected the cornerstone objective of limiting the likelihood of those events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the inadequate technical reviews associated with the revision of procedures for placing standby RFPs into service in startup level control and valve control (manual), and procedures for placing standby RFPs into service in FCM and valve control (manual), resulted in two reactor power transients up to five percent and two significant reactor vessel water level transients and challenged the stability of the plant. Additionally, this issue is similar to Example 4b described in IMC 0612, Appendix E, "Examples of Minor Issues," issued August 11, 2009, which states that issues are not minor if procedure issues cause a reactor trip or other transient.
The inspectors evaluated the finding using Attachment 0609.04, "Initial Characterization of Findings," worksheet to IMC 0609, "Significance Determination Process," issued June 2, 2011. The attachment instructs the inspectors to utilize IMC 0609, Appendix A,
"Significance Determination Process for Findings At-Power," issued June 19, 2012. The 28 Enclosure inspectors determined this finding did not cause a reactor trip and the loss of mitigation equipment relied upon to transition the plant from the onset of the trip to a stable shutdown condition (e.g. loss of condenser, loss of feedwater) and is therefore of very low safety significance (Green).
A cross-cutting aspect was assigned in the area of Human Performance, Change Management, because leaders did not use a systematic process for evaluating and implementing change so that nuclear safety remained the overriding priority.
Specifically, a clear focus on nuclear safety when implementing changes to the general operating procedure for shutdown to minimum power was not maintained and resulted in unintended procedure discrepancy. [H.3]
Enforcement.
TS 5.4.1 requires that written procedures be established, implemented, and maintained covering the applicable procedures recommended in Section 2, "General Plant Operating Procedures," of Appendix A to Regulatory Guide 1.33, "Quality Assurance Program Requirements," Revision 2, February 1978, which includes procedures for "Cold Shutdown to Hot Standby." Contrary to the above, between February 2014 and March 2014, PPL did not adequately maintain procedures for plant shutdown to minimum power, which resulted in two significant reactor vessel level transients and corresponding power transients of approximately five percent on March 20, 2014. PPL's corrective actions included mitigating the RPV level transients, collecting personnel statements, and entering the events of March 1, 2014, and March 20, 2014, into the corrective action program (CR-2014-08941 and CR-2014-10388). Because this violation is of very low safety significance and has been entered into PPL's corrective action program, this finding is being treated as an NCV consistent with Section 2.3.2.a of the NRC Enforcement Policy. (NCV 05000388/2014008-01, Inadequately Maintained Procedures for Plant Shutdown to Hot Standby led to Reactor Water Level and Power Transients) Observations
.2 Operator Performance
On March 1, 2014 during a Unit 2 transfer of the 2B RFP from FCM to standby and the 2C RFP from standby into Discharge Pressure Mode (DPM), the reactor operator performing the evolution recognized an unintended flow path from the 2C RFP through the 2B RFP startup isolation valve existed. The 2B RFP startup isolation valve was left open due to a conflict in required valve position between GO-200-004, "Plant Shutdown to Minimum Power," and OP-245-005, "Infrequent Manual RFP System Operations."
The reactor operator corrected the flow path without obtaining adequate concurrence from supervision, without implementing the appropriate procedure change process of NDAP-QA-0004, 'Procedure Change Process', and without subsequently writing a condition report to document the discrepancy.
Due to the uncorrected procedural discrepancy, on March 20, 2014, during a transition of the 2B RFP from FCM to standby and the 2C RFP from standby to DPM, Unit 2 experienced two high level transients peaking at approximately +50 inches and +48 inches respectively. On March 20, the control room operating crew did not adequately conduct required "Just in Time Training" prior to the evolution due perceived time pressure. After recovering from the initial level transient, the crew made the decision to move forward in the procedure in an attempt to stabilize the plant without seeking additional assistance from plant engineering, 29 Enclosure operations, or the outage control center. These actions directly contributed to the two level transients.
The inspectors noted that the operator actions exhibited during the March 1 and March 20, 2014 events were similar to several of the contributing and root causes identified as a part of the RCEs and CCAs for the November 9, December 16, and December 19, 2012, Unit 2 events. The continued presence of these behaviors in spite of established corrective actions calls into question the effectiveness of these corrective actions. Following a review of the corrective actions that PPL identified and implemented in the previous evaluations, the inspectors determined that the existing corrective actions are sufficient to address the station performance issues if they are rigorously implemented and adhered to and assessed that this observation is a general weakness.
.3 Accountability with Respect Model
PPL developed the "Accountability with Respect Model" initiative as a corrective action to prevent recurrence for both root causes identified by the CCA "Unit 2 2013 Degraded Cornerstone Evaluation." The "Accountability with Respect Model" is a structured approach to ensure that standards and expectations are clearly defined, communicated and implemented throughout the station organization. The "Accountability with Respect Model" represents the PPL's efforts to cause change in the organizational behavior of the station and ingrain desired behaviors. The inspectors reviewed the implementation details of the model, interviewed the model process owners and included the model as a point of discussion in the safety culture interviews held with station employees. The inspectors identified three general weaknesses with the Accountability with Respect Model:
The rollout of the Accountability with Respect Model was only to station managers and supervisors. The first level supervisors were tasked with briefing the workforce. The inspectors Safety Culture interview results identified that a significant portion of the workforce did not understand or know about this station initiative.
The effectiveness review for the initiative centered on determining the percentage of the first level supervisors that understood their role in the initiative model. Once the number reached a predetermined percentage, the model would be considered self-sustaining and part of the station's organizational culture. The inspectors observed that the effectiveness review was narrowly focused on the first line supervisors and should consider periodically measuring the effectiveness of implementing changed behaviors throughout the organization.
There were no plans to conduct initial training for new employees or periodic refresher training for existing employees for the "Accountability with Respect Model" initiative. The inspectors observed that without initial training to inform new employees that "this is the way PPL conducts business" or refresher training to remind employees that "this is the way PPL expects you to conduct business," the standard for organizational behavior desired by PPL would not be consistently emphasized over time.
The inspectors determined that these observations were not performance deficiencies and represented enhancements to PPL's Accountability with Respect Model 30 Enclosure initiative. PPL entered these observations into the corrective action program as condition reports CR-2014-23290 and CR-2014-23298.
.4 OE Program Key Effectiveness Measure
PPL determined that not all industry OE, such as vendor service advisories, bulletins, and notifications were being entered into the station's OE program for formal tracking and evaluation. Specifically, some OE was being delivered directly to station engineering personnel and informally evaluated, without being entered into station's OE program, as required by LS-115, "Operating Experience Program." In response to this issue, PPL performed a technical re-review of OE received by engineering that had bypassed the station's OE program. PPL issued a station level notification to remind all site staff that OE from any source must be submitted to the OE coordinator and formally screened. As an effectiveness measure, PPL verified that the OE program was being implemented satisfactorily, in part, by checking that OE delivered directly to station engineering was now being routed through the OE coordinator, and had an adequate review in a timely manner. The inspectors determined that PPL's effectiveness measure was limited to site engineering. The inspectors concluded that limiting the effectiveness reviews to only site engineering was a weakness because other station organizations, such as health physics, security, emergency planning, procurement, and maintenance might also have directly received OE that bypassed the station's OE program and may not have received an adequate technical review. In immediate response to this issue, PPL verified that Maintenance Planning and Procurement Management were appropriately handling OE. The inspectors determined that these observations were not performance deficiencies and the observations represented enhancements to PPL's corrective actions. In follow-up, PPL entered this issue into their corrective action program as CR-2014-23409, to verify that other station organizations were correctly handling OE.
02.04 Independent Assessment of Extent of Condition and Extent of
Cause.
a. Inspection Scope
IP 95002 requires that the inspection staff perform a focused inspection to independently assess the validity of the licensee's conclusions regarding the extent of condition and extent of cause of the issue(s). The objective of this requirement is to independently sample performance, as necessary, within the key attributes of the cornerstone(s) that are related to the subject issue(s) to ensure that the licensee's evaluation regarding the extent of condition and extent of cause is sufficiently comprehensive.
.1 November 9, 2012 Complicated Scram
The inspectors performed an independent assessment of the extent of condition and extent of cause of this event. In conducting this independent review, the inspectors interviewed station management and staff, reviewed program and process documentation, and reviewed existing station program monitoring and improvement efforts, including review of corrective action documents. PPL completed two RCEs and a CCA associated with the event. Each of the evaluations addressed extent of condition and extent of cause, and assigned corrective actions based on the identified extent of condition and extent of cause. PPL determined 31 Enclosure that the actual extent of condition for this event is limited to applications of complex digital control systems that could have direct impact to continued operation of the plant, and identified the Unit 1 and Unit 2 ICS core switches as within the scope of the extent of condition. PPL initiated actions to replace the core switches in both Unit 1 and Unit 2, as well as an interim action for the Operations Department to develop procedural guidance to recover from ICS lockup until the core switches are replaced with an updated model that is not susceptible to lockup. PPL utilized a date range of January 1, 2009 through November 9, 2012 to determine the extent of cause and further bounded the determination by design deficiency issues. PPL identified no additional causes that could have impacted other plant processes, equipment or human performance.
The inspectors concluded that PPLs determination of extent and condition and extent of cause for this scram event, as well as the corrective actions assigned to each, were valid and sufficiently comprehensive.
.2 December 16, 2012 Complicated Scram
The inspectors performed an independent assessment of the extent of condition and extent of cause of this scram during the turbine valve cycling surveillance. In conducting this independent review, the inspectors interviewed station management and staff, reviewed program and process documentation, and reviewed existing station program monitoring and improvement efforts, including review of corrective action documents.
PPL completed two RCEs and a CCA associated with the event. Each of the evaluations addressed extent of condition and extent of cause, and assigned corrective actions based on the identified extent of condition and extent of cause. PPL determined that the actual extent of condition for this event was limited to two station surveillance procedures that introduce a half scram signal during plant operations at power. PPL initiated actions to revise the two surveillance procedures to require Instrumentation &
Control (I&C) technical support and installation of a RPS "test box" during turbine control valve testing.
PPL utilized a date range of January 1, 1997, through December 17, 2012, to review the extent of cause related to implementing lessons learned from OE and worker practices. As a result of the extent of cause review, PPL initiated actions to re-evaluate the 2005 BWROG scram reduction frequency effort recommendations, verify adequate wiring connections for Unit 1 turbine control valves, and re-evaluate five vendor technical communications related to turbine control.
The inspectors concluded that PPLs determination of extent of condition and extent of cause for this scram, as well as the corrective actions assigned to each were valid and sufficiently comprehensive.
.3 December 19, 2012 Scram
The inspectors conducted independent extent of condition and extent of cause reviews of the scram. The RCE identified issues associated with completing plant manipulations without a formal risk evaluation of impacts; not documenting lessons learned in the corrective action program, such as the inability to complete a procedure as written; and operations management not recognizing and correcting a pattern of operator procedure use behaviors related to operating plant equipment without certainty of the consequence of action. The inspectors' independent review focused on the primary root causes 32 Enclosure associated with the scram in addition to PPL's identified contributing causes of the scram.
The inspectors assessed whether PPL's extent of condition and extent of cause evaluations sufficiently identified and bounded all operations issues. The inspectors also assessed whether PPL's extent of condition and extent of cause evaluations sufficiently determined the actual extent of similar organizational issues that potentially existed in other station departments, programs, and processes.
In conducting this independent review, the inspectors interviewed station management and personnel, reviewed program and process documentation, and reviewed existing station improvement efforts, including review of corrective action documents.
The inspectors determined that PPL conducted a comprehensive extent of condition and extent of cause review. The inspectors did not identify any substantive extent of condition and extent of cause issues that PPL was not aware of and had not already identified with corrective action plans in place.
With the exception of the findings and observations documented in section 4OA4.02.03.f, the inspectors concluded that PPLs determination of extent of condition and extent of cause for this scram, as well as the corrective actions assigned to each were valid and sufficiently comprehensive.
.4 September 14, 2013 Scram
The inspectors conducted independent extent of condition and extent of cause reviews of the scram. The RCE identified issues associated with the overall process for implementation of ICS software changes lacking the appropriate level of rigor and formality for maintaining digital control systems and procedure use and adherence of information use procedures for work order planning. The inspectors independent review focused on the primary root cause associated with the scram in addition to PPL's identified contributing cause of the scram.
The inspectors assessed whether PPL's extent of condition and extent of cause evaluations sufficiently bounded all engineering issues. The inspectors also assessed whether PPL's extent of condition and extent of cause evaluations sufficiently determined the actual extent of similar organizational issues that potentially existed in other station departments, programs, and processes.
In conducting this independent review, the inspectors interviewed station management and personnel, reviewed program and process documentation, reviewed existing station improvement efforts, including review of corrective action documents, conducted walkdowns of the ICS installation, including the plant computer room, and observed demonstrations of ICS usage and capabilities in the plant simulator.
The inspectors determined that PPL conducted a comprehensive extent of condition and extent of cause review that sufficiently identified most relevant areas. The inspectors did not identify any substantive extent of condition and extent of cause issues that PPL was not aware of and had not already identified with corrective action plans in place.
33 Enclosure The inspectors concluded that PPLs determination of extent of condition and extent of cause for this scram, as well as the corrective actions assigned to each were valid and sufficiently comprehensive.
.5 Common Cause Analysis for Unit 2 Degraded Cornerstone Evaluation
The inspectors independently assessed PPL's common cause analysis to determine whether PPL's conclusions regarding the identification of common issues and common causes were sufficiently comprehensive. The inspectors evaluated PPL's root causes, key corrective actions, and key effectiveness measures for the identified common causes to assess whether PPL's planned corrective actions, once completed, could reasonably prevent recurrence of the degraded performance issues. PPL's CCA evaluated six unplanned scrams in Unit 2 (i.e., four scrams while the reactor was critical, and two scrams that occurred after the reactor was shutdown) and identified 3 common causes: Less than adequate design and implementation of the ICS; Less than adequate review, evaluation, and follow-up on industry OE; and Less than adequate operating procedure content and less than adequate procedure use and adherence. The inspectors interviewed station management, engineering and operations personnel, reviewed selected station level, engineering, and operations department programs and processes, and reviewed PPL's effectiveness measures to verify the adequacy of the completed and in-progress corrective actions. In addition, the inspectors reviewed selected CRs for associated or similar issues. The inspectors conducted walkdowns of the ICS installation, including the plant computer room and main control room, observed demonstrations of ICS usage and capabilities in the plant simulator, and observed a demonstration of software testing and verification activities using engineering's Foxboro Simulator and Foxboro Test Bed. The inspectors utilized portions of IP 90700,
"Feedback of Operational Experience Information at Operating Power Reactors," and IP 42700, "Plant Procedures," to assess PPL's conclusions and corrective actions.
With the exception of the findings and observations documented in section 4OA4.02.03.f, the inspectors concluded that PPLs determination of extent of condition and extent of cause for the degraded cornerstone, and the corrective actions identified, were of reasonable breadth and depth to prevent recurrence of the degraded performance issues.
02.05 Safety Culture Consideration
a. Inspection Scope
IP 95002 requires that the inspection staff perform a focused inspection to independently determine that the licensee's RCE appropriately considered whether any safety culture component caused or significantly contributed to any risk significant issue. The inspectors independently assessed the relationship between the safety culture aspects and the performance issues through the use of focus groups and interviews, as 34 Enclosure well as review of cause evaluations, self-assessments, and corrective action documents provided by PPL. The inspectors interviewed a total of 70 staff, including 21 supervisors and senior management personnel. Based on review of the applicable cause evaluations, the inspectors selected participants for the focus groups and interviews from the following organizations: Operations, Station Engineering, Design Engineering, I&C, Maintenance, Operations Training, and Work Management. Focus groups did not combine supervisors with staff-level personnel. The inspectors also designed the focus groups and interviews to gather information on the safety culture at the station with some questions directed towards specific safety culture aspects. The questions covered the following areas2: Leadership safety values and actions, including station resources and change management; Problem identification and resolution; Personal accountability, including PPL's "Accountability with Respect" model; Work processes, including work management, procedure adequacy, and procedural adherence; Continuous learning, including initial and continuing training and use of OE; Safety conscious work environment and alternative processes for raising concerns; Effective safety communication; Questioning attitude; and Decision-making, including station processes used for decision-making. b. Assessment Based on input from the focus groups, interviews, and review of documentation, the inspectors concluded: PPL staff expressed a willingness to use the station's corrective action program to identify plant issues and deficiencies. Station personnel stated that they were willing to raise safety issues, and were aware of the availability of alternate reporting channels, including the Employee Concerns Program. Additionally, workers noted that they would not hesitate to stop work in the event they receive an unexpected response, or if there is a problem with any step in a procedure. PPL staff also mentioned that there has been an increased emphasis on procedural compliance at the station. Nonetheless, during the inspection, the inspectors noted general weaknesses associated with operator procedural adherence issues. These weaknesses are discussed in Section 4OA4.02.03.f of this inspection report.
The station is effective at distributing operating experience to PPL staff. Most staff remarked that operating experience is routinely provided and discussed via email, pre-job briefings, various station meetings, and training.
It is not apparent that PPL's "Accountability with Respect" model has been fully embraced at all levels. In some cases, staff perceived this initiative to be a "once-and-done" occurrence that will be replaced by another new management initiative later. Additionally, though all levels of management are expected to implement this 2 For more information on the specific topics included in these areas, refer to NRC Inspection Manual Chapter 0310, "Aspects Within the Cross-Cutting Areas," issued 12/19/2013 (ADAMS Accession Number ML13351A028).
35 Enclosure model, some supervisors stated that they did not receive any specific training in the recent past on methods to enforce accountability. Further insights related to implementation of this model as a CAPR are discussed in Section 4OA4.02.03.f of this report.
There is a common perception amongst most groups that staffing levels are lower than optimal, especially in the Operations and Training organizations. In addition, most groups are very aware of potential knowledge management issues that face the station due to retirements. PPL staff perceives that there is no succession planning in place for critical positions and qualifications, and provided specific examples where there are only one or two people onsite who are qualified to perform certain tasks. Based on interviews with management, the inspectors concluded that there are plans in place to address succession planning and knowledge management. However, it does not appear that these plans have been adequately communicated to internal stakeholders. The inspectors noted that this is similar to an observation documented during the safety culture review conducted during the previous 95002 inspection in 2012 (ML12125A374).
PPL staff noted that while the station was effective at using the corrective action program to identify issues, the station was not as effective at resolving issues. Staff speculated that some of the factors affecting this include insufficient documentation of the issue in the CR, which then requires more follow-up before actions are assigned; inexperience of some of the CR screening committee members in plant systems and operations, which can result in inappropriate prioritization or disposition of some issues; and an incorrect threshold for identifying issues, which may overload the process. PPL staff also noted that in general, the station applies an appropriate amount of effort to fix significant equipment problems, but repairs for lower-level, non-safety related equipment issues are frequently delayed.
In general, PPL staff acknowledged that the actual work management process was effective. However, the staff also recognized that the station was not as effective in implementing this process. PPL staff noted that improvements are needed in enforcing accountability when organizations fail to meet work planning milestones.
Most groups also provided multiple examples where shortages in staffing and qualifications adversely affected work management at the station. Finally, most groups observed that improvements are needed in coordination of emergent and normally scheduled work. PPL staff did have a positive response towards the establishment of the Engineering Fix-it-Now team and addition of a Senior Reactor Operator to the Fix-it-Now group, and noted that both benefitted the work management process at the station.
PPL staff indicated that change management was a weakness at the station. Specific examples provided to the inspectors included the transition to a new corrective action program (Actionway) software, and implementation of new station-level procedures. The most common concerns with the change management process included a lack of communication prior to implementing a change, using the wrong method of communication (e.g., email versus face-to-face), failure to get input from affected departments during development of a change, and/or not thoroughly evaluating the impact of a change before implementation.
36 Enclosure The inspectors discussed all of these observations during the individual interviews with station senior management. In most cases, the senior management team at the station was aware of the perceptions of plant staff. Overall, the inspectors determined that the cause evaluations associated with this inspection appropriately considered safety culture components as they related to the various root and significant contributing causes. The inspectors did not considered any of these observations to be issues of concern or performance deficiencies.
02.06 Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues PPL did not request credit for self-identification of an old design issue; therefore, the risk-significant issues were not evaluated against the IMC 0305 criteria for treatment of an old design issue.
4OA6 Meetings
Exit Meeting Summary On July 31, 2014, the inspectors presented the inspection results to Mr. J. Franke, Site Vice President, and other members of his staff, who acknowledged the findings. The inspectors asked PPL if any of the material examined during the inspection should be considered proprietary. PPL did not identify any proprietary information.
On September 3, 2014, the NRC held a regulatory performance meeting and public exit with PPL to discuss the corrective actions associated with PPL's White PIs and subsequent entry into the degraded cornerstone column of the ROP Action Matrix.
Mr. B. Frannsen, General Manager of Operations, and other members of the staff represented PPL. The purpose of the meeting was to provide a forum in which to develop a shared understanding of the performance issues, underlying causes, and PPL's planned actions for each safety significant assessment input.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
J. Bella Senior Engineer
- R. Bingman Unit Supervisor, Senior Reactor Operator
- J. Boyer Support Engineer M. Boyle Unit Supervisor, Senior Reactor Operator
- E. Brice Unit Supervisor, Senior Reactor Operator
S. Carpenter Root Cause Analyst
E. Carter Simulator Supervisor
T. Creasy Assistant Operations Manager for Shift Operations W. DeLuca Supervisor Computer Engineering E. Dudick Performance Improvement Process Leader
- D. Duttry Shift Manager, Senior Reactor Operator
- K. Dyer Acting Supervisor, Corrective Action
- R. Francis General Manager, Operations L. Fuller Senior Design Engineer J. Glaser System Engineer
J. Goodbred Jr. Manager Procedure Upgrade Project
K. Green Reactor Operator
- J. Grisewood Manager, Performance Improvement
- C. Hoffman Manager, Nuclear Fuels
- N. Hyduk Support Engineer K. Karidyer Acting Supervisor, Corrective Actions
A. Kissinger PPL Consultant
K. Kluk Maintenance Rule Coordinator
D. Kostelnik Senior Design Engineer K. Landis PPL Consultant C. Manges Regulatory Affairs
C. Markley 95002 Support Team
T. Mogavero CAPCO Performance Improvement Coordinator B. Morris Licensed Senior Reactor Operator
- E. Myers Computer Engineer T. Page PPL Consultant A. Price Supervisor, Performance Improvement
F. Purdy System Engineer
T. Roth Operations Engineer
P. Scanlan Engineering Manager) J. Schultz Senior Computer Systems Analyst M. Schwiker Senior Project Manager for Management Model Project
H. Strahley Assistant Operations Manager for Work Control
- J. Waclawski Programs Supervisor, Programs Engineering
Enclosure
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Closed
- 05000388/LER-2012-003-01 LER Unit 2 Automatic Scram While Performing Turbine Control Valve Surveillance Testing (Section
- 4OA3.1)
Opened/Closed
- 05000388/2014008-01 NCV Inadequately Maintained Procedures for Plant Shutdown to Hot Standby led to Reactor Water Level and Power Transients (Section 4OA3 02.03.f.1)
LIST OF DOCUMENTS REVIEWED
Procedures
- AD-2006, Accountability With Respect Model, Revision 0
- Design Considerations Applicability Sheet No. 36, Setpoint Selection and Control, Revision 9
- Design Considerations Applicability Sheet No. 78, Operating & Design Margin Analysis, Revision 2 Design Considerations Applicability Sheet No. 80, Interface and Related Analysis, Revision 2
- EO-000-100, Operating Cautions, Revision 10
- GDG-17, Design Guide for Failure Modes and Effects Analysis, Revision 2
- GO-100-002, Plant Startup, Heatup, and Power Operation, Revision 89
- GO-100-004, Plant Shutdown to Minimum Power, Revision 68
- GO-200-002, Plant Startup, Heatup, and Power Operation, Revision 75
- GO-200-004, Plant Shutdown to Minimum Power, Revision 65
- HU-AD-003, Human Performance Tools Implementation Guide, Revision 10
- LS-115, Operating Experience Program, Revision 0
- LS-115-1004, Manual for Processing Non-IER Operating Experience Documents, Revision 0
- LS-125, Corrective Action Program, Revision 1
- LS-125-1001, Root Cause Analysis Manual, Revision 0
- LS-125-1003, Apparent Cause Evaluation Manual, Revision 0
- LS-125-1004, Effectiveness Review Manual, Revision 0
- LS-125-1005, Work Group Evaluation Manual, Revision 0
- MFP-QA-1220, Engineering Change Process Handbook, Revision 18 and 19
- NDAP-00-0007, Governance, Oversight, Support and Perform Model, Revision 3
- NDAP-QA-0029, Procedure and Work Instruction Use and Adherence, Revision 24
- NDAP-00-0032, Human Performance (HuP) - Standards for Error and Event Prevention,
- Revision 16
- NDAP-00-0036, Management of Observations, Revision 7
- NDAP-00-0708, Corrective Action Review Board, Revision 16
- Experience Report (SOER) Review Program, Revision 8
- Enclosure
- NDAP-00-0752, Cause Analysis, Revision 18
- NDAP-00-0752, Cause Analysis, Revision 20
- NDAP-00-0753, Common Cause Analysis, Revision 2
- NDAP-QA-0004, Procedure Change Process, Revision 11
- NDAP-QA-0027, Station Component Verification Requirements, Revision 12
- NDAP-QA-0029, Procedure and Work Instruction use and Adherence, Revision 21
- NDAP-QA-0029, Procedure and Work Instruction Use and Adherence, Revision 24
- NDAP-QA-0043, Work Package Standard, Revision 1
- NDAP-QA-0302, System Status and Equipment Control, Revision 28
- NDAP-QA-0701, Action Request and Condition Report Process, Revision 41
- NDAP-QA-0702, Action Request and Condition Report Process, Revision 34
- NDAP-QA-0702, Action Request and Condition Report Process, Revision 38
- NDAP-QA-0725, Operating Experience Review Program, Revision 16
- NDAP-QA-1213, Control and Use of Vendor Technical Information, Revision 9
- NDAP-QA-1902, Integrated Risk Management, Revision 12
- NSEI-AD-506, Computer System Problem Reporting, Revision 22
- NSEI-AD-506, Computer System Problem Reporting, Revision 23
- OI-AD-096, Operator Burdens, Revision 10
- OI-AD-096, Operator Burdens, Revision 11
- OI-AD-096, Operator Burdens, Revision 12
- ON-183-001, Stuck Open Safety Relief Valve, Revision 30
- ON-245-001, RPV Level Control System Malfunction, Revision 35
- ON-RPV-102, Stuck Open SRV, Revision 0
- OP-AD-002, Standards for Shift Operations, Revision 43
- OP-AD-001, Operations Standards for System and Equipment Operation, Revision 50
- OP-AD-003, Shift Surveillance Scheduling, Log Sheets, Turnover Sheets and Rounds, Revision
- 46
- OP-AD-004, Operations Standards for Error and Event Prevention, Revision 31
- OP-AD-004, Operations Standards for Error and Event Prevention, Revision 32
- OP-AD-055, Operations Procedure Program, Revision 14
- OP-145-005, Infrequent Manual RFP System Operations, Revision 13
- OP-145-005, Infrequent Manual RFP System Operations, Revision 18
- OP-245-001, Feedwater System Operation, Revision 79
- OP-245-005, Infrequent Manual RFP System Operations, Revision 3
- OP-245-005, Infrequent Manual RFP System Operations, Revision 8
- OP-245-005, Infrequent Manual RFP System Operations, Revision 11
- PSP-39, Outage Scope Control Process, Revision 1
- Enclosure
- TP-245-030, Feedwater Level Control Calibration, Revision 0
Drawings
- D107278, Feedwater Heater Drain Valves, Revision 9
- 07F717310-FD-2384, Reactor Feedpump 2A Isolation Valves, Revision 5, Sheet 1 07F717310-FD-2384, Reactor Feedpump 2A Isolation Valves, Revision 6, Sheet 2 FF62201, ICS Block Diagram, Revision 0, Sheet 1
- FF62201, ICS Logic Diagram, Revision 1, Sheet 144-152
- FF62208, Symbols and Legend, Revision 0, Sheet 674-677
Condition Reports
(*denotes NRC identified during this inspection)
- 1640540
- 1652377
- 1652942
- 1653480
- 1654543
- 1654546
- 1659749
- 1659793
- 1660805
- 1661456
- 1661470
- 1661485
- 1661759
- 1661762
- 1663285
- 1665479
- 1668242
- 1676146
- 1747717
- 1257416
- 1262545
- 1264167
- 1264270
- 1268244
- 1492016
- 1229913
- 1244072
- 1259764
- 1262202
- 1265360
- 1265467
- 1265808
- 1266341
- 1276263
- 1280496
- 1290487
- 291128
- 1291610
- 1295385
- 1307282
- 1320241
- 1324257
- 1332762
- 1333056
- 1334477
- 1335391
- 1335400
- 1336149
- 1336150
- 1336623
- 1352609
- 1357242
- 1358708
- 1359821
- 1364253
- 1364331
- 1539893
- 1290487
- 1364331
- 1653480
- 1751294
- 1447441
- 1654543
- 1659749
- 1676146
- 1734446
- 1746169
- 1264270
- 1496453
- 1539729
- 1496453
- 1539729
- 219069
- 1539729
- 1694052
- 1687440
- 1712099
- 1746169
- 1652338
- 1653480
- 1659749
- 1665479
- 1676146
- 1735339 *1653828 *1661680 *1661681
- 2013-01143 2013-04833 2013-06368 2014-16284 2014-16419 2014-19105
- 2014-22660 2014-22672 2013-04833 2014-22612 2014-24532 2014-08941
- 2014-09518 2014-10388 2014-24288 2013-01143 2014-09434 2013-06174
- 2013-06132 2014-01801 2013-02080 2014-19154 2013-01143 *2014-23005 *2014-24469 *2014-24471 *2014-24616 *2014-23458 *2014-23298
- 2014-24288 *2014-24447 *2014-24451 *2014-24486 *2014-24532 *2014-23597
- 2014-24286 *2014-23299 *2014-23357 *2014-23409 *2014-24160 *2014-24340
- 2014-24402 *2014-24458 *2014-24550 *2014-23028
Work Orders
- 1293989
- 1456387
- 1632895
- 1654350
- 1667779
- 1750451
- 1313894
- 1823265
Miscellaneous
- Maintenance Rule (a)(1) Systems List, dated July 24, 2014
- Operations Directive 12-07, January 29, 2013, Revision 3
- Operations Directive 12-08, December 28, 2012, Revision 0
- Operations Directive 14-02, June 12, 2014, Revision 0
- Enclosure Hot Box 12-48, Operation's Phase I Package, dated December 23, 2012 Hot Box 13-22, Revision to
- OI-AD-096 Operator Burdens Procedure, dated May 2, 2013
- Maintenance Rule Basis Document, System 245-Feedwater, dated October 9, 2012 MRFF Evaluation Summary, System 245-Feedwater,
- CRA 1654245
- CRA 1746905
- RCA 1746169, dated November 15, 2013 Operations Stand Down,
- CR-2014-10388 Attachment M
- EC-048-1018, Reactor Feed Pump Turbine ICS Setpoints, Revision 2
- EC-1694052 ICS Changes to Address 2012 Scram Items, Revision 0
- EC-1811069, ICS Single Point Vulnerability & Diversity Project, Revision 0
- SIP-0881
- IDCN-1, Site Installation Procedure for
- PTR-110912, Simulator Test Procedure - Plant Transient Review of Unit 2 Manual Scram following Lockup of ICS, dated 11/9/12
- PTR-121612, Simulator Test Procedure - Plant Transient Review of Unit 2 Trip during Turbine Control Valve Testing, dated 12/16/12
- SI-245-501, Feedwater & Turbine Logic System Functional Test, dated 5/31/11
- TP-164-042, Reactor Recirculation Scoop Tube Calibration, dated 6/8/14
- TP-245-028, Initial Operation of ICS in Modes 4 & 5, dated 6/23/11
- ACT-1456123, Focused Self-Assessment of the Station Procedure Program, dated 3/24/14
- CSPR-1724646, Revise Simulator Level 3 RPS Trip Setpoints, Revision 0
- Enclosure
LIST OF ACRONYMS
- CA [[]]
PR Corrective Action to Prevent Recurrence
CCA Common Cause Assessment
CDF Core Damage Frequency
DPM Discharge Pressure Mode EC Engineering Change
- HP [[]]
ICS Integrated Controls System IMC Inspection Manual Chapter
IP Inspection Procedure
LERP Large Early Release Probability NCV Non-Cited Violation
NOV Notice of Violation
NRC U.S. Nuclear Regulatory Commission
- PPL [[]]
- RC [[]]
IC Reactor Core Isolation Cooling
- SEP [[]]
- SS [[]]
ES Susquehanna Steam Electric Station
TS Technical Specification
Unit 1 Susquehanna Steam Electric Station Unit 1 Unit 2 Susquehanna Steam Electric Station Unit 2