IR 05000311/2017011: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
||
(9 intermediate revisions by the same user not shown) | |||
Line 2: | Line 2: | ||
| number = ML17297A437 | | number = ML17297A437 | ||
| issue date = 10/24/2017 | | issue date = 10/24/2017 | ||
| title = | | title = Supplemental Inspection Report 05000311/2017011 and Assessment Follow-Up Letter | ||
| author name = Bower F | | author name = Bower F | ||
| author affiliation = NRC/RGN-I/DRP/PB3 | | author affiliation = NRC/RGN-I/DRP/PB3 | ||
| addressee name = Sena | | addressee name = Sena P | ||
| addressee affiliation = PSEG Nuclear, LLC | | addressee affiliation = PSEG Nuclear, LLC | ||
| docket = 05000311 | | docket = 05000311 | ||
Line 18: | Line 18: | ||
=Text= | =Text= | ||
{{#Wiki_filter: UNITED STATES NUCLEAR REGULATORY COMMISSION | {{#Wiki_filter: UNITED STATES NUCLEAR REGULATORY COMMISSION | ||
SUBJECT: SALEM GENERATING STATION UNIT 2 - SUPPLEMENTAL INSPECTION REPORT 05000311/2017011 AND ASSESSMENT FOLLOW-UP LETTER | ==REGION I== | ||
2100 RENAISSANCE BLVD., Suite 100 KING OF PRUSSIA, PA 19406-2713 October 24, 2017 Mr. Peter P. Sena, III President and Chief Nuclear Officer PSEG Nuclear LLC - N09 P.O. Box 236 Hancocks Bridge, NJ 08038 SUBJECT: SALEM GENERATING STATION UNIT 2 - SUPPLEMENTAL INSPECTION REPORT 05000311/2017011 AND ASSESSMENT FOLLOW-UP LETTER | |||
==Dear Mr. Sena:== | ==Dear Mr. Sena:== | ||
On September 15, 2017, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95001, | On September 15, 2017, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95001, Supplemental Inspection Response to Action Matrix Column 2, at your Salem Generating Station Unit 2. On September 15, 2017, the NRC inspection team and an NRC Region I Branch Chief discussed the inspection results and the implementation of your corrective actions with Mr. C. McFeaters and other members of your staff during an inspection exit meeting and subsequent Regulatory Performance Meeting. The results of the inspection are documented in the enclosed report. | ||
In accordance with the NRC Reactor Oversight Process (ROP) Action Matrix, this supplemental inspection was conducted within the Regulatory Response Column of the | In accordance with the NRC Reactor Oversight Process (ROP) Action Matrix, this supplemental inspection was conducted within the Regulatory Response Column of the NRCs ROP Action Matrix because of one White performance Indicator (PI) for Unplanned Reactor Scrams per 7,000 hours Critical. Specifically, Salem Unit 2 crossed the threshold from Green to White in the second quarter of 2016 following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor trip in June 2016. The PI remained White when a fifth reactor trip occurred in August 2016. The details behind each reactor trip are discussed in this inspection report. In response to this Action Matrix input, the NRC informed you in the 2016 mid-cycle letter dated August 31, 2016 (Reports 05000272/2016005 and 05000311/2016005; ML16242A0991), that a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, would be required. The NRC staff was informed on July 27, 2017, of your staffs readiness for this supplemental inspection. | ||
. Specifically, Salem Unit 2 crossed the threshold from Green to White in the second quarter of 2016 following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor | |||
. In response to this Action Matrix input, the NRC informed you in the 2016 mid-cycle letter dated August 31, 2016 (Reports 05000272/2016005 and 05000311/2016005; ML16242A0991), that a supplemental inspection using Inspection Procedure 95001, | |||
The NRC | The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes of risk-significant performance issues were understood; (2) the extent of condition and extent of cause of risk-significant performance issues were identified; and (3) corrective actions, taken or planned, for risk-significant performance issues are sufficient to address the root and contributing causes and prevent recurrence. The inspection consisted of examination of activities conducted under your license as they related to safety, compliance with the Commissions rules and regulations, and the conditions of your operating license. Based on the results of this inspection, the NRC concluded that PSEGs evaluation of the root and contributing causes associated with the five unplanned reactor scrams at Salem Unit 2, as well as the overall adverse scram trend, is appropriate. PSEG appropriately identified the root causes to include inadequate management focus on processes and programs to ensure equipment reliability (ER), such as the mitigation or elimination of single point vulnerabilities that led to several of these reactor scrams, as well as weaknesses in the implementation of the corrective action program. Overall, PSEG satisfactorily identified the individual and collective performance issues associated with this White PI and has appropriate corrective actions either implemented or planned to address the issues that led to this White PI. Key corrective actions included: 1) replacing certain high consequence protective relays and/or converting them to digital; 2) implementing causal evaluation training for the engineering staff; 3) installing a modification to prevent future water hammers when returning containment fan cooler units to service; 4) mitigating or eliminating a large number of single point vulnerabilities; 5) implementing management tools to focus on monitoring and improving ER; and, 6) increasing corrective action closure oversight. | ||
Based on the results of this inspection, the NRC concluded that PSEG performed a satisfactory evaluation of the White PI, the supplemental inspection objectives were met, and no significant weaknesses or findings of significance were identified. Based on the guidance in Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program, and the results of this inspection, the White PI will be closed. Additionally, since Salem Unit 2 has successfully completed this inspection, a Regulatory Performance Meeting has been held, and the PI returned to Green during the first quarter of 2017, Unit 2 will transition from the Regulatory Response Column of the NRCs ROP Action Matrix to the Licensee Response Column effective the date of this inspection report. | |||
In accordance with Title 10 of the Code of Federal Regulations (10 CFR) 2.390 of the NRC's Rules of Practice, a copy of this letter, its enclosure, and your response (if any), will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System component of the NRC's Agencywide Documents Access and Management System (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). | |||
Sincerely, | |||
/RA/ | |||
Fred L. Bower, Chief Reactor Projects Branch 3 Division of Reactor Projects Docket No. 50-311 License No. DPR-75 | |||
===Enclosure:=== | ===Enclosure:=== | ||
Inspection Report | Inspection Report 05000311/2017011 w/Attachment: Supplementary Information | ||
== | REGION I== | ||
Docket No. 50-311 License No. DPR-75 Report No. 05000311/2017011 Licensee: PSEG Nuclear LLC (PSEG) | |||
Facility: Salem Generating Station (Salem) Unit 2 Location: Hancocks Bridge, New Jersey Dates: September 11 - 15, 2017 Inspectors: R. Barkley, PE, Senior Project Engineer, Team Leader N. Warnek, Senior Allegation Coordinator M. Hardgrove, Project Engineer (Observer) | |||
Approved by: Fred L. Bower, Chief Reactor Projects Branch 3 Division of Reactor Projects Enclosure | |||
PSEG Nuclear LLC (PSEG) Facility: | |||
Salem Generating Station (Salem) Unit 2 Location: | |||
R. Barkley, PE, Senior Project Engineer, Team Leader N. Warnek, Senior Allegation Coordinator M. Hardgrove | |||
, Project Engineer (Observer) | |||
Approved by: Fred L. Bower, Chief Reactor Projects Branch 3 Division of Reactor Projects | |||
=SUMMARY= | =SUMMARY= | ||
Inspection | Inspection Report 05000311/2017011; 09/11/2017 - 09/15/2017; Salem Generating Station (Salem) Unit 2; Supplemental Inspection - Inspection Procedure (IP) 95001 A Senior Project Engineer from the Division of Reactor Projects, U.S. Nuclear Regulatory Commission (NRC) Region I and the Senior Allegation Coordinator (inspector qualified) from the Office of the Regional Administrator, NRC Region I, performed this inspection. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 6. | ||
Report 05000311/2017011; | |||
U.S. Nuclear Regulatory Commission (NRC) Region I and the Senior Allegation Coordinator (inspector qualified) from the Office of the Regional Administrator | |||
, NRC Region I, performed this inspection. | |||
The | |||
===Cornerstone: Initiating Events=== | ===Cornerstone: Initiating Events=== | ||
The NRC staff performed this supplemental inspection pursuant to IP 95001, | The NRC staff performed this supplemental inspection pursuant to IP 95001, Supplemental Inspection Response to Action Matrix Column 2 Inputs, because of a White performance indicator (PI) associated with the Initiating Events Cornerstone. The PI involved Unplanned Reactor Scrams per 7,000 hours Critical. The PI turned White following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor trip in June 2016. The PI remained White when a fifth reactor trip was incurred in August 2016. The PI returned to Green as of the first quarter of 2017. The details behind each reactor trip is discussed within this inspection report. The NRC staff was informed on July 27, 2017, of your staffs readiness for this supplemental inspection. | ||
. The details behind each reactor trip is discussed within this inspection report. The NRC staff was informed on July 27, 2017, of your | |||
Based on the results of the inspection, no significant weaknesses or findings were identified. | Based on the results of the inspection, no significant weaknesses or findings were identified. | ||
The inspectors concluded that PSEG had adequately performed a total of six (6)root cause analyses for the five events as well as the overall adverse scram trend. PSEG appropriately identified the root | The inspectors concluded that PSEG had adequately performed a total of six (6) root cause analyses for the five events as well as the overall adverse scram trend. PSEG appropriately identified the root causes to include inadequate management focus on processes and programs to ensure equipment reliability (ER), such as the mitigation or elimination of single point vulnerabilities (SPVs) that led several of these reactor scrams, as well as weaknesses in the implementation of the corrective action program (CAP). With regard to the safety culture of the organization, the most significant issues were noted in the attributes of problem identification, decision-making, leadership safety values and actions, and questioning attitude. | ||
: | |||
-inch bypass line on the service water ( | Key corrective actions to address these root and contributing causes included: 1) placing the heater drain pumps (HDPs) in the Long Term Asset Management (LTAM) program; 2) replacing certain high consequence protective relays and/or converting them to digital (42 in total); 3) implementing plans for the replacement of multiple 25 kV monitoring system components that are obsolete or out of service; 4) implementing Causal Evaluation training for the Engineering staff; 5) installing a 1-inch bypass line on the service water (SW) supply line to the containment fan cooler units (CFCUs) to prevent future water hammers when returning the system to service; 6) installing modifications to eliminate a number of SPVs (61 in total); | ||
SW) supply line to the containment fan cooler units (CFCUs) to prevent future water hammers when returning the system to service; 6) installing modifications to eliminate a number of SPVs (61 in total); | |||
7) implementing management tools to focus on monitoring and improving ER, including an ER Heat Map; 8) providing Line-Of-Sight review for CAP evaluations; and, 9) performing Corrective Action Closure Boards on a monthly basis. | 7) implementing management tools to focus on monitoring and improving ER, including an ER Heat Map; 8) providing Line-Of-Sight review for CAP evaluations; and, 9) performing Corrective Action Closure Boards on a monthly basis. | ||
The team also reviewed the corrective actions that addressed the White PI for Unplanned Scrams per 7000 Hours Critical at Salem Unit 1 since the conclusion of its 95001 inspection in August 2015. The station identified that the corrective actions implemented to address that White PI had not effectively corrected the significant conditions adverse to quality because the station received a repeat White PI for Unplanned Scrams on Unit 2 one year later. PSEGs review noted that the four reactor trips at Unit 1 in 2014-2015 were for different reasons, and the focus of the prior efforts was on the recovery of margin and how to prevent crossing the threshold for this White PI, versus improving ER to prevent future scrams/trips. PSEG found that some of the corrective actions implemented were either too broad or were not effective in the long-term. Moreover, some corrective actions were changed over time such that they did not meet the intent of the action originally planned. These changes were made without adequate review and approval by the PSEG senior management team. The root cause evaluations (RCEs) for the Unit 2 White PI did touch on the issues noted at Unit 1 in 2014-2015, but focused predominately on enhancing the corrective action process and improving ER in order to prevent reactor scrams/trips. The RCEs performed in support of this Unit 2 supplemental inspection also looked back to 2010 to ensure that the causes identified were aligned with the reactor scram/trip issues noted at both Salem units since then. | |||
The station identified that the corrective actions implemented to address that White PI had not effectively corrected the significant conditions adverse to quality because the station received a repeat White PI for Unplanned Scrams on Unit one year later. | |||
-2015 were for different reasons, and the focus of the prior efforts was on the recovery of margin and how to prevent crossing the threshold for this White PI, versus improving ER to prevent future scrams/trips. PSEG found that some of the corrective actions implemented were either too broad or were not effective in the long-term. Moreover, some corrective actions were changed over time such that they did not meet the intent of the action originally planned. These changes were made without adequate review and approval by the PSEG senior management team. | |||
The numerous corrective actions, both completed and planned, for this Unit 2 White PI were reasonable to address the related issues. Based on the guidance in IMC 0305, Operating Reactor Assessment Program, dated November 17, 2016, and the results of this inspection, the White PI will be closed. Additionally, a regulatory performance meeting was held immediately following the inspection exit meeting on September 15, 2017. Since Salem has successfully completed this inspection, a Regulatory Performance Meeting has been held, and the PI returned to Green effective the first quarter of 2017, Salem Unit 2 will transition to the Licensee Response Column of the reactor oversight process (ROP) Action Matrix effective the date of this inspection report. (Section 4OA4) | |||
, 2017. Since Salem has successfully completed this inspection, a Regulatory Performance Meeting has been held, and the PI returned to Green effective the first quarter of 2017 | |||
, Salem Unit 2 will transition to the Licensee Response Column of the reactor oversight process (ROP) Action Matrix effective the date of this inspection report | |||
. | |||
=REPORT DETAILS= | =REPORT DETAILS= | ||
Line 110: | Line 68: | ||
==OTHER ACTIVITIES== | ==OTHER ACTIVITIES== | ||
{{a|4OA3}} | {{a|4OA3}} | ||
==4OA3 Follow-up of Events and Notices of Enforcement== | ==4OA3 Follow-up of Events and Notices of Enforcement Discretion== | ||
(Closed) Licensee Event Report (LER) 05000311/2015-002-01: Reactor Trip Due to Loss of 4kV Non-Vital Group Bus | |||
-Vital Group Bus | |||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
This LER (05000311/2015 00) was originally submitted by PSEG in October 2015, following an August 5, 2015, Unit 2 reactor trip. The LER presented the results of | This LER (05000311/2015-002-00) was originally submitted by PSEG in October 2015, following an August 5, 2015, Unit 2 reactor trip. The LER presented the results of PSEGs RCE for this event, which concluded that there was no definitive cause. PSEG had determined that the most probable cause was a ground fault on the 21 HDP that was not isolated by its associated neutral overcurrent relay, resulting in the loss of the 2H bus, including the 21 reactor coolant pump (RCP), and leading to the subsequent reactor trip. The LER was reviewed and closed by the NRC under inspection report 05000311/2016001 (ML16133A087). | ||
In preparation for the 95001 inspection, PSEG revised the RCE for this event and definitively concluded that the cause of the trip was that the 21 HDP motor neutral overcurrent relay did not actuate in response to a ground fault on the 21 HDP motor. | |||
The relay failed to actuate due to aging or foreign material. PSEG submitted a revised LER (05000311/2015-002-01) to ensure alignment with this root cause. The inspectors reviewed the revised LER, the revised RCE and corrective actions, and interviewed PSEG staff. No findings or violations of NRC requirements were identified. This LER is closed. | |||
. This LER is closed. | |||
====b. Findings==== | ====b. Findings==== | ||
No findings were identified (Closed) Licensee Event Report 05000311/2016 01: | No findings were identified (Closed) Licensee Event Report 05000311/2016-002-01: Automatic Reactor Trip due to Main Turbine Trip | ||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
This LER (05000311/2016 00) was originally submitted by PSEG in April 2016, following a February 4, 2016, Unit 2 reactor trip. The LER presented the results of | This LER (05000311/2016-002-00) was originally submitted by PSEG in April 2016, following a February 4, 2016, Unit 2 reactor trip. The LER presented the results of PSEGs original RCE for this event, which concluded that the reactor trip was caused by a main generator protective relay (model STV1) that actuated unexpectedly due to setpoint drift. The LER was reviewed and closed by the NRC under inspection report 05000311/2016002 (ML17101A407), which documented a Green, self-revealing FIN for PSEG work orders not specifying the appropriate modification testing for the protective relay. | ||
-revealing FIN for PSEG work orders not specifying the appropriate modification testing for the protective relay. | |||
In preparation for the 95001 inspection, PSEG revised the RCE for this event. The revised RCE considered the inadequate work order to be a contributing cause, but pointed to a new root cause: | In preparation for the 95001 inspection, PSEG revised the RCE for this event. The revised RCE considered the inadequate work order to be a contributing cause, but pointed to a new root cause: that PSEG did not develop and implement adequate corrective actions after Unit 1 experienced a similar trip in December 2012 due to STV1 relay setpoint drift. The inspectors reviewed the revised LER (05000311/2016-002-01),the revised RCE and corrective actions, and interviewed PSEG staff. No additional findings or violations of NRC requirements were identified. This LER is closed. | ||
that PSEG did not develop and implement adequate corrective actions after Unit 1 experienced a similar trip in December 2012 due to STV1 relay setpoint drift. The inspectors reviewed the revised LER (05000311/2016 01), the revised RCE and corrective actions, and interviewed PSEG staff. No additional findings or violations of NRC requirements were identified. This LER is closed. | |||
====b. Findings==== | ====b. Findings==== | ||
No findings were identified (Closed) Licensee Event Report 05000311/ | No findings were identified (Closed) Licensee Event Report 05000311/2016-005-00: Automatic Reactor Trip Due to Main Generator Trip | ||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
This LER presented the results of | This LER presented the results of PSEGs RCE for this event, which found that this June 2016 generator trip occurred due to a fault internal to the recently replaced A main transformer. The new transformer had wiring running from a current transformer to a connection point. The wiring was wrapped with an adhesive cotton tape. However, over time, the ground wire in this bundle worked loose due to an inadequate termination on one end (which was not visible at fabrication) as well as eventual loosening of the cotton tape that held the wiring bundle together. The wire was suspended in the transformers oil and moved in response the oil flow in this area. The wire eventually caused an intermittent ground fault when it touched the low voltage connections inside the transformer bushing compartment that ultimately resulted in a main generator trip. | ||
PSEG performed extensive monitoring and troubleshooting as well as equipment cleaning (i.e., carbon dioxide pressure washing of the 25 kV bus bar support dielectric coils) and replacement of select components to identify the source of this generator/reactor trip. As the ground fault was intermittent, it was difficult to locate. | |||
Electrical experts from outside PSEG were brought in to assist with the troubleshooting effort. The source of the ground fault was readily identifiable once the decision was made to drain and enter the A main transformer following the elimination of numerous other possible causes over a multi-week troubleshooting effort. | |||
-week troubleshooting effort. | |||
The inspectors reviewed the LER, the final RCE on this scram and pictures of the wiring bundle: 1) at the end of fabrication, 2) following opening of the faulted transformer, and 3) following repairs to the wiring. He also spoke with the PSEG staff member involved in the oversight of the main transformer procurement | The inspectors reviewed the LER, the final RCE on this scram and pictures of the wiring bundle: 1) at the end of fabrication, 2) following opening of the faulted transformer, and 3) following repairs to the wiring. He also spoke with the PSEG staff member involved in the oversight of the main transformer procurement. No findings or violations of NRC requirements were identified. This LER is closed. | ||
. No findings or violations of NRC requirements were identified. This LER is closed. | |||
====b. Findings==== | ====b. Findings==== | ||
No findings were identified | No findings were identified {{a|4OA4}} | ||
==4OA4 Supplemental Inspection (IP 95001)== | |||
{{a|4OA4}} | |||
==4OA4 Supplemental | |||
Inspection (IP 95001) | |||
===.1 Inspection Scope=== | ===.1 Inspection Scope=== | ||
The NRC staff performed this supplemental inspection in accordance with IP 95001, | The NRC staff performed this supplemental inspection in accordance with IP 95001, Supplemental Inspection Response to Action Matrix Column 2 Inputs, to assess PSEGs evaluation of a White PI, which affected the Initiating Events cornerstone in the Reactor Safety strategic performance area. The inspection objectives were: | ||
to assess | * To assure that the root causes and contributing causes of individual and collective significant performance issues are understood. | ||
* To independently assess and assure that the extent of condition and extent of cause of significant individual and collective performance issues are identified. | |||
The inspection objectives were: | * To assure that corrective actions taken to address and preclude repetition of significant performance issues are prompt and effective. | ||
To assure that the root causes and contributing causes of individual and collective significant performance issues are understood. | * To assure that corrective plans direct prompt actions to effectively address and preclude repetition of significant performance issues. | ||
To independently assess and assure that the extent of condition and extent of cause of significant individual and collective performance issues are identified. | |||
In accordance with the NRC ROP, the inspectors performed a supplemental inspection since Salem Unit 2 was in the Regulatory Response Column of the Action Matrix because of one White PI for Unplanned Reactor Scrams per 7,000 Critical hours. | |||
Specifically, Salem Unit 2 crossed the Green-White threshold in the second quarter of 2016 following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor trip in June 2016. The PI remained White when a fifth reactor trip was incurred in August 2016. The five | |||
: (5) reactor scrams/trips at Salem Unit 2 that led to this White PI are described below: | |||
Rx Trip #1: On August 5, 2015, Unit 2 experienced an automatic reactor trip from full power due to a loss of the 21 RCP. The 21 RCP, which is powered from a breaker on the 2H bus, tripped when power to the 2H 4 kV non-vital bus was lost following opening of the infeed breaker from the 2B auxiliary power transformer due to a fault on 21 HDP motor. | |||
Rx Trip #2: On February 4, 2016, Unit 2 automatically tripped from 75 percent power on a generator trip initiated by a main generator automatic voltage regulator (AVR)volts/hertz over-excitation protection (STV1) relay trip. Power had been reduced to 75 percent to support removal of a 500 kV line offsite. The STV1 relay was found to have a manufacturing defect which caused the relay setpoint to drift low and actuate when the volts/hertz ratio increased during the power reduction. | |||
On February 4, 2016, Unit 2 automatically tripped from 75 percent power on a generator trip initiated by a main generator automatic voltage regulator (AVR) volts/hertz over | |||
-excitation protection (STV1) relay trip. Power had been reduced to 75 percent to support removal of a 500 kV line offsite. The STV1 relay was found to have a manufacturing defect which caused the relay setpoint to drift low and actuate when the volts/hertz ratio increased during the power reduction. | |||
Rx Trip #3: | Rx Trip #3: On February 14, 2016, Unit 2 experienced a main generator protection trip signal which led to a reactor trip from 100 percent power. The trip was later found to be caused by water dripping into the stator water cooling trip Agastat relay (62-C1), causing an electrical short. | ||
On February 14, 2016, Unit 2 experienced a main generator protection trip signal which led to a reactor trip from 100 percent power. The trip was later found to be caused by water dripping into the stator water cooling trip Agastat relay (62 | |||
-C1), causing an electrical short. | |||
Rx Trip #4: | Rx Trip #4: On June 28, 2016, Unit 2 experienced a main generator and subsequent reactor trip. The cause was ultimately traced to a loose current transformer wire on the A main power transformer, which caused an intermittent ground. | ||
On June 28, 2016, Unit 2 experienced a main generator and subsequent reactor trip. | |||
Rx Trip #5: On August 31, 2016, Unit 2 automatically tripped due to the loss of the 21 RCP. The breaker for the 21 RCP opened due to a fault within the inner enclosure assembly of its electrical penetration. The fault was caused by brackish service water falling on the enclosure, which occurred after a water hammer on the 22 CFCU blew out a section of rubber gasket on its motor cooler. The water hammer occurred as flow was being reestablished to the cooler following SW accumulator discharge valve testing. | |||
Rx Trip #5: | |||
On August 31, 2016, Unit 2 automatically tripped due to the loss of the 21 RCP. The breaker for the 21 RCP opened due to a fault within the inner enclosure assembly of its electrical penetration. The fault was caused by brackish service water falling on the enclosure, which occurred after a water hammer on the 22 CFCU blew out a section of rubber gasket on its motor cooler. The water hammer occurred as flow was being reestablished to the cooler following SW accumulator discharge valve testing. | |||
In each of the reactor scrams/trips, safety systems performed as expected and the operators responded to the events properly. Thus none of the five events above involved operator performance issues. | In each of the reactor scrams/trips, safety systems performed as expected and the operators responded to the events properly. Thus none of the five events above involved operator performance issues. | ||
In response to each of these five reactor scrams/trips, PSEG created five RCEs to evaluate the root and contributing causes for these events as well as proposed corrective actions. PSEG also performed an RCE on the overall adverse trend in reactor scrams/trips between August 2015 and August 2016 | In response to each of these five reactor scrams/trips, PSEG created five RCEs to evaluate the root and contributing causes for these events as well as proposed corrective actions. PSEG also performed an RCE on the overall adverse trend in reactor scrams/trips between August 2015 and August 2016. | ||
The inspectors also conducted an in-plant walkdown of the affected components in the Unit 2 turbine building, noting modifications that had been completed since these reactor trips occurred. | The inspectors reviewed all six of PSEGs RCEs, reviewed applicable CAP documents, and interviewed licensing, engineering, maintenance, and operations management personnel to ensure that the root and contributing causes were understood and corrective actions taken or in progress were appropriate. The inspectors also conducted an in-plant walkdown of the affected components in the Unit 2 turbine building, noting modifications that had been completed since these reactor trips occurred. | ||
===.2 Evaluation of the=== | ===.2 Evaluation of the 95001 Supplemental Inspection Requirements=== | ||
===.2.1 Problem Identification=== | |||
a. IP 95001 requires that the inspection staff determine that PSEGs evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC identified) and the conditions under which the issue was identified. | |||
In each of the five reactor trips that caused the PI to turn White, PSEG documented who identified the cause of the trip and the conditions under which the issue was identified. | |||
In each case, the cause of the trip was either self-revealing or identified by PSEG after troubleshooting. The exception was the first reactor trip, which involved an electrical failure of the 21 HDP and subsequent loss of the 2H 4kV bus. The reason why the protective relay (which should have isolated the 21 HDP by opening its breaker before the 2H bus was lost) did not operate properly could not be definitely identified, although the potential causes were narrowed to aging (the relay was original equipment) or foreign material on a key internal component. As a precaution, the relay was replaced. | |||
-revealing or identified by PSEG after troubleshooting | |||
. The exception was the first reactor trip, which involved an electrical failure of the 21 HDP and subsequent loss of the 2H 4kV bus. The reason why the protective relay (which should have isolated the 21 HDP by opening its breaker before the 2H bus was lost) did not operate properly could not be definitely identified, although the potential causes were narrowed to aging (the relay was original equipment) or foreign material on a key internal component. As a precaution, the relay was replaced. | |||
The inspectors determined this inspection objective was met. | The inspectors determined this inspection objective was met. | ||
b. IP 95001 requires that the inspection staff determine that | b. IP 95001 requires that the inspection staff determine that PSEGs evaluation of each issue documents how long the issue existed and prior opportunities for identification. | ||
In each of the five reactor trips, PSEG documented how long the issue which caused the reactor trip existed and prior opportunities for identification. PSEG ultimately determined the following: | In each of the five reactor trips, PSEG documented how long the issue which caused the reactor trip existed and prior opportunities for identification. PSEG ultimately determined the following: | ||
In the first reactor trip, the testing data over several years indicated that the 21 HDP motor was acceptable, but showed a clear trend that the motor insulation was degrading; In the second reactor trip, the generator | * In the first reactor trip, the testing data over several years indicated that the 21 HDP motor was acceptable, but showed a clear trend that the motor insulation was degrading; | ||
. During the most recent Unit 2 | * In the second reactor trip, the generator STV1 relay had experienced performance problems since 2005, with multiple replacements over the years. This same relay had caused a reactor trip on Unit 1 in 2012. During the most recent Unit 2 STV1 relay replacement, a few months before the 2016 trip, problems were experienced during acceptance testing that could have led to the identification of the degraded relay; | ||
-located stator water line and the relay dated to the replacement of the Unit 2 main generator in 1992; | * In the third reactor trip, no definitive time was found for when the water leak on a relay developed, although the design of the panel that housed the co-located stator water line and the relay dated to the replacement of the Unit 2 main generator in 1992; | ||
SW)-induced water hammers since a 1998 modification | * In the fourth reactor trip, the current transformer wire came loose at some point following its entry into service in November 2015 and the trip in June 2016, without prior, definitive warning; and, | ||
, and apparent cause evaluations (ACEs) had been performed on four separate occasions for significant leaks since that time. | * The fifth reactor trip was caused by a CFCU motor cooler gasket leak that led to the wetting of the RCP penetration immediately prior to the trip. However, the system was susceptible to service water (SW)-induced water hammers since a 1998 modification, and apparent cause evaluations (ACEs) had been performed on four separate occasions for significant leaks since that time. | ||
The inspectors determined this inspection objective was met. | The inspectors determined this inspection objective was met. | ||
c. IP 95001 requires that the inspection staff determine that | c. IP 95001 requires that the inspection staff determine that PSEGs evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issue(s). | ||
In each of the five trips, the operators responded as expected and the trips were uncomplicated. The trips all involved equipment that was not safety-related, with the exception of the safety-related SW leak from the CFCU which sprayed down on a non-safety related reactor coolant pump electrical penetration and caused a short to ground. | |||
Specific regulatory compliance concerns, if any, were addressed by the resident inspectors in their quarterly inspection reports. | |||
The increased frequency of reactor trips during this 12 month period (August 2015 | The increased frequency of reactor trips during this 12 month period (August 2015 - | ||
August 2016) does impact the Probabilistic Risk Assessment (PRA) model for Salem, but not to a significant degree as uncomplicated trips are a small percentage | |||
, but not to a significant degree as uncomplicated trips are a small percentage (~10 percent) of the PRA initiating events that result in increasing the core damage frequency | (~10 percent) of the PRA initiating events that result in increasing the core damage frequency. | ||
. | |||
The inspectors determined this inspection objective was met. | The inspectors determined this inspection objective was met. | ||
Line 245: | Line 172: | ||
a. IP 95001 requires that the inspection staff determine that PSEG evaluated the issue using a systematic methodology to identify the root and contributing causes PSEG used a systematic methodology to review each event, identifying the root and contributing causes of each of the five reactor trips as well as the overall adverse scram trend. The following root cause evaluation (RCE) methodologies were utilized for the PSEG evaluation: | a. IP 95001 requires that the inspection staff determine that PSEG evaluated the issue using a systematic methodology to identify the root and contributing causes PSEG used a systematic methodology to review each event, identifying the root and contributing causes of each of the five reactor trips as well as the overall adverse scram trend. The following root cause evaluation (RCE) methodologies were utilized for the PSEG evaluation: | ||
The WHY Staircase Barrier Analysis Failure Modes and Effects Analysis Event and Casual Factor (E&CF) Chart Safety Culture Analysis Organizational and Programmatic (O&P) Review Extent of Condition (EOCo) and Extent of Cause (EOCa) Review: | * The WHY Staircase | ||
o Same - Same o Same - Similar | * Barrier Analysis | ||
* Failure Modes and Effects Analysis | |||
* Event and Casual Factor (E&CF) Chart | |||
* Safety Culture Analysis | |||
* Organizational and Programmatic (O&P) Review | |||
* Extent of Condition (EOCo) and Extent of Cause (EOCa) Review: | |||
o Same - Same o Same - Similar o Similar - Same o Similar - Similar | |||
* Operating Experience (OE) Assessment | |||
* Review of Previous Internal Occurrences The inspectors determined this inspection objective was met. | |||
b. IP 95001 requires that the inspection staff determine that | b. IP 95001 requires that the inspection staff determine that PSEGs RCEs were conducted to a level of detail commensurate with the significance of the issue. | ||
The six RCEs performed by PSEG were conducted to a level of detail commensurate with the significance of the issue. Each RCE used a variety of techniques to identify the root and contributing causes, and had documentation as necessary to support the conclusions drawn. The amount of detail varied by the complexity of the issue, with the RCEs on the CFCU leak that ultimately resulted in Rx trip #5 and the adverse scram trend RCEs being the most lengthy and involved. | The six RCEs performed by PSEG were conducted to a level of detail commensurate with the significance of the issue. Each RCE used a variety of techniques to identify the root and contributing causes, and had documentation as necessary to support the conclusions drawn. The amount of detail varied by the complexity of the issue, with the RCEs on the CFCU leak that ultimately resulted in Rx trip #5 and the adverse scram trend RCEs being the most lengthy and involved. | ||
Line 254: | Line 189: | ||
The inspectors determined this inspection objective was met. | The inspectors determined this inspection objective was met. | ||
c. IP 95001 requires that the inspection staff determine that | c. IP 95001 requires that the inspection staff determine that PSEGs RCE included a consideration of prior occurrences of the issue and knowledge of Operating Experience. | ||
The RCEs on each of the five reactor trips included a review of industry operating experience and prior occurrences of the issue. Prior occurrences varied by event, with the CFCU leak that ultimately resulted in a reactor trip (Rx trip #5) having the most, followed by the inadvertent actuation of the STV1 relay (Rx trip #2) being the second. | |||
However some events had little or no precedent, such as the stator cooling water dripping on a relay that caused Rx trip #3. Industry experience was also considered in each case, but provided limited information due to either the uniqueness of the event and/or the uniqueness of the system design (e.g., Salem SW system, which is the only plant in the country equipped with accumulators). | |||
The inspectors determined this inspection objective was met. | |||
d. IP 95001 requires that the inspection staff determine that | d. IP 95001 requires that the inspection staff determine that PSEGs RCE addresses the extent of condition and extent of cause of the issue. | ||
PSEGs six RCEs documented considerable work to address the extent of condition and the extent of cause. For example, inspections were conducted on a range of similar equipment in the plant, such as the remaining two main transformers and a spare at Unit 2, the six HDPs at both units (including a spare in storage), and multiple electrical penetrations that were wetted by SW recently during past events, causing corrosion). In the case of the electrical penetrations, cleaning and repairs were performed on those penetrations boxes found to have significant corrosion. While additional inspections are planned (i.e., internal inspection of the Unit 1 main power transformers (MPTs) in the fall 2017 outage), most were completed at the time of this inspection and discrepancies were documented and addressed. For the adverse scram trend, the extent of cause was focused on reviewing an expanded sample of RCEs and ACEs since 2013 to ensure the evaluations and resulting actions met procedural requirements. For instance, a large number of conditions requiring corrective action (CRCAs) were reviewed for proper closure and implementation. In addition, corrective actions to preclude recurrence (CAPRs) which had a change of intent since their creation received additional management scrutiny. | |||
internal inspection of the Unit 1 main power transformers (MPTs) in the fall 2017 outage | |||
), most were completed at the time of this inspection and discrepancies were documented and addressed. For the adverse scram trend, the extent of cause was focused on reviewing an expanded sample of RCEs and ACEs since 2013 to ensure the evaluations and resulting actions met procedural requirements | |||
. For instance, a large number of conditions requiring corrective action (CRCAs) were reviewed for proper closure and implementation. In addition, corrective actions to preclude recurrence (CAPRs) which had a change of intent since their creation received additional management scrutiny | |||
. | |||
The inspectors determined this inspection objective was met. | The inspectors determined this inspection objective was met. | ||
e. IP 95001 requires that the inspection staff determine that | e. IP 95001 requires that the inspection staff determine that PSEGs root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0305. | ||
PSEG concluded that the various corrective actions proposed for each of the five individual reactor trip RCEs will address most of the safety culture attributes, although an additional performance analysis was performed and corrective actions will be taken to address issues with leadership decision-making. | An analysis was performed which looked at each of the 10 safety culture traits described in IMC 0305. While behaviors were found to have been met in 3 of the 10 traits, weaknesses were identified in the remaining seven traits, with the most significant weaknesses found to be in the areas of: 1) Problem Identification; 2) Decision-Making; 3) Leadership Safety Values and Actions; and, 4) Questioning Attitude. PSEG concluded that the various corrective actions proposed for each of the five individual reactor trip RCEs will address most of the safety culture attributes, although an additional performance analysis was performed and corrective actions will be taken to address issues with leadership decision-making. | ||
The inspectors determined this inspection objective was met. | The inspectors determined this inspection objective was met. | ||
Line 287: | Line 214: | ||
===.3 Corrective Actions=== | ===.3 Corrective Actions=== | ||
a. IP 95001 requires that the inspection staff determine that | a. IP 95001 requires that the inspection staff determine that: | ||
: | : (1) PSEG specified appropriate corrective actions for each root and/or contributing cause, or | ||
: (2) an evaluation that states no actions are necessary is adequate. | |||
PSEG specified appropriate corrective actions for each root and/or contributing cause; none of the evaluations stated that no action was necessary. Key corrective actions included: | |||
* Placing the HDPs in the LTAM program | |||
* Replacing certain high consequence protective feeder relays and/or converting them to digital (42 in total) | |||
* Implementing plans for the replacement of multiple 25 kV monitoring system components that are obsolete or out of service | |||
* Implementing causal evaluation training for the Engineering staff | |||
* Placing a 90-day limit for Long Term Asset (LTA) manager items to be in draft status | |||
* Installing a 1-inch bypass line on the SW supply line to the CFCUs to prevent future water hammers when returning the system to service | |||
* Installing modifications to eliminate a number of SPVs (61 SPVs in total) | |||
* Implementing management tools to focus on monitoring and improving ER, including an ER Heat Map | |||
* Providing Line-Of-Sight review for CAP evaluations | |||
* Performing Corrective Action Closure Boards on a monthly basis. | |||
The inspectors determined this inspection objective was met. | |||
a. IP 95001 requires that the inspection staff determine that PSEG prioritized corrective actions with consideration of risk significance and regulatory compliance. | |||
Each of these reactor trips involved balance of plant or non-safety related equipment (the exception being the CFCU SW leak), so the risk significance was low and regulatory compliance was not an issue. However, PSEG completed (or will soon complete) a number of the corrective actions involving equipment changes. In the case of the CFCU SW leak, which was the only one involving safety | Each of these reactor trips involved balance of plant or non-safety related equipment (the exception being the CFCU SW leak), so the risk significance was low and regulatory compliance was not an issue. However, PSEG completed (or will soon complete) a number of the corrective actions involving equipment changes. In the case of the CFCU SW leak, which was the only one involving safety-related equipment that is inside containment (which limits both access and available outage work windows), interim corrective actions have been implemented to prevent another water hammer until a small bypass line is installed to allow slow refilling of the CFCUs. | ||
-related equipment that is inside containment (which limits both access and available outage work windows), interim corrective actions have been implemented to prevent another water hammer until a small bypass line is installed to allow slow refilling of the CFCUs. | |||
The risk significance and regulatory compliance issues for the overall root | The risk significance and regulatory compliance issues for the overall root causes were addressed in the five individual root causes. An overall risk significance for the multiple reactor trips was provided in the sixth adverse trend RCE. | ||
PSEG procedure ER | The prioritization of corrective actions was generally subjective, with the exception of actions to address SPVs. The SPVs are individual components which, if they fail or actuate, lack redundancy (e.g., 2 out of 3 in a logic circuit) and thus trip a component that will subsequently lead to a reactor trip. PSEG procedure ER-AA-2004, System Vulnerability Review Process Description, Attachment 1 contained a SPV Prioritization Matrix for risk ranking vulnerabilities. | ||
-AA-2004, System Vulnerability Review Process Description, 1 | |||
PSEG used this structured prioritization matrix to rank SPVs and help determine which ones should be eliminated (versus mitigated), as well as which SPVs should be addressed first. | PSEG used this structured prioritization matrix to rank SPVs and help determine which ones should be eliminated (versus mitigated), as well as which SPVs should be addressed first. | ||
Line 314: | Line 246: | ||
b. IP 95001 requires that the inspection staff determine that PSEG established a schedule for implementing and completing the corrective actions. | b. IP 95001 requires that the inspection staff determine that PSEG established a schedule for implementing and completing the corrective actions. | ||
PSEG established a comprehensive schedule for implementing and completing the roughly 150 corrective actions (CAPRs, CRCAs)and action items (ACITs) laid out between the six RCEs. Almost all of the CAPRs, CRCAs and ACITs involving non | PSEG established a comprehensive schedule for implementing and completing the roughly 150 corrective actions (CAPRs, CRCAs) and action items (ACITs) laid out between the six RCEs. Almost all of the CAPRs, CRCAs and ACITs involving non-equipment issues have been completed or are scheduled for completion before the end of 2017. | ||
-equipment issues have been completed or are scheduled for completion before the end of 2017. | |||
The nature of the equipment modifications planned in response to these events (or other related plant or industry initiatives) are, by their nature, long | The nature of the equipment modifications planned in response to these events (or other related plant or industry initiatives) are, by their nature, long-term since they can only be performed during outages and on only one train at a time. For example, PSEG plans to perform actions to mitigate the trip risk posed by the 700+ SPVs identified by their reviews, a time consuming, long-term action. The following other equipment modifications are planned for the following number of plant cycles (on both units): | ||
-term since they can only be performed during outages and on only one train at a time. For example, PSEG plans to perform | * SW to CFCU 1 inch fill bypass lines - 2 cycles | ||
-term action. The following other equipment modifications are planned | * CFCU gasket design changes - 5 cycles | ||
* Relay conversions to digital - 2-3 cycles | |||
- | * Eliminating 61 SPVs - 2-3 cycles | ||
- | * 25 kV monitoring equipment replacements - 2 cycles Although, the progress of the implementation of the above modifications, as well as the completion of key procedural and process CAPRs and CRCAs and their subsequent effectiveness reviews are expected to be inspected during the conduct of future inspections, the inspectors determined this inspection objective was met. | ||
- | |||
, the inspectors determined this inspection objective was met. | |||
c. IP 95001 requires that the inspection staff determine that PSEG developed quantitative and/or qualitative measures of success for determining the effectiveness of the CAPRs. | c. IP 95001 requires that the inspection staff determine that PSEG developed quantitative and/or qualitative measures of success for determining the effectiveness of the CAPRs. | ||
Multiple effectiveness reviews (EFFRs) were planned to confirm the successful implementation of the corrective actions to address each RCE. The EFFRs used a balance of qualitative versus quantitative measures based on the nature of the corrective action (e.g., a definitive hardware change versus training or a programmatic or process change). Although many of the EFFRs cannot be completed until the completion of planned plant modifications | Multiple effectiveness reviews (EFFRs) were planned to confirm the successful implementation of the corrective actions to address each RCE. The EFFRs used a balance of qualitative versus quantitative measures based on the nature of the corrective action (e.g., a definitive hardware change versus training or a programmatic or process change). Although many of the EFFRs cannot be completed until the completion of planned plant modifications, interim EFFRs were done where possible. The EFFRs that were not completed at the time of this inspection are expected to be inspected during the conduct of future inspections; therefore, the inspectors determined this inspection objective was met. | ||
, interim EFFRs were done where possible | |||
. The EFFRs that were not completed at the time of this inspection are expected to be inspected during the conduct of future inspections; therefore, the inspectors determined this inspection objective was met. | d. IP 95001 requires that the inspection staff determine that PSEGs planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable. | ||
This supplemental inspection was based on a White PI versus an inspection finding. | |||
However, three Green NCVs were issued by the resident inspectors in 2016 for performance issues related to the causes of three of the five reactor trips. | |||
====e. Findings==== | ====e. Findings==== | ||
No findings were identified. | No findings were identified. | ||
===.4 Evaluation=== | ===.4 Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues=== | ||
PSEG did not request credit for self-identification of an old design issue involving any of these five reactor trips. | |||
-identification of an old design issue involving any of these five reactor trips. | |||
===.5 Review of PSEG Corrective Actions taken in Response to the 95001 Inspection=== | ===.5 Review of PSEG Corrective Actions taken in Response to the 95001 Inspection=== | ||
Conducted at Salem Unit 1 in August 2015 The team reviewed the corrective actions that addressed the White PI for Unplanned Scrams per 7000 Hours Critical at Salem Unit 1 since the conclusion of the 95001 inspection (05000272/2015 | Conducted at Salem Unit 1 in August 2015 The team reviewed the corrective actions that addressed the White PI for Unplanned Scrams per 7000 Hours Critical at Salem Unit 1 since the conclusion of the 95001 inspection (05000272/2015-009; ML15258A467) in August 2015. The station identified that the corrective actions implemented to address that White PI had not effectively corrected the significant conditions adverse to quality because the station received a repeat White PI for Unplanned Scrams on Unit 2 one year later. PSEGs review noted that the four reactor trips at Unit 1 in 2014-2015 were for different reasons, and the focus of the prior efforts was on the recovery of margin and how to prevent crossing the threshold for this White PI, versus improving ER to prevent future scrams/trips. The involvement of PSEG staff in the prior RCE efforts were also heavily weighted with Regulatory Affairs staff, versus a representative cross-section of the station Operations, Maintenance, and Engineering staff who are focused on equipment issues. | ||
-009; ML15258A467)in August 2015. | |||
In the 2015 RCEs for the Unit 1 scrams/trips, PSEG identified two root causes: | |||
1) behaviors of site leadership did not effectively enforce existing processes to ensure specific and adequate actions were developed, challenged, and implemented to prevent operational challenges leading to reactor scrams; and, 2) ineffective knowledge and use of existing processes to develop plans and actions that could have mitigated the risk of the reduced scram margin. PSEG determined that governance and oversight was ineffective in identifying weaknesses in the application of fundamentals to prevent reactor scrams was also a contributing cause. Thus the corrective actions for the previous White PI were focused on enhancing procedural compliance, especially for new employees, as well as enhancing leadership and accountability. However, PSEG found that some of the corrective actions implemented were either too broad or were not effective in the long-term. Moreover, some corrective actions were changed over time such that they did not meet the intent of the action originally planned. These changes were made without adequate review and approval by the PSEG senior management team. | |||
By contrast, the RCEs prepared for the Unit 2 White PI involved a cross-section of station staff to ensure the inclusion of viewpoints and buy-in of the involved departments. | |||
The | The RCEs did touch on the issues noted at Unit 1 in 2014-2015, but focused predominately on enhancing the corrective action process and improving ER in order to prevent reactor scrams/trips. The RCEs performed in support of this supplemental inspection also looked back to ensure that the causes noted were aligned with the reactor scram/trip issues noted at both Salem units since 2010. | ||
In response to the observation that some corrective actions for the prior White finding experienced significant changes of intent over time, PSEG elected to impose management controls on changes of intent to ensure that such changes receive the proper level of management review and scrutiny. In addition, in contrast to the corrective actions taken for the prior White PI, PSEG is making efforts to improve the reliability of the equipment involved in each of the five reactor trips that contributed to this PI. Thus the team had sufficient basis to conclude that the corrective actions taken in response to this White PI will be more effective than the scram prevention efforts previously undertaken by PSEG. However, since the effectiveness reviews for many of these corrective actions will not be completed for some time, the full effectiveness of these measures are expected to be inspected during the conduct of future inspections. | |||
In response to the observation that some corrective actions for the prior White finding experienced significant changes of intent over time, PSEG elected to impose management controls on changes of intent to ensure that such changes receive the proper level of management review and scrutiny. In addition, in contrast to the corrective actions taken for the prior White PI, PSEG is making efforts to improve the reliability of the equipment involved in each of the five reactor trips that contributed to this PI. Thus the team had sufficient basis to conclude that the corrective actions taken in response to this White PI will be more effective than the scram prevention efforts previously undertaken by PSEG. However, since the effectiveness reviews for many of these corrective actions will not be completed for some time, the full effectiveness of these measures are expected to be inspected during the conduct of future inspections | |||
. | |||
{{a|4OA6}} | {{a|4OA6}} | ||
==4OA6 Exit Meetings== | ==4OA6 Exit Meetings== | ||
On September 15, the inspectors presented the inspection results to Mr. C. McFeaters | On September 15, the inspectors presented the inspection results to Mr. C. McFeaters, Salem Site Vice President, and other members of the PSEG staff. The inspectors verified that no proprietary information was retained by the inspectors or documented in this report. PSEG management acknowledged and did not dispute the findings. The results of the inspection were visually summarized using slides that can be found in ADAMS (ML17291B082). | ||
, Salem Site Vice President, and other members of the PSEG staff. The inspectors verified that no proprietary information was retained by the inspectors or documented in this report. | |||
PSEG management acknowledged and did not dispute the findings. | |||
The results of the inspection were visually summarized using slides that can be found in ADAMS (ML17291B082 | |||
). | |||
=SUPPLEMENTARY INFORMATION= | =SUPPLEMENTARY INFORMATION= | ||
Line 389: | Line 296: | ||
===Licensee Personnel=== | ===Licensee Personnel=== | ||
: [[contact::P. Sena]], Chief Nuclear Officer | : [[contact::P. Sena]], Chief Nuclear Officer | ||
C. | : [[contact::C. McFeaters]], Site Vice President | ||
: [[contact::P. Martino]], Plant Manager | : [[contact::P. Martino]], Plant Manager | ||
: [[contact::P. Davison]], | : [[contact::P. Davison]], Engineering Vice President | ||
Engineering Vice President | |||
: [[contact::R. DeNight]], Engineer Manager | : [[contact::R. DeNight]], Engineer Manager | ||
: [[contact::K. Anderson]], Salem Operations, RCE team leader | : [[contact::K. Anderson]], Salem Operations, RCE team leader | ||
: [[contact::J. Wearne]], Regulatory Assurance | : [[contact::J. Wearne]], Regulatory Assurance Manager | ||
Manager | : [[contact::S. Nevelos]], Manager - Salem Organizational Effectiveness | ||
: [[contact::S. Nevelos]], Manager | |||
- Salem Organizational Effectiveness | |||
: [[contact::M. Brummitt]], I & C Supervisor | : [[contact::M. Brummitt]], I & C Supervisor | ||
: [[contact::F. Possessky]], Salem Performance Improvement | : [[contact::F. Possessky]], Salem Performance Improvement | ||
: [[contact::S. Markos]], Design Engineering Manager | : [[contact::S. Markos]], Design Engineering Manager | ||
: [[contact::K. Hernandez]], Salem Engineering | : [[contact::K. Hernandez]], Salem Engineering | ||
: [[contact::J. Fleming]], Regulatory Assurance | : [[contact::J. Fleming]], Regulatory Assurance Director | ||
Director | |||
: [[contact::R. Flessner]], Contractor, Landis Consulting | : [[contact::R. Flessner]], Contractor, Landis Consulting | ||
: [[contact::K. Landis]], Contractor, Landis Consulting | : [[contact::K. Landis]], Contractor, Landis Consulting | ||
: [[contact::P. Essner]], Engineer | : [[contact::P. Essner]], Engineer | ||
: [[contact::J. Stead]], Engineer | : [[contact::J. Stead]], Engineer | ||
: [[contact::A. Zheng]], Engineer | : [[contact::A. Zheng]], Engineer | ||
Other | Other | ||
: [[contact::E. Rosenfeld]], State of New Jersey Bureau of Nuclear Engineering | : [[contact::E. Rosenfeld]], State of New Jersey Bureau of Nuclear Engineering | ||
==LIST OF ITEMS== | ==LIST OF ITEMS== | ||
Line 418: | Line 321: | ||
===Closed=== | ===Closed=== | ||
: 05000311/2015 | : 05000311/2015-002-01 LER Reactor Trip Due to Loss of 4kV Non-Vital Group Bus (Section 4OA3) | ||
-002-01 LER Reactor Trip Due to Loss of 4kV Non | : 05000311/2016-002-01 LER Automatic Reactor Trip due to Main Turbine Trip (Section 4OA3) | ||
-Vital Group Bus (Section 4OA3) | : 05000311/2016-005-00 LER Automatic Reactor Trip Due to Main Generator Trip (Section 4OA3) | ||
: 05000311/2016 | |||
-002-01 LER Automatic Reactor Trip due to Main Turbine Trip | |||
: 05000311/2016 | |||
-005-00 LER Automatic Reactor Trip Due to Main Generator Trip | |||
==LIST OF DOCUMENTS REVIEWED== | ==LIST OF DOCUMENTS REVIEWED== | ||
}} | }} |
Latest revision as of 10:37, 19 December 2019
ML17297A437 | |
Person / Time | |
---|---|
Site: | Salem ![]() |
Issue date: | 10/24/2017 |
From: | Fred Bower Reactor Projects Branch 3 |
To: | Sena P Public Service Enterprise Group |
References | |
IR 2017011 | |
Download: ML17297A437 (21) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION
REGION I
2100 RENAISSANCE BLVD., Suite 100 KING OF PRUSSIA, PA 19406-2713 October 24, 2017 Mr. Peter P. Sena, III President and Chief Nuclear Officer PSEG Nuclear LLC - N09 P.O. Box 236 Hancocks Bridge, NJ 08038 SUBJECT: SALEM GENERATING STATION UNIT 2 - SUPPLEMENTAL INSPECTION REPORT 05000311/2017011 AND ASSESSMENT FOLLOW-UP LETTER
Dear Mr. Sena:
On September 15, 2017, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95001, Supplemental Inspection Response to Action Matrix Column 2, at your Salem Generating Station Unit 2. On September 15, 2017, the NRC inspection team and an NRC Region I Branch Chief discussed the inspection results and the implementation of your corrective actions with Mr. C. McFeaters and other members of your staff during an inspection exit meeting and subsequent Regulatory Performance Meeting. The results of the inspection are documented in the enclosed report.
In accordance with the NRC Reactor Oversight Process (ROP) Action Matrix, this supplemental inspection was conducted within the Regulatory Response Column of the NRCs ROP Action Matrix because of one White performance Indicator (PI) for Unplanned Reactor Scrams per 7,000 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> Critical. Specifically, Salem Unit 2 crossed the threshold from Green to White in the second quarter of 2016 following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor trip in June 2016. The PI remained White when a fifth reactor trip occurred in August 2016. The details behind each reactor trip are discussed in this inspection report. In response to this Action Matrix input, the NRC informed you in the 2016 mid-cycle letter dated August 31, 2016 (Reports 05000272/2016005 and 05000311/2016005; ML16242A0991), that a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, would be required. The NRC staff was informed on July 27, 2017, of your staffs readiness for this supplemental inspection.
The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes of risk-significant performance issues were understood; (2) the extent of condition and extent of cause of risk-significant performance issues were identified; and (3) corrective actions, taken or planned, for risk-significant performance issues are sufficient to address the root and contributing causes and prevent recurrence. The inspection consisted of examination of activities conducted under your license as they related to safety, compliance with the Commissions rules and regulations, and the conditions of your operating license. Based on the results of this inspection, the NRC concluded that PSEGs evaluation of the root and contributing causes associated with the five unplanned reactor scrams at Salem Unit 2, as well as the overall adverse scram trend, is appropriate. PSEG appropriately identified the root causes to include inadequate management focus on processes and programs to ensure equipment reliability (ER), such as the mitigation or elimination of single point vulnerabilities that led to several of these reactor scrams, as well as weaknesses in the implementation of the corrective action program. Overall, PSEG satisfactorily identified the individual and collective performance issues associated with this White PI and has appropriate corrective actions either implemented or planned to address the issues that led to this White PI. Key corrective actions included: 1) replacing certain high consequence protective relays and/or converting them to digital; 2) implementing causal evaluation training for the engineering staff; 3) installing a modification to prevent future water hammers when returning containment fan cooler units to service; 4) mitigating or eliminating a large number of single point vulnerabilities; 5) implementing management tools to focus on monitoring and improving ER; and, 6) increasing corrective action closure oversight.
Based on the results of this inspection, the NRC concluded that PSEG performed a satisfactory evaluation of the White PI, the supplemental inspection objectives were met, and no significant weaknesses or findings of significance were identified. Based on the guidance in Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program, and the results of this inspection, the White PI will be closed. Additionally, since Salem Unit 2 has successfully completed this inspection, a Regulatory Performance Meeting has been held, and the PI returned to Green during the first quarter of 2017, Unit 2 will transition from the Regulatory Response Column of the NRCs ROP Action Matrix to the Licensee Response Column effective the date of this inspection report.
In accordance with Title 10 of the Code of Federal Regulations (10 CFR) 2.390 of the NRC's Rules of Practice, a copy of this letter, its enclosure, and your response (if any), will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System component of the NRC's Agencywide Documents Access and Management System (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Fred L. Bower, Chief Reactor Projects Branch 3 Division of Reactor Projects Docket No. 50-311 License No. DPR-75
Enclosure:
Inspection Report 05000311/2017011 w/Attachment: Supplementary Information
REGION I==
Docket No. 50-311 License No. DPR-75 Report No. 05000311/2017011 Licensee: PSEG Nuclear LLC (PSEG)
Facility: Salem Generating Station (Salem) Unit 2 Location: Hancocks Bridge, New Jersey Dates: September 11 - 15, 2017 Inspectors: R. Barkley, PE, Senior Project Engineer, Team Leader N. Warnek, Senior Allegation Coordinator M. Hardgrove, Project Engineer (Observer)
Approved by: Fred L. Bower, Chief Reactor Projects Branch 3 Division of Reactor Projects Enclosure
SUMMARY
Inspection Report 05000311/2017011; 09/11/2017 - 09/15/2017; Salem Generating Station (Salem) Unit 2; Supplemental Inspection - Inspection Procedure (IP) 95001 A Senior Project Engineer from the Division of Reactor Projects, U.S. Nuclear Regulatory Commission (NRC) Region I and the Senior Allegation Coordinator (inspector qualified) from the Office of the Regional Administrator, NRC Region I, performed this inspection. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 6.
Cornerstone: Initiating Events
The NRC staff performed this supplemental inspection pursuant to IP 95001, Supplemental Inspection Response to Action Matrix Column 2 Inputs, because of a White performance indicator (PI) associated with the Initiating Events Cornerstone. The PI involved Unplanned Reactor Scrams per 7,000 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> Critical. The PI turned White following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor trip in June 2016. The PI remained White when a fifth reactor trip was incurred in August 2016. The PI returned to Green as of the first quarter of 2017. The details behind each reactor trip is discussed within this inspection report. The NRC staff was informed on July 27, 2017, of your staffs readiness for this supplemental inspection.
Based on the results of the inspection, no significant weaknesses or findings were identified.
The inspectors concluded that PSEG had adequately performed a total of six (6) root cause analyses for the five events as well as the overall adverse scram trend. PSEG appropriately identified the root causes to include inadequate management focus on processes and programs to ensure equipment reliability (ER), such as the mitigation or elimination of single point vulnerabilities (SPVs) that led several of these reactor scrams, as well as weaknesses in the implementation of the corrective action program (CAP). With regard to the safety culture of the organization, the most significant issues were noted in the attributes of problem identification, decision-making, leadership safety values and actions, and questioning attitude.
Key corrective actions to address these root and contributing causes included: 1) placing the heater drain pumps (HDPs) in the Long Term Asset Management (LTAM) program; 2) replacing certain high consequence protective relays and/or converting them to digital (42 in total); 3) implementing plans for the replacement of multiple 25 kV monitoring system components that are obsolete or out of service; 4) implementing Causal Evaluation training for the Engineering staff; 5) installing a 1-inch bypass line on the service water (SW) supply line to the containment fan cooler units (CFCUs) to prevent future water hammers when returning the system to service; 6) installing modifications to eliminate a number of SPVs (61 in total);
7) implementing management tools to focus on monitoring and improving ER, including an ER Heat Map; 8) providing Line-Of-Sight review for CAP evaluations; and, 9) performing Corrective Action Closure Boards on a monthly basis.
The team also reviewed the corrective actions that addressed the White PI for Unplanned Scrams per 7000 Hours Critical at Salem Unit 1 since the conclusion of its 95001 inspection in August 2015. The station identified that the corrective actions implemented to address that White PI had not effectively corrected the significant conditions adverse to quality because the station received a repeat White PI for Unplanned Scrams on Unit 2 one year later. PSEGs review noted that the four reactor trips at Unit 1 in 2014-2015 were for different reasons, and the focus of the prior efforts was on the recovery of margin and how to prevent crossing the threshold for this White PI, versus improving ER to prevent future scrams/trips. PSEG found that some of the corrective actions implemented were either too broad or were not effective in the long-term. Moreover, some corrective actions were changed over time such that they did not meet the intent of the action originally planned. These changes were made without adequate review and approval by the PSEG senior management team. The root cause evaluations (RCEs) for the Unit 2 White PI did touch on the issues noted at Unit 1 in 2014-2015, but focused predominately on enhancing the corrective action process and improving ER in order to prevent reactor scrams/trips. The RCEs performed in support of this Unit 2 supplemental inspection also looked back to 2010 to ensure that the causes identified were aligned with the reactor scram/trip issues noted at both Salem units since then.
The numerous corrective actions, both completed and planned, for this Unit 2 White PI were reasonable to address the related issues. Based on the guidance in IMC 0305, Operating Reactor Assessment Program, dated November 17, 2016, and the results of this inspection, the White PI will be closed. Additionally, a regulatory performance meeting was held immediately following the inspection exit meeting on September 15, 2017. Since Salem has successfully completed this inspection, a Regulatory Performance Meeting has been held, and the PI returned to Green effective the first quarter of 2017, Salem Unit 2 will transition to the Licensee Response Column of the reactor oversight process (ROP) Action Matrix effective the date of this inspection report. (Section 4OA4)
REPORT DETAILS
OTHER ACTIVITIES
4OA3 Follow-up of Events and Notices of Enforcement Discretion
(Closed) Licensee Event Report (LER) 05000311/2015-002-01: Reactor Trip Due to Loss of 4kV Non-Vital Group Bus
a. Inspection Scope
This LER (05000311/2015-002-00) was originally submitted by PSEG in October 2015, following an August 5, 2015, Unit 2 reactor trip. The LER presented the results of PSEGs RCE for this event, which concluded that there was no definitive cause. PSEG had determined that the most probable cause was a ground fault on the 21 HDP that was not isolated by its associated neutral overcurrent relay, resulting in the loss of the 2H bus, including the 21 reactor coolant pump (RCP), and leading to the subsequent reactor trip. The LER was reviewed and closed by the NRC under inspection report 05000311/2016001 (ML16133A087).
In preparation for the 95001 inspection, PSEG revised the RCE for this event and definitively concluded that the cause of the trip was that the 21 HDP motor neutral overcurrent relay did not actuate in response to a ground fault on the 21 HDP motor.
The relay failed to actuate due to aging or foreign material. PSEG submitted a revised LER (05000311/2015-002-01) to ensure alignment with this root cause. The inspectors reviewed the revised LER, the revised RCE and corrective actions, and interviewed PSEG staff. No findings or violations of NRC requirements were identified. This LER is closed.
b. Findings
No findings were identified (Closed) Licensee Event Report 05000311/2016-002-01: Automatic Reactor Trip due to Main Turbine Trip
a. Inspection Scope
This LER (05000311/2016-002-00) was originally submitted by PSEG in April 2016, following a February 4, 2016, Unit 2 reactor trip. The LER presented the results of PSEGs original RCE for this event, which concluded that the reactor trip was caused by a main generator protective relay (model STV1) that actuated unexpectedly due to setpoint drift. The LER was reviewed and closed by the NRC under inspection report 05000311/2016002 (ML17101A407), which documented a Green, self-revealing FIN for PSEG work orders not specifying the appropriate modification testing for the protective relay.
In preparation for the 95001 inspection, PSEG revised the RCE for this event. The revised RCE considered the inadequate work order to be a contributing cause, but pointed to a new root cause: that PSEG did not develop and implement adequate corrective actions after Unit 1 experienced a similar trip in December 2012 due to STV1 relay setpoint drift. The inspectors reviewed the revised LER (05000311/2016-002-01),the revised RCE and corrective actions, and interviewed PSEG staff. No additional findings or violations of NRC requirements were identified. This LER is closed.
b. Findings
No findings were identified (Closed) Licensee Event Report 05000311/2016-005-00: Automatic Reactor Trip Due to Main Generator Trip
a. Inspection Scope
This LER presented the results of PSEGs RCE for this event, which found that this June 2016 generator trip occurred due to a fault internal to the recently replaced A main transformer. The new transformer had wiring running from a current transformer to a connection point. The wiring was wrapped with an adhesive cotton tape. However, over time, the ground wire in this bundle worked loose due to an inadequate termination on one end (which was not visible at fabrication) as well as eventual loosening of the cotton tape that held the wiring bundle together. The wire was suspended in the transformers oil and moved in response the oil flow in this area. The wire eventually caused an intermittent ground fault when it touched the low voltage connections inside the transformer bushing compartment that ultimately resulted in a main generator trip.
PSEG performed extensive monitoring and troubleshooting as well as equipment cleaning (i.e., carbon dioxide pressure washing of the 25 kV bus bar support dielectric coils) and replacement of select components to identify the source of this generator/reactor trip. As the ground fault was intermittent, it was difficult to locate.
Electrical experts from outside PSEG were brought in to assist with the troubleshooting effort. The source of the ground fault was readily identifiable once the decision was made to drain and enter the A main transformer following the elimination of numerous other possible causes over a multi-week troubleshooting effort.
The inspectors reviewed the LER, the final RCE on this scram and pictures of the wiring bundle: 1) at the end of fabrication, 2) following opening of the faulted transformer, and 3) following repairs to the wiring. He also spoke with the PSEG staff member involved in the oversight of the main transformer procurement. No findings or violations of NRC requirements were identified. This LER is closed.
b. Findings
No findings were identified
4OA4 Supplemental Inspection (IP 95001)
.1 Inspection Scope
The NRC staff performed this supplemental inspection in accordance with IP 95001, Supplemental Inspection Response to Action Matrix Column 2 Inputs, to assess PSEGs evaluation of a White PI, which affected the Initiating Events cornerstone in the Reactor Safety strategic performance area. The inspection objectives were:
- To assure that the root causes and contributing causes of individual and collective significant performance issues are understood.
- To independently assess and assure that the extent of condition and extent of cause of significant individual and collective performance issues are identified.
- To assure that corrective actions taken to address and preclude repetition of significant performance issues are prompt and effective.
- To assure that corrective plans direct prompt actions to effectively address and preclude repetition of significant performance issues.
In accordance with the NRC ROP, the inspectors performed a supplemental inspection since Salem Unit 2 was in the Regulatory Response Column of the Action Matrix because of one White PI for Unplanned Reactor Scrams per 7,000 Critical hours.
Specifically, Salem Unit 2 crossed the Green-White threshold in the second quarter of 2016 following one reactor trip in August 2015, two reactor trips in February 2016, and a fourth reactor trip in June 2016. The PI remained White when a fifth reactor trip was incurred in August 2016. The five
Rx Trip #1: On August 5, 2015, Unit 2 experienced an automatic reactor trip from full power due to a loss of the 21 RCP. The 21 RCP, which is powered from a breaker on the 2H bus, tripped when power to the 2H 4 kV non-vital bus was lost following opening of the infeed breaker from the 2B auxiliary power transformer due to a fault on 21 HDP motor.
Rx Trip #2: On February 4, 2016, Unit 2 automatically tripped from 75 percent power on a generator trip initiated by a main generator automatic voltage regulator (AVR)volts/hertz over-excitation protection (STV1) relay trip. Power had been reduced to 75 percent to support removal of a 500 kV line offsite. The STV1 relay was found to have a manufacturing defect which caused the relay setpoint to drift low and actuate when the volts/hertz ratio increased during the power reduction.
Rx Trip #3: On February 14, 2016, Unit 2 experienced a main generator protection trip signal which led to a reactor trip from 100 percent power. The trip was later found to be caused by water dripping into the stator water cooling trip Agastat relay (62-C1), causing an electrical short.
Rx Trip #4: On June 28, 2016, Unit 2 experienced a main generator and subsequent reactor trip. The cause was ultimately traced to a loose current transformer wire on the A main power transformer, which caused an intermittent ground.
Rx Trip #5: On August 31, 2016, Unit 2 automatically tripped due to the loss of the 21 RCP. The breaker for the 21 RCP opened due to a fault within the inner enclosure assembly of its electrical penetration. The fault was caused by brackish service water falling on the enclosure, which occurred after a water hammer on the 22 CFCU blew out a section of rubber gasket on its motor cooler. The water hammer occurred as flow was being reestablished to the cooler following SW accumulator discharge valve testing.
In each of the reactor scrams/trips, safety systems performed as expected and the operators responded to the events properly. Thus none of the five events above involved operator performance issues.
In response to each of these five reactor scrams/trips, PSEG created five RCEs to evaluate the root and contributing causes for these events as well as proposed corrective actions. PSEG also performed an RCE on the overall adverse trend in reactor scrams/trips between August 2015 and August 2016.
The inspectors reviewed all six of PSEGs RCEs, reviewed applicable CAP documents, and interviewed licensing, engineering, maintenance, and operations management personnel to ensure that the root and contributing causes were understood and corrective actions taken or in progress were appropriate. The inspectors also conducted an in-plant walkdown of the affected components in the Unit 2 turbine building, noting modifications that had been completed since these reactor trips occurred.
.2 Evaluation of the 95001 Supplemental Inspection Requirements
.2.1 Problem Identification
a. IP 95001 requires that the inspection staff determine that PSEGs evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC identified) and the conditions under which the issue was identified.
In each of the five reactor trips that caused the PI to turn White, PSEG documented who identified the cause of the trip and the conditions under which the issue was identified.
In each case, the cause of the trip was either self-revealing or identified by PSEG after troubleshooting. The exception was the first reactor trip, which involved an electrical failure of the 21 HDP and subsequent loss of the 2H 4kV bus. The reason why the protective relay (which should have isolated the 21 HDP by opening its breaker before the 2H bus was lost) did not operate properly could not be definitely identified, although the potential causes were narrowed to aging (the relay was original equipment) or foreign material on a key internal component. As a precaution, the relay was replaced.
The inspectors determined this inspection objective was met.
b. IP 95001 requires that the inspection staff determine that PSEGs evaluation of each issue documents how long the issue existed and prior opportunities for identification.
In each of the five reactor trips, PSEG documented how long the issue which caused the reactor trip existed and prior opportunities for identification. PSEG ultimately determined the following:
- In the first reactor trip, the testing data over several years indicated that the 21 HDP motor was acceptable, but showed a clear trend that the motor insulation was degrading;
- In the second reactor trip, the generator STV1 relay had experienced performance problems since 2005, with multiple replacements over the years. This same relay had caused a reactor trip on Unit 1 in 2012. During the most recent Unit 2 STV1 relay replacement, a few months before the 2016 trip, problems were experienced during acceptance testing that could have led to the identification of the degraded relay;
- In the third reactor trip, no definitive time was found for when the water leak on a relay developed, although the design of the panel that housed the co-located stator water line and the relay dated to the replacement of the Unit 2 main generator in 1992;
- In the fourth reactor trip, the current transformer wire came loose at some point following its entry into service in November 2015 and the trip in June 2016, without prior, definitive warning; and,
- The fifth reactor trip was caused by a CFCU motor cooler gasket leak that led to the wetting of the RCP penetration immediately prior to the trip. However, the system was susceptible to service water (SW)-induced water hammers since a 1998 modification, and apparent cause evaluations (ACEs) had been performed on four separate occasions for significant leaks since that time.
The inspectors determined this inspection objective was met.
c. IP 95001 requires that the inspection staff determine that PSEGs evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issue(s).
In each of the five trips, the operators responded as expected and the trips were uncomplicated. The trips all involved equipment that was not safety-related, with the exception of the safety-related SW leak from the CFCU which sprayed down on a non-safety related reactor coolant pump electrical penetration and caused a short to ground.
Specific regulatory compliance concerns, if any, were addressed by the resident inspectors in their quarterly inspection reports.
The increased frequency of reactor trips during this 12 month period (August 2015 -
August 2016) does impact the Probabilistic Risk Assessment (PRA) model for Salem, but not to a significant degree as uncomplicated trips are a small percentage
(~10 percent) of the PRA initiating events that result in increasing the core damage frequency.
The inspectors determined this inspection objective was met.
d. Findings
No findings were identified.
.2.2 Root Cause, Extent of Condition, and Extent of Cause Evaluation
a. IP 95001 requires that the inspection staff determine that PSEG evaluated the issue using a systematic methodology to identify the root and contributing causes PSEG used a systematic methodology to review each event, identifying the root and contributing causes of each of the five reactor trips as well as the overall adverse scram trend. The following root cause evaluation (RCE) methodologies were utilized for the PSEG evaluation:
- The WHY Staircase
- Barrier Analysis
- Failure Modes and Effects Analysis
- Event and Casual Factor (E&CF) Chart
- Safety Culture Analysis
- Organizational and Programmatic (O&P) Review
- Extent of Condition (EOCo) and Extent of Cause (EOCa) Review:
o Same - Same o Same - Similar o Similar - Same o Similar - Similar
- Operating Experience (OE) Assessment
- Review of Previous Internal Occurrences The inspectors determined this inspection objective was met.
b. IP 95001 requires that the inspection staff determine that PSEGs RCEs were conducted to a level of detail commensurate with the significance of the issue.
The six RCEs performed by PSEG were conducted to a level of detail commensurate with the significance of the issue. Each RCE used a variety of techniques to identify the root and contributing causes, and had documentation as necessary to support the conclusions drawn. The amount of detail varied by the complexity of the issue, with the RCEs on the CFCU leak that ultimately resulted in Rx trip #5 and the adverse scram trend RCEs being the most lengthy and involved.
The inspectors determined this inspection objective was met.
c. IP 95001 requires that the inspection staff determine that PSEGs RCE included a consideration of prior occurrences of the issue and knowledge of Operating Experience.
The RCEs on each of the five reactor trips included a review of industry operating experience and prior occurrences of the issue. Prior occurrences varied by event, with the CFCU leak that ultimately resulted in a reactor trip (Rx trip #5) having the most, followed by the inadvertent actuation of the STV1 relay (Rx trip #2) being the second.
However some events had little or no precedent, such as the stator cooling water dripping on a relay that caused Rx trip #3. Industry experience was also considered in each case, but provided limited information due to either the uniqueness of the event and/or the uniqueness of the system design (e.g., Salem SW system, which is the only plant in the country equipped with accumulators).
The inspectors determined this inspection objective was met.
d. IP 95001 requires that the inspection staff determine that PSEGs RCE addresses the extent of condition and extent of cause of the issue.
PSEGs six RCEs documented considerable work to address the extent of condition and the extent of cause. For example, inspections were conducted on a range of similar equipment in the plant, such as the remaining two main transformers and a spare at Unit 2, the six HDPs at both units (including a spare in storage), and multiple electrical penetrations that were wetted by SW recently during past events, causing corrosion). In the case of the electrical penetrations, cleaning and repairs were performed on those penetrations boxes found to have significant corrosion. While additional inspections are planned (i.e., internal inspection of the Unit 1 main power transformers (MPTs) in the fall 2017 outage), most were completed at the time of this inspection and discrepancies were documented and addressed. For the adverse scram trend, the extent of cause was focused on reviewing an expanded sample of RCEs and ACEs since 2013 to ensure the evaluations and resulting actions met procedural requirements. For instance, a large number of conditions requiring corrective action (CRCAs) were reviewed for proper closure and implementation. In addition, corrective actions to preclude recurrence (CAPRs) which had a change of intent since their creation received additional management scrutiny.
The inspectors determined this inspection objective was met.
e. IP 95001 requires that the inspection staff determine that PSEGs root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0305.
An analysis was performed which looked at each of the 10 safety culture traits described in IMC 0305. While behaviors were found to have been met in 3 of the 10 traits, weaknesses were identified in the remaining seven traits, with the most significant weaknesses found to be in the areas of: 1) Problem Identification; 2) Decision-Making; 3) Leadership Safety Values and Actions; and, 4) Questioning Attitude. PSEG concluded that the various corrective actions proposed for each of the five individual reactor trip RCEs will address most of the safety culture attributes, although an additional performance analysis was performed and corrective actions will be taken to address issues with leadership decision-making.
The inspectors determined this inspection objective was met.
f. Findings
No findings were identified.
.3 Corrective Actions
a. IP 95001 requires that the inspection staff determine that:
- (1) PSEG specified appropriate corrective actions for each root and/or contributing cause, or
- (2) an evaluation that states no actions are necessary is adequate.
PSEG specified appropriate corrective actions for each root and/or contributing cause; none of the evaluations stated that no action was necessary. Key corrective actions included:
- Placing the HDPs in the LTAM program
- Replacing certain high consequence protective feeder relays and/or converting them to digital (42 in total)
- Implementing plans for the replacement of multiple 25 kV monitoring system components that are obsolete or out of service
- Implementing causal evaluation training for the Engineering staff
- Placing a 90-day limit for Long Term Asset (LTA) manager items to be in draft status
- Installing a 1-inch bypass line on the SW supply line to the CFCUs to prevent future water hammers when returning the system to service
- Implementing management tools to focus on monitoring and improving ER, including an ER Heat Map
- Providing Line-Of-Sight review for CAP evaluations
- Performing Corrective Action Closure Boards on a monthly basis.
The inspectors determined this inspection objective was met.
a. IP 95001 requires that the inspection staff determine that PSEG prioritized corrective actions with consideration of risk significance and regulatory compliance.
Each of these reactor trips involved balance of plant or non-safety related equipment (the exception being the CFCU SW leak), so the risk significance was low and regulatory compliance was not an issue. However, PSEG completed (or will soon complete) a number of the corrective actions involving equipment changes. In the case of the CFCU SW leak, which was the only one involving safety-related equipment that is inside containment (which limits both access and available outage work windows), interim corrective actions have been implemented to prevent another water hammer until a small bypass line is installed to allow slow refilling of the CFCUs.
The risk significance and regulatory compliance issues for the overall root causes were addressed in the five individual root causes. An overall risk significance for the multiple reactor trips was provided in the sixth adverse trend RCE.
The prioritization of corrective actions was generally subjective, with the exception of actions to address SPVs. The SPVs are individual components which, if they fail or actuate, lack redundancy (e.g., 2 out of 3 in a logic circuit) and thus trip a component that will subsequently lead to a reactor trip. PSEG procedure ER-AA-2004, System Vulnerability Review Process Description, Attachment 1 contained a SPV Prioritization Matrix for risk ranking vulnerabilities.
PSEG used this structured prioritization matrix to rank SPVs and help determine which ones should be eliminated (versus mitigated), as well as which SPVs should be addressed first.
The inspectors determined this inspection objective was met.
b. IP 95001 requires that the inspection staff determine that PSEG established a schedule for implementing and completing the corrective actions.
PSEG established a comprehensive schedule for implementing and completing the roughly 150 corrective actions (CAPRs, CRCAs) and action items (ACITs) laid out between the six RCEs. Almost all of the CAPRs, CRCAs and ACITs involving non-equipment issues have been completed or are scheduled for completion before the end of 2017.
The nature of the equipment modifications planned in response to these events (or other related plant or industry initiatives) are, by their nature, long-term since they can only be performed during outages and on only one train at a time. For example, PSEG plans to perform actions to mitigate the trip risk posed by the 700+ SPVs identified by their reviews, a time consuming, long-term action. The following other equipment modifications are planned for the following number of plant cycles (on both units):
- SW to CFCU 1 inch fill bypass lines - 2 cycles
- CFCU gasket design changes - 5 cycles
- Relay conversions to digital - 2-3 cycles
- Eliminating 61 SPVs - 2-3 cycles
- 25 kV monitoring equipment replacements - 2 cycles Although, the progress of the implementation of the above modifications, as well as the completion of key procedural and process CAPRs and CRCAs and their subsequent effectiveness reviews are expected to be inspected during the conduct of future inspections, the inspectors determined this inspection objective was met.
c. IP 95001 requires that the inspection staff determine that PSEG developed quantitative and/or qualitative measures of success for determining the effectiveness of the CAPRs.
Multiple effectiveness reviews (EFFRs) were planned to confirm the successful implementation of the corrective actions to address each RCE. The EFFRs used a balance of qualitative versus quantitative measures based on the nature of the corrective action (e.g., a definitive hardware change versus training or a programmatic or process change). Although many of the EFFRs cannot be completed until the completion of planned plant modifications, interim EFFRs were done where possible. The EFFRs that were not completed at the time of this inspection are expected to be inspected during the conduct of future inspections; therefore, the inspectors determined this inspection objective was met.
d. IP 95001 requires that the inspection staff determine that PSEGs planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable.
This supplemental inspection was based on a White PI versus an inspection finding.
However, three Green NCVs were issued by the resident inspectors in 2016 for performance issues related to the causes of three of the five reactor trips.
e. Findings
No findings were identified.
.4 Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues
PSEG did not request credit for self-identification of an old design issue involving any of these five reactor trips.
.5 Review of PSEG Corrective Actions taken in Response to the 95001 Inspection
Conducted at Salem Unit 1 in August 2015 The team reviewed the corrective actions that addressed the White PI for Unplanned Scrams per 7000 Hours Critical at Salem Unit 1 since the conclusion of the 95001 inspection (05000272/2015-009; ML15258A467) in August 2015. The station identified that the corrective actions implemented to address that White PI had not effectively corrected the significant conditions adverse to quality because the station received a repeat White PI for Unplanned Scrams on Unit 2 one year later. PSEGs review noted that the four reactor trips at Unit 1 in 2014-2015 were for different reasons, and the focus of the prior efforts was on the recovery of margin and how to prevent crossing the threshold for this White PI, versus improving ER to prevent future scrams/trips. The involvement of PSEG staff in the prior RCE efforts were also heavily weighted with Regulatory Affairs staff, versus a representative cross-section of the station Operations, Maintenance, and Engineering staff who are focused on equipment issues.
In the 2015 RCEs for the Unit 1 scrams/trips, PSEG identified two root causes:
1) behaviors of site leadership did not effectively enforce existing processes to ensure specific and adequate actions were developed, challenged, and implemented to prevent operational challenges leading to reactor scrams; and, 2) ineffective knowledge and use of existing processes to develop plans and actions that could have mitigated the risk of the reduced scram margin. PSEG determined that governance and oversight was ineffective in identifying weaknesses in the application of fundamentals to prevent reactor scrams was also a contributing cause. Thus the corrective actions for the previous White PI were focused on enhancing procedural compliance, especially for new employees, as well as enhancing leadership and accountability. However, PSEG found that some of the corrective actions implemented were either too broad or were not effective in the long-term. Moreover, some corrective actions were changed over time such that they did not meet the intent of the action originally planned. These changes were made without adequate review and approval by the PSEG senior management team.
By contrast, the RCEs prepared for the Unit 2 White PI involved a cross-section of station staff to ensure the inclusion of viewpoints and buy-in of the involved departments.
The RCEs did touch on the issues noted at Unit 1 in 2014-2015, but focused predominately on enhancing the corrective action process and improving ER in order to prevent reactor scrams/trips. The RCEs performed in support of this supplemental inspection also looked back to ensure that the causes noted were aligned with the reactor scram/trip issues noted at both Salem units since 2010.
In response to the observation that some corrective actions for the prior White finding experienced significant changes of intent over time, PSEG elected to impose management controls on changes of intent to ensure that such changes receive the proper level of management review and scrutiny. In addition, in contrast to the corrective actions taken for the prior White PI, PSEG is making efforts to improve the reliability of the equipment involved in each of the five reactor trips that contributed to this PI. Thus the team had sufficient basis to conclude that the corrective actions taken in response to this White PI will be more effective than the scram prevention efforts previously undertaken by PSEG. However, since the effectiveness reviews for many of these corrective actions will not be completed for some time, the full effectiveness of these measures are expected to be inspected during the conduct of future inspections.
4OA6 Exit Meetings
On September 15, the inspectors presented the inspection results to Mr. C. McFeaters, Salem Site Vice President, and other members of the PSEG staff. The inspectors verified that no proprietary information was retained by the inspectors or documented in this report. PSEG management acknowledged and did not dispute the findings. The results of the inspection were visually summarized using slides that can be found in ADAMS (ML17291B082).
SUPPLEMENTARY INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- P. Sena, Chief Nuclear Officer
- C. McFeaters, Site Vice President
- P. Martino, Plant Manager
- P. Davison, Engineering Vice President
- R. DeNight, Engineer Manager
- K. Anderson, Salem Operations, RCE team leader
- J. Wearne, Regulatory Assurance Manager
- S. Nevelos, Manager - Salem Organizational Effectiveness
- M. Brummitt, I & C Supervisor
- F. Possessky, Salem Performance Improvement
- S. Markos, Design Engineering Manager
- K. Hernandez, Salem Engineering
- J. Fleming, Regulatory Assurance Director
- R. Flessner, Contractor, Landis Consulting
- K. Landis, Contractor, Landis Consulting
- P. Essner, Engineer
- J. Stead, Engineer
- A. Zheng, Engineer
Other
- E. Rosenfeld, State of New Jersey Bureau of Nuclear Engineering
LIST OF ITEMS
OPENED, CLOSED AND DISCUSSED
Closed
- 05000311/2015-002-01 LER Reactor Trip Due to Loss of 4kV Non-Vital Group Bus (Section 4OA3)
- 05000311/2016-002-01 LER Automatic Reactor Trip due to Main Turbine Trip (Section 4OA3)
- 05000311/2016-005-00 LER Automatic Reactor Trip Due to Main Generator Trip (Section 4OA3)