IR 05000272/2002008

IR 05000272-02-008, on 9/30-10/04/02, Salem Units 1&2; Supplemental Inspection. Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area

ML023090058
Person / Time
Site:	Salem
Issue date:	11/04/2002
From:	Linville J NRC Region 1
To:	Keiser H Public Service Enterprise Group
References
IR-02-008
Download: ML023090058 (14)

Text

ber 4, 2002

SUBJECT:

SALEM UNIT 1 POWER PLANT - SUPPLEMENTAL INSPECTION FOR WHITE PERFORMANCE INDICATORS REPORT 50-272/02-008

Dear Mr. Keiser:

On October 11, 2002, the NRC completed a supplemental inspection at the Salem Unit 1 Generating Station. The enclosed report documents the results of the inspection, which were discussed with Mr. David Garchow, Mr. John Carlin, and other members of your staff on October 11, 2002.

The NRC performed this supplemental inspection to assess PSEG Nuclears evaluation of the white performance indicator (PI) associated with the Salem Unit 1 Unplanned Power Changes per 7000 Critical Hours. During this supplemental inspection, performed in accordance with Inspection Procedure 95001 - Inspection for One or Two White Inputs in a Strategic Performance Area, the inspector determined that PSEG Nuclear completed evaluations which determined the primary causes of each of the seven transient and a common cause evaluation which found several underlying issues which influenced all the events. However, the inspector did find some weaknesses in the corrective action program related to the identification of underlying issues which contributed to the transients.

PSEG Nuclears common cause evaluation of the PI was sufficient to identify many of the underlying issues that contributed to events that caused the PI to cross the white threshold.

PSEG Nuclears evaluation found inadequacies in addressing previously identified issues that caused the transients, the actions taken to correct those issues are complete, and you intented to implement several programs to address the underlying causes of the events associated with the white performance indicator. Therefore, the white PI was only considered in assessing plant performance until the transient trend crossed the green level threshold, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program. Implementation of the corrective actions and reassessment of weaknesses observed during this inspection will be reviewed during the next Problem Identification and Resolution Baseline Inspection per Inspection Procedure 71152 scheduled for March 2003.

Mr. Harold In accordance with 10 CFR 2.790 of the NRCs Rules of Practice, a copy of this letter and its enclosure will be available electronically for public inspection at the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

James C. Linville, Chief Electrical Branch Division of Reactor Safety Docket No. 50-272 License No. DPR-70 Enclosure: Inspection Report 50-272/02-008 cc w/encl:

M. Friedlander, Director - Business Support J. Carlin, Vice President - Nuclear Reliability and Technical Support D. Garchow, Vice President - Operations G. Salamon, Manager - Licensing R. Kankus, Joint Owner Affairs J. J. Keenan, Esquire Consumer Advocate, Office of Consumer Advocate F. Pompper, Chief of Police and Emergency Management Coordinator M. Wetterhahn, Esquire State of New Jersey State of Delaware N. Cohen, Coordinator - Unplug Salem Campaign E. Gbur, Coordinator - Jersey Shore Nuclear Watch E. Zobian, Coordinator - Jersey Shore Anti Nuclear Alliance

Mr. Harold

SUMMARY OF FINDINGS

IR 05000272/02-008, on 9/30-10/11/02, Salem Units 1&2; Supplemental Inspection.

Inspection Procedure 95001, Inspection for One or Two White inputs in a Strategic Performance Area.

The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Initiating Events

The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess PSEG Nuclears evaluation associated with the white performance indicator at Salem Unit 1. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspector determined that PSEG Nuclear performed comprehensive evaluations to determine the causes for each of the down-power transients. PSEG Nuclear completed an evaluation of all the transients related to the white performance indicator (PI)and identified several influencing factors which contributed to the events. They included timeliness of corrective actions, preventative maintenance program issues, human performance issues and other programmatic problems that lead to a reactive vs. pro-active response. In addition, the inspector found several examples in which the corrective action program did not determine all the germane factors related to the transients.

Given PSEG Nuclears acceptable performance related to the final resolution of each of the issues and implementing several programs to address the underlying causes of the events which caused the seven transients associated with the white performance indicator, the white PI will only be considered in assessing plant performance until the transient trend crosses the green level threshold, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program. Implementation of the corrective actions will be reviewed during the next Problem Identification and Resolution Baseline Inspection per Inspection Procedure 71152.

Report Details 01

INSPECTION SCOPE

The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess PSEG Nuclears problem identification, root cause and extent of condition review, and related corrective actions associated with the seven transients which caused the Unplanned Power Transients performance indicator (PI) to cross the white threshold. This white PI is related to the initiating event cornerstone in the reactor safety strategic performance area.

The inspector reviewed the root or apparent cause reports performed by PSE&G Nuclear for each of the transients that caused the PI to cross the white threshold. These down power transient events were:

2. Moisture Separator Stop Valve Failure

Due to Actuation Rod Failure (12RS11) 27 May 01

4. Moisture Separator Stop Valve Failure

Due to Actuation Rod Failure (13RS12) 8 Sept 01

7. Loss of Circulation Water During Maintenance Due to Failure

of Bearing Cooling Line 26 Feb 02 The inspector reviewed the notifications, root or apparent cause evaluations and status of corrective actions for each event. In addition, the licensees common cause evaluation, which accessed if any common cause or contributing causes for down-power transient events at Salem Unit 1, was reviewed. This evaluation also included an assessment of down-power transient events that had occurred at Salem Unit 2 and Hope Creek Generating Stations. The inspector also interviewed licensee personnel, reviewed procedures, and walked down plant systems.

02 EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.

Determination of who identified the issue and under what conditions The inspector found that with the exception of the steam generator low-low level set point adjustment

(6) all of the events were self-revealing. The inspector found that in all of the self-revealing cases the licensee was either aware of the equipment degradation or there was information available which could have been used to evaluate that a degraded condition existed. The licensee determined that all of the transients occurred during normal power operations with events 1, 2, 4 and 7 being initiated as a result of maintenance activities causing the equipment to fail.

b.

Determination of how long the issue existed, and prior opportunities for identification The performance indicator assesses events that had occurred over the previous twelve-month period. The licensee determined that many of these issues were long standing equipment problems that had been previously identified and/or information was available to identify the degradation. Their evaluation found that corrective actions had not been effective or had not been implemented in a timely fashion. Specifically:

Event 1 - The licensee review found that the first time this event occurred was on August 20, 1999. The root cause analysis determined that the modification package which installed a digital controller for feed pump governor failed to determine that the power supply and controller were not compatible. The licensee identified that immediately before the power supply transfer between power source it created a momentary high voltage condition and cause the controller to shutdown on over-voltage.

A total of three controller shutdowns and the resulting plant transient occurred, over a 11/2 year period, before the modification was installed to correct the problem.

Events 2 and 4 - The licensee determined that the failure of the valve was the result of an unauthorized modification to the valve actuator rod performed during bi-annual maintenance on the valves and actuator. When event 2 occurred the licensee extent of condition evaluation found five additional valves that were susceptible to the failure mechanism because of the modification. One of the valves that was found to have the degradation subsequently failed during event 4. However, the inspector determined that this modification issue had been brought to the attention PSEG engineers during the previous outage. The PSE&G turbine maintenance team discovered, during reassembly of the valve and actuator after reconditioning, that the pre-drilled hole for the connector rod and the valve stem hole did not lined up. The team asked PSEG engineering for an evaluation to determine how to connect the rod to the two components. The evaluation was provided which created a special procedure for reassemble when this problem occurred. PSEG Nuclear never recorded this evaluation, a notification was not written and the procedure was not changed to address the problem. Subsequent modifications to the actuator rods were not drilled in accordance with the engineering evaluation. This caused the rods to weaken and fail when the valves were reopened after actuation.

This prior opportunity to identify the degraded condition was not identified or evaluated in either of the licensees apparent cause evaluations or the common cause evaluation.

Events 3 and 5 - The licensees evaluation determined that the gap type lightning arresters, which had been installed for 25 years on the stations high-voltage-power transformers, were known in the industry to begin to fail after 20 years of service.

Additionally, PSEG Transmission and Distribution (T&D) was also aware of the lightning arrester problems and stated that they had advised PSEG Nuclear to incorporated these components into the preventive maintenance program when the ownership of the equipment was transferred from T&D to Nuclear in the mid 90s. The evaluation found no documented information from T&D and no maintenance activities to replace the components were ever created. Additionally, the inspector noted that an opportunity to identify design and procedure problems which caused the loss of the B Circulating Water (CW) bus to be more severe (responsible for the initiation a manual scram during event 5) was not evaluated, as part of event 3, when the A CW bus was lost. Although the A and B CW busses split the power supplies to the six CW pump motors the second event revealed that the differential pressure transmitters, transformer tap changers and intake screen motors for all the CW trains were powered from the B bus.

Event 7 - The licensee determined that there had been many opportunities to address the problems related to the intake systems. The licensee has significant challenges with the system due to grass growth in the bay. The specific problem related to installation of the bearing cooler line had been discussed with the non-nuclear PSEG repair facility, but the information had not been incorporated into the installation procedure.

c.

Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue The licensee found that there were no specific plant-specific risk consequences as a result of the plant transients because none of the station safety systems were effected.

The performance issues that resulted in the white PI involved non-safety-related balance of plant equipment. The inspector determined that the components that had failed are monitored only as potential scram initiators for risk based analysis. Although two of the transients resulted in a reactor scram, the plants probabilistic risk assessment (PRA)accounts for this frequency, therefore, no additional risk was incurred. Any changes to the reliability input into the stations PRA for the individual components as a result of the failures should be documented as part of 10CFR 50.65-A3 evaluations during the next refueling outage. Additionally, there were no violations associated with the down-power transients because the equipment is not subject to technical specifications requirements.

02.02 Root Cause and Extent of Condition Evaluation a.

Evaluation of method(s) used to identify root cause(s) and contributing cause(s).

The inspector determined that the methods PSEG Nuclear used for their level 1 root cause analysis provided adequate evaluation techniques to determine the causes for the plant transients. Events 5, 7 and the original 1999 evaluation of the precursor to event 1 were evaluated using level 1 root cause techniques. The licensee used several methods including Hazard-Target-Barrier Analysis, Human Errors/Inappropriate Actions Failure, and Event & Causal Factor Charting in the evaluation process. The root cause evaluation of the white PI which used a common cause analysis technique was also found to be an adequate evaluation technique. The five remaining events were examined using a level II Apparent Cause evaluation. The apparent cause evaluation is used to: understand and correct the problem, understand if a significant condition exists, identify apparent cause for trending purposes, and allow consideration of corrective action to prevent recurrence. It does not require a specific root cause technique and in the case of event 3 the evaluation stated, No root cause techniques were used since the events involved equipment failures that were clearly identified.

Additionally, the inspector found that recurring events (i.e., same component failure) did not have separate evaluations performed. The apparent cause evaluations for events 1 and 4 did not fully analyze the specific causes associated with the event. Generally, the previously performed root or apparent cause was reused with some minor modifications.

b. Level of detail of the root cause evaluation The inspector found that the common cause evaluation of the plant transients went into sufficient detail to identify most apparent and contributing causes related to the performance issue. However, the inspector noted that the evaluation stated that it did not identify problems with the corrective action program. In reviewing the individual apparent cause and root cause evaluations, the inspector found inadequacies in several the evaluations. They included:

1 - The inspector found that the apparent cause did not evaluate why it had taken a 11/2 years and two additional transients to complete a minor modification to the power supply circuit for the digital controller. The majority of the evaluation was a rewrite of previously performed evaluations.

2 - The inspector found that the apparent cause investigation did not determine how maintenance workers determined that drilling extra holes into steam isolation valve actuator rods was an acceptable action. The inspector determined that an evaluation performed by a PSEG engineer provided an acceptable method to re-drill the hole in the actuator rod, but because the issue was not documented subsequent modifications to the rods were not performed in accordance with the evaluation.

3 - The inspector found that the apparent cause evaluation missed an opportunity to evaluate design and procedure deficiencies related to the effects of a loss of the A CW bus during their evaluation of the two events which resulted the loss of the B CW bus.

The Root Cause Manual (NC.CA-TM.ZZ-0003(Z)) provides that an apparent cause evaluation should confirm that a more significant condition is not present. The failure to find the procedure and design problems caused the event 5 transient to be more severe.

4 - The inspector found that the second valve failure apparent cause was a rewrite of the first apparent cause. The failure mechanism and extent of condition was well understood after the evaluation of the first failure but, this apparent cause did not evaluate any of the specific circumstances related to the failure. It also did not address the how maintenance personnel determined that it was acceptable to drill extra holes in the rod.

c.

Consideration of prior occurrences of the problem and knowledge of prior operating experience The inspector determined that the three level one root cause evaluations conducted by the licensee evaluated prior occurrences and operating experience. All the evaluations found prior opportunities to identify and correct the equipment problems. The common cause evaluation related to the white PI found that the inability to fix known issues that were clearly documented in our corrective action program was an underlying deficiency and also found that the apparent cause for the events, the need for prompt corrective maintenance caused by equipment failures, was not performed. Additionally, the evaluation found that many of the same underlying issues that were found during the licensee common cause evaluation Common Cause

Analysis:

Salem Unit 1 Reactor Trips - September 22, 2000" which included; human performance issues, equipment failure issues , procedure issues and preventative maintenance program issues; were also contributing causes to these events. The inspector agreed with this assessment.

d.

Consideration of potential common cause(s) and extent of condition of the problem The inspector found that the PSEG common cause evaluation related to the PI did consider the potential common cause and extent of condition associated with underlying contributing causes for the seven events. However, the inspector found that underlying issues related to the corrective action program were not addressed. PSEG did not evaluate why several of the individual evaluations did not address contributing causes (3,7), extent of condition(3), investigation of the specific event(1,4) or evaluate the actual cause (1,2,4) of the events. The inspector also noted that several of the events (2,3,4,5,7) involved ineffective communications between PSEG Nuclear and PSEG non-nuclear workforces and this was not evaluated.

02.03 Corrective Actions a.

Appropriateness of corrective action(s)

The inspector found that actions taken to correct the equipment and/or procedure issues related to the events that were the direct causes for the down-power transients have been addressed sufficiently to prevent reoccurrence. The actions taken included:

1 - Voltage filters were installed, on the power supply to the feed pump controller, shortly after the third down-power transient event that prevent the high-voltage condition which had caused the controller shutdown.

2,4 - All rods, on the moisture separator stop valve, that were incorrectly drilled were identified and replaced. Procedure changes now prevent reuse of rods if the pre-drilled holes do not lineup during reassembly.

3,5 - All gap-type lightning arresters, that were installed on high-voltage transformers, have been identified. Arresters whose failure could cause a down-power transient have been replaced. The remaining gap-type arresters still in operation have been scheduled for replacement.

6 - Adjustments to the steam generator level setpoint were completed.

7 - The procedure to install the bearing cooling line on the CW pump motors was revised. Additionally, a comprehensive review of maintenance practices related to the CW and service water systems is being conducted.

The inspector determined that there were only two new corrective actions designed to address deficiencies found during the root cause evaluation for the white PI. The evaluations cites several programs at the site, that had been previously implemented, that would address the underlying causes of the transients. They include the Nuclear Reliability Program, and Equipment Reliability Plans. Additionally, the licensee discussed several other new programs that would address these issues. They included a newly formed Human Performance Branch, Preventive Maintenance Optimization Program, Precision Maintenance Group, Quality Improvement Action Plan and Manager Top 20 List.

b. Prioritization of corrective actions The PSEG common cause evaluation found the failure to properly prioritize and complete corrective actions in a timely manner was the main underlying contributor to the down power transients. The inspector agreed with this assessment and noted that actions subsequent to the down-power transients were properly prioritized. The CW system recovery plan has been designated as the 1 item in the Station Manager Top 20 List and corrective actions for the other individual events have been completed.

Additionally, the site has created organizations to address the programmatic issues on a site wide bases.

c. Establishment of a schedule for implementing and completing the corrective actions PSEG had previously established several programs that were created to address many of the underlying issues found in the common cause evaluation. Interviews conducted by the inspector with personnel from several of the programs found them to be in the early stages of development, and they did not have actions associated with the common cause evaluations.

d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The inspector determined that PSEG revised the requirements for the stations System Health Reports to require that all long standing issues be discussed in quarterly reports as part of the corrective actions for the common cause evaluation. Additionally, PSEG monitors all of these components via the Maintenance Rule.

03 MANAGEMENT MEETINGS

.1 Exit Meeting Summary

The results of this inspection were discussed at a regulatory performance and exit meeting conducted on October 11th, with Mr. David Garchow, Mr. John Carlin and other members of the PSE&G Nuclear staff. At that time, the inspector asked whether any of the information was considered to be proprietary. None of the information was identified as proprietary.

ATTACHMENTS PARTIAL LIST OF PERSONS CONTACTED PSE&G Nuclear J. Carlin VP - Nuclear Reliability and Technical support D. Garchow VP - Operations T. OConnor VP - Maintenance L. Waldinger Director - Site Operations M. Mosier Regulatory Affairs L. Curran Reliability Engineering K. Botto Station Technical Advisor G. Delp Precision Maintenance Program T. Sacora Reliability Engineer G. Figueron Maintenance Supervisor C. Kapes Reliabilty Engineer M. Welker Reliability Engineer Service Water and Circulation Water S. Mannon Manager, Project Service Water and Circulation Water J. OConnor Design Engineering Nuclear Regulatory Commission R. Lorson, Sr. Resident Inspector F. Bower, Resident Inspector J. Linville, Chief, Electrical Branch LIST OF ACRONYMS PI Performance Indicator NRC Nuclear Regulator Commission PSEG Public Service Electric and Gas CW Circulating Water T&D Transmission and Distribution PRA Probability Risk Assessment

DOCUMENTS REVIEWED

Procedures

S1.OP-SO.CN-0002(Q), Rev 13 - Steam Generator Feed Pump Operation

SC.MD-PM.MS-0003(Q), Rev 1- Main Turbine Stop Valve Disassembly, Inspection, Repair and

Reassembly

S1.OP-AB.LOOP-0003(Q) - Partial Loss of Offsite Power

S1.OP-PT.TRB-0001(Q), Rev 12-Turbine Auto Trip Mechanism Operational Test

S1.OP-SO.TRB-0001(Q), Rev 16 - Turbine Generator Startup Operation

S1.OP-PT.TRB-0002(Q), Rev 1- Turbine Generator Shutdown Operation

S1.OP-AR.ZZ-0007(Q), Rev 17-Overhead Annunciators Window G

NC.DE-DG.ZZ-0100(Z), Rev 3 - Quality of Products Desk Top Guide

NC.DE-AP.ZZ-0010(Q), Rev 4 -Review/Checking and Design Verification

NC.DE-AP.ZZ-0001(Q), Rev 11-Design Bases/Input

NC.DE-AP.ZZ-0009(Q), Rev 3 -Peer Review

Deficiency/Event Reports

20037536 20061482 20067535 20069302 20070510 20076743

20078117 20084964 20091633 20092755 20092918 70000766

70009498 70015717 70017332 70017692 70019684 70019935

70023061 70023280 70023341 70023583

Miscellaneous Documents

PSEG Nuclear,LLC Updated FSAR

NC.CA-TM.ZZ-0003(Z) Rev 0 - Root Cause Manual

2003-2007 PSEG Nuclear Business Plan

System Health Reports 2nd Quarter-2002

Living PM Program Strategy Rev 72402.r3