IR 05000261/2010012

IR 05000261-10-012; on 10/12/2010 - 11/18/2010; H. B. Robinson Steam Electric Plant Unit 2 Special Inspection

ML103440401
Person / Time
Site:	Robinson
Issue date:	12/10/2010
From:	Randy Musser NRC/RGN-II/DRP/RPB4
To:	Duncan R Carolina Power & Light Co
References
IR-10-012
Download: ML103440401 (29)

Text

UNITED STATES mber 10, 2010

SUBJECT:

H. B. ROBINSON STEAM ELECTRIC PLANT - NRC SPECIAL INSPECTION REPORT 05000261/2010012

Dear Mr. Duncan:

On October 15, 2010, the U.S. Nuclear Regulatory Commission (NRC) completed the onsite portion of a special inspection at your H. B. Robinson facility. The inspection reviewed the circumstances surrounding the reactor trip on October 7, 2010, failure of the C reactor coolant pump (RCP) motor, number 2 RCP seal, and issues associated with the restoration of the normal steam generator feed system. A special inspection was warranted based on the risk and the deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program. The determination that the inspection would be conducted was made by the NRC on October 11, 2010, and the inspection started on October 12, 2010. The preliminary inspection results were discussed with Mr. Eric McCartney and members of the licensee staff on October 15, 2010. Subsequent in-office reviews were conducted and the enclosed inspection report documents the inspection results which were discussed with you on November 18, 2010.

The inspection was performed in accordance with Inspection Procedure 93812, Special Inspection, and focused on the areas discussed in the inspection charter described in the report. The inspection examined activities conducted under your license as they relate to safety, compliance with the Commissions rules and regulations, and with the conditions of your license. The team reviewed selected procedures and records, conducted field walk downs, and interviewed personnel.

The report documents one NRC-identified finding (FIN) and one self-revealing finding of very low safety significance (Green). One of these findings was determine to involve a violation of NRC requirements. However, because of the very low safety significance and because it is entered into your corrective action program (CAP), the NRC is treating the finding as non-cited violation (NCV) consistent with Section 2.3.2 of the NRC Enforcement Policy. If you contest the NCV or finding, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN.: Document Control Desk, Washington DC 20555-0001; with copies to the Regional Administrator, Region II;

CP&L 2 the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the H.B. Robinson facility. In addition, if you disagree with the cross-cutting aspect assigned to any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region II, and the NRC Senior Resident Inspector at the H.B. Robinson facility.

In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Randall A. Musser, Chief Reactor Projects Branch 4 Division of Reactor Projects Docket Nos.: 50-261 License Nos.: DPR-23

Enclosure:

Inspection Report 05000261/2010012 w/Attachment: Supplemental Information

REGION II==

Docket No.: 50-261 License No.: DPR-23 Report No.: 05000261/2010012 Licensee: Carolina Power and Light (CP&L)

Facility: H.B. Robinson Steam Electric Plant, Unit 2 Location: 3581 West Entrance Road Hartsville, SC 29550 Dates: October 12, 2010, through November 18, 2010 Inspectors: P. OBryan, Senior Resident Inspector, Brunswick (Lead)

S. Rose, Senior Project Engineer, DRP F. Ehrhardt, Senior Reactor Inspector, DRS Approved by: Randall A. Musser, Chief Reactor Projects Branch 4 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000261/2010012; 10/12/2010 - 11/18/2010; H. B. Robinson Steam Electric Plant Unit 2

Special Inspection.

This report documents special inspection activities performed onsite and at NRC offices by a senior resident inspector, a senior project engineer and a senior reactor inspector to review the circumstances surrounding the reactor trip on October 7, 2010, failure of the C reactor coolant pump (RCP) motor, number 2 RCP seal, and issues associated with the restoration of the normal steam generator (S/G) feed system. One NRC identified finding and one self-revealing violation is documented is this report. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). The cross cutting aspects were determined using IMC 0310,

Components within the Cross Cutting Areas. Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review.

Cornerstone: Mitigating Systems

Green.

A self-revealing Green NCV of Technical Specification (TS) 5.4.1, Procedures, was identified when the licensee bypassed the feedwater isolation safety function, in Mode 3, a condition prohibited by TS and an action contrary to procedural requirements.

On October 7, 2010, Unit 2 was in Mode 3 after a reactor trip that occurred earlier in the day. Steam generator feedwater was being supplied by the auxiliary feedwater (AFW)system and the main feedwater system was not running because of an automatic feedwater isolation, which occurred shortly after the reactor trip due to high C S/G water level. Contrary to procedure OP-403, Feedwater System, control room operators overrode the feedwater isolation safety function by placing the feedwater logic switches in Override/Reset, and leaving them in that position for three hours and twenty minutes. Upon realization of the error, licensee operators isolated S/G feed flow, placed the feedwater isolation logic switches in the Normal position, and restarted S/G feed flow with the AFW system. This issue was entered into the licensees CAP as AR 425643.

The failure to operate the feedwater isolation logic switches in accordance with plant procedures is a performance deficiency. The finding is more than minor because it affects the human performance attribute of the Mitigating Systems cornerstone and the objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e. core damage). Specifically, the finding affected the ability for the feedwater isolation (FWIS) to isolate the S/Gs and prevent overfeeding/overcooling events. The inspectors evaluated this finding using IMC 0609, Significance Determination Process Phase 1 screening and determined that the finding represented a complete loss of the FWIS function which required further evaluation under SDP Phase 2. The Robinson Phase 2 Pre-Solved SDP Worksheet and Phase 2 SDP Notebook did not include the FWIS function therefore a phase 3 SDP analysis was performed by a regional Senior Regional Analyst (SRA) in accordance with NRC IMC 0609. The plant was already shutdown prior to the performance deficiency so Anticipated Transients Without Scram scenarios would not be valid. The impact of the loss of the FWIS function would be that the S/Gs would not be isolated on Hi-Hi S/G water level and plant overfeed scenarios could result in Safety Injection initiation on low Reactor Coolant System (RCS) pressure and potential overfill/moisture carryover scenarios could overspeed the turbine driven AFW pump. The phase 3 analysis considered a potential overcooling and safety injection scenario using the licensees full scope Robinson Probabilistic Risk Assessment (PRA) model data and a potential moisture carryover induced overfeed scenario causing a loss of the turbine driven AFW pump on overspeed using the NRC Robinson Standardized Plant Analysis Risk (SPAR)model and data. The core damage frequency increase for both scenarios was <1E-6 per year. The risk was mitigated by the short exposure period. The finding is characterized as Green, a finding of very low safety significance. The finding has a cross-cutting aspect in the Human Performance area, Decision Making component because the licensee failed to make a safety-significant or risk-significant decision using a systematic process to ensure safety was maintained when faced with uncertain or unexpected plant conditions. Specifically, the licensee intentionally bypassed the safety function of feedwater isolation instrumentation while it was required with the reactor plant in Mode 3. (H.1(a)) (Section 4OA5.02)

Green.

The inspectors identified a Green finding for failure to correct a known equipment deficiency which adversely affected the operators ability to respond to reactor trip transients. Contrary to the licensees corrective action program, as described in procedure CAP-NGGC-0200, the licensee failed to address and correct an abnormal or unexpected equipment condition that affected and complicated plant events. The turbine building lubrication oil (lube oil) area fire protection detectors were known to actuate the turbine building lube oil deluge system after reactor trips when the 6A and 6B feedwater heater relief valves lifted. After the October 7, 2010, reactor trip, steam from the relief valves drifted to the area of the turbine building fire detectors, causing them to actuate the turbine building lube oil deluge system. This actuation caused distractions in the main control room because of several fire protection alarms sprayed fire protection water in the turbine building, and required the diversion of field operators to isolate the spuriously actuated deluge system. As an immediate corrective action after the October 7, 2010, reactor trip, the licensee directed the steam relief valve discharges from the 6A and 6B feedwater heaters and the 1A and 1B moisture separator drain tanks to an area outside the turbine building (NCR 425437).

The failure to correct a long-standing deficiency that adversely affected operator response to reactor trips is a performance deficiency. The finding is more than minor because it is associated with the mitigating systems cornerstone attribute of human performance in that the performance of operators was adversely affected by the fire protection actuation after the reactor trip. This adverse effect included diverting operator attention and resources away from initiating event response. Using the Inspection Manual 0609, Significance Determination Process Phase 1 Worksheet, the inspectors concluded that the finding is of very low safety significance (Green) because it is not a design or qualification deficiency, does not represent a loss of safety function, does not represent the loss of safety function of a single train of TS equipment, does not represent the loss of risk-significant equipment, and is not potentially risk significant due to an external events. The cause of this finding is directly related to the corrective action program component of the problem identification and resolution cross cutting area because appropriate and timely corrective actions were not taken for a known adverse condition. (P.1(d)) (Section 4OA5.08)

Licensee-Identified Violations

None.

REPORT DETAILS

Summary of Plant Events On October 7, 2010, at 12:13 a.m., a reactor trip occurred due to an over-current trip of the C RCP. Approximately 17 seconds after the reactor trip, C RCP number 1 seal leak-off flow lowered to zero gallons per minute (gpm) and C RCP number 2 seal leak-off flow went to 2.5 gpm. The B main feedwater pump (MFP) also tripped. After the loss of the C RCP, steam generator C water level rose due to AFW actuation and normal feedwater system flow leaking through the feedwater regulating valve. The C steam generator water level reached the Hi-Hi level setpoint and a feedwater isolation occurred, which tripped the A MFP. This occurred approximately eleven minutes after the reactor trip.

Two events associated with restoration of normal feedwater occurred. The first occurred at 4:05 a.m., when the licensee attempted to start the A MFP and it immediately tripped. The A MFP tripped because the feedwater isolation logic was not reset. The second event occurred at 10:18 a.m. on October 7, 2010, when the licensee positioned the feedwater isolation switches to the Override/Reset position and maintained them in that position to support starting and running the A MFP. This resulted in defeating the feedwater isolation function, which is required to be operable while the reactor plant is in Mode 3. TS Limiting Condition for Operation (LCO) 3.0.3 was applicable from the time the feedwater isolation switches were placed in the Override/Reset position, until the feedwater isolation function operability was restored at approximately 1:29 p.m. on October 7, 2010.

Special Inspection Charter Inspection Scope Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a Special Inspection was initiated in accordance with NRC Inspection Procedure 93812, Special Inspection Team. The inspection focus areas included the following special inspection charter items:

1. Develop a sequence of events from the time of the reactor trip until the unit was placed

in Mode 5.

2. Assess the operator performance during and following the reactor trip until cold

shutdown was achieved. Assessment should include a review of the adequacy of procedures that were used during the event response (reactor trip, restoration of normal feedwater). In addition, evaluate adequacy of the training provided to the operators regarding the feedwater isolation logic, and specifically resetting the feedwater isolation signal.

3. Review the licensees immediate corrective actions (CAs) associated with this event,

including the adequacy of any remediation training provided to shift personnel.

4. Review operations performance and related procedural issues along with associated

corrective actions identified in the licensees root cause from the March 28th event.

Compare those operator performance issues to those identified with this event, and determine if similarities exist.

5. Review the performance of all RCP pump seals and motors following the units restart

from the refueling outage until the October 7 reactor trip. Review the as-found condition of the C RCP seals and potential causes for its degradation.

6. Review the corrective actions completed by the licensee regarding all the RCP pump

seals and motors following the March 28th event. In addition, review how the licensee assessed the readiness of the RCPs prior to the unit restart from the refueling outage.

7. Review this event to determine if it meets the criteria for the Scrams with Complications

performance indicator.

8. Collect data necessary to develop and assess the safety significance of any findings in

accordance with IMC 0609, Significance Determination Process.

9. Identify any potential generic safety issues and make recommendations for appropriate

follow-up actions (e.g. Information Notices, Generic Letters, and Bulletins).

4OA5 Other Activities - Special Inspection

.01 Develop a sequence of events from the time of the reactor trip until the unit was placed

in Mode 5.

October 7, 2010, 12:13 a.m.: The C RCP tripped and the reactor automatically tripped.

The main turbine also automatically tripped as a result of the reactor trip. The plants alarm and annunciation system indicated that the reactor trip was due to a single loop low flow condition, which is expected when a single RCP trips. Seventeen seconds after the C RCP trip, the C RCP number 1 seal leak-off flow lowered to zero gpm. The B MFP also tripped.

October 7, 2010, 12:14 a.m.: The steam relief valves for the 6A and 6B feedwater heaters lifted in the turbine building. Steam from the relief valves was carried by wind (the turbine building is open to the outside environment) and caused the turbine building lubricating oil fire protection system (deluge system) to actuate. This actuation released water into the turbine building and caused several alarms in the main control room.

October 7, 2010, 12:15 a.m.: A security guard near the turbine building reported to the main control room by telephone that some type of explosion had occurred in the turbine building. This report was later determined to be the result of the loud noise that steam reliefs make combined with the large steam cloud in the turbine building after the steam reliefs lifted.

October 7, 2010, between 12:15 a.m. and 12:20 a.m.: A two inch fire protection system pipe ruptured, spilling water into the second level of the turbine building.

October 7, 2010, 12:19 a.m.: In response to a high water level in the C steam generator, the control room balance of plant (BOP) operator stopped the turbine driven AFW pump and stopped flow from the motor driven AFW pump to the C steam generator. AFW flow to C steam generator was stopped with C steam generator level at 67%.

October 7, 2010, 12:20 a.m.: The main control room received a report of flooding in the turbine building. The turbine building lubricating oil deluge system was isolated by operators in the turbine building using manual valves.

October 7, 2010, 12:21 a.m.: The ruptured fire system piping was isolated by operators in the turbine building using manual valves. This also isolated fire protection system water to approximately half of the turbine building.

October 7, 2010, 12:24 a.m.: The C steam generator water level reached the Hi-Hi level setpoint, which resulted in a feedwater isolation and the A MFP to trip. Water level in the C steam generator continued to rise between 12:19 a.m. and 12:24 a.m.

due to heating of the water in the steam generator and leakage past the C feed regulating valve.

October 7, 2010, between 12:24 a.m. and 2:10 a.m.: Operators used procedure EPP-004, Reactor Trip Response to stabilize plant parameters, including pressurizer water level, which experienced a transient due to initial primary plant cooldown and the associated system response. Operators also used AOP-018, Reactor Coolant Pump Abnormal Conditions due to the low C RCP seal leak-off flow.

October 7, 2010, 2:10 a.m.: Operators exited EPP-004, Reactor Trip Response and entered GP-004, Post Trip Stabilization.

October 7, 2010, 4:05 a.m.: Operators in the main control room attempted to reset the feedwater isolation actuation by temporarily placing the feedwater isolation switches in the Override/Reset position. The switches were then returned to the Normal position.

Operators then attempted to restart the A MFP. However, with the feedwater isolation switch in Normal, the initiation logic also required that the reactor trip logic be reset prior to the feedwater actuation logic being reset. Therefore, when the A MFP was started, it immediately tripped and the AFW system logic reinitiated the AFW system.

Operators continued to use AFW to maintain steam generator water level.

October 7, 2010, between 6:00 a.m. and 6:37 a.m.: The night shift operations crew turned the watch over to the day shift operations crew.

October 7, 2010, 10:18 a.m.: Operators bypassed the feedwater isolation logic by placing the feedwater isolation switches in the Override/Reset position and leaving them in this position. They then restored the normal feedwater system to operation using the A MFP.

October 7, 2010, 10:20 a.m.: The AFW system was placed in standby.

October 7, 2010, 1:29 p.m.: Main control room operators realized that TS 3.3.2 was not met with the feedwater isolation logic bypassed and applied TS 3.0.3. Operators then closed the feedwater isolation valves to satisfy TS 3.3.2 and exited TS 3.0.3.

October 7, 2010, 1:36 p.m.: AFW was placed back into service with the A motor-driven pump running.

October 7, 2010, 1:38 p.m.: The A MFP was stopped and the feedwater isolation logic switches were placed in Normal.

October 8, 2010, 2:00 a.m.: The reactor trip logic was reset by closing the reactor trip breakers.

October 8, 2010, 2:01 a.m.: The feedwater isolation logic was reset by temporarily placing the logic switches in Override/Reset and the A MFP was placed into service.

October 8, 2010, 2:04 a.m.: The reactor trip breakers were opened.

October 8, 2010, 2:05 a.m.: The AFW system was placed in standby.

October 9, 2010, 8:02 p.m.: The reactor plant entered Mode 4.

October 10, 2010, 11:52 a.m.: The reactor plant entered Mode 5.

.02 Assess the operator performance during and following the reactor trip until cold

shutdown was achieved. Assessment should include a review of the adequacy of procedures that were used during the event response (reactor trip, restoration of normal feedwater). In addition, evaluate adequacy of the training provided to the operators regarding the feedwater isolation logic, and specifically resetting the feedwater isolation signal.

a. Inspection Scope

The inspectors conducted an independent review of control room activities to determine if licensee staff responded properly during the event. The inspectors specifically focused on the effectiveness of control board monitoring, technical decision making, and work practices of the operating crew. The inspectors reviewed the procedures used to mitigate the event and reestablish main feedwater flow to the steam generators to verify their proper use and that transitions between procedures were appropriate. The inspectors also reviewed the procedures for adequacy. The inspectors review included training lesson plans, training scenarios and simulator performance compared to the actual event. Inspectors evaluated the adequacy of training provided to the operators prior to the event. Inspectors performed the following activities in order to understand and evaluate the control room operating teams actions in response to the event:

• Conducted interviews with control room operations personnel on shift during the reactor trip and subsequent reset of the main feedwater isolation logic;

• Reviewed operating and administrative procedures, narrative logs, event recorder data, system drawings, and plant computer data;

• Observed a simulated plant response to this event as demonstrated on the plant simulator;

• Reviewed the crews implementation of emergency, abnormal, alarm, and normal operating procedures as well as TS; and

• Reviewed licensed operator training lesson plans and attendance records pertaining to the feedwater isolation logic, operation of associated components and controls, and associated administrative requirements.

.1 Inspector Observations

Through a review of plant data, the inspectors determined that the crews response to the reactor trip effectively stabilized the plant. The crew effectively monitored and controlled plant parameters.

Through interviews and a review of plant procedures, the inspectors determined that the night shift and day shift operating crews used direction contained in GP-004, Post Trip Stabilization, to attempt to reset the feedwater isolation signal. The inspectors observed that procedure GP-004, Attachment 10.2, Starting a Main Feed Pump, step 6, directs operators to reset the feedwater isolation signal if it had been previously received.

However, the procedure did not contain specific instructions for manipulating switches and components necessary to reset the signal nor did it reference an operating procedure to provide specific directions for accomplishing the task. The inspectors concluded that operators relied on their knowledge of system operation in order to perform the task, rather than using the procedural directions for performing the task contained in OP-403, Feedwater System.

Through a review of training material and attendance records, the inspectors determined that licensed operator requalification training conducted in March 2009 included a lecture covering design and operation of the feedwater isolation logic as well as applicable TS.

Inspectors concluded that this training adequately instructed operators to monitor automatic operation of the feedwater system, reset a feedwater isolation signal, and be aware of the TS associated with the feedwater isolation instrumentation. During interviews, operators could not recall any specific instances when they had operated the feedwater isolation logic switches during normal operations (e.g. during performance of GP-004) in the plant or on the simulator, but the majority of operators were familiar with operation of these key switches during simulator scenarios requiring performance of FRP-H.1, Response to Loss of Secondary Heat Sink. Additionally, the more experienced operators were aware that the key switches are used during outages to allow surveillance testing of the main feedwater regulating valves.

The inspectors verified that the plant reference simulator properly modeled the event and provided no negative training associated with plant response. The inspectors observed that the simulator did not model the leak by of the main feedwater regulating valves, which resulted in a slightly different steam generator level. This leak by contributed to over-filling the C steam generator and the main feedwater isolation in the plant on October 7, 2010. However, this difference was not considered to be significant in affecting the operator response to the event. The inspectors verified that the simulator properly modeled the actions required to properly reset a main feedwater isolation signal.

Through interviews, inspectors determined that the night shift operating crew interpreted GP-004, Attachment 10.2, step 6 to mean momentarily place the feedwater isolation logic switch for each steam generator to the Override/Reset position, and then place the switch back to the Normal position. This action did not reset the feedwater isolation logic because the reactor trip signal, which seals in the feedwater isolation signal, also needed to be reset by closing the reactor trip breakers. Believing that the feedwater isolation signal was reset, operators attempted to start the A MFP. The A MFP started, but immediately tripped due to the sealed in feedwater isolation signal.

Additionally, the AFW system reactuated because no MFPs were running (the AFW actuation logic was reset when the A MFP switch was manipulated).

Through interviews, inspectors determined that night shift operating crew personnel did not initially remember that the reactor trip breakers needed to be closed in order to reset the feedwater isolation signal. Additionally, although plant procedure OP-403, Feedwater System, Section 8.4.4, Resetting Feedwater Isolation Signals, contained detailed guidance for resetting the feedwater isolation signal, the inspectors determined that the crew was unfamiliar with the guidance contained in Section 8.4.4 and did not reference this procedure. After failing to restore main feedwater, night shift operators did not make any additional attempts to reset the feedwater isolation signal. The crew requested technical assistance from the Outage Control Center (OCC), continued to feed the steam generators using the AFW system, and conducted shift turnover with the oncoming crew. The turnover included discussions regarding the problem experienced when attempting to restore main feedwater.

The day shift operating crew, with the assistance of plant engineering staff, investigated the cause of the failed attempt to restore main feedwater. Using logic diagrams, the crew identified that, in order to remove the sealed in feedwater isolation signal, either the reactor trip signal had to be reset or the feedwater isolation logic switch had to be positioned to the Override/Reset position and maintained in that position. While this conclusion was correct with respect to system operation, operators failed to identify that maintaining the feedwater isolation reset switches in the Override/Reset position was a condition not allowed in Mode 3 by TS 3.3.2, Engineered Safety Feature Actuation System (ESFAS) Instrumentation. In order to reset the feedwater isolation signal, the reactor trip breakers must be closed (closing the reactor trip breakers resets the reactor trip signal, which seals in the feedwater isolation signal). However, procedural requirements prevented closing the reactor trip breakers until several surveillance tests were either performed or verified to have been previously performed within their required periodicities (31 days). Through interviews, the inspectors determined that the day shift crew attempted to determine if the most recent surveillance tests had been performed within the past 31 days by checking the controlled document electronic database. The crew determined that the most recent performance of the surveillance tests recorded in the electronic database were in June and July 2010 (outside the 31 day window.) The crew concluded that the surveillance tests, which require several hours to complete, would have to be performed before closing the reactor trip breakers. However, the required surveillance tests were most recently performed prior to plant startup after a reactor trip in September 2010 (within 31 days prior to October 7, 2010), and, unbeknownst to the operations crew, were not required to be performed before closing the reactor trip breakers. Operators did not find a record of the September surveillance tests in the electronic database because it had not been updated to reflect completion of these tests.

Through interviews and a review of post trip written statements by the operators, the inspectors determined that the day shift operating crew was motivated to restore the main feedwater system for two primary reasons. The first reason is that steam generator levels are significantly easier for operators to control using the main feedwater system as compared to the AFW system. The second reason was because the operating crew was aware of discussions that took place between the Operations Manager and Superintendent of Shift Operations regarding the potential regulatory consequence of being unable to prove restoration capability of main feedwater. During interviews, crew members characterized the consequences of not being able to restart the main feedwater pump as a regulatory concern, a red finding, and a red cornerstone.

At 10:18 a.m. on October 7, 2010, after consultation with the Operations Manager, operators placed the feedwater isolation switches in Override/Reset, which bypassed the automatic feedwater isolation signal, an engineered safety feature actuation system function.

The inspectors concluded that the failure of the operating crew to perform a formal pre-job brief was a missed opportunity for the crew to identify the applicable TS requirements prior to placing the feedwater isolation logic switches in the Overrride/Reset position. During interviews, operators stated that they determined a pre-job brief was not necessary due to the extensive and lengthy discussions that had taken place over the course of the shift concerning restoration of main feedwater. Had the operating crew conducted a pre-job brief using Attachment 10.6 of PLP-037, Conduct of Infrequently Performed Tests or Evolutions and Pre-Job Briefs, they likely would have identified that maintaining the feedwater isolation switches in Override/Reset was not allowed per TS 3.3.2. The following two required items in 10.6 of PLP-037 may have prompted the crew to review TS 3.3.2:

• Describe any applicable LCO conditions and the requirements if not met; and

• Describe the evolution and expected plant response including expected alarms.

After placing the feedwater isolation key switches in Override/Reset the control room received engineering safeguards annunciator panel alarm C-1, Feedwater Isolation / CV Spray Override/Reset. Through interviews, the inspectors determined that the operating crew acknowledged the alarm, and recognized that the alarm was caused by repositioning the feedwater isolation key switches. However, the crew did not review the associated annunciator response procedure, APP-002-C1, either before or after the alarm was received, contrary to OPS-NGGC-1000, Fleet Conduct of Operations. Had the operating crew reviewed APP-002-C1, either as part of a pre-job brief when discussing expected alarms, or in responding to an unexpected alarm, they likely would have identified that maintaining the feedwater isolation key switches in Override/Reset was not allowed per TS 3.3.2. APP-002-C1 lists the following possible plant effects resulting from placing the feedwater isolation key switches in Override/Reset:

• Loss of the FW Isolation Signal from the Safeguards System to any feedline with the switch in the OVRD/RESET position; and

• Tech Spec LCO.

Additionally the References section of APP-002-C1 explicitly lists TS Table 3.3.2.1-1, Item 5, which is the feedwater isolation safety function. These failures to perform a pre-job brief and review applicable annunciator procedures were contributors to the violation listed below in paragraph 4OA5.02.b.2.

Through interviews and a review of plant operating logs, inspectors determined that the crew failed to recognize that the feedwater isolation switches were governed by TS 3.3.2 until shortly before 1:29 p.m. on October 7, 2010. The inspectors determined that the crew became aware that TS 3.3.2 and TS LCO 3.0.3 were applicable when they discussed contingency actions associated with main feedwater in the event a safety injection actuation occurred. The crew discussed that, in the event of a safety injection actuation, manual actions would be required to accomplish the required feedwater isolation safety function, either by realigning individual main feedwater components or returning the feedwater isolation switches to the Normal position. This discussion resulted in the crew realizing the error. At 1:29 p.m. the crew closed the feedwater bypass valves to all three steam generators in order to meet the requirements of TS 3.3.2 and exit TS LCO 3.0.3. Operators subsequently restarted the AFW system and, at 1:38 p.m. on October 7, 2010, placed the feedwater isolation key switches in the Normal position.

.2 Procedure Violation for Overriding Feedwater Isolation Safety Function in Mode 3.

Introduction:

A self-revealing Green NCV of Technical Specification (TS) 5.4.1, Procedures, was identified when the licensee bypassed the feedwater isolation safety function, in Mode 3, a condition prohibited by TS and an action contrary to procedural requirements.

Description:

At 12:13 a.m. on October 7, 2010, the reactor tripped due to a trip of the C RCP. Eleven minutes after the reactor trip, the C S/G reached the Hi-Hi water level setpoint, which automatically actuated the feedwater isolation signal. Automatic actuation of the feedwater isolation signal shut the main feedwater system isolation valves, feedwater regulating valves, and the feedwater regulating bypass valves, and stopped the running main feed pump. The AFW system was already running because it automatically initiated due to low S/G levels immediately after the reactor trip, and the AFW system continued to supply feedwater to the A and B S/Gs. Operators isolated AFW to the C S/G six minutes into the event. However, level in the C S/G continued to increase due to leak by past the main feedwater regulating valve and expansion of the water in the S/G as it heated. Operators continued to use AFW to supply the S/Gs until 4:05 a.m. on October 7, 2010, when they stopped the running AFW pumps and attempted to reset the feedwater isolation signal, which would have allowed them to restore main feedwater flow to the S/Gs. During this attempt to reset the feedwater isolation signal, operators used guidance contained in procedure GP-004, Post Trip Stabilization. This procedure instructed operators to Reset the feedwater isolation, but did not provide specific details on exactly how to reset the system logic. Lacking this specific guidance, operators interpreted GP-004 to mean momentarily place the switch in the Override/Reset position, and then place the switch back in the Normal position.

This action did not reset the feedwater isolation logic. In order to reset the feedwater system logic, the reactor trip signal needed to be reset which requires closing the reactor trip breakers. Detailed guidance for resetting the feedwater isolation signal is contained in plant procedure OP-403, Feedwater System, but the operators were unfamiliar with the guidance contained in OP-403 and did not reference this procedure.

Day shift operators, with the help of plant engineering staff, investigated the cause of the failed isolation system signal reset attempt by analyzing the system electrical schematics and logic diagrams. Operators identified that, in order to reset the feedwater isolation signal, the reactor trip signal had to be reset, or the feedwater isolation logic switches had to be positioned to the Override/Reset position and maintained in that position.

While this conclusion was technically correct, operators failed to consider that maintaining the feedwater isolation logic switches in the Override/Reset position was a condition not allowed by TS 3.3.2, ESFAS Instrumentation, and would therefore place the plant in conditions governed by TS 3.0.3. TS 3.0.3 is applicable when the feedwater isolation logic switches are in the Override/Reset position because the feedwater isolation safety function is bypassed. Operators also mistakenly thought that surveillances were required to be performed prior to cycling the reactor trip circuit breakers, which would take several hours to perform.

At 10:18 a.m. on October 7, 2010, after consultation with the Operations Manager and without consulting the correct governing procedure, operators placed the feedwater isolation logic switches in Override/Reset, which bypassed the feedwater isolation safety function, and did not apply TS 3.0.3. Operators then started the A MFP and restored main feedwater flow to the steam generators. The feedwater isolation logic was bypassed in this manner until 1:29 p.m. on October 7, 2010, after operators referenced TS 3.3.2 and realized that it did not provide for completely bypassing the feedwater isolation logic in Mode 3, and isolated the main feedwater system. They then restarted the AFW system and, at 1:38 p.m. on October 7, 2010, placed the feedwater isolation logic switches in the Normal position.

Analysis:

The failure to operate the feedwater isolation logic switches in accordance with plant procedures is a performance deficiency. The finding is more than minor because it affects the human performance attribute of the Mitigating Systems cornerstone and the objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e. core damage). Specifically, the finding affected the ability for the FWIS to isolate the steam generators and prevent overfeeding/overcooling events. The inspectors evaluated this finding using IMC 0609, Significance Determination Process Phase 1 screening and determined that the finding represented a complete loss of the FWIS function for three hours and eleven minutes which required further evaluation under SDP Phase 2. The Robinson Phase 2 Pre-Solved SDP Worksheet and Phase 2 SDP Notebook did not include the FWIS function therefore a phase 3 SDP analysis was performed by a regional SRA in accordance with NRC IMC 0609. The plant was already shutdown prior to the performance deficiency so Anticipated Transients Without Scram scenarios would not be valid. The impact of the loss of the FWIS function would be that the S/Gs would not be isolated on Hi-Hi S/G water level and plant overfeed scenarios could result in Safety Injection initiation on low RCS pressure and potential overfill/moisture carryover scenarios could overspeed the turbine driven AFW pump. The phase 3 analysis considered a potential overcooling and safety injection scenario using the licensees full scope Robinson PRA model data and a potential moisture carryover induced overfeed scenario causing a loss of the turbine driven AFW pump on overspeed using the NRC Robinson SPAR model and data. The core damage frequency increase for both scenarios was <1E-6 per year. The risk was mitigated by the short exposure period.

The finding is characterized as Green, a finding of very low safety significance. The finding has a cross-cutting aspect in the Human Performance area, Decision Making component because the licensee failed to make a safety-significant or risk-significant decision using a systematic process to ensure safety was maintained when faced with uncertain or unexpected plant conditions. Specifically, the licensee intentionally bypassed the safety function of feedwater isolation instrumentation while it was required with the reactor plant in Mode 3. (H.1(a))

Enforcement:

TS 5.4.1, Administrative Control (Procedures), requires that written procedures shall be established, implemented, and maintained, covering applicable procedures recommended in Regulatory Guide 1.33, Appendix A, November 1972.

Section 3 of Regulatory Guide 1.33, Appendix A, November 1972 states that operation of systems that affect the safety of the nuclear power plant, including the feedwater system, should be conducted in accordance with written procedures. The licensee established OP-403, Feedwater System, as the governing procedure for operation of the feedwater system. Contrary to OP-403, on October 7, 2010, operations personnel completely bypassed the feedwater isolation safety function by placing the feedwater isolation logic switches in Override/Reset for three hours and eleven minutes.

Because this issue is of very low safety significance and has been entered into the licensees CAP as AR 425643, the violation is being treated as a Non-Cited Violation consistent with Section 2.3.2 of the NRC Enforcement Policy and is designated as NCV 05000261/2010012-01, Procedure Violation for Overriding Feedwater Isolation Safety Function in Mode 3.

.03 Review the licensees immediate corrective actions (CAs) associated with this event,

including the adequacy of any remediation training provided to shift personnel.

a. Inspection Scope

Inspectors reviewed the licensees immediate and short term corrective actions, portions of the licensees preliminary Significant Adverse Condition Investigation, and Reactor Trip Reports for the event in order to 1) independently assess the licensees investigation of the event, 2) verify the licensee had appropriately assessed plant and operator response to the event, and 3) assess the adequacy of the licensees immediate corrective actions. Inspectors also conducted interviews with plant personnel, performed observations of plant operations personnel, and visually inspected plant equipment.

b. Observations and Findings

No findings were identified.

.1 Remediation Training for Shift Personnel.

The inspectors concluded that the licensees investigation report for this event adequately identified and addressed the issues arising from this event. The inspectors verified the licensee adequately evaluated operator response to the event and identified factors contributing to operator performance. The inspectors determined that the licensee adequately identified and documented causes specific to the event as well as immediate and proposed corrective actions for identified discrepancies.

The inspectors concluded that the remedial training provided to the operating crews was effective in correcting individual and crew performance gaps related to the event. The inspectors concluded that the classroom training provided to operating crews following the event was effective in communicating lessons learned from the event. The training reinforced management expectations for procedure adherence, leadership behaviors, use of human performance tools, and procedure changes. The training increased operator knowledge concerning the operation of plant control systems and associated TS. Inspectors observed portions of the remedial training provided to the operating crews, including classroom and simulator training. Licensee management provided specific training on control room procedure use and adherence, and the inappropriateness of supervisions communication of perceived regulatory influence on control room operations. The inspectors concluded that training adequately covered the deficiencies identified during the events of October 7, 2010, and actions to be taken if nondescript procedure steps are encountered in the future.

.2 Other Immediate Corrective Actions.

Inspectors also determined that the licensee completed additional corrective actions.

Plant and corporate management conducted extensive oversight observations of main control room operations for several weeks following the event in order to verify that plant operators were meeting their expectations for safe operation of the plant. General operating procedures were reviewed and revised in order to eliminate vague or poorly described steps, including GP-007, Plant Cooldown to Mode 5, which was revised prior to the plant being cooled down to cold shutdown.

The licensee also determined that the fire protection system pipe failure was caused by galvanic corrosion due to two dissimilar metals being used at a junction without dielectric insulation. Immediate corrective action for the fire protection failure included inspection of the other junctions in the fire protection system, repair of the failed junction, and restoration of the system to a functional condition. The licensee also installed discharge piping on the 6A and 6B feedwater heater relief valves in order to direct steam to an area that would not actuate the turbine building lube oil deluge system. Longer term corrective actions included replacement of the C RCP motor and seals prior to the restart of the unit.

.04 Review operations performance and related procedural issues along with associated

corrective actions identified in the licensees root cause from March 28th event.

Compare those operator performance issues to those identified with this event, and determine if similarities exist.

a. Inspection Scope

Inspectors reviewed the issues associated with this event as well as the licensees Significant Adverse Condition Investigation Report (Action Request Number 390095)and the NRC Augmented Inspection Team Report (Report Number 05000261/2010009)issued following the March 28, 2010 event.

b. Observations and Findings

No findings were identified.

Inspectors observed the following similarities, with respect to operator performance, between the March 28 event and this event. The following observations contributed to the violation discussed in Section 4OA5.02.

Operator Knowledge of System Operation During the March 28, 2010, event, operator knowledge of the main generator lockout was not sufficient to have a complete understanding of inputs and expected outcomes when reset during the event. Resetting the generator lockout relays caused a plant fire.

Similarly, during the October 7, 2010, event, night shift operator knowledge of the feedwater isolation logic was not sufficient to have a complete understanding of inputs and expected outcomes when attempting a reset. Incorrectly resetting the feedwater isolation signal resulted in an unexpected AFW system actuation.

Operator Use of Normal Operating Procedures (GP-004)

During the March 28, 2010, event, operators performed a step in GP-004 related to electric plant realignment without questioning whether performance of the action was appropriate given the electric plant transient that had occurred and uncertainty regarding the nature and extent of damage to the electric plant.

Similarly, during the October 7, 2010, event, operators performed a step in GP-004 related to restoration of main feedwater without questioning whether performance of the action was appropriate given current plant conditions (Mode 3.) Overriding the feedwater isolation signal resulted in defeating the feedwater isolation automatic actuation logic, a condition not allowed by TS 3.3.2.

Operator Use of Annunciator Response Procedures During the March 28, 2010, event, operators did not reference an annunciator response procedure (ARP) prior to incorrectly attempting to manually align the suction of the charging pumps to the refueling water storage tank. The ARP contained directions for performing this alignment. Additionally, operators did not reference the annunciator response procedure for an alarm indicating an abnormal condition before attempting to reset generator lockout relays per GP-004. Resetting the generator lockout relays caused a plant fire.

Similarly, during the October 7, 2010, event, operators did not reference an ARP before or after overriding the feedwater isolation automatic actuation logic. The ARP contained explicit, detailed information concerning the effect on the plant, including TS.

Operator Recognition of Technical Specification Applicability During the March 28, 2010, event, the crew did not recognize that the reactor coolant system cooldown rate exceeded the limiting value specified in TS 3.4.3 and the requirement to evaluate the actions contained in TS 3.4.3 was not recognized by the crew at any time during or after the cooldown.

Similarly, during the October 7, 2010, event, the crew did not recognize that the position of the feedwater isolation switches was governed by TS 3.3.2 until several hours after the crew repositioned the switches, placing the plant in TS LCO 3.0.3.

.05 Review the performance of all RCP pump seals and motors following the units restart

from the refueling outage until the October 7 reactor trip. Review the as-found condition of the C RCP seal and potential causes for its degradation.

a. Inspection Scope

The inspectors reviewed data collected from plant instrumentation for the C RCP including frame vibrations, shaft vibrations, bearing temperatures, labyrinth seal differential pressure, seal leak-off flows, seal leak-off temperatures, motor current, and cooling water return temperatures for the time period from plant startup following the spring, 2010 refueling outage until the plant trip on October 7, 2010. Inspectors also reviewed the as-found condition of the C RCP seals and reviewed the conclusions contained in the licensees C RCP seal laboratory report.

.1 Inspector Observations

All C RCP parameters indicated a normal trend, and consistent with the A and B RCPs except for a slight increasing trend in shaft vibrations for the C RCP in the month of September (approximately 5% increase in magnitude). However, the magnitude of the increase is not significant, and still well below the manufacturers limit for indication of an abnormality.

The licensee concluded that the number 2 and number 3 seals sustained damage during the October 7, 2010, event. The damage to the number 2 and number 3 seals was consistent with high voltage arcing across the seal components, and therefore, the C RCP seals were damaged after the C RCP motor faulted and fault current travelled through the seals. Inspectors compared metallurgical and visual evidence to report conclusions and found the licensees determination of the cause of the seal failures (arcing due to motor fault current) to be the most likely cause of the C RCP seal failures.

.2 Unresolved Item (URI): C RCP Motor Failure and Seal Damage

Introduction.

The inspectors identified an URI associated with the failure of the C RCP motor and damage to two of the C RCP seals. Upon review of the licensees root cause evaluation of the failure of the C RCP motor and seals, the inspectors identified that the licensee was aware of vibration and age related degradation vulnerabilities of the C RCP motor windings, and planned to rewind the motor in order to eliminate these vulnerabilities. However, the licensee did not rewind the motor prior to its failure. This item is unresolved pending further review and evaluation of the licensees refurbishment plans and failure susceptibility analysis.

Description.

The licensees root cause evaluation for the failure of the C RCP motor identified that vibration induced winding degradation, combined with age related thermal degradation of motor winding insulation, led to a turn-to-turn fault in the motor stator end windings. This turn-to-turn fault propagated to a phase-to-phase fault and the fault current traveled to the motor rotor and eventually through the number two and three seals. Laboratory analysis showed that the seals had evidence of arcing damage. The licensees root cause evaluation report also states that the licensee was aware of these degradation vulnerabilities and, in 2003, formulated a motor rewind plan for all of their RCPs. However, the C RCP that failed on October 7, 2010, had not been rewound to eliminate the end winding vibration degradation mechanism prior to its failure.

Additionally, the replacement RCP motor currently installed as C RCP motor has not been rewound to eliminate the end winding vibration degradation mechanism. Pending the results of this additional inspection an Unresolved Item will be opened and designated as URI 05000261/2010012-02, C RCP Motor Failure and Seal Damage.

.06 Review the corrective actions completed by the licensee regarding all the RCP pump

seals and motors following the March 28th event. In addition, review how the licensee assessed the readiness of the RCPs prior to the unit restart from the refueling outage.

a. Inspection Scope

The inspectors reviewed corrective action program and maintenance records regarding the RCPs following the plant electrical transient and reactor trip which occurred on March 28, 2010. These records included the 4kV recovery plan, which detailed which large motors were potentially affected by the March 28, 2010 transient, the work orders associated with the C RCP, the test procedures for the RCPs, and the RCP test results.

Specific procedures are listed in the attachment to this report.

b. Observations and Findings

No findings were identified.

After the March 28, 2010, event, the licensee conducted testing on several motors and electrical loads that were susceptible to damage as the result of the electrical transient that occurred. The C RCP was identified as being potentially affected. The motor testing included insulation resistance testing of the cables to the C RCP motor, motor winding resistance testing, and motor insulation resistance testing. Testing methodology included bridge, megger, and polarization index. All motor testing results met applicable acceptance criteria and were similar to previous test results for the C RCP from April, 2004.

After the March 28, 2010, event, the licensee also disassembled and inspected the C RCP seals. Some components did not pass inspection and were replaced including the number 1 seal insert, the number 3 seal ring, and the number 3 seal runner. According to licensee maintenance records, the C RCP seals were rebuilt to satisfactory condition.

Post maintenance performance of the motor and seals was satisfactory.

.07 Review this event to determine if it meets the criteria for the Scrams with complications

performance indicator.

a. Inspection Scope

Inspectors reviewed this event to determine if the circumstances of the event meet criteria for being classified as a scram with complications per guidance contained in NEI 99-02, Appendix H, Unplanned Scram with Complications Basis Document. Inspectors reviewed plant procedures, operator logs, and the plant post trip report, and interviewed plant personnel. Documents reviewed are in the attachment to this report.

b. Findings

Per the General Reporting section of NEI 99-02, licensees determine performance indicator statistics and report them to the NRC quarterly. At the time of this inspection, the licensee had not determined if they would report this event as a scram with complications. Therefore, inspectors were unable to review the licensees rationale for their final determination. NRC inspectors will, after the licensee details their position, determine if they are in agreement with the determination.

.08 Collect data necessary to develop and assess the safety significance of any findings in

accordance with IMC 0609, Significance Determination Process.

a. Inspection Scope

The inspectors reviewed licensee procedures, corrective action program documents, work orders, root cause evaluations, operability assessments, engineering evaluations, and operating experience information to gather data necessary to develop and assess the safety significance of any findings.

.2 Operator Transient Response Adversely Affected by Uncorrected, Known Plant

Deficiency

Introduction.

The inspectors identified a Green finding for failure to correct a known equipment deficiency which adversely affected the operators ability to respond to reactor trip transients.

Description.

On October 7, 2010, the reactor tripped at 12:13 a.m. Shortly afterwards, several fire alarms were received at the main control room fire protection alarm panel and the fire protection system indicated that the motor driven fire pump had automatically started. At approximately the same time the fire protection alarms were sounding, a report was received by telephone in the main control room that an explosion occurred in the turbine building. Operators were dispatched to the turbine building to investigate the cause of the alarms and discovered that the turbine building deluge system was actuated. Operators in the turbine building also discovered that a two inch pipe leading to a fire hose station had ruptured. At 12:20 a.m., the main control room received a report by telephone that there was flooding in the turbine building.

Since the reactor tripped at 12:13 a.m., plant operators were forced to perform reactor trip response actions coincident with the multiple distractions described above.

Inspectors determined that spurious actuations of the turbine building lube oil deluge system had previously occurred after reactor trips on May 15, 2007, and November 6, 2009. On neither of these occasions did the licensee address this adverse condition in the CAP. Therefore, contrary to the licensees corrective action program, as described in procedure CAP-NGGC-0200, the licensee failed to address and correct an abnormal or unexpected equipment condition that affected and complicated plant events.

Analysis.

The failure to correct a long-standing deficiency that adversely affected operator response to reactor trips is a performance deficiency. The finding is more than minor because it is associated with the mitigating systems cornerstone attribute of human performance in that the performance of operators was adversely affected by the fire protection actuation after the reactor trip. This adverse effect included diverting operator attention and resources away from initiating event response. Using the Inspection Manual 0609, Significance Determination Process Phase 1 Worksheet, the inspectors concluded that the finding is of very low safety significance (Green) because it is not a design or qualification deficiency, does not represent a loss of safety function, does not represent the loss of safety function of a single train of TS equipment, does not represent the loss of risk-significant equipment, and is not potentially risk significant due to an external events. The cause of this finding is directly related to the corrective action program component of the problem identification and resolution cross cutting area because appropriate and timely corrective actions were not taken for a known adverse condition (P.1(d)).

Enforcement.

The inspectors determined that this finding did not involve a violation of NRC requirements and therefore is not subject to enforcement action. The licensee entered this issue into the CAP as NCR 425437. Because the finding does not involve a violation of regulatory requirements and has very low safety significance, it is identified as a Finding, FIN 05000261/2010012-03, Operator Transient Response Adversely Affected by Known Plant Deficiency.

.09 Identify any potential generic safety issues and make recommendations for appropriate

follow-up actions (e.g., Information Notices, Generic Letters, Bulletins) (Charter Item 9)

a. Inspection Scope

The inspectors reviewed the licensees internal operating experience database, preliminary root cause evaluation, corrective action program documents, and work orders to determine the potential for generic safety issues related to this event.

b. Findings

No findings were identified. Based on the information reviewed, the inspectors did not identify any generic safety issues. However, the licensees preliminary conclusions with regard to the failure mode of the C RCP number 2 and number 3 seals is a failure mode that other plants may be susceptible to. Further NRC review of the licensees conclusions will be necessary after the licensee completes their report on the event.

4OA6 Meetings, Including Exit

On October 15, 2010, the special inspection team leader presented the preliminary inspection results to Mr. Eric McCartney, then H.B. Robinson Steam Electric Plant Vice President, and members of his staff. Subsequently, additional in-office reviews were conducted and the final inspection results and preliminary significance determination were discussed with Mr. Robert J. Duncan on November 18, 2010. No proprietary information is included in this inspection report.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

R. Duncan, Site Vice President

NRC Personnel

Randall

A. Musser, Chief, Reactor Projects Branch 4, Division of Reactor Projects Region II

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

NCV

05000261/2010012-01 NCV Procedure Violation for Overriding Feedwater Isolation Safety Function in Mode 3 (Section 4OA5.02)

FIN

05000261/2010012-03 FIN Operator Transient Response Adversely Affected by Uncorrected, Known Plant Deficiency (Section 4OA5.08)

Closed

None.

Opened

URI

05000261/2010012-02 URI C RCP Motor Failure and Seal Damage (Section 4OA5.05)