IR 05000440/2005005

IR 05000440-05-005, on 01/07/2005 - 02/18/2005, Perry Nuclear Power Plant, Unit 1; Special Inspection for Recirculation Pump Downshift Events of December 23, 2004, and January 6, 2005

ML050830113
Person / Time
Site:	Perry
Issue date:	03/23/2005
From:	Reynolds S Division Reactor Projects III
To:	Richard Anderson FirstEnergy Nuclear Operating Co
References
IR-05-005
Download: ML050830113 (60)

Text

rch 23, 2005

SUBJECT:

PERRY NUCLEAR POWER PLANT NRC SPECIAL INSPECTION REPORT 05000440/2005005

Dear Mr. Anderson:

On February 18, 2005, the U.S. Nuclear Regulatory Commission (NRC) completed a special team inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection findings, which were discussed with you and other members of your staff on February 18, 2005.

The special inspection team was established by Region III, on January 7, 2005, using the guidance in Management Directive 8.3, NRC Incident Investigations Procedures. The special inspection was charted to evaluate the facts, circumstances, and your actions in response to the events of December 23, 2004, and January 6, 2005, when both reactor recirculation pumps unexpectedly shifted from fast to slow speed without operator action. The downshift events were similar and each initiated a sequence of events that ultimately resulted in a reactor scram.

Based on the results of this inspection, there were five NRC-identified and one self-revealed findings of very low safety significance, three of which involved violations of NRC requirements.

However, because these violations were of very low safety significance and because the issues were entered into the licensees corrective action program, the NRC is treating these findings as Non-Cited Violations in accordance with Section VI.A.1 of the NRCs Enforcement Policy. This inspection, its observations, and the associated findings highlight the importance of minimizing transient initiators. The December 23, 2004, and January 6, 2005 transients, at the Perry Nuclear Power Plant, challenged personnel and equipment, and during the second transient, mitigating equipment failed compounding operator response. The objective of the Initiating Events Cornerstone of Reactor Safety is to limit the likelihood of those events that upset plant stability and challenge critical safety functions; the failure to identify root causes and implement appropriate corrective actions related to transient initiators directly affected this objective. If you contest the subject or severity of a Non-Cited Violation, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C.

20555-0001; with a copy to the Regional Administrator, U.S. Nuclear Regulatory Commission -

Region III, 2443 Warrenville Road, Suite 210, Lisle, IL 60532-4352; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the Resident Inspector Office at the Perry Nuclear Power Plant.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Steven A. Reynolds, Deputy Director Division of Reactor Projects Docket No. 50-440 License No. NPF-58

Enclosure:

Inspection Report 05000440/2005005 w/Attachments: 1. Supplemental Information 2. Charter for Special Inspection

REGION III==

Docket No: 50-440 License No: NPF-58 Report No: 05000440/2005005 Licensee: FirstEnergy Nuclear Operating Company (FENOC)

Facility: Perry Nuclear Power Plant, Unit 1 Location: P.O. Box 97 A200 Perry, OH 44081 Dates: January 7 through February 18, 2005 Inspectors: S. Burton, Senior Resident Inspector, Team Lead R. Powell, Senior Resident Inspector S. Sheldon, Reactor Engineer R. Walton, Reactor Engineer Z. Falevits, Senior Reactor Engineer Observer: D. Reeser, Reactor Engineer Approved by: C. Lipa, Chief Branch 4 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000440/2005005; 01/07/05 - 02/18/05; Perry Nuclear Power Plant. Special Inspection for recirculation pump downshift events of December 23, 2004, and January 6, 2005.

This special inspection examined the facts and circumstances for two events where both reactor recirculation pumps unexpectedly shifted from fast to slow speed without operator action. On December 23, 2004, and January 6, 2005, both reactor recirculation pumps unexpectedly shifted from fast to slow speed without operator action. The downshift events were similar and each initiated a sequence of events that ultimately resulted in a reactor scram. On December 23, an automatic scram was initiated by the Oscillation Power Range Monitor System (OPRM),

while on January 6, a manual scram was initiated following the trip of one reactor recirculation pump. The inspection identified five NRC-identified and one self-revealed findings of very low safety significance, three of which involved violations of NRC requirements. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review.

The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

NRC-Identified

and Self-Revealed Findings

Cornerstone: Initiating Events

Green.

A self-revealed finding of very low significance and a Non-Cited Violation of 10 CFR 50.65(a)(3) was identified. The finding involved the transient initiated as a result of a trip of A reactor recirculation pump low frequency motor generator and subsequent manual scram of the reactor. The Non-Cited Violation was associated with a failure to incorporate industry operating experience into preventive maintenance activities that would have prevented the failure of the A reactor recirculation pump. The primary cause of this violation was related to the cross-cutting area of Problem Identification and Resolution.

The finding was more than minor because the event caused an actual upset in plant stability and operation resulting in a plant transient, thereby directly affecting the objective for the Initiating Events Cornerstone. Additionally, the trip affected the equipment performance attributes of availability and reliability of the Initiating Events Cornerstone of Reactor Safety. The issue was of very low safety significance because the finding did not result in exceeding the Technical Specification limit for identified reactor coolant system leakage and did not affect other mitigation systems; the finding did not contribute to both the likelihood of a reactor trip AND the likelihood that mitigation equipment or functions will not be available; and the finding did not increase the likelihood of a fire or internal/external flood. Proposed and completed corrective actions included a formal root cause analysis, replacement of the defective voltage regulator, and establishment of a process to review post-transient performance data. (Section 1.2)

Green.

A finding of very low significance was identified by the inspectors. The finding involved the licensees failure to quarantine equipment after both reactor recirculation pumps experienced an unplanned downshift from fast to slow speed on December 23, 2004. The inspectors determined that the failure to quarantine equipment impaired the licensees ability to identify the associated failure mechanism for the simultaneous downshifting of both reactor recirculation pumps. The primary cause of this finding was related to the cross-cutting area of Human Performance.

The finding was more than minor because the failure to quarantine equipment impaired the licensees ability to identify the associated failure mechanism, and as a result, a plant transient was initiated on January 6, 2005, that caused an actual upset in plant stability, which directly affected the objective for the Initiating Events Cornerstone.

Additionally, the reactor recirculation pump downshifts affected the equipment performance attributes of availability and reliability of the Initiating Events Cornerstone of Reactor Safety. The issue was of very low safety significance because the finding did not result in exceeding the Technical Specification limit for identified reactor coolant system leakage and did not affect other mitigation systems; the finding did not contribute to both the likelihood of a reactor trip AND the likelihood that mitigation equipment or functions will not be available; and the finding did not increase the likelihood of a fire or internal/external flood. No violation of NRC requirements occurred. (Section 2.2(1))

Green.

A finding of very low significance was identified by the inspectors. The inspectors concluded that the licensee failed to properly assess and document the assessment for the removal of restart restraints prior to resuming reactor operation subsequent to the December 23, 2004, scram, and that the failure to appropriately close and document the basis for resolving a mode restraint prior to startup impaired the licensees ability to identify the associated failure mechanism for the December 23 recirculation pump downshift event. The primary cause of this finding was related to the cross-cutting area of Human Performance.

The finding was more than minor because a plant transient was initiated on January 6, 2005, that caused an actual upset in plant stability, which directly affects the objective for the Initiating Events Cornerstone. Additionally, the reactor recirculation pump downshifts affected the equipment performance attributes of availability and reliability of the Initiating Events Cornerstone of Reactor Safety. The issue was of very low safety significance because the finding did not result in exceeding the Technical Specification limit for identified reactor coolant system leakage and did not affect other mitigation systems; the finding did not contribute to both the likelihood of a reactor trip AND the likelihood that mitigation equipment or functions will not be available; and the finding did not increase the likelihood of a fire or internal/external flood. No violation of NRC requirements occurred. (Section 2.2(2))

Cornerstone: Mitigating Systems

Green.

The inspectors identified a finding having very low safety significance and an associated Non-Cited Violation of Technical Specifications for inadequate safety-related breaker maintenance procedures. The inspectors determined that maintenance procedures for overhauling safety-related breakers were inappropriate, because they did not contain guidance to refurbish breakers within the vendors specified time frames or provide reasonable alternative preventative maintenance practices to ensure that safety-related breakers remained operable.

The finding was more than minor because the procedure quality attribute of the Mitigating Systems Cornerstone was affected when the licensee failed to evaluate industry and vendor recommended changes and incorporate the changes into their breaker maintenance procedures. The issue was of very low safety significance because the deficiency did not result in any loss of function; the finding was not risk significant due to a seismic, a flooding, or a severe weather initiating event; and because other plant-specific analyses that identify core damage scenarios of concern were not impacted. The finding was a Non-Cited Violation of Technical Specification Section 5.4, and Regulatory Guide 1.33, for inadequate maintenance procedures. The issue was entered into the licensees corrective action program and is being evaluated under multiple condition reports (CR 05-0187. CR 05-00230, CR 05-00253, CR 05-00274,

CR 05-00283, CR 05-00295, CR 05-00359, CR 05-00459). (Section 1.3(2))

Green.

The inspectors identified a finding having very low safety significance and associated Non-Cited Violation of Technical Specifications for inadequate procedures associated with safety-related breaker maintenance procedures. The inspectors determined that maintenance procedures for overhauling safety-related breakers were inappropriate, because they did not contain guidance to measure and monitor critical measurements identified by the vendor.

The finding was more than minor because the procedure quality attribute of the Mitigating Systems Cornerstone was affected when the licensee failed to evaluate industry and vendor recommended changes and incorporate the changes into their breaker maintenance procedures. The issue was of very low safety significance because the deficiency did not result in any loss of function; the finding was not risk significant due to a seismic, a flooding, or a severe weather initiating event; and because other plant-specific analyses that identify core damage scenarios of concern were not impacted. The finding was a Non-Cited Violation of Technical Specifications Section 5.4, and Regulatory Guide 1.33, for inadequate maintenance procedures. The finding was entered into the licensees corrective action program and is being evaluated under condition reports CR 05-00364 and CR 05-00095. (Section 1.3(3))

Green.

A finding of very low significance was identified by the inspectors. The inspectors concluded that the licensee failed to quarantine equipment. The inspectors determined the failure to quarantine the motor feed pump (MFP) breaker cubicle impaired the licensees ability to identify the associated failure mechanism for the January 6, 2005 failure of the MFP breaker to close. The primary cause of this finding was related to the cross-cutting area of Human Performance.

The finding was more than minor because the failure to quarantine the MFP breaker after the January 6, 2005 failure, if left uncorrected, could become a more significant safety concern. The finding affected the short term heat removal element of the Mitigating System Cornerstone and that the issue was not a design deficiency that resulted in a loss of function. The finding was of very low safety significance because the system was not a safety system and that the system was not a TS system. In addition, the finding did not represent an actual loss of safety function or equipment designed as risk-significant per 10 CFR 50.65 for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, the finding was not risk significant due to a seismic, a flooding, or a severe weather initiating event, therefore the finding screened as

Green.

No violation of NRC requirements occurred.

(Section 2.4(1))

REPORT DETAILS

Background and Overview On December 23, 2004, Perry Nuclear Power Plant experienced a transient and subsequent scram as a result of an unplanned downshift of both reactor recirculation pumps to slow speed.

Subsequently, on January 6, 2005, Perry Nuclear Power Plant experienced a transient and subsequent scram, again initiated by an unplanned downshift of both reactor recirculation pumps to slow speed. Although the scrams were not directly the result of the pump speed downshift, both transients were initiated by the same event and each challenged plant operators and equipment. Notably, any event that results in a plant transient directly affects the Initiating Events Cornerstone of Reactor Safety, the objective of which is to limit the likelihood of those events that upset plant stability and challenge critical safety functions during shutdown as well as power operations.

As a result of the second transient on January 6, 2005, the transients were determined to meet the criteria of Management Directive 8.3, NRC Incident Investigation Program, for a special inspection. Management Directive 8.3, NRC Incident Investigation Program, recommended a special inspection for situations, including significant operational power reactor events that involved a major deficiency in design or had generic safety implications, involved repetitive failures or events involving safety-related equipment or deficiencies in operations, or involved questions or concerns pertaining to licensee operational performance. Because these transients contained elements that potentially impacted each of these criteria, and directly involved repetitive failures, a Special Inspection was chartered in accordance with the guidance contained in IP 0309, Reactive Inspection Decision Basis for Reactors. The inspection was conducted using IP 93812, Special Inspection, and IP 71153, Event Followup. The charter for the Special Inspection is included as Attachment 2 to this report.

4OA3 Event Follow Up

A.

Description and Chronology of the Events (93812) (Charter Items 1, 2, and 3)

On December 23, 2004, while operating at approximately 100 percent power, both reactor recirculation pumps (RRPs) unexpectedly downshifted from fast to slow speed.

The resulting power and flow reduction resulted in reactor operation in the Immediate Exit region of the Power-to-Flow map. Reactor power was approximately 55 percent rated power after the pumps downshifted. Operators first attempted to exit the Immediate Exit region by opening the associated reactor recirculation flow control valves and increasing flow. Opening the flow control valves resulted in little change in flow because the flow control valves were already approximately 80 percent full open.

This, combined with the low differential pressure developed across these valves on low speed RRPs, created a condition where the valves provided very little throttling; therefore, valve manipulations were ineffective in obtaining sufficient flow.

Subsequently, operators were preparing to insert control rods as an alternate method to respond to this event when intermittent oscillating power range monitor (OPRM) alarms were received. Approximately 9 minutes after the RRPs downshifted, an automatic reactor scram occurred as a result of valid signals from the OPRM system.

On January 6, 2005, while operating at approximately 100 percent power, both RRPs downshifted from fast to slow speed. This again resulted in the reactor operating in the Immediate Exit region of the Power-to-Flow Map. Reactor power subsequent to the transient was approximately 44 percent of rated power. While operators were inserting control rods in accordance with procedures, RRP A tripped from slow speed to off. As a result of the additional equipment problem, the operators elected to shut down the reactor by inserting a manual reactor scram. Following the scram, the motor feedwater pump (MFP) failed to start, which resulted in the operators using the turbine driven feedwater pump to provide water to the reactor. The effects of the addition of cold water coupled with the demand for steam from plant steam loads, including the turbine driven feedwater pump, resulted in the operators approaching the 100/F/hr Technical Specification (TS) cool-down limit. To prevent exceeding the TS limit, operators closed the main steam isolation valves (MSIVs) and used the reactor core isolation cooling (RCIC) system to provide water to the reactor. The reactor was successfully brought to a cold shutdown condition.

The inspectors reviewed plant logs, the sequence of event recorders, plant trend traces, the licensees scram reports and other plant information to establish the following detailed sequence of events.

December 23, 2004 11:45 p.m.

RRPs A and B downshifted from fast to slow speed 11:47 p.m.

RRP A high to low speed transfer annunciator cleared with no operator action 11:48 p.m.

First OPRM alarm, which cleared 11:51 p.m.

Second OPRM alarm, which cleared 11:52 p.m.

Third OPRM alarm, which cleared 11:54 p.m.

Operators start increasing recirculation flow in an unsuccessful attempt to exit the Immediate Exit Region of the power/flow curve.

11:54 p.m.

Fourth OPRM alarm, which did not clear and OPRM initiated an automatic reactor scram 11:55 p.m.

Reactor Level 8 - turbine driven feedwater pump turbines automatically tripped December 24, 2004 00:04 a.m.

Reactor Level 8 cleared 00:05 a.m.

MFP started for level control December 26, 2004 06:00 a.m.

Reactor start-up commenced (Mode 2)07:38 p.m.

Reactor power operation commenced (Mode 1)11:16 p.m.

Plant synchronized to electrical grid January 5, 2005 01:17 a.m.

Plant reached 100 percent power January 6, 2005 01:06 a.m.

RRPs A and B downshifted from fast to slow speed 01:10 a.m.

Commenced insertion of control rod 26-15 01:10 a.m.

RRP A motor generator lockout annunciator; RRP A tripped 01:12 a.m.

Manual reactor scram initiated 01:14 a.m.

Reactor Level 8 - Turbine driven feedwater pumps automatically tripped 01:16 a.m.

Reactor Level 8 cleared 01:20 a.m.

Attempted to start the MFP for level control; the pump failed to start due to breaker springs not being charged 01:22 a.m.

Successfully started RCIC for level control 01:28 a.m.

Successfully restarted turbine driven feedwater pump turbine A for level control 01:33 a.m.

Shutdown RCIC to standby status 01:52 a.m.

Restarted RCIC 01:56 a.m.

Manually tripped the turbine driven feedwater pump turbine to reduce cooldown rate 01:59 a.m.

Shut the inboard MSIVs to reduce cooldown rate 02:10 a.m.

Shut the outboard MSIVs and drain valves to reduce the cooldown rate 03:17 a.m.

Opened the outboard MSIVs January 7, 2005 09:00 a.m.

Cold Shutdown (Mode 4)

B.

Probable Contributing Causes of the Event or Degraded Condition (93812)

Equipment Failures 1.1 December 23, 2004, and January 6, 2005, Unplanned Downshift of the RRPs (Charter Items 4 and 5)

a. Inspection Scope

The licensees root cause investigation for the December 23, 2004 event, was not started prior to restart and was in progress when the January 6, 2005 event occurred.

When the Special Inspection Team arrived on January 7, 2005, the licensee was conducting a root cause investigation for both the December 23, 2004, and January 6, 2005, events. The inspectors monitored the licensees root cause investigations and associated activities as they progressed. The inspectors reviewed the root cause reports from both events to assess the licensees corrective actions. Root cause reports

[condition report] CR 2004-06766, Reactor Recirculation Pump Speed Shift, and CR 2005-00094, Reactor Recirculation Pump Speed Shift, were assigned to the December and January events respectively.

The inspectors performed a detailed review of the licensees in-process root cause analysis, including troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process. The inspectors also performed in-plant observations of selected troubleshooting and diagnostic activities. Associated work orders and logs were reviewed to assess troubleshooting activities. The team also interviewed the day and night shift instrumentation and control (I&C) supervisors overseeing the troubleshooting to understand the events and the documentation. Additionally, the licensees corrective action database was reviewed to evaluate if the licensee had prior opportunities to identify the associated failure mechanism. As part of this inspection, the documents in 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Following the December 23, 2004 event, the licensee evaluated the equipment associated with the RRP logic to support reactor restart and assess the failure mechanism which caused the downshift of both RRPs. The inspectors noted that the licensee had not performed a root cause evaluation to support the restart decision. The decision to restart was based upon the results of an immediate investigation following the event. The inspectors reviewed Work Order 200134452, and the Work In Progress log from the December 23, 2004, event. The inspectors concluded that there was not a separate troubleshooting plan for the December event. The inspectors also found that equipment was not quarantined after the December event to allow investigation and data gathering to the extent necessary to identify the root cause. Because the troubleshooting effort did not reveal any data that indicated the cause of the December 23, 2004, RRP downshift, the licensee decided to restart and monitor the suspect equipment. The inspectors concluded that the scope of the troubleshooting effort for the December event was inappropriately limited by the licensees assumptions that the cause was related to the feedwater interlock and therefore did not consider other causes. The inspectors noted that the equipment which was determined to have caused the downshift was not part of the licensees monitoring activities.

The formal root cause investigation for the December 23, 2004, event was initiated on January 3, 2005, and was in-progress when the January 6, 2005, event occurred. As a result, the troubleshooting for the January event continued under the same work order that was opened for the December event. Following the January event, equipment associated with the RRP downshift was quarantined, which facilitated the troubleshooting used to support the root cause investigation.

The inspectors reviewed the troubleshooting plans and associated fault trees for the January event to verify that the scope of the equipment investigation was broad enough to capture all potential failure mechanisms which could have caused a simultaneous actuation of the downshift circuitry for both RRPs. The formal root cause report CR 05-00094, Reactor Recirculation Pump Speed Shift, used a combination of Fault Tree Analysis and Failure Modes and Effect

Analysis.

The failure mode was determined to be a spurious actuation of a degraded optical isolator card. The root cause of the degradation and subsequent spurious actuation was determined to be a design inadequacy in the surge suppression for a relay located on the output of the optical isolator. The suppression scheme was ineffective because surges beyond the design capacity of the optical isolator were generated when the output relay was de-energized.

Over time, this excessive voltage condition degraded a transistor in the output section of the optical isolator. Due to the degradation, the transistor produced an output voltage when there was no input to the optical isolator. Additionally, the degraded card was shown to be sensitive to variations in temperature, power supply voltage, and power supply noise which may have contributed to the failure mechanism.

The inspectors evaluated and monitored the troubleshooting plans and activities after the January event. As troubleshooting progressed, the plans were expanded by adding additional revisions, which were included as addenda to the report. The inspectors concluded that the troubleshooting plans, including the addenda, provided sufficient data to enable the licensee to proceed through the Fault Tree Analysis and Failure Modes and Effect

Analysis.

Utilizing this data, the licensee was able to identify the failure mechanism and root cause stated above.

A key aspect of the troubleshooting was duplication of the sequence of actions that occurred during the pump downshift event and looking for the characteristic signature of the event. This characteristic signature included both RRPs shifting to slow speed at the same time, and the Recirc A Hi to Lo Speed Transfer annunciator clearing with the Recirc B Hi to Lo Speed Transfer annunciator remaining illuminated. The licensee duplicated this characteristic signature by gradually raising and lowering the voltage across the failed (K154B) relay. Using this same technique on other relays produced different results. Notably, the licensee had used this technique on some candidate relays following the December event. Because the licensees investigation into the December 23, 2004, event was narrowly focused, its troubleshooting efforts did not test for this characteristic signature on the K154A,B or K172A,B relays which individually could cause a simultaneous downshift of the recirculation pumps. The inspectors concluded that this was a weakness in the licensees troubleshooting plan to identify the failure mechanism for the December event.

After the licensee determined that all available data had been gathered locally, the degraded optical isolator and relay were released from quarantine for replacement.

Next, the components were sent to a local laboratory for analysis following a test plan developed by the root cause investigation team. The licensee also had a representative at the laboratory to observe testing. Upon completion of the laboratory testing, the degraded card was sent to General Electric Nuclear Energy for final destructive testing.

Laboratory reports confirmed that the card had experienced over-voltage conditions and concluded that the surge suppression on the output of the card was inadequate. The inspectors concluded that the root cause investigation appeared to be adequate and thorough.

The inspectors reviewed the licensees evaluation and actions associated with the extent of condition. The original extent of condition for the degraded optical isolator included all types of optical isolators at the site. Once the failure mode and root cause were determined, the licensee narrowed the extent of condition to optical isolators with 125VDC outputs, resistor/capacitor (RC) suppression, and used in applications other than those specified for the original design. The team agreed that this decision was reasonable, based on the identified failure mechanism.

The licensees corrective actions were based on the optical isolator failure mode and the design inadequacy established as the root cause. Corrective actions included replacement of all optical isolator cards identified during the extent of condition evaluation. The corrective actions also included a modification of the surge suppression scheme on relays associated with these cards. The modification replaced the current RC suppression scheme with a reverse biased diode scheme. The inspectors concluded that the corrective actions appeared to be reasonable, consistent with industry experience, and should eliminate the potential vulnerability for a repeat occurrence of the same failure mechanism.

1.2 January 6, 2005 Spurious trip of RRP A (Charter Items 4 and 5)

a. Inspection Scope

The inspectors performed a detailed review of the licensees formal root cause analysis, including troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process.

The inspectors performed in-plant observations of selected troubleshooting and diagnostic activities. Associated work orders and logs were reviewed to evaluate the adequacy of troubleshooting activities. Additionally, the inspectors reviewed the licensees corrective action database to determine if the licensee had prior opportunities to identify the associated failure mechanism. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction A self-revealed finding of very low significance (Green) related to the Initiating Events Cornerstone and a Non-Cited Violation (NCV) of 10 CFR 50.65(a)(3) were identified by the inspectors. The finding involved the transient initiated as a result of a trip of RRP A low frequency motor generator (LFMG) and subsequent manual scram of the reactor on January 6, 2005. The NCV was associated with a failure to incorporate industry operating experience into preventative maintenance activities that would have prevented a failure of RRP A.

Description The RRP A trip investigation was documented in root cause report CR 05-00094. The inspectors reviewed the troubleshooting plans and fault trees to verify that the equipment investigation scope was broad enough to capture all potential failure mechanisms that would cause a trip of RRP A. The licensees investigation also used a combination of Fault Tree Analysis and Failure Modes and Effect

Analysis.

The RRPs are powered in low speed by a LFMG. The licensee determined that the voltage regulator on the LFMG failed, causing a momentary loss of voltage. When voltage recovered, the inrush current to the RRP motor tripped the LFMG on overcurrent.

The inspectors evaluated the troubleshooting plan, Work Order 200135641, and associated Work in Progress log. The inspectors concluded that the licensee had established troubleshooting activities which contained the actions needed to successfully complete the Fault Tree Analysis that was the basis for the root cause evaluation.

The licensee identified the failure mechanism during its review of computer data associated with the event. The data revealed that there was a loss of voltage from the LFMG for approximately 7 seconds. The licensee determined that when voltage recovered, there was an inrush of current to the RRP motor and the LFMG tripped on overcurrent. While the voltage regulator became the primary focus of investigation, the licensees process appropriately included other possible failure mechanisms, and troubleshooting appropriately disproved these alternate possibilities. Additionally, the licensee sent the suspect voltage regulator to a laboratory for evaluation. The laboratory found a faulty operational amplifier, which would explain the momentary loss of voltage control.

During the investigation, the licensee reviewed the performance of the LFMG during previous operations. The licensee determined that voltage regulator degradation was readily apparent from traces of generator voltage, which were captured by the plant integrated computer system. The inspectors reviewed this information and agreed that, had this performance data been reviewed, it would have been reasonable to conclude that the voltage regulator was not operating properly. The inspectors reviewed the licensees trending methodologies that required the review of post-event data. The inspectors determined that the licensee lacked requirements to trend system response during operation, particularly following transients and concluded that this was a weakness. Supporting this conclusion was the fact that the licensee determined that the root cause was a failure to adequately trend and correct voltage regulator degradation prior to it failing.

The inspectors reviewed the licensees data, conclusions, and root cause analysis for this event and found that the licensee had determined that prior event history, specifically CR 01-2145, identified that a previous voltage regulator installed in this location demonstrated a failure emerging over time. Additionally, this CR concluded that an indicator of degradation should be able to be established to assure that a voltage regulator is replaced before the degradation proceeds to failure, and that the voltage regulator should be replaced when output voltage shows fluctuations or when frequent adjustments are required. Notably, data from May 23 and December 23, 2004 events indicated regulator degradation. Had the data been trended, it would have alerted the licensee to the impending failure of the LFMG voltage regulator.

The licensee reviewed other voltage regulation equipment within the plant in order to establish an extent of condition. With the exception of the RRP LFMGs, it was determined that other voltage regulation equipment was either monitored on-line or periodic maintenance tasks were scheduled to replace components at appropriate frequencies.

Analysis The inspectors determined that the failure to trend data when prior information, specifically the recommended actions from CR 01-2145, indicated this equipment warranted trending was a performance deficiency.

The objective of the Initiating Events Cornerstone was to limit the likelihood of those events that upset plant stability and challenge critical safety functions during shutdown and power operations. The inspectors determined the finding was more than minor, because the event caused an actual upset in plant stability and operation resulting in a plant transient, thus directly affecting the objective for the Initiating Events Cornerstone.

Additionally, the trip affected the equipment performance attributes of availability and reliability of the Initiating Events Cornerstone of Reactor Safety, further supporting that the issue was more than minor. Also, the inspectors determined that the issue affected the Initiating Events Cornerstone, because it had an actual, although small, risk impact on the licensees transient event initiating frequency.

In order to determine the risk/safety significance of the event, the inspectors performed a Phase 1 SDP in accordance with Inspection Manual Chapter (IMC) 0609. The inspectors concluded that the event increased the likelihood of a initiating event and, therefore, impacted the Initiating Events Cornerstone. Specifically, the event was a transient initiator contributor, which resulted in a reactor trip. Because the finding did not result in exceeding the TS limit for identified reactor coolant system (RCS) leakage and did not affect other mitigation systems; the finding did not contribute to both the likelihood of a reactor trip AND the likelihood that mitigation equipment or functions will not be available; and the finding did not increase the likelihood of a fire or internal/external flood, the finding screened as Green.

Enforcement Title 10 CFR 50.65.A(3) states that Performance and condition monitoring activities and associated goals and preventive maintenance activities shall be evaluated at least every refueling cycle provided the interval between evaluations does not exceed 24 months.

The evaluations shall take into account, where practical, industry-wide operating experience. Adjustments shall be made where necessary to ensure that the objective of preventing failures of structures, systems, and components through maintenance is appropriately balanced against the objective of minimizing unavailability of structures, systems, and components due to monitoring or preventive maintenance. Contrary to this requirement, the licensee failed to incorporate operating experience when it was practical to do so. Specifically, the recommendations from CR 01-2145, to replace a voltage regulator when output voltage shows fluctuations or when frequent adjustments are required, could have prevented the ultimate failure of the voltage regulator and resulting trip of the RRP A. Additionally, the RRP A LFMG is a component covered under this rule. Because this violation was of very low safety significance (Green) and it was entered into the licensees corrective action program, this violation is being treated as a NCV consistent with Section VI.A of the NRC Enforcement Policy. (NCV 05000440/2005005-01). The licensee entered this issue into their corrective action program as CR 05-00094, Reactor Recirculation Pump Speed Shift.

Completed corrective actions include a formal root cause and evaluation of the conditions contributing to the LFMG voltage controller failure. The licensee replaced the voltage regulator on the A LFMG and reviewed performance data from the B LFMG and determined that the B LFMG voltage regulator did not require replacement at this time. An additional corrective action was initiated to establish a method or process to review equipment performance data following any major plant transient. The intent was to expand the current formal trending program to identify equipment degradation and to reduce the potential vulnerability for a repeat occurrence of a similar failure. Other actions had been created to address root and contributing causes identified as a result of the formal evaluation.

1.3 January 6, 2005 Failure of the MFP to Start (Charter Items 4 and 5)

(1) Licensees Root Cause Determination and Failure Analysis

a. Inspection Scope

The inspectors performed a detailed review of the licensees root cause analysis and failure analysis (CR 50-00095) and related documentation, including operating experience reports, vendor breaker refurbishment reports, troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process. The inspectors also interviewed technicians and engineers, and performed in-plant observations of selected troubleshooting and diagnostic activities. Associated work orders, logs, breaker historical data, maintenance procedures, and vendor manuals were reviewed to assess the adequacy of troubleshooting activities. Additionally, the licensees corrective action database was reviewed to assess if the licensee had prior opportunities to identify the failure mechanism. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction The inspectors identified an unresolved item (URI) requiring further inspection. The URI was related to numerous breaker failure modes, or contributors, that had been identified by the licensee during their root cause analysis, but were not evaluated sufficiently to support full qualification of safety-related and safety significant breakers.

Description Following the January 6, 2005 reactor scram, the licensee unsuccessfully attempted to manually start the MFP from the control room. Investigation by operations personnel identified that the closing spring for the MFP breaker, in cubicle L 1007, was not charged. Following the event, the licensee initiated a root cause investigation to determine the associated failure mechanism. As part of the licensees evaluation, the affected breaker was sent to their vendor for analysis. The failed breaker was replaced by a recently refurbished breaker. On January, 18, 2005, while performing post-maintenance testing on the replacement breaker, the closing springs failure to charge. As a result of the second failure, the licensee established a more comprehensive plan to investigate the root cause for the breaker failures in Cubicle L1007.

The breaker installed in Cubicle L1007 was a 15HK 1000 (15HK) type breaker manufactured by I-T-E Gould, which is now Asea Brown-Boveri (ABB). The breaker was powered by 13.8kV Bus L10, which was a non-safety-related bus; however, this bus supplies power to the Class 1E 4.16kV buses and other risk significant loads, including the MFP. Closing springs in the breaker are used as the motive force to close the breaker. These springs are compressed (charged) using an electric motor (spring charging motor). The closing springs can also be charged via a manual ratcheting mechanism.

Assessment The inspectors reviewed the licensees root cause analysis and troubleshooting efforts for the failed MFP breaker. After the January 6, 2005, MFP breaker failed to close, the subject breaker was sent to the vendor for analysis. Following analysis and refurbishment, the vendor concluded that the failure was caused by an improper gap between the timing lever and the control device limit switch roller. The interaction of these two components satisfies the necessary interlocks and circuitry to recharge the breakers closing springs subsequent to opening the breaker, thereby placing the breaker in a standby condition for the next demand. The inspectors reviewed the licensees investigation results and concluded, after review of the January 18, 2005, event, that the initial root cause investigation was ineffective because it failed to identify the cause. The inspectors concluded that the cause identified during this investigation was one of the contributing causes, an observation supported by the licensees final analysis.

Subsequent to the January 18, 2005, failure of the breaker in Cubicle L1007, the licensee established a more comprehensive plan to investigate the failure mechanism and identify the root cause. Improved licensee actions included increasing the number of individuals on the root cause team by adding experienced individuals from the industry and the vendor. The root cause investigation team was primarily comprised of engineers, operators, and maintenance individuals from Perry, Davis Besse, Beaver Valley, ABB, [Electric Power Research Institute] EPRI, and contract personnel. The root cause team used both Failure Mode Analysis and Event and Causal Factor methodologies to identify areas for further investigation, failure modes, and the root cause. The inspectors concluded that the root cause methodology used appeared appropriate for identifying the root cause and contributing causes.

After the second failure of the MFP breaker, the licensee conducted extensive tests of the breaker in Cubicle L1007 using high speed video. This allowed them to observe actual breaker operating characteristics in slow motion. Additionally, the licensee evaluated the switchgear cubicle dimensions and critical measurements using a three dimensional laser technique. To assist in identifying any potential disparities, the licensee measured the critical surfaces inside two 13.8kV switchgear cubicles and performed a comparative analysis of any significant differences. Using the data gathered, the licensee attributed the failure of the breaker springs to charge and the breakers to close on January 6 and 18, 2005, to a design deficiency within the breaker cubicle. The licensee determined that the rejection (interference) plate in the Cubicle L1007 was located higher, to the left, and deeper into the cubicle than other 13.8kV cubicles. This created a condition where the breakers spring charging link arm was repeatedly striking the floor mounted interface plate, causing wear on the interface plate installed in the suspect cubicle. Additionally, the licensee concluded that five prior failures of various breakers installed in Cubicle L1007, which occurred between 1987 and 1992, were most likely attributed to this newly identified root cause.

The licensee documented the following causal factors and the associated contributing causes in their root cause report:

C That routine operation of the suspect 15kV breaker resulted in the closing spring charging link arm hitting the cubicle interference plate, which was welded to the cubicle floor. The licensee concluded that this action removed some kinetic energy from the operating mechanism which resulted in the breaker timing cam failing to rotate sufficiently, a condition which failed to actuate the control device.

Another contributing cause was determined to be an ineffective breaker management program, which contained less than adequate oversight of generic breaker maintenance and refurbishment issues. An additional contributing cause was the failure to apply internal breaker operating experience.

C Deficient trending and analysis of historical breaker cubicle performance had not resulted in the identification and correction of breaker performance issues. The licensee indicated that the current corrective action program requirements to document failures combined with the implementation of the Maintenance Rule Program in 1997 provided trending abilities that had not occurred prior.

C The breaker vendor had failed to update the 15kV vendor manual to include the need to periodically monitor the gap between the timing lever and the control device limit switch roller. The inspectors noted that this information was updated for 5kV breakers but the licensee failed to question the applicability to the 15kV breakers. The licensee also identified that the vendor oversight program had not incorporated up-to-date information, which would ensure breaker reliability.

C The lack of site specific guidance to prevent unauthorized modifications. This was important because there was evidence of undocumented modifications to the control relay limit switch arm and mounting bracket holes.

C That less than adequate oversight of vendor refurbishment performance resulted in defective parts not being replaced. This condition contributed to degraded breaker performance.

The inspectors reviewed the licensees root cause analysis methodology, cause determination, and completed and proposed corrective actions. The licensee documented these items in CR 05-00095, Motor Feed Pump Failure to Start. The inspectors concluded that there was reasonable assurance that the licensee identified the root cause. However, the inspectors believed that additional effort was needed to ensure adequate resolution of all identified contributing causes. For example, Section 3.3, of CR 05-00095 stated that since implementation of the 10-year breaker overhaul frequency, the reliability of the 15kV breakers had improved dramatically. The inspectors agreed that breakers, which were refurbished in the 1994 to 1996 time frame, had not experienced any failure of the operating mechanisms until January 6, 2005.

Because the breaker to cubicle interference and the location of the interference plate were unchanged after the breakers were refurbished in the 1992 to 1994 time frame, the inspectors concluded that the licensee had not adequately ruled out the potential that other time-related failure mechanisms, such as hardened grease, were adequately assessed as possible contributing causes of the failure.

In order to assess the effectiveness of licensees corrective actions and causal analysis, the inspectors reviewed the troubleshooting plans used to evaluate the failure mechanism of the MFP breaker. The initial scope of the licensees investigation focused on the MFP failure to start on January 6, 2005. Subsequently, the licensee expanded the scope because the results of the vendors analysis identified issues that were potentially applicable to all 15kV, 5kV, and 480V circuit breakers. The licensee used Procedure NOP-ER-3000-1 to develop a detailed plan to investigate the cause of the MFP breaker failure. The plan was used as a living document to gather and evaluate data as information was obtained and potential causes proven or disproved. The plan was modified and updated until the main cause and the contributing causes were identified.

During its investigation into the January 18, 2005, breaker failure, the licensee revised its troubleshooting plan as new information was received. Various revisions of the troubleshooting plan were used to investigate both the MFP breaker failure to charge on January 6, 2005, and the actual failure mechanism of the related auxiliary contacts. The plans included the title of the issue, the issue owner, an associated problem statement, possible causes, data gathered during the investigation, an evaluation of causes, and a significance assessment. The inspectors found that the individuals implementing the plan included licensee and vendor representatives with extensive knowledge of breaker operations. The inspectors concluded that the licensee implemented the plans using a logical process.

The inspectors reviewed licensees immediate corrective actions following the two events in January where the MFP breaker failed to close. The inspectors reviewed the licensees corrective actions to assess the adequacy and timeliness of the initial response to the breaker failures. The inspectors concluded that the corrective actions for the January 6, 2005, failure were not effective in identifying the cause; however, corrective actions for the second failure were more thorough. Actions taken after the January 6th breaker failure to close included: the operator racked out and in the MFP breaker twice and the closing springs did not charge; CR 05-00095 was generated to document that the MFP breaker closing springs did not charge; The MFP breaker was shipped offsite to the vendor for failure analysis; in-service 13.8kV and 4.16kV circuit breakers were verified to have the closing springs charged; and Instruction GEI-0135 was placed on hold to prevent inadvertent use.

The inspectors were concerned that one of the immediate corrective actions for the January 6, 2005, event appeared inappropriate; specifically, that the operators re-racked the MFP breaker in an attempt to charge the springs. The inspectors concluded that this action implies two possibilities: first, that the practice of re-racking a breaker to enable it to charge was considered acceptable and may indicate that a knowledge-based operator workaround existed associated with breaker deficiencies; and second, that there may have been undocumented problems with this type of failure prior to this event.

Additionally, the safety aspects of re-racking a breaker that may have failed may need consideration. The inspectors believe that this issue warrants further investigation and review by the licensee. Corrective actions for the January 18, 2005, failure of the breaker to close included generation of CR 05-00460 to evaluate the issue; removal of the interference plate from the floor of the MFP electrical supply breaker Cubicle L1007; and inspection of interference plates for other 15kV breakers that possessed an auto-close function for potential signs of impact/wear. Interference plates observed to exhibit wear were removed from breaker cubicles L1005, L1006, L1007, and L1009; performance of extent of condition reviews for similar potential failure mechanisms.

The inspectors determined that the licensee had issued numerous condition reports to address the root cause investigation findings. The corrective actions to address the findings were listed in the table of corrective actions in CR 05-00095. Additionally, the licensee developed an Effectiveness Review Plan and documented the plan in Section 7.0 of CR 05-00095. The effectiveness review plan for this CR included a review of the corrective actions to address the findings and to prevent recurrence. The inspectors concluded that the licensees proposed corrective actions would be effective in correcting the identified failure mechanism and preventing recurrence of the issue.

The inspectors also evaluated licensees efforts to determine the extent of condition for the breaker failures. The inspectors concluded that the licensees extent of condition reviews for the January 6, 2005, failure of the MFP breaker was narrowly focused. The inspectors noted that the licensees scope for determining the extent of condition changed as more facts were gathered. This observation was supported by the fact that on January 25, 2005, subsequent to the second failure of the MFP breaker, the licensee concluded that all 15HK breakers rated at 1200 or 2000 amps were suspect due to interference plate design. Additionally, the licensees root cause team selected 14 of 72 safety-related ABB 5kV breakers that were past the 10-year refurbishment due date for inspection and testing as part of the extent of condition sample. As the investigation progressed, and after a review of past preventative maintenance activities and procedure revisions, five additional non-safety-related, but important to safety breakers were added to the sample.

The licensee performed an extensive review of industry operating experience (OE)related to 5kV and 15kV ABB breaker problems. Attachment 10 of CR 05-00095 listed the results of this review. The inspectors reviewed the root cause and the extent of condition efforts and noted that no extent of condition reviews were specified for the identified failure contributors and for failure modes that could not be refuted in the root cause report. Potential failure modes not reviewed included; the control device gap setting; the hardened or improper breaker lubrication; the indentation on the timing lever found in the breaker from the January 6, 2005, event; the impact between the operating mechanism and the breaker cubicle found in Cubicle L1007; the physical switchgear cubicle interference with the breaker; the elongated mounting holes on the control device holding plate, and the breakers that had not been refurbished after 10 years in-service. Also, taking these findings collectively may require additional investigations and corrective actions to address potential underlying issues. Because there were numerous failure modes or contributors that the licensee had not fully evaluated, the inspectors concluded that the evaluation which the licensee performed to support operability of these breakers appeared incomplete. These facts caused the inspectors to question the operability evaluation performed on these breakers; therefore, this is an unresolved item warranting further inspection (URI 05000440/2005005-02).

The inspectors sampled the effectiveness of the licensees extent of condition evaluation by verifying that known operating experience had been evaluated. Specifically, the inspectors asked the licensee to verify if the plant had evaluated the recommendations of Significant Operating Event Report (SOER) 98-02, Circuit Breaker Reliability, dated September 22, 1998. This SOER highlighted operating experience where six circuit breaker failures in the nuclear industry had issues similar to some of the breaker failure mechanisms noted with the licensees MFP breaker and other recently refurbished breakers. The licensee was unable to provide the related information and indicated that a corrective action item would be generated to re-evaluate the implementation of the recommendations in the SOER.

The inspector also noted that the licensee did not trend items identified during breaker refurbishment to identify potential common cause or extent of condition issues. The inspectors noted that numerous problems were identified with breakers that were refurbished at the vendor, then returned to the licensee. Examples were documented in CR 05-00654, Breaker L1007 Truck Bent at Spring Point; CR 05-00618, 15kV Breaker Refurbishment (L1105 - CWP A); CR 05-00396, Refurbished Breaker Improperly Reassembled; CR 05-00522, Refurbished Breaker Aux Switches Not Lubricated; and CR 05-00707, Refurbished Breaker Not Acceptable for Use.

The inspectors reviewed historical information related to breaker problems to determine if there were prior opportunities for the licensee to trend, identify, and correct breaker-related problems; and include related operating experience in their root cause analysis. The inspectors identified the following information, which was available to the licensee:

C Inspection Report 50-440/94006(DRS) documented a corrective action violation for failure to take adequate corrective actions, and for inadequate maintenance and engineering efforts in resolving an extended ABB circuit breaker grease hardening issue.

C The inspectors determined that the ABB breakers in the plant were not being maintained as recommended by the vendor. The 1989 vendor manuals stated, The circuit breaker requires no lubrication during its service life. In 1991, ABB revised the breaker manuals to recommended a mandatory 10-year refurbishment which included disassembly, inspection, cleaning, and lubrication.

At that time, Perry took no action to revise the procedures and refurbish the breakers even though the age of some ABB breakers at Perry were over 10 years in 1994.

C ABB 13.8kV breakers at Perry had numerous failures since 1985. The lubrication problem was first noticed in 1987. Condition Report 92-011 was issued on January 29, 1992, to address three prior failures of the Unit 1, 13.8kV ABB MFP breaker. The CR noted that the breaker indicated a sticky lube condition and that this problem was widespread in ABB breakers throughout the plant. The breaker was refurbished on September 8, 1992. The vendor report stated that due to severe hardening of lubricant in the breaker, some linkages had to be removed with a hammer and drift pin.

C A licensee internal memorandum from D. Strach to L. Teichman, dated February 11, 1992, stated that there have been at least 15 documented failures of 13.8kV breakers to close, involving 9 different breakers over the last 6 years.

In each case, the problem was due to operating mechanism problems that prevented the breaker closing springs from being charged/compressed. There was supporting information indicating that this problem also occurred prior to 1985. The latest occurrence of this problem involved three separate failures of the breaker installed in Cubicle L1007 and was found to exist when the breaker was racked full in; notably, the problem did not occur when the breaker was in the test position. In 1982, the vendor indicated that the operating mechanism may be sticking and the lubrication, Anderol 757, should be renewed by overhaul of the breaker. However, the licensee did not overhaul the breaker and in an attempt to soften the grease in the operating mechanism, grease was applied as close to the pivoting areas as could be reached, then heated with a heat gun in the hope that it would flow to the affected area. Other techniques were also attempted to loosen the grease with various success. In addition, the licensees engineering staff submitted Bright Idea #1147 proposing a design change to the 13.8kV breaker operating circuits to modify the breaker (green) indicating lights on the switchgear and control room panels. This modification would have modified the green light to indicate when the breaker was open and ready to close (i.e., that the closing springs were charged); however, this design change was not implemented. The memo concluded that there were indications of a lubricating problem with the breaker operating mechanisms. The inspectors found related issues documented in CRs89-010, 91-0174,92-011, 92-244,92-452, 94-306,94-901, 99-2013, 00-1483 and in various work orders.

C In the response to the corrective action violation from Inspection Report 50-440/94006(DRS), the licensee concluded that the basis for failing to incorporate known in-house operating experience was a direct result of the untimely completion of related corrective actions. The licensee stated that CR-92-0244 was issued in late 1992 and repetitive tasks were generated to ensure the performance of the 10-year breaker refurbishment required by ABB; however, no corrective actions were taken to ensure that these repetitive tasks were planned, scheduled, and completed. As a result of the observed hardened grease on the ABB breakers, the licensee performed an operability evaluation and concluded that although early stages of hardened lubrication on ABB breakers had been identified, the breakers were operable in their installed condition. The licensee also stated that schedules and priorities had been established to ensure that the ABB breakers would be inspected and refurbished as required to ensure a high degree of operability. The licensee identified critical breakers that were required to cycle and operate automatically to perform their intended design function and placed emphasis on inspecting and refurbishing these critical breakers during refueling outage RFO4 (February to August 1994).

Additionally, the licensee planned to perform an extent of condition review to ensure that all applicable preventative maintenance and repetitive tasks have been identified and implemented. Subsequently, and in response to the violation that was based on past historical data related to ABB breaker problems at Perry, all breakers were refurbished between 1994 and 1996.

Based upon the above information, the inspectors concluded that there may have been prior opportunities to trend, identify and correct age-related problems with ABB breakers and breaker cubicles at Perry (See URI 05000440/2005005-02 above).

(2) Refurbishment of Safety-Related and Important-to-Safety ABB Breakers

a. Inspection Scope

The inspectors performed a detailed review of the licensees repair and refurbishment practices for safety-related and risk significant non-safety-related circuit breakers. The inspectors reviewed work orders, logs, breaker historical data, maintenance procedures, and vendor manuals to assess the adequacy of maintenance practices. The inspectors also interviewed technicians and engineers, and performed in-plant observations of selected troubleshooting and diagnostic activities. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction The inspectors identified a finding having very low safety significance (Green) relating to the Mitigating Systems Cornerstone and an associated NCV of TS for inadequate procedures associated with safety-related breaker maintenance procedures. The inspectors determined that maintenance procedures used to ensure that safety-related breakers are being overhauled in a timely manner were inappropriate. Specifically, the maintenance procedures did not contain guidance to refurbish breakers within vendor-specified time frames or provide reasonable alternative preventative maintenance practices to ensure that safety-related breakers remained operable.

Description The inspectors identified that the licensee was applying a 15 percent grace period to the 10 year maximum vendor specified refurbishment periodicity for safety and non-safety-related circuit breakers, and that this grace period was applied without a documented engineering basis or evaluation. The inspectors also noted that a similar corrective action violation was cited in 1994 in Inspection Report 50-440/94006(DRS).

Assessment The inspectors reviewed the licensee breaker maintenance records. The inspectors determined that the licensee had not refurbished nine safety-related 480V ABB breakers; seven safety-related 4.16kV ABB breakers; and eight 13.8kV high safety significant non-safety-related ABB circuit breakers within the vendor specified 10-year maximum overhaul periodicity. The inspectors noted that the licensee was applying a 15 percent grace period to the vendor (ABB MS 3.2.1.9-1) recommended 10-year maximum breaker refurbishment period without an engineering basis or evaluation. The licensee indicated that they were utilizing Electric Power Research Institute (EPRI) related guidance, which permits a 12-year overhaul frequency. The inspectors compared the EPRI guidance against the vendor-recommended maintenance requirements and found that the licensee was not performing some overlapping activities. As a result, the inspectors requested the basis for not performing all of the recommended maintenance activities. The licensee was unable to produce an engineering evaluation that allowed the use of the EPRI guidance versus the vendor guidance. Additionally, the inspectors found that the licensee had failed to update their in-use guidance when operating experience or new vendor information was issued.

Because the licensee was unable to produce documentation which qualified the practice of extending refurbishment intervals, the inspectors were concerned about the operability of the safety-related and safety significant breakers that had not been overhauled within 10 years. The licensee initiated two operability determinations (ODs)and CRs (CR 05-00230 and CR 05-00283) on January 11, 2005, to evaluate the operability of susceptible breakers. The licensee concluded that the 15 percent grace period was acceptable for these breakers. The licensee informed the inspectors that based on the conclusion of CR 05-00230, no change to the existing breaker refurbishment schedule would be made, except for the A emergency service water (ESW) pump breaker. A corrective action was written, 05-00095-007, that required the breaker in A ESW pump cubicle to be replaced with a refurbished breaker by the end of RFO10 (spring 2005). The ESW pump breaker was designated for replacement because of a high number of duty cycles and because it was well into the 15 percent grace period that the licensee assumed allowable for refurbishment.

The inspectors reviewed the related operability determinations and did not agree with the basis for the conclusions made in CR 05-00230. The inspectors based their assessment on several facts including:

C Item 5.1 in CR 05-00230 references an EPRI/Nuclear Maintenance Assistance Center (NMAC) document that provides the basis for an 8 to 12 year refurbishment frequency; however, the licensee could not demonstrate that they complied with all the EPRI-recommended maintenance activities, in comparison Table 4.4-1, Industry and Maintenance Recommendations, necessary to support the 8 to 12 year frequency.

C Item 5.2 in CR 05-00230 stated that the breakers are barely beyond the vendor-recommended frequency of 10 years, with the longest breaker being 10 years and 9 months. Item 5.2 further stated that the 10-year vendor requirement was based on breakers manufactured prior to 1975 with petroleum-based grease and that in 1975 ABB changed to a synthetic-based grease, Anderol 757, which does not dry out as fast and extends the useful life of the lubrication. The inspectors did not agree with this conclusion because the licensee experienced grease hardening in ABB breakers that contained the Anderol 757 grease and this condition occurred in early to mid-1990s, a period of time after the vendor had commenced using synthetic grease (IR 50-440/94006(DRS)). Also, the 10-year maximum requirement was established by ABB following the 10 CFR Part 21 report issued by D.C. Cook on March 3, 1989. This report was issued after the failure of 2 ABB 4.16kV breakers to close on demand due to hardened grease in their operating mechanism. Further supporting that grease hardening can occur in less than ten years was a condition report issued for the Perry, 4.16kV, C residual heat removal (RHR) pump breaker. This condition report identified various anomalies during the process of disassembling the breaker and stated that, the lubricant within the operating mechanism appears to be hardened. Notably, this breaker, Serial Number 51958B-103036, had an overhaul period that was from May 5, 1994, through March 2004, a period slightly less than 10 years. Based upon the May 1994 overhaul date, the inspectors noted that synthetic grease would have been used. This provided further evidence that synthetic grease can degrade in less than 10 years.

C The NRC resident inspectors also raised concerns with CR 05-00230 (IR 05000440/2005002). On January 18, 2005, the licensee identified that the OD prepared for CR-05-00230 and 05-00283 exhibited shortfalls with regards to technical rigor, documentation and procedural compliance and issued CR 05-00450 to address these concerns. The licensee also identified that the OD did not have tracking of corrective actions. The inspectors determined that the operability questions related to the issues above were unresolved and documented this issue in Section 1.3(1) above.

C The inspectors also determined that the licensees corrective actions for the violation identified during the 1994 inspection discussed above had not been fully effective in ensuring that the breakers were refurbished within the ABB specified 10-year maximum. Specifically, the licensee noted a varying degree of grease degradation during the initial inspection of the January 6, 2005 failed MFP breaker, Serial Number SN 51951C-11106. Notably, industry operating experience for related ABB breaker historical data showed that the lubrication in the operating mechanism tends to harden if not refurbished within 10 years, and that this condition can cause sluggish breaker operation.

Analysis The inspectors concluded that the licensee failed to evaluate industry and vendor recommended changes and incorporate the changes into their breaker maintenance procedures. Additionally, the inspectors found that the evidence supporting the vendor recommended 10-year maximum refurbishment periodicity was conclusive and that the licensee did not have a proper evaluation to support extending the maintenance frequency. Further, the inspectors did not identify any special or additional preventative maintenance requirements that would provide assurance that the breakers would remain operable beyond the vendor-recommended ten year maximum overhaul periodicity. The inspectors concluded that each of these items were performance deficiencies.

The objective of the Mitigating System Cornerstone of Reactor Safety is to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. The inspectors determined that this issue affected the procedure quality attribute for maintenance procedures of the Mitigating System Cornerstone of Reactor Safety. Specifically, the issue was more than minor because the failure to incorporate the vendor required maintenance frequency or fully incorporate EPRI maintenance recommendations for extending the service interval into maintenance procedures for safety related breakers, if left uncorrected, affected the availability, reliability, and capability of mitigating systems that respond to initiating events to prevent undesirable consequences because the reliability of safety-related breakers refurbished using the deficient procedures can not be predicted.

The inspectors performed a Phase 1 screening of the issue using the NRCs significance determination process (SDP) and determined that the deficiency did not result in any loss of function; that the finding was not risk significant due to a seismic, flooding, or severe weather initiating event; and that other plant-specific analysis that identify core damage scenarios of concern were not impacted; therefore, the issue screened as Green.

Enforcement Technical Specification Section 5.4, requires that procedures be developed and implemented in accordance with Regulatory Guide 1.33. Regulatory Guide 1.33, Section 9, states that, Maintenance that can affect the performance of safety-related equipment should be properly preplanned and performed in accordance with written procedures, documented instructions, or drawings appropriate to the circumstances.

Contrary to this requirement, maintenance procedures for overhauling safety-related breakers were not appropriate to the circumstance, because they did not contain guidance to refurbish breakers within vendor-specified time frames or provide reasonable alternative preventative maintenance practices to ensure that safety-related breakers remained operable.

Because this violation was of very low safety significance (Green) and was entered into the licensees corrective action program (CR 05-0187. CR 05-00230, CR 05-00253, CR 05-00274, CR 05-00283, CR 05-00295, CR 05-00359, CR 05-00459), this violation is considered a Non-Cited Violation, consistent with Section VI.A.1 of the NRC Enforcement Policy (NCV 05000440/2005005-03).

(3) Breaker Control Device Relay Gap Setting

a. Inspection Scope

The inspectors reviewed selected ABB installation and maintenance instructions, maintenance and surveillance instructions for medium-voltage power circuit breakers, technical bulletins, 10 CFR Part 21 reports, and vendor supplemental guidance issued for ABB breakers. The inspectors also performed a detailed review of the licensees root cause analysis, including troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process. In conjunction with this review, the inspectors performed a detailed review of the licensees repair and refurbishment practices for safety-related and risk significant non-safety-related circuit breakers. The inspectors reviewed work orders, logs, breaker historical data, maintenance procedures, and vendor manuals to assess the adequacy of maintenance practices. The inspectors also interviewed technicians and engineers, and performed in-plant observations of selected troubleshooting and diagnostic activities. As part of this inspection, the documents in 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction The inspectors identified a finding having very low safety significance (Green) relating to the Mitigating Systems Cornerstone and an associated NCV of TS for inadequate procedures associated with safety-related breaker maintenance procedures. The inspectors determined that maintenance procedures for overhauling safety-related breakers were inappropriate, because they did not contain guidance to measure and monitor critical measurements identified by the vendor. Specifically, the inspectors noted that the vendor-specified control device gap setting between the breakers control device lever and the limit switch crank on 4.16kV ABB safety-related breakers was not correctly incorporated into the maintenance procedures being used for maintenance activities on 4.16kV safety-related breakers.

Description Following the January 6, 2005, MFP breaker failure, the vendor reported that the gap clearance between the control device arm and the timing lever was found to be outside of the manufacturers tolerance for the initial gap setting on refurbished breakers. The licensee believed that this was the primary failure mechanism prior to January 18, 2005, when a newly refurbished and different breaker installed in Cubicle L1007 failed post-maintenance testing. After the second MFP breaker failure the licensees root cause team concluded that the gap setting may have been a contributing cause for the breaker failure on January 6th, but not the January 18th, because the breaker that failed on the 18th had been recently refurbished. Because the licensees vendor considered this to be a potentially important contributor, and because that clearance measurements for this gap appeared to be critical, the inspectors reviewed the issue.

During the inspectors review of the licensees safety-related circuit breaker maintenance procedure and associated vendor information, the inspectors noted that an ABB ERRATA sheet was issued on October 9, 1998, to change the gap setting between the control device lever and the limit switch crank. This ERRATA changed the allowable range from 0.060" to 0.090" to a new range of 0.010" to 0.090. On May 31, 2000, ABB issued supplemental guidance which further modified the specified breaker control device gap and adjustment requirements for 5HK 1200, 2000 and 3000 amp breakers.

This guidance modified the requirement to have the control device lever and the limit switch crank gap measured in two positions, one with the closing springs discharged (bottom of travel) and one with the closing springs charged (top of travel). The gap specified for the 1200, 2000 amp, Model 5HK breakers with the springs charged was between 0.010" and 0.090" and for springs discharged between 0.200" and 0.275.

Similarly, the vendor modified the requirements for the 3000 amp, Model 5HK breakers.

The gap specifications for this model breaker was from 1/64" to 1/32" with the springs charged, and from 1/64" to 1/16" with the springs discharged. The supplemental guide also provided instructions on adjusting the gap; however, the inspectors determined that the vendor-specified control device gap setting between the control device lever and the limit switch crank on 4.16kV safety-related breakers with the closing springs charged (0.010" to 0.090") was not incorporated into [Perry Nuclear Power Plant] PNPP Operations Manual Instructions GEI-0135, ABB Power Circuit Breakers 5KV Types 5HK 250 and 5HK 350 Maintenance, Revision 7, Section 5.8. Consequently, the procedure did not include the requirement to measure the 0.010" to 0.090" gap.

On January 28, 2005, the licensee issued CR 05-00364 to assess operability of all safety-related 5HK breakers. The purpose of the CR was to address an operability question raised concerning seven safety-related 5HK breakers that were found to have control device (control relay) gap settings that were outside the range recommended by ABB. The inspectors were concerned that this was a non-conforming condition that potentially impacted proper operation of safety-related breakers. Ultimately, the licensees OD concluded that breakers with control device settings in excess of the ABB-specified limits were acceptable. The licensee based their conclusion on the following facts; that all of the breakers were appropriately set during the last time they were refurbished; that the set up value specified in ABB IB 6.2.1.7.D was not an acceptance value used in determining breaker functionality; and that once overhauled or refurbished, and with the breaker gaps properly set, the breakers should function reliably for 1000 cycles without the need to further adjust the gaps. The inspectors noted that the assumption that the gaps would not require readjustment presumes that there are no other concerns or problems with the breaker. The inspectors concluded that the licensees OD did not adequately evaluate the interactive effects on operability of the breaker control device gap settings being outside the ABB-recommended gap values, and breakers which have been in-service over 10 years without being refurbished (See URI, Section 1.3(1)).

The inspectors also noted that the licensee checked the gap setting on a sample of breakers in support of the root cause evaluation effort (CR 05-00095). To aid in determining which type and size breakers should be in the sample, the licensee identified which safety significant 13.8kV breakers were required to cycle and operate automatically in order to perform their intended design function. The licensee inspected a representative sample of 4.16kV and 13.8kV breakers for control relay gap values and identified that 5 of the14 safety-related 4.16kV breakers, and 11 of the 12 non-safety-related, but high safety significant, 13.8kV breakers had gaps settings outside the vendor-recommended values.

After the vendor analyzed the January 6, 2005 MFP breaker failure, they reported that the gap clearance between the control device arm and the timing lever was found outside of the manufacturers tolerance for initial gap setting on refurbished breakers.

Additionally, the licensee indicated that the breaker functioned properly with these deficiencies corrected. Based on this finding, the root cause team concluded that the gap setting may have been a failure mode not refuted for the January 6th event and may have been a contributing cause for the breaker failure on January 6th, but not for the failure that occurred on January 18, 2005.

The issues discussed above further support the critical nature of the gap measurement and the potential for an improper gap setting to contribute to the failure of a safety-related circuit breaker. Additionally, the inspectors concluded that the failure to incorporate the gap setting in related maintenance procedures could leave the breaker in a condition where operation of the breaker could cause damage to the device.

Analysis Based upon the licensees operability evaluation that indicated that the control device gap setting was critical as part of overhaul and refurbishment activities, and because related maintenance procedures did not contain all of the vendor-recommended tolerances, the inspectors concluded that the failure to incorporate the gap setting in the maintenance procedures was a performance deficiency. Specifically, if the control device gap were improperly set or the breaker configuration or tolerances were altered as a result of a maintenance activity, then damage could occur to the components necessary to ensure proper breaker operation.

The inspectors determined that this performance deficiency resulted in an inadequate maintenance procedure for safety-related breakers; therefore, this deficiency affected the procedure quality attribute of the Mitigating System Cornerstone of Reactor Safety.

The objective of the Mitigating System Cornerstone of Reactor Safety is to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. As such, the objective of ensuring the availability and reliability of the safety-related 4.16kV breakers in various plant systems to respond to initiating events to prevent undesirable consequences was impacted; therefore, this issue was more than minor.

The inspectors performed a Phase 1 screening of the issue using the NRCs SDP and determined that the deficiency did not result in any loss of function; that the finding was not risk significant due to a seismic, flooding, or severe weather initiating event; and that other plant-specific analyses that identify core damage scenarios of concern were not impacted; therefore the issue screened as Green.

Enforcement Technical Specification Section 5.4, requires that procedures be developed and implemented in accordance with Regulatory Guide 1.33. Regulatory Guide 1.33, Section 9, states in part that, Maintenance that can affect the performance of safety-related equipment should be properly preplanned and performed in accordance with written procedures, documented instructions, or drawings appropriate to the circumstances. Contrary to this requirement, maintenance procedures for overhauling safety-related breakers were inappropriate because they did not contain guidance to ensure that critical measurements were maintained within vendor specifications; thereby ensuring that safety-related breakers remained operable.

Because this violation was of very low safety significance (Green) and because the issue was entered into the licensees corrective action program (CR 05-00364 and CR 05-00095-008), the failure of the licensee to incorporate the appropriate vendors gap requirements into the PNPP Operations Manual Instructions GEI-0135 is considered a Non-Cited Violation, consistent with Section VI.A.1 of the NRC Enforcement Policy (NCV 05000440/2005005-04).

Human Factor and Procedural Issues (93812)2.1 Evaluation of Operator Response to the December 23, 2004 Transient Caused by an Unplanned Downshift of the RRPs (Charter Items 3 and 6)

a. Inspection Scope

The inspectors reviewed plant logs; procedures, including off normal procedures and annunciator response procedures; the sequence of event recorders; plant trend traces; the licensees scram reports; and other related plant information. The inspectors also interviewed various personnel involved with responding to the event. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Description On December 23, 2004, control room operators were completing a brief in the control room when the panel operators received alarms indicating an unanticipated downshift of both RRPs from high speed to low speed, with a resultant power decrease to approximately 55 percent. Operators verified the plant response and requested assistance from crew members. Senior operators confirmed that the combination of both reactor power and core flow placed them in the region of instability known as the Immediate Exit Region on the Power-to-Flow map. Senior operators reviewed [Off Normal Instruction] ONI-C51, Unplanned Change in Reactor Power or Reactivity.

Operators were instructed to monitor the neutron detectors for indications of instability and were directed to scram the reactor if instability was identified. Senior operators discussed the effect of implementing the mitigating actions of increasing reactor recirculation flow to exit the Immediate Exit Region. Concurrently, control room operators acknowledged three OPRM alarm annunciators, which had alarmed and cleared.

About 81/2 minutes after the RRPs downshifted, operators commenced increasing recirculation flow in an unsuccessful attempt to exit the Immediate Exit Region in accordance with ONI-C51 procedure. About 20 seconds later, a fourth OPRM annunciator alarmed and did not clear. Senior operators directed panel operators to insert control rods. At about 91/2 minutes after the RRP downshift, operators were approaching the reactor control panel to insert control rods when the reactor scrammed on high OPRM period. The operators did not expect the OPRM trip since there were no visible oscillations seen on the nuclear instruments.

Assessment For the December 23, 2004, event, the inspectors determined that the OPRMs successfully tripped the reactor on the period algorithm when core wide oscillations exceeded the predetermined setpoint. The OPRM period was a calculated algorithm and the setpoint was low enough that operators were unable to detect oscillations upon entering the region of instability using less sensitive control room instrumentation, (i.e.

average power range meters (APRMs), local power range meters (LPRMs), and source range meters (SRMs)). Additionally, no procedural requirements or management guidance to insert a manual scram on an OPRM alarm existed. The lack of procedural guidance existed because the licensees reactor engineering department did not want operators to manually scram the reactor on converging OPRM oscillations. This was because converging OPRM oscillations would not result in the fuel exceeding minimum critical power ratio (MCPR) limits. Although diverging oscillations could result in the fuel exceeded MCPR limits, the OPRM was installed to ensure that the reactor would be shutdown during diverging oscillations prior to exceeding thermal limits. The inspectors determined that the OPRMs adequately protected the core independent of actions taken by the operators on December 23, 2004.

The inspectors were concerned that the OPRM did not provide indications that would allow operators to anticipate OPRM-initiated reactor scrams. With respect to the December OPRM scram, the inspectors concluded that instrumentation in the control room was appropriate for the conditions; however, because the licensee had installed an OPRM with minimal indications, the indications available to the operators to anticipate the OPRM scram were not sensitive enough to give prior warning of the trip. The inspectors determined that the OPRM modification was installed in accordance with design instructions. The OPRM design consisted of three separate algorithms that had annunciators as outputs, but no indication. The OPRM trip algorithm was set low enough to trip the reactor before existing neutron monitoring equipment would detect oscillations, but high enough to avoid getting spurious trip signals. The licensees design was reviewed by the NRC and received approval for implementation in a Safety Evaluation Report dated February 26, 2001.

The inspectors determined that operators properly implemented ONI-C51 and adhered to the annunciator response procedure for the OPRM alarm; however, ONI-C51 gave a higher order of preference to exit the Immediate Exit Region by increasing core flow rather than inserting control rods. The automatic downshift of the RRPs placed the recirculation system in a configuration with both pumps operating in low speed with the associated flow control valves approximately 80 percent open. Subsequent analysis showed that increasing core flow by fully opening the flow control valves would not be effective in exiting the Immediate Exit Region under the low flow - low differential pressure conditions that existed in the post-downshift configuration. After operators recognized that their attempts to increase flow were ineffective, they ordered insertion of control rods as required by ONI-C51. The inspectors determined that there were no procedure non-adherence issues associated with this event, but there were procedure weaknesses because control rod insertion was more effective at exiting the region of instability than increasing recirculation flow.

During operator training after this event, operators provided feedback to change ONI-C51 to allow inserting control rods as a higher priority than increasing flow for exiting the Immediate Exit Region. Additionally, [annunciator response instruction]

ARI H13-P680-0004-A3(A12), Recirc Pump High to Lo Speed Xfer, and ARI H13-P680-0006-A3, OPRM [alarm], did not provide adequate operator guidance.

The RRP downshift ARI directed entry into ONI-C51 only if the low frequency motor generator tripped. The inspectors believed that the licensee should consider whether the procedure should direct entry into ONI-C51 for any unplanned downshift of the RRPs while at power.

During a requalification classroom training session dated June 21, 2004, reactor engineering told licensed operators that, in a slow moving transient, the time from the

[OPRM] alarm ... to the OPRM trip [was] in the order of 45 seconds to a minute...

[affording time for the] possibility of operator interaction. On December 23, 2004, the fourth OPRM alarm came in and did not clear, followed by the OPRM trip 42 seconds later. Although the classroom training adequately predicted the onset of the trip, operators still believed that the oscillations would be seen on existing neutron indication. Operators stated that a white annunciator tile clearing was a good thing.

During this event operators did not recognize that the increasing frequency of the OPRM alarms was indicative of power oscillations, since the operators did not see oscillations from existing indicators (i.e., APRMs, LPRMs, and SRMs). Additionally, training provided negative reinforcement to the operators because the simulator modeling of core power oscillations indicated that oscillations would be seen prior to receiving an OPRM trip. Also, the OPRM alarm ARI stated that if the [OPRM] alarm has NOT cleared in 2 minutes, and reactor engineering is not available, then insert cram rods until alarm clears. In this event, 2 minutes after the OPRM alarm was locked in, control rods were already inserted by the scram. Based on the above, the inspectors determined that a weakness existed in operator training on the operational characteristics of the OPRM.

The inspectors verified that operators had received training on the OPRM modifications, changes to TS, ONI-C51 flow chart usage, and changes to the Power-to-Flow map, including actions to take when in the region of instability. Both classroom and simulator training was provided to all crews to discuss the December 23, 2004 event and the need to insert control rods in a timely manner upon entering into the Immediate Exit Region.

The inspectors determined that the post-December 23 event training provided to operators was effective in combating the second RRP that occurred on January 6, 2005.

2.2 Evaluation of Licensee Response to the December 23, 2004, and January 6, 2005, Equipment Failures That Caused an Unplanned Downshift of the RRPs (Charter Items 4 and 7)

(1) Evaluation of the Immediate Response to the Failed Equipment After the December 23, 2004 Event

a. Inspection Scope

The inspectors performed a detailed review of the licensees root cause analysis, including troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process.

The inspectors also reviewed plant logs; procedures, including off normal procedures and annunciator response procedures; the sequence of event recorders; plant trend traces; the licensees scram reports and other related plant information. The inspectors included in their assessment interviews with various personnel involved in responding to the event. Associated work orders and logs were reviewed to assess the adequacy of the licensees response to the equipment failures. Additionally, the licensees corrective action database was reviewed to assess if the licensee had prior opportunities to identify related conditions. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction A finding of very low significance (Green) without a violation related to the Initiating Events Cornerstone was identified by the inspectors. The inspectors concluded that the licensee failed to quarantine equipment, and performed troubleshooting without full benefit of a troubleshooting plan. The inspectors determined that the failure to quarantine equipment impaired the licensees ability to identify the associated failure mechanism for the simultaneous downshifting of both RRPs. Because the RRPs are not 10 CFR 50, Appendix B systems, and because the procedures that the licensee failed to follow were not TS-required procedures, there was no violation associated with the finding.

Description The inspectors found that subsequent to the December 23, 2004 scram, the licensee performed evaluations on equipment associated with the RRP logic to assess the failure mechanism and support subsequent reactor restart. The inspectors also noted that the licensee did not perform a root cause evaluation to support a restart decision. The decision to restart the reactor was based upon the results of an immediate investigation following the event. Additionally, the primary focus of the investigation was on related feedwater circuitry. The licensee believed that this circuitry, which had previously failed, was the most likely cause for the event. The fact that the licensee failed to quarantine related equipment and believed that vital information relative to the interfacing feedwater circuitry had been lost, contributed to its failure to identify the actual failure mechanism.

Assessment The inspectors noted that the licensees troubleshooting activities for the December 23, 2005, event appeared to be hurried. Equipment was not appropriately quarantined and troubleshooting commenced under a high priority work order without a detailed troubleshooting plan; the plan was generated in parallel with the investigation activities.

Further, the restart meeting was moved up and then allowed to slip to coincide with the completion of the immediate investigation. In addition, licensee technicians indicated that in-field engineering support was limited because the engineers were involved with preparations for the restart meeting. Based on these observations, the inspectors were concerned that time pressures, whether perceived or real, may have contributed to the licensee not appropriately evaluating the December 23 event. The licensee generated CR 05-00989 to evaluate potential restart pressure issues.

Because the licensee did not quarantine equipment, performed troubleshooting without full benefit of a troubleshooting plan, and focused their efforts on equipment that had previously failed, it inappropriately limited the scope of its investigation. Because of the limited investigation scope, the information necessary to identify the cause of the RRP downshift was not identified. The inspectors concluded that the decision to restart was made without an identified failure mechanism and was based entirely on historical experience related to feedwater components. Additionally, adequate corrective actions were not implemented until the root cause for the January event was completed.

Measures implemented after the December event were specific to monitoring of equipment and components related to the feedwater pump inputs to the RRP logic and did not include equipment which ultimately was determined to have caused the downshift.

The inspectors identified that multiple procedural barriers for quarantine failed for this event. The root cause analysis reference guide, condition report process, and problem-solving decision-making procedures all contain requirements to quarantine equipment, and the failure to do so resulted in the loss of valuable information for technicians and engineers. Had certain data been available, more in-depth troubleshooting may have resulted. Specifically, resetting of various logics caused the licensees staff to assume that critical data associated with the feedwater cards was lost. This caused the licensees staff to believe that those cards were the contributors to the downshift.

Analysis Procedure NOP-LP-2001, Condition Report Process, requires notification of the on-shift SRO of any condition appearing to jeopardize plant operation and when an Immediate Investigation is needed on equipment installed in the plant, THEN coordinate with a designated on-shift licensed SRO to quarantine the equipment . . . Procedure, NOP-ER-3001, Problem Solving and Decision Making Process, states that the issue owner is responsible for quarantining affected equipment, when applicable to ensure evidence is not disturbed or destroyed. Procedure NOBP-LP-2011, FENOC Root Cause Analysis Reference Guide, discusses the preservation of data and indicates that a root cause trained individual should be among the initial responders to the event, and that an event should include a prompt investigation, which can include the need to quarantine a work site or equipment. The inspectors determined that the failure to quarantine as required by these procedures, was a performance deficiency.

The objective of the Initiating Events Cornerstone is to limit the likelihood of those events that upset plant stability and challenge critical safety functions during shutdown and power operations. The inspectors determined the finding was more than minor because the failure to quarantine impaired the licensees ability to identify the associated failure mechanism. Subsequently, on January 6, 2005, both RRPs again downshifted and initiated a plant transient, a transient where challenged equipment failed and plant operators were challenged with combating equipment failures. As a result, a plant transient was initiated that caused an actual upset in plant stability which directly affects the objective for the Initiating Events Cornerstone. Additionally, the RRP downshifts affected the equipment performance attributes of availability and reliability of the Initiating Events Cornerstone of Reactor Safety, further supporting that the issue is more than minor. Also, the inspectors determined that the issue affected the Initiating Events Cornerstone because it had an actual, albeit small, risk impact on the licensees transient event initiating frequency.

In order to determine the significance of the event, the inspectors performed a Phase 1 SDP in accordance with IMC 0609. The inspectors concluded that the event increased the likelihood of a initiating event and impacted the Initiating Events Cornerstone.

Specifically, the event resulted in a reactor trip and was a transient initiator. Because the finding did not result in exceeding the TS limit for identified reactor coolant system (RCS) leakage and did not affect other mitigation systems; the finding did not contribute to both the likelihood of a reactor trip AND the likelihood that mitigation equipment or functions will not be available; and the finding did not increase the likelihood of a fire or internal/external flood, the finding screened as Green.

Enforcement The inspectors determined that reactor recirculation system is not a 10 CFR 50, Appendix B, system, and because the aforementioned procedures are not required by licensee TSs, specifically Regulatory Guide 1.33, there is no violation of NRC requirements. Because this finding was of very low safety significance (Green), the inspectors determined that the failure to quarantine equipment as required by procedures was a finding without a violation (FIN 05000440/2005005-05). The licensee entered this issue into their corrective action program as CR 05-00239.

(2) Evaluation of the Decision to Restart Following the December 23, 2004 Event

a. Inspection Scope

The inspectors performed a detailed review of the licensees root cause analysis, including troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process.

The inspectors also reviewed plant logs; procedures, including off normal procedures and annunciator response procedures; the sequence of event recorders; plant trend traces; the licensees scram reports and other related plant information. The inspectors included in their assessment interviews with various personnel involved with responding to the event. Associated work orders and logs were reviewed to assess the adequacy of the licensees response to the equipment failures. Additionally, the licensees corrective action database was reviewed to assess if the licensee had prior opportunities to identify related conditions. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction A finding of very low significance (Green) without a violation related to the Initiating Events Cornerstone was identified by the inspectors. The inspectors concluded that the licensee failed to properly assess or document an assessment for the removal of restart restraints prior to resuming reactor operation subsequent to the December 23, 2004 scram. The inspectors determined the finding was more than minor because the failure to appropriately close a mode restraint prior to startup, and the failure to properly document the basis for resolving a mode restraint in the associated condition report impaired the licensees ability to identify the associated failure mechanism for the dual RRP downshift event. Because the RRPs are not 10 CFR 50, Appendix B, systems and because the procedures that the licensee failed to follow were not TS-required procedures, there was no violation associated with the finding.

Description Subsequent to the investigation, station management elected to restart the facility using the data gained in the immediate investigation. Evidence showed that management recognized the staff had not identify the failure mechanism for the RRP downshift and were concerned about a second event. Although station management was aware of the event and documented the decision to restart in a matrix, they did not follow the process for resolving mode restraints prior to restarting the facility.

Assessment The inspectors noted that the restart package contained conflicting information with respect to requirements related to restarting the reactor. Specifically, the restart package included forms indicating that no restart restraints existed; however, there was a form indicating that CR 2004-06766 Reactor Recirculation Pump Speed shift, remained open as a restart restraint.

The inspectors also identified that multiple procedures required that mode restraints be properly resolved prior to reactor restart. Specifically, Procedure NOBP-OM-4010, Restart Readiness for Plant Outages, Attachment 2, Item 14, required that any condition reports, corrective actions, work orders, and modifications required for restart be completed or that they be tracked on the mode restraint list. The inspectors noted that CR 2004-06766 was designated as a mode restraint; however, Attachment 2 was closed without the benefit of explanation and with this CR remaining active as a mode restraint. Additionally, NOP-LP-2001, Condition Reporting Process, required that the basis for resolving a mode restraint be documented in, or attached to, the associated condition report. Not only was restart conducted with this CR active as a mode restraint, no information was documented in the CR stating the basis for lifting the restraint.

The inspectors identified that the licensee missed two opportunities to further evaluate the lack of an identified failure mechanism when the licensees staff did not document the basis for restart with an active mode restraint per the 4010 Procedure, and in the CR per the 2001 Procedure.

Analysis The inspectors determined that the failure to follow the requirements to track and close mode restraints, as required by Procedures NOBP-OM-4010 and NOP-LP-2001, represented a performance deficiency.

The objective of the Initiating Events Cornerstone is to limit the likelihood of those events that upset plant stability and challenge critical safety functions during shutdown and power operations. The inspectors determined that the finding was more than minor because the failure to appropriately close a mode restraint prior to startup, and to properly document the basis for resolving a mode restraint in the associated condition report, impaired the licensees ability to identify the associated failure mechanism for the dual RRP downshift event. On January 6, 2005, both RRPs again downshifted and initiated a plant transient, a transient where challenged equipment failed and plant operators were challenged with combating the equipment failures. As a result, a plant transient was initiated that caused an actual upset in plant stability, which directly affected the objective for the Initiating Events Cornerstone. Additionally, the RRP downshifts affected the equipment performance attributes of availability and reliability of the Initiating Events Cornerstone of Reactor Safety, further supporting that the issue was more than minor. Also, the inspectors determined that the issue affected the Initiating Events Cornerstone because it had an actual, albeit small, risk impact on the licensees transient event initiating frequency.

In order to determine the risk/safety significance of the event, the inspectors performed a Phase 1 SDP in accordance with IMC 0609. The inspectors concluded that the event increased the likelihood of a initiating event and impacted the Initiating Events Cornerstone. Specifically, the event was a transient initiator contributor, which resulted in a reactor trip. Because the finding did not result in exceeding the TS limit for identified RCS leakage and did not affect other mitigation systems, the finding did not contribute to both the likelihood of a reactor trip AND the likelihood that mitigation equipment or functions will not be available; and the finding did not increase the likelihood of a fire or internal/external flood, the finding screened as Green .

Enforcement The inspectors determined that reactor recirculation system is not a 10 CFR 50, Appendix B, system and because the aforementioned procedures are not required by licensee TSs, specifically Regulatory Guide 1.33, there is no violation of NRC requirements. Because this finding was of very low safety significance (Green),the inspectors determined that the failure to evaluate and document the removal of a mode restraint, as required by procedures, was a finding without a violation (FIN 05000440/2005005-06). The licensee entered this issue into their corrective action program as CR 05-00380.

2.3 Evaluation of Operator Response to the January 6, 2005 Transient Caused by an Unplanned Downshift of the RRPs, Trip of the A RRP LFMG, Failure of the MFP to Start, and Cooldown Rate Approaching the TS 100/F/hr Limit (Charter Items 3 and 6)

a. Inspection Scope

(93812)

The inspectors reviewed plant logs; procedures, including off normal procedures and annunciator response procedures; the sequence of event recorders; plant trend traces; the licensees scram reports and other related plant information. The inspectors also interviewed various personnel involved with responding to the event. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Description On January 6, 2005, control room operators received indication of an automatic unanticipated downshift of both RRPs from 100 percent power. Operators verified correct plant response and informed senior operators who were monitoring the event.

Senior operators verified that the combination of both reactor power and core flow placed them in the Immediate Exit Region of the Power-to-Flow map. Senior operators reviewed ONI-C51 and quickly ordered panel operators to increase recirculation flow and insert control rods. Three minutes after the RRP downshifted panel operators had almost fully inserted the first control rod when RRP A tripped to off. In the next 2 minutes, senior operators discussed a panel operators suggestion to shutdown the unit. After weighing possible recovery and shutdown options, senior operators ordered the unit scrammed.

During recovery of reactor water level, control room operators could not start the MFP.

A plant operator, dispatched to the MFP electrical breaker, reported that the breaker closing coil was not energized. The plant operator was ordered to re-rack in the breaker in an attempt to get the auxiliary contacts on the breaker to pick up, allowing the closing coil to become energized. After two attempts, the closing coil would not energize. In response, control room operators started the reactor core isolation cooling (RCIC) pump and a turbine-driven feedwater pump to recover reactor vessel water level.

During subsequent plant cooldown, a senior operator ordered panel operators to observe the cooldown rate and not to cool the plant below 400 psig. About 48 minutes after the scram, control room operators observed a calculated cooldown rate in the mid-90s on the safety parameters display system and plant pressure approaching 400 psig. Control room operators then closed the MSIVs to prevent further cooldown.

Assessment The inspectors reviewed the operator response to the downshift of both RRPs. The inspectors determined that the training provided to control room operating crews after the December 23, 2004, scram was effective because operators appropriately responded to and mitigated the event in both a timely manner and without receiving OPRM alarms. In general, the inspectors concluded that control room operator and senior operator activities surrounding the January scram were methodical and prudent, with the exception of some of the initial activities for quarantining the MFP breaker after its failure to close.

During the event, the RRP A LFMG tripped. This trip, combined with the unplanned downshift for both RRPs, gave the operators cause to manually scram the reactor. The inspectors noted that the plant was licensed to operate in a single loop and determined that the decision to scram the reactor after a second equipment failure, the RRP A trip, was conservative and prudent.

Following the downshift, the MFP failed to start. As a result, turbine feed pumps and RCIC were used to mitigate the event. The inspectors concluded that the combination of the steam demanded by operating the turbine-driven feedwater pump, a high feedwater flow rate after the scram, other house steam loads, and a relatively low decay heat rate, resulted in a more rapid depressurization of the RPV than expected. The inspectors reviewed trend charts and estimated the cooldown rate to be 98/F/hr. The maximum cooldown rate allowed by TS was 100/F/hr. The inspectors determined that the operator action to close the MSIVs was appropriate.

2.4 Evaluation of Licensee Response to the January 6, 2005 Equipment Failures That Caused an Unplanned Downshift of the RRPs, Trip of the RRP A LFMG, and Failure of the MFP to Start (Charter Items 4 and 7)

(1) Failure to Quarantine the MFP Breaker Subsequent to the January 6, 2005 Event

a. Inspection Scope

The inspectors performed a detailed review of the licensees root cause analysis, including troubleshooting activities, corrective actions, extent of condition reviews, actions to prevent recurrence, challenge boards, and the management review process.

The inspectors also reviewed plant logs; procedures, including off normal procedures and annunciator response procedures; the sequence of event recorders; plant trend traces; the licensees scram reports and other related plant information. The inspectors included in their assessment interviews with various personnel involved with responding to the event. Associated work orders and logs were reviewed to assess the adequacy of the licensees response to the equipment failures. Additionally, the licensees corrective action database was reviewed to assess if the licensee had prior opportunities to identify related conditions. As part of this inspection, the documents in Attachment 1 were utilized to evaluate the potential for an inspection finding.

b. Findings and Observations

Introduction A finding of very low significance (Green) without a violation related to the Mitigating Systems Cornerstone was identified by the inspectors. The inspectors concluded that the licensee failed to quarantine equipment and performed troubleshooting without full benefit of a troubleshooting plan. The inspectors determined that the failure to quarantine the MFP breaker cubicle impaired the licensees ability to identify the associated failure mechanism for the January 6, 2005 failure of the MFP breaker to close. Because the MFP is not a 10 CFR 50, Appendix B, system, and because the procedures that the licensee failed to follow were not TS-required procedures, there was no violation associated with the finding.

Description Subsequent to the January 6, 2005 reactor scram the licensee unsuccessfully attempted to manually start the MFP from the control room. Following the event, the licensee initiated a root cause investigation to determine the failure mechanism. As part of the licensees evaluation, the affected breaker was sent to their vendor for analysis.

Following breaker analysis and refurbishment, the vendor concluded that the failure was caused by an improper gap between the timing lever and the control device limit switch roller. On January 18, 2005, while performing post-maintenance testing on a replacement breaker, another failure to charge the closing springs occurred.

The inspectors compared the licensee investigation initiated on January 6, 2005, to the investigation initiated on January 18, 2005, for the second failure of the MFP breaker to charge. The inspectors determined that the 13.8kV breaker that failed on January 6th had not been quarantined. After the January 18th event, valuable as-found data identified using proper quarantine techniques and better analysis methods was obtained.

The inspectors concluded that the licensee could have identified the actual failure mechanism had they properly quarantined the breaker and the associated cubicle after the January 6th event. After the January 18th MFP breaker failure, the licensee identified that there was sufficient historical data indicating that the L1007 cubicle was an outlier in the number of breaker failures, and a high probability existed for finding the failure mechanism had this historical data been assessed and the breaker quarantined. The licensee subsequently performed a review of this issue and identified an adverse trend regarding quarantining items and initiated CR 05-00575 to address this concern.

Analysis Procedure NOP-LP-2001, Condition Report Process, requires notification of the on-shift SRO of any condition appearing to jeopardize plant operation and when an Immediate Investigation is needed on equipment installed in the plant, THEN coordinate with a designated on-shift licensed senior reactor operator (SRO) to quarantine the equipment. Procedure NOP-ER-3001, Problem Solving and Decision Making Process, states that the issue owner is responsible for quarantining affected equipment, when applicable to ensure evidence is not disturbed or destroyed.

Procedure NOBP-LP-2011, FENOC Root Cause Analysis reference Guide, discussed the preservation of data and indicates that a root cause trained individual should be among the initial responders to the event and that an event should include a prompt investigation, which can include the need to quarantine a work site or equipment. The inspectors determined that the failure to quarantine as required by these procedures was a performance deficiency.

The inspectors determined that the finding was more than minor because the failure to quarantine the MFP breaker after the January 6, 2005 failure, if left uncorrected, could become a more significant safety concern. Specifically, if left uncorrected, a latent failure of the MFP to start could have existed thereby making a mitigating system unavailable during a transient where the system was needed. Additionally, the only reason that the issue was discovered was the failure of post-maintenance testing of a refurbished MFP breaker and, had the post-maintenance testing been successful the inspectors concluded that the licensee-identified root cause would not have been revealed. Further supporting that the finding was more than minor was the impact on the objective of the Mitigating Systems Cornerstone. The objective of the Mitigating Systems Cornerstone is to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences, and the failure to quarantine the MFP breaker directly affects the equipment performance attributes of availability and reliability of this cornerstone for the same reasons described above.

In order to determine the significance of the event, the inspectors performed a Phase 1 SDP in accordance with IMC 0609. The inspectors determined that the deficiency affected the short term heat removal element of the Mitigating System Cornerstone and that the issue was not a design deficiency that resulted in a loss of function, that the system was not a safety system, and that the system was not a TS system. In addition, the finding did not represent an actual loss of safety function or equipment designed as risk-significant per 10 CFR 50.65 for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, the finding was not risk significant due to a seismic, flooding, or severe weather initiating event, therefore the finding screened as Green.

Enforcement The inspectors determined that because MFP is not a 10 CFR 50, Appendix B, system and the aforementioned procedures are not required by licensee TSs, specifically, Regulatory Guide 1.33; there is no violation of NRC requirements for failing to follow the procedures. Because this finding was of very low safety significance (Green), the inspectors determined that this issue was a finding without a violation (FIN 05000440/2005005-07). The licensee entered this issue into there corrective action program as CR 05-00239.

(2) Evaluation of the Corrective Actions Human Performance Issues for the December 23, 2004, and January 6, 2005 Equipment Failures

a. Inspection Scope

The inspectors reviewed the root cause analysis for the December 23, 2004, and the January 6, 2005 events. The root cause was chartered to determine the cause of the RRP downshift, the RRP trip, to evaluate the response to the event, and to evaluate for common cause between the two scrams. The inspectors reviewed the corrective actions and barriers that the licensee established to mitigate related human performance issues in the future.

b. Findings and Observations

Description The inspectors noted that the licensee had identified in their root cause analysis contributing causes that were related to human performance issues at the facility.

Specifically, the failure to quarantine equipment after the December 23, 2004, was in part attributed to a lack of a full organizational commitment to the program implementation of NOP-ER-3001, Problem Solving and Decision Making. The report further stated that this lack of organizational commitment to the program implementation of the Problem Solving and Decision Making process resulted in the less than adequate management decision to restart the plant following the December 23, 2004, forced outage. Additionally, management had indicated that the decision to restart the plant on December 26, 2004, was based on insufficient information and had procedures been followed management would have recognized that insufficient information was available to make a decision as important as restarting the reactor.

Assessment During the inspectors review of both the December and the January root cause analyses (CR 04-06766 and CR 05-00094) the inspectors found that both analyses indicated that CR 05-00145 would investigate the operational decision-making of the plant restart on December 26, 2004, using an apparent cause process. The inspectors concluded that diverting this investigation was an inadequate decision. By diverting this investigation, the licensees root cause team was predisposed to defer the analysis to another process, which was an apparent cause evaluation. The condition reporting process prohibits the closure of a CR to one of a lesser process, and by allowing this action to occur the quality of the product and the rigor of the investigation became suspect.

With respect to the unplanned downshifting of RRPs during both events, the inspectors found that the January root cause report reflected the same causal factor as the December report. Because the event repeated itself, the inspectors anticipated that the second root cause would delve into the basis for the repeat issue; however, both analyses identified the same cause. The inspectors considered causal analysis for the repeat pump narrowly focused, thereby limiting the licensees ability to identify and correct broader human performance issues.

The inspectors concluded that the root cause process itself would lead licensee personnel to narrowly focus their analysis and was a contributor to the above observations. The inspectors noted that the licensees root cause procedure focused the analysis on technical issues and did not provide a good process to expand the analysis when broader human performance arise. The inspectors also identified that extent of condition reviews for non-technical issues was limited. Additionally, licensee procedures did not require extent of cause, extent of condition, or effectiveness reviews for contributing causes associated with a significant condition adverse-to-quality. The inspectors were concerned that the lack of extent-of-cause and extent-of-condition reviews for the broader issues that may have been identified by a broader analysis, could deprive the licensee of valuable insights in correcting station performance.

The inspectors concluded that the repeat nature of this event, along with the related decision-making errors discussed above, warranted further review. Because the root cause process focused the analysis on technical issues and did not have a good process for expanding the investigation when broader human performance were identified during the analysis, the inspectors were concerned about the interim barriers established for the broader issues. Specifically, Contributing Cause No. 2 in the licensees root cause analysis for the January 6, 2005, event, indicated that the lack of a full organizational commitment to the problem solving and decision-making process had resulted in less than adequate management decision to restart the plant following the December event.

Further supporting the need for the root cause process to have a provision to expand the scope of a root cause analysis was the fact that the inspectors had found prior condition reports which identified contributing and root causes containing human performance attributes similar to this apparent cause. Examples included: when the plant was restarted with inoperable diesel generators; restarted with inoperable low-pressure core spray and RHR systems; and in Revision 3 to the root cause analysis performed for the combined white findings dated September 2004.

The inspectors discussed these issues with Site Vice President on January 20, 2005.

Upon review of the first root cause, the inspectors concluded that Contributing Cause No. 2, Lack of a full organizational commitment to the program implementation of NOP-ER-3001, Problem Solving and Decision Making Process, indicated that conditions existed for this causal factor to re-surface in future processes. The Site Vice President indicated that elements which already existed in the Site Performance Improvement Initiative would provide the primary barriers to prevent a repeat occurrence of this contributing cause. Additionally, he cited other good practices that would normally be part of any strong managerial and oversight process. The inspectors noted that the barriers were all good, but many would take long term to implement and inquired about short-term barriers. The Site Vice President indicated he would provide a list of interim measures. Following the meeting, the site vice president provided a list of other compensatory measures and interim barriers to bolster managerial practices until the long term barriers became effective. The measures were entered into the corrective action program and included:

C As part of the performance improvement initiative, establish a requirement for an independent review of significant root cause analysis performed at Perry, until final corrective actions are in place and effective (05-00514 CA #1).

C Develop and implement compensatory measures or contingency plans identified during startup readiness review meeting (05-00145 CA #6).

C As an interim measure, the plant manager will request that PORC [Plant Operations Review Committee] review readiness for startup for safety issues and make a recommendation on restart to plant manager (05-00145 CA #7).

C As an interim measure, establish a requirement that all engineering tasks receive a pre-job brief, using "New Assignment and Briefing Card" (05-00543 CA #3).

C Establish expectations for the [Engineer Assessment Board] EAB including face-to-face meetings, improved quorum requirements, and the enforcement of an expectation that the EAB review is performed after the engineering supervisors review and signature.

C Establish an interim measure that Procedure PYBP-SITE-0014 is an in-hand use procedure during engineering development of operability determinations.

C As an interim measure for this start up, perform a collegial review of the decision to start up, using the INPO Guidelines for Operational Decision Making until corrective actions from CR 05-00145 are finalized (05-00145 CA #3).

C Establish an interim measure to require all operability determinations to be reviewed by the Engineering Director prior to submittal to Operations (05-00543 CA #4).

C Review organizational aspects of the December 26, 2004 restart decision with the management team, and as an interim action, establish requirement that Restart Readiness, Condition Report Process, and Problem-Solving Decision-Making procedures are in-hand use procedures for a 2-week period

[starting January 15, 2004] (05-00145 CA #4).

The inspectors concluded that the interim barriers could be effective and were appropriate; however, some weaknesses in the processes remained. Specifically, the failure to follow the Restart Readiness, Condition Report Process, and Problem-Solving Decision-Making procedures were paramount to the failure to quarantine equipment after the December, 2004 event. The inspectors concluded that a 2-week in-hand use period may not provide sufficient opportunity for all station personnel to re-familiarize themselves with these General Skill Reference procedures. Because knowledge of the procedures was not readily apparent during the December event and the quarantining of the MFP breaker in January, the inspectors concluded that making them an in-hand use procedure may not accomplish the desired level of content knowledge. The lack of knowledge with the referenced procedures is not consistent with a systematic approach to training (SAT) based training program, where infrequently performed tasks such as scram response or plant restart have some form of periodic reinforcement. Safety nets for these types of skills can include items such as periodic refresher training, verbatim procedural compliance, or evolution checklists that ensure personnel are familiar with those tasks or utilize the tools provided to cope with unplanned situations, should the need arise. The inspectors noted no related long-term corrective actions.

The inspectors were concerned that a root cause analysis was not performed on the repeat nature of the RRP trips. The condition report process guidance suggests that an unusual plant transient involving equipment performance that affected proper control of reactivity warrants a root cause analysis. The inspectors noted that the root cause analysis focused on the equipment failure and not the transient in general, which was a broader issue. When asked about the guidance requirement, multiple members of the plant staff, including the Site Vice President, indicated that the category descriptions found in the condition report procedure were only guidance. The inspectors further noted that when an issue is categorized at a lower level than the CR process suggests, no basis for the decision is documented. The licensee entered the above issues into their corrective action program for further evaluation (CRs: 05-00985, 05-00993, 05-00994).

4OA6 Exit Meeting

The Team presented the inspection results to Mr. L. Myers and members of his staff on February 18, 2005. The licensee acknowledged the results presented. No proprietary information was identified.

SUPPLEMENTAL INFORMATION

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000440/2005005-01 NCV Failure to Incorporate Industry Operating Experience into

Preventative Maintenance Activities (Section 1.2)05000440/2005005-02 URI Operability Evaluation of Safety Related Breakers Requires

Further Review (Section 1.3(1))05000440/2005005-03 NCV Failure to Provide Guidance to Refurbish Breakers Within

Vendor-specified Time Frames or to Provide Reasonable

Alternative Preventative Maintenance Practices to Ensure

That Safety-related Breakers Remained Operable

(Section 1.3(2))05000440/2005005-04 NCV Failure to Incorporate the Vendors Gap Requirements into

Operations Manual Instructions (Section 1.3(3))05000440/2005005-05 FIN Failure to Quarantine Equipment as Required by

Procedures (Section 2.2(1))05000440/2005005-06 FIN Failure to Evaluate and Document the Removal of a Mode

Restraint as Required by Procedures (Section 2.2(2))05000440/2005005-07 FIN Failure to Quarantine Equipment and Perform

Troubleshooting Without Full Benefit of a Troubleshooting

Plan (Section 2.4(1))

Closed

05000440/2005005-01 NCV Failure to Incorporate Industry Operating Experience into

Preventative Maintenance Activities (Section 1.2)05000440/2005005-03 NCV Failure to Provide Guidance to Refurbish Breakers Within

Vendor-specified Time Frames or to Provide Reasonable

Alternative Preventative Maintenance Practices to Ensure

That Safety-related Breakers Remained Operable

(Section 1.3(2))05000440/2005005-04 NCV Failure to Incorporate the Vendors Gap Requirements into

Operations Manual Instructions (Section 1.3(3))05000440/2005005-05 FIN Failure to Quarantine Equipment as Required by

Procedures (Section 2.2(1))05000440/2005005-06 FIN Failure to Evaluate and Document the Removal of a Mode

Restraint as Required by Procedures (Section 2.2(2))

Attachment 1

05000440/2005005-07 FIN Failure to Quarantine Equipment and Perform

Troubleshooting Without Full Benefit of a Troubleshooting

Plan (Section 2.4(1))

Discussed

None.

Attachment 1

LIST OF DOCUMENTS REVIEWED

The following is a list of documents reviewed during the inspection. Inclusion on this list does

not imply that the NRC inspector reviewed the documents in their entirety, but rather that

selected sections or portion of the documents were evaluated as part of the overall inspection

effort. Inclusion of a document on this list does not imply NRC acceptance of the document or

any part of it, unless this is stated in the body of the inspection report.

Procedures and Documents:

ARI H13-P680-0006-A3; OPRM Annunciator Response Instruction; Revision 3

C51AP-OPRM; OPRM Lesson Plan; Revision 0

GEI-0135; Perry Operations Manual, Generic Electrical Instruction, ABB 5 kV Power Circuit

Breakers, Types 5HK250 and 5HK350 Maintenance; Revisions 1 and 7

GEI-0136; Perry Operations Manual, Generic Electrical Instruction, ABB 15 kV Power Circuit

Breakers, Type 15HK1000 Maintenance; Revisions 1 and 4

NOBP-LP-2011; FENOC Root Cause Analysis Reference Guide; Revision 2

NOBP-OM-4010; Restart Readiness for Plant Outages; Revision 1

NOBP-OM-4010; Attachment 6; Restart Recommendation Approval (with all enclosures);

2/16/04

NOP-ER-3001; Problem Solving and Decision Making Process; Revision 0

NOP-LP-2001; Condition Report Process; Revision 8

ONI-C51; Unplanned Change in Reactor Power or Reactivity; Revision 17

PDB-A0006; Plant Data Book - Power Flow Map; Revision 10

PNPP Stability Project Completion - Lesson Plan; 6/21/2004

PYBP-SITE-0014; Operability Determination Reference Guide; Revision 2

Memo Attention Perry Plan Team from Rich Anderson to the Plant

Root Cause Task Assignment Document for CR 04-06766; 1/13/05

Plant Outage Notebook Contents from December 23-26, 2005 outage; Various Documents

Perry Nuclear Power Plant Electrical Element Diagram 208-0006-00001, 15kV Switchgear

Internals; Revision A

PNPP Switchgear Cubicle Measurements, Measurement Report - Cubicles L1210 and L1007;

dated January 27, 2005

Ten Year 480V Breaker Overhaul PM Tasks for R23 System (Sorted by Quality Class); dated

January 26, 2005

4.16kV Breaker Overhaul PMs; dated January 12, 2005

Charging Motor Failure to Charge Breaker Closing Springs by Cubicle; undated

List of 13.8kV Breakers; dated January 29, 2005

Ten Year Breaker Overhaul PM Tasks for R22 System (Sorted by Quality Class, Voltage, and

Cubicle); dated January 20, 2005

FirstEnergy BETA Laboratory, File No. 05-95223; Grease Sample Analysis from Perry Breaker

L1007; dated January 24, 2005

MDFP Breaker Team Organization; dated January 19, 2005

Problem Solving Plan (CR 05-00095); Motor Feed Pump Breaker Failure to Charge; Revision 0

Outage Control Center Log; dated January 25, 2005

MFP breaker 1007 IRT End of Day Shift Goals; dated January 24, 2005

Problem Solving Plan; Bus L10 Energized Testing and Inspection; Revision 7a-2000; dated

January 23, 2005

ABB Supplemental Guidance for IB 6.2.1.7D; Control Device Adjustment; dated May 31, 2000

Attachment 1

ABB Fax from Coy Haynes to Dave Workman; Factory Specifications on the Control Device

Adjustment on the 15HK 1000; dated January 8, 2005

ABB Supplemental Guidance for IB 6.2.1.7D; Operational Parameters of ABB Control Device

191921; dated January 22, 2005

ABB MS3.2.1.9-1D; Maintenance and Surveillance Medium Voltage Switchgear Equipment;

undated

IB 6.2.1.7D ABB Installation/Maintenance Instructions Medium-Voltage Power Circuit Breakers,

Type 5HK 1200 thru 3000 Amperes 5000 Volts; undated

IB 6.2.2.7-1G ABB Installation/Maintenance Instructions Medium-Voltage Power Circuit

Breakers, Type 7.5HK500, 15HK500 and 15HK750, 1200 thru 3000 Amperes, 7500 and 15000

Volts; undated

SOER 98-2; Circuit Breaker Reliability; dated September 22, 1998

Condition Reports:

89-010; Failure of Circulating Water Pump A to Start (Breaker L1105)

91-0174; Failure of MFP to Start From Its Control Room Control Switch (Breaker L1007)

2-0011; L1007 Failure to Close

2-244; Revised Maintenance and Surveillance Manuals for ABB Medium Voltage Switchgear

Were Not Being Updated at Perry

NR 94--503; Breakers L1006 (Supply to L11 Bus) and L1009 (Supply to L12 Bus) Closing

Springs Did Not Recharge

94-0306; Lack of Progress to Refurbish ABB 13.8 kV, 4160 kV, and 480V Breakers94-901; Charging Spring for the 13.8 kV L1103 Breaker (Feed Breaker to Interbus Transformer

LH-1-B) is Not Charged

99-2013; Charging Springs for Breaker EF1C03 Did Not Charge When Racked Into the

Connected Position

00-1483; Beaver Valley - ABB 5HK 3000 Amp Circuit Breaker Failed to Close on Demand

04-01207; OE17906 5kV ABB HK Breaker Charging Spring Cycling

04-06763; Post Scram Restart Report

04-06766, Reactor Recirculation Pump Speed Shift

04-06799; Evaluate Operator Response to Recirc Pump Downshift

04-06811; Just in Time Training - Recirc Pump Downshift Event

05-00094; Reactor Recirculation Pump Speed Shift

05-00095; MFP Breaker Failure to Charge

05-00096; Post Scram Restart Report

05-00187; PY-C-05-01: Grace Periods Beyond Maximum Intervals Are Non-Conservative

05-00191; RFA Control Devise Shim Material 4.16kV Breakers

05-00121; Main Turbine Generator May Not Have Tripped as Expected

05-00145; Evaluate Decision Making Following December 23 Scram

05-00198; PY-C-05-01: Plant Health Report May Not Be Accurate

05-00219; Prob. Solving During CR 04-06766 Immediate Investigation was Incomplete

05-00224; Control Device Does Not Repeat to Its Original Position on L2001 Breaker

05-00225; Control Device Does Not Repeat to Its Original Position on L1105 Breaker

05-00229; Control Device Does Not Repeat to Its Original Position on L2004 Breaker

05-00230; 5 & 15kv Breakers Beyond Their Due Date for 10 Year Overhaul (NRC Identified)

05-00239; Quarantine of Components Needs Improvement (NRC Identified)

05-00251; Documentation of Flow Charts After a Plant Transient (NRC Identified)

05-00253; 5 & 15kv Breakers Beyond Their Due Date for 10 Year Overhaul

05-00272; Level Control Following the 12/23/04 SCRAM (NRC Identified)

Attachment 1

05-00274; 5 percent 15kv Breaker Cycles Since Last Overhaul

05-00283; 480 Volt K-Line Breakers Scheduled Beyond Their Due Date for 10 Year Overhaul

05-00290; RX Recirc Pump A Breaker (L1106) Aux. Switch Requires Adjustment

05-00295; Breaker 10 Year Refurbishment/Programs

05-00313; RFA - Review Vendor Procedures for Breaker Refurbishment

05-00359; 5kV and 15kV Data Sheets for 10 Year Overhauls Missing Cycle Counter Reading

05-00364; Safety Related 4.16kV Breaker Issues

05-00368; RFA - Document Decision Process/Criteria for 5kV Breaker EOC for CR 05-00095

05-00396; Refurbished Breaker Improperly Re-assembled

05-00429; PY-C-05-01, CR-05-00230 & CR 05-00283 Missing Tracking CAs for OD

05-00449; RFA-Meeting Intent of GEI-0136 (Section 5.14.5.1)

05-00450; PY-C-05-01, CR-05-00230 & CR 05-00283 Operability Determination Shortfalls

05-00459; Bases for PM Frequency Not Adequately Identified (NRC Identified)

05-00460; MFP Breaker Charging Springs Fail to Charge During PMT

05-00489; OE17906 - 4kV ABB HK Breaker Charging Spring Cycling

05-00496; Restart Readiness for Plant Outages Potential Weaknesses (NRC Identified)

05-00522; Refurbished Breaker Aux. Switches Not Lubricated

05-00529; RFA: Fast Closure Time for HPCS Breaker EH1304

05-00548; Switchgear Internal Drawings (B-208-0006-00000 and 00001)

05-00575; PY-C-05-01: Negative Trend Regarding Quarantining Items

05-00611; L1210 Limiter Plate Bracket Spot Welds Broken

05-00618; 15kV Breaker Refurbishment (L1105-CWP A)

05-00654; Breaker L1007 Truck Bent at Left Trip Spring Point

05-00679; Foot Cause Investigation CR 05-00095 Failed to Identify Root Cause of Failure (NRC

Identified)

05-00707; Refurbished Breaker Not Acceptable for Use

05-00773; PY-C-05-01 - Common Cause Analysis for Self-Revealed Equipment Problems

05-00794; Extent of Condition is Narrowly Defined in NOP-LP-2001

05-00984; Vendor Independent Oversight of Activities May Be Deficient (NRC Identified)

05-00985; NRC Special Inspection Debrief: Infrequently Used Procedures (NRC Identified)

05-00986; NRC Special Inspection Debrief: CRs Identified by External Oversight

(NRC Identified)

05-00987; NRC Observation: Documenting Plant Equipment Deficiencies (NRC Identified)

05-00988; NRC Special Inspection Debrief: Team Observations Not Captured In Condition

Reports (NRC Identified)

05-00989; NRC Special Inspection Debrief: Restart Pressure may Have Affected 12/23

T/S Plan (NRC Identified)

05-00993; NRC Special Inspection Debrief: Root Cause Procedure Could Result In a Narrow

Investigation (NRC Identified)

05-00994; NRC Special Inspection Debrief: Condition Report 05-00095 Investigation

(NRC Identified)

05-01232; NRC Special Inspection Exit: MFP [Post-Maintenance Testing] PMT Performed Prior

to Investigation Completion (NRC Identified)

05-01235; NRC Special Inspection Exit: Finding: Equipment Quarantine Process Is Inadequate

(NRC Identified)

05-01240; NRC Special Inspection Exit: Unresolved Issue for Breaker Refurbishment

(NRC Identified)

05-00282; PY-C-05-01: Ineffective Investigation: CR 02-03972, [High Pressure Core Spray]

Attachment 1

HPCS Pump Failed to Start

Work Orders:

870008040; Charging Spring Motor Wont Energize, 13.8kV BOP Bus L-22; Revision 0;

dated September 18, 1987

900000991; Modify Breaker L1007 O/C Trip CKT/DCP 89-42C Motor Driven Feedwater

Pump; dated May 2, 1990

9100004063; Pump Failed to Start, Motor Driven Feedwater Pump; dated August 3,

1991

20000452; Springs for

M. Feed Pump Breaker Not Charged, Motor Driven Feedwater

Pump; dated January 29, 1992

20000637; 13.8kV Breaker Tests Using Spare Cub. L2110; Revision 0; dated

February 14, 1992

99-008798-000 (Repetitive); Metal Clad Switchgear (15kV and 5kV) Load Center

L10 Feed to 1N27C0004; Revision 0; dated October 5, 1999

200135473; Work In Progress Log; dated January 6, 2005

200135854; HPCS Pump Breaker EH1304; Verify Control Device Adjustment Gap to

Support Extent of Conditions for CR 05-00095; dated January 9, 2005

200136531; Control Rod Drive Pump A Breaker XH1101, Perform Breaker Inspection

Per the Test Plan; dated January 12, 2005

200136532; ABB 5HK Circuit Break (For Recently Refurbished and 10 Year Service

Life)

Test and Inspection Guide, Serial No. 04674A-01393, Attachment No. 2; dated

January 12, 2005

200137234; Work In Progress Log; dated January 22 - 23, 2005

Attachment 1

LIST OF ACRONYMS

ABB Asea Brown Boveri

APRM Average Power Range Monitor

ARI Annunciator Response Instruction

CA Corrective Action

CFR Code of Federal Regulations

CR Condition Report

DRS Division of Reactor Safety

EAB Engineer Assessment Board

EPRI Electric Power Research Institute

ESW Emergency Service Water

FENOC FirstEnergy Nuclear Operating Company

FIN Finding

GE General Electric

GEI General Electrical Instructions

HPCS High Pressure Core Spray

I&C Instrumentation and Control

IMC Inspection Manual Chapter

INPO Institute of Nuclear Power Operations

kV Kilo-Volt

LFMG Low Frequency Motor Generator

LPRM Local Power Range Monitor

MCPR Minimum Critical Power Ratio

MFP Motor Feedwater Pump

MSIV Main Steam Isolation Valves

NCV Non-Cited Violation

NMAC Nuclear Maintenance Assistance Center

NRC Nuclear Regulatory Commission

OD Operability Determination

OE Operating Experience

ONI Off-Normal Instruction

OPRM Oscillating Power Range Monitor

PNPP Perry Nuclear Power Plant

PMT Post-Maintenance Testing

RC Resistor/Capacitor

RCIC Reactor Core Isolation Cooling

RCS Reactor Coolant System

RFO Refueling Outage

RHR Residual Heat Removal

RRP Reactor Recirculation Pump

Rx Reactor

SAT Systematic Approach to Training

SDP Significance Determination Process

SN Serial Number

SOER. Significant Operating Experience Report

SRM Source Range Monitors

SRO Senior Reactor Operator

TS Technical Specification

Attachment 1

URI unresolved item

V Volt

VAC Volts Alternating Current

VDC Volts Direct Current

WO Work Order

Attachment 1

January 7, 2005

MEMORANDUM TO: Ray Powell, Senior Resident Inspector, Perry

Division of Reactor Projects

Stephen Burton, Senior Resident Inspector, Monticello

Division of Reactor Projects

FROM: Mark

A. Satorius, Director /RA/

Division of Reactor Projects

SUBJECT: SPECIAL INSPECTION CHARTER FOR REACTOR

RECIRCULATION PUMPS DOWN-SHIFT, AND RESULTING

MANUAL SCRAM, AND OTHER EQUIPMENT ISSUES AT THE

PERRY NUCLEAR POWER PLANT ON JANUARY 6, 2005

On January 6, 2005, at about 0106 (EST), the Perry Nuclear Power Plant was operating at full

power when Reactor Recirculation Pumps A and B unexpectedly down-shifted from fast to

slow speed which resulted in reactor power decreasing from 100 percent to approximately

percent. As operators started to respond to this event using control rod insertion, Reactor

Recirculation Pump A tripped. At 0112, a manual reactor scram was inserted due to operating

at undesirable power to flow conditions. Following the manual scram, the turbine driven

Feedwater pumps tripped as a result of high reactor vessel water level. The operators were

unable to start the Motor Feedwater Pump. The Reactor Core Isolation Cooling (RCIC) system

and a Turbine Driven Reactor Feed Pump were manually started and reactor water level control

was established. The Main Steam Isolation Valves were subsequently closed by operators to

limit the cooldown rate, and reactor level was controlled with RCIC. All control rods fully

inserted. The lowest reactor water level reached was 154 inches above Top of Active Fuel.

The licensee decided to cooldown the plant to Mode 4 (Cold Shutdown).

The cause of the Reactor Recirculation Pumps down-shifting and the subsequent trip of Reactor

Recirculation Pump A is still under investigation. This event was similar to the transient which

occurred on December 23, 2004, because the reactor recirculation pumps down-shifted

unexpectedly in both events. As a result, the January 6, 2005 transient was determined to meet

the criteria of Management Directive 8.3, NRC Incident Investigation Program to warrant the

establishment of a special inspection team.

Based on the criteria specified in Management Directive 8.3 (Part I criterion (g)) and

Inspection Procedure 71153, a Special Inspection was initiated in accordance with Inspection

Procedure 93812 and Regional Procedure RP-1219. The Special Inspection will commence on

January 7, 2005, and be led initially by Ray Powell. On Monday January 10, 2005, Stephen

Burton, Senior Resident Inspector at Monticello, will replace Mr. Powell as team leader.

Attachment 2

R. Powell/S. Burton -2-

Mr. Burton may draw upon the resources of the Perry Senior Resident Inspector, Ray Powell.

In addition to Mr. Burton, the team will add Stuart Sheldon DRS, Electrical/Instrumentation and

Controls as of January 8, 2005, and Keith Walton, DRS, Operating Licensing Examiner on

January 10, 2005, as members. Also, David Reeser, Operations Branch, will participate as an

observer and resource.

The Special Inspection will evaluate the facts, circumstances, and licensee actions surrounding

the January 6, 2005, and December 23, 2004, events. Elements of this inspection should

include the cause of the Reactor Recirculation Pumps down-shifting, the trip of Reactor

Recirculation Pump A and the subsequent manual scram, and the cause of Motor Feedwater

Pump failure to start, for the January 6, 2005 event. Additionally any relationship to the events

which occurred on December 23, 2004, should be evaluated. The team should focus on

assessing the adequacy of the licensees efforts to resolve the identified equipment problems. A

Charter was developed and is attached. An entrance meeting will be conducted on Friday,

January 7, 2005.

Attachment: As stated

cc w/att:

S. Sheldon, DRS, Electrical/Instrumentation and Control

K. Walton, DRS, Operating Licensing Examiner

W. Ruland, Project Director, Project Directorate III, NRR/DLPM

C. Pederson, DRS, Division Director

J. Caldwell, Regional Administrator Region III

G. Grant, Deputy Regional Administrator Region III

D. Weaver, Region III EDO Coordinator

DOCUMENT NAME: C:\MyFiles\Roger\ML050830113.wpd

To receive a copy of this document, indicate in the box: "C" = Copy without attachment/enclosure "E" = Copy with attachment/enclosure "N" = No copy

OFFICE RIII RIII NRR RIII

NAME CLipa for CLipa CLipa for (email) MSatorius

RSmith:dtp WRuland

DATE 01/07/05 01/07/05 01/07/05 01/07/05

OFFICIAL RECORD COPY

Attachment 2

PERRY SPECIAL INSPECTION (SI) CHARTER

This Special Inspection is chartered to assess the circumstances surrounding the January 6,

2005, and December 23, 2004, events. The inspection should include the cause of the Reactor

Recirculation Pumps down-shifting, the trip of Reactor Recirculation Pump A and the

subsequent manual scram, and the cause of Motor Feedwater Pump failure to start, for the

January 6, 2005, event. Additionally any relationship to the events which occurred on

December 23, 2004, should be evaluated. The January 6, 2005, event was similar to the

December 23, 2004, transient, where both reactor recirculation pumps also down-shifted

unexpectedly. The scram and loss of feedwater transient involved repetitive failures of the

reactor recirculation system. The Special Inspection will be conducted in accordance with

Inspection Procedure 93812, Special Inspection, and will include, but not be limited to, the

following items:

1. Establish a sequence of events of the January 6, 2005 event, including the Reactor

Recirculation Pumps down-shifting and the subsequent trip of Reactor Recirculation

Pump A resulting in a manual scram, and the failure of the Motor Feedwater pump to

start. (Event Number 41310)

2. Establish the sequence of events for the recirculation down-shift and scram on

Oscillation Power Range Monitors (OPRM) which occurred on December 23, 2004.

(Event Number 41290)

3. Interview operators, that were involved in both events. Determine and assess the

actions taken by the crews during both events, including the instruments monitored in

the control room for operations in the Immediate Exit Region of the Power to Flow Map,

procedure usage, and the decision-making associated with shutting the Main Steam

Isolation Valves following the January 6, 2005, scram. Specifically evaluate the

appropriateness of the actions the operators took following the entry into the undesirable

power to flow condition during the December 23, 2004, scram.

4. Monitor the licensees root cause determination of the down-shifting of the Reactor

Recirculation Pumps, trip of the A Recirculation pump, and the failure of the Motor

Feedwater pump to start. Consider including in your evaluation: the troubleshooting

plan used after the first down-shift event that occurred on December 23, 2004, and

subsequent corrective actions; and the troubling shooting and corrective actions for the

January 6, 2005, scram, and potential vulnerability to a repeat occurrence.

5. Evaluate the licensees efforts to determine the extent of condition for root causes for the

events of December 23, 2004, and January 6, 2005.

6. Assess the adequacy of the licensees implementation of the OPRM modification as

related to the December 23, 2004, and January 6, 2005, events. In this assessment

include operator training, procedures, and instrumentation.

7. Assess the adequacy of the root cause evaluation of the recirculation pump

down-shifting event of December 23 that supported the decision to restart from the

December 23 event, including the adequacy of compensatory measures taken by the

licensee given their inability to identify the specific causes.

Attachment 2