Text

Report: Unplanned Isolation of Spent Fuel Cooling (CAP056010)
	ML050870112
Person / Time
Site:	Point Beach
Issue date:	04/23/2004
From:	Guay T; - No Known Affiliation
To:	; Office of Nuclear Reactor Regulation
References
	CAP056010, FOIA/PA-2004-0282, RCE000256
	Download: ML050870112 (20)
	v • d • e

P B-E COH LA POINT BEACH NUCLEAR PLANT RCE000256 CAP056010 Principle Investigators:

Terry Guay, Team Leader Approvals:

Team Leader Date Management Sponsor Date A10,11hop"Y u

I X,;

RCE000256, Unplanned Isolation Of Spent Fuel Cooling Table of Contents Section Start Page I. Executive Summary............................................

3 II. Event Narrative............................................

6 III. Extent of Condition Assessment............................................

II IV. Nuclear safety Significance...........................................

14 V. Reports to External Agencies & the NMC Sites................................... 14 VI. Data Analysis

Information and Fact Sources.....................................

14

Evaluation Methodology & Analysis techniques......................... 19

Data Analysis Summary.....................................

19

Failure Mode Summary.....................................

20 VII. Root Causes 22 VIII. Contributing Causes

.......................... 22 IX. Corrective Actions............................

23 X. References............................

24 XI. Attachments A. ACEMAN Barrier Analysis..........................

27 B. ACEMAN "Picture of Excellence..........................

28 C. Cause/Corrective Action Table..........................

29 D. Root Cause Evaluation Charter..........................

30 2

RCE000256, Unplanned Isolation Of Spent Fuel Cooling I. Executive Summary

Purpose:

This evaluation is to determine how and why Point Beach Nuclear Plant (PBNP) Spent Fuel Pool (SFP) Cooling was isolated during "Bolted Faults" work when an Electrical Maintenance craft individual opened an incorrect breaker that was adjacent to the breaker that was to be opened.

Event Synopsis:

Electrical Maintenance (EM) personnel have been performing breaker cubicle stiffening as part of a project to reduce the potential for a seismic event involving breaker cubicle equipment (Bolted Faults Project.) This work involves a relatively large number of breaker manipulations. To the date of the event, breaker manipulations had been performed by PBNP EM personnel without incident.

EM personnel completed a detailed pre-job briefing including the use of the "Are You Ready Checklist" and proceeded to the _

work area. At the work area peer checking was applied as the correct breaker to open was identified. The EM Lead, who was also the breaker manipulator then proceeded away from the work area to a phone at which time he contacted the control room and reported that he would be opening breaker 2B52-321 IM (W21B Aux Bldg Fan). Following receipt of authorization to proceed with breaker opening this same EM Lead then returned to the work area and opened breaker 2B52-3212M (A SFP Cooling Pump) instead of Breaker 2B52-321 IM (W21B Aux Bldg Fan). These breaker were located adjacent to each other.

At 1705 on April 23, 2004, during Auxiliary Operator rounds, SFP cooling flow was found to be zero. The 'A' SFP cooling Pump (P-12A) had been operating previously.

The Auxiliary Operator reported the information to the control room and AOP-8F, "Loss of Spent Fuel Pool Cooling", was entered at 1705. Follow-up identified the breaker for P-12A SFP Cooling Pump was found in the "OFF' position.

At 1711, P-12B ('B' SFP Cooling Pump) was started per AOP-8F, step 3a, initiating 1250 gpm SFP cooling flow.

Operations personnel normally manipulate breakers at PBNP, as a routine function of their job. As a normal part of the job, Operators have developed specific error-reduction habits to ensure the work is performed safely and event free. As work load and programmatic changes have increased the demands on Operations personnel time, alternate ways to accomplish work have been considered. EM personnel have attained danger tagging qualifications to ease the Operations work burden. Although not a normal part of the Mechanic/Electrician job, EM personnel have taken on some electrical equipment manipulations that have normally been done by Operators. The frequency of breaker manipulation by qualified EM personnel has increased over the past few years as Operations has gained confidence in EM ability to conduct that activity safely and 3

RCE000256, Unplanned Isolation Of Spent Fuel Cooling without incident. This was taken into account during the planning and preparations for the 'Bolted Fault" project work being done during U1R28. Human error prevention tools (peer checking independent checking, etc.) were not institutionalized in "Bolted Fault" work control documentation during its development to prevent component manipulation errors.

In this specific case the involved personnel used error-prevention tools on multiple occasions during the briefing and start of the job. Unfortunately, a distraction diverted the EM Lead's attention away from the correct breaker and no additional error prevention tool was implemented following the distraction prior to breaker manipulation. The EM Lead then operated the incorrect breaker.

==

Conclusions:==

The EM Lead lost track of the correct breaker following the distraction of calling the control room from a phone not in the immediate area of the subject breaker and did not implement an error-reduction tool to ensure he was on the correct component prior to manipulation.

Nuclear Safety Sign ificance:

During the approximate 3.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months that SFP Cooling was isolated the SFP temperature increased from 83 to 87 degrees (4 degrees). The SFP temperature high alarm is set at and was never challenged. SFP temperature high alarm is set at approx. -

degrees below the technical specification temperature limit of

. Therefore, this event involved essentially no Nuclear Safety Significance.

Root Causes:

Contributing Causes:

Corrective Action Synopsis:

Upon discovery of SFP Cooling Isolation, Operations dispatched an individual to the associated SFP Cooling Pump breakers and the "OFF' status of the 'A' SFP Breaker was expeditiously identified. Essentially coincident with this discovery, Operations realized the likelihood that the incorrect breaker had been manipulated during the "Bolted Faults" work. Operations then closed breaker 2B52-32 1 2M, restarting the 'A' SFP Cooling Pump, and opened 2B52-321 IM (W21B Aux Bldg Fan), the intended "Bolted Faults" breaker.

"Bolted Fault" work was stopped pending completion of a "Human Performance Investigation" (HPI) and establishment of robust corrective action.

4

RCE000256, Unplanned Isolation Of Spent Fuel Cooling Operations then performed a verification of breaker positions in steps 3.6.4 through 3.6.10 of IWP 01-128*E-FN MCC IB42 Bus Bracing Installation.

The HPI was completed Additional corrective actions are described on pages 22 through 24 of this report.

II. Event Narrative 5

RCE000256, Unplanned Isolation Of Spent Fuel Cooling Date/Time Action Comment 4/32/04

-0540hrs Mtn work crew assigned to work "Bolted Faults" project Mtn Crew performs pre-job briefing Mtn Crew Starts work for shift 1430hrs Ops gives permission to place Step written to be performed by bkr 2B52-3211M (W21B-Aux Ops or EM Bldg Fan) in 'OFF' position 1705hrs AO finds SFP Cooling Flow @ 0 Immediately informs Control Room 1705hrs Ops enters AOP-8F "Loss of Speng Fuel Cooling" 1711hrs Ops starts P-12B SFP Cooling 1250 gpm Flow Observed Pump per AOP-8F Step 3a RNO Ops directs Mtn work on IWP 01-128*E-FN for MR01-128*E MCC 1B42 Bus Bracing Installation to Stop Mtn stops work per Ops directive 1732hrs Ops Recloses Bkr 2B52-3212M (P-12A) to restore power to P-12A SFP Cooling Pump 1733hrs Ops places Bkr 2B52-3211M Note added to IWP record copy (W21-Aux Bldg Fan) to "OFF' indicating that the Bkr had been Position found in the "ON" position.

Time/date Bke placed in "OFF' position also recorded.

Mtn initiates performance of a Human Performance Investigation III. Extent of Condition Assessment This issue is being evaluated against the criteria for being a Significant Condition Adverse to Quality.

Previous Similar Events 6

RCE000256, Unplanned Isolation Of Spent Fuel Cooling IN7. Nuclear Safety Significance V. Reports to External Agencies This issue did not result in any External Agency Reporting.

It is recommended that an industry notice and/or report to other NMC sites on the issues be initiated.

VI. Data Analvsis Infonnation & Fact Sources Documentation Facts During UIR28 following movement of all 80 fuel assemblies from the Reactor to the SFP the time to boil, following loss of SFP cooling is 20 hours2.314815e-4 days 0.00556 hours 3.306878e-5 weeks 7.61e-6 months .

A Tagging Qualified Mechanic Electrician Placed the Incorrect Bkr in the "OFF' Position The Intended Bkr to be Manipulated was located next to the Incorrect Bkr that was manipulated Interview Facts Involved Mtn personnel ran through the "Are You Ready Checklist" error prevention tool during their pre-job briefing.

Involved Mtn personnel performed peer checks to identify the Bkr intended to be manipulated when they initially arrived at the following a "2-Minute Timeout" to again run through the "Are You Ready Checklist" error-prevention tool.

Evaluation Methodology & Analysis Techniques A Human Performance Investigation was performed and documented using the station Human Performance Investigation tool. Personnel statements were obtained from the involved Electrical Maintenance personnel. Reviews of work control documentation, 7

RCE000256, Unplanned Isolation Of Spent Fuel Cooling internal and external operating experience, and various other station documents was a foundation of this evaluation. An "ACEMAN" barrier analysis was performed to identify pluses and deltas with prior successful similar evolutions. Failure modes were identified from relevant facts and inappropriate actions. Failure modes were then assessed to determine Root and Contributing Causes.

Data Analysis Summary Failure Mode Summary Organization and Management Failure Modes I

I I

Human Pejrmonance Failure Modes I

I I

VNII. Root Causes VIII. Contributing Causes 8

RCE000256, Unplanned Isolation Of Spent Fuel Cooling IX. Corrective Actions Interim Corrective Actions (mitigation)

Corrective Actions Completed or in Progress Corrective Actions to Prevent Reoccurrence (CATPR)

Additional Corrective Actions Effectiveness Review XI. References CAP056010, "P-12A Spent Fuel Pump breaker found OFF-results in loss of spent fuel cooling" CAP056039, "PBF-1562 requires eval. of SFP Cool ONLY during FULL CORE OFFLOAD" OM 3.4, "Operations Self-Checking Expectations", page 3 of 4 pages Work Order Work Plan, MR 01-128*E: MCC 1B42 BUS BRACING INSTALLATION

- CR/PAB Fan Realignment (IWP 01-128*E-FN) - Copy of Original Test Work Order Work Plan, MR 01-128*E: MCC IB42 BUS BRACING INSTALLATION

- CR/PAB Fan Realignment (IWP 01-128*E-FN) - Copy of Retest 9

RCE000256, Unplanned Isolation Of Spent Fuel Cooling XI.

Attachment A - ACEMAN Barrier Analvsis 10

RCE000256, Unplanned Isolation Of Spent Fuel Cooling XI.

Attachment B - ACEMAN "Picture of Excellence" 11

RCE000256, Unplanned Isolation Of Spent Fuel Cooling XI.

Attachment C - Cause/Corrective Action Table 12

RCE000256, Unplanned Isolation Of Spent Fuel Cooling XI. Attachment D - Root Cause Evaluation Charter 13

Loss of SFP Cooling Due to Breaker Miss Alignment Self-revealing fifth 4

0 a.YS Cb be-lja,%'

0/

V

.ci I_

f I/

AO y

)V

,r

Loss of SFP Coolina Due to Breaker Miss Alianment Information requested from Licensee. RCE in progress.

Self-revealing

==

Introduction:==

On April 23, 2004 an auxiliary operator discovered that the spent fuel pool (SFP) cooling flow was zero. The 'A' SFP cooling pump (P-12A) had been operating previously. The auxiliary operator reported the information to the control room and the operators entered AOP-8F, "Loss of Spent Fuel Pool Cooling," procedure. The operators identified that the breaker for P-12A was in the "OFF' position. Per AOP-8F, step 3a, P-1 2A, SFP cooling pump was started.

==

Description:==

Analysis:

Enforcement:

-.QNuclear Management Company Pagel1 of 2 E State Change History MRE Assign Work Initiate 4/26/2004 12:36:11 0

Owner by KREIL,

HENNESSY, JULIE WILLIAM Complete A and Close Done 5/2512004 o10:52:53 by KREIL.

Owner (None)

JULIE T

1 Section 1 t.

L Assign 0

by

HENNESSY, WILLIAM Conduct Work 4/27/2004 8:06:09 Owner

BARTEL, MICHAEL Work Complete 0

by BARTEL, MICHAEL Review &

Approval 5/1412004 10:43:07 Owner

HENNESSY, WILLIAM Approved 0

by

HENNESSY, WILLIAM Quality Check 5/1512004 1:52:47 Owner PBNP CAP Admin I"-

I.

Activity Request Id:

Activity Type:

Site/Unit:

i One Line

Description:

Activity Requested:

i CATPR:

Initiator Department:

MRE000194 Maintenance Rule Submit Date:

4/26/2004 12:36:11 Evaluation Point Beach -

Common P-12A Spent Fuel Pump breaker found OFF - results in loss of spent fuel cooling Per 4-26-04 Managers Meeting, perform Maintenance Rule Evaluation. 30 day due date.

(Ref. CAP056010).

N PO Operations PB Initiator:

JESSESSKY, TOM 3

Responsible Group Code: EESN Engineering Equipment Systems NSSS Mech PB Activity Supervisor:

HENNESSY, WILLIAM 3

Responsible Department: Engineering Activity Performer:

BARTEL, MICHAEL E Section 2 Priority:

3 Due Date:

5/26/2004 4 Mode Change Restraint: (None)

Management Exception From PI?: N OQA/Nuclear Oversight?: N V Licensing Review?:

N NRC Commitment?:

N D NRC Commitment Date:

0 Significance Level:

A E Section 3 Activity Completed: 5114/2004 10:43:07 - BARTEL, MICHAEL:

System:

SF Equipment ID:

OP-012A Work Order (WO):

0202527 AR:

CAP 56010 Affected Function: '2031 Ensure subcriticality, integrity and cooling of spent fuel. Remove decay heat from irradiate fuel stored in SF Pool after removal from RPV, Maintain SF."

FF=Y MPFF = Y Repetitive MPFF = N Did a component failure affect a system performance criteria? No Evaluation:

In the course of performing scheduled maintenance on the B-42 electrical bus, the wrong breaker was turned off. The breaker for OP-01 2A was taken to off instead of the intended breaker. The A-http://enw%,sO2/tmtrackltmtrack.dll?lsstuePaic&Tcmplatc=vticxvbody &,recordid=5995 11 &8tablci( 6/9/2004

Nuclear Management Company I

Page 2 of 2 train for spent fuel cooling was aligned for cooling at this time. Taking the breaker for OP-012A to off cause the pump to trip off and a subsequent loss of spent fuel cooling. Because the breaker was turned off, the train was no longer capable of performing the function requiring cooling of spent fuel.

Unavailability time is not tracked for this system, but spent fuel cooling was lost for about 3.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months prior to starting of the B-train cooling.

Because the A-train was aligned and was no longer able to perform its function, this is classified as a functional failure. Furthermore, because the loss of cooling was an error caused by Maintenance personnel during the performance of normal maintenance, this event is classified as a maintenance preventable functional failure.

This is not a repetitive functional failure. No components failed, the loss of spent fuel cooling was due to an inadvertent manipulation of a breaker.

Corrective actions will be determined and implemented through the associate root cause evaluation, RCE 256.

The criteria for SF 2 failures every 2 years. There have been no functions failures in the last 2 years. Therefore, the system remains a(2).

511512004 1:52:47 - HENNESSY, WILLIAM:

The requested evaluation is completed and is acceptable. The system remains a2. There are no other actions required. This Item may be closed.

5/1812004 8:54:24 - KREIL, JULIE:

Pending CAP Technical Panel review.

5/2512004 10:52:53 - KREIL, JULIE:

CLOSED per CAP Technical Review Panel (Bill Zipp, Joe Krentz, Ron Harper) meeting, 5-25-04.

Hot Buttons:

(None)

B Section 4 QA Supervisor: (None)

Licensing Supervisor: (None) 13 Attachments and Parent/Child Links s Subtask fromCAP056010: P-12ASpentuelPmp breaker found OFF - results in loss of spent fuel cooling by KREIL, JULIE (4/26/2004 12:36:13)

E Change History 5/1812004 8:54:24 by KREIL, JULIE Last Modifier Changed From ^-*.... To........

Activity Completed Changed From 'Original Text] To '[Appended:] Pending CAP Technical Panel review.'

5125)2004 10:52:53 by KREIL, JULIE Active/Inactive Changed From....... To *.......

Owner Changed From *-....

To ***...

Last Modified Date Changed From... **- To Last State Change Date Changed From *...

To.****.

Last State Changer Changed From ***'* To Close Date Changed From

To Activity Completed Changed From '[Original Textr To '[Appended: CLOSED per CAP Technical Review Panel (Bill Zipp, Joe Krentz, Ron Harper) meeting, 5-25-04.'

State Changed From *.*...* To *-*-

lhttp:!/cnwvsO2/tmtrackl/tmtrack.dll?lssutcPagc&Templatc=v icwbody&-recordid=5995 1 1 &tabl(... 6/9/2004

II t

I........

t ok 0-E State Change Histo Initiate X

0 byJESSESSKY,TOM E Section 1 AR Pre-Screen S u it to Screening Team 4/23/2004 6:43:42 PM C

1 Owner (None)

P, by HARPER, RON w l-AR Screening Que 4/23/2004 8:06:16 PM

)wner PBNP CAP Admin i

id Activity Request Id:

Activity Type:

0 One Line

Description:

0 Detailed

Description:

Initiator:

CAP056010 CAP Submit Date:

4/23/2004 6:43:42 PM P-12A Spent Fuel Pump breaker found OFF - results in loss of spent fuel cooling 4/23/2004 6:43:42 PM - JESSESSKY, TOM:

At 1705 on 4/23/2004, during Aux Operator rounds, spent fuel pool cooling flow was found to be zero. The 'A" Spent Fuel Pool Pump had been operating previously. The Aux Operator reported the information to the control room and AOP8F Loss of Spent Fuel Pool Cooling was entered at 1705.

At 1711, Started P-12B SFP Cooling Pump per AOP-8F Step 3a. RNO, flow 1250 GPM.

The breaker for P-12A Spent Fuel Pool pump was found in the 'OFF position.

At 1716, the SFP Level was logged as stable at 63.8 feet with no indications of leakage. SFP temperature is 87 degrees and trending lower. The SFP experienced a 4 degree temperature rise during the loss of cooling from 83 degrees last shift.

At 1732, Breaker 2B52-3212M was re-closed to provide power to P-12A SFP Cooling Pump.

JESSESSKY, TOM Initiator Department:

PO Operations PB Z E~J Date/Time of Discovery: 4/23/2004 6:08:47 PM Identified By:

Equipment # (1st):

Equipment # (2nd):

Equipment # (3rd):

Site/Unit:

Why did this occur?:

Site-identified P-012A PB i (None)

(None)

Date/Time of Occurrence:

System:

Equipment Type (1st):

Equipment Type (2nd):

Equipment Type (3rd):

4/23/2004 6:08:47 PM SF PB CENTRIFUGAL (None)

(None)

Point Beach - Common 4/23/2004 6:43:42 PM - JESSESSKY, TOM:

IWP 01-128*E-FN for MR01-128*E MCC 1B42 Bus Bracing Installation - CRIPAB Fan Realignment Step 3.6.11 states the following:

"At MCC 2B32, ensure that W-21 B breaker 2152-3211 M is OFF."

The breaker was found in the "ON" position and the breaker immediately to the right of 11 M was found in the 'OFF' position -- 2B52-3212M is the breaker for P-12A Spent Fuel Pool Cooling Pump. Apparently the inproper breaker was positioned during execution of the step.

Operations had given electrical maintenance permission to perform the step at approximately 1430 (the step is written to be performed by OPS or EM). Thus, the loss of spent fuel pooling is estimated to be approximately 3.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months .

Immediate Action Taken: 4/23/2004 6:43:42 PM - JESSESSKY, TOM:

As noted above, AOP8F for Loss of Spent Fuel Pool cooling was entered, and spent fuel pool cooling restored. Immediate investigation and knowledge of what work was on-going resulted in identifying performance of step 3.6.11 as the likely cause of the event. A field walkdown confirmed the status of both breakers.

When the cause was identified, work was immediately stopped on the IWP and maintenance supervision initiated a human performance investigation tool.

At 1733, the step was reperformed by operations to place breaker 2B52-3211 M W-21 B in the

'OFF" position. A note was added to the IWP record copy indicating that the breaker had been found in the "ON' position and time and date of placing breaker in the 'OFF' position was also recorded.

http://enwsO2/tmtrack/tmtrack.dll?IssuePage&Template=viewbody&recordid=599079&ta...

4/24/2004

Recommendations:

0 Notify Me During Eval?:

E Section 2 Operability Status:

Basis for Operability:

1D Unplanned TSAC Entry:

8 Section 3 41 Cc re

N 23/2004 6:43:42 PM - JESSESSKY, TOM:

nduct human performance investigation tool and take appropriate actions to prevent currence.

I SRO Review Required?:

Y Inoperable

@ Compensatory Actions:

N 4/23/2004 8:06:16 PM - HARPER, RON:

SFP cooling pump P-12A was inoperable due to its circuit breaker being open. The redundant pump, P-12B remained operable but not running (requires manual operation to start it). SFP cooling was inadvertently secured for an estimated 3 hours3.472222e-5 days 8.333333e-4 hours 4.960317e-6 weeks 1.1415e-6 months .

N 4 External Notification:

N Screened?:

N 0 Significance Level: (None)

INPO OE Reqd?:

N Potential MRFF?:

N ii QA/Nuclear Oversight?: N V Licensing Review?: N Good Catch/Well Doc'd?: NA E Section 4 Inappropriate Action:

Process:

(None)

Activity:

(None)

Human Error Type:

(None)

Human Perf Fail Mode: (None)

Equip Failure Mode:

(None)

Process Fail Mode:

(None)

OrglMgt Failure Mode: (None) 0 Group Causing Prob:

(None)

Hot Buttons:

(None)

E Section 5 CAP Admin:

PBNP CAP Admin Prescreener:

(None)

@ Project:

Corrective Actn Program (CAP) AR i State:

AR Screening Que ii Active/inactive:

Active f) Submitter:

JESSESSKY, TOM 4 Owner:

PBNP CAP Admin '

Last Modified Date:

4/2312004 8:06:16 PM I Last Modifier:

HARPER, RON 0i Last State Change Date: 4/2312004 8:06:16 PM V Last State Changer:

HARPER, RON E)

Close Date:

NUTRK ID:

of Children:

0

References:

Update:

Prescreen Comments:

Import Memo Field:

OPR Completed?:

N OLDACTION-NUM:

sub_tsid:

0 originalissue_id:

Site:

Point Beach Cartridge and Frame:

8 Change History 4/23/2004 8:06:16 PM by HARPER, RON System Changed From (None) To SF PB Equipment # (1st) Changed From (None) To P-012A PB original-projectId:

0 http://enwsO2/tmtrackltmtrack.dll?lssuePage&Template=viewbody&recordid=599079&ta 4/424 4/24/2004

I Operability Status Changed From (None) To Inoperable Basis for Operability Changed From To [Appended:] SFP cooling pump P-12A was Inoperable due to Its circuit breaker being open. The redundant pump. P-12B remained operable but not running (requires manual operation to start It). SFP cooling was Inadvertently secured for an estimated 3 hours3.472222e-5 days 8.333333e-4 hours 4.960317e-6 weeks 1.1415e-6 months .

State Changed From AR Pre-Screen To AR Screening Que Via Transition: Submit to Screening Team Owner Changed From (None) To PBNP CAP Admin Last Modified Date Changed From 4/23/2004 6:43:42 PM To 4/23/2004 8:06:16 PM Last Modifier Changed From JESSESSKY, TOM To HARPER. RON Last State Change Date Changed From 4123/2004 6:43:42 PM To 4/23/2004 8:06:16 PM Last State Changer Changed From JESSESSKY, TOM To HARPER, RON http://enwsO2/tmtrack/tmtrack.dll?IssuePage&Template=viewbody&recordid=599079&ta...

4/24/2004

ML050870112

Text

Purpose:

Description:

Description:

Description:

References:

Navigation menu

Search