ML030790312

Internal NMC Correspondence from Cayia to Peveler, Potential AFW Common Mode Failure Vent Resolution Team

ML030790312
Person / Time
Site:	Point Beach
Issue date:	10/31/2002
From:	Cayia A Nuclear Management Co
To:	Peveler K Office of Nuclear Reactor Regulation, Nuclear Management Co
References
FOIA/PA-2003-0094
Download: ML030790312 (7)

Text

, Nf INTERNAL Conu, m*,cd ro

• uctoem Exce,,cnc CORRESPONDENCE NPM 2002-0583 To: Ken Peveler

• From: A. J. Cayia 4 Date: October'31,

Subject:

POTENTIAL A COMMON MODE FAILURE EVENT RESOLUTION TEAM Copy To: D. Schoon S. Thorns J.P. Cown J. Anderson J. Freels T. VandenBosch T. Coutu K. Davison R. Flessner R. Wood T. Kendall B. Zipp M. Rosseau File I am forming an event resolution team to provide management oversight to drive resolution of the issue involving the potential to have a common mode failure due to orifice plugging in the AFW minimum flow recirc line orifices and am assigning you as the team leader for this effort.

This issue is described in CAP 029952 attached. The following organization is established which will report to you to ensure that we effectively resolve this issue:

> Incident Investigation - Stu Thomas

>On-line Work Risk Management - John Anderson Interim Corrective Actions - Duane Schoon

> Issue Resolution Team and Root Cause Evaluation - Jim Freels Fritzie Flentje is assigned to support you in your efforts. I would like you to step out of your role as Nuclear Oversight Manager and assign your supervisors to act in your stead until this issue is resolved. I expect that schedules and actions will be established for each of the focus areas identified. Additionally, you, Fritzie Flentje and each of the managers above will comprise our Inspection Interface Team for the special inspection, which will commence Thursday, October 31, 2002.

The assignments and tasks below should continue to be revised as you pursue resolution of this issue. The assignments for each of the above efforts are as follows:

Incident Investigation - Stu Thomas The scope of this effort is to conduct an incident investigation to collect the facts surrounding the maintenance conducted on the P-38 A motor driven auxiliary feedwater pump, the post maintenance testing, the evaluation of the corrosion products discovered in the recirc orifice and concluding with the decision to declare the auxiliary feedwater pumps inoperable on Tuesday, October 29,2002. The incident investigation is to be completed using applicable portions of NP 5.3.3 "Incident Investigation and Post-trip Review" for guidance.

On-line Work Risk Management-John Anderson The focus of this effort is to ensure that we comply with procedure 10.3.7 "On-line Safety Assessment" while we are in elevated risk due to the potential for common mode plugging of

NPM 2002-0583 October 31, 2002 Page 2 the AFW recirc orifices. This effort should continue and provide you daily updates until interim corrective actions have restored on-line risk to a "green" condition.

Interim Corrective Actions - Duane Schoon The focus of this effort is to complete the interim corrective actions which are comprised of administrative controls, training for the operators, and procedure changes and also to perform an independent objective review of the adequacy of each of these corrective actions.

1. Complete the interim administrative controls, operator training and EOP and AOP changes to establish appropriate operator guidance for all accident sequences of interest.

Issue Manager: Duane Schoon Team Leader: Terry Vandenbosch

2. Independently evaluate the procedure changes being made to the EOPs and AOPs and assess their adequacy to provide adequate operator guidance. Additionally, evaluate the adequacy of the temporary information tags and objectively evaluate our decision to declare the AFW pumps operable after taking interim actions to address this issue on Tuesday October 29, 2002.

Responsible Manager: Tom Coutu/K. Davison

3. Independently evaluate the briefings and training conducted or planned for operating crews. We would like an objective look to ensure that the training was adequate and well documented.

Responsible Manager: Kevin Davison/T. Coutu Engineering Resolution Team and Root Cause Evaluation In addition to completing the root cause evaluation, the efforts of this team are to include completing the risk evaluation on an expedited basis, analyzing the potential for the orifice to become plugged, and conducting testing as appropriate to provide input to the PRA efforts being conducted.

1. Ensure that the root cause evaluation team completes the RCE in a timely and objective manner. The root cause team must evaluate why the original modification that installed the recirc orifices did not address the potential for orifice plugging and as importantly why our organization did not identify the issue when the recirc line was changed to provide a safety related open function in response to the recent red AFW finding. The charter for the RCE team is attached.

Issue Manager: Jim Freels RCE team leader: Rich Flessner

2. Ensure that the PRA work to evaluate the risk significance of this potential common mode failure is completed expeditiously.

Responsible Manager: Jim Freels Team Leader: Rick Wood 4

NPM 2002-0583 October 31, 2002 Page 3

3. Evaluate hydraulic system response to determine if the orifices could become plugged.

Responsible Manager: Jim Freels Team Leader: Tom Kendall

4. Develop a test plan to evaluate plugging of a spare orifice. Use the vendor utilized by Kewaunee to evaluate the AFW suction orifice issue as appropriate.

Responsible Manager: Jim Freels Team Leader: Tom Kendall

-5. Analyze the sample of corrosion products for metallurgical content to enable us to determine the source of the corrosion products.

Responsible Manager: Jim Freels Team Leader: Bill Zipp Evaluate the auxiliary feedwater system to determine the sources and quantity of potential corrosion products.

Responsible Manager: Jim Freels Team Leader: Bill Zipp

7. Contact the vendor of the orifice to obtain test data and other relevant information that could be used as input to the PRA work.

Responsible Manager: Jim Freels Team Leader: Tom Kendall

8. Identify all other applications of this type of orifice in the Point Beach units and determine applicable operating history. Evaluate suitability of this orifice type in all other applications.

Responsible Manager: Jim Freels Team Leader: Tom Kendall

9. Ensure that modifications to the orifice design or a replacement orifice are pursued expeditiously to resolve any questions relating to the potential for the orifice to become plugged.

Responsible Manager: Jim Freels Responsible Design Lead: Mike Rosseau rip

Page 1 of 3 Nuclear Management Company STATE CHANGE HISTORY Initiate by WILLIAM ZIPP AR Pre-Screen 10/29/2002 12 41 27 PM Owner (None)

SECTION 1 Activity Request Id:

Activity Type:

@ One Line

Description:

0 Detailed

Description:

Initiator:

Date/Time of Discovery:

Identified By:

Equipment # (1st):

Equipment # (2nd):

Equipment # (3rd):

Site/Unit:

Why did this occur?:

CAP029952 CAP Submit Date:

10129/2002 12:41:27 PM Possible Common Mode Failure of Aux Feed Recirculation Lines 10/29/2002 12:41:27 PM -WILLIAM ZIPP:

Partial plugging of the mini-recirc orifice (RO-4008) in the recirc line from aux feed pump P-38A occurred during testing on 10/24. This is documented in CAP 029908. Flow through the recirc line remained above operational limits during the event, and the remaining aux feed pumps were run to check for extent of condition with no reduction in recirc flows found. Review of this event has led to the conclusion that it is possible the mini-recirc orifices for all the aux feed pumps (3 per unit) could plug during an emergency when aux feed is needed. Plugging of the recic onfice could then render its associated pump inoperable, as this flowpath provides a safety related function to maintain a required amount of flow for pump protection. This considers use of our credited water supply (service water), but may also be an issue using the normal CST water supply. This is because of the small orifice opening sizes compared to the size of material that might credibly be introduced into the system from service water, such as rust nodules from the carbon steel supply piping, sand/silt, and lake grass. Material could also be introduced from the CST water supply, though this is thought to be a very pure supply of water.

ZIPP, WILLIAM Initiator Department:

1012912002 11:21:55 Date/Time of Occurrence:

AM Site-identified (None)

(None)

(None)

System:

Equipment Type (1st):

Equipment Type (2nd):

Equipment Type (3rd):

EESB Engineering Equipment Systems BOP Mechanical PB 10/2912002 11:21:55 AM (None)

(None)

(None)

(None)

Point Beach Common 10/29/2002 12:41:27 PM - WILLIAM ZIPP:

My opinion: It is explained via considering the history for the aux feed recirc lines:

2-3 years ago. due to recirc line vibration, cavitation, and ensuing weld failures, the recirc orifices were redesigned and a modification was initiated to replace the existing onfices with anti-cavitation models. All four have been replaced, the last one being done last month during Ul R27. The new orifices have smaller passages than the original model. At the time the mod was conceptualized, the recirc line's function was to LIMIT flow, ensuring adequate forward flow to the steam generators. Potential for blockage of the onfices was explored, and considered possible, but justified based on the short length of time the recirc line is needed at the beginning of an accident. The recirc line's function to limit flow bounds any postulated onfice blockage.

NovlDec 2001: Issue of recirc line AOV failing closed on loss of instrument air subsequent "red" finding. PBNP design philosophy evolves to needing recirc line flow to maintain aux feed pump operability. EOPs and AOPs revised to ensure minimum forward flow required to protect the pumps from overheating is maintained, and absent that, that recirc flow is present. The means of verifying recirc flow is that the recirc AOV indicates open, as there is no control room indication of recirc flow. Apparently, the impact of the new orifice mod was not considered

.. I

-f I

-^

-~.,^

Nuclear Management Company Immediate Action Taken:

Recommendations:

• Notify Me During Eval?:

Unclear to me at this point.

Summer 2002: Recarc line AOV function to open becomes a safety related function. The internals of the check valve in the common return line to the CSTs was removed to eliminate a potential common mode failure. Again, the onfice appears to have not been considered.

1012912002 12:41:27 PM - WILLIAM ZIPP.

Discussed with engineering management. Ops management including the on-shift DSS, Licensing. The shift manager declared all four aux feed pumps inoperable at approx. 1025 on 10/29/2002. Came to concensus on immediate actions needed.

10/29/2002 12:41:27 PM -WILLIAM ZIPP:

1) Wnte and hang temp info tags on the pump handswitches, requiring that a minimum forward aux feed flow of greater than 50 gpm (P-38 ANB) or 75 gpm (1/2 - P29) be maintained, if not, stop affected pump.; brief the Ops crews on the issue. (Ops)

2) Make changes to affected AOPs, EOPs, and other critical procedures to ensure that minimum forward flow is maintained, or the affected pump is stopped. (Ops, Engr support 50.59)

3) Review for reportability; report the issue (Licensing)

4) Review from PSA significance determination process perspective. Evalute the present way the onfices are modeled (PSA).

5) Review for impact on aux feed pump unavailability for NEI cornerstone performance indicator (EESN).

6) Redesign the recirc line onfices to make use of an onfice design that has aperture size of

>118" (basis: service water zurn strainer size is 1/8"). Consider providing Control Room with indication of recirc line flow. (design engr)

N 0 SRO Review Required?:

Y SECTION 2 Operability Status:

Basis for Operability:

0 Unplanned TSAC Entry:

(None)

" Compensatory Actions:

" External Notification:

N N

N SECTION 3 Screened?:

INPO OE Reqd?:

N N

0 QA/Nuclear Oversight?:

N Good CatchIWell Doc'd?: NA SECTION 4 Inappropriate Action:

Process:

Human Error Type:

Equip Failure Mode:

Org/Mgt Failure Mode:

Hot Buttons:

(None)

(None)

(None)

(None)

(None) 0 Significance Level:

Potential MRFF?:

0 Licensing Review?:

Activity:

Human Perf Fail Mode:

Process Fail Mode:

0 Group Causing Prob:

SECTION 5 CAP Admin:

0 Project:

PBNP CAP Admin Corrective Action (None)

N N

(None)

(None)

(None)

(None)

Prescreener:

(None)

"At, Page 2 of 3)

I..

I t*..

I.

t,*

j-I

,.liT.

,or Q.T'*t, t--

r1t r)

,l, l(

/*'to i

l m o)ft*-

Nuclear Management Company 0 State:

D Submitter:

AR Type:

• Last Modifier:

0 Last State Changer:

NUTRK ID:

1. of Children:

References:

Update:

Prescreen Comments:

Import Memo Field:

OPR Completed?:

OLDACTIONNUM:

subtsid:

originaljssue.id:

Site:

Cartridge and Frame:

Program (CAP)

AR Pre-Screen WILLIAM ZIPP Parent DAN WEBER WILLIAM ZIPP 0 Active/Inactive:

0 Owner:

0 Last Modified Date:

0 Last State Change Date:

0 Close Date:

Active (None) 10129/2002 6:32:25 PM 10/2912002 12:41:27 PM 0

N 0

original_project id:

0 Point Beach ATTACHMENTS AND PARENTICHILD LINKS SPrincipal to OPROO0031" Possible Common ModeFajilurre ofAux Feed Recirculation Lines CHANGE HISTORY 10f2912002 6:23:38 PM by DAN WEBER Last Modified Date Changed From 10129/2002 12:41:27 PM To 10/29/2002 6:23:38 PM Last Modifier Changed From WILLIAM ZIPP To DAN WEBER 1012912002 6:32:25 PM by DAN WEBER Last Modified Date Changed From 10/29/2002 6:23:38 PM To 10/2912002 6:32:25 PM Attachment Added: Pnncipal to OPR000031: Possible Common Mode Failure of Aux Feed Recirculation Lines I

0 11r till tflnr...r.

Page 3 of 3 it I--%-

I A n "'I r%#,% I

Root Cause Investigation Charter CAP029952 RCEOOO191 Issue Manager:

Jim Freels Problem Statement:

Discovery during the evaluation of CAP029908 (P-38A, MDAFW Pump had inadequate recirc flow during IT-10) that the recirculation line restricting flow orifices had become plugged during plant operation causing a reduced flow (but above minimum required) and that a potential existed for a common mode failure where all AFW pump recirculation lines could have restricted flow rates resulting in eventual pump failure.

Investigation Scope:

Determine the following:

"* Timeline of key events

"* The root and contributing causes of why the condition exists, including any potential human performance issues

"* Why the problem was not identified previously Make recommendations for:

• Correcting the problem, including any remedial actions

• Preventing recurrence of the problem

• Applicability of the root cause to other areas (extent of condition), including verification that a safety-related AFW recirculation flow path exists for the postulated failure modes Team Members:

Team Leader - Richard Flessner, Engineering Processes Team Member - Kevin Bennett, Engineering Processes Team Member - Eric Schmidt, System Engineering Team Member - William Bosacki, KNPP Design Engineering Milestones:

Status Update - 11/4/02 Status Update - 11/11/02 Draft Report - 11/18/02 Final Report - 1 1/26/0*

Engineering Director