ML19325D623

Forwards Summary of 891002 Enforcement Conference Re Auxiliary Feedwater Pump Net Positive Suction Head,Apparent Lack of Mgt Attention & Inappropriate Corrective Actions. List of Attendees & Viewgraphs Encl

ML19325D623
Person / Time
Site:	Robinson
Issue date:	10/16/1989
From:	Ebneter S NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II)
To:	Eury L CAROLINA POWER & LIGHT CO.
References
NUDOCS 8910250165
Download: ML19325D623 (59)

Text

yW 7 n,. a

}

OW4w 16,1989 o

b Docket No. 50-261 j,

License No. DPR-23 j

l Carolina Power and Light Company l

c ATTN: Mr. Lynn W. Eury ll Executive Vice President i-Power Supply L

P.'O. Box 1551 L

Raleigh, NC 27602 l

Gentlemen:

SUBJECT:

ENFORCEMENT CONFERENCE

SUMMARY

L

[

This letter refers to the Enforcement Conference held at our request on October 2, 1989.

This conference concerned activities authorized for your e

H. B. Robinson facility.

The issues discussed at the meeting were related to Auxiliary Feedwater Pump Net Positive Suction Head, the apparent lack of management attention and inappropriate corrective actions.

A list of attendees, a sunnary, and a copy'of your handout are enclosed. We are continuing our review of those issues to determine the appropriate enforce-ment action.

In accordance with Section 2.790 of the NRC's " Rules of Practice," Part 2 Title 10, Code of Federal Regulations, a copy of this letter and its enclosures will be placed in the NRC Public Document Room, t

Should you have any questions concerning this letter, please contact us.

Sincerely, I

(ORIGINAL SIGNID BY S. D. EDNTIE:R) i i

Stewart D. Ebneter 5

Regional Administrator i

Enclosures.

1.

List of Attendees i

2.

Management Heeting Summary i

3.

Licensee Handout cc w/encls:

C. R. Dietz, Manager Robinson Nuclear Project Department H, B. Robinson Steam Electric Plant P. O. Box 790 Hartsville, SC 29550 l

(cc w/ enc 1 cont'd - see page 2) 8910250165 891016 I l PDR ADOCK 05000261 Q

FDC yjr p

'J C'

i e'

s P

C Carolina Power and Light Coinpany 2

Octctxe 16, 1989 cc w/encls cont'd:'

R. E. Morgan, Plant General Mana9er i

H

8. Robinson Steam Electric Plant.

P. O. Box 790 Hartsv111e. SC 29550 Dayne H. Brown, Chief Radiation Protection Branch Division of Facility Services N. C. Departinent of Human Resources 701 Barbour Drive Raleigh, NC 27603-E008 McCuen Morrell, Chairrran Darlington County Board of Supervisors County Courthouse Darlington, SC 29535 Richard E. Jones, General Counsel Carolina Power and Light Costpany i

P. O. Box 1551 Raleigh, NC 27602 H. A. Cole Special Deputy Attorney General State of North Carolina P. O. Box 629 Raleigh, NC 27602 Avery Upchurch, Chairnian Triangle J Council of Goverrwnents 100 Park Drive P. O. Box 12276 P,esearch Triangle Park, NC 27709 j

Robert Gruber i

Executive Director Public Staff.- NCUC P. O. Box 29520 Raleigh, NC 27626-0520 i

State of South Carolina

[

bec w/ enc 15:

Docurr.ent Control Desk NRC Resident inspector l

U.S. Nuclear Regulatory Commission Route 5, Box 413 Hartsville, SC 29550 j

RII:

Ril: RS R

DRS R MDRS R1 : RP RIl b

RI -

A

[

i o

i

\\

l LMe7 PKel Pdbles kins yS ns AFG t-

.10/k/89 10/g/89

/ /89 10//o /89 10/q/89 10/

10/

s p

o m.

a.

,f 4

?-

t I

,,G I

i ENCLOSURE 1 j

i.

LIST OF ATTENDEES l:

Carolina Power and Light l

. C. Crawford -

Manager - Corporate Nuclear Safety i

C. D etz -

'RNP Project Manager J. Kloosterman -

Director of Regulatory Compliance L. Loflin -

Manager - Nuclear Licensing l

L R. Morgan -

. Plant General Manager H

M. Page -

Manager - Technical Support R. Parsons -

Manager - Engineering Support J.' Sheppard -

Manager - Operations A. Watson -

Senior Vice President Nuclear Generation I

NRC' Region II i

L H. Dance -

Chief, Project Section 1A i

S. Ebneter -

Regional Administrator R. Gibbs -

Operational Programs Section (DRS)

?

A. Gibson -

Acting Deputy Regional Administrator f

R. Goddard -

. Regional Counsel G. Jenkins -

Director, Enforcement and Investigation Coordination l

Staff (EICS) t P. Kellogg -

Chief, Operational Programs Section (DRS) l L. Mellen -

Operational Programs Section (DRS) i i

K. Perkins -

Acting Director (DRS)

L. Reyes -

Director, Division of Reactor Projects B. Urye.-

Senior Enforcement Coordinator (EICS)

NRC - Resident inspectors r

L. Garner -

Senior Resident Inspector K. Jury -

Resident Inspector l

NRC -Headquarters i

G. Lainas -

AD for RII Reactors /NRR j

W. LeFave -

DST /NRR R..Lo -

Senior Project Marager 1

i k

L L

1

I

.e s,

l t

c-

?

i F,< CLOSURE 2 i

MANAGEMENT MEETING

SUMMARY

j The Regional Administrator opened the meeting by indicating the main issue to I

be discussed was Auxiliary feedwater (AFW) Net Positive Suction Head (NPSH) and the apparent lack of management attention to this issue.

He then requested the licensee to provide an explanation of the problem and the proposed corrective actions.

The Director of Enforcement and Investigation r

4 Coordination Staff gave a brief description of the applicable enforcenient r

criteria.

Af ter the introduction of both staffs and the acknowledgment of their understanding of NRC's concerns, CP&L began their presentation by l

l acknowledging that the apparent violations had occurred as stated.

l r

i The presentation continued with a brief introduction and a chronological overview of the AFW NPSH concerns.

This was followed by a list of lessons learned from the CP&L investigation of these events and the proposed recovery i

program.

CP&L continued with a discussion of the operability and safety significance of the AFW NPSH issues.

The presentation concluded that after reanalyses, the AFW flow Requirements for the Small Break Loss of Coolant Accident and the Anticipated Transient Without Scram are the same as all other AFW accident analyses. The licensee further stated that the AFW system, prior to any piping i

modifications, was capable of delivering the required post accident flow, i

although the system was not tested at minimum seismic design level or maximum temperature for the CST.

l The discussions continued with a description of the root cause investigation that was conducted by a Special Independent Investigation Team.

The team was l

comprised of members that were not previously involved in the AFW issues and i

were in organizations which report independent of plant management.

The findings of this team were similar to the findings of the NRC Augmented i

Inspection Team which also investigated this issue.

The licensee outlined their corrective actions which included AFW Pump suction pipe replacement and changes to program management.

The CP&L presentation

{

concluded with a management summary.

CP&L assured NRC that H. B. Robinson management is totally consnitted to producing auditable results and that this i

issue and the respective root causes will be corrected.

The Regional Administrator subsequently closed the meeting by agreeing with CP&L's consnitment to the safe operation of their plant and stating that we would i

inform them of the results of the enforcement action, j

L

.n,

=,- =,.

}

1

(

.f j

INCIf6URE 3

• ~,

r c

l l-l'

{

l

.1

-i i

l i

H. B. ROBINSON i

i CAROLINA POWER AND LIGHT COMPANY ENFORCEMENT CONFERENCE RELATED TO

)

1 1

AFW OPERABILITY 1

USNRC REGION ll l

t ATLANTA, GEORGIA OCTOBER 2,1989 l

l l

1o I

1 l

\\

l i

i i

1 i

I i

2 i

i INTRODUCTION i

i,.

o Objective 1

To Respondto Violationsin 50 261/89-18 e

Failure to MaintainThree AFW Pumps Operableas j

l Required by Technical Specification 3.4.1.b l

e 10CFR50, AppendixB, Criterion XVI, Failure to Promptly identify and Correct Conditior.s AssociatedWith the AFW System i

l l

Violation Response l

L e

Three AFW Pumps Not MaintainedOperable Failed to Respondto and Take Action to Correctthe e

Deficiencyin a Timely Manner l

l I

-m.. -

1 i

,l CHRONOLOGY 12/86 Reactor Tri incurred Where Operators Observed Less Than Desi Flow When All Three AFW Pumps Respon MDAFV! Pumps Demonstrated Design Flow When SDAFW Was Secured.

Suspected inadequate NPSH and Requested investigation.

Post Trip Review Nor LER Recognized the Flow Anomaly.

2/87 ONS Completed Assessment of the Flow Anomaly and Concluded That There Wasn't an NPSH Concern Nor Was There a Concern Related to Suction Line Sizing.

Report Contains Contradictory Evidence That Such a Problem L

Did Exist. The Conclusion Recommended Additional l

Testing and Returned the Conclusions to Technical i

Support for Disposition and Tracking.

l 3/87 System Engineer Concluded That the Auto Start of i

SDAFW Was Not Required to Satisfy the Requirements of Any UFSAR Chapter 15 Accident Scenario.

Engineer Resigned Same Month.

7/87 Reactor Trip Incurred With All Three AFW Pumps initiated.

ERFIS Data Reflected Degraded Performance.

i Post Trip Review and LER Did Not Document the l

Deficiency.

8/87 SSFl AFW lasued Noting Concerns With CST Level l

Requirements and Related inadequate NPSH Calculations.

RAIL ltems Were initiated to Resolve the Concerns.

7/88 RAll items Assigned to NED to Resolve Via the DBD process.

8/88 Initiated AFW DBD Development.

j 11/88 Performed Special Procedure During Shutdown For l

Refueling Outage to Obtain Operating Data on the SDAFW System During Low Pressure Conditions.

3/89 AFW DBD Phase il Response Manager Requested Hydraulic Calculations no Support Design Bases L

Validation.

b

. ~. _.. _ _, ~ _. -. _ _. _,....

CHRONOLOGY (Continued) 6/89 AFW System Engineer Obtained Operating Data During Monthly Suiveillance Test.

7/89 NED lasued Revision 0 Calculations That Reflected inadequate NPSH.

Finding Was Entered into the Discrepancy Resolution Program.

Site Was Notifled and Assessed Informally That This Was Not a Significant issue Since it Represented a Previously Known and Evaluated Condition.

ONS Continued an Independent Verification of NED Calculations and Determined That Actual Head Losses o

Exceeded Prediction by Darcy Ecuntion.

ONS Model Was Changed to Utilize Hazen-W,lliams Which Provided

)

Excellent Correlation / Predictability, i

f l

Site Reviewed Revision 0 Calculations and Determined That They Were in Error Since They Did Not Consider j

i Recirculation Flows and Utilized incorrect Fluid Temperature.

Requested That Calculations Be Corrected and Additional Models Be Run to Assess Flow Rate t

Sensitivity to NPSH.

[

8/89 RNP Was Notified of Revision 1 Calculation Results.

SDAFW Removed From Service Due to inadequate NPSH.

l Simultaneous Operation of All Three Pumps Created an Inoperability Situation For All Pumps.

Securing SDAFW Enabled the Two MDAFW Pumps to Remain Operable.

ONS Met With NED and Determination Was Made That the Revision 1 Calculations Were Non Conservative in Reflecting Actual Head Loss.

Revised Calculations Supported the Operability of Only One MDAFW Pump.

Plant Was Secured on 8/22/89.

+

L ISSUES DERIVED FROM CHRONOLOGY e

Post Trip Reviews Were inadequate, o

Pump Operability Not Challenged on Several Occasions Based Upon Available Analytical and Empirical Evidence, Conclusions Were Based Upon Informal Engineering e

Calculations, ONS Conclusions Were Misleading in 2/87 Report and Did Not e

Track Embedded Operability Concerns, o

System Engineer's Conclusions Were Misleading Related to AFW Operability, Design Bases Re Well Understood.quirements For the AFW System Were Not o

l RNP Management Was Not Appropriately Sensitive to e

Operability lasue and Demonstrated Non Conservatism Based Upon insufficient NPSH.

e NED and ONS Hydraulle Calculations Were incorrect.

e Management Practices Related to Work Control and Prioritization Were inadequate.

1 4

C g-g 4'

~

AFW INVESTIGATION RECOVERY TEAM STRUCTURE P,,O,CT - _ _

NOR CORPORATE CORPORATE.

I I

1 PLANT MANAGER OM/ MOD PROMCT l

R.E. MORGAN W.J. FLAMA9AN I

I i

I I

I I

RELATED ISSUES FLOW ANA1.YSf3 EQUIPMENT DESIGN ENO.

CMS EVALA#ATION gg E.M. HARMt3 W.F. PAGE D.R. OUIC9C R.M. PARSONS W. MARTIN l~

E l

BACSCLOO REveEW FLOW TESTING PUMP INSPECTION FLOW ANAYSIS MA8MTENANCE ASSESSMENT

- PRSORITY ASSESSMENT

- CNRONOLOGY PSPE ONSPECTION MCTOR INSPECTION 0800 DEVELOPMENT esoDeFICATIONS

- PEftFORMANCE j

PERSONAL $NTERVIEWS IMO9CATORS

- UNRESOLVED STEMS CST INSPECT 90N PW ROUTIMES DWU CONCUENS TESTINS

- MST PRACTICES

- OTHER PUMPS /MTRS - P850RrrY REVIEW SUstVEILLANCE TEST 1NG

- SYSTEM EVOLUTION 8800T CAUSE

- OTHER SYSTEMS

- CAUSAL FACTORS NMBN CONCEstMS

- OTHER SYSTEMS FLOW REOUsstEMENTS

- ALL ANALYS85

?

,)

,,e.- - '

'.,\\.,

i 1

i-l l

l l

s RECOVERY OVERVIEW e

Key issues j

System Modification Management Concerns e

Safety Significance Degraded Performance Represented Minimal Threat Operability Determination e

Non Conservative Decision Making Failure to Recognize Until 8/89 e

1 i

RECOVERY OVERVIEW (Continued) e Response Strategy i

Process Consideration e

Critical, Timely, and Comprehensive i

Develop Root Cause Understanding i

i e

Complex and Far-Reaching Corrective Actions e

Practical and Sound

[

^

o Preclude Recurrence e

Address Cultural lasues t

Operational Sensitivity

[

Procedural Guidance l

e Management Control Recognition, Capture, Prioritization,

]

implementation i

Apply Lessons-Learned I

e Other System o

Work Processes i

l

.i j

I i

l t

i ENFORCEMENT CONFERENCE j

AUXILIARY FEEDWATER OPERABILITY OCTOBER 2,1989 - ROBINSON UNIT 2 e

introduction C. R. Dietz j

i e

Operability and Safety Slanificance J. J. Shannard j

e Root Cause C. W. Crawford l

e Corrective Actions R. E. Morgan j

i o

e Management Summary C. R. Dietz R. A. Watson L

l l

.y.

3 j,

j c..

& 4.. -

L i

i i

t v

I l

4 I

i i

I i

l AUXILIARY FEEDWATER SYSTEM l

PUMP OPERABILITY

-w f

• s AUXILIARY FEEINATER SYSTEM I

FUNCTIONAL DESCRIPTION i

,m g '= =

T.

A" Steem.

[

heet e e

b l

7' M V2es l

Feesa Feedessee Syssene

-O<>

S.

v2-8 4--

M I

To "B" steese I

~

I anseen tw,em AFW Fweep f-

'l 8* CV FCV-6416 n

z w car I

M k M F. m -

j

^

1. ~_

I y

m rcy-ure y

A%g S

' ree.

$ c = r-E oc

't x

=

,a te Y2 30A te 5.ww

1. " 8 *"'

T."C' 3

T

E 7

5t* =

7}

90esar Desseen AFW V2 5

g c--

u Y22 re

~$

V r

n v2 esc d L

-#W 1425 l

'cw= rus

..___,___.___._,_____r*4__________________.____.___________,____

1..-

. _ _ _. :. e _., __

{

w l

p 4

AFW HISTORY e

Configuration Summary

)

Present Configu' ration is Two 300 m MDAFW Pumps and One 600 gpm SDAFW mp. Water Source is a 190,000 gal. Condensate Storage Tank

)

With Backup Supplies from Service Water and Deep l

Wells.

Pumps Receive Automatic Start Signals For the Following Conditions.

e MDAFW Pumps 2/2 Feed Pump Breakers Open Lo Lo Level in one S/G f

Safety injection

[

Loss of Emergency Bus j

i AMSAC Initiation SDAFW Pump e

Undervoltage on 4KV Busses 1 and 4 (MFP power supplies)

Lo Lo Level in two S/G 3

f AMSAC Initiation e

Technical Specifications Unchanged Since 1983

.. - ~.,,. -.-.. _..

m_..,._.,..._m..._,-.,_.._m_.

.)

i. *. -

)

)

i i

AFW HISTORY (Continued)

{

e Summary l

System Has Evolved From a Two Pumps Required f

" Auxillary" System to a Three Pumpa Required

" Equivalent to Safety Grade" System.

l Technical Specifications With Regard to the Pumps l

Are Largely Unchanged From Initial Licensing Other t

Than Addition of Th rd Pump Being Required to be

Operable, i

i Upgrades in 1984 Made the System Much More Reliable and ERFIS/SPDS Installation in 1988 Provided Vastly improved Trending Capability.

System as Configured Now, is a Relatively Simple, Easy to Operate and Reliable Source of Water to j

Remove Decay Heat.

3 t

k i

I

l

.i j'

T i

i i

i AFW REQUIREMENTS i

e Technical Specifications.

i Three Pumps Operable to Exceed 350*F Minimum of 3'5,000 gal. in Condensate Storage Tank (2 hrs at Hot Shutdown) r Unlimited Water Supply From the Lake Via Either -

Leg of Service Water Essential Features including Piping and Valves Operable e

Surveillance Test Requirements i

.Run Each MDAFW Pump Once a Month For 15 l

Minutes Run SDAFW Pump Once a Month For 15 Minutes j

Test Discharge Valves Once a Month Test initiation Signals a

l r

p.

u..

j

~

AFW FLOW REQUIREMENTS Transient / Issue Reference Reauired Flow Comments 1)

LB LOCA Not Used in Calcs.

l 2)

SB LOCA FSAR/WCAP-12034 900 gpm Accident is Not AFW j

R Ltr 8/21/89 240 gpm Sensitive.

j i

900 gpm Based on Loss of One EDG 3)

Loss of FW FSAR 240 gpm i

l 4)

ATWS WCAP 11993 600 gpm Generic Resolution t

W Ltr 8/21/89 (Initial)*

Assumes Two MDAFW

+

(Not a Design 240 gpm Pumps l

Basis Accident (long-term)*

l l

per SECY-83-293) j 5)

Main Steam FSAR 1300 gpm

[

Line Break (max.)

i 240 gpm (long-term)*

NOTE-Except for Station Blackout, All Transients Assume Auto Start and AFW Available in One Minute.

I

• Requirement Not Formally Documented.

i I

t i

i I

!+

--w-

--,-y

_y c- - -

s s,-

AFW FLOW REQUIREMENTS (Continued) 1 1

N i

l Transient / Issue Reference Required Flow Comments j

i I

6)

Station Blackout Generic 240 gpm Bounded Loss of FW.

Resolution (Hot Shutdown)

More Flow to coasdown.* Assume.

Manual Start of SDAFWP i

Bounded By Main Steam 7)

Feedwater Line Break Line Break i

i 8)

Long-Term 240 gpm Bounded By Loss of Cooldown Normal Feedwater NOTE-Except for Station Blackout, All Transients Assume Auto Start and AFW Available in One Minute.

i

• Requirement Not Formally Documented.

f I

I

-I

.l i

1 l

~

i.... ;

i i

i AFW PERFORMANCE e

Pump Operability History j

Pumps Tested as Required by Technical P

Specifications, j

In Service Testing Data Showed Satisfactory Pump I

Performance (Individuall'/) and No Degradation in Performance.

t Condensate Storage Tank Normallgal.)y Maintained l

l Greater Than 70% (~133,000 Pumps Were Assumed Operable Due to Meeting Required Surveillance Tests and No Evidence to the Contrary.

I 12/86 Reactor Trip Was the First Time That Empirical Data Was Available That Operability Might Be in Question.

Even With Reduced Flow Evidence in 1986 and 1987, Available Flow Was Three to Four Times Greater Than FSAR Values.

Pumps Were Evaluated 1

as Operable.

l 1

1

Q t

i l

l l

l l

i i

I AFW PERFORMANCIE (Continued) i e

Plant Shutdown t

August 1989 Calculations and Empirical Data l

Showed insufficient NPSH For Two or Three Pumps Running in Combination.

Two Out of Three Pumps Were Declared inoperable l

on August 22,1989, Due to Not Meeting NPSH For L

Pumps in Combination Over the Full Range of the i

Condensate Storage Tank and the Plant Was Shutdown.

e Plant investigation l

Inspection of Pumps Showed no Evidence of NPSH j

Damage on MDAFW Pumps and Minimal Damage / Wear to SDAFW Pump.

l Inspection of Suction Piping Showed a Buildup of 1

Corrosion Products Thereby increasinD Friction Factor and Reducing NPSH Available.

l l

l

l I

1.

g.

,l 2

..o I

a

)

I

'I i

i i

4 i

l 1

AFW PERFORMANCE (Continued) e Conclusion i

I Two Out of Three Pumps Were Technically Inoperable Due to insufficient NPSH Available For Pumps in Combination, i

Evidence Suggesting Operability Problems Was First Available in Late 1986.

Conclusive Evidence and Calculations Were Not Available Until August,1989.

j i

i

\\

l Upon Reviewing August,1989 Evidence, the Plant l

Was Shutdown.

i

.f l

1 e

i

+

i P

..9-.---.w-%..

.,.%,-.,m.

e.,..,.

ry.m.

c.

4 5

3.

.3

.6 1

o AUXILIARY FEEDWATER SYSTEM SAFETY SIGNIFICANCE OF-PUMP INOPERABILITY f

r

)

4 L 1.

1 l

AFW SAFETY SIGNIFICANCE j

l i

e Requirements Review P

240 gpm Required Within One Minute For FSAR l-l Analyses.

Other Non FSAR Analyses Have Other Assumptions Outside of Potential LCOs.

One MDAFW Pump Provides All the Cooling Required to Remove Decay Heat and Maintain the l

Plant in Hot Shutdown, i

i e

Mitigating Factors I

Pumps Have Provided Flow Far in Excess of FSAR i

Requirements on All Occasions. One MDAFW Pump Has Always Been Operable.

During Transients, pumps Only Run in Combination l

For Short Periods of Time (Typically Less Than 30 j

Minutes).

- t Emergency Operating Procedure Network Requires Monitoring AFW Flow on Any Reactor Trip or Safeguards Actuation (Minimum of 300 gpm Required).

Functional Restoration Procedure - H.1 Provides i

Mitigating Actions Should Minimum AFW Flow Not be Available.

i t

Operators Trained to Maintain and Recover AFW Flow.

3 Decay Heat Removal Capability Has Not Been Jeopardized.

e a

i 1

1

, ~

,e.-,-,-%,-e.-*.-----,,

t-,w,--,,,,---e v..ww..+,.e

+4

,vww.-.--..,--e,-----ve---

i l

~

4..'

l l

a i

i i

l i

.l l

AFW SAFETY SIGNIFICANCE (Continued) l l

o Conclusion Public Health and Safety Not Endangered, j

j Pumps Never Failed to Deliver Required Flow.

One MDAFW Pump Always Available.

Procedures and Training Were in Place to Mitigate i

insufficient AFW Flow, ii t

Actual Safety Slanificance Was Minimal.

e

,c l

't t

l l

L

~

t 9

e

>- v.>

y?._..

,}

i a

.Le ' c g

\\

uy,/

g

-j a

A. I 1

)..'

i a

I AFW OPERABILITY AND SAFETY SIGNIFICANCE l-e Overall Summary 1

L '.

Problem identified by CP&L.

o b

No Evidence Was. Available Until 1986 Which Would L

Challen'ge Pump Operability.

Pumps Were Only inoperable When Running in Combination.

-l Decay Heat Removal Capability Was Not l

Jeopardized.

u Safety Significance Was Minimal and the public

..f,

~ Healt1 and Safety Was Not Compromised.

p r

i 1

f

. i.,

.e-

~..

p'

+.

E ;,f,

.'n.

,- ; c,

m 1

.i n.

l-.

'I

^

,y i

,\\

?

i ENFORCEMENT CONFERENCE

]

AUXILIARY FEEDWATER OPERABILITY t

1

- OCTOBER 2,1989 - ROBINSON UNIT 2 i

e Introduction C. R. Dietz l l1 L

~ Operability ~and Safety Significance J. J. Sheppard e-L e

Root Cause C. W. Crawford e

Corrective Actions R. E. Morgan m

e Management Summary C. R. Dietz R. A. Watson

~

4 t'

(

r e

w,

,-e-,,-

4

., t..'..

l y). ;.

't l

r. e b

,J 1

.p

)

h s

)

INDEPENDENT ASSESSMENT TEAM r

i H. B. ROBINSON UNIT 2

.?

l AUXILIARY FEEDWATER NPSH ISSUE t

(

l r

3 i

4 f

.i l

c l

f i

e r

l' l

• I-

n,,

L'y

's j

1 l.

. ?.j '

G La.;-

s l-c j

.i o

1 i

i l

t l

L INTRODUCTION 1

a Special independent Investigation. Team On August 24,1989, l.

e Was Formed to investigate the Events Leading to the Plant-i Shutdown.

.i L'

. The Misslorrof the Special investigation Team Was to:

e L

" Conduct' an independent investigation of the Process'Used in identifying the Condition of inadequate AFW Pumps NPSH to Determine Causal Factors and Root Cause."

t

- e-The Team Members Selected Had No Previous involvement in Either the HBR. AFW SSFl or Any. Previous investigations of 1

AFW NPSH and Were in Organizations Which Report 1

independent of. Plant Management'

(

I 4i t'

a f

. +

3

'.ci.

-,..,,..~

-. c

,p

'o Q2T*W- ;

r

.4..

7-

. ID AFW INVESTIGATION RECOVERY

TEAM STRUCTURE PROJECT MANAGER HOR CORPORATE CORPORATE-I PLANT MANAGER OM/ MOO PROJECT R.E. MORGAN W.J. FLANAGAN I

I I

F I

RELATED ISSUES FLOW ANALYSIS EQUIPMENT DEStON ENG.

CNS EVALUATION MTM E.M. HARRIS W.F*. PAGE O.R. QUICK R.M. PARSONS W. hARTIN EACKLOG REVIEW FLOW TESTING PUMP INSPECTION F1.OW ANAYSIS MAINTENANCE ASSESSMENT

- PRIORITY ASSESSMENT

- CHRONOLOGY PIPE INSPECTION MOTOR INSPECTION MOD DEVELOPMENT MOOtFICATlONS

- PERFORMANCE IMO1CATORS PERSONAL INTERVIEWS

~

- MST PRACTICES

- UNT.ESOLVED ITEMS CST INSPECTION PW RJOTIMES DED CONCERNS TESTING

- OTHER PUMPS /MTRS - PRIORITY REVIEW SURVEILLANCE TESTlHG

- SYSTEM EVOLUTION ROOT CAUSE

- OTHER SYSTEMS

- CAUSAL FACTORS NPSH CONCERNS

- OTHER SYSTEMS FLOW REQUIREMENTS

- ALL ANALYSIS r

' g.

s

~

.J.c.: 'o l..

.,J*'

;,;. t i

t

/ w I:

L '(/

l j

a c.,-

L

.i 1.

[

[

t l,.

t 4

')./

g

• i i

l I'

L.

MAJOR ISSUES L1 i

Corrective' Action Program e

Technical Specification Interpretation e

\\.

t i

e Technical Support Work Control

[

e Calculational inadequacies e

I b

1 i

'g

, i a

,l' m,

l;b

~

b 4

'.., 'i

e, t

l ',

es

.g

-[

Iqk i MAJOR ISSUES (Continued) r

'The AFW NPSH Concern Was Not Placed in a Formal Corrective Action Program to identify, Document and Evaluate the Condition..

' Causal Factors e

ineffective Formal Corrective Action Process for Identification, Documentation,'Prioritization, Evaluation and Tracking of items Requiring Corrective Action 1

L e

Informal Management Reviews Based on Historical Experiences Not Technical Evaluations With Appropriate Documentation and' Review l

An Attitude That the Plant Was Built and Operated e

Correctly Unless Proven Otherwise (i.e, Lack of a l

l R'

Questioning Approach to Discrepancies) 1 l

g l

1

.o

7.

~

.b..

..z.:

c4 -

f, >

i

}

t MMOR ISSUES (Continued)

{

i The Definition' and Interpretation of AFW Technical Specification Operability Was inadequate.

]

1

' Causal-Factors l

e'-

Incomplete APN Technical Specification Operability l.

Requirements i

h7 No Formal Process for Documenting or Approving i

e Technical Specification Interpretation 1

o a-Improper Use of FSAR Information to Determine

[

Operability e

inadequate Design Basis information 4

E

.,,e---

c.

~

--m.,_,-.

____..-,.m

'.? a 4

y

,t e

il n

^

MAJOR ISSUES (Continued)

The Technical Support Unit Work Control and Prioritization Did Not Support Adequate initial Resolution of the NPSH Problem.

1 L.

L.

Causal Factors l:

e inadequate Plant Prioritization and Control of Technical Support Unit Assigned Action items e

inadequate Reporting and Tracking of Action items

~

l Lack of Management Oversight and Controls That e

Effectively Transfer Project Responsibility and

}

Accountability to Another Organization Misuse of DBD Process to Resolve Operability Questions e

e informality of Initial NPSH Calculations

.I Technical Support Unit Workload and Personnel Turnover e

.l l

y.

.r.

[:

n

.:. ;.c ur e

y' l

V l

MAJOR ISSUES (Continued) o The AFW NPSH Calculation Process Contained Significant l

Technical Inadequacies.

Causal Factors e

Non-use of Procedures and Informality of Initial Calculations, including the Absence of L

Checking / Verification l

t e

Desire to Achieve Technical Results That Support L

Continued Operation e-An Acceptance of Minimum Results From Analysis to Support Operational Needs i

e inadequate Consideration of Design / Operational Margin o

e Lack of Attention to and Use of Actual Plant Data (Where J-Applicable and Available) l, i

g

t.... '.

v' j d. '{,f: a.

3, ih.,

U

' ll s

i ROOT CAUSE The Root Cause for the AFW NPSH issue Not Being Resolved in' a Timely Manner Was Determined to be tho'Following Causal Factors Acting in Concert:

e A Lack of Attention to Aggressively identifying and Correcting Abnormal Conditions e

A Non-Conservative Interpretation of-What Constitutes AFWS Oserability Based on incomplete Design Informat on An inadequate Technical Support Unit Prioritization and e

Work Control Process o

i E

9

.. n. -

.,..,.~.n..-..~m g n.,

t.

m

.I u '.,,,..,.

y l

f

?c

]

1 6

KEY RECOMMENDATIONS I

1 e

Provide Plant Staff Instruction on Effective identification of Problems and the Need to Aggressively Pursue i

Correction of Abnormal' Operating Conditions.

e increase Management Emphasis in the implementation of.

7 PLP-026, " Corrective Action Program."

e Establish a Formal Technical Specification l

Assessment / Interpretation Process.

Provide Management Direction and Personnel Training e

that Minimum Requirements Are Not Accestable and That Adequate Operational Margin Must be Ma ntained.

i e

. Strengthen' Technical Support Unit Work Control Process l

to Provide. Effective Plant Technical Support and Orderly Completion of Assigned Action items.

O 5

4

..u

. n >..~ ~ n.

w...

1

. ' 7 *. '

g

(

a l.

t" ENFORCEMENT CONFERENCE AUXILl' RY FEEDWATER OPERABILITY

[

A OCTOBER 2,1989 - ROBINSON UNIT 2 I

l 'e i '

' e introduction-C. R. Dietz Operability and Safety Significance J. J. Sheppard

[

e

\\

e Root Cause C. W. Crawford 1

L e

, Corrective Actions R. E. Morgan _

e Management Summary C. R. Dietz R. A. Watson l,

s' k

.s q.

- m. 4

- v..

- - ^ ^. _ r_: =

't

.e '

f h'.

l p

g 03'

.<- e f

A.

.. e

\\ 'l f, -

3..

O, 4

1 k

/

P Y

J A

'I a I

b b

i i

CORRECTIVE ACTION T

s 9

d h

h n

-' 'f,s e

rf i

r

~ ~ '

w.:=, an. :.c.-..., awn +.,..m.,,:.=.mu-~-u..w..m u.w.-.

.a=. e ;n ::s.w.,u w.., - --

?

i

.y l

+,

g i

3

'j

.y:,

q 1

J P

1

-l J

OVERVIEW

-s i

e Nuclear Regulatory Commission-(NRC)

Senior Resident Special Inspection t

Augmented Inspection Team (AIT)

\\

.,3 u-

'1 o

Corporate Nuclear Safety (CNS) 1 L

Special CP&L Independent Assessment Team (IAT)

.l w

u Correlation of Findings and Corrective Actions-e i-3 t

,~ n..... -. -

. _,.. -..- - ~ ~.

.g...

5 I

E

'5

+

t i/

1,.

. ).'

.h'.

L.

}'-

VIOLATIONS (NRC)

I As identified in Inspection Report 89-18 i

g, s

L e

Failure to Have Three Operable Auxillary Feedwater 1

L.

Pumps (Technical Specliication 3.4.1.b)

. (

- e Failure to Promstly identify and Correct Condition's l

Adverse to.Qua ity (10CFR50, Appendix B, Criterion XVI)

- t s

s I

I j

l,

1 -

1

~..,,...

4. '.. !

'i m,

.9 n!

L

n..

g

.l 1^

1 l

I l1 i

1; r

L j

?

d o

l l

i MAJOR ISSUES (IAT) p.

e Other Internal Investigations / Issues L

l Auxiliary. Feedwater (AFW) Net' Positive Suction 1

l Head (NPSH) Concern Was Not Placed in Corrective a

If Action Program.

1 Definition and interpretation.of AFW Operability Was Inadequate.

Technical Support Work Control and Prioritization Did Not Support Resolution of NPSH Concern.

L NPSH Calculation Process Contained Significant L

Technical Inadequacies.

c

+

+.

l.

B L

y l-E 3

l-r I;

o P

....; y.n.

.. z. s.,...

'1,.

i

'. 59'.'

l l(,'

' t's l

f t,,-

. i

,?..

i i

i i

,l j

l j

'lj.. r.,

.)

L:

ci l.

l-o i

h CORRECTIVE ACTION I

i, L

i p

r e

Failure to.Have:Three Operable AFW Pumps 1.

I Auxiliary Feedwater System Modifications r

(MOD 1018) l l

1 l.

.4 j't !.

o

.4 l

-i p.

1 : 1

!'5 l

l.

1 o

e I

I,'

l C

,f.

b

, 7 ll

-).

i,,6 4

s-

-a

-5

'e

.m-nu um*..

,m'-_

a

_.ew.,

- - -,----.---ne---e=we--

--.-w.-se.- ~w-.

a--e-.w-,w--

e-,

= - +. - -, <

-me<w e

.m

- e e

- er

. ~.

-=,

g. g i.

sy "IW PUhP f

.i e.

CST u :

H i

- evl20'

~

l u,:

l i

i u :

l l :

u :

i M

h i l

l l :

l l :

i

,',ll 5

l l :

i i

t i.

rul5S' rviO' i

t

- ev25'

{

j CODE SDAFW PUMP l

i

! M SS 6" PIPE NU$$$$lRS5lRSS 8" PPE l

ggggggggggggggSS 12" PIPE l

NOT CODED-CS 6" PPE & VALVES (ORIGINAL) i l

MODIFIED AFW SYSTEM r

i

..m

lt 1

, O.

p '-

CORRECTIVE ACTION (Continued) e Failure to Promatly identify and Correct Conditions

?

Adverse to Qua Ity (10CFR50, Appendix B, Criterion XVI).

i Increase Management Visibility.and Attention to Technical Corrective Actions Within Various Plant-Problem identification and Tracking Systems

]

(Prior to Startup).

e implement interim Guideline to Provide Management Visibility and improve 1

Effectiveness of Corrective Action Program (Prior to Startup).

1 g.

Continue Management Emphasis on Effective e

implementation of PLP-026, Corrective Action i

L Program (Long Term Plan).

L-L Implement.Prioritization of identified Corrective Action Program, Significant Condition Reports S

(SCRs), in Accordance With Nuclear Generation Group' Guideline NGGG 305-05, Prioritization Process (Prior to Startup).

integrate Corrective Action Program SCRs into e

Plant Work Management Process Based on Priority (Prior to Startup).

Fully implement NGGG 305-05, Prioritization e

Process, into Plant Work Management Control Process (June 1,1990).

l l

1 f

.I

- -... - ~.. -... ~... -.... ~

, ~,.,

__-_ \\

.(

k, l. *m

. j 3..

. e....

n j

i i

]

)

i CORRECTIVE ACTION (Continued)

L u

l e

Failure to Promstly identify and Correct Conditions l

Adverse to Qua.ity. (10CFR50, Appendix B,. Criterion XVI)

(Continued).

Provide Formalized Technical Specification H

L L

Interpretation Process for Documentation and Approval of Interpretations involving Operability.

1

'1 u

6 e

interim Guideline - Management involvement (Prior to Startup)

Formal Procedure (December 31, 1989)-

e

. \\

n i

1

.4 t

'D-'

-., - - < - -., - - ~ -..

.,--,~~.,nn-..-----.~,+----.--..-...-~---------,--...n--n--.

.. - - - - ~ ~ -.. - -. ~... -. -.,..

. _. _. ~. _ _

I a

1

/.

', s. -

s HE :n lY t

,':j

}',

CORRECTIVE ACTION (Continued) l:]

e Other Internal investigations / issues Outstanding issues Technical Support L

NPSH-(Other Systems) e e

JCOs L

e ISl/IST Maintenance o

I e

Pump / Motor inspection History t

e PM Routines-Regulatory Compliance Technical Specification Interpretation e

e Non-Conservative Interpretation (Other l

l Systems) u It m

o 7

j i.* s Ji p i..'

.{

n,.i,. *

,,1 i

~ ;,.

F 1

i i

j 1

a l

s

[-

j CORRECTIVE ACTION (Continued) e i

'e Other Internal. Investigations / issues (Continued)

)

-Operations i

e Post Trip Review e

. Operability Determination Design Basis Document (DBD) e Accelerating CVCS DBD Reprioritizing the Validation Effort on e

Vu,nerable Systems Service Water L'

HVAC System (Post Accident Portions)

.\\

i l'

)

1 J

l' 1

-)

v-

-,i L,..L,, r-

..-a,

,,a.--,..,,,,

w,,.--

-,----,,-.-n-n w-~n..

--,. - - -. - - - ~ - - - - - - - - - - - -- - - -- - - -.. - ~.. - - - - ~ ~ - - - - - ~ ~ ~ ~ ~ ~ ~ - - - - - - - - - - - ~ ~ ~ - - - -

....... -... ~.........

... -..,.......... ~......... ~ ~...

,.m

.,: c,

V

\\

u.

\\

l CORRECTIVE ACTION (Continued) r e

Management issues Corrective Action Program e

10CFR50, Appendix B, Criterion XVI L'

e Process Design L

implementation L',

e i

e Prioritization e

Root Cause Determination j

Integration into Work Management Process e

Visibility l

l e

Control i

e Resource Allocation g

e Oversight Plant Support Unit Provide Focus on Management issues l

e L

L L

h L.:.

cf I

a.

_p^

~_

g

,T, e-E-

CORRECTIVE ACTION IMPLEMENTATION ~ CONCEPT l

i.

4 t.

CAPTURE l.

l i

t l

l SIGNIFICANCE I

DETERMINATION I

l t

f i

l i

MANAGEMENT i

VISIBILITY r

t t

i SIGNIFICANT NON-SIGNIFICANT f

CONDITION CONDITION'-

I i

WORK -

+

i l

i t

PRIORITIZATION 2

j i

4

(

j WORK 1

-MANAGEMENT I

PROCESS 4

I i

i L

)

=

m g,

1

.l

• i-b n :,..

. c :,y.:.

1 gg,

. 4 4

" ; e i

?'

q e,

CORRECTIVE ACTION (Continued) e Management lasues (Continued) n!

Culture / Attitude

't e

Accountability

'e Curious - Questioning e

Understand Level of Performance -

Goals-e-

Business Plan m,

o Prioritization Process Team Work-e Corporate e

Site e

Systems

)

.p

-2,-

.,.. - -..-........+.,_..

..-....,.....,-.-~.-_.------.-.m..

..n_.,,

, gc e

j

n m m... ;.

n zq y!

m ;

i 4-

. t p

r,.

j

.o

.. n-i g.

i

)'

N l

.l 8

CORRECTIVE ACTION

SUMMARY

o' Low' Safety Significance e

Low Sensitivity to Operability e

Root Cause Understood i

L

' I Management Controls e

I,.

(L' l

6 3

i e

b t

b S-4

m. *, -e e

• --+-,,%--....,....i+w.,,nv, w

c--.mvc

..w...=ww..,

w-,--w,,.

ae-we e---w w.,

ww wem y,+---

w e v e

-.- eree iv* - w

-reer--ems

._~

m.

4.

,,1 ;.,

.

• rs Lfp'.4l n.V '

l

. u, 1;.

u 2

,.- ( J -

u

'a 1

i V

r ENFORCEMENT CONFERENCE AUXILIARY FEEDWATER OPERABILITY j

' OCTOBER ~2,1989 - ROBINSON UNIT 2

+

i y

C. R. Dietz e

. introduction o

Operability and Safety Significance J. J. Sheppard u.

e Root Cause C. W. Crawford e

Corrective Actions R. E. Morgan e

Management Summary C. R. Dietz R. A. Watson i

.),:

f

t.=

4..

. f

. b.

i' s.

i.

,/

s

.f' MANAGEMENT

SUMMARY

m

1 L,.

l; I f ',..,,

L 1

1, a f p

SUMMARY

l.

e Response Strategy j

[,'

Timely Comprehensive l

l' Aggressive /Self Critical L

l e

Accomplishments L

Physical AFW Operability Restored e

[

e Technical Details - Startup Conference 1:

Administrative 1

E e

Critical Issues Prior to Startup i

PM Routines Surveillance Test Review o

Unresolved Issues Review J

m Operability Determination Considerations Technical Specification Interpretation Review Absence of Other NPSH Concerns Management Review of AFW lasues e

Long-Term Issues Scope and Commitment Are Actively Pursued.

1 ~

y a

1,

. i.

..

• v *:.

,g4 e.+

1 l

I 1

SUMMARY

i e

Key Challenges Understand Our Work Demands.

Establish Proper Priorities.

Assure. Proper Resources Allocated to Resolve.

j 1

Develop Regulatory and Internal Understanding of Our 1

Planning and implementation Strategy.

i Advocate inquisitiveness and Operational Pessimism in

)

Approaching issues.

i

)

H h, '

~ __,

..-.-.._.,.._...___4

e o. i.,

'f '

lI s

1

SUMMARY

o Conclusions Safety Significance Was Minimal, l

e No Situation Analytically or By Demonstrated Performance Where the AFW Did Not or Would Not Satisfy its Safety Requirements, o

Did Not Represent a Threat to the Safety or Health of the General Public.

Management lesues Were Dominant, e

Responsiveness Was Unsatisfactory, e

Work Control Processes Were Deficient, e

Demonstrated Non Conservatism in Our Decision Making and Exhibited a Willingness to Achieve Only Minimum Standards of Compliance.

l' ww.

. _,, _. _ _ _.,. _ _ _ _ _...,, _.., _..,. ~,,..., _., _

i

^t.,

~ os t. :

y d'

n-i j

i l

l

SUMMARY

e Conclusions I

t.

I ji Challenges are Significant i

t e

100% Responsible to Correct l

Recognition, Strategy, Plan and Commitment Are in l

l e

L Place L

i e

Not Committed to Promises - Committed to 1

Auditable Results j

I l'

l e

n

'I

)

1

• .c-y-.,

e-4-%,-

-y--%.,,,,3,%m,..._,_,,y.,,,%.,,,_

,,,_,,..-,,,m,,

,,_,,,,,,_,, __ _...,,,,,,,, _,.,,, _, _, _ _ _...,,,, _