ML050580002

Meeting Summary Discussing Palo Verde Apparent Violations

ML050580002
Person / Time
Site:	Palo Verde
Issue date:	02/25/2005
From:	Troy Pruett NRC/RGN-IV/DRP/RPB-D
To:	Overbeck G Arizona Public Service Co
References
EA-04-221
Download: ML050580002 (41)

Text

UNITED STATES N0 CLEAR REGULATORY C 0 M An I SS IO N REQION I V 611 RYAN PLAZA DRIVE, SUITE 400 ARLINGTON, TEXAS 76011~4006 February 25, 2005 Gregg R. Overbeck, Senior Vice President, Nuclear Arizona Public Service Company P.O. Box 52034 Phoenix, Arizona 85072-2034

SUBJECT:

MEETING

SUMMARY

DISCUSSING PAL0 VERDE APPARENT VIOLATIONS

Dear Mt. Overbeck:

This refers to the Pre-decisional Enforcement and Regulatory Conference conducted at the NRC Region IV Office, Arlington, Texas, on February 17, 2005. The meeting attendance list and a copy of the presentations are included as Enclosures 1 and 2. No commitments were made by the licensee during the conference.

In accordancewith Section 2.390 of the NRCs "Rules of Practice," Part 2, Title I O , Code of Federal Regulations, a copy of this letter and its enclosures will be available electronically for public inspection in the NRC's Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http:llwww.nrc.Qovlreadina-rmla~a~s. html (the Public Electronic Reading Room).

Should you have any questions concerning this matter, we will be pleased to discuss them with you.

Troy W. Pruett, Chief Project Branch D Division of Reactor Projects Dockets: 50-528 50-529 50-530 Licenses: NPF-41 NPF-51 NPF-74

Enclosures:

1. Meeting attendance list

2. Presentations

Arizona Public Service Company cc wlenclosures:

Steve Olea John Taylor Arizona Corporation Commission Public Service Company of New Mexico 1200 W. Washington Street 2401 Aztec NE, MS Z110 Phoenix, AZ 85007 Albuquerque, NM 87107-4224 Douglas K. Porter, Senior Counsel Cheryl Adams Southern California Edison Company Southern California Edison Company Law Department, Generation Resources 5000 Pacific Coast Hwy. Bldg. DIN P.O. Box 800 San Clemente, CA 92672 Rosemead, CA 91770 Robert Henry Chairman Salt River Project Maricopa County Board of Supervisors 6504 East Thomas Road 301 W. Jefferson, 10th Floor Scottsdale, AZ 85251 Phoenix, AZ 85003 Brian Almon Aubrey V. Godwin, Director Public Utility Commission Arizona Radiation Regulatory Agency William B. Travis Building 4814 South 40 Street P.O. Box 13326 Phoenix, AZ 85040 1701 North Congress Avenue Austin, TX 78701-3326 M. Dwayne Carnes, Director Regulatory AffairslNuclear Assurance Palo Verde Nuclear Generating Station Mail Station 7636 P.O. Box 52034 Phoenix, AT 85072-2034 Hector R. Puente Vice President, Power Generation El Paso Electric Company 310 E.Palm Lane, Suite 310 Phoenix, AZ 85004 Jeffrey T. Weikert Assistant General Counsel El Paso Electric Company Mail Location 167 123 W. Mills El Paso, TX 79901 John W. Schumann Los Angeles Department of Water 81Power Southern California Public Power Authority P.O. Box 5111I , Room 1255-C Los Angeles, CA 900516 100

Arizona Public Service Company Electronic distribution by RIV:

Regional Administrator (BSMI )

DRP Director (ATH)

DRS Director (DDC)

DRS Deputy Director (MRS)

Senior Resident Inspector ( G W 2 )

Branch Chief, DRPID (TWP)

Senior Project Engineer, DRPID (GEW)

Team Leader, DRPITSS (RLN1)

RlTS Coordinator (KEG)

RidsNrrDipmLipb SlSP Review Completed: -TWP-ADAMS: J Yes 0 No Initials: -TWP-f Publicly Available 0 Non-Publicly Available U Sensitive Non-Sensitive I I I OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax

PREDECISIONAL ENFORCEMENT & REGULATORY CONFERENCE ATTENDANCE LlCENSEE/FACILlTY Palo Verde Nuclear Generating Station DATE/TlME February 17,2005 8:OO a.m.

CONFERENCE USNRC Region IV Offices LOCATION Arlington, Texas Bruce Mallett USNRC Region IV Regional Administrator Art Howell 1 USNRC Region IV I Director, DRP Tony Vegel

~

I USNRC Region IV I Deputy Director, DRP Troy Pruett USNRC Region IV Chief, Projects Branch D Mike Hay USNRC Region IV Action Chief, Projects Branch C Scott Schwind I USNRC Region IV I

__ ___ -~

SRI, Cooper Nuclear Station

~~

Gary Sanborn I USNRC Region IV I Director, ACES ~

~ ___

Karla Fuller I USNRC Region IV I Regional Counsel David Loveless I USNRC Region IV I Senior Reactor Analyst Greg Warnick I USNRC Region IV I SRI, PVNGS Greg Werner I USNRC Region IV I - - ~ ~-

Senior Project Engineer, Branch D

-~

Nick Taylor USNRC Region IV Project Engineer, Branch D Jeff Clark USNRC Region IV Chief, Engineering Branch, DRS Linda Smith USNRC Region iV Chief, Plant Engineering Branch, DRS

I PREDECISIONAL ENFORCEMENT & REGULATORY CONFERENCE ATTENDANCE DATERiME February 17,2005 8:OO a.m.

CONFERENCE USNRC Region iV Offices LOCATION Arlington, Texas EA NUMBER EA-04-221 I NAME (PLEASE PRINT)

! NRC REPRESENTATIVE ORGANIZATION Victor Dricks USNRC Region IV Public Affairs Officer Neil Keith USNRC Region IV Plant Engineering Branch, DRS Enforcement Specialist 1 Charlie Stancil 1 USNRC Region IV Project Engineer, Branch B Mark Schaeffer USNRC Region IV Deputy Director, DRS John Huang USNRC Office of Nuclear Division of Engineering Reactor Regulation Charlie Stancil USNRC Region IV Project Engineer, Sranch A Steven Alferink USNRC Region IV Reactor Inspector, DRS Cale Young USNRC Region IV Reactor Engineer, DRP TSS Reactor Inspector, DRS Plant Engineering Branch, DRS Bo Pham USNRC Off ice of Nuclear DLPM Reactor Regulation Andrew Howe USNRC Office of Nuclear DSSNSPSB Reactor Regulation Bill Cook USNRC Region I Branch Chief, DRP I I Page -&. of 3

PREDECISIONAL ENFORCEMENT& REGULATORY CONFERENCE ATTENDANCE I

LlCENSEE/FAClLlTY Palo Verde Nuclear Generating Station DATEnlME February 17,2005 8:OO a.m.

CONFERENCE USNRC Region IV Off ices LOCAJlON Arlington, Texas EA NUMBER EA-04-221 NRC REPRESENTATIVE I

NAME (PLEASE PRINT) ORGANIZATION TITLE Warren Lyon USNRC Off ice of Nuclear DSSNSRXB Reactor Regulation

PREDECISIONAL ENFORCEMENT & REGULATORY CONFERENCE ATTENDANCE 1 LJCENSEE/FACILITY Palo Verde Nuclear Generating Station CONFERENCE USNRC Region 1V Offices LOCATION Arlington, Texas EA NUMBER EA-04-221

~ -.

1 NAME (PLEASE PRINT) ORGANtZATION I TITLE

LICENSEE/FACILITY Palo Verde Nuclear Generating Station

~

DATE/TlME February 17,2005 8:OO a.m.

CONFERENCE USNRC Region iV Offices LOCATION Arlington, Texas EA NUMBER EA-04-221 LICENSEE REPRESENTAT 'ES I'

I NAME (PLEASE PRINT)

I 0RGANlZATlON TITLE c I Page Z of E

PREDECISIONAL ENFORCEMENT & REGULATORY CONFERENCE AllENDANCE I

I1 I)ATE/TIME February 17,2005 8:OO a.m.

CONFERENCE USNRC Region IV Offices LOCATION Arlington, Texas I. EA NUMBER EA-04-221 I PUBLIC AITENDEES I

NAME (PLEASE PRINT) ORGANIZATION I TITLE I

5 1 1 U I H ~ S&+ld%elGcCT Page of 2

EiKLOSURE 2 PREDECISIONAL ENFORCEMENT AND REGULATORY CONFERENCE AGENDA CONFERENCE WITH ARIZONA PUBLIC SERVICE COMPANY February 17,2005 NRC REGIQN tV, ARL'INSTON, TEXAS 8:OO - 8:30 a.m.

Introductions& Opening Remarks Art Howell, Director, Division of Reactor Projects Tony Vegel, Deputy Director, Divisionof Reactor Projects Enforcement process Gary Sanborn, Director, Allegation Coordination t#

Enforcement Staff (ACES)

Apparent Violations Scott Schwind, Senior Resident Inspector, Project Branch C, Division of Reactor Projects 8:30 a.m. 12:OO p.m. (Breaks at 10 a.m. and 12:OO p.m. for public comment)

Opening Remarks Jim Levine, Executive Vice President, Generation System 8t Investigation Overview Mike Winsor, Director, Nuclear Engineering Testing Results and Safety Analysis Mark Radspinner, Section Leader, System Engineering, Mechanical NSSS Risk Significance Evaluation Gerry Sowers, PhD, Section Leader, Probability/Risk Assessment 1200-1:00 pm.

Lunch Break 1:OO - 4:30 p.m. (Break at 2 0 0 pm. for public comment)

Root-cause Investigation Gerry Sowers, PhD, Section Leader, Probability/RiskAssessment 10 CFR 50.59 Tom Weber, Section Leader, Licensing CRDFVOD George Andrews, Section Leader, Reactor Engineering Operational Decision-Making Terry Radtke, Director, Operations Lessons Learned

NRC Enforcement Program Predecisional Enforcement Conference with Arizona Public Service Co.

February 17,2005 Arlington, Texas Key Points

. Most violations at power reactors processed under SDP

.lo CFR 50.59 violations processed under Enforcement Policy

.Main difference are CP factors Significance of 50.59 violations

. determined by risk No final decisions have been made

Decision Process Determine whether violation occurred Determine significance of violation Evaluate all circumstances 1 Determine sanctions 3

Factors in Determining Significance Actual safety consequences Potential safety consequences Impact on NRCs regulatory process Willfulness

Significance of 50.59 violations Enforcement Policy, Supplement I:

Severity Level I11 .... A failure to obtain prior Commission approval required by 10 CFR 50.59 for a change, in which the consequence of the change, is evaluated as having low to moderate, or greater safety significance (Le., white, yellow, or red) by the SDP.

5 Possible Outcomes

=No action Notice of Violation HNOVwith Civil Penalty ($)

Order 6

Key Points About Civil Penalties Considered for Level I, 11, and III violations

.May be assessed for each violation or grouping of violations and for each day v

.violation occurred Based on type of license and significance of

.violations Current base penalty for power reactors is

$130,000 (for Level I violation)*

• $120,000 in this case \

Factors in Assessing Civil Penalties

.. History of significant violations

. Willhlness Circumstances surrounding

..identification Corrective action taken Discretion ('judgment) 8

CP Flow Chart YES severity Level I, II V i and IH vioiations &

VidatiOnsRelatedtO White, Yellow, 01 Red SDP Findings with ActualcMlseguence 2xBase

,MI penany Post-Conference Process

. Review all information

'Panel to reach fmal decision Develop actions

.Notify licensee & issue actions

.Issue press release for civil pena ties and orders 10

--______- __ ~-

Appeal Rights

'Any agency action may be challenged Challenge may resuit in

. reconsideration of action or hearing Civil penalties and orders provide hearing rights

Terry Radtke Director Operations aperational Decision Making Lessons Learned

+ Four major issues identified from the RAS sump event

- Control room notification af issues impacting operability not performed in a timely manner

- Operability assessment not performed in a timely manner

- Compensatory action credited without a completed 50.59 review

- Indeterminate condition not recognized, requiring entry into LCO 3.0.3 1

Operational Decision Making Lessons Learned

+ Management review team process implemented

- Engages diverse and specialized expertise to discuss issues

- Shift managers role has been clarified to station personnel and reinforced with shift managers to ensure they are Immediately notified Involved in the technical discussions as they evolve Continuously assessing operability Ultimately responsible for making operability determinations Operational Decision Making Lessons Learned I+ Management review team process implemented

- Operational decision-making practice utilized Defines scope of condition Operational significance determined Determines best technical solutions Considers operational challenges, licensing compliance and effects on safety margin Appoints designated skeptic 2

Operational Decision Making Lessons Learned

+ We have had a number of opportunities to exercise the MRT concept since the M S sump event

- Areas of improved performance Shift managers are engaged Heightened sensitivity to enter the operability determination process Roles and responsibilities are established for making and implementing decisions A designated skeptic challenges decision making Operational Decision Making Lessons Learned

+ Areas of improved performance

- Implementation plans are developed to effectively communicate actions, respensibilities, compensatory measures and contingencies to ensure successful outcomes

- Potential consequences of operational challenges are clearly defined, and alternative solutions are rigorously evaluated

- Provides structured, facilitative approach using a specialized and diverse group of people 3

Operational Decision Making Lessons Learned

+ Areas where improvement is still needed

- Promptly reporting conditions that potentially challenge safe, reliable operation to the control room for resolution

- Recognizingcorrective actions that, in themselves, create a degraded or non-conforming issue that require an operability determination

- Continuing to lower the threshold for when the operability determination process is entered Operational Decision Making Lessons Learned

+ Conclusions

- We have learned Prompt notification of issues impacting operability to the control room is essential implementation and execution of the operability determination and 50.59 processes has to be flawless Shift managers have to be engaged in the technical discussions, continuously assessing operability and ultimately making the operability calls Heighten the sensitivity and lower the threshold for entering the operability determination process 4

Operational Decision Making Lessons Learned

+ Conclusions

- What we have changed Implemented MRT process for significant events and issues Clarified and reinforced the shift managers role Revised the operability determination procedure

~

Updated procedures and communicated expectations to station personnel concerning prompt notification to the control room on issues impacting operability Established an operational decision making tool 5

Station Response George Andrews Section Leader Reactor Engineering Agenda

+ Review of NRC inspection report finding

+ Evolutionary/background information

+ Corrective action program

+ Operability determination program 1

NRC Inspection Report Finding Non-Cited Violation (NCV)

...failure of Engineering and Operations personnel to implement requirements in the stations condition reporting and operability determination procedure following identification of a degraded condition.

+ We agree with this finding Background Information

+ Station evolution to remove potential distractions from the control room

- Work Control process

- Site manager position

- Role of STA section leader

- Engineering desire to provide a solution to a problem 2

Corrective Action Program I Agenda

+ Sequence of eventsldiscussion

+ Causes

+ TransportabiIity

+ Corrective actions

+ Summary of conclusions Corrective Action Program Sequence of Events/Discussion Condition reporting procedure requirements If the condition meets either of the following criteria:

1. The condition requires immediate action to ensure the safety of plant personnel or equipment, OR

2. The condition is a non-conforming condition or may cause a degraded condition (i.e., loss of quality or function) in a plant system, structure or component Then the originator SHALL:

1. Promptly notify the Shift Manager of the affected unit(s)

2. Initiate andlor take any required immediate actions 3

Corrective Action Program Sequence of Events/Discussion Thursday, July 29

+ (1527) The design engineer initiated a condition report (CRDR) and electronically selected control room review required option but did not notify the shift manager

+ The Design Engineering section leader did not review the CRDR verbiage and also did not ensure that the shift manager was notified of the condition Corrective Action Program Sequence of Events/Discussion Friday, July 30

+ (-0700) The Design Engineering section leader requested operations support from the Shift Technical Advisor section leader

+ (-0730) STA Section Leader immediately informed the shift managers and Operations director

+ (-0800) Operations, including the shift managers discussed the condition and entered the operability determination program 4

Corrective Action Program Causes and Transportability

+ Causes

- Failure to follow the condition reporting procedure (timeliness of notification of shift manager)

- Inadequate knowledge of electronic condition reporting process

+ Transportability

- Work control process

- Deficiency work order (DFWO) process Corrective Action Program Cerrective Actions Condition reporting procedure revision (complete)

If the condition noted requires I. Immediate action to ensure the safety of personnel or plant equipment OR

2. The condition is a nonconforming condition or may cause a degraded condition (Le., a less of quality or function) in a plant system, structure or component The orininator SHALL (A) Immediately notify (in person or by telephone) the shift manager of the affected unit@)

AND (B) Initiate andlor take any required compensatwy actions 5

Corrective Action Program Corrective Actions

+ Site-wide communications via For Your Information (FYI) process (complete)

+ Formal briefing on condition reporting procedure requirement (complete)

+ Revise work control process (complete)

+ Revise DFWO process (Complete)

+ Revise electronic CRDR process to route to control room directly (complete)

Corrective Action Program Corrective Actions

+ Electronic CRDR tool enhancement (complete) 6

Corrective Action Program Corrective Actions

+ Engineering

- Industry Events Training,

- To be completed by March 31,2005

+ Classroom training for procedure use and adherence for all station personnel

- To be completed by December 37,2005 Corrective Action Program Conclusion

+ In this situation, station personnel failed to correctly implement the condition reporting procedure resulting in untimely notification of the shift manager of a significant condition adverse to quality 7

Operability Determination Program Agenda

+ Sequence of events/discussion

+ Causes

+ Transportability

+ Corrective actions

+ Summary of conclusions Operability Determination Program Sequence of Events/Discussion

+ Design engineer identified the condition

+ Previous design engineer was out of town, unavailable to provide support

+ Design Engineering section leader wanted to perform further review and believed he had three working days to complete review

+ Engineering contacted STA section leader for additional support 8

Operability Determination Program Sequence of EventslDiscussion Friday, July 30

+ (-9738j STA section ieader notified the shift managers and Operations management of issue

+ (-0800) Engineering, Operations and management believed the condition was too obvious to have gone undetected for such a long period of time

- Nothing had changed for 20 years

- A calculation or evaluation of the configuration must exist

+ (-0800) The operability determination program -

was entered Operability Determination Program Sequence of Events/Discussion Friday, July 30

+ (-0800) STA section leader stated immediate operability call must be made by end of shift

+ (-0800) Engineering began system behavior evaIuation

+ (-1000) Operations began developing process to fill the empty pipes

+ (-1 100) Compensatory action was identified to eliminate large portion of void 9

Operability Determination Program Sequence of Events/Discussion Friday, July 30

+ (-1300) Engineering stated that a calculation could be done but it would not be completed by the end of shift

- lssue#l

+ (-1 300) Engineering evaluation concluded that the void would not vent back to containment following a RAS

- lssue#l Operability Determination Program Sequence of Events/Discussion Friday, July 30

+ (-1400) Compensatory action was discussed and accepted by Operations management to eliminate large portion of void

- lssue#l

+ Shift managers were aware of condition but were not directly involved in discussions between Engineering and Operations management

- Issue #2 10

Operability Determination Program Sequence of Events/Discussion Friday, July 30

+ (-1400) Engineerings understanding: No further immediate support required as Operations was going to eliminate void via compensatory action and addition of water

+ (4700) Operations understanding: The compensatory action would eliminate the large portion of the void and only the small void must then be addressed prior to the end of the current shift Operability Determination Program Sequence of Events/Discussion Friday, July 30

+ (-1730) At the end of the shift, Engineering provided judgment that the small void would not impact pump operation

+ (-1 800) Operations completed the immediate operability assessment, concluding ECCS was operable based on compensatory action and engineering judgment on the small void

- Issue #3 11

Operability Deter ination Pro Sequence of Events/Discussion Friday, July 30 - Saturday, July 31

+ Recognizing the significance of relying on the compensatory action, direction was given to expeditiously fill the piping with borated water

+ In the process of developing a method to fill the piping, it was determined necessary to fill the containment sumps as well Operability Determination Program Sequence of Events/Discussion

+ Issue #I: The immediate operability determination was not timely

+ Issue #2: An indeterminate condition was not recognized and thus Technical Specification LCO 3.0.3 was not entered

+ Issue #3: A compensatory action was inappropriately credited to maintain operability and without completion of a 50.59 evaluation 12

Operability Determination Program Causes and Transportability

+ Causes

- Failure to follow the operability determination procedure

- Management failed to recognize the significance of the issue

- Operations management directive versus facilitative Took lead to ensure an evaluation complete by the end of shift Inadvertently insulated/failed to keep shift managers informed of evolution of information

- Misconception of how to apply 10 CFR 50.59 to compensatory actions to maintain operability Operability Determination Program Causes and Transportability

+ Transportability

- Previous operability determinationswith compensatory actions 10 CFR 50.59 reviews performed after crediting the action 13

Operability Determination Pro Corrective Actions Training 4 Training for Operations licensed personnel and shift technical advisors on operability determination program, process and requirements (Complete) 4 Targeted training on operability determination process and the use of compensatory actions

- To be completed by December 31,2005 Operability Determination Program Corrective Actions Operability determination procedure revision

+Requires the shift manager to make and document an immediate operability call ASAP and generally within two hours following notification of Operations (complete)

+Management review team concept is required (complete)

+ Clarifies requirements associated with crediting compensatory actions and review pursuant to I O CFR 50.59 (complefe)

+Includes a checklist and flow chart to aid the shift manager in the immediate operability determin (complete) 14

Operability Determination Program Corrective Actions Operational decision making process

+ External assessment of Operations leadership style and operational decision making

- in progress, to be completed by February 28,2005

+ Internal assessment of operational decision making

- scheduled for 3rdquarter 2005 Operability Determination Program Conclusion

+ In this situation, station personnel failed to correctly implement the operabiIity determination procedure resulting in an untimely and incorrect assessment of operability 15

containment ECCS sump Suction Line Root Cause Investigation Gerald Sowers, PhD Section Leader Probability Risk Assessment Agenda

+ Investigation team

+ Condition description

+ Cause of condition

+ Transportability

+ Extent of condition

+ Missed opportunities 1

Investigation T am Charter associated with ECCS sump suction piping found unfilled including

- Root cause and contributing causes Investigation Team

+ Engineering

- Team leader

- 3 design engineers

- 1 system engineer

- 1 maintenance engineer

+ Operations

- Ioperations standards advisor

- 1 STA(part time)

+ Performance Improvement International consultant

+ Essentially full time from September to December 2

Condition Description

+ A section of ECCS suction piping was discovered to be unfilled. This was

- Contrary to the original design intent, and

- Unanalyzed

+ This unanalyzed condition has existed since plant start-up and went unaddressed until July 2004 Cause Investigation Review of

+All operation, surveillance and test procedures related to the sump suction line and

+Interviews with plant operators confirmed that there never was a procedural requirement to fill the section of sump piping as a prerequisite to placing the system in operation 3

Cause Investigation

+ Three potential failure modes I.The design requirement was specified, but the user failed to consider the design requirement and did not incorporate it into station procedures

2. The design requirement was recognized, but there was a breakdown in communicating the desiqn requirement to the end user

3. The design requirement was not recognized by the responsible design organization Cause Investigation

+ A review of the historical documents used as references for the development of operating and test procedures found no mention of the need to maintain the sump suction line filled

- System Description Manual (Bechtel)

- Design Criteria Manual (Bechtel) 1

- CE procedure guidelines

- CE interface requirements

- Engineering Evaluation Requests (APS)

- Startup documents I

4

Cause l ~ v e ~ t i g a t i u ~

+ Three peripheral documents were identified that referred specifically to the need to fill this section of pipe

- Independent design review transcripts (May 1981)

- CE letter to Bechtel about leaking seats on the sump isolation valves and the need for Type C testing (1/5/1984)

- CE calculation providing information for NRC Bulletin 85-03 (MOV switch settings) response (11/26/1986, after Unit I license)

+ None of these documents were intended or expected to be used by procedure authors as references for operating procedure development Cause Investigation

+ Several documents (FSAR, PVNGS SER) refer to filling the SI piping in general with no reference to specific parts of the system, in the context of prevention of water hammer

+ Technical Specifications only required verifying that the discharge piping was full and did not mention the suction piping 5

Cause of Condition

+ Cause #I- A breakdown in communicatingthe design requirement to the end user

- The documents used as references for writing the operating and test procedures did not include the requirement to maintain the sump line in a filled condition

+ Corrective action -Added the requirement to the Safety Injection Design Basis Manual (complete)

+ Corrective action - Changed procedure to require filling the pipe with borated water (complete)

Cause of Condition

+ Cause #2 -The PVNGS technical specifications only required verifying f uII the discharge piping and did not mention the suction piping

- This is consistent with the prevention of water hammer

+ Corrective action -Added periodic verification that ECCS sump containment penetration lines are full to station procedures (complete) 6

Cause of Condition

?, Contributing cause - The design of the system did not facilitate filling this section of piping

- The original "vent and drain" connections were installed to facilitate leak rate tests

- The connections were not standard vent and drain connections

- No fill source was provided

+ Corrective action - Changed the design to add vents, drains and a fill source (complete U3; to complete U1 and U2 in 2005)

Transportability Approach

+ The transportability started with the obvious look for other sections of unfilled pipe

+ The transportability evaluation was extended to identify parts of systems not normally in use and not periodically functionally tested 7

Transportability Scope

+ Safety-related and selected important to safety systems

- Fluid systems

- I&C systems GL 96-01, testing of safety-related circuits

- Electrical systems Transportability Results

+ Fluid systems

- No other piping sections identified as not filled

- Not flow tested A Section of the containment spray piping

- CRDR 2760630, piping verified filled RMWT to AF pumps (RMWT is not the safety-related water source; it is a backup to the CST)

- No problems found 8

TransportabiIity ResuIts

+ I&C systems - Generic Letter 96-01 reviewed for untested actuation circuits

- Initial GL review was comprehensive

+ Electrical systems - Circuits either normally energized, periodically energized (e.g., alternate battery chargers) or periodically tested

- No problems found Extent of Condition Approach

+ All LERs since initial operation were reviewed to establish instances in which a similar condition was identified

- LERs documenting failure to implement design requirements into operating basis

- LERs categorized into three generic failure modes

+ Distribution of failures examined to establish extent of condition

- Global

- Local

- Isolated 9

Extent of Condition (LERs) 4 1 1 2 --

o t - I I 1 1 I l l I I I I I I I I I I t Extent of Condition

+ More instances (three) in SI than other systems

+ SI instances appear to be unrelated

+ A review of all I 9 LERs did not identify issues related by cause

+ Condition is considered to be isolated 10

Missed Opportunities 4 Two direct opportunities identified

- Instruction Change Request 58646 11/16/1992 -

Failed to initiate a corrective action document

- Design basis reconstitution project Failed to follow project procedures Action - Review sample of IDR transcripts (Sept. 2005)

Conclusions 4 Configuration was not maintained as the design intended due to failure of the original design organizations to adequately communicate the design requirement to the operating oraanization 4 Isolated case 11

Palo Verde Investigation of Unfilled Emergency Core Cooling Piping Mike Winsor Director Nuclear Engineering Investigation Director Agenda MORNING SESSION Evaluation of Condition and Significance Determination

- System & Investigation Overview Mike Winsor Director Nuclear Engineering

- Testing Results and Safety Analysis Mark Radspinner Section Leader System Engineering Mechanical NSSS

- Risk Significance Evaluation Gerry Sowers, PhD Section Leader Probability/Risk Assessment 1

Agenda AFTERNOON SESSION Evaluation of Condition and Significance Determination

- Rootcause Investigation Gerry Sowers, PhD Section Leader Probability Risk Assessment Plant Response

- 10CFR50.59 Tom Weber Section Leader Licensing

- CRDWOD George Andrews Section Leader Reactor Engineering

- Operational Decision-Making Lessons Learned Terry Radtke Director Operations

- Concluding Remarks Jim Levine System Overview

+ Combustion Engineering safety injection system

- Two independent trains High-pressure safety injection pump (HPSI)

Containment spray pump (CS)

Low-pressure safety injection pump (LPSI)

Recirculation sump

- Inboard and outboard containment isolation valves

- Recirculation actuation signal (RAS) opens valves and stops LPSl

- System response to a LOCA scenario 2

+ A significant investigation was initiated in to

- Determine root cause of why the condition existed, the missed opportunities to identify the condition and extent of the conditionkause

- Evaluate the safety significance of the as-found condition

- Perform an assessment of the adequacy of the station response upon discovery of the condition Investigation Overview

+ Three teams formed, comprised of more than 20 Palo Verde and consulting personnel

+ Contracted with testing, engineering and industry experts

- Westinghouse

- Fauske and Associates

- Wyle Labs I - MPR and Associates

- Performance Improvement International

- LeBlond and Associates

~ + Established an independent oversight group using recognized industry and academic 3

Investigation Oversight Committee 4 Chartered by executive management - to

- Review team charters

- Periodically review team direction

- Review findings and proposed corrective actions Provide critical feedback and consultation on methods, scope, results, corrective actions and conclusions for all areas of the investigation Executive Oversight Cornmittee Committee members Ed FOX Executive Chairman, APS VP Communication, Environmental and Safety Warren Peabody PVNGS Nuclear Oversight Committee, retired Utility Executive Joe Callan PVNGS Nuclear Oversight Committee, retired NRC Andy Kaufman - Senior Associate, Continuum Dynamics, Inc.

Norton Shapiro - Senior Consultant Engineer Safety Analysis, Westinghouse Michael Mancini - Consultant, former VP, lngersoll Pump Co.

4

Root Cause and Extent of Condition I

i

+ Charter

- Determine root cause for leaving the piping unfilled since plant startup

- Identify and evaluate causes for missed opportunities to correct the condition

- Determine the extent of cause and condition Plant Response

+ Charter

- Review adequacy of plant response Condition reporting Timeliness of control room notification Operability determination Compensatory actions 50.59 adequacy, including program review Human performance analysis 5

Safety Significance

+ Charter

- Develop a full understanding of the system response to the void and the resulting fluid conditions delivered to the pumps

- Determine pump performance from the resulting fluid conditions

- Using the pump and system response to the as-found condition, determine the safety significance 6

Plant Response Significant CRDR Investigation Tom Weber Section Leader Licensing Plant Response Significant CRDR Investigation Charter

- 10 CFR 50.59 program

- Corrective action program (CRDR)

- Operability determination program (OD) 1

Plant Response Significant CRDR Invest igation

+ !nvestigatbn dir&er

- Michael Winsor

+ Team members

- Jon Sears, qualified CRDR .,ivestiga,ar

- Thomas Weber, lead for 10 CFR 50.59

- George Andrews, lead for CRDWOD

- Peter LeBlond, LeBlond and Associates Violations

+ Apparent violation

- 1992 procedure change

+ Green non-cited violation

- Three examples 2

Apparent Violation 1992 Procedure Change 1992 Procedure Change

+ Technical specification surveillance test

- Emergency core cooling system (ECCS) leakage test Tested every 18 months during outages Performed in Mode 5 or lower when ECCS is not required to be operable

- Test sequence (prior to procedure change)

Attach temporary test rig Fill piping with demineralized water Pressurize line to >40 psig Walk down system to identify leakage Remove temporary test rig 3

1992 Procedure Change

+ ASME Section XI surveillance test

- ECCS sump valves stroke test Tested every quarter Performed in any mode Performed subsequent to ECCS leakage test

- Test sequence Valves stroked open then closed Resulted in demineralized water being drained into RAS sump 1992 Procedure Change

+ Instruction Change Request 61008, May 1992

- Written against ECCS leakage test

- Concerned with cleanup needed after demineralized water drained into RAS sump

- Requested steps to remove demineralized water after TS testing was complete

- TS surveillance test procedure changed -June 1992 (10 CFR 50.59 determined to be not applicable) 4

1992 Procedure Change i Frocedure change Was maintenance activity

- Removed demineralized water (test medium) after testing (prior to ASME test)

- Restored piping to as-found condition (no change)

- Perpetuated erroneous plant configuration (piping not filled) 10 CFR 50.59 Non-cited Violation

+ Manually open inboard ECCS suction valves

+ Filling the ECCS sump

+ I O cubic feet of air 5

Manually Open Inboard ECCS Suction Valves

+ Twc ccncerns with 50.59 review

- Not completed prior to implementation

- Ended at screening step I

Manually Open Inboard ECCS Suction Valves

+ Not completed prior to implementation

- Cause Failure to follow procedure

- Completed corrective actions STAs instructed to complete 10 CFR 50.59s prior to implementation Revised the operability determination procedure 6

Manually Open Inboard ECCS Suction Valves

+ Ended at Screening step

- Cause Focused on degraded condition Failed to understand change under review

- Completed corrective action Withdrew compensatory action and cancelled screening

- Pending formal training, licensing review required prior to implementation 10 CFR 50.59s needed for ODlcompensatory actions OD procedure revised to include this interim measure Filled ECCS Sump

+ Not completed prior to implementation

- Cause Failed to re-evaluatethe change after the decision was made to add water to sump

- Initially only piping was planned to be filled

- Scope changed to include sump

- Completed corrective actions Entered condition in corrective action program Performed the I O CFR 50.59 7

Filled ECCS Sump c Ended at screening step

- Cause Failed to demonstrate change was NOT adverse Evaluation criteria used in screening step

- Completed corrective action Completed 10 CFR 50.59 evaluation I O Cubic Feet of Air c Area of disagreement with inspection report

- The NRC cited Palo Verde for not performing a written safety evaluation prior to filling the line c NE1 96-07, Rev I,Section 4.4 states

- Three general courses of action are available to licensees to address nonconforming and degraded conditions. Whether or not I O CFR 50.59 must be applied, and the focus of the 10 CFR 50.59 evaluation if one is required, depends on the corrective action plan chosen by the licensee.

8

I O Cubic Feet of Air

+ Three courses of action

1. The licensee intends to restore the structure, system or component back to as-designed condition. The activity is not subject to 10 CFR 50.59

2. An interim compensatory measure is taken to address the condition. I O CFR 50.59 applies to the interim compensatory measure, not the degraded condition

3. The licensee intends to accept the condition as-is resulting in something different than its as-designed condition, or to change the facility or procedures. The final corrective action becomes the proposed change that would be subject to I O CFR 50.59 I O Cubic Feet of Air

+ The I O cubic feet of air was a non-conforming condition subject to GL 91-18

- The acceptability of the non-conforming condition was evaluated in the OD process

- Was not the final corrective action

- Final corrective action was to fill the piping with borated water to restore to original design condition

+ NE1 96-07, Rev I,Section 4.4 states

- This activity is not subject to I O CFR 50.59.

+ In compliance with NE1 96-07 and GL 91-18, no 10 CFR 50.59 performed 9

Extent of Condition

?, Reviewed 283 operability determinations

- Four pre@na&Xk M $0 CFR 50.59s associated with (itomwmatory actions All four -1 50.59s were completed after the actions were implemented Two of the four 10 CFR 50.59s inappropriately stopped at the screening step

- Condition was limited to STAs in completing 10 CFR 50.59s for OD compensatory actions Extent of Condition

+ Completed corrective actions

- Revised the OD procedure

- Completed the evaluation for the 10 CFR 50.59 that inappropriately stopped at the screening step

+ Additional corrective actions

- Provide formal training to STAs OD compensatory actions and the application of 10 CFR 50.59 by May 31,2005 10

I O CFR 50.59 Program Review r Oversight/Evaluations

- Program manager review I

80% of the screenings and evaluations

- NRC Inspection Report (10 CFR 50.59 program inspection) 8 evaluations and 14 screenings No findings

- PVNGS self-assessment I - Trend CRDR I O CFR 50.59 Program Review

+ Areas requiring attention

- Program entry requirements

- Screening vs. evaluation

- Thoroughness of documentation

+ Qualified I O CFR 50.59 personnel to complete enhanced requalification training

- Individuals who have not completed the training by March 31 will have their qualifications revoked 11

Summary

+ Problem identified with performing I O CFR 50.59s for compensatory actions within ODs

+ Corrective actions implemented

+ Areas requiring attention are being addressed 12

Risk Significance Determination Gerald Sowers, PhD Section Leader Probability Risk Assessment Significance CalcuIation assumptions Calculation Conclusion 1

Significance I Region IV analysis assumptions without testing resuits PVNGS analysis assumptions with teesting r@suats HPSI always fails on RAS HPSI fails on RAS for breaks C 2.3 diameter CS always fails on RAS CS never fails Operators recover HPSI No recovery Operators recover CS CS never fails No backup to HPSI Depressurize, LPSl or CS as backup per EOPs RCP seal LOCAs contribute to RCP seal LOCA does not risk . contribute to risk Significance Calculation Internal Events Initiating Event Region IV PVNGS Approx. (ACDP) (ACDP)

Large LOCA 1.44e-06 0 Medium LOCA I.06e-05 0 ISmall LOCA 1 9.15e-07 1 4.5e-6 I

Transients (PSV)

I 2.89e-06 I

2.7e-7 I I LOOP (RCP seal LOCA)

I 9.72e-08 I

I o

Total 1.59e-05 4.8e-6 I

2

Significance Calculation - External Events Initiating Events Region IV I PVNGS (ACDP) I (ACDP) 7.90e-06 4.7e-7 Internal Floods Fire ~-

2.44e-09 8.83e-06 1.Oe-8 2.3e-6 j,

I Significance Calculation Conclusions Events Region IV PVNGS (ACDP) (ACDP)

Internal Events 1.6e-05 4.8e-6 External Events 8.8e-06 Total 2.5e-05 7.0e-6 3