ML17304B214

Unit 3 Reactor Trip Following Large Load Reject.

ML17304B214
Person / Time
Site:	Palo Verde
Issue date:	05/19/1989
From:	ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR
To:
Shared Package
ML17304B213	List: ML17304B212 ML17304B214 ML17304B215
References
2-3-89-001, 2-3-89-1, TAC-73246, NUDOCS 8906050218
Download: ML17304B214 (136)

Text

PALO VERDE NUCLEAR GENERATZNG STATION INCIDENT INVFWTIGATIONREPORT NUMBER: 2-3-89-001 TXTILE: UNZIP 3 RE~ACTOR TG~JP

'FOLLOWING LARGE LOAD RE JECT, PPP DATE. MARCH 3, 1989 REPORT APPROVAL DATE: MAY 19, 1989 ANPP Encfdent Investigation Program 79DP-OIPOl, INCIDENT INVESTIGATION REPORT PREPARATION - Appendix A

PALQ VERDE NUCLEAR GENERATING STATION INCIDENT INVEKHQATIONREPORT NKMHEZh 2-3-89~01 TT]~: Unit 3 Reactor Tri Followin Lar e Load Re ect 5HPe Team hhmnher Date Reviewed By: 5 i~a Date Reviewed By:

Reviewed By-.

Revhmrad By:

Revere@ By:

~tl~lBg Investig~ Directnr EVICT DATE: March 3 1989 REPORT APPROVE DATE:

ANPP Incident Investigation Program 79DP-OIP01, INCIDENT INVESTIGATION REPORT PREPARATION - AppcndIx.A- 2 0

~c~~ ~~CATIONRI~QRT'~~g. 2-3-89 PP y UKIT 3 REACTOR TRIP FOLLOWXNGLARGE LOAD REJECT INCIDENTINVESTIOA 0 TeamLeaderi M. R. Oren / SI'A Print Name- Signature - epartment ate Team Ldr E. P. Sonn / SPA Wll PV Print Name - Signature Department Date Team Member > G. W. Sovrers / EED W iS'7 Print Name- Signature - Department Date Team Mem~er L. A, Florence / U3 OP 0 Print Name- Signature -

yeamMembezi M. J. Beyer U3 WC /

rtmen Print Name- Signature - Department Date Date Team Member t J ~. ~~olds

~ / U3-Main< 5 "/A-d)

Print Name- Signature Department TeamMember~ L. D. /

Stricker PS8tC Print Name- Signature - Department ate ember.. T. K P~ps / ISEG Print Name- Signature - epartment Date K%MTDATE: 3/3/89 RmORT AmROVALDATE: >/>9/89 ANPP Incident Investigation Program 79DP-OIPOl, INCIDENT INVESTIGATION REPORI'REPARATION - Appendix A- 5

PALO VERDE NUCLEAR GENERATING STATION 1h Ml g~~i I thth action. They do not indicate a review af the camp1eteaxess ax'f the imnmtigation pracess ar the reFart.

Item: 5.26

~p ViSi h6~ w/iz/&

VP Nuc Prod V3?.

J. G. Haynes, Item: 4.2. 5.16. 5..17, 5.18. 5.20. 5.21. 5.23. 5.25. 6.3.".8.7. I I . C. I Plant Director W. C. Marsh 2.3. 6.2. 7.8 0

NSG/ISG Director WS'. Quinn QA Director B. Ballard 2.1.1 IIT Leader M. R Oren Item:

ANPP Incident Investigation Program 79DP-OIPOI, INCIDENT INVESTIGATIONREPORT PREPARATION - Appendix A - 4

PALO VERDE NUCLEAR GENERATING STATION Th Ibll 8!~i h a=tian. Zh~ dn ILat indicate a rex~ af the campletexmss ar thtJ j~

af the invastigatian prcaess ar the repart.

gem: 5.5. 5.8. 5.22. 7.1. 10.5.2 PS & C Manager R. E. Younger Meaager Item: 2.1.4, 2.1.5. 3.2. 5.11.2

'r

/ 4'~ ~

Qp/pc~

G P PS&C I&C R.,E. Younger Manager Item: 2.2. 5.27. 5.28. 8.4 PS&CSTA II

'5'l 1~ s/ic. ~f E. Younger 5.4.a

~le@

c:

PS & C Mech.

R. E. Younger

</'tem:

5.10, 5.12, 11.B.1, 11.B.2,11.C.2.2 PS & C Ops.

f lVhaxager R. E. Younger Item: 5.6... 5.14, 5.15. 5.19, 5.20, 9.5, 9.8, 11.E Training W. F. Fernow ncaa ent vesuga con rogram 79DP-OIP01, INCIDENT INVESTIGATION REPORT PREPARATION-

PALO VERDE NUCLEAR GENERATING STATION

'1h Ibllo g~~ h6 action. Th~ do not iadimda a review of the completmmss or 4p~

of the investigation process or the reFart.

Item: 5.3 b 5 4.b. 5 5 b. 7 2. 9

.4 U-1 Work Control J. D., Dennis homager Item: 5.3.b, 5.4.b. 5.5.b. 7i2 U-2 Work Control Data P. J. WQey Itexn: 2.1.2. 5.3.b, 5.4.b..5.5.b. 7.2. 8.1. 8.5. 9.1. 10.5.1. 10.6. 10.8, 10.10 U-3 Work Control CM1 w~e~~ i i~ i/

C. D. Churchman Item. 9.10 Unit I Chem D. Fuller 'anager Item: 9.10 Unit 2 Chem R. Ferro yt . 9.10 Unit 3 Chem J. Scott ANPP Incident Investigation Program 79DP-OIPOl, INCIDENT INVESTIGATIONREPORT PREPARATION - Appendlv A - 4

PALO VERDE NU'CLEAR GENERATING STATION Th IM: g~ I d 8 action. Tb~ do not indicate a review of the complebene.m ar of the inveetigation process or the repart.

g d 5.4.a. 5.13

~/re/p U-1 Plant Manager W.E. Ide Item: 5.4.a. 5.13

/ I / / ~

U-2 Plant Manager D. Heinicke 5Ma,'5.13 U-3 Plant Manager Mix /ZF R, J. Adney Item: 5 17. 11 D u F'/

U-1 Operations J.J. Scott Date 5.17, 11.D c

U-2 Operations F. C. Buckingham 'anager Item: 5.17, Il.D U-3 Operations R E. Gouge ANPP Incident Investigation Program 79DP-OIP01. INCIDENT INVESTIGA'EON REPOR1'REPARATION - Appendix A - 4

PALO VERDE NUCLEAR

~

GENERATING STATION

'11 Iblh h th th g

comtian. Tb~ da nat'indimda a reviewer af the campletexmss ar af the iuveatigaki,an pnx~ ar the repart.

Ia  : 6.2 7 EED Manager G.W. Sowers Item: 1.1. 1.2. 6.1. 11M 11.B.l C

EED Electrical G.W. Sowers Item:2.1 3. 3 l. 5-11 l. 8 6 EED I Bc C G. lV. Sowezs Itazn: 5.1.1. 5.2. 5.3.a. 5.5.a. 5.22. 5.24. 10.1. 10.2. 10.3

-iz.

Item: 8.1. 8 2 EED NSSS G.W. Sowers Item: 5.1.2. 5.1.3. 7.7 EED Tech. Eng.

G, W. Sowers ANPP Incident Investigation Program 79DP-OIPOI. INCIDENT INVESTIGATION REPORI'REPARATION - Appendix A - 4

PALO VERDE NUCLEAR GENERATING STATION T Iblh I,'~ I h

actian. They do nat indicate a review af the

'8 campiness ar af the ixxvestigatiaa pra(loess ar the re(art.

Item: 10.11, 10.12 a"-f7-8 OCS J. F. Schmadeke Item.. 9.3, 9.4, 9.9'

($ ~ g Emergency Planning F. Bieling, Maaagex'.

Item: 1 1. 4.1'.3. 6 4. 7 3 7 4 7 5 7 6 8 3 9 11 c 5-if-J NED MGlaager E. Sterling I~: 9.6 CHEM STDRS Managex B. Cederquist Item:

Item:

ANPP Incident Investigation Program 79DP-OIPO1. INCIDENT INVESTIGATION REPORT PREPARATION - Appendix A- 4

PAGE 1 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD MM CTION TABLE OF CONTENTS I. Incident Investigation Report Forms pg II. Executive Summary pg III. Introduction pg 4 IV. Event Description V. Per'sonnel Performance Evaluation pg 20 VI. Plant Protection System Response Evaluation pg 26 VII. Control System Response Evaluation pg 29 VIII. Nuclear Safety Assessment pg 30 IK Issues Summary, pg 33

> '1. Issue ¹1 - Subsynchronous Oscillation (SSO) Relay pg 34

2. Issue ¹2 - Steam Bypass Control System pg 37

.3. Issue ¹3 - Safety Equipment Status System pg 41 4; Issue ¹4 - Fast Bus Transfer pg 44,

5. Issue ¹5 - Atmospheric Dump Valves pg 46

6. Issue ¹6 - MSSS Lighting pg 65

7. Issue ¹7 - Instrument Air pg 70 8.. Issue ¹8 - RCS Leak pg 74

9. Issue ¹9 - Potential Human Performance Deficiencies pg 80 During Post-'Mp Response by Radiation Protection/Chemistry Technicians

10. Issue ¹10 - Miscellaneous Equipment Issues pg 89 ll. Issue ¹11 - Human Performance Evaluations (HPES) pg 95 A

Kfguxm FIG. 1) Primary Event and Causal'Factors Chart (E&CF Chart)

FIG. 2) SSO Relay E&CF Chart FIG. 3) Steam Bypass Control Erratic Operations E&CF Chart FIG. 4) Fast Bus Transfer E&CF Chart FIG. 5) ADV Operation from the Control Room and Remote Shutdown Panel E&CF Chart FIG. 6) ADV Local Manual Operation E&CF Chart FIG. 7) MSSS Building Lighting E&CF Chart FIG. 8) Instrument Air E&CF Chart FIG. 9) RCS Leakage from RCP 1B E&CF Chart FIG. 10) RCS Leakage from CHV-435 E&CF Chart FIG. 11) OQ'-Site Dose Calculation Errors EBcCF Chart FIG. 12) Energy-BWer-Target

1) 2-3-89-001 Facts Database

2) Table of Issues, Conclusions. Facts and Recommendations

3) Personnel Statements

4) Shift Technical Advisor Log

5) Unit Log

6) Control Room Log

7) Diagnostic Flow Chart

8) Temporary Data Acquisition System Plots

9) Strip Chart

10) Plant Monitoring System Post Trip Logs ll) Plant Monitoring System Alarm Typer Printout

12) RONAN SOE

13) CPC/CEAC Trip Bufrers

14) Quarantined Material Inventory List

15) Action Plan Guidelines

PAGE 2 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD M&ECTION L INCIDENT INVESTIGATIONREPORT FORMS

PALO VERDE NUCLEAR GENERATING STATION PLANTTPAJ'ISIENT H&flE%'ASSESSMENT EVENT DATE/TIMEl BriefDescription: Unit 3 Reac or Trl followin Lar Load Re ect SHIFT PERSONNEL Shift Supervisor: Dwavne Carnes Asst. Shift Supervisor> Dave mith.

Primary ROi Steve Wackenstedt Secondary RO: Mike Barron Mike Sanch .

I Shift Technical Advisor t Other personnel Involvedin event Jim Proctor. Bill McCaII PLANT CONDITIONS PRIOR TO.THE EVENT.'o der. i Reaeror powers 97 i'> iriWe: 1307 CEA poririoo ~Grou 5 I UEL4 Pressuriser Pressuret 2250 Tavge 591-8 F Fressurjser Levels 51 6%

Boron Concentration: 127 Ppm S/GLevel (%Fide Range) 41/ 4'2: 75% / 75%

e EVOLUTIONS IN PROGRESS PRIOR TO THE EVENT:

Leak Check of AS-PSV-15 in progress ANPP Incident Investigation Program 79DP-OIP01, INCIDENT INVESTIGATION REPORI'REPARATION - Appendix B - 1

PALO VERDE NUCLEAR GENERATING STATION PL/84TTRANSIENT REWIEWASSESSMENT EVENT DATE/'HMEl 3/3/89 01:05 CONTROL SYSTEM STATUS PRIOR TO EVENT:

REACTOR REGULATINGS~USB (OPERATE)ez TEST Tevg Seleeeed I 2 (Ap CEDMCS: ~AS MS MO Ml STANDBY FEED%MER CONTROL MAS'IZR SYPH'/GP e \

5IANl~UTO 1 '/t 4' 5IAIIPAUTC)

DOVHCOMER REG VALVE AUTO)MANUALSTATION: MAN~UT5 MAII/~UTg ECONOIYIIZER REG VALVE AUTO/MANUALSTATION: MAN~V'DP MANQUTg)

PEED PUMP SPEED AUTO(MANUALSTATION:, MAN U MAI'I~V~TO BIAS SETTING +4 0

~~ gyP~

GE CONTROLLER COygggoL CO~QQ MAN U'IQ hIAN/gCUTQ) eEMCTOLCCAL-iCUT@ldAN STPT QIEMCTaiLocAL-+UTtPMANETPT

~R PQ~ ~~

eieeieeilzeeeeeeeeeeeoeeeoL SY~~:

e"ei"

~UPilAIIUALEELEUTeiAA002 RPS/ESFAS/BOP-ESFAS STATUS PRIOR TO EVENT:

(List anychannelstrippedor inbypasspriortothe event,)

NONE

Note

Control System status obtained 3/11/89 by L. Florence from D. Carnes Bc M. Harron.

ANPP Incident Investigatiou Program 79DP-OIP01, INCIDENT INVESTIGATION REPOV PREPARATION - Appendix 8-2

PALO VEHDE NUCLEAR GENERATING STATION PLAKTTRANSIENTREWIEVfASSESSMENT EVENT DATE/TTMEi 3/3/89 / 0 t: 05 REACTOR TRIP INITIATION.

RecordReactor 1st -out: S/G 82 Pressure Lo Record, Turbine 1st -out: No ~Sv Pres Trio Was the Initiating Reactor Trip Parameter (if available) within the required value? YES X NO IfNO, Explain:

I Was the response time {i.e. the time f'rom reaching the process setpoint until the bus undervoltage alarm occurs.) of the initiating trip signal within

'he required value as listed in the PVNQS Technical SpeciQcations?

YES x NO IfNO, Explain:

ESFAS)BOP-ESFAS ACTUATIONS; Record any ESFAS actuations or ESFAS-BOP actuations andthe channels actuated, SIAS, CIAS, i~ISIS (All channels A and 8/1-3 and 2-4)

ANPP Incident Investigation Program 79DP-OIPOI. INCIDENT INVESI'IGATIONREPORT PREMATION - AppcndL~ 8 - 3

PALO VERDE NUCLEAR GENERATING STATION PLANT I'TVQVSIENTREWIRE ASSESSMENT EVENT DATE/TIMEl ~

O IQ~

'f Were any RCS pressure/temperature violatedP YES ~NO +>> limits, as listed in Tech Specs v ~

p )( P q

)

(

YES, provide summaxy and evaluation in the Incident Investigation Report. Outline the circumstances defining the occurrence, the results of any Engineering Evaluations performed to evaluate the impact on the RCS and the actions taken to prevent recurrence of the event.

Does this event require an inspection of the hyydraulic and mechanical snubbers per Surveillance Requirement 4.7.9d? YES ~NO Wi(U'~>>.4rF IfYES, define which areas are to be inspected and provide the appropriate f ($

Inspections. The areas to be inspected are Inspection complete and Satis&ctory: ~

3 PP

~ $0 ~ 0 ~ e ~ e ~ 0 ~ e ~ e ~ 0 ~ Ose ~ 0 ~ 0 ~e 0~

EED Manager or Designee 0

Date

~ 0 ~ e ~ 0 ~ 0 ~ 0 ~ 0 ~ e ~ 0 ~ 0 ~ 0 ~ 0 ~ e ~ 0 ~ 0 ~ e ~ ~ 0 ~ 0 ~ 0 ~ 0 ~ 0 ~ 0 ~ a ~ ~ 0 ~ ~ e ~ ~ 0 ~ a ~ 0 ~ 0 ~ ~ 0 ~ ~ 0 ~ 0 ~ e ~ ~ e~ 00 ~ ~e ~v~0 ~0 ~0 ~0 ~0 ~0 ~0 SAFiETY LIMIT

~NO R&RJHRV'idRCS prescore exceed 2760 psiaP YES IfYES, provide a discussion in the Incident Investigation Report which enumerates the actions taken to comply with Tech Spec. 6.7,1.

Was a CPC - LPD or DNBR trip received'? ~YES NO IfYES, Reactor Engineering will perform an evaluation of the event to determine ifthe Safety Limi 'ted.

~ 0 ~ e ~ ~ e ~ ~ e ~ e ~ a ~ e ~ 0 ~ e ~ e ~ ~ e~~e ~~0~0~0~0~0~e ~~ ~e~~ ~ ~0 ~ ~0~0 0~0~0~~0~~0~~0~~e ~~0~~0~~e ~e ~0~ 0~0~0~

~ 0 ~ aeeaea ~ ~ ~ sea a as ~ e ~

San ty Limit Not violated:

eac or

~ a ~ e ~ ~ e ~ ~ e ~ a ~ e ~ e ~ e ~ e ~ e ~ ~ e ~ ~ Q ~ ~ e ~ 0 ~ e ~ e ~ e ~ Q ~ a ~ Q ~ ~ Q ~ as ngi eering Supervisor or Designee Date e ~ ~ 0 ~ ~ 0 ~ e ~ a ~ e ~ 0 ~ ao ~ e ~ e ~ ~ e

~ ~ e ~ ~ e ~ ~ e ~ ~ 0 ~ ~ e ~ 0 ~ e ~ ~ 0 ~ ~ ~ 0~ ~ ~ 0 ~

e

) e 0~ ~ a ~ ~ 0~ ~ ~ ~ ~ ~ ~ 0 ~ 0 ~

Ifthe Safety limit(s) were violated provide a discussion in the Incident ~

Investigation Report enumerating the actions taken to comply with Tech Spec 6.7.1..

ANPP Incident Investigation Program 79DP-OIP01, INCIDENT INVESTIGATIONREPORT PREPARATION - AppcndI~ g

PALO VERDE NUCLMLB.

GENERATING STATION INCIDENTINVESTIGATIONPLANTRESTMZT AtHWOKZATIQN TIXZ~ICALandCROSS -MANAGIVEEh7I' RKVIEM

~>n Report Technical Review Complete: Y P - <+'< ~ o. &

STA Superviso Date /time

( This shall include a review of the concerns Rom previous events to 8ssure that previously identified corrective actions +rare adequate and appropriate for addressing the concerns identified.)

PRB review complete Unit Mode 2 entry Review based on implementation of'concern resolu'tions as identified iri this report.

r By: 1!i ~~71., 'std" .= i Z4W PRB'hairman Date/time OPERATIONS RE'VHVPF AND APPROV'AL FOR MODE '2 KKITRY Report Review Complete: Unit Mode 2 Entry Recommended based on Implementation of concern resolution as identified in this report, Bye Plant Manager Date/time Corrective Actions required for Mode 2 Entry, as identified in this report, are complete. Mode 2 entry is authorized.

Bye Shift Supervisor Date/time ANPP Incident Investigation Program 79DP-OIPOI. INCIDENT INVESTIGATIONREPORT PREPARATION - AppcndLv B - 5

PALO VERDE NUCLEAR GENERATING STATION INCIDENT IhTVESTIGATIONREPDRT CHECKLIST PART I ERKGtxve Sumxzlaxy Emmet Description F8chs KxBct Id bll' Ccxrrectxve Actions Identifled Organindian Br Indivichxal Mentifled Rr each Carry Action Due Dates Ra all Corrective Actions Mentifled

~

PART II ( Categaxy 1 R 2 Emits axQyy)

E d EddEE I I Iddd E I El EE IEEP Control Sysltexn Evalmdian PART III lg' .. Cvmr Page Revmmr and Apparel Page(s)

.C EE Rl All ts with Ccxrrective Actians speci6ed are inchxded d E EE chart."; (EBT ar ESCFj are inchxded.

Concern Sumxxuay (Ifxxxultiple canc~).

El~

I I FIIEE

'Appezuhx A completed I

I dd d~

ANPP Incident Investigation Program 79DP-OIP01, INCIDENI'NVESTIGATIONREPORT PREPARATION - Appendix D

PAGE 3 IIR 2-3-89-00 1 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP POLLO~Q LARGE LOAD REJECTION II. EXECUTIVE SUMMITRY At 0102 on March 3. 1989, Unit 3 experienced a I.arge Load Rejection event. The Steam Bypass Control System (SBCS) malfunctioned due to a failure of one permissive timer and the Reactor subsequently tripped on Low

'Steam Generator pressure. The events following the trip were complicated by a number of equipment failures and inappropriate actions. The most significant was the failure of the Atmospheric Dump Valves (ADVs) to open remotely. As a result, the Control Room Supervisor elected to manually open the ADVs from the Main Steam Support Structure (MSSS). The Auxiliary Operators opened two of the ADVs manually. Significant problems had to be overcome in the process of accomplishing the local manual operation. These problems included procedural inadequacies, a valve component failure, inadequate lighting, incomplete valve labeling, high noise levels, and poor human engineering in the arrangement of the manual operators for the ADVs. One ADV manual operator was damaged as a direct result of these problems.

The Plant Director formed the Incident Investigation Team (IIT) as required by the Incident Investigation Program. The Nuclear Regulatory Commission issued a confirmatory action letter and formed an Augmented Investigation Team (AIT) to investigate the event.

The major issue of-l;oncern in this event was the failure of the ADVs to operate from the Control'oom. The failure of all four. ADVs to open remotely was due to an indeterminant combination of causal fa'ctors including: 1) Packing interference, 2) Bonnet pressure, 3) an "extra" spring in 3 of the 4 valve operators, and 4) too small a demand signal imposed for too short a time to allow valve response.. Based upon as-found conditions, cold valve operating forces, and assuming the design residual bonnet pressure of 15 psig, it is highly unlikely that 3 of the 4 valves would have operated on nitrogen at 95 ps'nd unlikely that the other valve would have opened with the short duration, low demand signal imposed.

Operator training on the actual response of the ADVs during remote operation and local manual operations have been conducted. The valves will be modified to eliminate the potential for excessive bonnet pressure, and the actuator's two-spring configuration will be assured prior to restart of any unit. Additionally, comprehensive testing and preventative maintenance will be in place to assure ADV system operability. The Emergency and the Essential lighting have been upgraded in the MSSS.

The event also illuminated that Palo Verde failed to take effective corrective actions on previously identified performance deficiencies in. three 'other circumstances related to this event: the failure to correct lighting deficiencies; the failure to correct deficiencies in calculating dose assessment, and the failure of corrective actions for deficiencies noted in previous post-trip and Special Plant Event Evaluation Report (SPEER) reviews.

PAGE 4 IIR 2-3-89-00>

EVENT DATE: MARCH 3. 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD XUMZCTION IIL INTRODUCTION:

UNIT 3 INCIDENT INVESTIGATION 2-3-89-001 On March 3, 1989 Unit 3 experienced a large load rejection due to a fault on the Devers line in Southern California. Due to a failure of one permissive timer in the Steam Bypass Control System (SBCS). a low Steam Generator pressure condition resulted. A Reactor Trip with a Main Steam Isolation Signal (MSIS), Safety Injection Actuation Signal (SIAS), and Containment Isolation Actuation Signal (CIAS) followed. A loss of non-class electrical power and inability to remotely operate the Atmospheric Dump Valves complicated the event..A Notification of Unusual Event (NUE) was declared.

In accordance with APS procedures. APS classified the Unit 3 event as a Category 2 incident and established an Incident Investigation Team (IIT). A preliminary Events and C'ausal Factors Chart was developed to establish a time line of events and identify issues of concern. Additionally, areas of the plant determined to be relevant to the investigation by the IIT team were quarantined by APS QC personnel.

The Region V Office of the Nuclear Regulatory Commission issued a confirmatory action letter to document the actions that APS would perform as a result of the event. These actions were to include a thorough investigation of the Unusual Event to obtain a full.understanding of the event and define pre-startwp and post-start-up corrective actions. Additionally, the NRC sent an Augmented Inspection Team (AIT) to review the event as well as the investigation activities.

The Incident Investigation Team was comprised of key individuals from a number of departments to provide a s'ource 'of technical expertise to analyze and evaluate the event. The team collected and evaluated information to determine the sequence of events as well as identify causative factors which impacted the course and severity of the event. The Event and Evaluation sub-group was tasked with the following responsibilities:

1) Identification and control of factual information regarding the sequence of events.

2) Identification and control of issues uncovered during the fact gathering process. A detailed listing of potential issues involving equipment operation, operator actions, and apparent equipment malfunctions was maintained. This information became the foundation of Engineering evaluation teams which subsequently developed individual action plans for issue resolution. This information also led to the requirement to quarantine equipment.

A Quarantine list was maintained'to control maintenance activities on equipment which required further investigation, as specified by the NRC and the IIT.

' 3) Development of detailed Events and Causal Factors charts.

.PAGE 5 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EK~:CTION IIL INTRODUCTION (cont'd)

4) Coordination of the Management versite Risk Tree (MORT) and Human Performance Evaluation System (HPES) evaluations.

The IIT also developed specific - guidance for the development of troubleshooting action plans. This was developed to detail the methodology employed in conducting investigative maintenance activities on quarantined,~

equipment. Highest priority was given to ensuring that troubleshooting activities did not inadvertently result in the loss of information necessary to confirm the postulated causes of equipment malfunctions. Toward this end, action plans developed by the individual engineering teams were required to be approved by the Engineering Evaluations Manager, the Investigation Director and the NRC AIT leader. Action plans developed for this event also included detailed reviews of equipment maintenance history, as well- as surveillance and test histories. Repairs and other corrective actions deemed necessary also required the concurrence of the AIT leader.

t Individual action plan teams were designated for significant issues which represented major equipment or system'malfunctions, and items that the AIT leader requested be investigated. These teams, typically, consisted of Engineering Discipline Supervisors, System Engineers, Design Engineers, and additional support personnel as required. {the organization of these groups is provided iriMe attached). Information obtained from the activities of the engineering. evaluation teams was fed back to the Event Evaluation Team for incorporation of significant factual information, and of any additional issues. the'dentification During the Qrst 2 weeks of the investigation, daily meetings of the IIT were held, during which investigation participants provided status updates of assigned activities. Updated facts lists as well as the preliminary issues lists were provided to all participants to ensure rapid dissemination of investigative findings. Facts from operator statements, control room logs, personnel interviews, and engineering evaluation team findings were validated against facts obtained from computerized data acquisition systems where possible. Identified issues were cross referenced to supporting facts as well as the fact source to assist the efforts of the engineering evaluation teams. The detailed sequence of events was determined primarily from information obtained from the Plant Monitoring System (PMS). The PMS provides Digital Sequence of Events, time tagged to the nearest millisecond, as well as Alarm printouts for digital and analog alarm information when monitored parameters reach a predetermined alarm state. Additionally, events were logged by the Temporary Data Acquisition System (TDAS), a separate digital Sequence of Events Recorder (SER), and the Emergency Response Facility Data Acquisition and Display System (ERFDADS).

This . report provides a detailed event description, and a discussion of ll major issues associated with the event: The IIT utilized Energy-Barrier-Target (EBT) analysis and MORT techniques to identify the causal factors associated with this event.

PAGE 6 IIR 2-3-89-001 EVENT DATE MARCH 3 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RE JECTION III. INTRODUCTION (cont'd)

The concept of EBT analysis is one of recognizing that in anv process th<<e, is a potential hazard ( or energy) that can impaci a. desirable condition target. i.e. maintaining the reactor at l009o power. In order to prevent the hazard from affecting the target a series of barriers. both administrative and physical. are instituted or constructed to prevent the Hazard from reaching the t'arget. These include adequate design. procedures. training.

maintenance, management policies. and personnel performance standards.,

During the course of this investigation the barriers which should have acted to prevent the, undesirable consequences were examined and corrective actions have been generated to "shore-up" the barriers in such a way as to prevent recurrence of the undesirable aspects of this event.

MORT analysis is a sophisticated and detailed approach to the identiQcation of factors which can result in adverse consequences. The implementation of, MORT integrates the results of the EBT. HPES. and Events and Causal Factors methodologies into one overall set of conclusions regarding the causes of a specific incident. The unique aspect, of the iMORT method over traditional methods of identifying the causes of an event is that a greater focus is placed on management's role and responsibilities in preventing adverse consequences. This includes the identification of systemic areas of concern. In this investigation the MORT techniques were utilized,and the results integrated into the individual issues in terms of conclusion that are associated with management related systems.

h Two of the management related issues which were of importance in the course of this event include:

Risk A's essment S s em:

This analysis addresses the questions: "Does the risk assessment system provide management with the information it needs to assess residual risk and to take appropriate'action. if the residual risk is found unacceptable?

Does the system also provide: (1) comparative evaluation of hvo or more systems and (2) development and evaluation of methods supporting the hazard analysis process' Technical Information transfer This analysis addresses the questions: "XVas the technical information adequate'? Was there an accessible. open line, of information in both directiops between upper and lower levels'? XVas the responsibility for the study of problems shared? %fere the results of the study provided to +ose who would use the information?

These analyses are, in some respects, subjective indications o f programmatic problems. The decision to formulate corrective actions based on the outcome of the analyses is left as a decision for management to implement or to accept the risk of rtot pursuing the changes to the programs which indicate potential problems.

PAGE 7 .

IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION III. INTRODUCTION (cont'd)

These techniques are discussed in more detail in the Incident Investigations Methods Procedure. 79DP-OIP02.

PAGE 8 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION IV. EVENT DESCRIPTION The following narrative provides a broad overview of the event.

Those items which are analyzed in the Incident Investigation Report <<e assigned with an Issue iVumber and summarized in the "Issue Summary" following this Event Description. For a more detailed sequence of events, refer to the "Facts List" (which is sorted by time sequence), and the

'%vents and Causal Factors Charts" (Figures I - l 1).

Event description times are from the Plant monitoring System (PMS) ALarm Typer printout. Approximate times are from Operator Statements or Log entries.

A. INITIALCONDITIONS conditions prior to the event; Palo Verde Unit 3 was. operating I'nitial in Mode 1 at approximately 98 percent power. in-plant non-class IE electrical loads were being supplied by the Main-Turbine-Generator via the Unit Auxiliary Transformer. in-plant Class IE electrical loads were being supplied by off-site power via the Start-up Transformers. No abnormal evolutions were in progress.

The Operations on-shift compliment consisted of a Shift Supervisor.

an Assistant Shift Supervisor/Control Room Supervisor (CRS). a Primary Operator; a Secondary Operator, an Extra (Third) Reactor Operator, six Aumliary Operators (AOs), a Nuclear Operations Technician. and a Shift Technical Advisor. In addition to the normal shift complement, the following personnel were involved during the event: a Unit 2 Shift Supervisor (acting as Emergency Coordinator), a AVork Control Reactor Operator, a Work Control Evaluator (with an inactive Reactor Operator's License), and two other Shift Technical Advisors from Units 1 and 2.

R INITIATINGEVENT At approximately 01:02 on March 3, 1989 a fault occurred near the Devers, California switchyard which resulted in an electrical disturbance on the off-site power supply system.

At 01:02:18. Palo Verde Unit 3 generator output breakers PL-985 and PL-988 tripped open. The breakers tripped due to operation of the Sub synchronous Oscillation (SSO) relay (Issue ¹1) which is part of the sub-synchronous resonance protection system. This large load rejection resulted in the automatic actuation of the following systems as designed:

Steam Bypass Control System (SBCS), Reactor Power Cutback System (RPCB), and the Power Load Unbalance circuitry in the Main Turbine Control Syst: em. At this time the Main Turbine-Generator. continued to supply in-plant non-class IE loads.-

G GENERAL CONTROL SYSTEM RESPONSE The SBCS and RPCB Systems w'ork together following a large load rejection to allow the Nuclear Steam Supply System (NSSS) to remain at power. The SBCS functions to bypass steam around the Main Turbine

PAGE 9 IIR 2-3-89-001 EVENT DATE'ARCH 3 1989 UNIT 3 REACTOR TRIP FOLLO~G LARGE LOAD REJECTION IV. EVENT DESCRIPTION (cont'd)

(Mp) during situations requiring the removal of excess ifuciear Steam Supply System energy. The RPCB System rapidlv reduces core therm>>

power output bv dropping preseiected Control Element Assemblv (CE>>

Subgroups. The power Load Unbalance circuitry initiates fast closing of the turbine control valves and iintercept valves to preclude rapid acceleration, overspeed. and conse'quent tripping of the turbine Once the power-load imbalance is cleared, the steam admission valves would.

normally reopen.

The RPCB caused 12 CEAs (subgroups 4, 5, and 22) to be dropped into the core as designed. This lowered reactor power to approximately 45/o. In response to the large load reject. SBCS generated a Quick Open (Q.O.) signal and all eight SBCS valves opened. Following the initial Quick Open, response of the valves, erratic fluctuations of the Feed Water Control System (FWCS) and SBCS were observed by the Secondary Operator. FWCS 01 and 02 demand, controllers were swinging from 100/0 to 600/0 with the

,Steam Generator Econemizer Valves attempting to follow the demand. The

'FWCS was responding appropriately to the fluctuations imposed on the secondary system due to the erratic behavior of the SBCS. SBCS valves 1001, 1003, 1004. and 1006 cycled between full open and full closed due to a permissive timer failure (Issue "2J while valves 1002. 1005. 1007, and 1008 cycled from approximately 75/o to 1000/0. open due to the influence of the 1001, 1003, lOQ4 and 1006 valve behavior. Ten separate Quick Open cycles occurred causing Steam Generator pressures to decrease. The Secondary Operator did not take manual control of either the FWCS or SBCS while diagnosing the anomalies in accordance with management policy, as he was unable to come to an understanding of what was wrong with the system prior to the unit trip.

At Ol:03:48 a Reactor trip and Main Steam Isolation Signal (MSIS) were generated due to Steam Generator 02 Low Pressure. The Control Room Supervisor (CRS), directed the Control Room Operators to monitor Safety Functions 'nd he began event diagnosis as required by procedure (43EP-3ZZ01).

At 01:03:54, a Safety Injection Actuation Signal (SIAS) and Containment Isolation Actuation Signal (CIAS) were generated due to Low Pressurizer Pressure resulting from the Reactor Coolant System (RCS) cooldown.

D. INITIALOPERATOR RESPONSE The Primary and Secondary Operators monitored safety functions and the CRS diagnosed the event as an Excessive Steam Demand.

During the monitoring of safety functions, it was observed by the Operators that the Safety Equipment Status System (SESS) indicated that certain valves and dampers. did not fully reach their actuated position and that two 'switches indicated dual position without any attendant SESS alarms (Issue 03). The Control Roora Staff addressed the Safety Equipment

PAGE 10

, IR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION IV. EVENT DESCRIPTION (cont'd)

Actuation System (SEAS) alarms by verifi~ng proper component lineup~

with the exception of one damper (MFA-XI06) i~.hich is located within an Air Filtration Unit (AFU) and is inaccessible for position verification, The Shift Supervisor informed the Radiation Protection Department of releasing steam to the atmosphere.

PRIMARY OPERATOR At 01:04:00, in response to the SIAS,'he Primary Operator secured two out of the four running Reactor Coolant Pumps (RCP's IB and 2B) as required by procedures. RCP 2B handswitch had to be taken to "Stop" twice in order to secure the pump (Issue ¹10.5). The Primary Operator verified proper High Pressure Safety Injection (HPSI) header flows for the eMsting RCS pressure. he was however unable to determine if"SIAS injection had actuallv occurred. He noted that RCS pressure had "bottomed" at approximately 1800 psia as noted on RCS Pressure Recorder on Control Board B02. The Primary Operator also noted that Sai'ety Injection Tank (SIT) check valve leakage pressure indicated approximately 2250 psia on B02 recorder (Issue ¹10.7).

Due to fhe MSIS the Reactor Coolant System temperature and pressure started to increase. Later in the event the Primary Operator utilized auxiliary pressurizer spray to reduce primary pressure since normal pressurizer spray was 'unavailable when RCPs were not running. He logged each time the spray valve was cycled including the initial and final pressures for each cycle. The Operator however, did not log the Au~liary Spray line temperature prior to each cycle as required by procedure (Issue ¹11).

The Primary Operator isolated RCP seal bleed-off in response to the loss of Nuclear Cooling XVater (as a result of the loss of power). This action was not required with seal injection still supplied by the Charging System (Issue ¹ll). Later in the event when injection and bleed-off were secured by the Primary Operator a flow path was somehow reestablished via RCP bleed-off without operator action (Issue ¹8). This flow path allowed hot reactor coolant to circulate.up through'the RCP seals. The RCP 1B seal began leaking prior to the restoration of seal injection (Issue

¹8).

Due to the CIAS, RCS.letdown fiow was automatically secured with the charging system still in operation. The Primary Operator had to control Pressurizer level to prevent level from exceeding the maximum allowed by Technical Specifications. He did this by cycling charging pumps three times.

Later. the Primary Operator had to secure charging altogether. Each time charging was secured and reinitiated the Primary Operator did not secure and reinitiate weal injection pcr the prescribed procedural method (ie.

gradually reduce and initiate seal injection respectively to reduce pressure surges on the RCP seals) (Issue ¹11).

SECONDARY OPERATOR The Secondary Operator verifled the plant parameters relative to the

PAGE 11 IIR 2-3-89-OOI EVENT DATE'- MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION IV. EVENT DESCRIPTION'(cont'd)

Engineered Safety Features (ESF) setpoints. He recognized. informed and verified that the CRS and Primary Oper'ator were aware of the ESF actuations.

To maintain Steam Generator (SG) level the Operator utilized the "B" Essential Aumfiary'Feedwater Pump.

The Secondary Operator was unsuccessful in opening all four ADVs from the control room after giving each of them a 15% to 20% open demand 'signal (Issue ¹5). The Secondary Operator announced to the Control Room StaQ'hat the ADV's would not open.

The Unit 3 Duty STA was called after the Reactor Trip, and when he arrived began monitoring plant response per procedure (Standard Appendix BB). He reviewed the Diagnostic Qow chart used by the CRS and based on the parameters available on the trend recorders reached the same

'conclusion of Excessive Steam Demand. The STA was directed by the CRS to verify shutdown margin and this was accomplished by the Unit 1 STA.

E. LOSS OF NON-CLASS ELECTRICAL POWER The Reactor trip generated a Turbine trip as designed. After a turbine trip, the Main Generator normally trips when a reverse power condition is sensed and a fast transfer of house loads to the Start-up Transformers should occur. In this event, the Main Generator was already separated from the grid, so no reverse power condition was sensed. The Main Generator began to coast down after the turbine trip while still carrying house loads. When the Main Generator was at a frequency of approximately 30 Hz, it trjpped on Hi Volts/Hertz. A Fast Bus Transfer (FBT) was appropriately prevented by the syncro-check circuit since the house loads were not in synchronization with the offsite grid (Issue ¹4).

At OI:06:OI, a loss of power to the non-classl3.8KV busses (3E-NAN-SO I and 3E-NAN-S02) and the non-class 4160 V busses (3E-NBN-SOI and 3E-NBN-S02) occurred since the generator tripped and no FBT occurred. This de-energized the remaining two RCPs, resulting in a loss of forced circulation of the RCS. Class power was continuously maintained throughout the event.

This loss of non-cIass power also caused the plant Instrument Air compressors to stop. The nitrogen back-up valve opened to supply the instrument air header 139 seconds after the loss of non-class power (Issue ¹7):

Also because of the loss of non-class power there were no Radiation Monitoring Sy'tem (RMS) displays available in the Control Room (Issue ¹9),

and containment temperature and humidity recorders and sump level indicators were unavailable (Issue ¹8).

PAGE 12

,IIR 2-3-89-001 EVENT DATE: iKARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION I

IV. EVENT DESCRIPTION (cont'd)

F. NOTIFICATIONS At 01:39. a Notification of Unusual Event (NUE) was declared pursuant to EPIP-02. "Emergency. Classification." by the Shift Supervisor based on the SIAS and the loss of both non-class busses (3E-NAN-S01 and 3E-NAN-S02).

The Unit 2 Shift Supervisor was called to act as the Emergency Coordinator and, the Satellite Technical Support Center (STSC) and Operations Support Cent'er (OSC) were activated.

At approximately 01:49 the appropriate state and local, agencies were notified via the Notification and Alert Network. This met the requirement for notification of state and local agencies within 15 minutes.

The STA from Unit 2 was tasked, with completing the Notification Worksheet form. Following completion of the form and review by'.the SS.

the STA notified the NRC Operations Center at 02:03. This met the requirement to notify i~C within, one hour,.

G DEGRADED HEAT REMOVAL CAPABILITIES As a result of the MSIS, steam flow to the main condenser and to the atmosphere through the SBCS valves was terminated. thus affecting heat removal capabilities. In order to remove decay heat without relying on the Main Steam Safety Valves or Primary Safety Valves, remote operation of the Atmospheric Dump Valves (ADVs). which are located upstream of the Main Steam Isolation Valves (MSIVs), was attempted. The Secondary Operator was unsuccessful in opening any of the four ADVs from the control room after giving each of them a 15% to 20% open demand signal (Issue C5). The Secondary Operator announced to the Control Room Staff that the ADV's would not open without specifically informing them of the demand signal given. The Primary Operator verified that the RCS was greater than 28 degrees subcooled throughout the event. and did recognize that the Secondary Operator was having difficulty achieving heat removal. The Primary Operator informed the Secondary Operator that the RCS was heating up.

At approximately 01:07, the Turbine Driven Auxiliary Feedpump had been manually started to provide an additional steaming path due to the unsuccessful attempts at opening the ADVs from the Control Room. AO's were also dispatched to the Main Steam Support Structure (MSSS) for manual operation of the ADV's.

K REMOTE SHUTDONN PANEL OPERATION OF THE ADV's The Secondary Operator thought that due to the loss of power the control of ADV's may have automatically shifted to the Remote Shutdown Panel (RSP). This was discussed with the CRS. and Shift Supervisor (SS) and they agreed that it might be a possibility The Area 5 Au~lia~

Operator (AO) was .dispatched to the RSP with the intention of "resetting" the ADV's back to Remote Control, ie. ADV control from the Control Room.

PAGE 13 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RIMECTION IV. EVENT DESCRIPTION (cont'd)

When the AO arrived at the RSP he was met by two oiher AO's. Thev were uncertain as to just how the ADV's were to be "reset".(Issue ¹11). The AO had problems with radio communications to the Control Room.-After calling the Control Room via telephone. the Area 5 AO. who was responsible for the

~

area which included the RSP, saw that ADV 178 had neither Local nor Remote indication of control (Issue ¹5). He then checked the other three ADV's and found they were indicating that control was still from the Control Room (ie. "Remote" ).

The Third RO then directed the AO to attempt to open the ADV from the RSP. The AO informed the Control Room that he did not know how to operate the valve controller.. The Third RO instructed the AO on how to operate the valve (Issue ¹5).

At 01:11:18 the Area 1 AO took Local control of ADV 178 at the RSP and slowly dialed in a 30% open demand as instructed by the Control Room.

After waiting a few minutes with no ADV response (Issue "5), the Control Room directed the AO to return ADV 178 to 0% demand and return control of ADV 178 to the Control Room.

At 01:26 the AO was then instructed to place the three remaining ADV's to the Local and then back to the Remote position with the intention of resetting them. The AO was given no further direction'. so he left the area.

The Secondary Operator then again tried to open all four ADV's from the Control Room. No valve response was indicated.

I. LOCALCYMBALOPERATION OF THE ADV's IN MSSS Main Steam Safety Valve (PSV'579) lifted to relieve Steam Generator

¹1 pressure. The Secondary Operator observed that PSV 579 was lifting at an indicated pressure of 1220 psia, approximately 31 psi'lower than it's design setpoint of 1250 (+/- 1%) psig (Issue @10.1). Per the PMS Alarm Typer, during the course of the event the Main Steam Safety Valve lifted 4 times.

The AO's who were dispatched to the MSSS were directed to manually open an ADV on each Steam Generator. Upon entering the MSSS building, the auxiliary operators noticed that the building was almost totally dark (Issue ¹6). Normal Lighting was deenergized due to the loss of power and Essential Lighting was deenergized due to the SIAS (D90 and D91 deenergized). Later, when Essential Lighting was reenergized the single Essential Lighting bulb in the ¹2'team Generator side of the MSSS was burned out). The operators used fiashiights for lighting while manual ADV operations were performed. The room was extremely noisv when Main Steam Safety Valve PSV 579 cycled intermittently to control Steam Generator pressure.

( Five Au<liary Operators responded tq the MSSS. The Area 1 AO O. obtained the procedure for manually operating ADV 178, Operating Department Guideline (ODG) ¹30, which was posted at the valve. The

K 2-3-89-OOZ EVENT DATE MARCH 3 1gsg UMT 3 REACTOR TRIP FOLLO~t" LARGE LOAD ZP~ CTrON I

IV. EVENT DESCRIPTION (cont'd) 4 procedure instructed them to open the equalizina valve and to isolat~

Instrument Air to the ADV. The AO's encountered diificultv in performing steps of the procedure. Instrument Air System valves. requiring operation.

were not labeled nor were locations given (Issue .- 5). The air lines had to be traced to locate the valves. Valve lock wires had to be cut requiring additional tools to be utilized (Issue ¹5).

. At approximatelv Ol:36, ADV 178 (Steam Generator ¹1) was in Manual control. with the'Instrument Air to ADV 178 unisolated and the equalizing valve open. As th'e Control Room was directing the Area 1 AO to open the ADV, the Main Steam Safety Valve PSV 579 once again lifted. Each opening of the MSSV interrupted communications with the Control Room.

The AOs then started opening the ADV. After the AOs opened the ADV the Area 2 AO traced down the Instrument Air line and isolated air to the valve.

" At 01:37, ADV 178 (Steam Generator ¹1) was manually opened to appqoximately 7%.

'uring this time the Area 4 AO was attempting to open a Steam Generator ¹2 ADV (ADV 185) in the South Room of the hISSS using the posted instructions (ODG ¹30). As the AO performed the steps of the procedure he experienced the same problems in locating the Instrument Air valves as the AOs had in the North Room. The Area 4 AO tried to trace down the valves but was unable to find them. The Area 4 AO then went to the North Room ahd asked those AOs for assistance. The Area 2 AO from the North Room traced'own the lines and isolated air to ADV 185.

After "the Area 4 AO had ADV 185 in'manual he cracked open the valve and informed the Control Room that he had manual control of the valve. The Control Room directed him to open it to 2%. Getting the valve off it's seat was difficult,'however, the valve was slowly beginning to open. As he was opening the valve the handwheel was intermittently "slipping" on the shaft. MSSV 579 then lifted in the North Room. When the MSSV lifted it was so loud and unexpected that the AO contemplated leaving the room. He was not wearing ear protection although he had some available in his pocket.

At this time he was by himself and was unable to tell if the valve was opening. It was dark and he required both hands to operate the valve and he could not use his fiashlight to monitor valve position. The Control Room was directing him to open the ADV more. so he continued to open the valve until the handwheel setscrew worked itself loose and the handwheel came off in his hands. The Control Room called for the valve to be opened to 5% to 6%.

He decided to leave ADV 185 approximately 2% open and to open the other valve for Steam Generator ¹2 (ADV 179).

The AO turned around and started to set up ADV 179 from memory.

Since he had already read the procedure for ADV 185 he felt he knew-how to setup ADV 179. He did not remember that this valve was configured opposite to ADV 185. While setting up ADV 179 he obtained some assistance from the AO who had helped him before in isolating Instrument Atr. When ADV 179 was set.up for manual operation, the Area 5 AO entered the room with some tools, since he haQ heard over the radio that the AO was having trouble with the handwheel. When the Area 5 AO got to the room

PAGE 15 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION IV. EVENT DESCRIPTION (cont'd) they decided to try to Ax ADV 185 handwheel. The AOs did not have the right size tools. so they decided to attempt to open ADV 179.

Since the Area 4 AO was tired from his attempt to open ADV 185.

the Area 5 AO went to ADV 179. The Area 5 AO asked the Area 4 AO in which direction the handwheel should be turned. recalling that manual operation of the ADV's handwheels were different from ordinary valves. The Area 4 AO told the Area 5 AO to turn same direction that ADV 185 opened.

~V 179 in the clockwise direction the However, ADV 179 handwheel operates fn the counter clockwise direction (Issue 05). The AO noted some handwheel movement but was unable to continue the rotation manually. He assumed that the valve'was stuck on its seat. The AO's decided that the urgent situation warranted the use of a 24" pipe wrench as a cheater bar, aware that it was not a good operating practice to do so (Issue ~5). After less than one-half turn there was a loud pop and the platform vibrated. At first they thought the valve popped off its seat. The Area 5 AO then took off the pipe wrench and tried to open the valve by hand. As he turned the handhvheel the valve actuator turned sideways. The Area 4 AO got his flashlight on the valve and discovered 4 bolts sheared on the manual operator and the top plate of the actuating cylinder of the ADV cracked.

'The AOs then contacted the Control Room and informed them that the valve was damaged and that it should not be operated. They decided to coritinue their attempt to open ADV 185.

The AO's placed the handwheel for ADV 185 back on the drive shaft and 'tightened the holding nut down with channel-lock pliers. The Area 5 AO opened the valve one-half turn, when a part of the drive shaft that holds a clevis pin fastener chipped. The clevis flew off in the dark and the valve slammed shut. The Area 4 AO took the clevis from ADV 179 and put it on ADV 185 in a position where the chipped part would not ~

affect its operation.

At approximately 02:24 the Area 4 AO was able to open ADV 185 to approximately 5% and then it took bvo AO's to open the valve to 7%.

The Coritrol Room then instructed the AO's to place ADV 185 back to the neutral/auto position and open the nitrogen supply to ADV 185 in an attempt to control ADV'185 from the Control Room. The Control Room indication for the valve was now oscillating bebveen 3% and 7% with a 50%

demand signal on the valve, with the AO's monitoring valve position locally.

J..PLANT RECOVERY Continuing attempts were made to achieve heat removal from. the Control Room while the AOs were in the IVfSSS.

The Secondary Operator attempted to open Ql Steam Generator MSIV Bypass (SGE-UV-169) by placing the control board switch in override.

(The override position was required due to the MSIS signal). The valve would not open from the control panel and had to be manually opened (Issue 010 and 011).

PAGE 16 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION IV. EVENT DESCRIPTION (cont'd)

At 02:22. ~l Steam Generator MSIV Bypass (SGE-UV-169) was opened. SBCS valves 1007 and 1008 were then opened providing a steaming path to the atmosphere.

At 02:24. Natural Circulation was verified as required.

At 02:30. 42 Steam Generator MSIV Bypass (SGE-UV-183) was also

, opened to allow steaming of both Steam Generators. Following establishment of the steaming path via SBCS valves 1007 and 1008, ADVs 178 and 185 were manually closed.

At 02:32 power was restored to 13.8 KV bus 3E-NAN-S01. allowing energization of non-class 480V Load Centers.

At approximately 02:36, an Aumliary Operator reenergized Motor Control Center (MCC) panels M-19 and M-20. which were shed on the SIAS as designed. This restored containment sump level indication.

At 02:38 plant recovery continued with the resetting of MSIS.

At 02:41 the SIAS/CIAS was reset.

At 02:41, the Unit 3 generator output breakers PL-985 and PL-988 were reclosed. with the main generator motor-operated disconnect open, in order to restore switchyard integrity.

At 02:43, 13.8KV Bus 3E-NAN-S02 was energized allowing restoration of additional non-class loads.

At 02:52, as a result of restoriqg power to the non-class IE electrical busses and resetting the SIAS, the NUE was terminated.

When power was restored the STA evaluated the parameters which were unavailable during the performance of the event diagnostic.

At 03:00 an increase in containment radwaste sump level was observed and the Shift Technical Advisor (STA) calculated an approximate 6 gpm in-leakage to the sump. This was later found'o be due to RCP 1B leakage and an increase in the known leakage from charging line check valve CHV-435 (Issue C8).

When the Radiation Monitoring System (RMS) was restored the STA checked the Radiation Monitors and noted no irregularities with the exception of Containment Atmosphere (RU-1). The monitor was automatically isolated by the SIAS. The CRS was then informed and an Operator was directed to unisolate RU-l.

At approximately 03:41 RCP seal injection was restored.

At 04:24 the Start-up Auxiliary Feedpump (AFN P01) was started.

At 04:38 the Steam Generator Ol Train A Feedwater Downcomer Isolation valve could not be opened (Issue 010.2) and AFN-P01 was secured, Au~liary feed was maintained utilizing AFB-P01. ~

At 04:49 forced Circulation was reestablished when RCP 1A was restarted.'t 04:55, RCP 2A was restarted. The normal General Operating Procedure 43OP 3ZZ10 (Mode 3 to Mode 5) was then entered to continue the plant shutdown.

PAGE 17 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD M'~'TION IV. EVENT DESCRIPTION (cont'd)

K. DOSE ASSESSMENT CALCULATION During the event a Radiation Protection (RP) technician appropriately performed the Off-site Dose Assessment per EPIP-14 (Dose Assessment). The calculations were. however. performed incorrectly.

to the loss of power disabling RMS indication and the MESOREM" computer, which is the normal method used to perform the assessment. the RP technician had to perform a manual calculation (Issue "9). Personnel errors made in - utilizing the procedures led to an inaccurate and non-conservative Dose Assessment calculation. Calculated dose was 5.47 E -3 REM. whereas if calculated properly using the RMS default values prescribed by the procedure with RMS indication unavailable. should have been 28 REM to the thyroid of a child at the Site Boundary (Issue 09).

ADDITIONALISSUES Several additional problems were noted during the event and recovery as well as by evaluation of raw data following the event. These problems are categorized into either human performance or equipment performance issues.

Human Perfoxxnance Issues I Event descriptions of the human performance issues are summarized as follows and have been analyzed in Issue 411: I' A Reactor Operator erroneously reported Containment temperature to be 220'F.

A communication error was made to the Assistant Shift Supervisor (A/SS) when a Work Control Evaluator (Reactor Operator) erroneously reported Containment temperature to be 220'F vice the actual value of 110'F. The Evaluator, who was on'ssignment to Work Control and.had come to the Control Room in response to the event, was tasked by the Control Room staff to read the recorders on Control Board B07.to determine Containment parameters. The A/SS reported the 220'F temperature to the Unit 2 STA and directed him to apply the instrument correction factors for determining plant parameters which are given by procedure due to inaccuracies incurred with high containment temperatures. The Unit 2 STA, and the Evaluator went back to the recorder and determined that the temperature previously reported was incorrect. The recorder was misread by the Evaluator by a factor of two, when he used the wrong scaling for the indicated temperature. The A/SS was informed and no further actions were required.

~ An AO was erroneously sexit to NHN-M19 during restoration of non-class power.

A communication error was made by the A/SS. when he dispatched an A~iiary Operator to NHN-M19 to restore its power vice sending him to the location of its power source.

PAGE 18 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT-3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION IV. EVENT DESCRIPTION (cont'd)

Problems were encountered during the reclosure of generator output breakers.

The problem associated with the reclosure of the switchyard breakers II was encountered when Operations was attempting to close breakers PL-985 and PL-988 after the Main Generator Motor Operated Disconnect (MOD) was opened. The Operator assigned to close these breakers identified the actuated lock-out relav (186G-9) by looking at the back panels (behind the Control Room). A plaque was posted above the relay which instructed the Operator to reset. the trips on the Main Generator Exciter Cubicle Building). The Operator then went to the Generex Exciter Cubicle(140'ontrol and noticed the following trips. "V/Hz (timing)", "PSS comparator", and the "Trip" modules'. - The Operator discussed the trips with the Snit 3 SS and determined that the System Engineer should be consulted prior to resetting the trips and the lock-out. since they were not sure what caused the lock-out actuation. The Operator was unable to contact the System Engineer, so he contacted Protective Relay and Controls (PR Bc C) (Issue "11).

The PR & C representative stated that he knew about the relays but was not knowledgeable about the, Generex system. He was also not confident that the cause for the lock-out was due to the Generex trips. To assure himself that the lock-out was due to the Generex trips. PR Bc C recommended that Operations attempt to reset the lock-out relay (189G-9) and if the lock-out relay actuated again, they could be sure that the lock-out was due to the Generex trips. The Operator then tried to reset the 186G-9 relay and upon releasing the handswitch the relay immediately actuated as expected. After this action was performed PR 8 C concurred that Operations'ould reset the trips and then reset 186G-9 relay. At this time the Unit 2 SS (EC) was aware of the situation and concurred that the 186G-9 relay does actuate. on this type of an event and that it should be acceptable to reset the relay. The Unit 3 SS then gavel the Operator permission to reset the relay.

The Operator subsequently reset the Generex trips and the relay without any problem. The PL-985 and PL-988 breakers were then closed.

~ SurveQlance test of Safety Injection Valves were not performed within the required 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> following initiation of Safety Injection.

The Surveillance Requirements in Technical Specification section 4.5.2.g, requires that within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> following the completion of Safety Injection valve stroking operation, that valve position. is verified.

Nonperformance of this requirement for both trains of Safety Injection placed the Unit into Tech Spec LCO 3.0;3. The Control Room Staff was not aware that they were in this LCO throughout their shift and consequently failed to comply with the Technical Specifications. During the next shift the oncoming crew identified the requirement and entered the LCO 3.03 They subsequently performed the required surveillance tests for one train and ected LCO 3.0.3 (Issue 011}.

PAGE 19 IIR 2-3-89-00 1 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION IV. EVENT DESCRIPTION (cont'd)

~ Communication difiIculties with radios and telephones due to high noise conditions.

Communication difficulties with radio and telephones were encountered during the event and are addressed in Issue ¹5.

Equipment Performance Issues The additional equipment performance issues are summarized as follows and are-analyzed in the Issue number referenced:

RCP 2A high pressure cooler inlet indicted a high temperature after restarting the pump. (Issue ¹8)

Several problems were identified during the restoration of non-class loads.

~ Normal Chillers A and B tripped when Normal Chiller C started after 3E-NBN-S02 was energized (Issue

¹10.4).

~ A circulating water pump discharge valve (CWN-HV-008) did not fully close following the restoration of power (Issue

¹10.6).

When "B". charging pump was restarted; a. high seal water pressure alarm was received with seal pressure locally verified at 12 psig (Issue

¹10.8) I Oil Lift Pumps 9A and 9H indication lights on B06 were observed to be flickering (Issue ¹10.9) .

Main Steam Safety Valve PSV 577 was discovered to have a missing bolt/fastener (Issue ¹10.10)

There were 428 delayed alarms on PMS (Issue ¹10.11)

There is no time synchronization behveen various computer data acquisition systems (Issue ¹10.12)

s1 PAGE 20 IIR 2-3-89-00>

EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD M~CTION V. PERSONNEL PERFORM~ANCE EVALUATION A review'of personnel actions has been performed to evaluate human performance during the event. The performance oi the Control Room staff Auxiliary Operators. and Radiation Protection/Chemistry technicians has been evaluated. The Human Performance errors have been tabulated. HPES techniques were used to identify the causal factors for the more significant identified errors as a part of this report (Issue 1 1).

Xnitia1 Res onse to the Event; I

The overall initial response of the Operating crew to this event was go'od.; Following the. Load Rejection and, the RPCB the Secondary Operator decided not to take manual control of either HVCS or SBCS.

This decision is consistent with management's guidarrce. since the Ope'rator was not confident that he could have mitiaated the transient.

The( CRS appr'opriately obtained the Load Rej ection procedure.

43AO-3ZZ02, and during his evaluation of the transient. a Reactor Trip and "Main Steam Isolation Signal (MSIS) occurred.

Immediately following the Reactor Trip the Control- Room Supervisor (CRS) appropriately directed the Control Room Operators to monitor Safety Functions, and began event diagnosis as required by 43EP.-3ZZOl. The Duty STA was called and his arrival to the Control Room uas within the required 10 minute response time.

Mit ation of the Event The 'overall effectiveness of the operations staff during the mitigation and recovery of this complicated event was adversely impacted by a lack of adequate procedure usage, while performing equipment The Control Room staff quickly recognized the ESFAS 'anipulations.

actuations and attempted to take the appropriate actions.

The Primary Operator appropriately. verified the SIAS and quickly secured two Reactor Coolant Pumps.

The'econdary Operator appropriately informed the CRS and the Primary Operator of ESF functions actuated in a timely manner.

The Secondary Operator quickly recognized the necessity to est'ablish a steaming path via the ADVs.due to the MSIS. When the ADVs did not respond as expected the Secondary Operator . appropriately informed the Control Room staff. but his communication of the problem was less-than-adequate. He should have clearly stated his specific actions (e.g. dialed demand signal and duration) and his uncertainty regarding the causes for the non-responsiveness of the valves at B06 (e.g. if more demand should have been applied). This would have alerted his Supervision that assistance divas needed at the

PAGE 21 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION PERSONNEL PERFORMANCE EVALUATION(cont'd) 1VHti ation of th'e Event continued Control Boards (Issue y5). During the Secondary Operator's attempts to open the ADVs at B06 neither the CRS, SS nor the STA were specifically involved with his board manipulation of the ADVs (e.g. aware that only a 15% to 20% demand was applied to the ADVs and that insufQcient time was allowed for valve response).

During the Secondary Operator's assessment of the ADV problem, he incorrectly concluded that control of the valves had transferred to the Remote Shutdown Panel (RSP), basing his conclusions. on the existing electrical conditions. The Control Room staff reacted inappropriately to this assumption. since there was no Control Board annunciator indicating that this transfer had occurred. The Control Room staff performed less-than-adequately as a team, with respect to this evolution. because they did not fully explore the problems with Control Room operation of the ADVs prior'to dispatching AOs to the RSP and the iVISSS. They concluded too quickly that the ADVs could not be opened remotely from the Control Room.

I After the Consol Room staff agreed that it was worth investigating, the Third Reactor Operator (RO) .directed an Auxiliary Operator to t4e RSP to "reset" the ADVs. The use of the. term",reset" was inappropriate since the ADVs are not actually "reset".

The task to be performed was to assure that ADV control was transferred to the "Remote" position. Furthermore, the; decision 'to direct AOs to.the Remote Shutdown Panel by the Control Room Staff was inappropriate since the AOs are nei6er qualified nor trained to operate equipment at the RSP. It would have been more appropriate had they dispatched the AOs to determine the ADV status at the RSP.

The Secondary Operator recommended that the Steam Driven Auxiliary Feedwater Pump be started as an alternate method of heat removal. This was appropriate since it provided another steaming path for the Steam Generators.

The Primary Operator inappropriately secured RCP seal bleed-off while seal injection was still being supplied after the loss of. Nuclear Cooling Water. While this was procedurally incorrect, it had no impact on the seal degradation. (Issue Ill)

Due to the CIAS, RCS letdown was lost. In order to contro Pressurizer Level the Primary Operator appropriately cycled Charging pumps, however. hc did not properly start and stop RCP seal injection when cycling the pumps. (Issue tel l)

The Primary Operators use of Auxiliary Spray to control RCS prcssure was appropriate and timely. Tlic Primary Operator log<<ed each time the spray valve was cycled including the initial and final

PAGE 22 IIR-2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWZNQ LARGE LOAD REJECTION V. PERSONNEL PERFORM'ANCE EVALUATION(cont'd)

Miti ation of the Event continued pressurizer pressure for each cycle. The Operator. however. did not log the Auxiliary Spray line temperature prior to each cycle. as required by 43ST-3RC01. (Issue ¹11).

Local Control Actions The Control Room staff performed adequately in coordinating and communicating with the AOs during this evolution.

Local operation of the ADVs was hampered by. the fact that there was inadequate lighting in the MSSS and. due to the noise level. it was difficult to,understand radio communications from the Control Room.

The posted procedures for ADV manual operation were less-than-adequate and hampered AOs manual operations of the ADVs.

In addition. the equalizing valve on the pneumatic actuator was not labeled (Issue ¹5).

The Area 4 AO inappropriately directed the Area 5 AO to turn ADV 179 in the clockwise direction to open'he valve. The Area 5 AO inappropriately used a 24 inch pipe wrench as a cheater and subsequently damaged the valve (Issue ¹5).

Later in the event, the Secondary Operator attempted to open SGE-UV-169 (MSIV bypass) in order to utilize SBCS valves 1007 and 1008 as a steaming path. This was the appropriate method for achieving heat removal and was arrived at'by the Operators despite the lack of Emergency Procedure or Functional Recovery Procedure guidance. The attempt to open the valve from the Control Room was not performed in accordance with the Operating procedure. The valve could not be opened from the Control Room and was subsequently opened manually. (Issue ¹10.3)

Emer enc Plan Res onse The event was appropriately diagnosed as an Excessive Demand by the CRS. The Shift Supervisor correctly classified the event per EPIP-02 as a Notification of Unusual Event and promptly notified the Unit 2 SS to relieve him as the Emergency Coordinator (EC). The Unit 2 STA assisted the SS in preparing the "Notification Worksheet" prior to making the call to the NRC. This worksheet provides assurance that the information provided to the NRC Operation Center is accurate and complete. The appropriate notifications were made in the required time intervals.

The Radiation Protection (RP) technician proceeded to, perform the 0 Off-site Dose Assessment per EPIP-14. During the performance ot the Dose Assessmcnt calculations several errors werc'made by the RP

PAGE 23 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION V. PERSONNEL PERFORMMICE EVALUATION (contid) technician (see Unit ¹3 Incident Investigation. RP/Chemistry.

~

Potential Concerns and Radiation Protection & Chemistry

'"'-'Perfonnance,'"Dose"Assessment Function- Assessment of Performance attached). Due to the loss of power and subsequent loss of the MESOREMT" program. which is the normal method used to perform the assessment, the RP technician had to perform a manual calculation (Issue ¹9). His performance of the manual calculation was less-thaii-adequate.

errors, made in the performance of the Dose Assessment

'ersonnel procedure, led to an inaccurate and non-conservative Dose Assessment calculation (calculated. dose was 5.47 E -3 REM, whereas if calculated properly using the prescribed RMS default values. should have been 28 REM to the thyroid of a child at the site boundary). The RMS default values were necessary, due to the loss of the Radiation Monitoring System (RMS) Display and Control Unit (DCU). Also the information available through an alternate source." the Kaman Electronic Portable Indicating Controller (KEPIC), located in the Post Accident Monitoring Unit (PAMU) could not be obtained. This information was not obtained because the RP technician at the PAMU could not locate the KEPIC since the area was dark.

0 i Had, the manual dose assessment been performed correctly, using the RMS default values prescribed by the procedure, the RP technician would have been required to recommend the Protective Action Recommendations (PARs) corresponding to those for a "General Emergency" per EPIP-:15. The fact that the RP: technician could not obtain the Main Steam Line Radiation monitors (RU139 and RU140) information was less-than-adequate. It. is .'nacceptable ',when information was available and could not be obtained. This information was critical to the performance of dose assessment calculations in providing a realistic assessment of the offsite release.

See Issue ¹ll (Human Performance Evaluation) for the tabulated matrix identifying the Behavior Shaping Factor (Causal Categories) that led to the inappropriate actions. for each issue.

Recove Actions The Operators reset the "86" relay lock-out,in accordance with posted plaques outlining procedure 43AO-3ZZ13. "Degraded Electrical power" by ilrst contacting the PR&C representative and obtain'ing his concurrence. However their actions ~vere not in accordance with the intent of. the procedure (Appends G). This Appends. requires that PR&C or Engineering should be coHtactcd for inspection or

.concurrence prior to resetting the "86" rclavs. It continues that certain items including inspection of the affected component. bus.

control circuit, and breakers shoii be performed prior to rcenergizing the affected component. The troubieshooung performed to reset the

PAGE 24 IIR 2-3-89-00'VENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REACTION V. PERSONNEL PERFORMANCE EVALUATION(cont'd)

Recove "'ctions continued relay with the trips still in place was less-r.han-adequate.

The Control Room Staff was not aware that they had entered LCO 3.0.3 by failing to perform Surveillance Requirement 4.5.2.9.1.

Consequently,, they failed to complv with the Technical Specifications for having both trains'of ECCS inoperable. During the next.shift the oncoming crew identified the requirement and subsequently performed the required surveillance and ected LCO 3.0.3. (Issue ~1 1)

A- communication error was made to the CRS when.a Work Control

,Evaluator (Reactor'perator) incorrectly reported Containment temperature to be 220'F vice the actual. value of 110'F. These recorders were changed out as previously',scheduled following this event with digital recorders eliminating the requirement to determine which scale is being read.

The error in dispatching the A/0 to the MCC NHN-M19 location had little impact on plant restoration, since a different AO was dispatched to the correct location for restoration of NHN-M19. However, it must be stressed that in all communications it is the responsibility of the sender to ensure that the receiver understands the message and to clearly transmit correct directions.

PAGE 25 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION V. PERSONNEL PERFORM'ANCE EVALUATION ( cont'd )

List of Procedures used durin the event recove The following'.list'ofprocedures was compiled bv the On-sniit crew based on input of each crew member for the procedures used and/or consulted during the reactor trip event on March 3.

1989.

l. 43AO-3ZZ02 LOAD REJECT

2. 43AO-3ZZ12 DEGRADED ELECTRICAL

3. 43AO-3ZZ05 LOSS OF NC

4. 43AO-3ZZ03 LOSS OF TC

5. 43AO.-3ZZ04... LOSS OF HV

6. 43RQ'-3ZZ02 EXCESSIVE STEAiVI DEMAiVD

7. 43EP-3ZZ01 EMERGENCY PROCEDURE'.

43EP-3ZZO1 (APP J) SIAS VERIFICATION

9. 43EP 3ZZ01 (APP Iq, CIAS VERIFICATION

10. 43EP-3ZZ01 (APP M) MSIS VERIFICATION

11. 43EP-3ZZO1 (APP E) ~

NATURALCIRCULATIONVERIFICATION

12. 43EP-3ZZO1 (APP P) SIAS/CIAS RESET

13. 43EP-3ZZ01 (APP R) MSIS RESET

14. EPIP 02 -

EMERGEVCY CLASSIFICATION

15. EPIP 03 NOTIFICATIONOF UNUSUAL EVE; iT

16. 43AO-3ZZ37 LQSS OF LETDOWN

17. 43AO-SZZ13, iVATURALCIRCULATION LETDQIVVi

18. 43OP-3ZZ08 i REACTOR SHUTDOKVVi

19. 43ST-3RC01 AUX SPRAY VALVECYCLES

20. 43OP-3RC01 .'CP

'AVUAL NORMAL OPERATIONS.

21. ODG 030' OPERATION OF ADVs

22. PL000014 (Swyd Proc) PALO VERDE PROCEDURE FOR N3 GENERATOR QFF Llb',E

23. 43OP-3CHO1

~ 'VCS OPERATIONS

24. 43OP-3ibIT02: MAINTURBIiVE

25. 43OP,-3FT01 HVPT "A"

26. 43OP-3FT02 WVPT "B".

27. 40AC-QZZ06 LOCKED VALVEAiVD BREAKER

28. 43AO-3ZZ14 RCS EXCESSIVE LEAK RATE'9.

43AO-3ZZ29 RCP MOTOR EMERGENCY

30. 43AO-3ZZ52 D/G ESFAS RESET

31. 43OP-3ZZ10 MODE 3 TO MODE 5 GOP

32. 43 ST-3RC02 RCS LEAKRATE 33 43QP-3NA01 13.8 KV

34. 43OP-3NBO1 4.16 KV

35. 43OP-3AF02 NQN-SAFETY AUX FEEDXVATER PUMP OPERATIOiV

36. 43OP-3 VV01 OPERATING THE CIRC lVATER SYSTEM

37. 43OP-3DG01 D/G "A" NORMAL OPERATIONS

38. 43OP-3DG02 D/G "B" NORMAL OPERATIONS

39. 43OP-3GH01. GENERATOR HYDROGEN

40. 43QP-3HA01 AUX BLDG. VENTILATION

41. 43OP-3HC01 CONTAINMENTVENTILATION

42. 42OP-3HJ01 FUEL BLDG. VEiVVILATION

43. 43OP-3HJ01 CQiVTROL BLDG. VENTILATION 44, 43OP-3IA01 .INSTRUMENT AIR

45. 43OP-3IA02 SERVICE/BREATHING AIR

46. 43OP-3MTQ 1 MOISTURE SEPARATOR REHEATER

47. 43OP-3NC01 NUCLEAR COOLliVG WATER

48. 43OP-3OS01 LUBE OIL STORAGE, PURIFICATIOiV & TK4VSFER

49. 43OP-3RVO I PLANT COOLING iVATER

50. 43QP-3TC01 TURBINE COOLliVG WATER

51. 43OP-3WC01 NQRiVJ. CHILLED WATER, Throughout the event, the Control Room Sta>T consulted 2Q alarm response procedarcs.

PAGE 26 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWVfG LARGE LOAD M~C ION VI. PLANT PROTECTION SYSTEM RESPONSE EVALUATION The..followin

. allo.ving. Plant..Protection Systems actuated during the event:

~ Reactor Trip

~ Main Steam Isolation Actuation Signal (MSIS)

~ Safety Injection Actuation Signal (SIAS)

Containment Isolation Actuation Signal (CIAS)

Per the Iacident Investigation Program. each actuation was evaluated individually addressing the following questions:

1) Was it required?

2) Did it occur at the proper setpoint?

3) Did all equipment function as designed?

4) Was response time within the required interval' Reactor Tri 6 MSIS The Reactor Trip and MSIS both occurred. as required. at 01:03:48 due to Low Steam Generator ~2 Pressure, above the required minimum pressure setpoint which is > 919 psia.

The Reactor Trip functioned as- designed by dropping all Control Element Assemblies (CEAs) into the core within the required response time, Per the PMS Sequence of- Events times it took 0.057'seconds from th ti channels of Low Steam Generator 02 Pressure Trip" were received until the "CEDM Power Bus Undervoltage" alarms for all 4 channels were received and an additional 0.012 seconds (or a total of .069 seconds) until the "Reactor Trip Circuit Breaker - Tripped" alarm for all 4 channels were received. The 0.069 seconds for the signal delay time is well within the 1.15

. seconds response time required by Techaical Specification Table 3.3-2. Note that the sensor response time or the time it takes.to generate the trip signal at the pressure detector is not monitored and assumed to be within tolerance since the Surveillance Tests are current.

The MSIS equipment functioned as designed by isolating the Steam Generators from the Secondary Plant to eliminate the ezcessive steam demand. All valves positioned as required although the Safety Equipment Actuation System (SEAS) indicated othe~vise. The SEAS alarmed for the following valves indicating that they did not reach their actuated position:.

SGA-UV-225 - SGQ2 Hot Leg Blowdown Sample Isolation SGA-UV-22? - SG52 Downcomer Blowdown Sample Isolation SGA-UV-1134 - SG42 Main Steam Header Steam Trap Isolation SGA-UV-1135 - SG" 1 Main Steam Header Steam Trap Isolation Also SGA-HV-201 (SGQ2 Chemical Addition Line Isolation) alarmed and had dual indication on the Control Board and SGB-UV-228 (SG" I Hot Lcg Blowdown Sample Isolation) indicated dual position on thc Control I3oard an

'0 received no SEAS alarm.

All the above valves were verified by thc Control Room to be in their actuated position (closed) and the alarms were attributed to faultv SEAS indications. For a further discussion of the SEAS alarms rcccivcd see Issue "3 for analysis of the SEAS Engineering Action Plan.

PAGE 27 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLORINQ LARGE LOAD REJECTION VI; PLANT PROTECTION SYSTEM RESPONSE EVALUATION(cont'd)

The following facts were discovered by the SEAS Engineering Action Plan:

The. position of the equipment was verified by the Control Room Staff per procedure 43EP-3ZZ01. It was found that SGA-UV-1134 and SGA-UV-1135 indicated closed by the handswitch position indication and SGB-UV-228 and SGB-HV-201 indicated dual position by the handswitch indication.

SGB-HV-201 war greviously identified as having a position switch problem by Work Order 327482. SGA-UV-223, SGA-UV-225. and SGA-UV-227 were. previously identified as not providing the correct SEAS indication during an MSIS on 1/7/89.and Work Request 320973 had been written. The problem had not been corrected at the time of this event.

The ERFDADS (Emergency Response Facility Data Acquisition and Display System) also receives inputs from the affected valves and a printout of the information .shows that SGB-UV-228 indicated open before the Engineered Safety. Features Actuation System (ESFAS) actuation and closed after the MSIS. SGA-UV-1135A and SGA-UV-1135B both indicated open bi fore the actuation. After the actuation. 1135A indicated open and 1135B indicated closed. SGA-UV-1134 indicated open before and after the actuation.

informMon demonstrates that all ESFAS actuated devices received

'his

, the correct actuation signal from the ESFAS system. The ESFAS system output to the actuated devices is a.contact actuation. A failure of the ESFAS system will not cause an actuated device to fail in a mid-position because the ESFAS actuation signal cannot be removed until the Plant Protection System trip condition has cleared an'd the operator resets the actuation at the ESFAS cabinet.

N The response time limits per Technical Specifications Table 3.3-5 for the MSIS is'.6 seconds for'he Main Steam Isolation Valves (MSIVs) and 10.6 seconds for the Feedwater Isolation Valves (FWIVs). The only data available which records the change in valve position of the MSIVs and ~VIV's with respect to time is from ERFDADS. The time resolution however is in one minute increments and therefore actual MSIS response time of the Isolation Valves is unobtainable. ERFDADS MSIS Train "A" and "B" points changed state between 01:03:29 and 01:04:29. All MSIVs and WVIVs changed state in the same one minute period.

The SIAS and CIAS occurred, as required, at 01:03;54 due to Low Pressurizer Pressure above the minimum pressure setpoint which is o 1837 psia.

All pumps. valves and dampers actuated as required for the SIAS and CIAS with the following exceptions. The Safety Equipment Actuation Svstem (SEAS) alarmed for HPA-UV-001 (Containment Combustible Gas Control Train "A" Suction Isolation) indicating that it did not reach its actuated position.

HPA-UV-001 was verified by the Control Room to be in its actuated position (closed) by handswitch position indication and the alarm was attributed to faulty SEAS indication. ERFDADS determined that HPA-UV-001 indicated closed both before and after the ESFAS actuation.

PAGE 28 IIR 2-3-89-0OZ EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TKP FOLLOWING Lc'AGE LOAD ~<~CTION VL PLANT PROTECTION SYSTEM RESPONSE EQUATION (cont'd)

- --. Also'AA-.HS-114 (HAA-M02 - Auziliarv Building Ezhaust Isolation Damper) did not alarm on the SEAS but indicated dual position. HFA-M06 (Fuel Building and Aux Building Essential AFU Au~ Building Suction Isolation Damper) alarmed and had dual indication o> the Control Board.

HAA-HS-114 dampers HAA-M01. M02. and M06 were later verified to be closed by ERFDADS indication. ERFDADS indicated that HAA-M04 and M05 dampers did not reposition closed. This problem, is being investigated by the SESS Engineering Action Plan (Issue @3).

HFA-M06 Damper is located within the Air Filtration Unit and is inaccessible.to verify that it is in its actuated position. ERFDADS later determined that HFA-M06A and HFA-M06B indicated open prior to the actuation and closed after the actuation.

See Issue l3 for a further analysis of the SEAS alarms. The troubleshooting is designed to verify if the components did reposition as required and.. determine the reason for the various indications. The components each have a different limit switch sending indication of position to the control board handswitch. SEAS. and ERFDADS.

The response time limits per Technical Specifications Table 3.3-5 for the SIAS and CIAS is:,.

High. Pressure Safety Injection (HPSI)- 30 seconds Low Pressure Safety Injection (LPSI) -. 30 seconds Containment Isolation - mini-purge valves - 10.6 seconds

.:;Other CIAS valves- 31 seconds The only data available which records the discrete change in valve or pump operating coadition of the above with respect to time is from ERFDADS. The time resolution available for providing the post-transient closure times is, however, in one minute increments and, therefore, actual SIAS and CIAS response times of the valves and pumps is unobtainable.

ERFDADS SIAS and CIAS Train "A" and "B" points changed state between Ol:03:29 and 01:04:29. All SI pumps (Train "A" and "B" HPSI, LPSI and Containment Spray (CS)) started in the same one minute period.

The Containment 8" mini-purge valves (CP-UV-4A, 5A. 4B and 5B) were already in the closed position prior to the event. Based on the fact that no other SEAS alarms than those previously discussed above there is no reason to suspect that any other valves response times were not within specification.

It should be noted that the techniques used to determine the actual closing times. per Technical Specification requirements. are incapable of assessing times with sufficient, accuracy to ensure that the surveillance 'losing requirements are met.

PAGF 29 IIR 2-3-89-OOZ EVENT >ATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJIG'TION VIL CONTROL SYSTEM RESPONSE EVALUATION Control Systems. have been evaluated by the Engineering Evaluations Department-I&C using-the criteria of 79DP-OIP01. namely: ~ ~ ~

1) Identify any abnormal indications or degraded performance

.2). Identibg any-events occurring out of the normal or out of anticipated sequence

3) Identify any failed or degraded response of equipment to control

- signals

4) Identify any unanticipated alarms The following Control Systems have been evaluated:

~ Reactor Power Cutback

~ Pressurizer Pressure Control

~ Pressurizer Level Control

~ Steam Bypass Control Feedwater Control

~ ~ Reactor Regulation

~ Turbine Control System The Reactor Power Cutback. Pressurizer Pressure Control. Pressurizer Level Control, Reactor Regulation and Turbine Control systems functioned properly

. with no"abnormal ~pdications or alarms.

The Feedwater Control System prior to the reactor trip functioned properly during the multiple quick openings of the SBCS by varying Main Feedwater Pump speed and Economizer Valve position in an attempt to control Steam Generator level. After the reactor trip the Feedwater Control System responded properly by shutting both Economizer Valves and controlling both Downcomer Valves. Due to a loss of power, bath Main Feedwater Pumps tripped and t therefore their performance was. not significant.

The Steam Bypass Control System did not function properly as later determined to be due to a failure in the Group X timer card. See Issue N2 for a detaQed analysis of this failure.

PAGE 30 IIR 2-3-89-001 EVENT DATE: MARCH 3. I989 3 REtACTOR TRIP FOLLPWQfQ LARGE LOAD EJECTION NUCLEAR SAFETY ASSESSMENT Introduction This safety analysis was performed by the Nuclear Safety Analysis group (reference memo to M. R. Oren. Safety Analysis Assessment f Uni 3 Re r Tri nn 3 8, 0162-03071-PFC-SJT. March 17, 1989). This assessment addresses the impact of the Unit 3 load.'eject/low steam generator pressure reactor trip event from the perspective of compliance with the design bases events presented in Chapters 6 & 15 of the PVNGS FSAR. This event was first characterized as an "increase in heat removal by the secondary system" due to the Steam Bypass Control System (SBCS) valves cycling; Later the event progressed to a "decrease in heat removal by the secondary system" type event caused by a Main Steam Isolation Signal (MSIS) with inoperable atmospheric dump valves (ADV).

Increase in Heat Removal Event is Bounded hv I.icensed Analyses The design criteria of concern for an increase in heat removal by the secondary system event is violation of the Specified Acceptable Fuel. Design. Limits (SAFDL). These events cause a decrease in the temperature of the reactor coolant, an increase in reactor power due to the negative moderator temperature coefficient and . a decrease in reactor coolant system and steam generator pressures.

A review of the transient data for this brief period during the d d. I << f h SAPDL . S~ffi conservatisms a lied in the limitin desi n bases event as document d in FSAR Se tion 15.1 ade uatelv bounded the Unit 3 i

trans en t. The primary conservatism was the fact that the overcooling due to heat removal through the SBCS valves was less then the heat removal that is assUmed in the i%fain Steam Line Break design bases event and the Inadvertent Opening of a Steam Generator Relief or Safety Valve anticipated operational occurrence (AOO).

Decrease in Heat Removal Event is Bounded hv Licensed Anal ses For an event characterized by a decrease in heat removal by the secondary system the design criteria of concern is violation of the Reactor Coolant System and Steam Generator Design Pressure Limits. This type of event causes an increase in RCS temperature and pressure duc to decreased heat removal capability. The Unit 3 heat-up event was initiated after the reactor tripped on low steam generator pressure wi'th a concurrent MSIS. The transient data shows that main steam flow stopped for a brief period of time during which primary prcssure increased (as expected). Review of this pressure spike confirmed that Unit 3 did not experience a heat-up event of worse conscqucnces than those prcriously analyzed

~ ~ e e

PAGE 31 IIR 2-3-89-OOX EVENT DATE: MARCH 3, 1989

'UNIT 3 REACTOR TRIP'FOLLOWING LARGE LOAD REJECTION NUCLEAR SAFETY ASSESSMENT (cont'd) d d d S 152 f h FBAR. Th in Heat Removal Even t is Bnu n clecl bv Licensed An a 1vses maximum RCS pressure remained well below the design limit of 2750 psia.

Malfunction of SBCS Valve and ADVs doe. not Im act License Analyses l

Overall the response of the Unit was complicated due to the malfunction of the SBCS valves and the ADVs. The affect of these malfunctions did not cause the Unit to experience initial conditions or consequences any more adverse than those previously analyzed inthe PVNGS FSARi Steam B ass Control S stem not credited in Safety Analyses ll The SBCS and Reactor Power Cutback System (RPCB) are not safety grade systems and are therefore. not credited in Safety, Analyses.

Thus, the steam relief that the SBCS provided in combination with the reduced reactor power due to the proper functioning of the RPCB, only served to.,move the unit. further away (i.e. in a more conservative direction) from the initial conditions assumed in the Safety Analyses.

Rea r C plan System Hea Removal System was available Safety Analyses assume operation of the ADV for long term heat removal and cooldown (see FSAR Section 6.3.3.4) and are not credited in Chapter 15 events until 30 minutes after the initiating event. For the long* term cooldown analyzed event. only one ADV per steam ',

generator is assumed available for the duration of the event. Eventually, the Unit 3 Operators were able to open one ADV per steam generator.

Had the Operators not been able to open the ADVs, the Main Steam Safety Valves (MSSV) would have prevented over-pressurization of the steam generators and increased heat-up of the RCS, as actually occurred during the Unit 3 event. During an analyzed transient. the MSSV are assumed to operate and provide secondary heat removal.

Reactor decay heat is removed through the cycling of the MSSV. The MSSV will continue to cycle in this manner keeping the RCS in a hot standby condition. The fact that the MSSV first lift setpoint was expected, was an error in the conservative direction. In addition lower'han to the MSSV, the safety grade aumliary feedwater system aided in the heat removal process as required. If the operator had not initiated feed to the steam generators, then eventually the auxiliary feedwater actuation signal (AFAS-1 and AFAS-2) would have occurred and .

initiated feed to the steam generators. Both essential auxiliary feedwater trains were operable and fully available.

PAGE 32 IIR 2-3-89-001 EVENT DATE; MARCH 3 f989

~T 3 REACTOR TRIP FOLLOWING LARGE LOAD EUMZCTIPN VIIL NUCLEAR SAFETY ASSESSMENT (cont'd)

Off-Site Dose Crit ria 8 unde Due to the relief of secondary side steam to the atmosphere. there is "a potential for rel'easing radioactive material to the environment. For this event, the most probable source would have been a primary to secondary steam generator tube leak. All analyses that evaluate for o~sife dose "criteria" assume as initial conditions one percent fuel failure.=and a minimum Technical Specification primary to secondary leak of 1 gpm. Prior to the Unit 3 event. there was no identified leakage greater than 1 gpm and present chemistry data indicates that fuel failures are well within the allowed limits.. Therefore, the off-site dose consequences of,the Unit 3 event are bounded by analyzed events documented in the PVNGS FSAR.

PAGE 33

~ 2-3-89-001 EVENT DATE: MARCH 3, 1989 GT 3 REACTOR TRIP FOLLO~G LARGE LOAD REJECTION It IX. ISSUE SUMQVQMY A...number. of. issues were identified as a result of the initial review of the event. These issues were collected under nine major headings. The remainder of the equipment related issues which ~vere incidental to the event have been collected and addressed as Issue 010. The Human Performance: issues identified during the Plant Personnel Performance Evaluation are collected under Issue 011.

The summaxy of issues for this event are:

1. Subsynchronous Oscillation (SSO) Relay Operation 2: Steam Bypass Control System Operation

3. Safety Equipment Status System Operation,

4. Fast Bus Transfer Operation

5. Atmospheric Dump Valves Operation

6. MSSS Lighting

7. Instrument Air System Operation

8. Reactor Coolant System Leakage

9. Off-Site Dose Assessment Error

10. Miscellaneous Equipment Concerns

11. Human Performance Evaluation System Issues

PAGE 34 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RE JECTION RELAY'IR'2-3-89-0OI ISSUE eZ - SVSSVmeHRONOVS OSCILLATION SSO Subsynchronous.resonance relays are installed on all three Palo ><<<>ts.

Electrical grid disturbances can cause the grid to become tuned to a natural resonant frequency of the turbine-generator equipment. The Subsynchronous Oscillation (SSO) relays are provided mainly to protect against the effects of.torsional interactions beeveen PVNGS units that can arise-whefl" these electrical grid disturbances occur.

On March.3,. 1989. Southern California Edison's Devers Valley Line experienced a "C" phase ground fault. Approximately 14.5 cycles later, one or both SSO relays actuated, causing breakers PL-985 and PL-988 to open.

An issue was identified in the initial phase of the investigation which has been addressed in the course of the event:

I: Palo Verde Unit3 generator breakers PL-S85 and PL-S88 tripped open, causing a large load reject.

Analysis of-the digital fault recorders data from Unit 3, Salt River Project switchyard, and Southern California Edison Devers switchyard. determined that the fault condition lasted approximately 110ms. The phase to ground fault will induce aaubsynchronous resonance condition. The minimum

-0 reaction time of the SSO s'et modules, when e~eriencing ideal SSO conditions, is approximately 400ms. A subsynchronous resonance condition of sufficient duration did not e~st to cause an SSO relay actuation. Also. since there were no fiags, a subsynchronous resonance condition of sufficient magnitude to cause an instantaneous trip was not present.

INVESTIGATIONACTION PLAN Engineering Action Plan: Engineering Evaluations - Electrical prepared an action plan 'to investigate this issue. The action plan was issued and initial troubleshooting performed. A revision has been issued to perform further troubleshooting.

Interviews

@ personnel statements: Individuals contacted for this issue:

1) MA System Engineer

2) EED Electrical Supervisor Techniques Utilized: 1) Fact Collection (attachment 1)

2) Events and Causal Factors Analysis

' 3) Energy-Barrier-Target Analysis

I PAGE 35 IIR 2-3-89-001 FVENT DATE: MA'RCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REACTION t

1 I ISSUE 01 (cont'd)-

IP REFERENC~~:

1) IIR 2-3-89-001 Fact Data Base

2) "Subsynchronous Resonance Relay Troubleshooting Action Plan. Rev 1.

3) WO 346231, 346775, 346780. 34667 5, (Implements the Troubleshooting Action Plan).

4) WR 341316 (Implements the Troubleshooting Action Plan. Rev. 1).

5) Memo E. C. Sterling to E. E. Chartier. 167-03163-JTB/SLY (Addresses a Digital Fault Recorder for the SSO relays), December 12, 1988.

RESOLUTION/ANALYSIS The Incident Investigation Team'reviewed this issue an'd has made the foHowing conclusions:

1) Breakers PL-985 or PL-988 opened due to an SSO relay actuation.

2) The SSO relay was found to be calibrated and within tolerance. No cause for the relay actuation has been determined.

II

3) The lack of a Digital Fault Recorder (DFR) monitoring the SSO relays impaired the ability of EED to determine the cause of the SSO actuation.

CORRECTIVE ACTIONS CORRECTIVE ACTION 1. 1 a) Determine the cause for the relay actuation and implement necessary corrective actions as identified by the SSO Relay Engineering Action Plan. Further troubleshooting and investigation will concentrate in four areas:

~ Subtle Board Level Component Failure

~ Possible subsynchronous resonance condition other than Valley line fault eg. (Power System Stabilizer Malfunction)

~ Possible combination of SSR conditions adding to the Valley line fault;

~ Verify that the current and potential inputs to the SSO relay are within SpeciQcations.

b) Consider increasing to "2 of 2" or "2 of 3" logic to minimize spurious SSO induced generator breaker trips.

RESPONSIBLE ORGANIZATION a) Electrical - EED, Larry Henson/G. W. Sowers b) Electrical - NED. J. T. Barrow/E. C. Sterling MODE RESTRAINT/DUE DATE NONE/60 days following report approval

~ ~ W

"~ =P

PAGE 36 IIR 2-3-89-O01 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD ~~CTION ISSUE 0'1 (cont'd)

CORREC~~. ACTION 1.2 Install a Digital Fault Recorder to monitor the SSO relays on all units as recommended in referenced Nuclear Engineering memo.

RESPONSIBLE ORGANIZATION.

Electrical - EED, Lazy Henson/G. %V. Sowers MODE RESTRAINT/DUE DATE Unit 1: Prior to closing Generator Breakers

. Unit 2: Prior to closing Generator Breakers II

)

Unit 3: Prior to closing Generator Breakers

PAGE 37 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UMT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE P2- STEAM BYPASS CONTROL SYSTEM The Steam Bypass control System operates in conjunction with the Reactor Power Cutback Svstem (RPCB) and other control systems to accommodate large load rejections due to inputs to the system from steam generator s'team fiow transmitters.

During this event the SBCS Group Z (1,3.4.6) was noted to have quick opened ten times instead of a single quick open followed by modulation control of allvalves. This resulted in th'e reduction of S/G pressure and, ultimately the reactor trip. MSIS, SIAS, and CIAS.

An issue was identified in the initial phase of the investigation which has been addressed in the course of the event investigation:

2: Follomng the initial quick open response of the valves. erratic fluctuations ofSBCS valves mere observed by the secondary operators.",

INVESTIGATIONACTION PLAN:

Engineering Action Plan: The INC Engineering group within EED prepared an Engineering action plan to address'he issue raised in this event.

The action plan was issued and has been completed.

Interviews Bz personnel statements: The following Individuals were contacted in the course of this event:

1) Operating crew members

2) SF System Engineer

3) IRC Engineering Supe~<sor Techniques Utilized: 1) Facts Collection (see Attachment 1)

2) Events and Causal Factors Analysis

3) Energy-Barrier-Target (Engineering Action plan).

4) MORT REFERENCES

1) 2-3-89-001 Fact Data Base (Attachment 1)

2) Engineering Action Plan 6 KVork Orders 345657, 349278 (AVO which implement the action plan).. ~

3) EER 89-SF-014 - Addresses the performance of thc Root Cause of I ailure Analysis.

PAGE 38 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWZZG LARGE LOAD ~gZCTION ISSUE 02 (cont'd)

REFERENCES (continued)

4) 36iVIT-9SF03, Functional test of SBCS.

5) Letter M.R. Oren to G.W. Sowers,March 14, 1989, IIR 2- 00> I s<<s

-em1-0O4SZemO/MLC.

6) ICR 005233;. Change to'36MT-9SF03. to functionally t'est the timer card relay

7) Letter T. D. Shriver to W. F. Quinn, NRC AIT Preliminar'v Exit M tin<,

March 13, 1989, 0102-01161-TDS/RLMC, '

8) Letter W. M. Simko to Team Members. ADV SBCS SYSTEM REVIEW MEETING MINUTES, March 19, 1989, 0110-00446/WMS.

RESOLUTION/ANALYSIS The Incident Investigation Team has reviewed the event and has made the following conclusions: .

1) A relay timer card malfunction caused the multiple Quick Open operations of the group X Steam Bypass control Valves.

2) The SPEER process did not adequately identify and correct a previous.

SBCS timer failure in SPEER 88-03-003, "Main Transformer Phase Failure, July 31, 1988". 'B'ushing

3) The review of SPEER 88-03-003 by oversight groups, NSG, ISEG. and QA also failed to identify that SPEER 88-03-003 did not address corrective actions for the relay timer failure.

4) 36MT-9SF03 (the SBCS PIVf procedure) is inadequate in that'it does not functionally test or calibrate the relay timer card to ensure adequate reliability.

5) The Steam Bypass Control valves functioned as expected in response to the output of the SBCS.

PAGE 39 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR'IHIP FOLLOWING LARGE LOAD ~~CTION ISSUE 02 (cont'd)

CORRECTIVE ACTIONS Implement the-recommended corrective actions of the Engineering Action Plan:

CORREL'IVE ACTION 2.1. 1 The quyrantine on the SBCS components should be lifted and normal work. practices resumed.

RESPONSIBLE ORGANIZATION IITI., Mike Oren MODE RESTXULINT/DUE DATE COMPLETE CORRECTIVE ACTION 2.1.2 Replace the Group X Timer card and repeat the testing delineated in Work Order 0345657 to verify the replacement card corrects the problem as defined by Work Order 0349278.

RESPONSIBLE ORGANIZATION Unit 3 Work Control, C. D. Churchman MODE RESTRAINT/DUE DATE Unit 3 - Mode 2 Restraint CORRECTIVE ACTION 2;1.3 Perform a Root Cause Failure Analysis on the failed timer card.

This corrective action will be tracked by EER 89-SF-014. This action is to include a review of maintenance, modification history and evaluation of the circuit design and associated circuitry once the individual failure is identified. This action will be conducted in conjunction with the ADV/SBCS review group (reference 8).

RESPONSIBLE ORGANIZATION IRC - EED, Jeff Summy/ G. %V. Sowers MODE RESTRAINT/DUE DATE NONE/60 days following report approval CORRECTIVE ACTION 2.1.4 Add necessary instructions per Instruction Change Request 005233 to the SBCS Functional Test (36MT-9SI-03) to accomplish the functional testing of the timing of permissive Off Delay Timers.

PAGE 40 IIR 2>>3-89-001 EVENT DATE: MARCH 3, 1989 UNIT'3 REACTOR- TRIP FOLLOWING LARGE LOAD REHECTION IC II ISSUE 82 (cont'd)

I CORRECTIVE ACTIONS (continued)

RESPONSIBLE ORGANIZATION I&C - PS&C, Ken Cutler/ R. E. Younger MODE RESTRAINT/DUE DATE NONE/60 days following report approval CORRECTIVE ACTION 2.1.5 Review the requirements for modifying the testing and testing frequency of the SBCS functional test as recommended by the manufacturer's preventative maintenance program. Modify the SBCS functional 'test accordingly. I RESPONSIBLE ORGANIZATION I&C - PS&C, Ken Cutler/ R. E. Younger MODE RESTRAINT/DUE DATE NONE/60 days following report approval CORRECTIVE ACTION 2.2 Prepare a memo to all STAs that a thorough audit of draft incident investigations should be performed to verify all.concerns or issues which are addressed in the report are appropriately included in corrective actions.

RESPONSIBLE ORGANIZATION STA - PS&C, Lee Clyde/ R. E. Younger MODE RESTMZNT/DUE DATE NONE/ 60 days following report approval CORRECTIVE ACTION 2.3 The oversight groups should review their applicable programs/procedures governing their responsibilities with respect to IIR's reviews based on the fact that a concern was missed in a previous SPEER.

RESPONSIBLE ORGANIZATION NSG / KV. F.F. Quinn ISEG / %V. Quinn QA / B. Ballard MODE RESTRAINT/DUE DATE NONE/ 60 days following report approvai

PAGE 41 IIR 2-3-89-00 1 EVENT DATF,'ARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION ISSUE 03 -SAFETY E UIPMENT STATUS SYSTEM The Safety Equipment Status System (SESS) is divided into two subsystems the Safety System Actuated System (SEAS) and the Safety System Inoperable System (SBIS). - TheEIS monitors the status of safety systems and provides an annunciator and white alarm window for the system(s) and component(s) which are inoperable..The SEAS monitors the safety systems after an Engineered Safety. Featares Actuation (ESFAS). After a preset time delay which allows time for components to actuate, SEAS provides an annunciator and a blue alarm window for the system and for the individual component that faQs to fullyactuate.

During this"event the SEAS. received eight (8) alarms after ESFAS actuations occurred due to an MSIS on low steam generator pressure and SIAS/CIAS actuation on Low. Pressurizer Pressure.,

I An issue was ide4tified in. the initial phase of the investigation which has been addressed in the course of the event, investigation:

3: The Safety Equipment Actuation System indicated the follotving valves and darnpers did not reach their actuated position.

I ) HPA'UVOOI

2) SGAHV0201

3) SGAUV0223

4) SGAUV0225

5) SGAUVI 1 34

6) SGBUVI 135

7) SGAUV0227

8) HFAM06 Also, SGBUV228 and HAAHSI14 did not alarm on the SEAS although the handstvitches indicated dual position.

INVESTIGATIONACTION PLAN Engineering Action Plan: The I&C Engineering Group within EED prepared an Engineering Action Plan to address the issues raised in this event.

Interview

& personnel statements: The following individuals were contacted in the course of this event:

I) Operating crew members

2) SESS System Engineers

3) I&C Engineering Supervisor

PAGE 42 IIR 2 3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE S3 (cont'd)

QATESTIGATION ACTION PLAN (continued)

Techniques Utilized: '1) Facts Collection (see attachment 1)

2) Events and Causal Factors Analysis REFERENCES

'I

1) 2-3-89-004Fact Data Base (Attachment 1)

2) SESS Troubleshooting Action Plan

3) 43EP-3ZZOI (Emergency Operations)

II

4) ERFDADS DATA

5) EER 87-SG-121 (Replaces the SG valves springs)

6) DCP's 3FJ-SS-33, 3FJ-CH-237, 3FJ-SI-181, 3FJ-SG-160, 3FJ-SI-191, 3FJ-SC-132 (Replaces Reed Switches and Housing for Valcor Valves)

7) W.O.46727 (Implements the Troubleshooting Action Plan)

8) Unit Log RESOLUTION/ANALYSIS The Incident Investigation Team has reviewed the event and has made the following conclusions:

1) A hardware failure of either the component limit switches sending indication to SESS or of the actuated components themselves occurred which precluded proper operation of these devices.

2) All ESFAS relays actuated as required sending an actuation signal to the individual components.

3) The number of SEAS alarms (8) and dual indications due to faulty limit switches and damper, complicated the actions necessary to diagnose and mitigate the event.

4) The Qual indication and SEAS alarms on HFA-MO6 indicated that the

'A'rain ESF pump room essential exhaust system was not performing its design function. After the event. the SESS Engineering Action Plan determined from ERFDADS data that the dampers rcpositioned as required.

5) The HFA-M06 position cannot be locally verified unless the Fuel Building Essential Ventilation Air Filtration Unit (AFU) boundary is breached.

PAGE 43 IIR 2-3-89-001 EVENT DATE: MARCH 3. 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE S3 (cont'd)

RESOLUTION/ANALYSIS (continued)

6) All.Control Room and Control Building dampers actuated properly.

Auxiliary Building Isolation dampers M04A and M05A are associated 7)

-- "-ith HAA-HS-114. HAA-HS-1 14 indicated dual position on the control board indication and ERFDADS indicated that the dampers did not reposition closed.. The dampers are monitored by SESS but did not

, receive a SESS alarm.

H

8) The current SEIS/SEAS preventative maintenance program to verify that the alarms function properly is less-than-adequate;

9) The referenced DCP's for Valcor valves will'improve position indication reliability for those valves.

CORRECTIVE ACTIONS CORRECTIVE'CTION 3;1 Troubleshoot.and rework each individual indication in accordance with the-SESS&ction Plan. Determine, if the. problems noted are due to SESS indications or actual components not repositioning when actuated. Document results and initiate rework of any failed components. Consider ifgeneric problems eMst with ESFAS equipment position indication systems.

RESPONSIBLE ORGANIZATION I&C - EED, Jeff Summy/G. W. Sowers MODE RESTRAINT/DUE DATE Unit 3 - Mode 2 CORRECTIVE ACTION 3.2 Establish a SEIS/SEAS alarm testing program, incorporating checking SEIS/SEAS indicators as appropriate into PM's and ST's.

SEIS/SEAS indications should be veriQed for actual component state changes (eg. valve/damper actuations. pump starts/stops, etc.)

RESPONSIBLE ORGANIZATION ILC - PS&C, ICen Cutler/Ron Younger MODE RESTRAINT/DUE DATE NONE/120 days following report approval.

0

~ - r ~ ~ ~ I ~ ~

PAGE 4'4 IIR 2-3-89-00 1 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RFW'TION ISSUE 84 - FAST BUS TRANSFER The fast bus transfer automatically shifts the power supply to the non-c>>ss in-house busses, from the main generator to the offsite supply. when the main generator trips.

During this event, busses 3-E-NAN-SOl and 3-E-NAN-S02 lost power when the maingenerator..tripped....

An issue was identified in the initial phase of the investigation which has been addressed in theIcourse of the event investigation:

4: A Fast Bus- Transfer (FBTJ de not occur. Thfs resulted ln a loss ofNAN-SOl and NAN-S02.

IN'VESTIGATIVEACTION PLAN Engineering Action Plan: The Engineering Evaluations-Electrical group prepared a request for

'Removal of the Fast Bus Transfer

'cheme from the Quarantine plan. described. how the fast b'us List.'Huis transfer equipment functioned during this event. This plan was issued and the equipment removed from quarantine.

Interviews

& personnel statements: Individuals contacted for this concern:

1) NA System Engineer

2) EED Electrical Supervisor Techniques Utilized: 1) Facts collection (see attachment 1)

2) Events and Causal Factor Analysis REFERENCES

1) 2-3-89-001 Fact Data Base

1) Request for Removal of the Fast Bus Transfer Scheme from the

'0 Quarantine List.

2) EER 89-MA-003.

PAGE 45 IIR 2-3-89-001 EVENT DATE: MARCH 3. 1989 UNIT,3 REACTOR TRIP FOLLOWING LARGE LOAD REJEC ION ISSUE ¹4'(coat'd)

RESOLUTION/ANALYSIS The Incident Investigation Team has reviewed the event and has made the following conclusions:

1) For the conditions that existed during this event, the Fast Bus Transfer

.should.aot2xave. occurred and did not occur.

2) The current plant design will not maintain NAN-S01 and NAN-S02 energized whenever a generator breaker-only trip followed by a reactor trip sequence occurs. The units will be susceptible to Loss-of-non-class power events ifthe: 13.8KV busses are powered from the Unit Au~liarv Transformer.

lCORRECTIVE ACTIONS II 0'i CORRECTIVE ACTION 4. 1 Evaluate the present design of the 'breaker-only'ripping scheme per EER 88.-MA-003 for possible enhancements to the fast-bus transfer scheme.

RESPONSIBLE ORGANIZATION Electrical - NED, J. T. Barrows/E. C. Sterling MODE RESTXUIXVT/DUEDATE NONE/60 days following report approval CORRECTIVE ACTION 4.2 Based upon the evaluation performed in 4.1, Plant Management shall determine what enhancements shall be incorporated into plant design.

RESPONSIBLE ORGANIZATION Plant Director, W. C. Marsh MODE RESTRAINT/DUE DATE NONE/30 days following completion of 4.1

PAGE 46 IIR 2-3-89-Ooi EVENT DATE: MARCH 3, 1989 UNIT 3 REA'CTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹5 - ATMOSPHERIC DUMP VALVES The Atmospheric Dump Valves (ADV) are designed to provide a steaming path in the event that the condenser is unavailable for heat removal (e.g. due to an MSIS). The valves may be operated from the Control Room. Remote Shutdown Panel (RSP), or manually at the valve in the MSSS.

I During this event;:the operators'were not successful in opening the valves remotely. %hen the AOs were dispatched to manually open the valves.

several difQculties were encountered. The Events and Causal Factors Diagram (Qgures 5 and 6) provides a summary of the pathway utilized by the operators to achieve heat removal through the ADVs.

. Several issues were identiQed in the initial phase of the investigation which have been addressed in the course of the event investigation:

5.1: Attempts to'.open and controL the ADVsfrom the Control Room and from the Remote Shutdown Panel (RSP) mere unsuccessfuL 5.2: A Control ggoorq operator could not open the ADVsfrom the ControL Room.

5.3: The Auxiliary Operators could not operate an ADVfrom the Remote Shutdown PaneL, 5.4: The Auxiliary operator,had di~lty operating the ADVs locally.

5.5: During the attempt to manually open ADV179, the value was damaged.

INVESTIGATIONACTION PLAN Engineering Action Plan: The BOP Engineering group has developed an action plan to address the concerns relative to ADV operation.

The action plan was issued and is in progress.

Interviews

& personnel statements: The following individuals were contacted in the course of investigation:

l) The operating crew

2) BOP EED Supervisor

47 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UMT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION I

ISSUE ¹5 (cont'd)

ACTION PLAN (continued)

'NVESTIGATION,

.. Techniques Utilized: 1) Facts Collection (see attachment 1)

2) Events and Causal Factors Analysis

~r 3) EBT

~ aq 4)'ORT

5) HPES

~ I

~ e ~

REFERENCES

1) 2-3-89-001 Fact Data Base N

~ I

2) Engineering Action Plan and Work Orders 345580. 345698. 346717.

346729. 346728.

3) EERs/PCRs associated with ADVs:

'ER 83-SG-003 - ADVAccumulator capacity

'ER 85-SG-022 - ADV 178 Stroked Erratic

. '.. EER 85.-SG'-030= Question on ADV failing open EER 85-SG 049 - N2 Leakage on ADV supply valves

'ER

'ER 85-S9-061 - ADV 179,'84 Rapid Osc. open mode 85-SG-068 - Mtr from U3 to Ul for ADV 184

'ER 85-SG-100 - ADVs IA/GAsetpoint discrepancy.

EER 85-SG-140 - N2 Supply valves to ADV's

'ER

'ER 85-SG-160 - ADV 184 would not stroke 15%

86-SG-037 - ADVs Bz SBCS sticky

'ER

'ER 86-SG-144 - Leak on press tap ADV 185 86-SG-181 - Potential Loss of B train ADVS

'ER

'ER 86-SG-250 - ADV Control PWR shifted on Lop 87-SG-131 - N2 Leaks

'ER 87-SG-133 - N2 Leak thru PCV lifting PSV

'ER

'ER 88-SG-101 - ADV Operability question 88-SG-110 - ADV 179 didn't modulate close 0 - Provide MOVs for the ADVs .

PCR 85-13-SG-055 PCR 85-13-SG-056 - Inlet Isolation Valves for the ADVs PCR 85-13-SG-058 - Provide MOVs for the ADVs 0 - Increase LP setpoint where ADV XFRS to PCR 85-13-SG-078 N2 PCR 87-13-SG-001 - SMOD SM-SG-085 PCR 87-13-SG-002 - Revise N2 Alarm to 450psig PCR 87-13-SG-010 - Add dessicant filters to the

- Add dessicant filter to ADV air Equipment Index PCP 85-01-SG-069 line SMOD SM-SG-005 - Reset N2 Alarm to 450 psig SMOD SM-SG-017 - Install bonnet hot taps

\ '.

1 PAGE 48 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRXP FOLLOWING LARGE LOAD RECTION ISSUE ¹5 (cont'd)

REFERENCES (continued)

4) Post-Trips, Procedures and tests associated with ADVs:

WR ¹257247- WR ¹257247 was written to check ADV positioner in units.

WO ¹151096- Valve responds Qne ifthe thumbwheel is moved

'O ¹234236-4, WO

¹176223-at a moderate rate.

ADV 185 did not follow the demand signal No response to the demand signal from the HIC 0

for ADV 178

'3-SM-IA-003- Unit 3 doesn't have 3 micron Qlters installed on IA to ADVs.

PIRR 2-86-004- 100% demand signal needed to move Unit 2 ADVs.

FURR 2-86-011- Unit 2 ADV-179 response slow.

PTER 1-88-004 Unit 1 ADV-179 response slow and erratic.

73PA-3SG01- ADVs in Unit 3 last tested (6 35% power) 12/23/87.

PTRR 1-86-001- ADV 179 responded slowly.

4 PIRR 1-.85.-001 A

-. ADV SG 184 did not open with 15% demand 4

signal.

PTIRR 1-86-012- ADV SG 185 drifted between 5*and, 10%

(setpoint at 10%).

PTER 1-86-012- ADV SG 178 operated erratically.

PIER 1-86-003- ADV 178 operated erratically.

'J601A -.

ADVs Tech Manual.

73ST-3ZZ08 - The ADVs are tested on a 18 month period.

'3ST-3ZZ08-ADVs are tested in Modes 5 or 6.

43ST-3SG01- Main Steam 43OP-3SG01- ADV Operation

5) Memo Bill Simko to team members, ADV SBCS SYSTEM REVIEW MEETIN MINUTES, 110-00446-WMS, March 19, 1989.

6) Letter CCI to Ben Mendoza, Atmospheric Dump Valves potential significant deQciency under 10CFR-21, April 4, 1989.

7) Engineering Report: Atmos heric Dum Valve En ineerin'nalvsis-

~M..l 'I.ADVP g T .Ap I. 1989

PAGE 49 IIR 2-3-89-OO1 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REHECTION ISSUE ¹5 (cont'd)

RESOLUTION/ANALYSIS The Incident Investigation Team has reviewed the event and has made the following conclusions relative to the barriers available for increased reliability of ADV performance. Those barriers are:

~ Equipment Performance. barriers

~ Personnel Performance barriers

~ Procedure barriers

~ Training barriers

~ Design barriers

~ Administrative Control barriers A. ui ment Performance Barriers A. l) The BOP evaluation team has investigated the root cause of'failure of the ADVs in Unit 3. The investigation included testing of all ADVs in Units l, 2 and 3 to verify the operability of these valves. The investigation team effort also included review of past testing, design modifications. and maintenance activities associated with the ADVs.

0 The possible f'allure modes as identified in the initial Engineering a)

Plan included:

InsufQcient Demand Signal Given to Valves or InsufQcient Time Valves to Respond to Demand Signal Given.

'ction

'or b) Excessive Steam Leakage Around the Valve Plug Resulting in High Balance Chamber (Bonnet) Pressures.

c) Mechanical Binding of the Stem or Plug.

d) Failure of the Pneumatic Positioner.

e) Failure of the "A", "B", "R" or "S" ASCO Solenoid Valves.

f) Supply (Nitrogen or Instrument Air) Pressure Failure.

g) Excessive Leakage Through the Equalizing Valve on the Pneumatic Actuator Piston.

The following was discovered during the Engineering Action Plan (these issues were addressed in reference 7 above):

Valve Operation Anomalies Excessive Bonnet Pressure (seal ring leakage)-

Oscillations/Stability Packing interference Actuator configuration with 3 springs Corrective Action planned: Enhanced Test Program. Increase Nitrogen Supply Prcssure, revise Operating Procedures.

(Outage work - modify plug. piston ring and disk stack)

3 I

PAGE 50 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹5 (cont'd)

RESOLUTION/ANALYSIS (continued) 2.:- Preventative Maintenance Program Inadequacies Regulator Positioner Corrective Action Planned: Develop/Implement PM Tasks

~ I ~

3. Incomplete Implementation of DCP 3CM-SG-305 Positioner Gauges Actuator "0" Rings "Extra" Actuator Springs:

Corrective Action Planned: Remove gauges and install plugs.

3 3 . IIR~RR33 I dd incomplete implementation of DCP.

4. Surveillance Testing Program Inadequacies Nitrogen System Leakage Nitrogen/IA,Check Valves Corrective Action Planned: Develop/Implement Additional Testing.

CleanQness Concern 0

5.

Designed Class "C" - Requirement more stringent standard (for Nitrogen regulators/Tanks)

Corrective Action Planned: Flush/Verify System Cleanliness,

.Consider Design Changes..

Human Factor Considerations Packless Isolation Valves ADV Manual Operator Handwheel Operating Procedure Corrective Action Planned: Review Nitrogen System Design; Re-orient ADV Manual Operator Handwheel.

A.2) The PM/ST program did not test the ADVs under normal operating conditions. An ASME Section XI relief request is approved which only tests the ADVs during Cold Shutdown conditions. (Cold Shutdown Justification ¹18 -73PR-1XIO1)

A.3) The AO at the RSP was unable to easily ascertain ADV status due to burned out indicating lights on some controllers.

AR4) ADV185 manual operation was impeded due to slipping handwheel.

A.5) Lighting conditions were inadequate in the MSSS. (See Issue ¹6) 0 gg

PAGE 51 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UN' REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹5 (cont'd)

RESOLUTION/ANALYSIS (continued)

B. Personnel Performance Barriers B. 1) Communications between the RO in the Control Room and the AO at the RSP-were'imprecise with:respect to what was meant by "resetting" the ADVs.

B.2) The Control Room Staff should not have directed the AOs to operate controls on the RSP, as AOs are not trained or qualified to operate the RSP.

B.3) ADV 179 was broken because the AO did.not remember that it was manually operated in the reverse direction from ADV 185 because the procedure was not used and because a cheater bar was used.

'I B.4) AO communications with Control Room were impeded.

Communication betweeh the Control Room and the AOs in High Noise Areas is less-than-adequate.

AOs in the MSSS did not have all the tools and safety equipment readily available for the performance of their job (eg. cutters Br ear plugs).

The Secondary Operators face-to-face communications were less-than-adequate, because he did not specifically report his unsuccessful actions at the Control Boards and did not identify that he needed assistance-. (The evaluation concluded that the Operator did not apply more demand. due to the fact that he did not want to open the valves too much. He was concerned with causing an overcooling condition). His action in this case was inappropriate because he should have reported this concern to his Supervisor so that the problem could have received the proper attention.

B.7) The Control Room supervision missed the opportunity to correct the Secondary Operator because they did not fully explore the problems encountered by him at the Control Hoard. They should have given more attention to the ADVs specific manipulations prior to concluding that the ADVs would not operate from the Control Room.

H.8) The presence of steam in the MSSS, from thc lifting of the MSIV.

did not affect the ability of the operators to manually operate the ADVs.

PAGE 52 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP POLLPmrNQ LARGE LOAD REJECTION ISSUE 85 (cont'd)

RESOLUTION/ANALYSIS (continued)

C. ocedure Barriers C. 1) Procedural guidance for operating the ADVs from the Control Room was less-than-adequate. No procedural guidance was provided to the Operators with respect to previously identiQed ADV operating experience. The Operators were not given guidance on the amount of demand signal magnitude or response times necessary to open ADVs based on previous operating e~erience.

C.2) The Functional Recovery Procedure (4XRO-XZZ10) has insufficient guidance in the event that ADVs fail to open. The heat removal success path with MSIVs closed is to open ADVs, however. no provisions are included in the event that ADVs fail to open.

C.3) Previously identiQed ADV performance deficiencies were not effectively corrected by PVNGS organizations. Root cause(s) of the deficient ADV performance were not adequately identified to prevent recurrence.

C.4) The FI"iRR/SPEER closure process did not have, adequate criteria for assuring that corrective actions were adequate to meet the cencerns raised in the reviews.

C.5) Local instructions for ADV179 manual operation were not used:

additionally the instructions did not provide equalizing or isolation valve locations or numbers. The local operating instruction for ADV 185 was used in attempting to operate ADV 179.

D. Trainin Barriers D.1) PVNGS ADV operating experience from previous ADV operations was not adequately shared and/or transferred to appropriate departments.

D.2) The Simulator did not model ADV operation as e~yerienced in the Units. When.a demand was given on the simulator, the ADVs opened to that demand with little delays.

4 D.3) Oper'ations personnel were not trained on local manual operations There was no formal training or retraining for manual ot'DVs.

operation of thc ADVs.

D.4) The practice and hands-on experience the Control Room staff received for ADV operations was less-than-adequate.

D.5) Operational characteristics of the ADVs were not forma11y included in ADV (SG) classroom lectures.

PAGE 53 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWINGLARGE LOAD REJECTION ISSUE N5 (cont'd)

RESOLUTION/ANALYSIS E.

/-

E.1) The left hand/right hand valve orientation of the ADV's contributed to the damage of ADV 179 during manual operation.

E.2) ADV manual operation is complicated by the lack of visibility of the valve position indicator. The person operating the handwheel is not in a position to read the indicator.

E.3) ADV equalizing valves were hard for AOs to locate due to unfamiliarity. of valve operations and because the valves were not labeled.

E.4) The most probable causal factors for the failure of the Unit 3 valves to operate have been determined, although their relative contributions are indeterminate. There are several problems that identified in the EED anaysis report(reference 7) "...Specifically:

ADVs -178, -184. and -185 were found to have three actuator

'. springs vice the two required by design.

ADVs -178, -179, and -185 had their packing gland followers seized to the valve stem.

The, extra spring requires a minimum additional force of approximately 1500 pounds to operate. The seized packing follower requires an additional force of approximately 4000 pounds. The valve disassembly and inspection did not reveal anything else to prevent the valves from operating normally.

ADV-178 and ADV-185 had the esca spring and the seized packing follower. The force required to operate these valves very nearly exceeds the maximum force available from the actuator on nitrogen.

ADV-179 had a seized packing follower and ADV-184 had an extra These singular items probably added enough extra force to 'pring.

prevent the valves from opening when compared to the demand given and time allowed for the actuator force to build. All of the Unit 3 testing was performed in Mode 5. Cold Shutdown.

Therefore, it was not possible to determine if excessive piston ring leakage and the resultant high bonnet pressures were present in these valves.

The valve operator is capable of <<enerating <<pproximately 10.500

'0 pounds of force when supplied by 95 psig nitrogen. (The ADV reports (reference 7) contains the calculated and'measured forces required to operate each ADV.] The calculated force due to bonnet pressure assumes 15 psig steam pressure in the valve bonnet.

PAGE 54 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REACTION ISSUE ¹5 (cont'd)

Based on. average Mode 3 bonnet pressures measured on Unit 1 and 2 ADV's and frictional forces measured under Mode 5 conditions on Unit 3-ADV's, it is probable that the combination of the third actuator spring and packing follower friction'forADV-178 and ADV-185, prevented these valves from operation. The assumed prevented ADV-179 from operating.

ADV-184 probably failed to open based on the magnitude and duration of the open demand signa1. However, if a bonnet pressure of approximately 30 psig existed for this valve. it could not have been pneumatically opened regardless of the magnitude and duration of the demand signal."

In summary, based on the as-found conditions, cold valve operating forces. and assuming the design residual bonnet pressure of 15 psig, it is highly unlikely that 3 of the 4 valves would have operated on nitrogen at 95 psig and unlikely that the other valve would have opened with the short duration, low demand signal imposed.

F. Administrative Control S stem Barriers 0

F.1) Administrative policies were ineffective in identifying and correcting the problems encountered with the ADVs in previous PTRR and SPEER investigations.

F.2) Administrative Controls.did.not ensure that technical. information regarding the performance of the ADVs was disseminated in a timely manner.

CORRECTIVE ACTIONS:

The following corrective actions are sorted by priority using the Safety Precedence Sequence criteria. These criteria provide guidance in the development of corrective actions. Those highest on Safety Precedence Sequence are the most effective in preventing recurrence (but not necessarily the most cost effective). Namely:

~ Design for minimum hazard

~ Reduce hazard through safety devices Use warning devices to warn of hazards Develop or revise procedures to reduce or ameliorate hazards Personnel oriented ac'tions (e.g., Training, personnel actions)

Identify remaining residual hazards, and refer to management system for acceptance of remaining risks-

PAGE 55 IIR 2-3-89-00 1 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹5 (cont'd)

Desi for nunimum hazard CORRECTIVE ACTION 5.1.1 Engineering is to prov'ide recommendations to prevent recurrence of the inability to operate the ADVs remotely from the Control;,Room or RSP using the above safety precedence sequence with particular emphasis on:

RESOLUTION/ANALYSIS a) Design modiQcations necessary to be implemented b) Periodic, (Suiveillance) testing necessary to assure operability and the suggested frequency of testing c) Periodic preventative maintenance required d) Operating Procedures required to meet any design changes p

Based on the Engineering Action Plan findings this report will be supplemented to.include the necessary documentation for tracking the corrective actions required. (A. 1, C. 1)

RESPONSIBLE ORGANIZATION EED-BOP, K. i%I. Johnson/G. W; Sowers MODE RESTRAINT/DUE DATE:

Unit I, 2, 3 - Mode 2 CORRECTIVE ACTION 5; 1.2 Review the adequacy of the relief request for the ADVs and revise the ASME Section XI program to define appropriate surveillance testing for the ADVs. (A.2)

RESPONSIBLE ORGANIZATION EED- Tech Eng., R. Kropp/G. W. Sowers MODE RESTRAINT/DUE DATE:

Complete CORRECTIVE ACTION 5.1.3 Review the ASME Section XI relief requests for testing valves and pumps to assure that the requests are appropriate for safcty-related equipment.

Assure that the program does not exclude testing equipment which may be required to respond in transient events.

RESPONSIBLE ORGANIZATION EED, G. W. Sowers

' MODE RESTRAINT/DUE DATE NONE/90 days following report approval

PAGE 56 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR T92P FOLLOWING LARGE LOAD RFJZCTION ISSUE ¹5 (cont'd)

CORRECTIVE ACTION 5.2

'll Add labels to ADV equalizing valves and instrument air isolation valves in units. (E.3)

RESPDNSIBLE ORGANIZATION Unit 1 Work Control J. D. Dennis (WR 319970);

Unit 2 Work Control P. J. Wiley (WR 3 1-9971)',

Unit 3 Work Control C. D. Churchman (WR 319972).

MODE RESTRAINT/DUE DATE: Mode 2 CORRECTIVE ACTION 5.3 a) Evaluate and recommend a better (less failure prone) method of attaching ADV handwheels. EER 89-SG-144 submitted b) Implement above corrective action. (A.4)

RESPONSIBLE ORGANIZATION a) EED-BOP, K M. Johnson/G. W. Sowers b) Unit 1 Work Control, J. D. Dennis Unit 2 Work. Control, P. J. Wiley Unit 3 Work Control, C. D. Churchman MODE RESTRAINT/DUE DATE:

a) NONE/30 days following report approval b) NONE/60 days following completion of 5.3.a.

CORRECTIVE ACTION 5.4 a) Preferably eliminate or, as a minimum, identify all Safety Related

"- - and.haportant to'Safety "reverse operated valves" (e.g. ADVs) in the units.

b) Install label plates on above valves until replaced with normally operated valves. (E.1)

RESPONSIBLE ORGANIZATION a) Unit 1 Plant Manager, W. E. Ide Unit 2 Plant Manager, D. B. Heinicke Unit 3 Plant Manager, R. J. Adney b) Unit 1 Work Control, J. D. Dennis Unit 2 Work Control, P. J. Wiley Unit 3 Work Control, C. D. Churchman MODE RESTRAINT/DUE DATE:

a) NONE/60 days following report approval b) NONE/60 days following completion of 5.4.a.

CORRECTIVE ACTION 5.5 a) Evaluate modification and provide recommendations for local ADV valve position indication so that valve position can be 0 observed from the operating platform e.g., mirrors, widen, stem rider, etc or provide information relating the number of turns open to % open for ADV operating procedures. EER 89-SG-145 submitted.

PAGE 57 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 05 (cont'd)

.b) Install above modification ifrecommended by EED and as plant conditions permit.

c) Implement procedural changes as defined by EED.

RESPONSIBLE ORGANIZATION a) EED-BOP, K i%I. Johnson/G. W. Sowers b) Unit 1 Work Control'/J'. D. Dennis Unit 2 Work Control; P. J. Wiley Unit 3 Work Control, C. D. Churchman c) PS', R. E. Younger MODE RESTRAINT/DUE DATE:

a) NONE/60 days following report approval.

b) NONE/60 days following completion of 5.5.a.

~

c) NONE/60 days follow'ing completion of 5.5.a.

CORRECTIVE ACTION 5.6 Revise the Simulator to model actual ADV operation in the units. (D.2)

I RESPONSIBLE ORGANIZATION Training Department - W. F. Fernow MODE RESTRAINT/DUE DATE:

Complete CORRECTIVE ACTION 5.7 Inadequate lighting issues are addressed in Issue 06.of the IIR.(A.5)

RESPONSIBLE ORGANIZATION N/A MODE RESTRAINT/DUE DATE: N/A Reduce hazard thro h safe devices interlocks None required.

Use warnin devices to warn of hazards See Corrective Action 5.4 above.

Develo rocedures to reduce and ameliorate hazards CORRECTIVE ACTION 5.8 Revise procedures for analyzing and disscminatin<< incident investigation:

operations and maintenance experience to the appropriate dcpartnicnts.

per the charter for the PSKC Department. (D. I, F.2)

J RESPONSIBLE ORGANIZATION PSBcC, R. E. Younger

PAGE- 58 IIR 2-3-89-001 EVENT DATE: MARCH 3, 198'9 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EVICTION ISSUE ¹5 (cont'd)

MODE RESTRAINT/DUE DATE:

NONE/60 days following report approval CORRECTIVE ACTION 5.9 "Corrective Action 5.9 Intentionally Blank" CORRECTABLE ACTION 5.10.

i Revise 4XRO-XZZ10, Functional Recovery Procedure, as necessary, to include provisions for manual ADV operation or alternate steaming paths (eg. using steam driven auxQiary feed pump or bypassing MSIVs and using atmospheric SBCS valves) to meet heat removal success path. (C.2).

RESPONSIBLE ORGANIZATION PS&C-Operations, R. Buzard/ R.E. Younger

. MODE.RESTRAINT/DUE DATE:

Unit 1, 2, 3 - Mode 2 CORRECTIVE ACTION 5.11 A PM task for'testing the indicating lights at the RSP shall be developed.

This testing shou/d then be performed on a scheduled basis and lights replaced as necessary. (A.3)

CORRECTIVE ACTION 5. 1 1. 1 Develop a method for testing the RSP indicating lights per EER 89-XE-021.

CORRECTIVE ACTION 5.11.2 Develop a PM for periodic testing of the RSP indicating lights following disposition of the EER dispositioned.

RESPONSIBLE ORGANIZATION 5.11.1 EED - ILC, J. Summy/G. W. Sowers 5.11.2 PS&C - I&C, K Cutler, R. E. Younger MODE RESTRAINT DUE DATE 5.11.1 NONE/60 days following report approval 5.11.2 NONE/60 days following report approval CORRECTIVE ACTION 5.12 Review and make corrections to ODG-30 where deficiencies ezist with local instructions for equipmcnt operation. Verify that instructions provided are adequate for equipment operation. (C.5)

RESPONSIBLE ORGANIZATION PS&C-Operations. R. Buzard/ R.E. Younger MODE RESTRAINT/DUE DATE:

NONE/120 days following report approval

PAGE 59 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 05 (cont'd)

CORRECTIVE ACTION 5. 13 a-c) The Plant Managers shall identify Safety Related 8z Important To Safety [ITS) Tasks, from &e Job Task Analysis. which are done infrequently, for which the AO's need immediate training.

Identified'raining will be conducted by Unit staffs. (D.3)

//.r RESPONSIBLE ORGANIZATION a) Unit 1 Plant Manager, W. E. Ide b) Unit 2 Plant Manager, D. Heinicke

~

c) Unit 3 Plant Manager, R. J. Adney MODE RESTRAINT/DUE DATE:

a) Mode 2 b) Mode 2 c) Mode 2 CORRECTIVE ACTION 5. 14 The Lead Training Analyst and a representative of plant management shall review the lists of SRO/RO/AO Identified Safety Related or ITS tasks for training adequacy. (D.3)

RESPONSIBLE ORGANIZATION Training Department - W. F.'ernow 15ODX RESTRAINT/DUE DATE:

Mode 2 CORRECTIVE ACTION 5.15 Prepare Job Performance Measures (JPMs) for SRO/RO/AO tasks identified above:

a) Those which are required to be completed prior to Mode 2 entry. (D.3) b) Those which are NOT required to be performed prior to mode 2 entry.

RESPONSIBLE ORGANIZATION a) Training Department - W. F. Fernow b) Training Department - W, F. Fernow MODE RESTRAINT/DUE DATE')

Mode 2 b) During the first rcqual cycle following rcport approval.

PAGE'0 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 86 (cont'd)

I CORRECTIVE. ACTION 6. 16 Provide ongoing training to Operators who may be tasked with'ocal operation of equipment. (D.3)

RESPONSIBLE ORGANIZATION Director, W. C. Marsh )'lant MODE RESTXUZNT/DUE days following report approval DATE'ONE/30 CORRECTIVE ACTION 6. 17 a) Identify those tasks which the AOs are not allowed to perform. i.e.,

'that are required to be performed by a, licensed Operator only.

b) Operations Supervisory personnel shall be counseled that they should not direct the AOs to perform tasks for which they are neither trained nor speciQcally authorized to do. (B.2) c) AllAOs shall be informed that when problems are encountered in job performance. that these problems must be reviewed with their Supervisor prior to taking non-standard corrective actions. (eg. use of cheater bars for opening valves). The above shall also be reiterated to all Personnel in the Operations Department. (B.3)

RESPONSIBLE ORGANIZATION a) Plant Director, W. C. Marsh b) Unit 1 Operations Manager, J. J. Scott Unit.2.Operations Manager, F. C. Buckingham Unit 3 Operations Manager, R. E. Gouge c) Unit 1 Operations Manager, J. J. Scott Unit 2 Operations Manager, F. C. Buckingham Unit 3 Operations Manager, R. E. Gouge MODE RESTRAINT/DUE DATE:

a) Mode 2 U-l, U-2, U-3 b) Mode 2 U-l, U-2, U-3 c) Mode 2 U-l, U-2, U-3 CORRECTIVE ACTION 6.18 Operations personnel shall review the event with emphasis on the necessity of following speciAc local valve operating instructions even during transient conditions and the prohibition on using cheater bars.

(B.3)

RESPONSIBLE ORGANIZATION Plant Director, W. C. Marsh MODE RESTRAINT/DUE DATE:

Mode 2 U-l, U-2, U-3

PAGE 61 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EG~lCTION ISSUE ¹5 (cont'd)

CORRECTIVE ACTIONS:.,

CORRECTIVE ACTION 5. 19 Simulator training shall include review of ADV. operating experience li associated with erratic ADV performance and include scenarios associated with inability to open ADVs remotely. Training will be performed on the Recovery, Emergency, and Functional Recovery Procedure changes defined by 5.1.1. (C.l, C.3, D.l. D.2)

RESPONSIBLE ORGANIZATION Trgping Department - W. F. Fernow MODE RESTRAINT/DUE DATE:

'ONE/12/31/89 CORRECTIVE ACTION 5.20 a) Reemphasize and enforce the communication standard as described in 40AC-9OP02. (B. I) b) Require special emphasis..on formal communications on the simulator.

RESPONSIBLE ORGANIZATION 0 a) Plant Director, W. C. Marsh b) Training, W. F. Fernow MODE RESTRAINT/DUE DATE:

a) Mode 2 U-l, U-2, U-3 b) Mode 2 U-I,.U-2, U-3 CORRECTIVE ACTION 5.21 Develop a standard for required tools, safety apparel, etc. to be carried at all times by an AO. (B.5)

RESPONSIBLE ORGANIZATION Plant Director, W. C. Marsh MODE RESTRAINT/DUE DATE:

Mode 2 U-l, U-2, U-3 CORRECTIVE ACTION 5.22 a) Develop spcciiic instructions for ADV operations from the Control Room. The instructions as a minimum, should include dcsircd open demand signals, duration. and preferred closing method.

PAGE 62 IIR 2-3-89-OOZ EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTIO ISSUE 05 (cont'd)

CORRECTIVE ACTIONS:

P b) This information shall be incorporated as appropriate into the appropriate Operating, Surveillance, Recovery, Emergency, and Functional Recovery Procedures".'C.1, D.1)

RESPONSIBLE ORGANIZATION a) EED-BOP. K M. Johnson/G. W. Sowers b) 7'S&C,*R. E. Younger MODE RESTRPZNT/DUE DATE:

a) Complete

, b) Unit 1, 2, 3 - Mode 2

'ORRECTIVE ACTION 5.23 Standards of performance in procedural usage will be enforced and emphasized/measured during Simulator training and on shift. (C.5)

RESPONSIBLE ORGANIZATION Plant Director, W. C. Marsh MODE RESTRAINT/DUE DATE:

Unit 1, 2, 3 - Mode 2 Ident remainin residual hazards and refer to man ement s stem for acce tance or re'ection CORRECTIVE ACTION 5.24 Engineering shall review the following alternatives. utilizing a risk-based evaluation method (which considers, for example, the hazard associated with the potential for ADVs to "stick open" during stroking), to determine the optimum method for performance of ADV PMs and submit their recommendation to plant management for implementation (A.1):

~ Install block valves to facilitate periodic stroking,

~ Periodically stroke valves at power without block valves.

~ Modify ADV design and continue with present cyclin<<pcriodicitv.

RESPONSIBLE ORGANIZATION EED-I3OP, IC. M. Johnson/G. W. Sowers MODE RESTRAINT/DUE DATE:

NONE/Through completion of the ADV Action Plan

PAGE 63 IIR 2-3-89-801 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 85 (cont'd)

CORRECTIVE ACTION 5.25 Evaluate the use of radio earphones, or alternate methods for enhancing communications between the AOs and the Control Room. (B.l. B.4)

RESPONSIBLE ORGANIZATION

. - - - Plant Director; W; -C; Marsh MODE RES7XUIZNT/DUE DATE:

Mode 2 U-l, U-2, U-3 CORRECTIVE ACTION 5.26 Evaluate the Administrative Controls related deficiencies which resulted in the failure to adequately address previously identified ADU performance deficiencies. This shall include. but not be limited to the following:

- The flow of information concerning deficiencies between the Work Control groups and Engineering (Adequacy of EER procedure and System Engineering Program): between the Engineering groups and management. (F.2)

~ 6 'l f ~

- The policies and procedures for decision making including design/modification program (e.g., the PCR/DCP program. PM program), (F,l),

RESPONSIBLE ORGANIZATION Vice President - Nuclear Production, J. G. Haynes MODE RESTRAINT/DUE DATE:

NONE/90 days following report approval CORRECTIVE ACTION 5.27.

Develop procedural guidance to define what criteria is necessary to closeout/'complete a Recommended Corrective Action. (C.4)

RESPONSIBLE ORGANIZATION PSBcC STA, M. L. Clyde/R: E. Younger MODE RESTRAINT/DUE DATE:

NONE/90 days following r'eport approval

PAGE 64 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION ISSUE S5 (cont'd)

CORRECTIVE ACTION 5.28

. Conduct a review of all existing SPEER and FURR to ensure that Corrective Actions have been appropriately identiiled and are being tracked.

'i RESPONSIBLE ORGKNZZATION PS&C STA, M. L. Clyde/R. E. Younger MODE RESTMZNT/DUE DATE:

NONE/6 months following report approval

IIR 2-3-89-00 1 EVENT DATE: MARCH-3; 1989....

UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EMICTION ISSUE 06 - MSSS LIGHTING The Normal Lighting System provides approximately 90% of the illumination throughout the power block of each unit. The Normal Lighting System is fed by Normal Lighting load centers E-NGN-L17 and L18 w 'c are supplied from the 13.8 KV buses E-NAN-S01 and S02 respectively.

The Essential Lighting System provides approximately 10% of all lighting in the power block. The Essential Lighting S'tem is fed by load centers

'GA-L35 and PGB-L36. These load centersy feed main lighting s '

panels EQBN-D90 and D91. Panel D90 distributes power in the turbine and control buildings. Panel D9.1 distributes power to the auxiliary, containznent, and MSSS buildings.

The Emergency Lighting System provides reliable battery power lighting for proper ingress/egress along emergency rou es Emergency Lighting System energizes automatically upon loss o ssen and deenergizes upon the restoration of Essential Lighting. 'ighting From the Up d ate d Finaln Safety Analysis Report (UFSAR), the Essential Lig hting system supplements the Normal Lighting i htin an and p rovides a minimum n throu hout each unit in the event of a failure o e Li tin s stem. The Essentialal Lig tin feeeders L hting ers on D90 and D91 for ed on SIAS and can be manually reconnected after diesel generator sequeacing. The Essential g g sys energized and is supplied from twoo redundant Class lE loa centers. Th e DC-powered Emergency Lighting system is provi e n e m room, a t the remote shutdown panel room, associate oc con and along emergency exit routes where emergency main aintenance is 'xp ecte .

to be required. In th e even t o f the loss of Essential Lighting sources, the DC enc Li htin s stem is energized automatically. The DC Emer Em g en cy f Qxt th t have self-contained batteries, battery rs and switches that automatically energize the fixt:ures from their batteries in the event of the 1 oss o f th e AC ssource for their battery chargers.

vent, Normal Lighting was lost due to a loss of non-class power was lost due to a SIAS. Emergency lighting did ener ize er design. When Essenti al Lig htin iz d E Lighting g deenergize d an d because a light bulb was burned out in the Essential Lighting System, the MSSS south side was dark.

uildin Normal Lighting was also lost on the loss of non-class power and the Turbine Building Essen tial Lighti g al o lo to th SIAS.

'n issue was identi fie d in th e initiai ni phase of the investigation which has been addressed in the course of the event:

PAGE 66 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOW1ÃG LARGE LOAD REHECTION ISSUE 86 (cont'd)

INVESTIGATIONACTION PLAN (cont.)

6: Upon entering the MSSS building the auxiliary operators noticed that the building was almost totally dark. The limited amount'f lighting made task perfornuznce diffcult in the MSSS and Turbine Building.

IN'VESTIGATIONACTION PLA¹ Engineering Action Plan: The Electrical Engineering group within EED prepared an Engineering action plan to address the concerns raised in this event.

Interviews

& personnel statements: The following individuals were contacted in the course of this event:

1) Operating Crew members

2) QD System Engineer

3) Electrical Engineering Supervisor

4) Electrical Planner Scheduler Techniques Utilized: 1) Facts Collection (see attachment 1)

2) Events and Causal Factors Analysis

3) 'MORT

4) EBT REFERENCES')

2-3-89-001 Fact Data Base (Attachment 1)

2) Engineering Action Plan and Work Orders 342292. 345822, 346404 (WO's which implement the action plan).

3) PCR 87-13-QD-007-Addresses the use of inverters for Emergency Lighting.

4) PCR 87-13-QD-004-Addresses the redesign of the Emergency Lighting in the MSSS.

5) ICR f104829, Change Task 058652 to delete containment equipment and create a new task for deleted equipment.

PAGE 67 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹6 (cont'd)

IN'VESTIGATIONACTION PLAN (cont.)

6) ICR ¹04827. Change Task 058655 to delete containment equipment and create a new task for deleted equipment.

7) Work Order 336722. I M WO for Essential Lighting.

8) EER 86-QD-037-Addressed MSSS lighting deterioration as a result of heat and steam..

RESOLUTION/ANALTSIS:

The Incident Investigation Team has reviewed the event and has made the following conclusions:

The lighting in the MSSS was less-than-adequate during this event for Operator performance of tasks in the MSSS. As a result, additional burdens were placed on the AOs during manual ADV operations.

2) The Essential Light in the South MSSS Building did not work after power was restored. The PM which checks the bulb had been revised subsequent to the last time it was performed to install a "long-life" bulb. The revised PM was not scheduled to be', performed prior to the event, leaving a "short-life" bulb installed, which was burned out.

3) The Emergency Lighting PM task (Task 58655) was last performed on 8/23/87. The PM had been waived 4 times since then.

The review for waiver of Emergency Lighting PM tasks was less-than-adequate.- The combination of Containment and MSSS Emergency Lighting on the same task contributed to the inappropriate waiving of Emergency Lighting PM's. The entire PM was waived due to access considerations inside Containment which did not apply to the MSSS. Additionally the Maintenance Manager saw only the PM task cover sheet vice the actual task details and made his decision on the basis of incomplete information.

4) The failure to perform the PM Tasks did not contribute to the severity of this event. There is no evidence to support that the Emergency Lighting in the MSSS did not function. However, it should be noted that Emergency Lighting is provided for "ingress and egress only". not for equipment operation.

5) Most of the Turbine Building lighting was lost during this event. This made it difficult for the Operators to perform tasks in the Turbine Building, necessitating the use of flashlights.

PAGE 68 IIR 2-3-89-001 EVENT DATE: MARCH 3. 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EGA'ECTION ISSUE 06 (cont'd)

INVESTIGATIONACTION PLAN (cont.)

6) The System Engineer was unaware of pertinent changes in lighting requirements made to the Updated Final Safety Analysis Report (UFSAR).

7) The Planner'took appropriate action to write an ICR to have. the Containment Lighting and MSSS Emergency Lighting on separate tasks prior to this event.

8) The current Preventive Maintenance Procedure 30AC-9MP02 is inadequate with respect to the requirements for waiving PMs by the Maintenance Manager. The procedure did not require that the entire "package" be reviewed by the Maintenance Manager prior to waiving the PMs.

9) D90 and D91 are tripped from the class-buses after a SIAS since they energize non-class components.

CORRECTIVE ACTION 6. 1 Implement the recommended corrective actions of the Engineering Action Plan.

- Install an additional Essential Light in each of the rooms of the building 140'SSS

- Ensure that the future Essential Lighting PM program addresses

. replacement based on the bulb's designed life expectancy and not bulb failure only.

- Perform Work Orders 4303831 and 304291 and update PM tasks 064442, 064443, 022431, and 022425. These tasks establish baseline data for the batteries supplying power to the Unit Control Room Emergency Lighting; (The work has been completed for Unit 1, but the task has not been updated. Unit 2 needs to be fully completed.)

- Install Site Mod QD-022 in all three Units (Add 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> Emergency Lighting in the MSSS).

- Install Site Mod QD-023 in all three Units (Add 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> Emergency Lighting as identiQed in engineering walkdowns to the Diesel' Control, and Auxiliary Buildings.)

RESPONSIBLE ORGANIZATION Electrical - EED, Lazy Henson/G. W. Sowers MODE RESTRAINT/DUE DATE Units 1, 2, 3 - Mode 2 CORRECTIVE ACTION 6.2 a) Evaluate the present FSAR change procedure to ensure that changes to the FSAR.which affect Engineering are forwarded to the EED

PAGE 69 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD IH>ECTION ISSUE P6 (cont'd)

INVESTIGATIONACTION PLAN (cont.)

manager for review.

5) Prepare a memo to all EED Supervisors stating that the System Engineer shall review all USFAR changes which affect the system for which they have responsibility.

RESPONSIBLE ORGANIZATION a) Licensing, W. L. Quinn b) EED, G. W. Sowers 4

MODE RESTXUIQNT/DUE DATE-a) NONE/60 days following report approval b) NONE/60 days following report approval CORRECTIVE ACTION 6.3 Evaluate the AO emergency tasks involving manual manipulation of Safety Related and Important to Safety plant components and determine if they can be performed in the event of a loss of Normal and Essential Lighting condition. Provide Engineering with recommendations for any enhancements via an PCR as required.

RESPONSIBLE ORGANIZATION,',

Plant Director, W. C. Marsh",

NED. E. C. Sterling MODE RESTRAINT/DUE DATE Complete I

t CORRECTIVE ACTION 6.4 Evaluate the Power Block Emergency and Essential Lighting against the Design Basis and Criteria, respectively, and ensure it is adequate to allow Operators to perform their assigned safe shutdown tasks per procedures 41AO-lZZ44, Shutdown outside the Control Room due to fire and/or smoke, and 41AO-lZZ27, Shutdown Outside Control Room, during a loss of non-class power.

RESPONSIBLE ORGANIZATION Electrical - NED, John Barrow MODE RESTRAINT/DUE DATE NONE/60 days following report approval CORRECTIVE CTION 6.5 Preventive Maintenance Procedure 304C-9MP02 will be revised with respect to waiving of PMs by IIR 1-2-89-001.

RESPONSIBLE ORGANIZATIONN/A

PAGE 70 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR THQP FOLLOWING LARGE LOAD REJECTION ISSUE 87 - INSTRUMENT AIR The Instrument Air System provides a normal source of air for ADV operations.

During the event. the IA Compressors lost power, causing the N2 system to provide an automatic backup.

An issue was identiQed in the initial phase of the investigation which has been addressed in the course of the event:

7: The Instrument AirSt Jstem pressure may ha~e decreased too quickly after the loss of non class power.

ACTION PLAN Engineering Action Plan: BOP Engineering has evaluated the adequacy of the response of the Instrument Air System.

Interviews R personnel statements: NONE Techniques Utilized":- 1) Facts Collection (see attachment 1) ",

t REFERENCES H

1) Letter M. R. Oren to G. W.. Sowers. March 14, 1989. IIR 2-3-89-001 Issues 101-00432-MRO/MLC.

2) Letter 102-01161-TDS/ELMC, NRC AIT Preliminary Exit Meeting.

3) Instrument Air Final Report to Support Unit 1 and'2 Restart, Document no. 13-MS-036 RESOLUTION/ANALYSIS The Engineering Action Plan has evaluated the event with respect to the Instrument Air and Nitrogen Back-up Systems. They have also evaluated the system design considerations and PVNGS response to NRC Generic Letter 88-14, Instrument Air supply problems affecting Safety Related Equipment.

The following conclusions, regarding the IA system (The Engineering Evaluation team reviewed the entire compressed Gas System) response, were reached by the IA evaluation team:

1) The air compressors tripped as designed following the loss-of-non-class power.

PAGE 71 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWINg LARGE LOAD REJECTION ISSUE 07 (cont'd)

2) The N2 backup valve opened at 85 psig per the design requirements.

however the pressure in the IA system dropped to 64 - 67 psig whi~h was not consistent with the design.

3) The air pressure decrease in Unit 3 is consistent with the reduction observed in previous trips with concurrent loss-of-non-class power events.

4) In two of the previous low air pressure related events, ADVs operated successfully. There was no apparent safety related equipment failures due to the reduction in air header pressure;

5) A total of 43 FCRs have identified, installation of additional piping or air users to the plant compressed gas system. This as-built condition is not

, reflected in the e~sting calculations for normal and upset pressure drops in the system.

6) Following an earlier event, an EER was issued to document the air header pressure reduction to 65 psig. A check valve was identiQed to be the root-cause of the problem, but no corrective action was initiated.

7) The N2-IA interface spring loaded check valve. IAN-V056 was identified to have a 24:2 psi cracking pressure. The calculations for deQning pressure drops in the IA system assumed a 1 psid drop across the interface valve.

8) Until recently, APS did not have a program for monitoring the compressed gas air quality.

9) The N2 gas subsystem quality is consistent with the current IA subsystem valve design.

10) A review of the PVNGS IA-PMs identiQed some discrepancies between the PMs required by the manufacturer and those currently being performed.

11) Some PM tasks listed in the SIMS Repetitive Work Task database are not kept current.

12) The review of the APS response to Generic Letter 88-14 is continuing.

CORRECTIVE ACTION 7. 1 a) Review and develop PM Tasks defined by "Compressed Gas System Evaluation and Analysis for PVNGS.Units 1, 2, and 3."

PAGE 72 IIR 2-3-89-001 EVENT DATE: ILfARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWZNQ LARGE LOAD RF~CTION ISSUE 87 (cont'd)

RESPONSIBLE ORGANIZATION PS&C. R. E. Younger MODE RESTRAINT DUE DATE NONE/6 months following report approval CORRECTIVE ACTION 7.2 Perform IA PM Tasks developed in Corrective Action 7.1.

RESPONSIBLE ORGANIZATION Unit 1 Work Control, d. Dennis Unit 2 Work Control, P. Wiley Unit 3 Work Control, C. Churchman MODE RESTRAINT DUE DATE NONE/30 days following complete implementation of PM Tasks CORRECTIVE ACTION 7.3 Inspect a sample of the Unit 2 pneumatic system valves for evidence of dirt, moisture or corrosion of the valve components or IA subsystem piping. Analyze any containments found and repair. as applicable.

I RESPONSIBLE ORGANIZATION l~

NED/EED, E. C. Sterling MODE RESTRAINT DUE DATE Complete CORRECTIVE ACTION 7.4 Peform N~ Subsystem performance test to determine cause for pressure dropping to 65 psig. Correct design to prevent recurrence.

RESPONSIBLE ORGANIZATION NED Mechanical, M. F. Hodges/E. C. Sterling MODE RESTRAINT DUE DATE Mode 2- Unit 3 CORRECTIVE ACTION 7.5 Complete the correction of IA system design calculation deficiencies identified in the Intstrument Air Final Report.

RESPONSIBLE ORGANIZATION NED Mechanical, M. F. Hodges/E. C. Sterling MODE RESTRAINT DUE DATE NONE/3-3.1-90

~ ~ ~ r.

PAGE 73 IIR 2-3-89-001 EVENT DATE: MARCH 3, I989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD MMECTION ISSUE ¹? (cont'd)

CORRECTIVE ACTION 7.6 Replace the spring in IA valve IAN-V056,with one corresponding to a lower. pressure drop.

I RESPONSIBLE ORGANIZATION NED Mechanical, M. F. Hodges/E. C. Sterling ODE RESTRAINT DUE DATE Unit 1, 2,3 NONE/Next available IA system Outage CORRECTIVE ACTION 7.7 Include the spring loaded check valves between the N2 and IA systems for. the ADVs in the ASME XI leakage testing program.

RESPONSIBLE ORGANIZATION EED, R. Kropp/G. W. Sowers MODE RESTEVQNT DUE DATE NONE/90 days',following report approval CORRECTIVE ACTION 7.8 Complete the reevaluation of the response to NRC's Generic Letter 88-14 and determine ifadditional actions are necessary based upon this event.

RESPONSIBLE ORGANIZATION Licensing, W. F. Quinn MODE RESTRAINT DUE DATE

'ONE/60 days following report approval

PAGE 74 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 88 - REACTOR COOLANT SYSTEM LEAKAGE The Engineering Action Plan is included as an attachment to this report.

An important aspect of this event was an increase in RCS leakage. There were several factors that led to this condition and other factors that led to a lapse of time before it was recognized. After the condition of increased leakage was identified, a containment entry was performed to identify the sources and to quantify them. There were two sources of leakage. Valve CHV-435, which had previously undergone a Furmanite repair, was identified to be lealdng after the repair. Following this event, the valve was observed to be leaking at a greater rate. The other source was identified to be the RCP 1B seal.

Early in the event power was lost to the RCPs and to the Nuclear Cooling Water system (NC). NC is the normal cooling supply to the water supplied to the RCP seals. The design of the seal injection/seal bleedoff system is such that at hot standby operator action to secure bleedoff is necessary, almost immediately upon loss of NC and seal injection. to preclude damage to the RCP seals.

I '

Technical SpeciQcation LCO 3.4.5.1. requires that the following Reactor Coolant System leakage detection systems shall be OPERABLE in Mo'des 1-4:

a) A containment atmosphere particulate radioactivity monitoring system. b)

The containment sulnp level and flow monitoring system. c) The containment atmosphere gaseous radioactivity monitoring system. Ifnone of the systems are OPERABLE the Unit must be in Hot Standby within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and and in Cold Shutdown within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />. During the event, with the loss of non-class power and the SIAS, none of the systems were OPERABLE.

The following issues were identified in the initial phase of the investigation ll which have been addressed in the course of the event investigation:

8.1: RCS unidentified leakage was calculated to be 4.17 gpm at apprcuafmately 0600.

8.2: Reactor Coolant Pump (RCP) IB seal pressure anomalies mere observed at apprm~ately 0500.

8.3: A high temperature mas indicated on RCP 2A high pressure cooler inlet at 0630.

8.4: The primary operator isolated RCP seal bleedoff while seal injection was being supplied after the loss of Nuclear Cooling Waker.

8.5: The containment temperature and humidity recorders and the containment sump level indicators were inoperable.

PAGE ?5 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹8 (cont'd)

IN'VESTIGATIONACTION PLAN Engineering Action Plan: The NSSS Mechanical Group within EED prepared an Engineering Action Plan. The action plan will be completed during the upcoming refueling outage.

- Interviews

@ personnel statements: The following individuals were contacted. in the course of the investigation:

1) Operating Crew Members

2) EED-NSSS Engineers

3) EED-NSSS Mech. Supervisor

Techniques Utilized

1) Facts Collection (see attachment 1)

2) Events and Causal Factors analysis

3) Energy-Barrier-Target (Engineering Action Plan)

REFERENCES:

I 9

1); 2-3-89-001 Fact Data Base (Attachment 1)

' C ~

• 9

2) Engineering Action Plan and Work Orders 338162, 336503, 316360, 316361, 316362, 316363'(Action Plan implementation work orders).

1

3) 430P-3RC01, Reactor Coolant Pump Operation.

4) 31MT-9RC26, Reactor Coolant Pump Seal Disassembly and Assembly.

5) 31MT-9RC22, Reactor Coolant Pump Seal Housing Disassembly, Kcamination and Assembly.

6) DCP 3FE-RD-037-Reliable power to Unit 3 Containment Sump Level Indicator.

7) M M RG I 0 WR M 814 1989 ~IIRR 00

~lue~, ¹101-00432 MRO/MLC.

8) Memo T. D. Shriver to W. F. Quinn, NRC AIT Prelimin Exit Meetin, March 12, 1989, ¹102-01161-TDS/RLMC.

9) EER 89-RC-023-Analysis of automatic protection for RCP seals.

PAGE 76 IIR=2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~lCTION ISSUE 88 (cont'd)

RESOLUTION/ANALYSIS

. The Incident Investigation Team has reviewed this event and has made the following conclusions:

The primary cause of the RCP 1B seal leakage is considered 'to be degradation of the 3rd stage and possibly the 2nd stage seals.

2) The Primary Operator unnecessarQy secured RCP seal bleedoff whQe seal injection was still being supplied after the loss of Nuclear Cooling Water and the RCPs were in hot standby (not runjning).

Per the Reactor Coolant Pump and Motor Emergency Procedure (43AO-3ZZ29), Section 9.2...'The RCP(s) may be maintained in the hot standby condition for an indeflnite period of time with either Nuclear Cooling Water (NCW) low or seal injection, lost (but not both). Provided:

IfNCW i sin ul l lost n ure that the controlled bleedoff valve n fected um s rem 'n o n "or Ifseal injection flow is lost anytime an RCP is in hot standby.

ensure that the controlled bleedoff valve on the affected pump(s) is closed within 1 minute."

3) Operator action to isolate RCP bleedoff flow is relied upon to protect RCP seals'uring a loss of non-class power event. The consequence of not performing these actions in a timely manner is Reactor Coolant System leakage through the RCP seals.

4) The RCP 1B seal failure was most likely due to the reestablishment of controlled bleedoff flow to the Reactor Drain tank (RDT). This flow path occurred after the Primary Operator secured bleedoff flow. The leakage path appears to have occured past the seat of CHA-HV-507 (relief isolation to the RDT), since RDT temperatures were increasing after the valve was shut.

5) The Containment Sump Level indication lost power when the SIAS/CIAS occurred. A design change to upgrade the power supply to the containment sump level instruments had not yet been implemented.

The loss of-the sump level, temperature and humidity indications for containment complicated the diagnosis of the event.

7) Containment temperature and humidity recorders lost power when non-class. buses de-energized.

8) The Tech Spec leakage monitoring capability did not meet the LCQ in requirements and the action statement was entered.

PAGE 77 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLO%XNG LARGE LOAD RF~CTION ISSUE 88 (cont'd)

9) The loss of instrumentation delayed recognition of increased leakage from the RCP seal and CHV-435 (due to a failure of the furmani<<

- repair) until power was restored to the indicators.

10) The operators performed appropriately in not cross-connecting Nuclear Cooling Water (NC) and Essential Cooling Water (EW) systems.

This would have rendered EW inoperable and may have resulted in the loss of EW.

CORRECTD/E ACTIONS CORRECTIVE ACTION 8. 1 Implement the Engineering Action Plant recommended corrective actions.

II a) WO 0338162 - Rework CHV-435 Flange/gasket leak b) WQ 0346503 - Inspect RCP 1B seals per 31MT-9RC26 c) WO 0316360, 316361, 316362, 316363 - Inspect RCP )ournal bearings as necessary and replace seal cartridges on all 4 RCPs as originally planned during the refueling outage.

d) Conduct a design review or develop and implement a procedure to test CHA-HV-507, seal bleedoff isolation valve, to ensure it will perform as designed, including under a loss of Instrument Air conditions.

- "RESPONSIBLE ORGANIZATION a) Unit 3 Work Control, C. D. Churchman b) Unit 3 Work Control, C. D. Churchman c) Unit 3 Work Control, C. D. Churchman d) NSSS - EED, G. Waldrep/G. Sowerq MODE RESTEVlQNT/DUE DATE a) Unit 1 and Unit 3 - Mode 4/Following Refueling Outage b) Unit 1 and Unit 3 - Mode 4/Following Refueling Outage

" c) Unit 1 and Unit 3 - Mode 4/Following Refueling Outage d) Unit2-Mode 4 CORRECTIVE ACTION 8.2 Evaluate signiQcance of RCP-2A high pressure cooler inlet high temperature condition for root cause of failure determination and for any inspection and/or rework requirements. (EER 89-RC-038)

RESPONSIBLE ORGANIZATION NSSS - EED, G. Waldrep/G. Sowers MODE RESTRAINT/DUE DATE NONE/30 days following report approval

PAGE 78 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF JECTION ISSUE 88 (coat'd)

CORRECTIVE ACTION 8.3 Perform an evaluation (e.g. risk-based analysis) on a design change to automatically protect the RCP seals on a loss of non-class power (EER 89-RC-023)

RESPONSIBLE ORGRZGZATION Mechanical - NED, M. Hodges/E. C. Sterling MODE RESTXUZNT/DUE DATE NONE/90 days following report approval CORRECTIVE ACTION 8.4 Perform a HPES evaluation to address the operator action in securing seal bleedoff with seal injection still being supplied.

RESPONSIBLE ORGBZGZATION STA-PS&C, M. L.;Clyde/R. Younger MODE RESTRAINT/DUE DATE NONE/90 days following report approval I

I CORRECTIVE ACTION 8.5 Implement DCP 3FE'-RD-037 during this refueling outage, to provide reliable power to the containment sump level indicators.

f RESPONSIBLE ORGANIZATION Unit 3 Work Control, C. D. Churchman MODE RESTRAINT/DUE DATE Complete CORRECTIVE ACTION 8.6 Initiate a design change (PCR or Site Mod) for a reliable power supply to the containment temperature and humidity recorders.

RESPONSIBLE ORGANIZATION I & C - EED, J. Summy/G. Sowers MODE RESTRAINT/DUE DATE

.NONE/90 days following report approval

IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RPW'TION

'ISSUE 88 (cont'd)

CORRECTIVE ACTION Management shall re-evaluate the use of FurmaniteO for 8.7'lant primary system leaks.

RESPONSIBLE ORGANIZATION Plant Director, W. C. Marsh MODE RESTRAINT/DUE DATE Mode 2 - U-I, U-2, U-3

PAGE 80 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹9 - POTENTIAL HIJMAIVPERFORMANCE DEFICIENCIES DURING POST-TRIP RESPONSE BY RADIATIONPROTECTION CHEMISTRY TECHNICIANS

Background

During the course of the event on 3/3/89, a loss of non-class power occurred. This resulted in an inability to use normal methods to assess plant conditions. This was particularly important when it became necessary to complete the post-trip tasks assigned to Radiation Protection and Chemistry.

The actions of the Radiation Protection and Chemistry Technicians were analyzed by the Incident Investigation Team. One inappropriate action was identified. However, the investigators uncovered several items which led to difQculties in the performance of the assigned tasks. It should be noted that the Chemistry and Radiation Protection Technicians were able to successfully overcome these difficulties and correctly complete all requirements with the exception of the calculation of the off-site dose assessment.

ACTION PLAN I

Engineering Action Plan: None Interviews and Personnel Statements The following individuals were contacted in the course of this investigation:

1) Operations Crew Members

2) Radiation Protection Technicians

3) Supervisor of Chemistry Standards

4) Lead Site Emergency Planner Techniques UtilMd: Human Performance Evaluation (HPES) techniques were utilized to evaluate the identified inappropriate action and difficulties encountered by the Technicians

PAGE 81 IIR 2-3-89-001

'VENT DATE: MARCH 3. 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EGMECTION ISSUE ¹9 (cont'd)

REFERENCES

1) EPIP-14, Dose Assessment.

2) DCP 3FE-SQ-058- Reliable Power to Unit 3 RMS mini-computer.

3) 'O 304785- DCP 3FK-SQ-058 implementation.

4) WO 317270- DCP 3FE-SQ-058 implementation.

5) WO 317273- DCP 3FE-SQ-058 implementation.

6) DCP 1 FE-SQ-058- Reliable Power to Unit 1 RMS mini-computer.

7) WO 284976- DCP 1 FE-SQ-058 implementation.

8) Memo M. R. Oren to G. W. Sowers, March 14, 1989, IIR 2-3-89-001

~i~up, ¹101-00432 MRO/MLC.

9) Memo T. D. Shriver to W. F; Quinn, AIT Pr Iimin Exi Me n, March 12, 1989, ¹102-01161-TDS/RLMC.

10) Memo M. R. Oren to W. C. Marsh, dated March 21, 1989, Dose Calculation Concern- Unit 3 Investi ation: 2-3-89-001.

¹'101-00437-MVL/MRO

11) Memo T. J. Warren to P. W. Hughes . Dated March 23, 1989, ~Meetin with NRC on March 23 1989. ¹237-00160-TJW/JBC.

RESOLUTION/ANALYSIS The HPES evaluation identified the following causal factors which contributed to the following:

1.) The incorrect calculation of off-site dose assessment 2.) The difficulties encountered by the Technicians in the performance of their post-trip tasks:

VERBAL COKQEUNICATION 1.) The information exchange between the RP Technician in the STSC and the Control Room Staff was less than adequate. The RP technician based his initial calculation on a steam release through the Atmospheric Dump Valves. The actual release was through one Main Steam Safety Valve (The default value per EPIP-14, Appendix B, for flow through a MSSV, 2.39 curies/sec, is larger than the default valve for flow through an ADV, 0.0387 curies/sec.

due to highly conservative assumptions used in the Safety Analysis.

No attempt to verify'the actual release path was made. The RP

~ v A

PAGE 82 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD tuMZCTION ISSUE ¹9 (cont'd)

RESOLUTION/ANALYSIS Technician utilized the ADVs as the release path based on conversations, overheard in the Control Room. %hen the RP Technician reported to the Control Room. he did not receive any speciQc plant data related to release paths from the Emergency Coordinator. The RP Technician did not communicate the conservative assumptions (i. e. 1% failed fuel, 1 ruptured Steam Generator tube, and a stuck open Main Steam Safety for 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />) utQized in his manual calculation per EPIP-14.

This communication is important because the Emergency Coordinator is tasked with making the appropriate Protective Action Recommendations (PARs) to offsite organizations based on actual plant conditions and recommendations of the RP Technician.

2.) The Chemistry Technicians did not receive any plant specific information from the Control Room because the Chemistry Technicians did not go to the Control Room. They believed. based on past experiences, that they would not be able to "get in". This caused the Chemistry Technicians to perform tasks which were not required (e.g. sampling main condenser air removal exhaust following the Main Steam Isolation).

WRITTEN COMMUNICATION 1.) EPIP-14 (CALCULATIONOF OFFSITE DOSE ASSESSi~NT) contains a step which causes the performer to complete an App'endix which is not required when using default values. This can lead to an incorrect solution. In this case. this did not cause an incorrect calculation because this step was omitted. It should be noted that had EPIP-14 been completed as written using conservative default values for failed fuel. steam generator the'pecified leakage, and a stuck open safety, Protective Action Recommendations (PARs) consistent with a General Emergency would have been derived. This would have been inappropriate since no actual release was in progress and the actual plant conditions supported no Protective Action Recommendations.

2.) The Chemistry Technicians had several post-trip tasks to complete.

The required tasks are contained in parts of several procedures which are not adequately cross-referenced. There is no method to ensure that all required tasks are completed in the appropriate time frame. In this event. the Chemistry Technicians were able to identify.and complete all required tasks but the potential exists for important tasks to be omitted in future events. Some of these tasks are requhed for regulatory compliance with the plant Technical SpeciQcations but there is no central procedure to ensure these are performed.

PAGE 83 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD ~~lCTION ISSUE ¹9 (cont'd)

RESOLUTION/ANALYSIS (continued)

INTERFACE DESIGN OR E UIPMENT CONDITION 1.) The Mesorem and RMS mini-computer both lost power as a result of the initiating event. This was a ma]or causal factor in the incorrect calculation of the off-site dose calculation.

A Design Change Package (DCP 3FE-SQ-058) had been approved'to provide a class lE power source to the RMS mini-computer.

However. this modification had not been fully implemented in Unit 3 and is scheduled for the current refueling outage.

The unavailability of actual radiation readings for the plant effluent monitors made it necessary to utilize conservative default values in the offsite dose assessment.

The MESOREM program for calculating off-site dose assessment is kept on the IBM PC in the Satellite Technical Support Center (STSC). Power to the computer is provided by a non-class 120 VAC source which was unavailable due to the initiating event. The capabQity to utilize the preferred method, for calculating offsite dose (MESOREM) was not. available necessitating the use of the manual calculation per EPIP-14.

EPIP-14 is adequate for off-site dose assessment. Had the RMS values been available, a Protective Action Recommendation consistent with plant conditions would be calculated assuming the procedure was completed correctly.

2.) At the time of the event. RU-141 (Condenser Air=Removal Exhaust

'onitor) was inoperable. The auxiliary sample cart was in service as required by Technical Specifications. The auxiliary sample cart is powered via an eMension cord to a non- class receptacle. When non-class power was lost. the ability to monitor the air removal exhaust Was also lost. The Chemistry Technician obtained a portable generator to power the auxiliary sample cart but could not get the generator started. The generator was in-place and compatible with the sample cart. however, it was not a design requirement to have the cart available. The loss of capability to sample the air removal exhaust was not consequential in this event due to the MSIS isolating this release path via the condenser.

However, for other events, such as a Steam Generator Tube Rupture coincident with a Loss of Power (LOP), the ability to monitor air removal exhaust is important to ensure offsite dose commitments are satisfied. PCR 88-13-ZZ-008 was submitted in April 1988 to change the auxiliary sample cart receptacle to a class power supply.

PAGE 84 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD R3MECTION ISSUE ¹9 (cont'd) '

RESOLUTION/ANALYSIS (continued)

EMTIRONMENTALCONDITIONS 1.) The lighting in the area of the Post Accident Monitoring Unit (PAMU) contributed to the inability of a Radiation Protection Technician to obtain actual Main Steam Line Monitor readings. The

~

Technician unspccessfuIIy attempted to locate a Kaman Electronic Portable Indicating Controller (KEPIC) in a storage box utilizing a Qashiight. These readings are critical to performing a realistic off-site dose calculation. The lack of this information resulted in the use of conservative default values in the EPIP-14 calculation.

2.) The Chemistry Technician reported that the Hot and Cold chemistry labs were dark initially after the event. Essential Lighting was restored with the re-energization of D90 and D91. Additionally, the chemistry lab instrumentation was lost until D90 and D91 was restored. The Chemistry Technicians were able to complete all required sampling following the restoration of power to M-19 arid M-20.

WORK PRACTICES 1.) The failure of the BP Technician to utilize effective work practices was the primary causal factor involved in the incorrect off-site dose calculation. Several math errors were made and not caught by the Technician prior to communicating the Protective to the Emergency Coordinator. The Technician Action'ecommendations did not correctly follow the steps contained in EPIP-14.

The combination of math errors and procedure non-compliance resulted in an incorrect calculated 2 hour2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> child thyroid dose at the site boundary. As mentioned above, had the calculation been performed correctly utQizing the required default values, PARs consistent with a General Emergency would have resulted.

2.) Not applicable TRAININGMETHODS CONTENT 1.) The Radiation Protection Technicians are required to receive annual training in off-site dose assessment and the use of EPIP-14; This is accomplished in a General Training Course. The individual involved had successfully completed this course in August 1988. A review of the course content and objectives was performed by the Investigation Team. The course adequately covers methodology and

'required background information to successfully complete an off-site dose assessment utilizing either the MESOREM or EPIP-'14.

The Technicians are instructed that it is necessary to complete the

PAGE 85 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~CTION ISSUE 89 (cont'd) 4 TRAIIVIIVGMETHODS CONTENT (continued) ~

assessment within 15 minutes. A review of applicable references was performed by the Investigation Team and it is unclear whether it is necessary to complete the calculations within 15 minutes. The perception by the technician that the calculation had to be completed in 15 minutes and the length of time since he

~

had performed EPIP-14 contributed to the incorrect calculation.

2.) Not applicable CORRECTIVE ACTION 9. 1 Ensure that DCP 3FE-SQ-058 is installed by implementation of WOs 304785, 317270 and 317273. (Reliable Power to Unit 3 RMS

,mini-computer).

RESPONSIBLE ORGANIZATION Unit 3 Work Control, C. Churchman (Unit 3 Nuclear Construction, WO 304785)

(Unit 3 Electrical Maintenance, WO 317270, 317273)

MODE REST~ZNT/DUE DATE Mode 4 following Unit 3 first refueling CORRECTIVE ACTION 9.2 Ensure that DCP 1FE-SQ-058 is installed by implementation of WOs 284976. (Reliable Power to Unit 1 RMS mini-computer).

RESPONSIBLE ORGANIZATION Unit 1 Work Control, J. Dennis (Unit 1 Nuclear Construction - WO 284976)

MODE RESTRAINT/DUE DATE Mode 4 following Unit 1 second refueling CORRECTIVE ACTION 9.3 Provide an uninterruptible power supply with a capacity of at least 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> duration for the MESOREM computer in the STSC.

RESPONSIBLE ORGANIZATION Emergency Planning, H. Bieling MODE RESTRAINT/DUE DATE U-1 Mode 2 U-2 Mode 2 U-3 Mode 2

PAGE 86 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 3 REACTOR TRIP FOLLOWING LARGE LOAD ~UNCTION 'NIT ISSUE ¹9 (cont'd)

CORRECTIVE ACTION 9.4 Change EPIP-14 to include the following provisions in the event that manual default values have to be used:

a) Direct that Geld radiation level measurements be immediately obtained, ifusing default values for Dose Assessment calculations.

b) To direct the EPIP-14 performer to advise-the Emergency Coordinator that value obtained are based on:

1) 1% failed fuel condition

2) A 400 gpm primary to secondary leak

3) A MSSV is fully open and has been or will be. open for 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> c) Review EPIP-14 for any deficiencies or enhancements.

RESPONSIBLE ORGANIZATION Emergency Planning, H. Bieling MODE RESTRAINT/DUE DATE a) Mode 2 b) Mode2-c) Mode2 CORRECTIVE ACTION 9.5 a) Implement a Job Performance Measure (JPM) for RP Technicians to perform EPIP-14 manual calculations under a variety of scenarios. Training Department to develop JPM and train selected Lead RP Technicians in use of JPMs to monitor and document performance by the individual Technicians.

b) The Training Department Management Action Center will evaluate EPIP-14 and conduct an assessment of the adequacy of continuing training on EPIP-14. Implement the appropriate recommendations.

RESPONSIBLE ORGANIZATION Training Department, W. Fernow MODE RESTRAINT/DUE DATE a) Mode 2 - U-l, U-2, U-3 b) NONE/within 90 days following report approval

PAGE 87

~

IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 89 (cont'd)

CORRECTIVE ACTION 9.6 Evaluate the creation of a procedure to address post-trip response Chemistry Technicians (ICR 008532 submitted). 'or RESPONSIBLE ORGANIZATION Chemistry Standards, J. Cederquist MODE RESTREIZNT/DUE DATE NONE/60 days following report approval CORRECTIVE ACTION 9.7 Initiate and complete the design change proposed in PCR 88-13-ZZ-008 (class power to effluent monitor au~liary sample carts)

RESPONSIBLE ORGANIZATION EED'G., Sowers MODE RESTRAINT/DUE DATE I NONE/180 days following report approval A II CORRECTIVE ACTION 9.8 'I Review and modify, as appropriate, the training course for Emergency Coordinators and RP Dose Assessment Technicians in relation to the human performance deficiencies identified in this ~

report.

RESPONSIBLE ORGANIZATION Training, W. F. Fernow MODE RESTRAINT/DUE DATE NONE/60 days following report approval CORRECTIVE ACTION 9.9 Evaluate the 15 minute requirement for generation of PARs to determine ifan actual off-site dose calculation is required or can be based on the default NUE-PAHs until radiation level measurements are available. Update E-Plan Procedures accordingly based on the evaluation.

RESPONSIBLE ORGANIZATION Emergency Planning, H. Bieling MODE RESTRAINT/DUE DATE NONE/60 days following report approval

PAGE 88 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REHECTION

. ISSUE ¹9 (cont'd)

CORRECTIVE ACTION 9.XO Review requirements for Control Room Access with Chemistry Personnel stressing that the Lead or Acting Lead Chemistry technician is part of the on-shift team and is allowed into the Control Room when it is necessary to communicate pertinent information to the Shift Supervisor.

RESPONSIBLE ORGANIZATION Unit Chemistry Managers, D. Fuller U-1 R. Ferro U-2 J. Scott U-3 MODE RESTRAINT/DUE DATE Mode 2- U-l, U-2, U-3 CORRECTIVE ACTION 9.1 1 Review Emergency/Essential lighting requirements and design for the chemistry labs and area around the PAMU to ensure exists to support required post trip analyses.

adequate'ighting RESPONSIBLE ORGANIZATION Electrical - NED, J. Barrow/E. C. Sterling MODE RESTRAINT/DUE DATE NONE/120 days following report approval

PAGE 89 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹10 - MISCELLANEOUS E UIPMENT ISSUES During this event other equipment abnormalities occurred. These and other equipment issues identified'uring the investigation are listed in this section.

Each issue is listed along with a description of the abnormality. When appropriate, a follow-up document or due date is also included.

Item 10.1 SGE-PSV-579 actuated at' pressure lower than its design setpoint.

CONCLUSION Main steam Safety Valve SGE-PSV-579 cycled several times. It was observed by a Control Room operator to be lifting at about 31 psi below its nominal setpoint of 1250 psig (+/- 1%). The PMS Alarm Typer recorded the Main Steam Safety Valve lifting on 4 occasions.

CORRECTIVE ACTION

,EED - BOP, K Johnson/G. W. Sowers has. initiated an action plan to

,troubleshoot the valve. The valve was removed and shipped to a test facQity for evaluation and rework per WO 346108..

MOD/ RESTRAINT/DUE DATE

.Unit 3- Mode 2 Item 10.2 The ¹I Steam Generator Downcomer Isolation Valve (SGA-UV-l72) would not open from the Control Room.

CONCLUSION SGA-UV-172 failed to open due to an apparent equipment malfunction.

This failure prevented use of the non-essential amdliary feedwater pump.

t CORRECTIVE ACTION

~ED-B Kdl /G.W.B 'll E t dd t the appropriate follow-up document. (per memo 101-00432-MRO/MLC of3/19/89)

MODE RESTRING/DUE DATE NONE/Within 30 days of report approval

IIR 2-3-89:001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD EUMECTION ISSUE 810 (cont'd)

MISCE~QCEOUS EQUIPMENT ISSUES (continued)

Item 10.3 SGE-UV-169 (01 Steam Generator MSIV Bypass) was manually opened after it was unable to be opened from the Control Room CONCLUSION The MSIV bypass did not open in override from the Control Room.

The valve was subsequently'pene'd manually. Contrary to procedure, the attempt to open the valve remotely and the local operation of the valve was done with the downstream throttle valve open vice closed.

The purpose of the throttle valve is to allow for controlled warning of the steam lines not possible using the MSIV bypass valve alone. The bypass valve is designed to open with full system differential pressure across it. Closing the throttle valve Qrst will not reduce that differ'ential pressure: hence, the procedural violation had no impact on the bypass valve failing to open remotely.

The manual operator was broken following the event. An evaluation will be performed.

CORRECTIVE ACTION

1) EED - BOP R. Johnson/G. W. Sowers will evaluate and generate the appropriate follow-up document. (per memo 'I 101-00432-MRO/MLC of 3/19/89).

2) Inappropriate operator action addressed in Issue Ol'1

3) Complete EER 89-SG-131 MODE RESTXUZNT/DUE DATE

1) NONE/Within 30 days of report approval

3) NONE/Within 90 days of report approval Item 10.4 NormaL Chillers A and 8 tripped when normaL chiller C started when NBN-S02 was energized.

CONCLUSION Normal Chiller C (WCN-EOlC) automatically restarted when NBN-S02 was reenergized. The resulting chilled water system perturbation caused normal chillers A and B to trip on low cooling water flow. The automatic restart of Normal Chiller C was a proper operation of the chiller controls. The trip of the A and B Normal Chillers is representative of a known problem. This problem is being addressed by EER 89-WC-002.

CORRECTIVE ACTION No additional actions required.

PAGE 91 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹10 (cont'd)

Item 10.5 The ControL Room handsmitch for RCP 2B had to be taken to "stop" tmice to secure the pump.

I CONCLUSION The Primary Operator stated that he had to take the handswitch to stop twice to secure the pump. An analysis of the Unit Digital Fault Recorder trends by Engineering Evaluations - Electrical showed that the RCP 2B trip coil was energized twice before the pump's circuit breaker opened.

Engineering-Electrical has initiated a troubleshooting action plan to analyze this occurrence. The most likely cause is due to sticking or misalignment of the armature extension shaft of the breaker 3ENANS02M. Lack of lubrication on the trip shaft could have contributed to.the problem.

A review of maintdnance history shows no corrective maintenance has been performed on this breaker. 32ST-9ZZ06 was last performed on 7/14/87. This ST is performed on a 60 month cycle. Presently. there is no PM task for this breaker.

CORRECTIVE ACTION

1) Unit 3 Work Control - C. D. Churchman Perform 32MT-9ZZ29 on breaker 3ENANS02M, which includes the trip latch and, trip coil armature inspection and adjustment.

WR¹ 341326 was initiated to perform this. Perform adjustments, lubricate and/or do corrective maintenance, perform minimum pick-up voltage test for the trip and closing coil and functionally test the breaker per 32ST-9ZZ06.

2) PS&C - R. E. Younger initiate PM task to inspect/test all RCP breakers per 32MT-QZZ29.

MODE RESTRAINT/DUE DATE

1) Unit 3 - Mode 4

2) Mode 4 - U-l, U-2, U-3 Item 10.6 A circulating mater pump discharge valve did not fully close folloming the restoration ofpomer.

CONCLUSION Control Room and auxiliary operators reported that CWN-HV-008 discharge valve for Circulating Water Pump D. did not fully close when power was restored to the valve. The valve was later fully closed manually.

CORRECTIVE ACTION Uni 3 Work Control - C. D. Churchman WO ¹348434 was generated by on-shift personnel to troubleshoot.'and 1

PAGE 92 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹10 (cont'd)

Item 10.6 rework the problem via the normal work control process. This was a previously noted problem and the valves are scheduled to be replaced during the Unit 3 outage.

1 MODE RESTMDNT/DUE DATE Unit 3 - Mode 2 Restraint Item 10.7'lT check valve leakage alarms and B02 pressurizer pressure recorder indicate cold leg injection occurred.

CONCLUSION These. conditions are normal for SIAS actuation. Engineering Evaluation-Mechanical NSSS confirmed that these alarms are normal for HPSI pumps running and injection valves open. and that injection into the RCS is assumed to have occurred on any SIAS actuation initiated by low pressurizer pressure. A pressure of 2250 was noted when the condition alarmed. This was most probably due to leakage of the "Qrst-off'CS check valve. The leakage of the check valves was subsequently verified to meet the check valves leakage surveillance requirements.

CORRECTIVE ACTION, No corrective actions are required. I Item 10.8 When 'B'harging Pump was restarted. a high seal water pressure alarm was received with seal pressure locally

ven~d at 12psig.

CONCLUSION The charging pump seal pressure Hi-Lo alarm setpoints are 5 psig and 19.5 psig. The alarm was received and pressure veriQed to be normal locally. The 'B'harging pump seal pressure alarm should not have come in with seal pressure at 12 psig actual pressure. This did not have an impact on the Unit 3 event.

CORRECTIVE ACTION Unit 3 Work Control - C. D. Churchman WR ¹319968 was generated by the IIT to troubleshoot and rework the problem via the normal work control process.

MODE RESTRAINT/DUE DATE NONE/Within 90 days of report approval

PAGE 93 IIR 2-3-89-001 EVENT DATE: 5fARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD RF~:CTION ISSUE 810 (cont'd)

Item 10.9 Oil lijtpumps 9A and 9H indication lights on B06 were observed to be flickering.

CONCLUSION Lift oil pumps LON-PO9A and LON-PO9H apparently stopped momentarily as the generator was coasting down after the turbine tripped. When the generator tripped at 01:06. all liftoil pumps stopped. After restoration of power, the turbine liftoQ pumps operated normally.

CORRECTIVE ACTION No corrective action are necessary.

Item 10.10 Main Steam Safety valve PSV 577 was discovered to have a damaged bolt/fastener.

I'ONCLUSION MSSV PSV 577 was originally quarantined to investigate the damaged bolt or fastener. Engineering Evaluations-Mechanical BOP had the quarantine released after they determined that the bolt was part of a test rig on PSV 577 and not a structural component.

CORRECTI'VE ACTION Uni Work Control - C. D. Churchman WR 0319969 was generated by the.IIT to replace the bolt via the noimal work control process..

MODE RESTEVQNT/DUE DATE NONE/120 days from report approval Item 10.11 There was an excessive number ofdelayed alarms on the PMS.

CONCLUSION The alarm capacity of the PMS is 4 alarms per second. The reconstructed peak alarm rate during the trip was 63 alarms per second. Since there was a large but not excessive amount of alarms, the Plant Monitoring System delayed the reporting of some alarm conditions. This did not have an impact on the Unit 3 event.

CORRECTIVE ACTION Operations Computer Support will perform an Engineering Evaluation of the alarm processing function of the Plant Monitoring Computer System and make recommendations.

PAGE 94 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 010 (cont'd)

The following EER's were initiated by the OCS group:

89-BJ-010 - Evaluate desirability of adding more Sequence of Events (SOE) alarm points.

89-HJ-Ol 1 - Evaluate program priorities with software capability of only 4 alarms/second throughput.

89-HJ-012 - Evaluate hardware output (alarm typer) capacity of only 4 lines/second.

89-HJ-013 - Evaluate time tagging capability of PMS.

MODE RESTXVZNT/DUE DATE:

NONE/Within 90 days of report approval.

Item 10.12 There is no time synchronization between uarious computer data acquisition systems.

CONCLUSION PCR 83-AO-SD-OOl, to synchronize data acquisition systems was previously written. This item has been identified as plant betterment.

The lack of time synchronization of independent data acquisition systems make manual synchronization of data necessary post event.

OCS has determined that implementation of this PCR would be cost prohibitive and therefore will not be pursued.

CORRECTIVE ACTION NONE. Manual synchronization is acceptable.

MODE RESTEVQNT/DUE DATE N/A

PAGE 95 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹ 1 1- EGJEKAN PERFORMANCE EVALUATIONSYSTEM The Human Performance issues have been tabulated on a matrix below identifving the Behavior Shaping Factors (Causal Categories) that led to the inappropriate actions for each case. These identified errors have been evaluated utilizing HPES techniques'.

/

I The Human Performance evaluations associated with the ADVs are addressed in Issue ¹5. The Human Performance evaluation associated with the Radiation Protection and Chemistry issues are addressed in Issue ¹9.

The matrix conclusions are described on the following pages.

INVESTIGATIONACTION PLAN Action Plan: Utilize HPES techniques to evaluate issues.

Interviews Bc Plant Standards and Control conducted personnel statements: interviews with the Operations staff.

Identification of the causal factors associated with human errors show the following:

I A. Vezbal Communic'ation RE SO'LUTION/ANALYSIS (Mhtrix Error ¹7) Problems wer'e encountered during reclosure of generator output breakers:

. Problems were encountered when the Control Room staff was not able to contact the Electrical System Engineer. This deficiency led them to contact a PROC representative who may not have been fully qualiQed to recommend the resetting of the relay..

CORRECTIVE ACTIONS EED to supply the Units with a current and accurate phone list of qualified Engineers and PRBtC representatives.

'I RESPONSIBLE ORGANIZATION EED-Electrical, G. W..Sowers/L. L. Henson MODE RESTRAINT/DUE DATE Mode 2- U-l, U-2. U-3

PAGE 96 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE ¹ 1 1 HIJ1KAN PERFORM~ANCE EVALUATIONSYSTEM (continued)

B. Written communication I RESOLUTION/ANALYSIS B.l (Matrix Error.07) Problems were encountered during reclosure of generator output breakers:

Procedure to reset "86" lockouts was less-than-adequate. The causes for the 186G-9 relay actuation were not identiQed. The appendices in the "Degraded Electrical" procedure describe the required action for resetting "86" lockouts in general which is less-than-adequate.

The existing procedure allows resetting of "86" Relays with the concurrence of the System Engineer or PR&C representative without the preferred actions performed'e.g. without inspections).

B.2 (Matrix Error P8) The procedures which provide guidance for resetting SIAS/CIAS actuations and equipment was less-than-adequate:

The applicable procedures do not provide guidance for performing the required Surveillance Testing after the SI valves have been cycled.

CORRECTIVE ACTIONS B.l Modify Degraded Electrical Power Procedure (4XAO-XZZ12),

Appendix G, to included better and more specific guidance on resetting "86" Relays. This guidance shall be developed by EED-I Electrical group. The guidance shall include the Plant's philosophy on resetting "86"lockouts, strict guide lines on actions required to,be performed prior to resetting "86" Relays, and consequences of equipment damage for inappropriate actions.

B.2 Modify Emergency Operator Procedure (4XEP-XZZ01) Appendix P-Resetting SIAS/CIAS and Inadvertent SIAS and/or CIAS (4XAO-XZZ28) to "trigger" the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> Technical Specification requirement.

RESPONSIBLE ORGANIZATION

~

B. 1 EED-Electrical, G. W, Sowers/L. L. Henson PS&C Operations, R. E. Younger/R. E. Buzard B.2 PS&C Operations, R. E. Younger/R. E. Buzard MODE RESTEUIZNT/DUE DATE B.l Mode 2- U-l, U-2, U-3 B.2 NONE/60 days following report approval

~ '

PAGE 97 IIR 2-3-89-00 1 EVENT DATE: MARCH 3, 1989 U5ET 3 REACTOR TRIP FOLLOWING LARGE LOAD EGMZCTION ISSUE ¹11 HlB+LN PERFORMANCE EVALUATIONSYSTEM (continued)

C. Work ractices RESOLUTION/ANALYSIS C.l (Matrix Error ¹5) MSIV bypass valve SG-UV-169 was not operated in accordance with procedure.

(Matrix Error ¹6) Primary Operator isolated RCP bleed-off with seal injection & loss of NCW (Matrix Error ¹7) Problems were encountered during reclosure of generator output breakers (Matrix Error ¹8) STs for SI valves were not performed within the required 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />s:

Review of applicable Technical Specification Surveillance Requirements was less-than-adequate.

C.2 (Matrix Error ¹9) The Primary Operator did not log Au~liary Spray usage as required:

Required procedures were not used by the Operators. this is less-than-adequate.

These Human Performance errors would have been prevented if the appropriate procedures were used and correctly adhered to.

V CORRECTIVE ACTION C.1 1) Management shall reemphasize to Plant personnel the C.2 importance of procedure use arid strict adherence.

2) Management shall. provide an evaluation of the procedure preparation methodology (i.e., the Procedure Writers Guide) and determine ifadditional changes are required to improve procedure format and content.

C.2.1 Evaluate using PMS points to automatically log Aumliary Spray temperature and Pressurizer temperature when AuxQiary Spray is

., used.

C.2.2 Evaluate modifying the EP and/or applicable RO procedures to include steps which require the logging of aQ required Au~liary Spray information.

C.1 ')

RESPONSIBLE ORGAN12'ATION Plant Director. W. C. Marsh

2) Plant Director, W. C. Marsh C.2.1 EED-Electrical. G. W. Sowers/L..L. Henson

-0 C.2.2 PS&C Operations, R. E. Younger/R. E. Buzard

PAGE 98 IIR 2-3-89-001 EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION ISSUE 011 XGBLAN PERFORMANCE EVALUATIONSYSTEM C. Work ractices (continued)

MODE RESTRAINT/DUE DATE C. 1 1) Mode 2 entry restraint

2) NONE/120 days following report approval C.2.1 NONE/60 days following report approval C.2.2 NONE/60 days following report approval D. Su erviso methods RESOLUTION/ANALYSIS (Matrix Error 07) Problems were encountered during reclosure of generator output breakers:

The Control Room supervision performed inappropriately. because they reset the relay assuming that the relay "lock-out" would actuate with the Generex trips. They-should have used drawings or other appropriate documentation which identifies the causes for the 186G-9 "lockout" actuations. This research would have identified that the Generex trips are the only cause for this. particular relay actuation. This would'have eliminated the troubleshooting test which was performed and would have eliminated any assumptions made prior to the resetting of the relay.

CORRECTIVE ACTIONS Operations shall review the modifications to the Degraded Electrical Power Procedure (4XAO-XZZ12), as identified in the Written Communication's section. The Operations crew sh~ then adhere to the procedure and not allow or perform unnecessary troubleshooting.

RESPONSIBLE ORGANIZATION Operations, J. J. Scott, F. C. Buckingham, R. E. Gouge MODE RESTXUZNT/DUE DATE Mode 2 - U-l, U-2, U-3

PAGE 99 IIR 2-3-89-001-EVENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLO%ING LARGE LOAD REJECTION ISSUE ¹1 1 HU1KAN PERFORMANCE EVALUATIONSYSTEM (continued)

E. Traixiin uaHQcation content and methods RESOLUTION/ANALYSIS (Matrix Error ¹7) Problems were encountered during reclosure of generator output breakers:

Training for resetting "86 lock-outs" is less-than-adequate: Training on identifying potential consequences of inappropriate actions is less-than-adequate.

CORRECTIVE ACTION Provide training to Operations and PR&C personnel on resetting lockouts (including the consequences of resetting "86" lockouts without proper troubleshooting) per 4XAO-XZZ12, following

'ompletion of Corrective Action 11.B; l.

RESPONSIBLE ORGANIZATION Training, W. F. Fernow MODE RESTRAINT/DUE DATE Mode 2- U-l, U-2, U-3

SUMMARY

The'most notable deficiencies in human performance were attributed L. to Work PraCtices. This Causal Category accounted for 7 of the 9 errors This causal'factor has been identified on previous HPES 'oted.

evaluations. As a corrective action Training Development is in the process of developing a course on Work Practices for Maintenance and personnel. This training is scheduled for an upcoming 'perations requalification cycle.

I

~ '\ ~ 8

PAGE 100 IIR 2-3-89-001

'VENT DATE: MARCH 3, 1989 UNIT 3 REACTOR TRIP FOLLOWING LARGE LOAD REJECTION LIST OF FIGURES FIG. 1) Primary Event and Causal Factors Chart (E&CF Chart)

FIG. 2) SSO Relay E&CF Chart FIG. 3) Steam Bypass Control Erratic Operations E&CF Chart FIG. 4) Fast Bus Transfer E&CF Chart FIG. 5) ADV Operation from the Control Room and Remote Shutdown Panel E&CF Chart FIG. 6) ADVLocal Manual Operation E&CF Chart FIG. 7) MSSS Building Lighting E&CF Chart FIG. 8) - Instrument Air E&CF Chart FIG.') RCS Leakage from RCP 1B E&CF Chart FIG. 10) RCS Leakage from CHV-435 E&CF Chart FIG; 11) Off-Site Dose Calculation Errors E&CF Chart FIG. 12) Energy-Barrier-Target Charts (for Issues Ol - 010)

l

~ Su RY EVENT AND CAUS'AL FACTORS CHART I.I.R. 2-3-89-004 SBCS FAST BUS ADVS FAILED ADVS LOCAL SSORElAY OPERATION TRANSFER DID TOOPERATE, OPERATION OPERATES 0 ~ ERRATIC NOT OCQP REMOTELY DIFFICULT

~ 0 ~ ~

REACTOR LOSS OF DEGRADED HEAT NON.CLASS LARGE LOAD NON-CLASS CONTROL OF TRIP IESFAS REMOVAL BY POWER REJECT ACTUATION POWER HEAT SBCS RESTORED 0102:19 0103:48 0106 01 REMOVAL 0230 0243 MSSS LIGHTING INADEQUATE SEAS ANOMALIES INSTRUMENT AIR SYSTEM DEGfIADED DOTTED LltIES INDICATE FACTORS WHICH RCS LEAKAGE HAVE NOT BEEN VERIFIED INCREASED

///////// ~ ~

ItIDICATES DETAILED EVENTS AND EFfLUENTDOSE CAUSAL 'CHART FOR THAT TOPIC (ATTACHED) CALCULATION ERfKAS

FIGURE 2 ~ .

~

SSO RELAY EVENTS AND CAUSAL FACTORS CHART I.I.R. 2-3-89-001

~ ~

I s SSO'RELAY OPERATED AT PVNGS, UNIT 3 DEVERS LINE DEVERS GENERATOR C PHASE BREAKERS BREAKERS LARGE LOAD FAULT PL 992 & PL 985 & REJECT PL 995 PL 988 OPEN OPEN

FIGURE 3 STEAM BYPASS CONTROL ERRATIC OPERATION EVENTS AND CAUSAL FACTORS CHART I.I.R. 2-3-89-001.

PREVIOUSLY AUTO PERM NO PM TASK IDENTIFIED BY NO REPAIRS TIMER CARD FOR TIMER SPEER INITIATED FAILURE CARD 7/31/88 REACTOR ALL SBCS SBCS VALVES LARGE LOAD SECONDARY TRIP/MSIS VALVES 1001-1008 REJECT PRESSURE ON LOW S/G CLOSE ON CYCLE 0102:19 DECREASES PRESS Q.O. BLOCK 0102:30 0103:48 0103:51

FIGURE 4 FAST BUS TRANSFER EYENTS AND CAUSAL FACTORS CHART I.I.R. 2-3-89-001 GENERATOR AND GRID OUT OF SYNCH GENERATOR COASTS SBCS DOWN ERRATIC OPERATION GENERATOR SYNCH CHECK SEPE RATED WORKS AS FROM GRID HI DESIGNED VOLTS/HERTZ REACTOR FAST BUS NAN-SO I, LARGE I.OAD TRIPI GENERATOR TRANSFER NAN 602 REJECT TURBINE ~ 'TRIP DID NOT DE.ENERGIZE 0102119 TRIP 0106 OCCUR 0106:01 0103:48 0106 FAST BUS GENERATOR TRANSFER SUPPLYING ENABLED HOUSE LOADS

RGURE 5 ADY OPERATtQN FROM THE CONTROL ROOM AND REMOTE SHUTDOWN PANEL EYENT AND CAUSAL FACTORS CHART I.I.R.. 2-3-89-00 t SIMULATOR DOES NOT IDENTIFIED POTENTIAL 1 MODEL PREVIOUS DESGN AOs NOT TRAltlED ACTUALADVs ERRATIC DERCIENCIES IN REtAOTE RESPONSE OPERATION SI IUTOOYINPAIIEL MAIN STEAM OP ERATIOt I ISOLATION SGNAL OPERATORS ADV STEAMING UNABLE TO CONTROL OF DIFFICULTY CONTROL PATH TO OPEN ALL ADVs SHIFTED TO IN REMOTE CONDENSER ADVs FROM VERIFIED TO SHUTDOWN REIJIOTE ISOLATED CONTROL BE IH THE PANEL SHUTDOWN 0103 t48 ROOM COtlTROL OPERATIOtl PANEL 0106:30 ROOM O'111 OPERATOR INSERTS OtlLYADV IXII 15.20'le DEMAND ATTEt.lPTED FRY.'. ~

SIGNAL REt IOTE Sl IIIIDOVetl PN II I

~ I',

ADV(s) ADV ADV 178 ADV UHABLE TO CONTROL UHAOLE TO CONTROL TO BE OPERATED SHIFTED BE OPEttED MANUAL . FROM BACK TO FROM LOCAL CONTROL COtlTROL REMOTE 013T ROOM ROOM PANEL PANEL 0126 0126 0120 A.O. INSERTS 30%

DEMANDSIGNAL

.0

FIGURE 6 ADV LOCAL MANUAL OPERATION EVENTS AND CAUSAL -FACTORS CHART I.I.R. 2-3-89-00 i

'ANUALHANOWHEEL CLEVIS PIN CAME OFF AND WAS DSEIIGAGED CAUSING RE.ATTACHED VALVECLOSURE ADV 185 OPEIIED WITH DIFFICULTIES 0224 HEAT AOs ADV 184 NOT REMOVAL VIA DISPATCHED TO LOCALLY OPERATEO ADVs ACHIEVED OPERATE "ADVs 0137 0125 BARRIERS TO AOV 178 SUCCESS'ASK OPENED SUCCESSFULLY COMPLETICN 0137 ADV 179 BROKEN DURING MANUAL OPERATION EXTREME 0140 ENVIRONMENTAL AOs NOT CONImIONS TRAINED ON.

LOCAL'DV OPERATICN OPERATES LOCAL CIIEATER BAR OPPOSITE OF INSTRUCTIONS i%ED ADV 185 LESS THAN SAFETY ADEQUATE VALVE UFTING INSTRUCTIONS SPECIFIC TO VALVE IZING ADV 179 NOT LOCATIONS NOT NOT UTILIZED

- IDENTIFIEO LEO

FIGURE 7 MSSS BUILDING LIGHTING EVENT AND CAUSAL FACTORS CHART 2-3-89-001 AS BUILT NOT PER DEEDA

. SIAS LOAD 0.90/0.91 SCOSWWeR RESTCRED SUPPLY I P.C.R ChtICELLEO LOSS OF OtILY I LAMP NON.CLASS IN EACH POWER 140'SSS ROOt.1 LIGHTING Eh'IERGENCY LESS THAN NO ESS.

ESSENTIAL EMERGENCY LOSS OF LIGHTS ADEOUATE LIGHTS IN LIGHTS LIGHTS tIORMAL DE-EIIERGIZEI 140'OUTH FOR MANUAL DE.ENERGIZED ENERGIZE LIGHTING ESSENTIAL MSSS ADV 0103:54 0103:54 0106:01 OPERATIONS LIGHTS BUILDING EtIERGIZE 0125 SEVERAL DESIGNED FOR P.M.s WNVEO SHORT LITE LAI.IP BURNED BULB INSTALLED OUI'S AND NT LIGHTS PM TASK

FIGURE 8 INSTRUMENT AIR EVENTS AND CAUSAL FACTORS CHART I.I.R. 2-3-89-001 RX TRIP TURBINETRIP

~ )P

~ ~

I LOSS OF MANY PLANT IA PRESS NON-CLASS VALVES OBSERVED AT POWER OPERATING 64 PSIG INSTRUMENT INSTRUMENT AIR NITROGEN NON.CLASS AIR AIR PRESS COMPRESSORS BACK.UP POWER COMPRESSORS NORMAL DEENERGIZED VALUE OPENS RESTORED RE.EIIERGIZED 0106!01 0108:42 0243 0401

(

~

~ '~

I~

INSTRUMENT INSTRUMENT AIR LOW AIR PRESS PRESSURE DECREASES ALARM

~

>>'lGURE RCS LEAKAGE FROM RCP B 9-EVENTS AND CAUSAL FACTORS 1

~ ~ I N

CHART LLR. 2-3-89-001 RX TRIP BIAS/CIAS 0103:48 LOSS OF NAN S01 AND NAN 602 0106:41 ALL 4 RCPI DE-ENERGIZED 0106:01 NUCLEAR COOLt4G WATER UNAVAILABLE RCP SEAl.

COOLING LOST 0106:41

'H LETDOWN ISOLATED SEAL BLEED.OFF VAI.VES CLOSED SY OPgRATORS-0114 UV.507 BYOR CLOSED BLEED.OFF FLOW RE-ESTABLISHED

'EAKS FAILS TO REMAIN OPERATORS CYCLE CHARGltlG PUMPS RCP SEAL INJECTION STOPPED RCP SEALS EXPOSED TO ELEVATED TEMPERATUTIES >>

0135 ~

j>>

LOCAL l RCP 18 SEAL INSPECTION STAGING OBSERVES PRESSURES APPROX. 1.25 OBSERVED GPM LEAK FROM ABNORI>>IAL RCP 1B SEAL 1900 0250

,0 .0

I FIGURE 10 RCS LEAKAGE FROM CHY 435 EYENTS AND CAUSAL FACTORS CHART I.I.R. 2-3-89-001 NO LETDOWN LETOOYN FLOW THANH ISOLATEO DUE TO REGEN. HEAT SIAS EXCHANGER LOCAL hlEASUAEMENT FOLLOWt&

CHARGING TAANSIENT TEMPERATURE REDUCED CH V435 CHV 435 LEAKAGE FUAMANITED TO EXPERIENCES FUAMANITE LEAKAGE VERIFIED 3.3 REDUCE THEAhlAL MATERIAL INCREASES GPhl LEAKAGE PRIOR TRANSIENT SHRINKS 1900 TO EVENT 0103:54

///////////////////

CHV 435 LEAKAGE8 GPM

,0 . ~

FIGURE 11 OFF-SITE DOSE CALCULATION ERRORS

- EVENT AND CAUSAL FACTORS CHART I.I.R. 2-3-89-QQ1 ACTUAL RELEASE CALCULATED LOSS OF BE!.OW TECH NON CLASS SPEC LIMITS POWER 0106 41 REVIEW OF NOTIFICATION RMSNESOREh'I E PIP-14 OF UNUSUAL COMPUTERS NOT DISCOVERS EVENT AVAILABLE. MATH ERROAS DECLARED

'139 RP RP TECH BEGINS RECOLIMENDS MANUAL CALCULATION OFFSITE DOSE TO EC THAT NO NUE CALCULATION COMPLETED CALC PER PROTECTIVE TERMINATED NECESSARY ERAONEOUSLY EPIP-14 0139 0150 ACTIONS 0252 0139 NECESSARY 0150 DOSE RATE FOR MATH CONSERVATIVE ALL RELEASE ERAOASIN CORRECT DEFAULTVALUES POINTS NOT CONVERSION USED IN ESFAS NON CLASS CALCULATED FACTORS CALCULATION ACTUATIONS POWER RESET RES TCAED AU-139 AND AU 140 READINGS NOT OBTAINED

. ~

>>nleiuEIe Ilnlelciue Un uu R IinlciuEIe>>nicluLlc i nleiuEIc IniclclLlc Tnrec;I;-I Nl N2 ¹3 N4 ¹5 NG ¹7 NOTE: Tl IIS Glenl'I IICAI. IeEI'IeESENTATION OF TIIE ENE leGY-IIAHIeIERTAIeGETANAI YSIS IS ONI Y IN I'ENI)ED To IN I)ICnl'E IVIIICI I Cn'I'EGOIelvs SVElek VVVVCnVVVOle TIIIS ISSUE AND Svillcii WEIIE NOr. Willi I'. rr iS lel COGNirICI) iiin'r'IIILICEnieV MANY i OSSI>>LE I nlc ii I I:.i.

c)I'InieielEleS ANI) SEIeIES COM>>INA'I'IONS OF 11IESE CATEGOIeIES Ov IiniueIEIeS . IT IS No'I"II IE IN'I'LNI'vI'I IIS IeLI'IeESEN'I'n'I'ION Ov'I'I IE ll To sl loiv losE coMIIINAlloNs. E->>-'I'ANAI.Ysls ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001

'ENERGY BnleluER llnlelelEIe I'nlcG Nl N5 EI'~ABGE g~ERGY FAII.ED UNDER SSO RELAY Evni I'leo I ECr SPURIOUS EQUI I'h!iElf ACIUATION I IN l'VITI Sl'E(;II'ICn I'ION

.S ADEQUATE LESS 11 IAN UNDEIe UsED Nol UsED NQTAppLlcnl3I.E ADEQUATE EVALUA11ON SSO IeEIWY I( EQUIPMENT PERFORMANCE nci ilnl'loN

2. PERSONNEL PERFOIeMANCE

3. PROCEDURES

4. mAINING S. DESIGN DFIe INS'I'nl.lA'IION

6. MANAGEMENT

7. OTIIER

Ilht(t(IEl( I.ANGL-r Nt ((4 t(5 NOTE: Tt Its Gt(APlltchL I(Et I(ESENTw>ON oF TIIE ENERGY-ttnluuEI(-Tnl(GLr nNAI Ysls ts oNt.Y tNFENDFD To INnlcATE wt t tel 1 cn'I'I'.GoRIES oF DhtuuEI(s wEI(E EFFEcnvE roR Tilts lssUI; ANI) wtltcll wERE Nor. wt III.t; rr ls I(EC()GNlzt;o aslnr TIIEI(E nt(t;MANYI osstttI.I; I nl(nu.EI.

AND SEIuES Coh1IIINATIONS OF TIIESE CA I'EGORIES OF IthtuuEI(s, I'I'S NOT 'lllE IN'I'LN'I OF TIIIS t(EI'RESENTATION OF I'IIE I'l-'I'NAI.YSIS TO Sl IOW Tl IOSE COMIIINATIONS.

ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001

~

~

(

I ENERGY 13ARIuER tthluuER 13$ Rlu ER, IIARIuEI( I'hl(G I tt I t(2 N3 N5 "I'nt(G EHEBQX FAILED Fhll.ED UNDER +

Evhl Iit'.St'ONI) TO SDCS ERRATIC Ml I'IGATE RESPONSE I)I:SIGN Tl(ANSII'N'I'S E BE AND DESCRI 0 RR REF E ~SSUI~Uhll3ER 2 ADEQUATE I.ESS Tl IAN = UNDER USED. NOT USEI) NOT API'I.ICAIII.E ADEQUATE EVAI.UA'I'ION

-I(L'IAY'I'IMI:R Chl(I)

1. EQUIPMENT PERFORMANCE MAI.VIINC."I'ION

- ovEI(slolrr Glu>ttl S

2. PERSONNEL PERFOIWANCE - S'I'h Glu)ttl' Sl'EI'.I( I'((l(:I h'4

3. - PROCEDURES - I'ht I'I(( i(:I'.l)I Jl(l'.

4. TRAINING

-1(o(rl'.AIISI')I'AI I.UI(E () V

5. DESIGN Tlhl I'.I( Chill)

6. MANAGEMENT

7. OTIIER

llht(I(IER ENERGY 1 Allot-.l Nl N2 N3 Ns NOTE: Tllls GIIAI IIICALREI RESENTATION OF TIIE ENERGY-IIARRIEliThllol;-r ANAI Ysls Is ONI Y INILNDEDTo INDlchlE Wlllcll ch'll'.(:Otnl:.6 OF l3AI<<(it Its WEIIE EFFEClIVE Folt TIIIS ISSUE AND Wl IICII IVEHE NOT. Wl Ill.t'TIS Itt:.COGNI/LD llIATTI IEIZE Al(l: h1ANY I Osslltl.l; I'hlt Xl.t.t;I.

AND SEI(IES COMBINATIONS OF TIIESE CATEGOI<<ES OF BARICILIIS . I'I'S NOT 11IE IN1'EN'I'F flllS REI'l(ESEN'I'ATION OF 1'I II'. I II.T ANALYSISTo Sl IOiV TI IOSE Coh113INATIONS.

ENERGY-BARRIER-TARGET ANALYSIS FOR -IIR 2-3-S9-001 BARI<<ER ' Bht<<<<ER ttntu<<EI< I'Alt(:I ~ I ENERGY

=-

Nl N3 NG EHEBQX FAILED FAIIZD UNDER ~~G IMPROPER EVAI INDICATE SESS h(."I'Uhl.

ALARMS COND I I1ONS Fott OVEIIATOtt DthaNOStS RIE BE AND DESCRI 0 ARRIER FE .NESS IEEUEgfUhtllER I3 ADEQUATE LESS TI IAN UNDER UsED Nor USED NOTAvvt.tchltLE I'hler ADEQUATE EVAI.UATtoN DUhl. INI)lch'I'IONb

l. EQUIPMENT PERFORMANCE ~

RAN Olt NON AIAI(ht

? PERSONNEI PERFORMANCE 3, PROCEDURES

~.ir I Ito(;Ilhht

4. TRAINING I.lht IT SWI I'CI

5. DESIGN DESI(IN lhi'AI.VI'.

6. MANAGEMENT

7. OTIIER

Ohletet Ete hlelel vie Uhlelet e hle elEle 1 th le let EI e Itht el el v.t lh eHIEie ENERGY 1hteGI;I Nt N3 Nn Ns NG N7 NOTE: TlllS Glehl>tllchl. IIEI>leESENTATION OF TIIE ENEleGY IIAHIIIERThleGLI ANhl YSIS IS ONI V IN'I ENDED TO INDICA'I'EWIIICIICh'I I;GOlelt:-6 et t ohtetetvtes wEteE EFFECTIVE Fole1t tls tssUE AND wl l tel t wvteE Nor. wl 111 i;rr Is tel ce)GNtzt:.D n IAT1III;.IeE hteE MANY I os.'it tl I; I ate iu I;t.

AND SEleIES Coh1IIINAI'IONS Ol'Tt IESE CATEGOIeIES OF i)Ate/el!;Ies, IT IS Nol"II IE IN I1'N'I'l:Tl I IS IeL'I'IIL'SEN'I'hI'ION OF I'I II'-II-To Sl IOW 'H IOSE COMIIINATIONS. I'NALYSIS ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001 EVAI'hlle

'ENERGY 13hlelell'.Ie NS Q~E~G UNDEIe ET FBT MAINI'AIN I>owt;le 1'0 DESlGN NAN-sOI /s02 wrl'II A GENE!eh'I'Ote ItleLAREle-ONI.Y llell> Fot.l.owLD IIYA leEACI'Ole '11ell' UT N4 ADEQUATE I.LSS 11 IAN UNDEle USED No I'SED NOT APPLICAIIt.'E At)EQUATE EVAI.UA110N

l. EQUIPMENT PERFORI>fANCE RQaiR

2. PERSONNEL PERFORMANCE

3. PROCEDURES

4. 'HeAINING EVAI.UA'I'E

5. DESIGN CU tel eL'Nr I)V>IGN

6. MANAGEMENT

7. MIIER

IIAItltIElt Atl El he t eI UAI(HII It llhleHIEIt )hi(I IEIt ENERGY TnleGI:I Nl N2 ~ N3= N4 NS NG N7 NOTE: TlllS Glehl'IIICAI.IKI'IKSENI'ATIONOFTIIE ENERGY IIAIeltIEltThltGIT ANAIYSIS IS ONI Y INTENI)EI)TO INI)ICAll'VIIICII Ch'I'EGOIell".S OF 13AltltIEIeS iVEIeh: EFFECllVE FOlt Tl IIS ISSUE ANI) WVI IICI I iVEIKNOT. iVIIII h; CI'S III CO(lNIrrEI) 'I I IAT'II IEIK hltE h'IANY I'Osbllll I; I'hlehl I I I.

AND SENES COhIIIINAI1ONSOl'TIIESE CATEGOIIIES OF Bhltl\IEltS . I'I IS NO'I'llll'.IN Il.N'I'l"IIIIS IIEI'ItESENI'ATIONOF I'I IE TO SIIOiV Tl IOSE COMlllNATIONS.

E.ll:I'NAI.YSIS ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001 ENERGY OAIIItIER DAItltlEIt BAItltIER DhltltIE!t DhltlelEIt DAlelelER

<glthhI. INFO

'll(ANSI'E le

7. OTIIER

IIAHHIER ENERGY rht<GI;-r It I NOTE: Tl IIS GHAPIIICAI. HEI'l<ESENTATION OF TI IE ENERGY-lthtuuEH-TARGE'NAI.YSIS IS ONI.Y Ihrl'ENDED TO INDICA'IE WI IICII Ch'I'I'.GOHIES OF ll DAIuuEI<S WEI<E EFFECllVE FOll IIS ISSUE AND WI IICII WERE NOl'. WI III.E IT IS Ht COGNRLD 11IAT TI IEHE AHE hthNY I'OSSII II.I; I'hi<At.t.t<I.

AND SERIES COMI3INATIONS OF 11IESE CATEGOHIES OF l)AHluEI<S . Il IS NOT 11IE IN'I'ENI OF Tl IIS IKI'HESEN'I'h'IION OF I'I IE E TO Sl IOW Tl IOSE COMIIINATIONS.

Il-'I'NAI.YSIS

~ ~ (

ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001 ENERGY llhHIuEH l)ntuuER I)ARtuER UAHI<IER Iht<ca-,r al a2 a3 a5 KHEBQX FAILED FAILED FAILED FAILED INADEQUATE Al)EQUA I'E MSSS I.ICl I I I'IN G LIGIITING T<) I'LHFOHIII TASKS BER D ESCR1 0 @LHRIER-E FECTIVE ESS 1SSlJ~EUJWBQR a6:

~ '!

ADEQUATE LESS TIIAN UNDER USED NOT USED NOT APPI.IChl3I.E C~

ADEQUATE EYAI.UATION ESSEN'I'IAL I.teil ITtNG

.I. EQUIPMENt'ERFORMANCE EhtLHCI NCY I.lcitITIN hISSS I'ht ll'Atilt;[<4i;

2. PERSONNEL PERFORMANCE

3. PROCEDURES FSAH HI Vll;iV...i

4. TRAINING

6. DESIGN ~ I'U IIII I 4 I; I ll,l)G I.IGi I I I.

MSSS uct i'I'ING

6. MANAGEMENT V. OTIIER mls

,0 0

0 llhltltlEI( 13hltl(IE hi(I(IE t- -

Ilhl(I(IEI( llh t (II;It = Ilhltl(IEI( 3h I (I (I EI t I'hl t G I;"I' ENERGY Nl N2 N3 N4 N5 NG NV

'OTE: Tl I IS Glthl'I I ICAI. IKI'IKSENTATIONOF Tl IE ENEI(GY.llhltl(IEI(TARO EI'NAIYSIS IS ON I Y IN I'EN I)Ef) I'0 INDICATE1VI I ICI I Ch'I EGOI(l I& I <<F I)AI(l(IEI(SKVEIKEFFECllVE FOI( TIIIS ISSUE AND ivlIICII ivEI(E No I'. ivlIII I'T Is I(l Ct<<GNI/I;I) 'I'I Ih I"II IEI(E hl(E MANYI'oSsllll I I'hlt Xf I j I ll AND SEI(IES COh1IIINAl'IONS OF IESE CA1'EGOI(IES OF llhltltlEf(S . I I'S NO I"I'IIE IN'1)EN'I'<<I Tl IIS I(EI'l(LSEN1'ATION OF I'I IE E II T ANALYSISTO Sl 10'l IOSE COMI3INA11ONS.

ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001

'ENERGY IIAI(1(IER llhltl(IEI( Thl(GEI Nl N5 E!K~RG - UNnEI( UNnvlt JARGET FVAI I'El(FOI(M INSIHUMENT AIR AS DI SIGNEI) 'I'0 DEFICIENCIES Ml'I'IGA'IE

'I1(ANSIEN I' E ES S 0 53 -Ni' ADEQUATE LESS 11IAN UNI)ER USEI) NO1'SEI) NOT API<<IACAIII E ADEQUATE EVAl.UAllON

l. EQUIPMENT PERFORhlANCE AS I)EI'I lthIIHh,l) g In ENGINIXI(iNG AC l1ON I I WH

2. PERSONNEl PERFOI(MANCE

3. PROCEDURES

4. %RAINING

- EVAI.UATE I'Elt

5. DESIGN Nl(C GI'.NEI(IC LEIlEI(88 l4

6. MANAGEMENT

7. O11IER

13hlt HI E [4 ENERGY Nl N5 TAIIGEI'OTE:

Tl US GIIAVIIICAL REVRFSENTATtON OF TIIE ENEIIGY-IthtttuEII-TAIIGIWANAIYSIS IS ONI.Y IN rLNI)ED TO INDtCATE Wt ttCti Ch'IEC<)Illa'.'ht:.,

BAIIIIIEIISIVEIIE EFFECllVE Fott TIIIS ISSUE hNt) IVtllcltlVEI<E NOT. KVIIII&IT IS ttt,"COGNIZED TIIA'I"IllEREAISLE MANY I'OS.'IIIII.I'..I'Atoll.l'.Et.

AND SEIVES COM131NATIONS Ol'TIIESE CglEGoltlES OF I3hltttIEI3S, II'IS NOTTIIE IN'I'ENTOI'TltlS ltEI'ttESENTATION Ol 1'flE I To Sl los TI tOSE COM!3INATIONS. 13"I'Nhl.YSIS ENERGY.-BARRIER-TARGET ANALYSIS FOR Illa'. 2-3-89-001 ENERGY BAIIIIIER llhltltlEII 42 N5 Th'I'iGI::L'xbHQEy MEBQX FAILED FAILED FAILED RCS LEAKAGE AND HCV SEht> /

DETECTION TEClt SIEC a IIECOi)EIIY

'l'Et thTIONS BE D C 0 ARRIE EF EC NESS ~SSUE ~m~llLR~ 8 ADEQUATE I.ESS Tl IAN UNDElt USEI) NO I'SEI) NOT AVVI.ICAIILE; hl)EQUA'IE EVALUA'lloN

- SEAI. Ilt.l;lf)olI I. EQUIPMENT PERFORMANCE IIEII~IAltI'.ISI IEf)

- SLAI. Fhll.UHE

2. PERSONNEL PERFORMANCE - I'.O. Itt'.)I'ONSE (SE('Ill t I'. 131.h',El) hl INJ I Ot'I'V/'~t

3. PROCEDURES

4. TRAINING

- CON'I'AINMEN'I'UMI

5. DESIGN I.l;VF.I. 8 ISIS INI)ICsYI'ION

6. MANAGEMENT

7. OTIIER

l3hltltlER ENERGY

¹I TnltGEl NOTE: TIIIS GIthl'IIICALIIEI'IU;SENTATIONOF TIIE ENEIIGY-DARRIElt-TARGETANALYSIS IS ONI.Y INTENDED TO INDICATE IVIIICIICh'I'EGOItlh;b OF ll BAIUIIEIISiVEIIE EFFECTIVE FOIt Tl IIS ISSUE AND iVIIICII iVEItE NOT. iVIIILE IT IS ItECOGNIZED IATTl I ERE AltE hlhNY I'OSSI DI.L I'hlthl.l.EI.

ll

'AND SERIES COMBINATIONS OF TIIESE CATEGOItIES OF l3hltltIERS . IT IS NOT 11IL INTENT OF IIS ItEI'ItESENTATION ll ANALYSISlX) Sl IOiV IOSE COMBINATIONS.

OF TI IE E D-T ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001 ENERGY DAltltIER Dh!UUEIt BAltRIE!t DAIUtlElt ThltG I

¹2 ¹3 ¹4 ¹5 "I'ht FAIIZD -UNDER FAILED . FAII.ED G~

EVAI-OFFSITE I'ItOI'LR DOSE I'ltO'I'l;C."I'IVE CALCULATION A("I'ION ERRORS IIL'COhlhII;Nl)AI'ION

~ 1K@ IW 4 ~

BE D DESC 0 Rl ESS ADEQUATE I.ESS Tl IAN UNDElt USED NOT USEI) NOT APPUCADI.E ADEQUATE EVALUATION

l. EQUIPMEN1'ERFORMANCE

- CAI.CUI.YI'Il)iV

2. PERSONNEI PERFORMANCE EltltOHH

3. PROCEDURES I'IIO('ll) I'JAKIINCE-hlEN IP 'I'0 PHOCI;I) IIIU;

4. TRAINING

5. DESIGN - MESOItl;hI/lthIS I'ONLltSUI'I'I.Y

6. MANAGEMENT

7. OTIIER

UhltltIEIt ENERGY 1'AltG L"I Nl NOTE: TIIIS Glthl'IIICALltEI'IIESENTATION OF TIIE ENEIIGY-13ARRIER-ThltGET ANAI.YSIS IS ONI.Y IN Il;NI)I'.I)'I 0 INDICA'I'E IVIIICII CATEGOIIIES Ov ir xvEreE EFFEcnVE FOR Tilts Issuv. ANI) wllrcll iVEieE NOT. willi.E IS Itvn)GNrzED 'nwr TltvwE hleE MnNY I ossrtlt.t; I'hrehu.EI.

'IAIVIIElts AND SERIES COMIIINATIONSOF TIIESE CAl'EGOIIIES OF BAItltlEltS . IT IS NOT ANALYSISlX) Sl IO'LV Tl IOSE COMBINATIONS.

'lilt'NI'I'Nr'F 11IIS ltLI'ltESEN'I'htlON Ov Tllb I.ll T ENERGY-BARRIER-TARGET ANALYSIS FOR IIR 2-3-89-001 ENERGY DAIIRIER DARltlER IIAIVIIER llhltlVElt 'I'AIIG Nl N2 N3 N5 EHEBQX FAILED FAILED FAILED UNI)EIt t':IttOPER EQUIPMENT EVAI PERFORMANCE ANOMALIES 4

EAU I!eh'I'ION Phl EN'I'l'V BER D ESCRI 0 A RIE BF CTIVE ESS ~SSlrg ~Uh'fi)FR tllO LESS Tl IAN UNI)EIt -

USEI) NOT USED,NOT hl'I'IACAIIIX 1.~ EQUIPMENT PERFORMANCE ADEQUATE EVALUA'llON

'O. I, It).2, IO 3, IO.S It),tL It>.8. It). IO PROCEDURES'DEQUATE

.,2. PERSONNEL PERFORMANCE - IO3)

I ~ ~ ~

3.- - IOS

4. TRAINING

- IO. I I IO. I2

5. DESIGN ~

6. MANAGEMENT

7. OTIIER