ML030760376

NMC Internal Correspondence, Root Cause Evaluation 01-069, Revision 1

ML030760376
Person / Time
Site:	Point Beach
Issue date:	05/15/2002
From:	Hettick D Nuclear Management Co
To:	Warner M Office of Nuclear Reactor Regulation, Nuclear Management Co
References
CR 01-3595, FOIA/PA-2003-0094, NPM 2002-0252 RCE 01-069, Rev 1
Download: ML030760376 (73)

Text

9F7 INTERNAL Committed to Nuclear CORRESPONDENCE NPM 2002-0252 To: M.E. Warner From: D. A. Hettick Date: May 15, 2002

Subject:

ROOT CAUSE EVALUATION 01-069, Revision I Copy To: S. J. Nikolai D. D. Schoon A. J. Cayia T. Coutu - KNPP S. J. Thomas L. J. Armstrong V. A. Kaminskas J. Purcell - KNPP G. A: Corell T. Taylor D. Weaver J. R. Anderson T. Sullivan (P458) J. J. Walsh K. Peveler R. Repshas - KNPP B. Day M. McCarthy S F. Putman - KNPP R. Nicolai - KNPP M. B. Arnold T. L. Zifko R. Wood R. Milner C. Krause J. P. Schroeder T. Staskal L. Peterson L. A. Schofield/JOSRC T. Y Fessler/OSRC R. Flessner File Attached is Root Cause Evaluation (RCE)01-069, Revision 1, for your review. This RCE is an evaluation of Increased CDF in AFW PRA Model Due to Procedural Inadequacies Related to Loss of Instrument Air. The corrective actions associated with this RCE will be tracked under CR 01-3595.

If you have any questions or would like to discuss the report, please call me at Extension 6498.

Approved: .c2' "D.l*.tettick tlz Attachment

NMC >,.

Comminrtted to Nuclear Excellenc Point Beach Nuclear Plant Increased CDF in AFW PRA Model Due to Procedural Inadequacies Related to Loss of Instrument Air RCE 01-069 Revision 1 I (CR 01-3595)

Event Date: November 29, 2001 Report Date: May 14, 2002 I Principal Investigators:

R. Flessner - Team Leader C. Krause J. P. Schroeder T. Staskal R. Wood Approved By:

Issue Manager - Lori Armstrongc Da e I Date A Manager - DOe/l* s'lettick

RCE 01-069 Rev. I IncreasedCDF in AFW PRI4 Model Due to Procedural InadequaciesRelated to Loss of Instrument Air Table of Contents 3

1. Executive Summary ..................................................................................

Event Narrative ..................................................................................... 5 II.

9 III. Extent of Condition Assessment ......................................................

10 IV. Nuclear Safety Significance ............................................................

11 V. Report to External Agencies ........................................................

12 VI. Data Analysis ................................................................................

12 Information & Fact Sources ..................................................................

32 Evaluation M ethodology & Analysis Techniques ...............................

32 Data Analysis Summary .......................................................................

35 Failure M ode Identification .................................................................

36 VII. Root Causes & Contributing Factors ............................................

37 VIII. Corrective Actions ...........................................................................

40 IX. References ....................................................................................

41 X. Attachments ...................................................................................

Team Charter .............................................................. 42 Attachment A:

Event Timeline ........................................................... 43 Attachment B: 49 Attachment C: Why Staircase ..............................................................

Event & Causal Factor Chart ..................................... 51 Attachment D:

2

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of Instrument Air I. Executive Summary (Note: This RCE required revision because additional information and insight were developed during preparations for the NRC regulatory conference held on this issue.)

Purpose:

The purpose of this investigation is to determine the root and contributing causes of why the emergency operating procedural inadequacies existed that contributed to the increased core damage frequency (CDF) for the Auxiliary Feedwater System during a loss of instrument air event, and why these inadequacies where not identified previously.

Event Synopsis:

During a review of the AFW PRA model in June 2001, it was discovered that the AFW recirculation valves were not modeled. Subsequent discussions disclosed that under a loss of instrument air condition (IA), operators might close the AFW discharge valves to stop AFW flow. Because the recirculation valves fail close on loss of IA, these actions could deadhead the AFW pumps and result in pump damage. Initially the procedural concern was directed at AOP-5B, but it was later realized that the AOP was not the only concern. Operator actions could be taken earlier in an accident scenario to control or stop AFW flow, as directed by steps in EOP-0. 1, prior to taking manual actions directed by AOP-5B. PRA modeling of the AFW system continued and on 11/26/01 a factor of 2.3 risk increase in CDF was identified. As discussions with site personnel continued, additional initiating events were identified and on 11/28/01 arevised PRA model was run that changed the risk estimate to a factor of 4 to 5 increase in CDF. Condition report CR 0 1-3595 was initiated at 1445 on 11/29/01 and an NRC event notification was made at 1705 the same day.

==

Conclusions:==

The investigation found that the EOP validation process is the barrier that failed, causing the weakness in EOP-0.1. The EOP validation process failed because it did not evaluate the interaction among design, procedures, and human error timeline analysis. It was only from this integrated perspective that a loss of instrument air causing the recirculation valves to fail closed, combined with a possibility that an operator could close the discharge valve on an AFW pump, and the timing of this action prior to implementation of the abnormal procedure for loss of instrument air (AOP-5B) could the potential be seen to damage multiple AFW pumps. The combination of FMEA, timeline studies, and human error analysis is a recently implemented practice in the industry unique to PRA.

Without the use of these combined analyses, it was not reasonable that previous evaluations would have identified this vulnerability.

3

m IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air Nuclear Safety Significance:

Preliminary PRA results show that the vulnerability described in this report, prior to the procedural changes, was potentially risk significant. Although the initiating event frequencies are low to moderate, when an unrecoverable IA scenario is considered risk becomes significant due to the consequences of a total loss of all AFW pumps requiring feed and bleed without the pressurizer PORVs. The risk results are highly dependant upon human interactions. PBNP operators are trained on AFW system operations and have experience with degraded IA scenarios. Because of this training and experience, it is reasonable to assume that operators would have successfully handled this combination of conditions in the unlikely event that it would have occurred.

Root Cause:

The root cause of the EOP procedural weaknesses was the failure of the original EOP validation process barrier to identify that specific operator directions were needed to ensure the operator would properly control or stop AFW flow under a loss of instrument air condition. This barrier failed because the analytical tools needed to identify this vulnerability did not exist at that time. This resulted in a misalignment between plant design and procedural guidance.

Significant contributing causes to this condition continuing to exist were:

" The original PRA model fault trees evaluated system performance primarily on functions described in design documents and only considered operator actions taken to mitigate a failure

"* Previous evaluations focused on delivery of the minimum required AFW flow for providing decay heat removal Corrective Action Synopsis:

"* EOP-0, EOP 0.1 and ECA-0.0 revised to address AFW control under loss of IA

"* Back-up pneumatic supply added to AFW recirculation valves

"* AOP-5B revised to incorporate back-up pneumatic supply for recirculation valves

• EOP validation process revised to include PRA

• Simulator enhanced to model potential for AFW pump failure on loss of IA

• Evaluated EOP steps to ensure successful implementation on loss of IA

• Completed detailed evaluation of PRA model for the four top risk-significant systems

• Validated PRA assumptions on next two risk-significant systems (these six systems comprise 80% of CDF risk)

• Continuing detailed evaluations of PRA model for other risk-significant systems

• Enhancing CDF risk reduction by incorporating PRA human error reduction methods into operator training and operating procedures 4

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev I 1 InadequaciesRelated to Loss of Inst rumeni Air 1!. Event Narrative the AFW portion of the PRA In June, 2001 the PRA group was reviewing and revising the minimum flow recirculation valves model. During this review it was discovered that modes and effects analysis was were not modeled within the PRA. Therefore, a failure A discussion was held with past performed to determine potential failure modes.

within the AOPs and EOPs. It operations personnel about how the system was operated was then determined that upon a complete loss of instrument air, the operators may use MOV or the flow control valve.

the EOPs and stop AFW flow by closing the discharge a loss of instrument air, the AFW However, since the recirculation valve fails closed on This issue was discussed with a pump would not have adequate recirculation flow.

the AFW pumps could be damaged in design engineer who informed the PRA group that flow.

a short period of time without adequate recirculation Training personnel who reviewed the This issue was then discussed with Operations be. The operator actions were also EOPs and discussed what operator actions would were that upon a complete loss confirmed with an Operations crew. The actions assumed Reactor Trip or Safety Injection, and of instrument air, entry would be made into EOP-0, in these procedures would ensure that then into EOP-0.1, Reactor Trip Response. Steps 1, if S/G level is high the operator is at least one AFW pump was available. In EOP-0.

the discharge valve, the AFW directed to STOP flow. If flow were stopped, by closing by the recirculation valve failing pump would fail due to lack of minimum flow caused could be repeated on additional closed. The potential exists that this same evolution both units in a similar configuration, the AFW pumps. Since this is a dual unit event with same problem could also happen on the second unit.

Air, had a specific note to gag open the It was noted that AOP-5B, Loss of Instrument well into the procedure and timing recirculation valves, but the information was located closing the discharge valves. PRA showed that it would not be adequate to preclude the potential to be risk significant even personnel understood that this failure mode had since the PRA model development was not though the actual significance was not known on 7/6/01 to document this problem yet completed. PRA personnel initiated CR 01-2278 steps addressing the need to gag the and identify potential corrective actions to place of AOP-5B. It was assumed that the recirculation valves open earlier in the sequence the timing of the action could be AOP was sufficient to address the concern, but improved to ensure that the action would be successful.

Procedure group with a An action item was created on 7/10/01 for the Operations

24) to a more prominent position in the recommendation to move the step (AOP-5B step action item priority was set at 4 and procedure and consider using a foldout page. The were held between PRA and the due date was established as 8/21/01. Discussions PRA group evaluation to determine the Operations personnel and it was expected that a by 8/20/01. Initial Operations review of significance of the issue would be completed priority to restore instrument air, AOP-5B indicated that the procedure was laid out in a The evaluation of the risk significance of which is the correct response for that procedure.

dependent on quantifying the entire PRA the as found configuration of the procedure is 5

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air model. This was not completed until October, due to the complexity of developing a complete two-unit model. The original model used a single unit and simplified common systems. The PRA group informed Operations on 8/20/01 that the evaluation was not completed as expected and additional time was required to evaluate the actual significance and the type of action that should be done. At that time modifications and procedural changes were being considered.

The PRA group completed some preliminary modeling on 10/19/01 that indicated the potential for a high risk was involved and informed Operations that the AFW pump recirculation valves should be procedurally addressed. Based on further discussion, it was decided that a change to the Alarm Response Procedure for instrument air low header pressure (ARP COI A 1-9) could address the concern. The PRA group was to submit a procedure feedback form for the desired change. The original action item was closed on 11/14101 and a new action item was created on 11/14/01 to track the changes to the ARP and assigned to Operations. Operations discussed the request with PRA personnel and gave the new action item a priority of 3 with a due date of 12/26/01, based on expected completion of the PRA model and Safety Monitor update in December.

During that discussion some concerns were raised by Operations about the adequacy of procedural changes to address the issue. Specifically, the concern was that the ARP may not be the most effective way of protecting the AFW pumps during high activity in the Control Room, i.e., the loss of instrument air may not take priority and the ARP may not be referred to.

Additional discussions took place between Operations, PRA and a design engineer concerning the appropriate corrective actions and what risk might be involved if the procedural remedy was not completed or was inadequate. On Monday, 11/26101, the PRA modeling adjustments were completed and a factor of 2.3 risk increase in Core Damage Frequency (CDF) was identified, which is considered high. Additional discussions took place between Engineering and Operations to determine further actions that may be appropriate.

A meeting between Operations and Engineering was held at 1300 on Wednesday, 11/28/01, to discuss significance and actions. During the discussion it was discovered that the loss of instrument air was more than just a random loss, a loss of offsite power (LOOP) or other events could also initiate the event. A re-evaluation of risk including the LOOP event resulted in an estimated factor of risk increase of 4 to 5 in CDF.

Operability was also discussed. It was concluded that there was no operability concern because no equipment degradation, failure, or non-conformance had been identified.

Regardless, the level of concern was great enough that further prompt actions were felt to be justified. The Design Engineering Manager briefed the Operations Manager on the situation later that afternoon. The Operations Manager also updated the Plant Manager on the situation.

On Thursday morning, 11/29/01, the Operations Manager briefed the NRC Resident Inspectors on the issue and informed them that we were evaluating this apparent vulnerability and the risk significance. Operations decided that use of temporary 6

Increased CDF in AFIV PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of InstrumentAir information tags and briefing of all watch standers would be an important step to reduce risk; an evaluation of possible procedure enhancements was also initiated. At 1000, PRA personnel briefed the STA and Shift Manager on the issue and discussed potential wording for temporary placards to be placed on the control panels.

At 1100, PRA personnel discussed potential reportability concerns with Licensing. It was not clear if this issue was reportable because it involved a procedure and was not an equipment issue - additional discussions were needed. At 1130, PRA personnel briefed the NRC Resident Inspector on the issues and answered questions regarding risk impact and human error probabilities. During the afternoon, Licensing and Engineering personnel evaluated the reportability aspect further. It was concluded that the conservative decision would be to report the issue, even though a specific reporting criteria could not be identified. At 1445, PRA personnel initiated Condition Report 01 3595 and brought it to the Work Control Center for SRO screening at 1538. The Operations Manager took part in discussions involving operability and the need for an Operability Determination (OD). Since the issue identified in CR 01-3.595 did not affect equipment, the decision was made that an OD was not required; however, the details of those discussions were not captured in either the CR or the screening comments. The SRO screening was completed at 1553 with the event determined to be reportable as a procedural inadequacy and not requiring an OD.

At 1520, the oncoming crew was briefed on the concerns of this potential event and temporary information tags were placed adjacent to the controls for 1/2P-29 and P-38 A/B that provided a reminder of the minimum flow requirements for each AFW pump.

At 1700, the Operations Manager provided the Plant Manager with an update on the issue. At 1705, Event Notification EN 38525 was made to the NRC via the ENS phone.

(See Section V. for details)

On Friday morning, 11/30/01, the Licensing Manager received a phone call from the acting NRC-NRR Project Manager for Point Beach, concerning confusion over the event notification. A return conference call was made with Engineering personnel to address NRR questions. A decision was made to provide a supplemental event notification providing additional details. The Operations Manager had additional conversation with the NRC Resident Inspectors and concluded that to formally document the operability of the AFW system, an OD would be initiated to capture the discussions held during the previous 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Operations requested that Engineering provide an OD and informed the Shift Manager that it was expected to be completed that afternoon. At Noon, the Operations Manager met again with the NRC Resident Inspectors and their supervisor to address NRC concerns regarding AFW operability prior to 11/29/01 and in its current configuration. The Plant Manager and Operations Manager had a conference call with NRC Region 11I to discuss operability of the AFW system.

At 1400, a simulator scenario was run to obtain information on plant response to a loss of offsite power coincident with a rapid loss of instrument air pressure. Additional scenarios were run on 11/30 and 12/I.

7

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev I InadequaciesRelated to Loss of Instrument Air At 1645, temporary procedure changes were completed for EOP-0 and EOP-0. 1 to reflect the guidance provided earlier to operators via the temporary information tags.

At 1700, the Plant Manager was informed that a 5 person NRC incident investigation team would arrive on 12/3/01. At 1746, a supplemental event notification was made to the NRC to clarify the discussion on the potential for an AFW system failure as described in the original event notification (EN 38525).

At 1755, Engineering completed Revision 0 of the OD that concluded that the AFW system was Operable but Non-Conforming. This was based in part on a statement in the FSAR that "each pump has an AOV controlled recirculation line back to the condensate storage tanks to ensure minimum flow to dissipate pump heat." The compensatory actions already in effect were listed in the OD as required actions. The Plant Manager and Operations Manager reviewed the OD content and then briefed the Senior NRC Resident Inspector. The OD was then brought to the Control Room and accepted at 2015. On Friday evening, just-in-time (JJT) training was provided to the swing shift crew on the simulator on this event; JIT was also provided to the mid-shift crew on the simulator prior to assuming the watch.

On Saturday, 12/1/01, at 0720 JIT was provided to the oncoming dayshift crew on the simulator prior to assuming the watch. A staff meeting was held from 0930 to 1200 to prepare for the NRC inspection team. A revised OD was prepared at 1500 to expand the discussion on AFW pump motor duty cycles. The Control Room accepted it at 1515.

On Monday, 12/3/01, CR 01-3595 was screened and assigned to Engineering to perform an apparent cause evaluation. Another meeting was held from 1000 to 1200 in preparation for the NRC inspection team. At that meeting it was decided that a root cause evaluation would be a more appropriate response to this event. The Plant Manager approved the RCE Charter on 12/4/01.

The NRC Inspection Team arrived onsite on 12/3/01 and conducted a technical debrief on 12/7101. A preliminary exit meeting was held on 12/13/01.

An expert on Human Error Probabilities was brought onsite on 12/4/01 to help quantify the risks associated with the procedural weaknesses that were identified. His evaluation estimated that there was about a 50% chance that the operator would shut the discharge valve and fail to recognize that the minimum flow recirculation valve did not open when flow was stopped as S/G levels rose above 65% on the narrow range.

On 12/4/01, CR 01-3633 was initiated by Engineering on the ability of the Motor Driven Auxiliary Feedwater Pumps (MDAFWP) to respond to an Appendix R fire coincident with a loss of offsite power and instrument air because of a lack of documentation related to the potential for closure of the recirculation valves due to loss of instrument air. CR 0 1-3648 was initiated by Engineering on 12/5/01 on the same issue when four specific fire zones were identified as having the potential to cause an AFW pump auto-start 8

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. 1 I InadequaciesRelated to Loss of Instrument Air coincident with discharge and recirculation valve closure, resulting in pump damage. An OD was completed for CR 01-3468 on 12/7/01 that concluded the MDAFW Pumps were Operable but Non-Conforming, with the required compensatory measures of performing hourly fire rounds in the specified fire zones. An event notification on this issue was made at 1926 on 12/05/01 (EN #38541)

Permanent revisions to EOP-0 and EOP-0.1 were implemented on 12/14/01. As PRA reviews continued, it was recognized that the closure of the AFW recirc valves could occur after an operator had already taken action to put the pumps in the recirculation mode. Additional changes were made to those procedures and ECA-0.0 on 12/20/01 to address this concern. As additional information becomes available, procedure improvements are often implemented to continually improve their quality.

i11. Extent of Condition Assessment The root cause of this event is attributed to a weakness in the original EOP validation process where the effects of a loss of instrument air were not adequately evaluated. This occurred because the validation process did not evaluate the interaction between design, procedures and human error timeline analysis. It was only from this perspective that a loss of IA causing the recirc valves to fail closed combined with a possibility that an operator could close a discharge valve on an AFW pump and the timing of this action prior to implementation of the abnormal procedure for loss of IA (AOP-5B) could the potential be seen to damage multiple AFW pumps. This validation process was believed to be consistent with industry practices.

Because of this event, the previously held belief that AOP-5B, Loss of Instrument Air, adequately directed the required operator actions was found to be faulty because actions were required while in an EOP, prior to performing AOP subordinate actions. This event identified a specific concern with AFW control, but there may be other operator actions that'are unique to a loss of instrument air condition that were not adequately considered in the EOPs. A review of EOP steps was performed to ensure that the stated operator actions could be performed under a loss of instrument air condition.

The original PRA model fault trees evaluated system performance primarily on functions described in design documents and did not adequately consider human actions. The current PRA model review uses a methodology that integrates system performance with potential human actions to obtain a spectrum of plant responses. This more rigorous approach should identify any other assumptions used in risk-significant systems that have not adequately considered human actions and any risk-significant vulnerabilities in the emergency operating procedures. The four highest risk-significant systems have had a detailed review of the PRA model completed already. The assumptions for operator actions for the next two highest risk-significant systems have also been validated. These six systems comprise 80% of the CDF risk. The detailed review of the PRA model for the remaining risk-significant systems is continuing.

9

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air The lack of integration of human error reduction methods into operations training and emergency procedure development processes may allow situations to exist where PRA risk reduction has not been optimized. Procedures and training associated with high-risk human error events will be reviewed against human error reduction methods to ensure that reasonable risk reduction has been achieved.

IV. Nuclear Safety Significance Any complete loss of IA for a significant time is expected to result in a reactor trip and an AFW start signal due to a loss of normal feedwater (the normal feed water regulating valves fail closed on loss of air). Under this postulated condition, all components of the AFWS are now and continue to be fully capable of performing their design functions supporting automatic starting and supplying sufficient flow to the steam generators to mitigate any transient or accident by removal of decay heat. It is the continued function of the AFWS, in response to directed operator actions to control AFWS flow, and the lack of specific guidance contained within the EOPs regarding a loss of IA, that is the issue identified in this report.

A PRA assessment of the possible failure modes and effects associated with an IA failure identified a previously unrecognized vulnerability. This failure would have been caused by a combination of a design limitation, a specific sequence of postulated operator actions, and a lack of clear guidance within the EOPs. This combination could result in failure of one or more of the AFW pumps due to aggressive AFW flow reduction (as may be expected in response to a steam generator overfill or RCS over cooling) after automatic system start and flow had been established. The likelihood of success or failure in the postulated scenario is highly dependent upon plant transient response (which may vary with the nature of the initiating event, initial power levels, etc.) and operator response. Operator response is highly dependent upon prior training, procedural usage, system knowledge and awareness, experience, and other human effectiveness (HE) factors. It should be noted that a control board alarm is provided (Instrument Air Header Pressure Low) to alert the operator to the existence of an initiating condition for this event and that established plant procedures direct the restoration of IA (both Emergency Operating Procedures and Abnormal Operating Procedures), and the manual gagging open of the minimum flow recirculation valves in the event that IA cannot be promptly restored (AOP-5B). PBNP has experienced partial losses of IA, including one event involving the loss of all off-site power and another involving a low IA header pressure alarm following a reactor trip. In each of these cases the operators demonstrated the ability to cope with the loss of IA casualty and recover IA header pressure before it had an adverse affect on plant equipment or response.

Preliminary PRA results show that the vulnerability described in this report, prior to the procedural changes, was potentially risk significant. Although the initiating event frequencies are low to moderate, the unrecoverable IA scenario was risk significant due to the consequences of a total loss of all AFW pumps requiring feed and bleed 10

IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of InstrumentAir without the pressurizer PORVs. The risk results are highly dependant upon human interactions. PBNP operators are trained on AFW system operations and have experience with degraded IA scenarios. Because of this training and experience, it is reasonable to assume that operators would have successfully handled this combination of conditions in the unlikely event that it would have occurred.

Although the AFWS met, and continues to meet, all of its design and licensing requirements, the postulated initiating event of a loss of IA, in conjunction with a misaligned procedure, had the potential to affect redundant trains of the AFWS, a safety related system. Since it could be postulated that the same operator action could have impacted all the AFWS pumps, the result could be the complete loss of the AFWS safety related function. Accordingly, this event has also been identified as a possible safety system functional failure (SSFF).

V. Report to External Agencies Condition Report 01-3595 was initially brought to the PBNP Work Control Center for an SRO screening at 1538 on November 29,2001. During this screening, a determination was made that this event should conservatively be reported to the NRC in accordance with 10 CFR 50.72(b)(3)(v) as a condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to:...(D) Mitigate the consequences of an accident. This is an eight-hour non-emergency notification. During the discussion of reportability it was noted that 10 CFR 50.72 Paragraph (b)(3)(vi) clarifies paragraph (b)(3)(v) by noting that, "Events covered in paragraph (b)(3)(v) of this section may include one or more procedural errors, equipment failures, and/or discovery of design, analysis, fabrication, construction, and/or procedural inadequacies." The last of these items appeared as though it may be applicable in this situation. The following elements also entered into the notification determination:

• NUREG-1022 notes that the level of judgment for reporting an event is a reasonable expectation that the event or condition could lead to preventing fulfillment of a safety function. The intent of these criteria is to capture those events regardless of whether there was an actual demand.

SExample (20) in NUREG-1022 Page 64 directs that system interactions that are found as a result of ongoing routine activities may be reportable.

• When in doubt concerning issues of reportability, it is our policy (consistent with the directions in NUREG-1022) to make the report.

The NRC notification was made using the Emergency Notification System (ENS) telephone at 1705 on November 2 9th. Event number EN 38525 was assigned to this notification.

On the morning of November 3 0 'h, as a courtesy, the PBNP acting Project Manager at NRC-NRR was telephoned to advise him of the event notification. He had several 11

IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. ] I InadequaciesRelated to Loss of Instrument Air questions that were answered in a follow-up call later in the morning. At 1746 on the November 30, 2001, the ENS event notification was supplemented to further clarify discussion of the specific failures postulated and to reiterate that the potential failure would involve only the AFWS pump recirculation valves.

of this A Licensee Event Report (LER 266/2001-005-00) was submitted within 60 days event as required by 10 CFR 50.73.

V1. Data Analysis Information & Fact Sources Document Review Results Modifications

- M-623 / 624 - TDAFP Alternate Bearing Cooling Supply, issued 9/1179

Description:

In response to an NRC Evaluation of the AFW system, this MR of provided a cooling water supply to the TDAFP bearing coolers that is independent AC power. The supply is taken from the diesel powered Fire Water system.

are Evaluation: The MR enabled the TDAFP to cope with a SBO. Since the TDAFPs of the SBO, the only pumps available for decay heat removal during the first hour This operation of the pumps at low flows requiring recirculation flow is not probable.

modification was performed prior to the original EOP-0.1 being issued in 1985.

EOP Therefore, it is not reasonable that this modification would have identified the procedural vulnerability.

2/1180 IC-274 - Modify Logic To Keep Recirculation Valves Open, issued (Canceled 8/32/82)

Description:

Modify the control scheme of the recirculation valves to keep valves first off check normally open. The reason for this change was to provide a path for the from lifting valve leakage back to the CST. This change would prevent the leakage since it was only solving a the pump suction relief. The modification was canceled intended to symptom of the real problem; check valve leakage. The modification still have the recirculation valves fail to the shut position.

associated with Evaluation: The modification was attempting to resolve symptoms check valve leakage. The modification would not have permitted a continuous to the recirculation path. This modification was originated and cancelled prior not reasonable that reviews original EOP-0.1 being issued in 1985. Therefore, it is associated with this modification would have identified the EOP procedural vulnerability.

12

Increased CDF in AFW PRA Model Due to Procedural RtC 01-0o9 R~ev. I InadequaciesRelated to Loss of InstrunientAir MR 83-104 -"AFW System Discharge MOV Controls, issued 8/1/83 automatic

Description:

The MDAFP discharge valves were modified to provide MDAFPs.

actuation of the valves similar to the automatic starting logic for the provided to the Evaluation: The MR was a response to NUREG-0737 to ensure AF is on loss of air or S/Gs without operator action. The recirculation valves either failed in compliance shut as flow to the S/G increased therefore, these valves were already and the design with the NUREG. This MR deals with eliminating an operator action action is taken limitation of the recirculation valves is not introduced until an operator prior to the (i.e. throttling AF discharge flows). This modification was performed it not is original symptom-based EOP-0.1 being issued in 1985. Therefore, reasonable that this MR would identify the EOP procedural vulnerability.

MR 88-099 - AFW Pump Mini-Recirculation Line Improvements, issued 7/7/88 the recirculation

Description:

In response to NRC IE Bulletin 88-04 and GL 89-04, instability.

line flows were increased to prevent pump degradation due to hydraulic MR increased this to The minimum pump flow pror to this MR was 30 gpm. The The MR minimum flow to 70 gpm for the MDAFPs and 100 gpm for the TDAFPs.

did not change the operation of the recirculation valves.

for adequate Evaluation: PBNP did a design review of the recirc capacity needed was initiated to increase long-term protection of the AFW pumps. This modification did not alter the the recirc flow capacity to the required levels. The modification were reviewed to operating modes of the recirc valves. System operating procedures very specific design the extent that this design change impacted them. Therefore, this change and review would not identify the EOP procedural vulnerability.

- MR 92-091/0921093 - IST Testability of AF Recirculation Line AOfs, issued 6/19/92 bypass valves were

Description:

In order to simplify stroke testing of these AOVs, installed around the control solenoid.

need to bypass the Evaluation: The MR was small scope focusing only on the IST Program had solenoid to allow stroke testing of the valve. At this time, the for these valves. The already identified the shut position as the safety related position scope of this MR was not an opportunity to identify the issue.

- MR 97-038*A/B - MDAFP Discharge Pressure Control Valve Backup Nitrogen Supply and Cable Separation, issued 4/15197 AOVs (common electrical

Description:

The MR prevented redundant failures of the MR 97-038*B fault) and pump runout due to loss of IA (Ref. LER 97-014-00).

with the discharge provided physical separation for electrical cables associated control pressure control valves (AF-4012 and AF-4019) and their associated as a backup pneumatic supply.

components. MR 97-038*A installed nitrogen bottles of the of the functions The design description for MR 97-038*B states that one S/Gs to cool the associated pump discharge AOVs is to allow enough flow to the 13

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of InstrumentAir during a scenario when pump recirculation is required and the associated recirculauon valve fails closed.

Evaluation: The intent of the MR was to prevent pump runout due to a failed open discharge AOV as a result of a loss of instrument air and low SIG pressures. It appears the focus of the MR was to ensure control capability of the discharge pressure control valves. The MR does recognize that the discharge AOVs are needed to provide pump cooling flow if the recirculation valves fail shut. This appears to support the idea that the flow to the SIG is the safety related function and failure of the recirculation valves is acceptable. System operating procedures were reviewed for the impact of this design change. Since the recirc valves were not being modified, it was not reasonable to review procedures associated with those valves. The failure modes and effects analysis of the system performed on this modification did not consider failures caused by operator actions. The ability to throttle the pump discharge flow during a loss of instrument air provides another opportunity (in addition to throttling the MOV) for operator action to cause pump damage.

Procedures AOP-SB, Loss of Instrument Air: This AOP was first issued on 5/2186. The procedure contained an "immediate action - manual" step (step 6.0) emphasizing the understanding that AOVs may not function depending on IA header pressure and referred the operator to Appendix A for individual system information. Section R of Appendix A was for Auxiliary Feed, and listed the AFW pump recirculation valves as failing shut with a corresponding note on manual gag override. The additional information in that section included monitoring of AFW pumps for sufficient flow to prevent overheating due to no "minirecirc", and to use the manual gag on the "minirecirc" valve to provide maximum recirculation unless continuous feed was verified through each AFW pump. The procedure content remained essentially the same until Revision I1 was issued on 9/26/97, which moved time critical actions from the appendices into the main body of the procedure. At that time a specific step (step 21) was added for control of AFW flow. A note was placed before that step informing the operator "the manual gag on each AFW pump mini4ecirc valve must be used to provide minimum recirc flow if continuous flow through the pump can NOT be verified." The current procedure content is equivalent.

Evaluation: The AOP contained sufficient information identifying the correct failure mode of the AFW pump recirculation valves on loss of IA, the required manual actions, the concern with pump overheating, and the need to monitor pump flow. The content of the note that directed the operator to continuously monitor pump flow and use the manual gag if flow could not be verified, met the requirements of OM 4.3.1 for note content. OM 4.3.1 allows notes to advise on actions to be taken in the event of changing plant conditions (see discussion on OM 4.3.1 below).

EOP-0.1, Reactor Trip Response: Emergency Operating Procedures, specifically EOP-0. 1, is the PBNP procedure that would be used in the event of concern; EOP-0. I 14

RCE 01-069 Rev. I Increased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air WOG ERG has ever addressed is based on a WOG ERG. Neither EOP-0. 1 nor the EOP-0.1, in one step (step 3),

the function of the AFW mini recirc flow valves.

bypass valves for feed flow directs the operator to use main feedwater regulator AFW use is directed, and has a control. As a response-not-obtained (RNO) action, recirc valves are not included. A substep to "verify AFW alignment". The mini pumps will trip due to over current NOTE containing the flow rate at which AFW flow control step. In another step induced by pump runout precedes the feedwater level but is not provided details on (step 4), the operator is directed to stabilize S/G specified is to "stop feed flow to that how to accomplish the task. The RNO action being provided by main feedwater S/G." This action applies whether feed flow is There is also reference to (via the bypass) or by the auxiliary feedwater pumps.

RCS temperature.

controlling feed flow in step 1 related to maintaining control have basically existed since The steps on S/G level stabilization and feed flow as a result of NUREG-0737.

the symptom-based EOPs were created in July 1985, instrument air on the mini recirc They have never addressed the impact of loss of pump runout) was introduced in valves. The effect of excessive AFW flow (i.e.,

about 1995.

not address loss of instrument air, nor The WOG ERGs for Reactor Trip Response do flow capability. The WOG do they specifically address AFW pump mini recirc be addressed by the owner. The considers such aspects to be plant specifics, to contains little information on what original WOG developmental guidance from 1984 or how. This trend continues through (plant specific) systems should be addressed, that plant specific electrical loads 1997, Rev 1C, which does generically identify should be a plant specific (which covers one major cause of IA loss, compressors) addressed. The WOG has always list. AFW and S/G level control specifics are not needed in EOPs and the Deviation and recognized that plant specific information is to manage such information.

Background Document concepts were provided 1 the importance of AFW in At various times throughout the history of EOP-0.

has been recognized at PBNP. For general (but not mini recirc flow in particular) actuation was step number I of EOP-0.1.

example in Rev 7, 10/11/91, checking AFW 1995. Loss of IA due to electrical bus AFW pump runout concerns were added in For example in Rev 11, 11/22/94 (prior availability was addressed similarly to AFW.

train specific equipment operation) to the development of AOP-18A and -18B for Electrical Loads, which included Appendix A to EOP-0. 1 contained a list of Priority when AOP-18A & -18B were created.

an IA compressor. Appendix A was deleted on ERG guidance. The ERGs Evaluation: PBNP EOP-0.1 is based appropriately need to be included in EOPs and consider that plant specific information may Deviation document the same (Background and provides means and mechanisms to by the ERG (V&V) process described documents). The verification and validation to identify plant specific needs to be procedure development process is intended not include operator guidance in included in the plant specific EOPs. PBNP did a loss of IA condition.

EOP-0. I on AFW minimum recirc flow under 15

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rei. 1I InadequaciesRelated to Loss of Instrument Air OM 4.3.1, AOP and EOP Writers' Guide: The Writers' Guide contains the usage rules for notes and cautions that specify (in part):

"* A note is used to present advisory or administrative information necessary to support performance of the subsequent step(s).

"* Each document should provide enough information to accomplish the purpose of the document without relying on information contained in notes or cautions.

"* Notes and cautions should be declarative statements of fact and not commands or action statements unless they are advising on actions to be taken in the event of changing plant conditions.

The references listed in OM 4.3.1 were reviewed with the following results:

"* NUREG-0899, Guidelines for the Preparation of Emergency Operating Procedures - 8/82: Note statements provide operators with supplemental information concerning specific steps or sequences of steps in the EOP. These statements should provide operators with enough information, and be located so as to ensure that they can easily relate the note to the step or steps to which it applies. Because they are supplemental, notes should not direct operators to perform actions. (p24)

"* NUREG-1358, Lessons Learned From the Special Inspection Program for Emergency Operating Procedures - 4/89: In many cases action statement were found embedded in notes and cautions. Again, this increases the chance that the step will be overlooked and that an error will occur. (p4) Cautions and notes are not intended to direct operator action, but rather to warn of possible consequences or to provide supplemental information to the procedure steps. Inclusion of actions in a caution or note can be disruptive and confusing to an operator. More importantly, the action could be entirely overlooked if embedded in a caution or note. Any cautions or notes containing operator actions, including conditional actions or transitions, should be restructured so as to provide an action step plus a caution or note. (pC-3)

"* NUREG-1358, Supplement 1, Lessons Learned From the Special Inspection Program for Emergency Operating Procedures - 10/92: Cautions and notes:

notes (1) provide only supplemental information, and (2) no actions included.

(P16)

"* NUREG/CR-2005, Checklist for Evaluating Emergency Operating Procedures in Nuclear Power Plants - 4/83: Do explanatory notes avoid the use of action statements? (Statements directing personnel to perform actions must not be imbedded in explanatory notes.) (p7)

"* PBNP Procedures Writers' Guide - 11/27/00: Cautions and notes shall NOT direct or infer actions. All required actions shall be stated in action steps. (p50)

This procedure is not applicable to the AOPs or EOPs.

", WOG ERG Writers Guide - 7/1/87: Because the present action step wording is reduced to the minimum essential, certain additional information is sometimes desired, or necessary, and cannot be merely included in a background document.

This non-action information is presented as either a NOTE or a CAUTION. (p2 2 )

NOTE is used to present advisory or administrative information necessary to 16

Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of InstrumentAir support the following action instruction. A CAUTION or NOTE may also be used to provide a contingent transition based on changes in plant condition. As a general rule, a CAUTION or NOTE will not contain an instruction/operator action. However, passive action statements in CAUTIONS or NOTES, which typically contain the words should, may or must, may be appropriate under certain conditions. An example is when continuous monitoring of a specific plant condition and an associated action is required.

Evaluation: OM 4.3.1 guidance on the content of a note is consistent with the WOG ERG Writers Guide, but contradictory to all of the other references cited. Some statements within the OM contradict others; specifically, the statement that "Each document should provide enough information to accomplish the purpose of the document without relying on information contained in notes or cautions" contradicts the intent of "unless they are advising on actions to be taken in the event of changing plant conditions."

Training Continuing Training: The overall content of the continuing training program is determined based on a two-year cycle. Presently the 2001/2002 LOR (license operator requalification) Long Range Training Plan is in effect. The Long Range Plan concept is very organized and structured with respect to content of the topics to be covered; it has been used since the mid-1990s. The content of the Long Range Training Plan is based, in part, on PRA information and includes a focus on systems with high safety significance. Prior to the Long Range Plan implementation, the content of LOR training was determined in a much less rigorous manner and on a much shorter time frame, typically on a 6 week-to-6 week cycle. Content was based on needs suggested by students, operations management and instructors plus inputs based on current events (such as design change implementation, procedure changes, plant and industry events).

The 2001/2002 plan contains a number of topics pertinent to the issue of concern. The tasks for Loss of Instrument Air and Loss of Offsite Power were covered as well as a system review of Auxiliary Feedwater. The training devices used by instructors to cover the topics are LPs (Lesson Plans) and SGs (simulator guides). Both these devices present information in outline form, containing topical areas to be covered.

The LPs are primarily oriented for classroom environment, whereas SGs are targeted for the simulator, mostly the instructor/ simulator operator. LPs clearly identify references and materials to be used as handouts. Typical support documents are drawings, procedures and OE documents. The LPs used in continuing training are the same LPs used for initial training. Training personnel indicated that LPs and SGs are reviewed prior to use and, to the best ability of the individual trainer, are updated to be current.

- Initial Operator (CO and SRO) Training: The highest-level document in Initial Training is the Program requirements (TRPR). They are position based. For example 17

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I Inadequacies Related to Loss of InstrunentAir TRPR 18 is Control Operator Trainee. The TRPRs are primarily administrative documents rather than technical. The TRPRs do identify the Training Courses (TRCRs) that comprise the Program. The TRCRs are a little more technical than needs Programs in that they identify some general areas of knowledge that the trainee to cover. For example, under TRPR 18, two of the courses are TRCR 52, Secondary LPs are Systems and TRCR 55, Integrated Operations. The TRCRs identify LPs. The to the event the same as those used in continuing training. Some of the LPs specific includes are LP 0169 AFW system, LP 0405 Reactor Trip or SI Response (which EOP 0.1), LP 0338 Instrument and Service Air (which includes AOP-5B) and LP 2439 Secondary Coolant System Malfunctions (AFW is one of those).

feedwater and Evaluation: LPs contain enough specific information about auxiliary effects.

instrument air systems to accurately describe system operations, causes and evolutions.

Training documents do not contain extremely specific details on specific as directed in For example, the specific method for controlling steam generator level not covered nor is EOP-0. I in concert with compounding events such as loss of IA, is air.

the need to locally gag an AFW pump mini recirc valve upon loss of instrument make changes in Instructors review material to be taught in advance and are able to change course content in order to add information, including current events and to training appear to areas of emphasis. The Simulator Guide topics used in continuing be marginally related to the topic area they are listed under. PRA and human used as performance information is not included in LPs. PRA and CDF values are training.

input to select the content of the Long Range Training Plan for continuing Other Documents

- DBD-01, Auxiliary Feedwater System Design Basis Document: Revision 0 of Control DBD-01 was issued on 4/4/94. In Section 4.8, AFW Pump Recire Flow that "These valves Valves, there was a statement under Safety-Related Functions from shall open automatically and remain open to provide a recirculation flowpath insufficient AFW pump discharge to the CST when flow in the AFW discharge line is The DBD also to prevent pump damage." The reference cited was MR 88-099.

diversion of stated "These valves shall close automatically to prevent the unnecessary adequate pump discharge AFW pump discharge during high-flow conditions where requirements flow is removing pump heat." Section 4.8.4 addressed these competing and a less significant function stating "Since this valve has a safety function to close, the valve fail to open (long-term pump protection) it is most reliable therefore to have section also (upon loss of power or instrument air) to the closed position. This valve discussed a potential worst-case flow condition with both the recirculation (single active closed (due to loss of IA) and the associated discharge MOV closed and licensing basis.

failure), but concluded that this was outside the system design but was not This worst-case concern was based on NUREG-0800 assumptions, NUREG-0800 into its considered applicable since PBNP had not incorporated licensing basis.

18

RCE 01-069 Rev. I Increased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of InstrumentAir major changes included Revision I of DbBD-01 was issued on 3/31/00. One of the valves for AFW was "Deleted safety-related function to OPEN for mini-recirculation remained in the DBD.

pumps." The worst-case flow condition discussion function for the Evaluation: The basis for including an OPEN safety-related the modification that recirculation valves in Revision 0 was cited as MR 88-099, protection. A review of increased the recirculation flow orifice size for AFW pump declaring a safety-related the modification paperwork did not identify any statement validation documentation function for the valves to OPEN. A review of the DBD recirculation valve position.

indicated that in-service testing of the valves checked was added to the IST Testing of the recirculation valves in the OPEN direction below.)

Program in 1991. (See discussion of 1ST Program function. This appears to be Revision 1 of DBD-01 deleted this OPEN safety-related later). Testing of the valves a result of actions coming from CR 97-3363 (discussed on 9/30/98, also as a result in the OPEN direction was deleted from the IST Program this function to the DBD of CR 97-3363. Overall, the basis for adding and deleting was not well documented or justified.

EOPs issued in 1985 were EOP Verification and Validation (V&V): The original an approved procedure with a verified by a multi-disciplined verification team using generated over 2500 detailed checklist of attributes to be evaluated. That effort 40 team meetings over a period discrepancy sheets and involved a series of more than for EOP-0. 1 did not raise any of several years. The discrepancy sheets generated stopping feed flow to a S/G if a concerns with the step for controlling feed flow or level increase above the desired value occurred.

the basic version of the ERGs at The validation process involved a WOG review of I ERGs at the Seabrook the Calloway simulator in 1982 and on the Revision specific procedures were taken to simulator in 1983. Early drafts of some of the plant generated many suggested the Zion simulator in March and April of 1983, which through the previously described procedure changes. The procedures were then put were used by operating crews at verification process. Following this, the procedures spent a week mitigating accidents the Kewaunee simulator (8/84-11/84). Each crew the actions to control feed using the procedures. No concerns were raised regarding the desired level range. Finally, a flow or stop feed flow if SIG level increased above was expanded to provide another portion of the detailed control room design review mock-up of the PBNP control room validation of the EOPs. A full size photographic typical 5 or 6) were evaluated was created and fourteen scenarios (increased from the performed walkthroughs in an attempt to ensure that every EOP was used. Operators also videotaped for later review, and of the EOPs during these scenarios, which were was validated using a Reactor then interviewed for their comments (1985). EOP-0.1 instrument air). Again, no Trip without SI scenario (without a concurrent loss of feed flow or stop feed flow if concerns were raised regarding the actions to control SIG level increased above the desired level range.

19

Increased CDF in AFW PRA Model Due to Proccdural RCE 01-00*L Key. I I InadequaciesRelated to Loss of Instrument Air I (GL 82-33)

The EOP V&V process was also part of a NUREG-0737 Supplement submitted to the commitment. The EOP procedure generation package (PGP) was that found the PGP NRC on 6/1/84. The NRC responded with a draft SER on 5/7/87 back to the to be unacceptable. The PBNP revisions to the draft SER were submitted NRC issued the NRC on 11/10/87, addressing each of the identified concerns. The identified final SER on 4/9/90 that contained additional programmatic improvements NRC by the staff. The SER transmittal letter also referred to the June 1989 the results of that Inspection of the EOPs and recommended that PBNP consider both the next major inspection and the SER discussion and utilize them as appropriate in are OM revision of the EOPs. Current procedures governing the EOP V&V process 4.3.2, EOP Verification Procedure, and OM 4.3.3, EOP Validation.

the WOG ERGs, Evaluation: During the development of the PBNP EOPs from reference plant information was to be included to address differences between the of those used by WOG and the Point Beach plant. Following development ensure the adequacy procedures, verification and validation reviews were applied to for usability by of those procedures. Validation is the process of evaluating the EOPs with plant hardware and the operators and operational correctness (e.g., compatibility a loss of IA condition.

control board layout). EOP-0. 1 was operationally incorrect for and implementation Therefore, it was the validation step in the EOP development of instrument air process that failed. The need to evaluate EOP-0.1 using a loss not evaluate condition was not recognized because the validation process did procedures, design and human error/timeline analysis concurrently.

Operator Actions in EPRI Report TR-100259, An Approach to the Analysis of by the PRA group in Probabilistic Risk Assessment - 6/92: This document is used identifies attributes of evaluating human interactions for the probability of an error. It that the mechanisms certain failure mechanisms that influence the overall probability Relevant Step in will contribute to a human interaction (HI). One mechanism, in a decision Procedure Missed, has four attributes that are considered and evaluated tree:

. Obvious vs. Hidden: Is the relevant instruction a separate, stand-alone numbered step, in which case the upper branch is followed, or is it "hidden" in some way in a that makes it easy to overlook, e.g., one of several statements in a paragraph, note or caution, or on the back of a page?

more a Single vs. Multiple: At the time of the HI, is the procedure reader using one column of a than one text procedure or concurrently following more than flowchart procedure?

, Graphically Distinct: Is the step governing the HI in some way more conspicuous than surrounding steps?

• Place Keeping Aids: Are place keeping aids, such as checking off or marking by all crews?

through completed steps and marking pending steps used a procedure step A hidden step had a 10% probability of being missed, whereas 0.1%, a reduction by exhibiting the best of all four attributes had a probability of only 20

RCE 01-069 Rev. 1 Increased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air step is only 1.3%, which is a factor of 100. The worst probability for an obvious about a factor of 8 lower than a hidden step.

line features FSAR: The FSAR did not include a description of AFW recirculation update involved the addition of until updates were made in 6/97 and 6/98. The 1997 via the recirculation line to the a paragraph describing the diversion of AFW flow This was an original design feature CST for a 3-minute period following pump start.

of the AFW system. The 1998 that had never been included in the FSAR description the FSAR Review and Upgrade update was an extensive change resulting from the AFW system and its licensing Project that provided a more detailed description of had an AOV controlled basis. This change added the wording that each pump tanks to ensure minimum flow to recirculation line back to the condensate storage time period for AFW flow dissipate pump heat. This change also revised the 45 seconds.

diversion during pump start from 3 minutes to original IPE for Point Individual Plant Evaluation, Revision 0 dated 6/30193: The and procedures as of 9/5/90. Many Beach was developed from a snapshot of the plant model were based on design basis of the success criteria for systems in the IPE PRA for Auxiliary Feedwater, it was assumptions. In the original PRA system notebook failed closed on a loss of recognized that the minimum recirculation flow valves the PRA model as a failure mode instrument air. However, this was not included in failing to open did not result in for AFW because it was assumed that these valves of the notebook states:

pump failure. Assumption 22 in Section 4.6.7.1 lines back to the CSTs.

The discharge lines of the AFW pumps have recirculation closed on loss of power or These lines are normally isolated by AOVs that fail upon a pump start and when instrument air. Although they receive open signals does not fail the AFW pump.

pump flow is low, it is assumed that failure to open is assumed to fail the Failure of one of these AOVs in a full open position associated AFW train due to diversion of pump flow.

of flow was mentioned briefly in The potential to damage the AFW pumps with lack the following discussion is the notebook. In Section 4.6.2.2 on Support Systems, found under the "Instrument Air" heading:

AFW pumps (AF-4002)

The mini-recirculation valves on both the turbine-driven fail shut on a loss of and the motor-driven AFW pumps (AF-4007 and AF-4014) pumps on low flow instrument air. This could cause overheating of these conditions with no recirculation flow available.

However, controlling (reducing)

These two sections seem to contradict each other.

transient so there was plenty of time AFW flow was assumed to take place later in the This was based on decay heat for the operators to perform this action correctly.

emphasis on ensuring that enough removal curves. Again, there appeared to be an it was not recognized how early in the flow was available in the transient initially and prevent overfilling the Steam event that AFW flow needed to be reduced to 13 where operator actions to control Generators. This is evidenced by Assumption 21

lnct eased CDF in AFlW PRA Model Due'to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air AFW flow later in the transient are discussed. No mention is made of ensuring a minimum flow path is available:

Operator actions to control AFW flow later in an accident sequence are not explicitly modeled in the AFW system fault trees. Operator actions are necessary to prevent the AFW system from overfilling the steam generators as their pressures decrease and AFW flow likewise increases. This was not modeled since there is a long time available and the function would be alarmed.

of In addition, the operator would have to successfully supply an alternate source of the AFW pumps (not automatic) and then forget to control water to the suction flow or check steam generator level.

It seems from these statements in the notebook that some injection flow was always assumed to be required. The need for the operator to shut off flow to the Steam was Generators entirely from one or more AFW pumps at some time in the event apparently not considered.

are In Section 4.6.4.2 of the notebook, initiating event impacts on the system valves for discussed. Under the "Loss of Instrument Air" heading, only the discharge valves the motor driven pumps are considered. The closure of the mini-recirculation for the AFW pumps was not documented as a possible effect of the Loss of Instrument Air event:

flow rates A loss of instrument air will degrade the operators' ability to throttle the AFW pumps.

of that portion of the AFW system associated with the motor-driven to 200 The discharge pressure control valves, which are intended to limit flow and would fail open on a loss of gpm per pump, (AF-4012, 4019) are air-operated use the instrument air. Under this condition the operator is directed to turbine-driven pump to supply feed per AOP-5B, "Loss of Instrument Air" per 01 (Reference 4.6-12) or use the local gag to control AF-4012 and AF-4019 62A, "Motor-Driven Auxiliary Feedwater System (P-38A&B)".

for the The notebook also contains a discussion of potential common cause failures pumps minimum AFW system. This review did not identify the closure of each mechanism.

recirculation valve on a loss of instrument air as a potential failure of these valves to open However, this is consistent with the assumption that failure does not fail the AFW pumps.

of the plant Updates to the original IPE PRA model (1990) were based on snapshots (due to the long taken in 1993 and again in 1996, and implemented a few years later updates was to time required to perform the model update). The focus of these changes that incorporate new plant-specific failure data and to incorporate model completed this year is reflected plant modifications. The PRA model update being examined from the first time since the original IPE effort that critical systems were are captured. This was the ground up in a detailed review to ensure all failure modes trees. Adding accomplished in part by use of detailed failure modes and effects fault make the model more this detail was considered to be necessary at this point to 22

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I hiadequaciesRelated to Loss of InstrurnentAir flexible for Fisk-informed applications. It was the use of this approach that identified the concern with operator actions to control AFW flow.

IST Program: In December, 1990 the 3rd interval program (Revision 0) was implemented. There is a line item in the general valve section that states "Due to isolation of S/G by EOPs, it may be necessary for an operating pumps recirc path to be available." The testing to verify the open function was not included in the tabular section of the IST program that identified the actual testing to be done. A valve program relief request (VRR-28) was added to the IST Program under Revision I on 5/28/91 that described the recirculation valves function to be "These valves open to ensure minimum recirculation flow from the pumps to prevent pump damage." A cold shutdown test frequency was being sought.

The NRC issued a Technical Evaluation Report (TER) on 4/17/92 that denied the relief request because the valves had a safety function in the closed position and noted that the recirculation valves were not tested by the IST Program in the open position.

The TER referenced the VRR-28 function statement and went on to state "The program should be revised to address these valves' safety function in the open direction." PBNP responded to the NRC on 7/30/92 to clarify that the valves could not be stroked except by use of hand wheels until modifications were made that allowed manual stroking using air. The response also stated "Since the AF pumps are capable of delivering feedwater at any steam generator pressure, the minimum flow valves are not required to open to protect the AF pumps under any anticipated accident conditions. The valves will, nevertheless, be stroke time tested in the open direction, as well as in the shut direction, once the modification to permit stroke time testing is completed." A follow-up letter dated 3/2/93, informed the NRC that the modifications would be completed by the completion of the spring 1993 refueling outage and VRR-28 relief request was being withdrawn. Revision 3 to the IST Program was implemented on 3/30/93 deleting relief request VRR-28.

On 10/15/97, CR 97-3363 raised a question about a discrepancy between the open function testing of the AFW recirculation line check valves (not in the IST Program) compared to the recirculation flow control valves (in the IST Program). The evaluation of this concern concluded on 2/5/98 that there was no safety related function for the recirculation valves or check valves to open, and the IST Program would be revised. Revision 5 of the IST Program was issued on 9/30/98 and deleted the open function testing of the recirculation flow control valves.

Interview Results involved in Personnel Statements: Written statements were obtained from key personnel the the evolution of this issue covering the period of initial discovery to its reporting to incorporated into the NRC. The information derived from those statements has been timeline included in Attachment B and involved the following personnel:

- PRA Engineer Design Engineer 23

IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of instrument Air

- Design Engiheering Manager

- Regulatory Compliance Engineer

- AFW System Engineer

- Operations Manager

- PRA Supervisor Interviews: Interviews were conducted with the following individuals to obtain additional information:

PRA Engineer: An interview was conducted with the PRA Engineer that identified the concern with operator actions to control AFW flow. That interview identified the following points:

- The PRA group reviewed the effect of the EOP change made (addition of foldout page information) but did not make recommendations on the best method of accomplishing the incorporation of that information. Use of the foldout page resulted in a reduction of the Human Error Probability (HEP) from 0.5 to 0.05.

Use of a foldout page is treated as a continuous step with some additional credit for other control room personnel and training; it does not have as high of a CDF reduction factor as a specific check.

Credit was given in the recovery factor calculated for use of a procedure reader; it was treated the same as an extra crew.

- The PRA Engineer received information in June or July 2001 that operators stop AFW flow by using valves versus stopping pumps. The information was obtained during discussions with an operating crew. This information was verified later via operator interviews conducted by the HEP expert.

- The PRA group provides feedback to Training, via informal communications, on high-risk accident sequences, but not on specific procedure steps that have high HEPs.

EOP Coordinator: An interview was conducted with the EOP coordinator and identified the following points:

- The direct work item system is a process that allows procedure changes to be made.

Direct work items are changes that are issued by the WOG after review by the appropriate WOG subcommittee. Essentially they are revisions to the ERGs. Any member of WOG can initiate a possible direct work item but it does not become one until issued by the WOG.

- Changes to the EOPs can also be initiated internally without going through the WOG using the procedure feedback process. When this mechanism is used, the EOP Coordinator and an Operations Procedure Writer evaluate the request to decide if it should be processed, and the EOP set changed. There is no procedurally defined process that describes the evaluation methodology. There does not seem to be any 24

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I I hIadequaciesRelated to Loss of Instrument Air guidance on determining specific technical content of a change if it is outside the ERG.

- Foldout page content is expected to be memorized by the operator. Foldout page information is intended to trigger operator memory. The addition of foldout page information to EOP-0 and EOP-0.1 is applicable at all times to continually control AFW flow correctly; this includes transition out of EOP-0 and EOP-0.1. The EOP Coordinator did not consider the PRA value of foldout page use versus other methods of incorporating the desired actions into procedures when the decision to use a foldout page was made.

- No formal V&V was performed on the foldout page change to the EOPs; a serial review was performed.

- The EOP Coordinator believes that Operations generally keeps Training informed of training needs.

- The EOP Coordinator thinks the changes made to the EOPs are done to streamline the procedures.

Other Information During preparations for the NRC Regulatory Conference held on this issue, discussions with the participants identified the following:

- The timing of operator actions for S/G level control assumed in the original IPE was based on decay heat curves. Diversion of flow (by gagging open the recirc valves) was not envisioned earlier in the accident sceriario. The timing of operator actions to throttle AFW flow to a level requiring a recirculation flow path due to SIG overfilling or RCS overcooling concurrent with a loss of IA was not recognized.

- The EOP procedure weakness was very difficult to identify. It was only from an integrated perspective of evaluating AFW system design, procedural guidance, and F1IEA, overlaid with human error probability analysis and timeline studies that the issue could be identified.

- The PBNP instrument air system has multiple cross-ties between units and redundancies that requires a dual unit event to cause a complete loss of IA. The EOPs are single-unit emergency procedures and do not consider dual unit casualties.

- During a SBO event, based on the required condition for decay heat (100% power for 100 days), the need to throttled AFW flow to levels requiring the recirculation valves to open would not occur for about 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />, well after the time that IA is restored.

Therefore, the review of this event would not identify the EOP procedural vulnerability.

25

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rcv. I InadequaciesRelated to Loss of Instrument Air Industry and Stafion Operating Experience Internal Operating Experience CR 97-3363, IST Program Design Basis for AFW Minimum Flow Recirculation Valves: This CR was initiated on 10/15/97 to address a concern with a conflict between the IST Program and the AFW DBD. The IST Program stated that the AFW recirculation line check valves did not have an active safety function to open and that the minimum flow recirculation lines were not needed since there was always adequate flow to the SIGs under accident conditions. This conflicted with the AFW DBD that did not address the check valves, but had an open safety function for the recirculation valves. The IST Program tests the recirculation valves in the open and close directions. The DBD group performed an evaluation on 2/5/98 that concluded the check valves have no safety related function in either direction and that the recirculation valves only have a safety related function in the closed direction. The basis stated that the main safety related function of AF was to supply water to the S/Gs and that flow to the S/Gs was the most important flow path to'inaintain. The mini-recirc line was considered a diversion path, and since the AF system was capable of a cold start, a recirculation path was not necessary. The potential to deadhead a pump was considered, but establishment of a flow path through the discharge lines was used to eliminate the concern and the mini-recirc path was deemed to not be needed for pump protection. The evaluation noted that DBD-01 (Rev. 0) was being revised to reflect that there was no open safety function. The evaluation went on further to consider an AFW pump scenario where the associated discharge MOV failed to open or the pressure control valve inadvertently closed along with the recirculation path being blocked. In this event, the recirculation line would be required to prevent pump destruction, but the emergency function to feed the SIGs is defeated anyway. This active single component failure scenario would only apply to one pump, so it would be acceptable and recirculation flow for AFW pumps was not a required safety related function.

QCR 99-0115, Code Testing Conflict With the Aux Feedwater Mini-Flow Recirc Check Valves: This CR was initiated on 5/24/99 and addressed a concern that conflicting information existed about the safety related function of check valves AF 115 and AF-1 17 to OPEN compared to the AFW recirculation valves that have a safety related function to CLOSE. Further, the 1ST Program did not include these check valves. An evaluation performed on 5127/99 concluded that the concern identified was in error and had already been addressed by CR 97-3363. Additional evaluation on 6/15/99 concluded that some clarification to the IST Program documentation was needed to address how AFW single failure affected the decision on testing. A new action item was generated to revise the IST Program documentation and closed on 6/19/00 with issuance of Revision 4 of Appendix A of the IST Background Document.

- RCE 98-148, P-38A AFW Pump Recirc Valve Found Failed Shut, dated 1/29/99:

This RCE documented an event where an operator was in the process of starting an 26

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air AFW pump and noted that the recirculation line valve did not open as expected and then quickly secured the pump. This event showed that operators monitor recirculation valve position during AFW system manipulations.

R,1PO Operating Experience SEN 174 - Loss of Nonvital Bus Causes Dual Unit SCRAM and Degraded Auxiliary Feedwater System, dated 11/10/97

Description:

At the McGuire plant, a loss of non-safety related 120V AC instrument and control power caused both units to SCRAM. Also, the recirculation valves for all 3 U-I AF pumps failed shut. The control board indication for these valves was also lost. As water level in the S/G was recovered, operators eventually shut the pump discharge valves. The pumps were operated for 20 to 60 minutes with their discharge and recirculation valves shut. Valve leakage was adequate to prevent pump damage.

Evaluation: This event is very similar to our case. Our evaluation of the SEN focused only on the power supply failure. AF pump operation without recirculation flow was discussed in the SEN and one question raised was "what procedures require operators to ensure that adequate pump flow is maintained?" This question was not addressed in the evaluation of the SEN. CARB requested that this SEN be reviewed again.

CA004279 was initiated to track this evaluation.

SOER 88 Instrument Air System Failures, dated 5/18/88

Description:

This document provides a review and evaluation of industry events associated with failures and degradations of instrument air systems.

Recommendations 1, and 2 from this SOER are relevant to this event.

Recommendation 1 (Operations) was to provide procedures to assist operators in the identification, control, and recovery from partial or total loss of instrument air events.

A list of attributes that the operating, abnormal, and emergency procedures should provide included (in part) the following: identification of critical components operated by instrument air and the positions in which they fail, expected system and plant responses to a loss of 1A and the consequences of these responses, actions to take if critical components do not fail in their intended position, and manual actions the operator should be expected to take to respond to a loss of IA event. The PBNP response was that AOP-5B, Loss of Instrument Air, contained the necessary instructions and information to assist operators in the identification, control, and recovery from partial or total loss of IA, and fully satisfied that recommendation. At that time, AOP-5iB had an appendix for the AFW systemi that identified the recirculation valves as failing shut and requiring a manual gag override to open.

Recommendation 2 (Training) from the SOER was to provide classroom and simulator training on loss of IA events to operators. The training was to provide the bases for such things as failure modes of critical components and expected operator actions, so that the operators would understand the major concerns involved in a loss of IA event. The PBNP response was to initiate Training Needs Analysis (TNA) 88-27

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. 1 I InadequaciesRelated to Loss of Instrument Air 0425 for the'PBNP Training group to evaluate. The result was that classroom training on loss of IA was included in cycle 89-8 of AO, RO, SRO, and DTA Air continuing training. LP 1782, Revision 0 dated 11/1/89, Instrument and Service a section was developed and approved to address this need. That lesson plan included that lists concerns with a loss of IA that focused on four areas: heat removal, auxiliary feedwater, inadvertent safety injection, and containment isolation. For AFW, the and lesson plan identified that on the electric driven AFW pumps, the PCV fails open, training on loss on all AFW pumps the recirculation valves fail closed. No simulator of IA of IA was provided because PBNP was using the KNPP simulator then and loss could not be adequately modeled on it.

Evaluation: The PBNP response to recommendation 1 addressed the need for information in abnormal operating procedures, but did not directly address operating and emergency procedures. The reliance on AOPs for addressing specific plant conditions and using EOPs for general response and mitigation probably influenced that the AFW the scope of the review. The classroom training specifically identified pump recirculation valves failed close on loss of IA, but did not identify concerns with pump damage or the need to gag open the valves, as dictated by AOP-5B.

valves However, there was a notation relating to the SI recirculation/test line isolation and reference to an OPS failing shut causing pump overheating in a few minutes training Special Order 85-05 that had the valves currently gagged open. Simulator address the was not performed due to modeling difficulties. Overall, the response did air. The issue of the AFW recirculation valves failing closed on loss of instrument air was reliance on AOP-5B for operator actions resulting from a loss of instrument reasonable based on what was known at that time.

OE 10727 - PRA Risk Insight to Improve Operator Actions, dated 9111/00 NRC

Description:

This document describes an event at another utility where the risk insight to improve the timeliness identified that they did not effectively use PRA in loss and reliability of mitigating operator actions prior to an actual event resulting of all RCP seal cooling to 2 RCPs. For this event, it was determined that PRA core damage.

updates were not being used to train operators on plant vulnerabilities to Evaluation: At PBNP, procedure ESG 5.1, PRA Maintenance and Update vulnerabilities Guidelines, requires the generation of a condition report whenever new in the ESG that addressed who are identified. However, there were no provisions was issued on should be trained. In response to OE 10727, a revision to ESG 5.1 updates and 12/19100 that specified what groups should receive training on PRA newly identified vulnerabilities.

Other Operating Experience Zion Station LER 90-002, 1A Auxiliary Feedwater Pump Cavitation, dated pump was 2/15/90: This LER describes an event where the 1A turbine-driven AFW of run in a deadheaded condition resulting in pump damage. Due to a combination with both management error and procedural deficiency, the AFW pump was operated 28

Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of Instrument Air the discharge 'valve and recirculation valve shut for a period of about eight minutes until an operator stationed locally at the AFW pump noted an abnormal temperature rise on the pump's thrust bearing, water hammer sounds, and that the oil cooling water relief valve had lifted. This event demonstrates that pump damage can occur in a short period of time when operating a pump in a deadheaded condition. The pump impeller was found to be damaged and required replacement.

NRC Generic Communications Generic Letter 81-14, Seismic Qualification of AFW Systems, dated 2/10/81 The purpose of this GL was for licensees to determine the extent to which their AFWS are seismically qualified and to walk-down the non-seismic portions of the system and identify deficiencies. Our original response was submitted on July 16, 1981, in which we concluded that the PBNP AFWS is adequately protected for a seismic event. No specific mention was made of the AFWS recirculation valves or piping. In a response to the NRC follow-up request for additional information dated May 4, 1982, we specifically noted that the recirculation piping connections to the seismic AFWS piping were inspected and that the recirculation valves close upon receipt of a pump discharge flow signal. The NRC's Technical Evaluation Report (TER) of November 12, 1982, concluded that the PBNP AFWS did not provide reasonable assurance to perform its SR function following a seismic event. In our response dated December 15, 1982, we stated that the recirculation valves fail closed and the discharge AOVs fail open and concluded that the instrument air system that powers these valves is not required for AFWS functioning. Because of the questions concerning the recirculation piping not being well supported, we committed in this letter to independently support each air operated recirculation valve. Finally, in our letter dated April 26, 1985, we responded to the NRC request for comments on their revised TER. In the TER the staff postulated a failure during a seismic event of the non-seismic AFWS piping or a failure of the pump recirculation valves to shut following the switchover of the AFWS supply to service water. In our response we stated that under either condition the operator are trained to recognize off normal condition and that adequate time existed for manual operator actions.

Evaluation: PBNP performed a design review that evaluated the seismic adequacy of foundations, supports and structures associated with the AFWS. Review of system operating procedures was not a reasonable response to the Generic Letter. Therefore, this very specific design review would not identify the EOP procedural vulnerability.

Information Notice 87-28, Air Supply Problems at US Light Water Reactors, dated 6/22/87 The internal evaluation of this IN consisted of a review of all systems that perform safety functions and contain air operated valve operators, for the effect that the loss of air would have on those safety functions. The failure positions of the AFWS valves are identified. The concern for pump damage or failure due to less than minimum pump flow with the recirculation valves failing shut is also discussed. However, the focus of the evaluation was on demonstrating that the AFWS pumps would always 29

Increased CDF in AFWV PRA Model Due to Procedural RCE 01-069 Rev. 1 I InadequaciesRelated to Loss of Instrument Air feed the S/Gs with sufficiently high flow to protect the pump This was documented in calculation N 87-041. At that time the discharge AOV for the electric AFW pumps failed open on loss of air; therefore, there was no identified concern with the recirculation valves failing shut.

Evaluation: PBNP verified the performance of safety-related functions with a loss of IA and that the AFW recirc valves must fail closed to assure the AFW safety-related function of providing flow to the S/Gs. It was also verified that adequate procedures existed (AOP-5B) to address a loss of IA, including the manual actions needed to gag open the recirc valves. Since PRA tools were not available yet, it is not reasonable that the EOP procedural vulnerability would have been identified.

NRC Bulletin No. 88-04, Potential Safety-Related Pump Loss, dated 5/5/88 This bulletin requested licensees to investigate and correct as appropriate two mini flow design concerns. The first concern was the potential for deadheading one or more pumps that have a common mini-flow line. The second concern is whether or not the installed mini-flow capacity is adequate to prevent damage to safety related pumps. In a response dated June 28, 1988, we acknowledged that each of the pumps in the AFWS have their own recirculation lines with an AOV isolation valve and an orifice upstream of the common return line to the CST. We discussed the logic of the recirculation valves to open or shut dependent on AFWS forward flow but did not address the potential to lose recirculation on an instrument air failure. We also acknowledged that the flow orifice for the pumps will need to be replaced with higher flow orifices to ensure sufficient flow for indefinite pump cooling via the recirculation lines.

Evaluation: PBNP did a design review of the recirc capacity needed for adequate long-term protection of the AFW pumps. Modifications were initiated to increase the recirc flow capacity to the required levels. Review of system operating procedures was not a reasonable response to this Bulletin. Therefore, this very specific design review would not identify the EOP procedural vulnerability.

10 CFR 50.63 Loss of All Alternating Current Power, effective 7/21/88 The NRC amended its regulations at 10 CFR 50.63 to require all nuclear power plants to be capable of withstanding and recovering from a station blackout (SBO) of a specified duration. Our initial response to this regulation, which addressed the appropriate guidance from Reg. Guide 1.155 and NUMARC 87-00 was submitted on April 17, 1989. In that response we stated that no air-operated valves are required to operate to cope with a SBO for one hour. We also completed an analysis on condensate inventory necessary to cope with the one hour SBO. We concluded that we had sufficient CST inventory, along with the initial S/G fluid inventory to maintain S/G decay heat removal capability. Clearly, for a SBO, only the TDAFW pumps would be available. The concern appeared to be assurance that sufficient water would be fed to the S/Gs until AC power was restored and AFW could be shifted to the safety related service water supply. The first NRC SER on SBO was dated October 3, 1990. The NRC agreed, based on our statement, "that the 30

hIcreased CDF in AFWV PRA Model Due to Procedural RCE 01-069 Rev. 1I InadequaciesRelated to Loss of Instrument Air compressed air is not needed to cope with an SBO for one hour and, after I hour, the Alternate AC power source will supply the compressed air." The Technical Evaluation Report (TER Page 16) also stated agreement that operation of the AFWS is independent of AC and IA for one hour. Indeed the concern identified in the Technical Evaluation Report was that the minimum volume of 10,000 gallons in the CST per unit, was insufficient and ultimately we had to revise our Technical Specificatiops to change that minimum CST volume to 13,000 gallons.

Evaluation: During a SBO event, only the TDAFW pumps are available (one per unit). The conditions for this event assume a decay heat load based on 100 days of operation at 100% power. Based on the high decay heat load and one TDAFW pump, it is not credible to stop or reduce AFW flow to a point where pump damage is incurred in the first hour. Therefore, it is not reasonable that the EOP vulnerability would have been found during reviews associated with a SBO event.

Generic Letter 88-14, Instrument Air Supply System Problems Affecting Safety Related Equipment, dated 8/8/88 In a February 20, 1989, response to this GL we stated that all safety related pneumatic equipment at PBNP is designed to fail to a safe condition with the safety function being tested in the PBNP IST Program. The AFWS discharge AOVs were specifically discussed and the concern expressed that the fail open position could potentially lead to over feeding of the S/Gs. There was additional correspondence to the NRC on July 27, 1989, in the form of a supplemental response concerning the potential problem with the discharge valves failing open. We also responded to an inspection report dated January 16, 1991, in which the NRC determined that PBNP had not fully complied with statements in our original GL response regarding testing of safety related AOVs. In this response we clarified that safety related valves with "4passive" functions (do not perform a mechanical motion during the course of accomplishing a system safety function) were excluded from IST fail safe testing.

We also noted that since the 1989 submittal the IST program was revised and reissued for the third 10-year interval and that the AFWS mini-recirculation valves were now fail safe tested.

Evaluation: PBNP verified the performance of safety-related functions with a loss of IA and that the AFW recirc valves must fail closed to assure the AFW safety-related function of providing flow to the S/Gs. It was also verified that adequate procedures existed (AOP-5B) to address a loss of IA, including the manual actions needed to gag open the recirc valves. Since PRA tools were not available yet, it is not reasonable that the EOP procedural vulnerability would have been identified.

Generic Letter 89-04, Guidance on Developing Acceptable In-service Testing Programs, dated 4/3189 The attachment to the GL listed eleven specific generic deficiencies related to IST programs and procedures. Item 9 addressed pump testing using minimum flow return line with or with out flow measuring devices. The concern for this item was for those pumps that could only be IST tested using minimum flow return. In our response 31

Increased CDF in AFW PRA Model Due to Procedural RCE 01-0U9 Key. j InadequaciesRelated to Loss of Instrument Air dated Octobei 3, 1989, we confirmed that SI, RHR and AFW are tested in compliance for with the GL position 9. The GL advised licensees that meeting the guidelines above).

Code testing does not supercede the thrust of Bulletin 88-04 (See discussion for Evaluation: This review of this issue does not appear to be a missed opportunity evaluation of the EOP procedural vulnerability.

Evaluation Methodology & Analysis Techniques The analytical techniques used in this root cause evaluation were:

- Document Review

- Interviewing

- Event and Causal Factor Charting (Attachment D)

- Timeline Development (Attachment B)

- Why Staircase Development (Attachment C)

Data Analysis Summary Identification of Causal Factors the Information A "Why Staircase" was constructed based on the information obtained in asking of the

& Facts Sources section of this report. This technique results in a repetitive The "Why question "why" until a detailed understanding of the problem is obtained.

identified three Staircase" for this event is provided in Attachment C. This approach main causal factors that contributed to this event.

of RCS cool down EOP-0.1 contains a step (step 1) to CONTROL feed flow because to a steam generator if an considerations and another step (step 4) to STOP feed flow these steps do not increasing level cannot be maintained below the desired setpoint -

postulated that an specify the method to be used to CONTROL or STOP flow. (It is a loss of instrument air operator could throttle the AFW discharge valves closed and with would dead-head and when the recirculation valves are failed closed, the running pumps failure.)

destroy themselves in short period of time; a potential common mode not provided in the There were two reasons influencing why specific information was specific operator actions EOP. First, reliance had been placed on AOP-5B for providing AFW discharge valves for a loss of instrument air scenario, and second, closure of the failure mechanism.

due to operator action was not previously considered as a possible Reliance on AOP-5B:

AFW flow (under loss Reliance on AOP-5B was faulty because operator action to control This need had not of instrument air conditions) was needed in the early steps of EOP-0.1.

identified this need was via been identified prior to this event. A key opportunity to have steps was done in 1985 the EOP validation process. The original validation of EOP-0.I a concurrent loss of using a Reactor Trip w/o Sl scenario. This scenario did not include method an operator instrument air condition. Consequently, it would not matter what 32

t hicreased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. 1I InadequaciesRelated to Loss of InstrumentAir used to control flow since either throttling flow or shutting off pumps would be successful. These steps have not changed since Revision 0, so additional validation would not have been required. It was the EOP validation barrier that failed. Validation was to ensure the operational correctness of the EOPs. The reason the barrier failed was because the interaction between design, procedures, and human error/timeline analysis was not evaluated concurrently, and the need for specific operator actions under a loss of instrument air condition was not recognized. The Human Error/Timeline Analysis method was not available at the time the EOPs were originally validated.

Another key opportunity to identify the need for operator action while in EOP-0.1 was when the initial PRA model was developed to support the IPE submittal in 1993. The original PRA model did not model operator actions to control AFW flow in the system fault trees because it was assumed (based on decay heat removal requirements) that there was a long time available and the function (S/G overfill) would be alarmed (assumption 13). The flaw in this assumption was not identified during the PRA model review because the fault trees were based primarily on functions described in design documents.

Also, only operator actions taken to mitigate a failure were evaluated. The selection of the evaluation method using fault trees focused on design functions over other FMEA methods was based on an assumption that the design function approach was more conservative. The current PRA model review uses a methodology that integrates system performance with potential human actions to obtain a spectrum of plant responses. The original PRA Model was based on system functions, and only operator actions to mitigate failures were evaluated.

Finally, routine performance of accident scenarios on the PBNP simulator should also have provided an opportunity to identify this need for operator action. Simulator Guides are presented in outline form and do not contain detailed information on evaluation of all actions performed during the scenario. PRA information has been used to identify which scenarios are important to teach from a risk perspective, but information on which steps in emergency procedures are risk-significant has not been incorporated into scenario evaluation criteria. The operator action to control AFW flow had not been identified as a human interaction with a human error probability assigned to it (because Human Error/Timeline Analysis was not available yet). Consequently, scenarios often went quickly through the loss of air condition to other conditions such as loss of secondary heat sink without evaluating the intermediate steps such as S/G level control. The interface between the PRA and Training programs is less than adequate.

Operator Action was not Previously Considered as a Possible Failure Mechanism Previous evaluations of the effects of the AFW recirculation valves failing closed on loss of IA concluded that the AFW pumps would not be damaged because forward flow was always available. Closure of a single discharge valve due to component failure concurrent with the AFW recirculation valve failing closed was evaluated and considered to be outside the design and licensing basis. (This used NUREG-0800 assumptions and PBNP was not committed to that NUREG.) Closure of all the discharge valves due to operator action was not considered. The two reasons identified for not considering 33

-..

• I Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 Rev I I InadequaciesRelated to Loss of Instrument Air mode analyses operator actions were the lack of integrating human actions into failure pump damage.

and the lack of insight that a specific operator action could result in result from human Although the concept of determining the potential failures that could often utilized in the PRA errors has been around since at least the TMI accident, it is most of failure modes from a area. The current design process does not prompt an evaluation AOVs were modified human action perspective. When the MDAFW pump discharge was created that did not exist on with a nitrogen back-up system, a throttling capability of the MOVs that that valve before (under a loss of instrument air condition). Throttling existed, so this was an direct AFW flow to the respective steam generators had already Only recent additional opportunity to perform that same action on another component.

identification of the use of failure mode fault tree tables in the PRA program allowed human interactions in concern on AFW control. The knowledge learned from evaluating and effects analysis the PRA program has not been transferred into the failure modes the PRA and Design element of the design control program. The interface between Control programs is less than adequate.

to a "CONTROL or Insight was needed to understand that the actual operator response would be closure of STOP feed flow" command under a loss of instrument air scenario expected operator the discharge valves instead of stopping the AFW pumps. The a loss of instrument air response to the "CONTROL or STOP feed flow" command under that operation of the scenario was not clearly stated in training documents. Knowledge with it could have AFW discharge valves had a human error probability associated identified the potential for resulted in focused training on that evolution that may have human interactions was not pump damage. However, the information on risk-significant between the PRA and effectively incorporated into the training program. The interface Training programs is less than adequate.

Other Conclusions interactions are based on The assumptions used by the PRA group in evaluating human procedures is established.

industry guidelines that determine how the effectiveness of writing. One These same rules have not been applied to our process for procedure is clearly not to example is the use of action steps in notes. The industry guidance Guide (and WOG ERG include actions in notes. However, the AOP and EOP Writers' an action in a note.

Writers Guide) allows the use of condition monitoring that initiates in a note. Procedure Under PRA rules, little credit is given for an action embedded into our procedure effectiveness can be improved by incorporating PRA rules PRA and procedure development development process. The interface between the processes is less than adequate.

the governing document for ESG 5.1, PRA Maintenance and Update Guideline, is interfaces with departments administration of PRA updates. That procedure contains may be more appropriate for outside of Engineering. The use of a higher tier document process lack formality.

this process. Organizational interfaces for the PRA update 34

Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of Instrument Air There was a lack of consistency between different design basis and licensing documents regarding the description and function of the AFW recirculation valves. The predominant position taken in various licensing correspondence was that AFW flow could always be provided to the S/Gs and the recirculation valves were not required to provide an open safety function. However, the initial AFW DBD (1994) contained a statement that the valves had an open safety function, and the basis was not clear. The open function was removed from the AFW DBD in 2000. The IST program did not include an open safety function, but did test the valves in the open direction based on prior NRC correspondence (1992). That testing was removed from the IST program in 1998. The FSAR did not include any discussion of the recirculation line function until updates made in 1997 and 1998. Consistency between AFW licensing and design basis documents is less than adequate.

The subject of AFW flow and recirculation capability was part of many prior evaluations.

However, the combined evaluation of design, procedures and human error timeline analysis only occurred during the recent PRA model update process. Without the use of these combined analyses, it was not reasonable that previous evaluations would have identified this vulnerability.

FailureMode Identification RR5 Actions Not Tied to Another Process When Necessary - Actions required by one program not belonging to any program, which is needed to ensure consistency.

• Information on risk-significant human interactions was not effectively incorporated into the operations training program, including scenario development

• Knowledge learned from evaluating human interactions in the PRA program has not been transferred into the failure modes and effects analysis element of the design control program o PRA concepts are not included in the emergency procedure development process o Consistency in the licensing and design basis for the AFW system was not maintained between the FSAR, AFW DBD and IST program RR2 I Actions Not Clear - Inadequate program design The original validation of EOP-0.1 steps done in 1985 using a Reactor Trip w/o SI scenario did not include a concurrent loss of instrument air condition because the analytical tools (Human Error/Timeline Analysis) needed to identify this were not available at that time F2 Inadequate Communications Among Organizations - Lack of interface Iformality 35

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I Inadequacies Related to Loss of InstrumentAir The PRA update interface requirements with other organizations are contained in an Engineering Supplemental Guideline, and lack formality 1J4 Wrong Assumptions - Erroneous assumptions used in decision making Only operator actions taken to mitigate failures were evaluated in the original PRA model The selection of the original PRA model evaluation method using fault trees focused on design functions over other FMEA methods was based on an assumption that the design function approach was more conservative VII. Root Causes & Contributing Factors Conclusions causing The investigation found that the EOP validation process is the barrier that failed, evaluate the weakness in EOP-0.1. The EOP validation process failed because it did not analysis. It was only the interaction among design, procedures, and human error timeline from this integrated perspective that a loss of instrument air causing the recirculation close the valves to fail closed, combined with a possibility that an operator would discharge valve on an AFW pump, and the timing of this action prior to implementation potential be of the abnormal procedure for loss of instrument air (AOP-5B) could the timeline studies, and seen to damage multiple AFW pumps. The combination of FMEA, unique to PRA.

human error analysis is a recently implemented practice in the industry that previous Without the use of these combined analyses, it was not reasonable evaluations would have identified.this vulnerability.

Root Cause EOP The root cause of the EOP procedural weaknesses was the failure of the original needed to validation process barrier to identify that specific operator actions were air condition. This barrier properly control or stop AFW flow under a loss of instrument did not exist at that failed because the analytical tools needed to identify this vulnerability guidance.

time. This resulted in a misalignment between plant design and procedural Contributing Causes Significant contributing causes to this condition continuing to exist were:

A model fault trees evaluated system perform ance prim arily on origin al P RSThe actions functions described in design documents and only considered operator taken to mitigate a failure

• Previous evaluations focused on delivery of the minimum required AFW flow for providing decay heat removal 36

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. 1I InadequaciesRelated to Loss of InstrumentAir Other causes that were not significant contributors were:

"* The failure to consider human actions during FMEA reviews in the design control processes,

"* The lack of integration of human error reduction methods into the operations training process,

"* The lack of integration of human error reduction methods into the emergency procedure development process,

"* The lack of formality of organizational interfaces in the PRA update process, and

"* The inconsistencies between the FSAR, AFW DBD, and the IST program concerning the description and function of the AFW recirculation valves.

Viii. Corrective Actions Interim Corrective Actions (mitigation)

CA #1 Responsible Group: Qperations, Comrnletion Due Date: Complete Revise EOP-0, EOP-0. I and ECA-0.0 to address AFW control under loss of instrument air conditions.

Corrective Actions to Prevent Recurrence (CATPRs)

• CA #1 Responsible Group: Engineering (PRA), Priority: 2, Completion Due Date:

Complete [CA003691]

Assist Operations in determining what initiating events should be included in the EOP validation process by formally providing information on which initiating events considered risk-significant for each EOP.

• CA #2 Responsible Group: Operations, Priority: 2, Completion Due Date: 8/5/2002 (90 days after CATPR #1 is completed) [CA003692]

Revise the EOP validation process to ensure that appropriate initiating events are included. Utilize PRA input in determining what initiating events are applicable.

37

IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air Corrective Actions to Restore (broke - fix)

" CA #1 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date.

10/4/2002 [CA003693]

Complete the analysis portion of the PRA model review to identify any other risk significant vulnerabilities in the current EOPs.

" CA #2 Responsible Group: Operations, Priority: 3, Completion Due Date: Complete

[CA003694]

Review the operator actions specified in AOP-5B to determine if they should be included in applicable EOPs to ensure timeliness of the actions, and initiate revisions as required.

" CA #3 Responsible Group: Engineering (PRA), Priority: 3, Comple.tion Due Date:

6/5/2002 [CA003695]

Formally provide Operations and Training with an updated list of high-risk human error events based on the PRA model.

"* CA #4 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:

6/5/2002 [CA0036961 Formally provide Operations and Training with a description of the human error reduction methods used in evaluating operator actions in the PRA model.

" CA #5 Responsible Group: Operations, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA #2 and CA # 3 are completed) [CA003697]

Review EOPs and AOPs containing high-risk human error events against human error reduction methods used in the PRA model and revise where appropriate to achieve significant CDF risk reduction.

" CA #6 Responsible Group: Operations, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA # 3 is completed) [CA003698]

Revise OM 4.3.1, AOP and EOP Writers' Guide, to incorporate human error reduction methods used in the PRA model that can significantly reduce CDF risk.

" CA #7 Responsible Group: Training, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA 4f2 and CA # 3 are completed) [CA003699]

Review initial operator training materials and methods associated with high-risk human error-events against human error reduction methods used in the PRA model and revise where appropriate to achieve significant CDF risk reduction.

38

RCE 01-069 Rev. I IncreasedCDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of InstrunentAir Date: 10/4/2002 CA #8 Responsible Group: Training, Priority: 3, Completion Due (120 days after CA # 3 is completed) [CA0037001 error reduction methods Revise operator training procedures to incorporate human risk.

used in the PRA model that can significantly reduce CDF Due Date:

• CA #9 Responsible Group: Engineering (PRA), Priority: 3, Completion 6/5/2002 [CA00370 1]

system performance.

Revise the AFW PRA model to accurately reflect 3, Completion Due

" CA #10 Responsible Group: Engineering (Systems), Priority:

Date: 6/5/2002 [CA003702]

in the FSAR, DBD-01, Review the description of the AFW recirculation line function and initiate revisions as required.

and the IST Program for consistency and accuracy, 3, Completion Due Date:

" CA #11 Responsible Group: Engineering (Design), Priority:

6/5/2002 [CA003703]

of human action induced failure Revise the design process to include consideration modes.

3, Completion Due Date:

" CA #i2 Responsible Group: Engineering (PRA), Priority:

6/5/2002 days [CA003704]

is the appropriate procedural Evaluate if an Engineering Supplemental Guideline tier document such as a Nuclear method for controlling PRA updates, or if a higher interfaces involving other Procedure (NP) should be used considering the from that evaluation.

departments. Initiate any procedure changes resulting 3, Completion Due Date:

"CA#13 Responsible Group: Engineering (PRA), Priority:

6/5/2002 [CA003705]

include identification of the formal Revise the procedure governing PRA updates to to other groups. Use of existing methods to be used for providing information procedure feedback forms, should be processes, such as training work requests and used whenever possible.

Due Date:

" CA #14 Responsible Group: Assessment, Priority: 3, Completion Complete [CA003982]

items if not fully addressed.

Review SEN 174 response and re-open the OE 39

RCE 01-069 Rev. I Increased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air Due Date: Complete

" CA #15 Responsible Group: Operations, Priority: 3, Completion

[CA004279]

adequate pump Review SEN 174 and verify that procedures exist for maintaining flow, including pumps other than AFW.

Due Date:

" CA #16 Responsible Group: Engineering (PRA), Priority: 4, Completion Complete [CA004388]

for validity for the top risk Review operator action assumptions in the PRA model significant systems.

Due Date: Complete

"* CA #17 Responsible Group: Training, Priority: 3, Completion due to less than required Update the PBNP simulator to model AFW pump failure minimum recirculation flow.

Due Date: Complete

" CA #18 Responsible Group: Operations, Priority: 3, Completion Revise the EOP validation process to include PRA involvement. OM/4'-3/ O'MZ'/ _

3, Completion Due Date:

" CA #19 Responsible Group: Engineering (Design), Priority:

Complete back-up pneumatic supply to allow Modify the AFW recirculation valves to provide a time for operator actions.

IX. References AOP-5B, various revisions, Loss of Instrument Air Basis for AFW Minimum Flow CR 97-3363, dated 10/15/97, IST Program Design Recirculation Valves Loss of Instrument Air CR 01-2278, dated 7/6/01, AFW PRA Model for CR 01-3595, dated 11129/01, PRA for AFW System to an Appendix R Fire CR 01-3633, dated 12/4/01, Response of MDAFWPs Mode Failure Information for CR 01-3641, dated 12/4/01, AFW Pumps Common CR 01-3595 RCE to an Appendix R Fire CR 01-3648, dated 12/5/01, Response of MDAFWPs Opportunity CR 01-3654, dated 12/6/01, AFW System DBD Missed Feedwater System DBD-01, Revision 0, dated 4/4/94, Auxiliary System DBD-0 1, Revision 1, dated 3121/00, Auxiliary Feedwater

- Reactor Trip or Safety Injection DD-EOP-0, various revisions, Deviation Documents EOP-0, various revisions, Reactor Trip or Safety Injection EOP-0. 1, various revisions, Reactor Trip Response to the Analysis of Operator Actions EPRI Report TR-100259, dated 6/92, An Approach in Probabilistic Risk Assessment 40

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Ret' I InadequaciesRelated to Loss of histruMent Air Event Notificatiori Worksheet EN#38525, dated 1 1/29/01 Event Notification Worksheet EN#38525 Supplemental, dated 11/30/01 FSAR, Chapter 10, various revisions, Auxiliary Feedwater System (AF)

Individual Plant Evaluation, Revision 0, dated 6/30/93 IST Background Document -Appendix A, dated 5117/00 IST Program Interval, various revisions Internal Memorandum, dated 12/3/01, CR 01-3595 Reportability Recommendations NUREG-0899, dated 8/82, Guidelines for the Preparation of Emergency Operating Procedures NUREG-1358, dated 4/89, Lessons Learned From the Special Inspection Program for Emergency Operation Procedures NUREG-1358 Supplement 1, dated 10/92, Lessons Learned From the Special Inspection Program for Emergency Operation Procedures NUREG/CR-2005, dated 4183, Checklist for Evaluating Emergency Operating Procedures Used in Nuclear Power Plants OD 01-3595 Rev. 0 dated 11/30/01, and Rev. I dated 12/1/01 OD 01-3648 Rev. 0 dated 12/7/01 OM 4.3.1, Revision 1, dated 6/4/99, AOP and EOP Writers' Guide OM 4.3.2, Revision 1, dated 6/14/95, EOP Verification Procedure OM 4.3.3, Revision 0, dated 7/30/93, EOP Validation PRA System Notebook - AFW, Revision 0, dated 1991 QCR 99-0115, dated 5/24/99, Code Testing Conflict With the AFW Mini-Flow Recirc Check Valves RCE 98-148, dated 1/29/99, P-38A AFW Pump Recirc Valve Found Failed Shut S-A-ENG-01-03, PBNP PRA Peer Review Report (Draft Report - 7/0 1)

SEN 174, dated 11/10/97, Loss of Nonvital Bus Causes Dual Unit Scram and Degraded Auxiliary Feedwater System WOG ERG Executive Manual WOG ERG Writers Guide, dated 7/1/1987 WOG LP-ERGs Zion Station LER 050-295/90-002-00, dated 2/15/90, 1A Auxiliary Feedwater Pump Cavitation X. Attachments Attachment A: Team Charter Attachment B: Timeline Attachment C: Why Staircase Attachment D: Event & Causal Factor Chart 41

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. ]

InadequaciesRelated to Loss of InstrumentAir Attachment A: Team Charter Root Cause Investigation Charter CR 01-3595 RCE 01-069 Issue Manager:

Rick Mende Problem Statement:

involving loss of Discovery during the review of the AFW PRA model for transients may not adequately instrument air that emergency and abnormal operating procedures flow to prevent AFW pump address maintaining minimum AFW pump recirculation failure.

Investigation Scope:

Determine the following:

• the root cause of why the condition exists d why the problem was not identified previously Make recommendations for.

"* correcting the problem

"* preventing recurrence of the problem

"* applicability of the root cause to other areas (extent of condition)

Team Members:

Team Leader - Richard Flessner, Engineering Processes Team Member - R. Wood, PRA Team Member - J.P. Schroeder, System Engineering Team Member - T. Staskal, Site Assessment Team Member - C. Krause, Licensing Milestones:

Status Update - 12/11/01 Draft Report - 12/20/01 Final Report - 1/10/02 Date: 12/4/2001 Approved: (Original sigmed by F. Cavia)

Fred Cayia, PBNP Plant Manager 42

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev.t I InadequaciesRelated to Loss of Instrument Air Attachment B: Event Timeline DATE / TIME DESCRIPTION 9/1/79 M-623/624 TDAFP alternate bearing cooling supply modification issued 2/1/80 IC-274 AFW recirculation valve logic (keep open) modification issued 2/10/81 GL 81-14 issued on Seismic Qualification of AFW System (response is dated 7/16/81) 5/4/82 Additional response to GL 81-14 due to NRC RAI - response says that AFW recirc valves close on receipt of AFW pump discharge flow signal 6/82 WOG Basic ERGs validated on Calloway Simulator 8/82 NUREG-0899, Guidelines for the Preparation of EOPs, is issued 8/31/82 IC-274 AFW recirculation valve logic (keep open) modification cancelled 11/12/82 NRC issues TER concluding that PBNP AFW system did not provide reasonable assurance to perform its SR function following a seismic event 12/15/82 PBNP response to NRC TER on AFW - concluded that IA is not required for AFW system functioning (based on recirc valves FC and discharge valves FO);

commit to independently supporting each recirc valve 4/83 NUREG/CR-2005, Checklist for Evaluating EOPs, is issued 8/1/83 MR 83-104 AFW system discharge MOV controls modification issued 4/26/85 PBNP response to revised NRC TER on AFW - conclude that AFW piping failure or failure of AFW recirc valves to close will be handed by operators trained to recognize off normal condition that adequate time exists for manual action 7/1/85 Revision 0 of the EOPs issued 5/2/86 AOP-5B, Loss of Instrument Air, Revision 0 issued 6/22/87 IN 87-28 issued on Air Supply Problems at US Light Water Reactors 7/1/87 WOG ERG Writers Guide issued 12/20/87 IN 87-28 Supplement I issued on Air Supply Problems at US Light Water Reactors 3/23/88 NPERS evaluation of IN 87-28 issued via NEPB 88-090 5/5/88 IEB 88-04 issued on Potential SR Pump Loss (response is dated 6/28/88) 5/18/88 INPO issues SOER 88-01 on Instrument Air Failures 7/7/88 MR 88-099 AFW pump mini-recirculation line improvements modification issued 7/21/88 SBO Rule (10CFR5O.63) became effective (response is dated 4/17189) 8/8/88 GL 88-14 issued on Instrument Air Supply System Problems Affecting SR Equipment (response is dated 2/20/89) 4/89 NUREG-1358, Lessons Learned From the Special Inspections Program for EOPs, is issued 4/3/89 GL 89-04 issued on Guidance on Developing Acceptable IST Programs (response is dated 10/3/89) 5/8/89 MSS approves response to SOER 88-01 2/15/90 Zion Unit I LER issued on AFW Pump Cavitation 12/90 3rd interval IST Program is implemented

.- 1991 Original IPE Notebooks developed 43

Increased CDF in AFW PRA Model Due to Procedural RCE OJ-069 Rev. I InadequaciesRelated to Loss of InstrumentAir DATE / TIME DESCRIPTION 5/28/91 Revision I to IST Program adding VRR-28 on recirc valves 4/17/92 NRC issues TER on IST Program denying VRR-28 and requesting OPEN safety function be added for recirc valves 6/92 EPRI Report TR-100259, An Approach to the Analysis of Operator Actions in PRA, is issued 6/19/92 MR 92-091/092/093 IST testability of AFW recirculation line AOVs modifications issued 7/30/92 PBNP response to NRC TER clarifying that recirc valves are not required to OPEN to protect AFW pumps 10/92 NUREG-1358 Supplement 1, Lessons Learned From the Special Inspections Program for EOPs, is issued 3/2/93 PBNP informs NRC that mods will be completed for testing recirc valves and withdraws VRR-28 3/30/93 Rev. 3 of IST deletes VRR-28 4193 DBD-01 validation considers worst-case flow (discharge and recirc valves closed) outside design and licensing basis 6/30/93 Revision 0 of IPE PRA model is issued 4/4/94 DBD-01, AFW System, Revision 0 is issued

-1995 Affects of excessive AFW flow introduced into EOPs 4/15/97 MR 97-038*A/B MDAFP discharge pressure control valve backup nitrogen supply and cable separation modifications issued 6/97 Update to FSAR adding AFW recirc feature for 3 minute closure on pump start 9/26/97 AOP-5B, Revision 11 issued that moved time critical steps from appendices to main body of the procedure 9/30/97 Revision 1C of WOG ERGs issued 10/15/97 CR 97-3363 initiated on IST Program Design Basis for AFW Minimum Flow Recirculation Valves (closed 10/5/98) 11/10/97 INPO issues SEN 174 on Loss of Nonvital Bus Causes Dual Unit Scram and Degraded AFW System (McGuire Units) 1998 Update to IPE PRA model is issued 1/6/98 Evaluation of SEN 174 completed - focus was on power supplies and did not address degradation of AEW recirculation valves 6/98 Update to FSAR adding detailed description of recirculation line function 6/29/98 CR 98-2575 (RCE 98-148) initiated on P-38A AFW Pump Recirc Valve Found Failed Shut 9/30/98 Rev. 5 of IST Program issued deleting testing of AFW recirc valves in the open direction 5/24/99 QCR 99-0115 initiated on Code Testing Conflict With the AFW Mini-flow Recirc Check Valves 3/31/00 DBD-01, AFW System, Revision I is issued 9/11/00 OE 10727 initiated on industry event involving PRA 44

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 7/6/01 While revising the Probabilistic Risk Assessment (PRA) model for the Auxiliary Feedwater system, a potential procedural shortcoming was identified in AOP-5B, Loss of Instrument Air. Condition Report 01-2278 was originated to document the above finding 7/10/01 A CR action item #1 was created for Operations to move the step in AOP-5B, "Loss of Instrument Air," for gagging open the AFW minimum recirculation valves to an earlier location in the body of the procedure. (CR 01-2278) 7/30/01 Operations discussed issue with PRA group. PRA to run an evaluation to determine the significance of the issue. Analysis was expected to be completed by 8/20/01 (CR 01-2278) 8120/01 The analysis is not ready yet. The evaluation is expected to determine the actual risk significance of the condition and address the type of actions that may be recommended. (CR 01-2278) 10/19/01 Per discussion with the PRA group, the PRA model is showing a higher risk and the recirculation valve should be procedurally addressed. The AOP is sequenced properly to address the loss of instrument air. PRA Group is requesting that the ARP for low instrument air pressure be changed to address this concern. This should be adequate rather than changing the sequence of the AOP. PRA will follow up with a procedure feedback. (CR 01-2278) 10/24/01 CR 01-2278 Action #1 was completed with direction to create a new action item to track issuance of a change to ARP COI A 1-9 for low instrument air pressure. (CR 01-2278)

Early Operations had discussions with PRA Group regarding whether procedure November, changes were adequate.

2001 Week of Nov PRA Group went to work to adjust the PRA model to evaluate the risk if the 13th- 2001 procedure change was not complete or would not be adequate.

11/26/01 Modeling adjustments were completed. A risk evaluation was done for the minimum recirculation valves. A factor of 2.3 risk increase was identified.

This was considered high-risk significance. A discussion was held with Operations and Engineering. Decided we needed to determine what the scope of this was and what further actions may be appropriate.

11/28/01 - A meeting was held with Operations, Engineering and PRA personnel to 1300 discuss the significance and appropriate actions. The mechanistic details of the issue were well understood and developed by all present. The consensus was that this item represented a real possibility, and that it required further attention. Various possible actions were discussed, focusing primarily on enhancing Operator awareness of the system design, as well as modifications or procedural changes that may be desirable to eliminate it.

The subject of Operability was discussed during the meeting, and it was agreed that there was no operability concern because no equipment degradation, failure, or non-conformance had been identified. Regardless, the level of concern was great enough that further prompt action was felt'justified.

45

Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 11/28/01 - The Operations manager had discussions with Engineering about this potential Late afternoon concern regarding significantly increased CDF risk resulting from an event where instrument air was lost and during the subsequent EOP actions, operators may take inappropriate action which could cause one or more AFW pumps to fail.

11/29/01 - AM Operations manager briefed the resident inspectors on the concerns of the issue and that we were evaluating the condition and risk.

11/29/01 - Following discussions with the staff SRO, operations concluded that use of Late AM temporary information tags and a briefing of all watch standers, would be an important step to reduce the risk of the event. We also started evaluating procedure changes that might help improve the safety of the plant and reduce the risk profile.

11/29/01 - PRA briefed the STA and Shift Manager on the issue and discussed potential 10:00 wording for control board placards.

11/29/01 - PRA discussed potential reportability concerns with licensing.

11:00 11/29/01 - PRA briefed the RI and provided estimated risk impact values.

11:30 11/29/01 - CR 01-3595 documenting the increased risk was written. The CR was brought 14:45 to the WCC and screened by an SRO. At that time, extensive discussion regarding whether an OD was required had already occurred, and extensive discussion on operability had occurred. My discussions with engineering and others focused on the fact that there was not an equipment problem, no equipment is degraded such that operability is in question, that this is a risk issue upon which we are relying on operator action to mitigate, and therefore, use of the OD was not appropriate. Those discussions were not captured in either the CR, or the associated screening.

11/29/01 - The oncoming crew was briefed and temporary information tags placed 1520 adjacent to the controls for 1/2P-29 and P-38A/B. This briefing summarized the concerns of this potential event. The temporary information tags provided a reminder that the minimum flow requirements for the AFW pumps are 50 GPM for the motor driven pumps and 75 GPM for the steam driven pumps.

11/29101 - CR 01-3595 was screened by the WCC SRO (CR 01-3595) 1553 11/29/01 - Operations Manager briefed Plant Manager on this issue.

1700 11/29/01 - Event Notification 38525 made to NRC via ENS phone.

1705 11/30/01 - AM Licensing manager received a call from the NRC-NRR backup PM concerning confusion over the event notification. A return conference call was made with engineering to address NRR questions.

46

Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelated to Loss of InstrumentAir DATE / TIME DESCRIPTION 11130/01 - AM Friday morning, after discussing this with the residents, Operations Manager concluded that to properly document the operability of the AFW system. we should initiate an operability determination to ensure the discussions we had the previous 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> regarding operability were properly documented.

Engineering was requested to start on the OD. 'The Shift Manager was informed that an OD on the issue was being performing it and that it was expected to be completed mid to late afternoon.

11/30/01 - Operations Manager met with Sr. Resident, Resident, and their supervisor to Noon discuss situation. At that point NRC brought forward their concerns regarding whether AFW was operable in the condition that existed prior to Thursday afternoon and whether it was currently operable. The Plant Manager called NRC Region III along with the Operations Manager and had a discussion regarding operability of the system.

11/30/01 - Ran a simulator scenario to get information on plant response to a loss of 1400 offsite power coincident with a rapid loss of instrument air pressure.

NOTE: Additional simulator scenarios were run on 11/30 and 12/1.

11/30/01 - Temporary procedure changes were completed to EOP-0 and EOP-0.1 to 1645 reflect the guidance provided earlier to operators on the temp info cards.

11/30/01 - Plant Manager informed that a five-man incident investigation team would

-1700 arrive on 12/3.

11/30/01 - A supplement to the Event Notification was provided to the NRC to clarify the 1746 discussion of the potential for an AFW failure as described in the original event notification 38525 11/30/01 - The OD was approved. This OD evaluated the current operability of the AFW

-1830 system and included a discussion of the compensatory measures already taken to assure compliance with our licensing basis.

12/1/01 - 0930 Staff meeting to prepare for NRC inspection team.

to 1200 12/1/01 - 1515 Revision 1 to the Operability Determination was approved. The discussion of the AFW pump motor duty cycle was revised.

12/3/01 - 0830 CR 01-3595 screened as requiring an ACE.

12/3/01 - 1000 Inspection Team meeting to prepare presentation for NRC entrance meeting.

to 1200 12/3/01 - 1200 SVP and Plant Manager agree that CR 01-3595 requires a RCE.

12/3/01 - 1400 NRC Inspection Team has entrance meeting.

12/4/01 HEP expert onsite 12/4/01 - 0700 Initial RCE Team meeting held.

1214/01 - 1200 Plant Manger approves RCE Charter.

12/4/01 - 1620 CR 01-3633 initiated on Appendix R concerns associated with MDAFW pump and LOOP and loss of IA and coincident fire. (CR 01-3633) 12/5/01 - 1545 CR 01-3648 initiated on response of MDAFW Pump to an Appendix R fire coincident with a LOOP and loss of IA. Potential existed for auto-start with discharge and recirc valves failed closed causing pump damage. (CR 01-3648) 12/7101 - 0900 NRC Inspection Team has technical debrief.

47

RCEOI-069 Rev. I ]

Increased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 12/13/01 - NRC Inspection Team has exit meeting.

1400 12/14/01 Permanent Revision to EOP-0 and EOP-0.1 implemented.

12/20/01 Additional revision made to EOP-0, EOP-0.1, and ECA-0.0 48

Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 Rev. I I InadequaciesRelaied to Loss of Instruneni Air Attachment C: Why Staircase Problem: There is an increased CDF during a loss of instrument air scenario due to a common mode failure of all AFW pumps.

Why?: EOP-0. 1 contains a step (step 1) to CONTROL feed flow because of RCS cool down considerations (RCS overcooling) and another step (step 4) to STOP feed flow to a steam generator if an increasing level cannot be maintained below the desired setpoint (S/G overfill) - these steps do not specify the method to be used to CONTROL or STOP flow. (It is postulated that an operator could throttle the AFW discharge valves closed and with a loss of instrument air when the recirculation valves are failed closed or fail closed later, the running pumps would dead-head and destroy themselves in a few minutes; a common mode failure.)

Problem: EOP-0.1 contains insufficient information to direct operators to take the correct actions for controlling AFW flow or stopping AFW flow to S/Gs under a loss of instrument air scenario.

Whyl?: Reliance had previously been placed on AOP-5B for directing operator response to a loss of instrument air scenario; however, it was just recently recognized by the PRA group that action by operators would be required earlier in the scenario while still in EOP-0.1 (e.g., controlling S/G level without the availability of the AFW recirculation valves).

ProblemI: The need for specific operator response actions for AFW flow control due to a loss of instrument air scenario while in EOP-0. I was not previously identified.

Whyl-l ?: The original validation of EOP-0.1 did not evaluate the interaction between design, procedures and human error/timeline analysis.

This analytical method was not available at that time. (Human Error/Timeline Analysis Not Available)

Whyl-2?: The original PRA model did not model operator actions to control AFW flow in the system fault trees because it was assumed that there was a long time available and the function (SIG overfill) would be alarmed (assumption 13). The flaw in this assumption was not identified during the PRA model review because the fault trees were based primarily on functions described in design documents. Also, only operator actions taken to mitigate a failure were evaluated. The selection of the evaluation method using fault trees focused on design functions over other FMEA methods was based on an assumption that the design function approach was more conservative. The current PRA model review uses a methodology that integrates system performance with potential human 49

Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 Rev. I InadequaciesRelated to Loss of Instrument Air actions to obtain a spectrum of plant responses. (PRA Model based on system functions)/(Only mitigating actions were evaluated)

Whyl-3?: The operator action to control AFW flow had not been identified as a human interaction with a human error probability assigned to it. (Human Error/Timeline Analysis Not Available)

Why2?: Previous evaluations of the effects of the AFW recirculation valves failing closed on loss of IA concluded that the AFW pumps would not be damaged because forward flow was always available. Closure of a single discharge valve due to component failure concurrent with the AFW recirculation valve failing closed was evaluated and considered to be outside the design and licensing basis. (This used NUREG-0800 assumptions and PBNP was not committed to that NUREG.) Closure of all the discharge valves due to operator action was not considered.

Problem2: Closure of the AFW discharge valves due to operator action was not previously considered as a possible failure mechanism.

Why2-1?: The consideration of human actions in failure modes and effects analyses has occurred primarily only in the PRA area and the integrated method of evaluating FMEA, human error probabilities, and timeline studies is a recent development. (Human Error/Timeline Analysis Not Available)

Why 2-2?: Insight was needed to understand that the actual operator response to a "CONTROL or STOP feed flow" command under a loss of instrument air scenario would be closure of the discharge valves instead of stopping the AFW pumps.

Problem: The expected operator response to the "CONTROL or STOP feed flow" command under a loss*of instrument air scenario was not clear.

Why?: Training materials did not contain specific information on operator actions for controlling steam generator level (and AFW flow) under a loss of instrument air condition.

Problem: Training materials did not specify the actions required for successful control of AFW flow under loss of instrument air conditions.

Why?: The importance of the AFW control evolution was not previously recognized. (Human Error/Timeline Analysis Not Available) 50

RCE 01-069 Rei Inc,,Lsed CDF in AFW PRA Model Due to Procedural Air InadequaciesRelated to Loss of Instrument Chart Attachment D: Event & CausalFactor I KEY a

T UNVERIFIE

• EVENT 5".

OO.... CAUSAL F<ACTOR 51

RCE 01-069 Re I In, .sed CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of InstrumentAir KEY INýAPPROPRIA1TE r -----......... CTION i EVENT 'ondto

---

D El (S)r Inve It- CONTReIBUiNG FACTORDF OOT (-CONDITION..,

CAUSAL AS nit to idently ach AFW valve r___________

5/82 issued nca tr) 52

RCE 01-069 Rc In. .jsed CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air KEY UNVERIFIED VET (lcorf I EVENT  !

Scope Is based on no operator actions "OO CAUSAL A CAUSTE 0\"CONOI~O. !D Timeline Analysis

• Not Available I,,

"Doexplanatory notes avoid the US e No change needed on AFW recirc vlvs of action statement?

3/10/83 - 4/18/83 Early drafts of PBNP EOPs validated on Zion Simulator "Scopeis MDAFP discharge valves auto actuation 53

RCE 01-069 Re In,,. ased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air 4/26/85 PBNP responded to revised NRC AFW TER KEY UNVEMtPED (Fietral) INAPPROPRIATE EVENT (CONTR¶S1NO\

R.......CAU.AL 54

In(., ..sed CDF in AFW PRA Model Due to Procedural RCE 01-069 Re Inadequacies Related to Loss of InstrumentAir KEY

... A...O...A..

UNVERIFIED EVENT Lnditon IEVENAi CTIOt L --------

Root CAUSA nv si-J FACTOR CAUSE "'"--CONDITION"*

55

RCE 01-069 Re, I Inc, ....sed CDF in AFW PRA Model Due to Procedural Inadequacies Related to Loss of Instrument Air KEY 3" NA~PPOPRIAT UNVERIFIED EVENT

~Fý"Ul CI CONTRIBUTING

,-UNVERFE-. Investl. FACTOR

( CAUSAL CU .. CONDITION..'

Discrepancy 56

RCE 01-069 Rf. I lr, .ased CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air KEY VNVE~IFIEDNAPPSOPflIATE EETondillon CTIO for~N COTLTJUTN fOTCAUSAL 1 wSS11.) FACo I CAS '~CONDITON- t~!!2

'Design review not identify did" concern with FC recire vtvs, 57

RCE 01-069 Re IfL .,sed CDF in AFW PRA Model Due to Procedural inadequaciesRelated to Loss of Instrument Air KEY INAP'PnopnIATE UNVERIFIED I EVET Ictrs EVENT ond~diofS'CIr POT CAUSAL ~Ivs1) FACTOR CAS .. CONOT . atbo 2/g/89 2/20/89 PBNP response to NPERS evaluation of GL 88-14 issued SOER 88-01 Issued 58

RCE 01-069 Rei In. ,ed CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air KEY S...............

UNVERIFIED EVENT

• II \..cions TO CONTnIBUTING CAUSAL $,,I- FACTOn CAUSE "..'rCON.ITION..' ,io "AFW dlsch AOVsN FO could overfeed S/Gs -

59

RCE 01-069 Rev Inc, .ed CDF in AFW PRA Model Due to Procedural Inadequacies Related to Loss of Instrument Air KEY INAPPROPRIATE UNVERIFIED EciO EVENT odition

=...............

looT , cAUSAL f CAUSCT.CONDTION.' cT fl

.ATO VRR-28 recirc linedescribes function 60

RCE 01-069 Rei I Inc. .jed CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air KEY u'X":ugal 1PROPPIAT 1

UNVERIFIED EVENT (Pact)ri')

I EVENT NO CONTRBU

( CAUSAL Inve.ta a,0o br FABT CAUS coNC ,

"Scope Is analysis' of operator actions In PRA -

61

Inc. a*sed CDFin AFW PRA Model Due to Procedural RCE 01-069 Re I InadequaciesRelated to Loss of Instrument Air KEY A

UNVERIFIED VE ENEN INAPPROPRIATE iEVENT I ~io CTION Based on NUREG- NUREG-0800 not CAUSE .ai CAUSAL

.C.N.T.O. SI FACTORl 0800 assumptions part of PBNP CLB Affects -1995 of excessive AFW flow put Into EOPs

'Scope Is analysis' of worst case AFW recirc flow ,

62

I11L. .sed CDF in AFW PRA Model Due to Procedural RCE 01-069 Re, I InadequaciesRelated to Loss of InstrumentAir KEY I

UNVrnIrIED EVN EVEN CIOn C0NMfIDUTINO.

l(O CAUSAL rACTOI1 CAUS . CONDIIIONo "ScopeIs movement of time critical steps from Appx. to main 63

1n. .sed CDF in AFW PRA Model Due to Procedural RCE 01-069 Re. I InadequaciesRelated to Loss of Instrumnent Air KEY UNVEpIIEIED EVENT amIlon NAPROPRIAT S.......

(CuIN T CAUSAL I \InveSt.J1 CAUSE ,.CON.DmION..'

64

I RCE 01-069 Re, I In,. .sed CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of Instrument Air KEY UNVEFIEDNAPPROPIA S EVENT nionCI l

CAtNVEIFIEb3/4 CONTAIBL NG (OT V CAUSAL I Sjnvnt FAT~

65

., I I In, sed CDF in AFW PRA Model Due to Procedural RCE 01-069 Re I InadequaciesRelated to Loss of InstrumentAir KEY

• UNVERIFIED ( a' tors;)PIAT EVENT ondllIoni ACTION Root('UVERIIED InlotI CONT11lIRUTING i

CAUSA InesilFACTOn CAUE .. CONOTO. 9110~y 6/01 Late June or Early 10/19/01 PRA group revising July '01 PRA group PRA evaluations AFW portion of PRA Identifies concern with showing higher risk model AOP-5B 66

- I . f RCE 01-069 Re. I Inc.. osed CDF in AFW PRA Model Due to Procedural InadequaciesRelated to Loss of InstrumentAir KEY EVENT F .0. 1NAýPPROPf j EVENT iondlO CTION CONTRIBUTING O " IJNVERIFIEtd'.

CAUSAL . m LV.; FACTOR

.A.....CcoN,oI ..

11/26/01 Risk evaluation performed on AFW recirculatlon valves 67

4* I RCE 01-069 Rev.

Incr, d CDF in AFW PRA Model Due to Procedural I

InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED NAP 0P fl ATE CT 0 N EVENT I---------------

CAUSAL I

• C0NONITION Ao 68

M INTERNAL Corm,ated to Nucear Exck,*fce j CORRESPONDENCE NPM 2002-0495 To: CARB Members From: Richard Flessner Date: September 16, 2002

Subject:

Addendum to RCE 0 1-069 Rev.1/ACE000314 Copy To: S. J. Nikolai S. A. Pfaff L. J. Peterson File The attached addendum to RCE 01-069 Rev. l/ACE000314 is being submitted for CARB review and approval. This addendum is being created to provide a more complete documentation record of items related to RCE 01-069 Rev. 1. The focus of the addendum is primarily on actions taken after the RCE was completed and accepted by CARB. A revision to the RCE is not deemed necessary because the basic conclusions and resulting recommended actions have not changed.

Additional discretionary actions have been implemented by NMC and are being included in the addendum for a more complete record.

Attachment

Addendum to RCE 01-069 Rev.I/ACE000314 This addendum to RCE 01-069 Rev.l (ACE000314) covers the following items:

I Inaccuracy in RCE report regarding IST program testing

2. Comments on Independent Review of RCE Report

3. Addition of the Open Safety Function to the AFW recirculation valves

4. Creation of action items to document corrective actions descnbed in RCE report

5. Expansion of Extent of Condition Review

6. Effectiveness Review Reason for Addendum- This addendum is being created to provide a more complete documentation record of items related to RCE 01-069 Rev. 1. The focus of the addendum is primarily on actions taken after the RCE was completed and accepted by CARB. A revision to the RCE is not deemed necessary because the basic conclusions and resulting recommended actions have not changed Additional discretionary actions have been implemented by NMC and are being included in the addendum for a more complete record.

1. Inaccuracy in RCE report regarding IST program testing On page 23 of RCE 01-069, Rev. 1, a statement is made regarding the deletion of open testing of the AFW recirculation valves from the IST program as a result of the evaluation made for CR 97-3363. Additional review has determined that testing of the AFW recirculation valves was not deleted, and that time testing data exists for all 4 AFW recirculation valves during the period 1993 to 2002.

2. Comments on Independent Review of RCE Report The independent review of the AFW RCE (CAP002612/CA004074) contained the following final conclusion:

"The following final conclusion is based upon the scope of the investigation as prescribed by the management team in the investigation charter. The RCE represents a high quality, detailed, integrated investigation into the problem statement described in the Team Charter. The report is well constructed and well written and allows a non-involved reader to understand the event and the investigation performed. The root cause is supported by the facts, evidence and failure modes identification. The corrective actions are appropriate for the scope of the investigation and will ensure higher quality EOP documents in the future. Questions regarding the adequacy of the overall scope of the investigation are contained in the main body of the report."

Specific issues discussed in the review are:

" Charter/scope of investigation does not investigate why the design allowed the recirculation valves to fail-closed on loss of instrument air and how this condition went uncorrected until discovered by the PRA review.

Comment: The fail-closed position was known and understood in the design and did NOT go uncorrecteduntil for discovered by the PRA review. What was not known was the timing of operator actions and the need specific guidance in the EOPs. The problem was determined to be a proceduralissue by PBNP and the NRC; hence the investigation scope was appropriate.

" No corrective actions exist to ensure that similar components do not have the same failure mode.

Comment: Since there was not a problem with the failure mode of the valve, there was no need to evaluate similar components. All operatoractions associatedwith a loss of instrument air condition were evaluatedand determined to be appropriate.

" Root cause may be too narrowly focused.

Comment: The RCE evaluated the mismatch benveen plant design and plant procedures. It was determined that the revised procedures could adequately support the plant design The cited violation is for a procedural problem and not a design issue: hence, the focus was appropriate.

" Barrier analysis might also be used (in addition to E&CF charting) on the EOP development and validation process Comment. This would be an enhancement. Since the EOPs have been through 3 major revisions by WOG and the current processes for verification and validation arc different (and enhanced by corrective actions in the RCE), it was felt that no value would be added by an additional barrier analysis

" Report does not discuss use of single failure analysis in deriving EOPs Comment: This comment was based on the misperception that the fail-closed mode of the recirculation valves was not correct. Single failure analysis would be in addition to the designedfailure mode of the valve and would not have been applicable

" RCE did not address timeliness or effectiveness of CA program in bringing issue to management's attention (initial CR 01-2278 written 7/6/01).

Comment: This issue was discussed between the RCE investigator, his Manager and the PRA Group Lead during the RCE evaluation and determined to be appropriate based on the complexity of the issue, the involvement of operations,and risk associatedwith the issue at that time, therefore, no concern was identified in the final RCE A statement of there being no problem was not added.

" Was deletion of testing the recirculation valves (in the open direction) from the IST program a dropped or missed commitment?

Comment: Evaluation of this item has determined that time testing of the AFW recirculationvalves in the open direction is occurringand has not been deleted.

" RCE does not discuss how PBNP specific design differences were identified through the original EOP development process.

Comment: The report describes the EOP verification process in general terms and the results obtained. The verification was via an approved procedure and checklist. There were more than 2500 discrepancy sheets identified, which is ample evidence that specific plant differences were considered.

" Is it a safety function for the recirculation valves to open?

Comment: The report clearly describes the plant's licensed position that there was no required OPEN safety function for the recirculation valves The NMC decision to add the OPEN safety function was based on improving equipment reliabilityand reducing CDFrisk.

" Report does not discuss any findings regarding design configuration control differences.

Comment: The report identifies that there were inconsistencies between the FSAR, IST and DBD documents and initiateda corrective action to review the current versionsfor consistency. This was treatedas a broke-fix issue since it was not a significant contributing cause to the event. The evaluator's perception of a design problem gave this issue more importance than warranted.

" There is no discussion on how the PBNP design compares to other similar plants AFW design.

Comment: A review of other plants AFW designs was performed and the PBNP design was found to be fairly unique; since there was no design deficiency, the issue was not discussedin the RCE report

" The design change for adding pneumatic back-up supply to the recirculation valves is not identified as a corrective action in the RCE Comment: This corrective action was added to Revision I of the RCE.

3. Addition of the Open Safety Function to the AFW recirculation valves During ongoing reviews of the AFW recirculation issue, NMC determined that there was increased nuclear safety benefit (improved reliability and reduced CDF risk) in the addition of an open safety function to the AFW recirculation valves beyond that credited by the pneumatic back-up supply modifications already installed.

Therefore, modification MR 02-029 was initiated to add the open safety function to the AFW recirculation valves.

This MR included removal of the internals of the AF-117 check valve to eliminate a common mode failure. The modification was accepted on 9/12/02.

4. Creation of action items to document corrective actions described in RCE report RCE 01-069, Rev. I identifies the corrective actions already taken and those being implemented in section VIII of the report, beginning on page 37. T-track references had been provided for the actions being implemented, but not for all of the actions already completed Subsequently, (-track records have been created to adequately document the completed actions discussed in the report The following action items have been created:

"* Interim Corrective Action #1 - CA026222

"* Corrective Action #17 - CA026223

"* Corrective Action #18 - CA026224

"* Corrective Action #19 - CA026225 Other t-track items related to this event are.

"* CA002592 - This item documents the review of the condition from a short-term Maintenance Rule risk monitoring perspective.

"* CA002593 - This item documented the OD review of the condition.

"* CA002594 -This item tracked issuance of the LER for this event.

"* OTH003541 -This item tracked presentation of the completed RCE to CARB.

"* CA003983 - This item brought closure documentation back for CARB review once CA00369 1, CA003692 and CA003693 were completed.

"* OTH004389 -This item tracked revision of the RCE to reflect information gained during preparations for the NRC regulatory conference.

"* OD Part 1 Rev 2 - This document is attached to the parent CAP001415 and documents the operability determination of the original condition.

" OTH0045 10 - This item tracks the correction of problems identified with some HEPs from the review performed under CA004388

"* CAP01201 I/CE010138 (KNPP) - These items document KNPP's review of the industry OE notification issued for this event.

5. Expansion of Extent of Condition Review The EOP weakness regarding controlling AFW flow was found during the PRA model update for the AFW system.

The PRA model update involved a simultaneous review of plant design, procedures, failure modes and timing of operator actions. However, the update process is not specifically designed to identify procedural errors. Therefore, an alternate approach was developed that combined the elements of the effects of a loss of support component function, the procedures that deal with resolving this function, and the timing of required actions. CAP029344 has been initiated to expand the extent of condition review for the AFW Red Finding using this alternate approach to provide an additional level of assurance that similar issues do not exist in other emergency procedures.

6. Effectiveness Review T-track action item CA003983 was created following the CARB Meeting on 3/5/02 to bring back closure documentation for review at a CARB Meeting once CATPRs I and 2 (CA003691 and CA003962), and corrective action #1 (CA003693) were completed CA003693 is associated with the overall PRA update project, which now has an approved action plan that extends to the end of 2004. It is recommended that the scope of CA003983 be modified to be an effectiveness review of the completed CATPRs as normally performed on RCEs