ML030730752
| ML030730752 | |
| Person / Time | |
|---|---|
| Site: | Point Beach  |
| Issue date: | 02/05/2002 |
| From: | Flessner R, Hettick D, Krause C, Mende R, Peterson L, Schroeder J, Staskal T, Wood R Nuclear Management Co |
| To: | Office of Nuclear Reactor Regulation |
| References | |
| FOIA/PA-2003-0094 CR 01-3595, RCE 01-069 | |
| Download: ML030730752 (66) | |
Text
Committed to Nuclear Exceffence.
Point Beach Nuclear Plant Increased CDF in AFW PRA Model Due to Procedural Inadequacies Related to Loss of Instrument Air RCE 01-069 (CR 01-3595)
Event Date: November 29, 2001 Report Date: February 5, 2002 Principal Investigators:
R. Flessner - Team Leader C. Krause J. P. Schroeder T. Staskal R. Wood Approved By:
Group Lead - Larry Peterson Date Issue Manager - Rick Mende Date
?J)
CAP Manager - Dennis Hettick Date J
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Table of Contents I. Executive Summary ............................................................................ 3 II. Event Narrative ................................................................................ 5 III. Extent of Condition Assessment ...................................................... 9 IV. Nuclear Safety Significance .......................................................... 10 V. Report to External Agencies ............................................................... 11 VI. Data Analysis ................................................................................. 12 Information & Fact Sources ................................................................. 12 Evaluation Methodology & Analysis Techniques ................................ 30 Data Analysis Summary ........................................................................ 30 Failure Mode Identification ................................................................... 33 VII. Root Causes & Contributing Factors ............................................ 34 VIII. Corrective Actions ...................................................................... 35 IX. References ..................................................................................... 38 X. Attachments ................................................................................... 39 Attachment A: Team Charter ............................................................... 40 Attachment B: Event Timeline ............................................................ 41 Attachment C: Why Staircase ............................................................ 47 Attachment D: Event & Causal Factor Chart ..................................... 49 2
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air I. Executive Summary
Purpose:
The purpose of this investigation is to determine the root and contributing causes of why the emergency operating procedural inadequacies existed that contributed to the increased core damage frequency (CDF) for the Auxiliary Feedwater System during a loss of instrument air event, and why these inadequacies where not identified previously.
Event Synopsis: During a review of the AFW PRA model in June 2001, it was discovered that the AFW recirculation valves were not modeled. Subsequent discussions disclosed that under a loss of instrument air condition (IA), operators might close the AFW discharge valves to stop AFW flow. Because the recirculation valves fail close on loss of IA, these actions could deadhead the AFW pumps and result in pump damage.
Initially the procedural concern was directed at AOP-5B, but it was later realized that the AOP was not the only concern. Operator actions could be taken earlier in an accident scenario to control or stop AFW flow, prior to entering AOP-5B, while still in EOP-0.1.
PRA modeling of the AFW system continued and on 11/26/01 a factor of 2.3 risk increase in CDF was identified. As discussions with site personnel continued, additional initiating events were identified and on 11/28/01 a revised PRA model was run that changed the risk estimate to a factor of 4 to 5 increase in CDF. Condition report CR 01 3595 was initiated at 1445 on 11/29/01 and an NRC event notification was made at 1705 the same day.
==
Conclusions:==
EOP procedural weaknesses existed because the actions required for AFW control under loss of instrument air conditions were not identified during the original EOP validation process. The recognition of the importance of specific AFW control steps occurring earlier in the EOPs prior to the transition to AOP-5B came only when human error probability methods were applied during the PRA model update process in 2001. These same human error probability methods have not been integrated into procedure development, training, and design control processes. Previous opportunities to identify this issue were missed due to a faulty assumption in the original IPE, an assumption that AOP-5B adequately addressed a loss of IA condition, and the failure to consider the effects of operator actions on system performance.
Nuclear Safety Significance: Preliminary PRA results show that the vulnerability described in this report, prior to the procedural changes, was potentially risk significant.
Although the initiating event frequencies are low to moderate, the unrecoverable IA scenario was risk significant due to the consequences of a total loss of all AFW pumps requiring feed and bleed without the pressurizer PORVs. The risk results are highly dependant upon human interactions. PBNP operators are trained on AFW system operations and have experience with degraded IA scenarios. Because of this training and experience, it is reasonable to assume that operators would have successfully handled this combination of conditions in the unlikely event that it would have occurred.
3
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Root Cause: The root cause of the EOP procedural weaknesses was the failure of the original EOP validation process to identify that specific operator actions were needed to properly control or stop AFW flow under a loss of instrument air condition. This resulted in a mismatch between plant design and procedural guidance.
Significant contributing causes to this condition continuing to exist were:
- the original PRA model fault trees evaluated system performance primarily on functions described in design documents and did not adequately consider human actions,
- the lack of integration of human error reduction methods into the operations training process.
- the failure to consider human actions during FMEA reviews in the design control processes, and
- the assumption that forward flow was the only necessary requirement to ensure AFW system availability without adequate consideration of intervening operator actions.
Corrective Action Synopsis:
- Revise the EOP validation process to ensure that appropriate initiating events are evaluated.
- Complete the analysis portion of the PRA model review to identify any other risk significant vulnerabilities in the current EOPs.
- Review the operator actions specified in AOP-5B to determine if they should be included in applicable EOPs to ensure timeliness of the actions, and initiate revisions as required.
- Provide Operations and Training with an updated list of high-risk human error events and human error reduction methods used in evaluating operator actions in the PRA model.
- Review EOPs and AOPs containing high-risk human error events against human error reduction methods used in the PRA model and revise where appropriate to achieve significant CDF risk reduction.
- Revise OM 4.3.1, AOP and EOP Writers' Guide, to incorporate human error reduction methods used in the PRA model that can significantly reduce CDF risk.
- Review operator training materials and methods associated with high-risk human error-events against human error reduction methods used in the PRA model and revise where appropriate to achieve significant CDF risk reduction.
- Revise operator-training procedures to incorporate human error reduction methods used in the PRA model that can significantly reduce CDF risk.
- Revise the AFW PRA model to accurately reflect system performance.
- Review the description of the AFW recirculation line function in the FSAR, DBD-01, and the IST Program for consistency and accuracy, and initiate revisions as required.
- Revise the design input checklist to include consideration of human action induced failure modes.
- Evaluate if an Engineering Supplemental Guideline is the appropriate procedural method for controlling PRA updates, or if a higher tier document such as a Nuclear Procedure should be used considering the interfaces involving other departments.
Initiate any procedure changes resulting from that evaluation.
4
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air
- Revise the procedure governing PRA updates to include identification of the formal methods to be used for providing information to other groups. Use of existing processes, such as training work requests and procedure feedback forms, should be used whenever possible.
II. Event Narrative In June, 2001 the PRA group was reviewing and revising the AFW portion of the PRA model. During this review it was discovered that the minimum flow recirculation valves were not modeled within the PRA. Therefore, a failure modes and effects analysis was performed to determine potential failure modes. A discussion was held with past operations personnel about how the system was operated within the AOPs and EOPs. It was then determined that upon a complete loss of instrument air, the operators may use the EOPs and stop AFW flow by closing the discharge MOV or the flow control valve.
However, since the recirculation valve fails closed on a loss of instrument air, the AFW pump would not have adequate recirculation flow. This issue was discussed with a design engineer who informed the PRA group that the AFW pumps could be damaged in a short period of time without adequate recirculation flow.
This issue was then discussed with Operations Training personnel who reviewed the EOPs and discussed what operator actions would be. The operator actions were also confirmed with an Operations crew. The actions assumed were that upon a complete loss of instrument air, entry would be made into EOP-0, Reactor Trip or Safety Injection, and then into EOP-0. 1, Reactor Trip Response. Steps in these procedures would ensure that at least one AFW pump was available. In EOP-0. 1, if S/G level is high the operator is directed to STOP flow. If flow were stopped, by closing the discharge valve, the AFW pump would fail due to lack of minimum flow caused by the recirculation valve failing closed. The potential exists that this same evolution could be repeated on additional AFW pumps. Since this is a dual unit event with both units in a similar configuration, the same problem could also happen on the second unit.
It was noted that AOP-5B, Loss of Instrument Air, had a specific note to gag open the recirculation valves, but the information was located well into the procedure and timing showed that it would not be adequate to preclude closing the discharge valves. PRA personnel understood that this failure mode had the potential to be risk significant even though the actual significance was not known since the PRA model development was not yet completed. PRA personnel initiated CR 01-2278 on 7/6/01 to document this problem and identify potential corrective actions to place steps addressing the need to gag the recirculation valves open earlier in the sequence of AOP-5B. It was assumed that the AOP was sufficient to address the concern, but the timing of the action could be improved to ensure that the action would be successful.
An action item was created on 7/10/01 for the Operations Procedure group with a recommendation to move the step (AOP-5B step 24) to a more prominent position in the procedure and consider using a foldout page. The action item priority was set at 4 and the due date was established as 8/21/01. Discussions were held between PRA and 5
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Operations personnel and it was expected that a PRA group evaluation to determine the significance of the issue would be completed by 8/20/01. Initial Operations review of AOP-5B indicated that the procedure was laid out in a priority to restore instrument air, which is the correct response for that procedure. The evaluation of the risk significance of the as found configuration of the procedure is dependent on quantifying the entire PRA model. This was not completed until October, due to the complexity of developing a complete two-unit model. The original model used a single unit and simplified common systems. The PRA group informed Operations on 8/20/01 that the evaluation was not completed as expected and additional time was required to evaluate the actual significance and the type of action that should be done. At that time modifications and procedural changes were being considered.
The PRA group completed some preliminary modeling on 10/19/01 that indicated the potential for a high risk was involved and informed Operations that the AFW pump recirculation valves should be procedurally addressed. Based on further discussion, it was decided that a change to the Alarm Response Procedure for instrument air low header pressure (ARP C01 A 1-9) could address the concern. The PRA group was to submit a procedure feedback form for the desired change. The original action item was closed on 11/14/01 and a new action item was created on 11/14/01 to track the changes to the ARP and assigned to Operations. Operations discussed the request with PRA personnel and gave the new action item a priority of 3 with a due date of 12/26/01, based on expected completion of the PRA model and Safety Monitor update in December.
During that discussion some concerns were raised by Operations about the adequacy of procedural changes to address the issue. Specifically, the concern was that the ARP may not be the most effective way of protecting the AFW pumps during high activity in the Control Room, i.e., the loss of instrument air may not take priority and the ARP may not be referred to.
Additional discussions took place between Operations, PRA and a design engineer concerning the appropriate corrective actions and what risk might be involved if the procedural remedy was not completed or was inadequate. On Monday, 11/26/01, the PRA modeling adjustments were completed and a factor of 2.3 risk increase in Core Damage Frequency (CDF) was identified, which is considered high. Additional discussions took place between Engineering and Operations to determine further actions that may be appropriate.
A meeting between Operations and Engineering was held at 1300 on Wednesday, 11/28/01, to discuss significance and actions. During the discussion it was discovered that the loss of instrument air was more than just a random loss, a loss of offsite power (LOOP) or other events could also initiate the event. A re-evaluation of risk including the LOOP event resulted in an estimated factor of risk increase of 4 to 5 in CDF.
Operability was also discussed. It was concluded that there was no operability concern because no equipment degradation, failure, or non-conformance had been identified.
Regardless, the level of concern was great enough that further prompt actions were felt to be justified. The Design Engineering Manager briefed the Operations Manager on the 6
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air situation later that afternoon. The Operations Manager also updated the Plant Manager on the situation.
On Thursday morning, 11/29/01, the Operations Manager briefed the NRC Resident Inspectors on the issue and informed them that we were evaluating this apparent vulnerability and the risk significance. Operations decided that use of temporary information tags and briefing of all watch standers would be an important step to reduce risk; an evaluation of possible procedure enhancements was also initiated. At 1000, PRA personnel briefed the STA and Shift Manager on the issue and discussed potential wording for temporary placards to be placed on the control panels.
At 1100, PRA personnel discussed potential reportability concerns with Licensing. It was not clear if this issue was reportable because it involved a procedure and was not an equipment issue - additional discussions were needed. At 1130, PRA personnel briefed the NRC Resident Inspector on the issues and answered questions regarding risk impact and human error probabilities. During the afternoon, Licensing and Engineering personnel evaluated the reportability aspect further. It was concluded that the conservative decision would be to report the issue, even though a specific reporting criteria could not be identified. At 1445, PRA personnel initiated Condition Report 01 3595 and brought it to the Work Control Center for SRO screening at 1538. The Operations Manager took part in discussions involving operability and the need for an Operability Determination (OD). Since the issue identified in CR 01-3595 did not affect equipment, the decision was made that an OD was not required; however, the details of those discussions were not captured in either the CR or the screening comments. The SRO screening was completed at 1553 with the event determined to be reportable as a procedural inadequacy and not requiring an OD.
At 1520, the oncoming crew was briefed on the concerns of this potential event and temporary information tags were placed adjacent to the controls for 1/2P-29 and P-38 A/B that provided a reminder of the minimum flow requirements for each AFW pump.
At 1700, the Operations Manager provided the Plant Manager with an update on the issue. At 1705, Event Notification EN 38525 was made to the NRC via the ENS phone.
(See Section V. for details)
On Friday morning, 11/30/01, the Licensing Manager received a phone call from the acting NRC-NRR Project Manager for Point Beach, concerning confusion over the event notification. A return conference call was made with Engineering personnel to address NRR questions. A decision was made to provide a supplemental event notification providing additional details. The Operations Manager had additional conversation with the NRC Resident Inspectors and concluded that to formally document the operability of the AFW system, an OD would be initiated to capture the discussions held during the previous 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Operations requested that Engineering provide an OD and informed the Shift Manager that it was expected to be completed that afternoon. At Noon, the Operations Manager met again with the NRC Resident Inspectors and their supervisor to address NRC concerns regarding AFW operability prior to 11/29/01 and in its current 7
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air configuration. The Plant Manager and Operations Manager had a conference call with NRC Region III to discuss operability of the AFW system.
At 1400, a simulator scenario was run to obtain information on plant response to a loss of offsite power coincident with a rapid loss of instrument air pressure. Additional scenarios were run on 11/30 and 12/1.
At 1645, temporary procedure changes were completed for EOP-0 and EOP-0. 1 to reflect the guidance provided earlier to operators via the temporary information tags.
At 1700, the Plant Manager was informed that a 5 person NRC incident investigation team would arrive on 12/3/01. At 1746, a supplemental event notification was made to the NRC to clarify the discussion on the potential for an AFW system failure as described in the original event notification (EN 38525).
At 1755, Engineering completed Revision 0 of the OD that concluded that the AFW system was Operable but Non-Conforming. This was based in part on a statement in the FSAR that "each pump has an AOV controlled recirculation line back to the condensate storage tanks to ensure minimum flow to dissipate pump heat." The compensatory actions already in effect were listed in the OD as required actions. The Plant Manager and Operations Manager reviewed the OD content and then briefed the Senior NRC Resident Inspector. The OD was then brought to the Control Room and accepted at 2015. On Friday evening, just-in-time (JIT) training was provided to the swing shift crew on the simulator on this event; JIT was also provided to the mid-shift crew on the simulator prior to assuming the watch.
On Saturday, 12/1/01, at 0720 JIT was provided to the oncoming dayshift crew on the simulator prior to assuming the watch. A staff meeting was held from 0930 to 1200 to prepare for the NRC inspection team. A revised OD was prepared at 1500 to expand the discussion on AFW pump motor duty cycles. The Control Room accepted it at 1515.
On Monday, 12/3/01, CR 01-3595 was screened and assigned to Engineering to perform an apparent cause evaluation. Another meeting was held from 1000 to 1200 in preparation for the NRC inspection team. At that meeting it was decided that a root cause evaluation would be a more appropriate response to this event. The Plant Manager approved the RCE Charter on 12/4/01.
The NRC Inspection Team arrived onsite on 12/3/01 and conducted a technical debrief on 12/7/01. A preliminary exit meeting was held on 12/13/01.
An expert on Human Error Probabilities was brought onsite on 12/4/01 to help quantify the risks associated with the procedural weaknesses that were identified. His evaluation estimated that there was about a 50% chance that the operator would shut the discharge valve and fail to recognize that the minimum flow recirculation valve did not open when flow was stopped as S/G levels rose above 65% on the narrow range.
8
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air On 12/4/01, CR 01-3633 was initiated by Engineering on the ability of the Motor Driven Auxiliary Feedwater Pumps (MDAFWP) to respond to an Appendix R fire coincident with a loss of offsite power and instrument air because of a lack of documentation related to the potential for closure of the recirculation valves due to loss of instrument air. CR 01-3648 was initiated by Engineering on 12/5/01 on the same issue when four specific fire zones were identified as having the potential to cause an AFW pump auto-start coincident with discharge and recirculation valve closure, resulting in pump damage. An OD was completed for CR 01-3468 on 12/7/01 that concluded the MDAFW Pumps were Operable but Non-Conforming, with the required compensatory measures of performing hourly fire rounds in the specified fire zones. An event notification on this issue was made at 1926 on 12/05/01 (EN #38541)
Permanent revisions to EOP-0 and EOP-0.1 were implemented on 12/14/0 1. Additional changes were made to those procedures and ECA-0.0 on 12/20/01.
III. Extent of Condition Assessment The root cause of this event is attributed to a weakness in the original EOP validation process where the effects of a loss of instrument air were not adequately evaluated by the scenarios used in that process. The plant response under a loss of instrument air condition is sometimes different than under normal accident conditions. Because of this event, the previously held belief that AOP-5B, Loss of Instrument Air, adequately directed the required operator actions, was found to be faulty because actions were required while in an EOP, prior to performing AOP subordinate actions. This event identified a specific concern with AFW control, but there may be other operator actions that are unique to a loss of instrument air condition that are not adequately considered in the EOPs. A review of the EOPs is needed to identify the impact a loss of instrument air condition may have on systems other than AFW and if the EOP directed operator actions associated with those systems are adequate.
The original PRA model fault trees evaluated system performance primarily on functions described in design documents and did not adequately consider human actions. The current PRA model review uses a methodology that integrates system performance with potential human actions to obtain a spectrum of plant responses. This more rigorous approach should identify any other assumptions used in risk-significant systems that have not adequately considered human actions. This review will also identify any risk significant vulnerabilities in current emergency operating procedures.
The lack of integration of human error reduction methods into operations training and emergency procedure development processes may allow other weaknesses to exist where PRA risk reduction has not been optimized. Procedures and training associated with high-risk human error events should be reviewed against human error reduction methods to ensure that reasonable risk reduction has been achieved.
9
IncreasedCDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air IV. Nuclear Safety Significance Any complete loss of IA for a significant time is expected to result in a reactor trip and an AFW start signal due to a loss of normal feedwater (the normal feed water regulating valves fail closed on loss of air). Under this postulated condition, all components of the AFWS are now and continue to be fully capable of performing their design functions supporting automatic starting and supplying sufficient flow to the steam generators to mitigate any transient or accident by removal of decay heat. It is the continued function of the AFWS, in response to directed operator actions to control AFWS flow, and the lack of specific guidance contained within the EOPs regarding a loss of IA, that is the issue identified in this report.
A PRA assessment of the possible failure modes and effects associated with an IA failure identified a previously unrecognized vulnerability. This failure would have been caused by a combination of a design limitation, a specific sequence of postulated operator actions, and a lack of clear guidance within the EOPs. This combination could result in failure of one or more of the AFW pumps due to aggressive AFW flow reduction (as may be expected in response to a steam generator overfill or RCS over cooling) after automatic system start and flow had been established. The likelihood of success or failure in the postulated scenario is highly dependent upon plant transient response (which may vary with the nature of the initiating event, initial power levels, etc.) and operator response. Operator response is highly dependent upon prior training, procedural usage, system knowledge and awareness, experience, and other human effectiveness (HE) factors. It should be noted that a control board alarm is provided (Instrument Air Header Pressure Low) to alert the operator to the existence of an initiating condition for this event and that established plant procedures direct the restoration of IA (both Emergency Operating Procedures and Abnormal Operating Procedures), and the manual gagging open of the minimum flow recirculation valves in the event that IA cannot be promptly restored (AOP-5B). PBNP has experienced partial losses of IA, including one event involving the loss of all off-site power and another involving a low IA header pressure alarm following a reactor trip. In each of these cases the operators demonstrated the ability to cope with the loss of IA casualty and recover IA header pressure before it had an adverse affect on plant equipment or response.
Preliminary PRA results show that the vulnerability described in this report, prior to the procedural changes, was potentially risk significant. Although the initiating event frequencies are low to moderate, the unrecoverable IA scenario was risk significant due to the consequences of a total loss of all AFW pumps requiring feed and bleed without the pressurizer PORVs. The risk results are highly dependant upon human interactions. PBNP operators are trained on AFW system operations and have experience with degraded IA scenarios. Because of this training and experience, it is reasonable to assume that operators would have successfully handled this combination of conditions in the unlikely event that it would have occurred.
10
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Although the AFWS met, and continues to meet, all of its design and licensing requirements, the initiating event of a loss of IA, in conjunction with a misaligned procedure, had the potential to affect redundant trains of the AFWS, a safety related system. Since it could be postulated that the same operator action could have impacted all the AFWS pumps, the result could be the complete loss of the AFWS safety- related function. Accordingly, this event has also been identified as a possible safety system functional failure (SSFF).
V. Report to External Agencies Condition Report 01-3595 was initially brought to the PBNP Work Control Center for an SRO screening at 1538 on November 29, 2001. During this screening, a determination was made that this event should conservatively be reported to the NRC in accordance with 10 CFR 50.72(b)(3)(v) as a condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to:... (D) Mitigate the consequences of an accident. This is an eight-hour non-emergency notification. During the discussion of reportability it was noted that 10 CFR 50.72 Paragraph (b)(3)(vi) clarifies paragraph (b)(3)(v) by noting that, "Events covered in paragraph (b)(3)(v) of this section may include one or more procedural errors, equipment failures, and/or discovery of design, analysis, fabrication, construction, and/or procedural inadequacies." The last of these items appeared as though it may be applicable in this situation. The following elements also entered into the notification determination:
" NUREG-1022 notes that the level of judgment for reporting an event is a reasonable expectation that the event or condition could lead to preventing fulfillment of a safety function. The intent of these criteria is to capture those events regardless of whether there was an actual demand.
"* Example (20) in NUREG-1022 Page 64 directs that system interactions that are found as a result of ongoing routine activities may be reportable.
"* When in doubt concerning issues of reportability, it is our policy (consistent with the directions in NUREG-1022) to make the report.
The NRC notification was made using the Emergency Notification System (ENS) telephone at 1705 on November 2 9 th. Event number EN 38525 was assigned to this notification.
On the morning of November 30th, as a courtesy, the PBNP acting Project Manager at NRC-NRR was telephoned to advise him of the event notification. He had several questions that were answered in a follow-up call later in the morning. At 1746 on November 30, 2001, the ENS event notification was supplemented to further clarify the discussion of the specific failures postulated and to reiterate that the potential failure would involve only the AFWS pump recirculation valves.
11
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air A Licensee Event Report (LER 266/2001-005-00) was submitted within 60 days of this event as required by 10 CFR 50.73.
VI. Data Analysis Information & Fact Sources Document Review Results Modifications
- M-623 / 624 - TDAFP Alternate Bearing Cooling Supply, issued 911/79
Description:
In response to an NRC Evaluation of the AFW system, this MR provided a cooling water supply to the TDAFP bearing coolers that is independent of AC power. The supply is taken from the diesel powered Fire Water system.
Evaluation: The MR enabled the TDAFP to cope with a SBO. Since the TDAFPs are the only pumps available for decay heat removal during the first hour of the SBO, operation of the pumps at low flows requiring recirculation flow is not probable.
Therefore, it is unlikely that this MR would have identified the need to have the recirculation valve open with a loss of IA.
- IC-274 - Modify Logic To Keep Recirculation Valves Open, issued 2/1/80 (Canceled 8/32/82)
Description:
Modify the control scheme of the recirculation valves to keep valves normally open. The reason for this change was to provide a path for the first off check valve leakage back to the CST. This change would prevent the leakage from lifting the pump suction relief. The modification was canceled since it was only solving a symptom of the real problem; check valve leakage. The modification still intended to have the recirculation valves fall to the shut position.
Evaluation: The modification was attempting to resolve symptoms associated with check valve leakage. The modification would not have permitted a continuous recirculation path. Therefore, it is not likely that this modification could have identified or resolved the current issue.
- MR 83-104 - AFW System Discharge MOV Controls, issued 8/1/83
Description:
The MDAFP discharge valves were modified to provide automatic actuation of the valves similar to the automatic starting logic for the MDAFPs.
Evaluation: The MR was a response to NUREG-0737 to ensure AF is provided to the S/Gs without operator action. The recirculation valves either failed on loss of air or shut as flow to the S/G increased therefore, these valves were already in compliance with the NUREG. This MR deals with eliminating an operator action and the design limitation of the recirculation valves is not introduced until an operator action is taken 12
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air (i.e. throttling AF discharge flows). Therefore, this MR had a low probability of identifying this issue.
- MR 88-099 - AFW Pump Mini-Recirculation Line Improvements, issued 7/7/88
Description:
In response to NRC IE Bulletin 88-04 and GL 89-04, the recirculation line flows were increased to prevent pump degradation due to hydraulic instability.
The minimum pump flow prior to this MR was 30 gpm. The MR increased this to minimum flow to 70 gpm for the MDAFPs and 100 gpm for the TDAFPs. The MR did not change the operation of the recirculation valves.
Evaluation: The MR was addressing the minimum flow requirements for the AF pumps. A detailed review of all operating modes of the pumps may have been able to identify that the pump discharge flow could not be reduced below the new values with the recirculation valves failed shut. The focus was probably on the fact that flow to the S/G required for accident mitigation provided more than the necessary pump minimum flow requirements. Therefore, the MR only addressed the increase in recirculation line capacity.
MR 92-091/092/093 - IST Testability of AF Recirculation Line AOVs, issued 6/19/92
Description:
In order to simplify stroke testing of these AOVs, bypass valves were installed around the control solenoid.
Evaluation: The MR was small scope focusing only on the need to bypass the solenoid to allow stroke testing of the valve. At this time, the IST Program had already identified the shut position as the safety related position for these valves. The scope of this MR was not a likely opportunity to identify the issue.
- MR 97-038*A/B - MDAFP Discharge Pressure Control Valve Backup Nitrogen Supply and Cable Separation, issued 4/15/97
Description:
The MR prevented redundant failures of the AOVs (common electrical fault) and pump runout due to loss of 1A (Ref. LER 97-014-00). MR 97-038*B provided physical separation for electrical cables associated with the discharge pressure control valves (AF-4012 and AF-4019) and their associated control components. MR 97-038*A installed nitrogen bottles as a backup pneumatic supply.
The design description for MR 97-038*B states that one of the functions of the discharge AOVs is to allow enough flow to the S/Gs to cool the associated pump during a scenario when pump recirculation is required and the associated recirculation valve fails closed.
Evaluation: The intent of the MR was to prevent pump runout due to a failed open discharge AOV as a result of a loss of instrument air and low S/G pressures. It appears the focus of the MR was to ensure control capability of the discharge pressure control valves. The MR does recognize that the discharge AOVs are needed to provide pump cooling flow if the recirculation valves fail shut. This appears to support the idea that the flow to the S/G is the safety related function and failure of 13
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air the recirculation valves is acceptable. Recognition that the discharge AOV is needed for pump cooling may also suggest that the author knew that AF discharge flows could be throttled in certain accident conditions. If this were true, it would have been prudent for the author to ensure procedural guidance was in place at this time to ensure minimum pump flow was maintained. It appears this MR had a high probability of identifying the current issue. The designer did not do a complete failure mode analysis of the system during installation of this new modification. The ability to throttle the pump discharge flow during a loss of instrument air provides an increased opportunity for operator action to cause pump damage.
Procedures AOP-5B, Loss of Instrument Air: This AOP was first issued on 5/2/86. The procedure contained an "immediate action - manual" step (step 6.0) emphasizing the understanding that AOVs may not function depending on IA header pressure and referred the operator to Appendix A for individual system information. Section R of Appendix A was for Auxiliary Feed, and listed the AFW pump recirculation valves as failing shut with a corresponding note on manual gag override. The additional information in that section included monitoring of AFW pumps for sufficient flow to prevent overheating due to no "minirecirc", and to use the manual gag on the "minirecirc" valve to provide maximum recirculation unless continuous feed was verified through each AFW pump. The procedure content remained essentially the same until Revision 11 was issued on 9/26/97, which moved time critical actions from the appendices into the main body of the procedure. At that time a specific step (step 21) was added for control of AFW flow. A note was placed before that step informing the operator "the manual gag on each AFW pump mini-recirc valve must be used to provide minimum recirc flow if continuous flow through the pump can NOT be verified." The current procedure content is equivalent.
Evaluation: The AOP contained sufficient information identifying the correct failure mode of the AFW pump recirculation valves on loss of LA, the required manual actions, the concern with pump overheating, and the need to monitor pump flow. The content of the note that directed the operator to continuously monitor pump flow and use the manual gag if flow could not be verified, met the requirements of OM 4.3.1 for note content. OM 4.3.1 allows notes to advise on actions to be taken in the event of changing plant conditions (see discussion on OM 4.3.1 below).
EOP-0.1, Reactor Trip Response: Emergency Operating Procedures, specifically EOP-0. 1, is the PBNP procedure that would be used in the event of concern; EOP-0. 1 is based on a WOG ERG. Neither EOP-0. 1 nor the WOG ERG has ever addressed the function of the AFW mini recirc flow valves. EOP-0. 1, in one step (step 3),
directs the operator to use main feedwater regulator bypass valves for feed flow control. As a response-not-obtained (RNO) action, AFW use is directed, and has a substep to "verify AFW alignment". The mini recirc valves are not included. A NOTE containing the flow rate at which AFW pumps will trip due to over current induced by pump runout precedes the feedwater flow control step. In another step 14
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir (step 4), the operator is directed to stabilize S/G level but is not provided details on how to accomplish the task. The RNO action specified is to "stop feed flow to that S/G." This action applies whether feed flow is being provided by main feedwater (via the bypass) or by the auxiliary feedwater pumps. There is also reference to controlling feed flow in step 1 related to maintaining RCS temperature.
The steps on S/G level stabilization and feed flow control have basically existed since the EOPs were created in July 1985. They have never addressed the impact of loss of instrument air on the mini recirc valves. The effect of excessive AFW flow (i.e.,
pump runout) was introduced in about 1995.
The WOG ERGs for Reactor Trip Response do not address loss of instrument air, nor do they specifically address AFW pump mini recirc flow capability. The WOG considers such aspects to be plant specifics, to be addressed by the owner. The original WOG developmental guidance from 1984 contains little information on what (plant specific) systems should be addressed, or how. This trend continues through 1997, Rev IC, which does generically identify that plant specific electrical loads (which covers one major cause of IA loss, compressors) should be a plant specific list. AFW and S/G level control specifics are not addressed. The WOG has always recognized that plant specific information is needed in EOPs and the Deviation and Background Document concepts were provided to manage such information.
At various times throughout the history of EOP-0.1 the importance of AFW in general (but not mini recirc flow in particular) has been recognized at PBNP. For example in Rev 7, 10/11/91, checking AFW actuation was step number 1 of EOP-0.1.
AFW pump runout concerns were added in 1995. Loss of IA due to electrical bus availability was addressed similarly to AFW. For example in Rev 11, 11/22/94 (prior to the development of AOP-18A and -18B for train specific equipment operation)
Appendix A to EOP-0. 1 contained a list of Priority Electrical Loads, which included an IA compressor. Appendix A was deleted when AOP-18A & -18B were created.
Evaluation: PBNP EOP-0.1 is based appropriately on ERG guidance. The ERGs consider that plant specific information may need to be included in EOPs and provides means and mechanisms to document the same (Background and Deviation documents). The verification and validation (V&V) process described by the ERG procedure development process is intended to identify plant specific needs to be included in the plant specific EOPs. PBNP did not include operator guidance in EOP-0.1 on AFW minimum recirc flow under a loss of IA condition.
OM 4.3.1, AOP and EOP Writers' Guide: The Writers' Guide contains the usage rules for notes and cautions that specify (in part):
"* A note is used to present advisory or administrative information necessary to support performance of the subsequent step(s).
"* Each document should provide enough information to accomplish the purpose of the document without relying on information contained in notes or cautions.
15
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air
- Notes and cautions should be declarative statements of fact and not commands or action statements unless they are advising on actions to be taken in the event of changing plant conditions.
The references listed in OM 4.3.1 were reviewed with the following results:
"* NUREG-0899, Guidelines for the Preparation of Emergency Operating Procedures - 8/82: Note statements provide operators with supplemental information concerning specific steps or sequences of steps in the EOP. These statements should provide operators with enough information, and be located so as to ensure that they can easily relate the note to the step or steps to which it applies. Because they are supplemental, notes should not direct operators to perform actions. (p24)
"* NUREG-1358, Lessons Learned From the Special Inspection Program for Emergency Operating Procedures - 4/89: In many cases action statement were found embedded in notes and cautions. Again, this increases the chance that the step will be overlooked and that an error will occur. (p4) Cautions and notes are not intended to direct operator action, but rather to warn of possible consequences or to provide supplemental information to the procedure steps. Inclusion of actions in a caution or note can be disruptive and confusing to an operator. More importantly, the action could be entirely overlooked if embedded in a caution or note. Any cautions or notes containing operator actions, including conditional actions or transitions, should be restructured so as to provide an action step plus a caution or note. (pC-3)
"* NUREG-1358, Supplement 1, Lessons Learned From the Special Inspection Program for Emergency Operating Procedures - 10/92: Cautions and notes:
notes (1) provide only supplemental information, and (2) no actions included.
(P16)
"* NUREG/CR-2005, Checklist for Evaluating Emergency Operating Procedures in Nuclear Power Plants - 4/83: Do explanatory notes avoid the use of action statements? (Statements directing personnel to perform actions must not be imbedded in explanatory notes.) (p7)
"* PBNP Procedures Writers' Guide - 11/27/00: Cautions and notes shall NOT direct or infer actions. All required actions shall be stated in action steps. (p50)
This procedure is not applicable to the AOPs or EOPs.
"* WOG ERG Writers Guide - 7/1/87: Because the present action step wording is reduced to the minimum essential, certain additional information is sometimes desired, or necessary, and cannot be merely included in a background document.
This non-action information is presented as either a NOTE or a CAUTION. (p22)
NOTE is used to present advisory or administrative information necessary to support the following action instruction. A CAUTION or NOTE may also be used to provide a contingent transition based on changes in plant condition. As a general rule, a CAUTION or NOTE will not contain an instruction/operator action. However, passive action statements in CAUTIONS or NOTES, which typically contain the words should, may or must, may be appropriate under certain conditions. An example is when continuous monitoring of a specific plant condition and an associated action is required.
16
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir Evaluation: OM 4.3.1 guidance on the content of a note is consistent with the WOG ERG Writers Guide, but contradictory to all of the other references cited. Some statements within the OM contradict others; specifically, the statement that "Each document should provide enough information to accomplish the purpose of the document without relying on information contained in notes or cautions" contradicts the intent of "unless they are advising on actions to be taken in the event of changing plant conditions."
Training Continuing Training: The overall content of the continuing training program is determined based on a two-year cycle. Presently the 2001/2002 LOR (license operator requalification) Long Range Training Plan is in effect. The Long Range Plan concept is very organized and structured with respect to content of the topics to be covered; it has been used since the mid- 1990s. The content of the Long Range Training Plan is based, in part, on PRA information and includes a focus on systems with high safety significance. Prior to the Long Range Plan implementation, the content of LOR training was determined in a much less rigorous manner and on a much shorter time frame, typically on a 6 week-to-6 week cycle. Content was based on needs suggested by students, operations management and instructors plus inputs based on current events (such as design change implementation, procedure changes, plant and industry events).
The 2001/2002 plan contains a number of topics pertinent to the issue of concern. The tasks for Loss of Instrument Air and Loss of Offsite Power were covered as well as a system review of Auxiliary Feedwater. The training devices used by instructors to cover the topics are LPs (Lesson Plans) and SGs (simulator guides). Both these devices present information in outline form, containing topical areas to be covered.
The LPs are primarily oriented for classroom environment, whereas SGs are targeted for the simulator, mostly the instructor/ simulator operator. LPs clearly identify references and materials to be used as handouts. Typical support documents are drawings, procedures and OE documents. The LPs used in continuing training are the same LPs used for initial training. Training personnel indicated that LPs and SGs are reviewed prior to use and, to the best ability of the individual trainer, are updated to be current.
Initial Operator (CO and SRO) Training: The highest-level document in Initial Training is the Program requirements (TRPR). They are position based. For example TRPR 18 is Control Operator Trainee. The TRPRs are primarily administrative documents rather than technical. The TRPRs do identify the Training Courses (TRCRs) that comprise the Program. The TRCRs are a little more technical than Programs in that they identify some general areas of knowledge that the trainee needs to cover. For example, under TRPR 18, two of the courses are TRCR 52, Secondary Systems and TRCR 55, Integrated Operations. The TRCRs identify LPs. The LPs are the same as those used in continuing training. Some of the LPs specific to the event 17
IncreasedCDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air are LP 0169 AFW system, LP 0405 Reactor Trip or SI Response (which includes EOP 0.1), LP 0338 Instrument and Service Air (which includes AOP-5B) and LP 2439 Secondary Coolant System Malfunctions (AFW is one of those).
Evaluation: LPs contain enough specific information about auxiliary feedwater and instrument air systems to accurately describe system operations, causes and effects.
Training documents do not contain extremely specific details on specific evolutions.
For example, the specific method for controlling steam generator level as directed in EOP-0.1 in concert with compounding events such as loss of IA, is not covered nor is the need to locally gag an AFW pump mini recirc valve upon loss of instrument air.
Instructors review material to be taught in advance and are able to make changes in course content in order to add information, including current events and to change areas of emphasis. The Simulator Guide topics used in continuing training appear to be marginally related to the topic area they are listed under. PRA and human performance information is not included in LPs. PRA and CDF values are used as input to select the content of the Long Range Training Plan for continuing training.
Other Documents DBD-01, Auxiliary Feedwater System Design Basis Document: Revision 0 of DBD-01 was issued on 4/4/94. In Section 4.8, AFW Pump Recirc Flow Control Valves, there was a statement under Safety-Related Functions that "These valves shall open automatically and remain open to provide a recirculation flowpath from AFW pump discharge to the CST when flow in the AFW discharge line is insufficient to prevent pump damage." The reference cited was MR 88-099. Section 4.8.4 discussed a potential worst-case flow condition with both the recirculation valve closed (due to loss of IA) and the associated discharge MOV closed (single active failure), but concluded that this was outside the system design and licensing basis.
This worst-case concern was based on NUREG-0800 assumptions, but was not considered applicable since PBNP had not incorporated NUREG-0800 into its licensing basis.
Revision 1 of DBD-01 was issued on 3/31/00. One of the major changes included was "Deleted safety-related function to OPEN for mini-recirculation valves for AFW pumps." The worst-case flow condition discussion remained in the DBD.
Evaluation: The basis for including an OPEN safety-related function for the recirculation valves in Revision 0 was cited as MR 88-099, the modification that increased the recirculation flow orifice size for AFW pump protection. A review of the modification paperwork did not identify any statement declaring a safety-related function for the valves to OPEN. A review of the DBD validation documentation indicated that in-service testing of the valves checked recirculation valve position.
Testing of the recirculation valves in the OPEN direction was added to the IST Program in 1991. (See discussion of IST Program below.)
18
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Revision 1 of DBD-01 deleted this OPEN safety-related function. This appears to be a result of actions coming from CR 97-3363 (discussed later). Testing of the valves in the OPEN direction was deleted from the IST Program on 9/30/98, also as a result of CR 97-3363. Overall, the basis for adding and deleting this function to the DBD was not well documented or justified.
EOP Verification and Validation (V&V): The original EOPs issued in 1985 were verified by a multi-disciplined verification team using an approved procedure with a detailed checklist of attributes to be evaluated. That effort generated over 2500 discrepancy sheets and involved a series of more than 40 team meetings over a period of several years. The discrepancy sheets generated for EOP-0. 1 did not raise any concerns with the step for controlling feed flow or stopping feed flow to a S/G if a level increase above the desired value occurred.
The validation process involved a WOG review of the basic version of the ERGs at the Calloway simulator in 1982 and on the Revision I ERGs at the Seabrook simulator in 1983. Early drafts of some of the plant specific procedures were taken to the Zion simulator in March and April of 1983, which generated many suggested procedure changes. The procedures were then put through the previously described verification process. Following this, the procedures were used by operating crews at the Kewaunee simulator (8/84-11/84). Each crew spent a week mitigating accidents using the procedures. No concerns were raised regarding the actions to control feed flow or stop feed flow if S/G level increased above the desired level range. Finally, a portion of the detailed control room design review was expanded to provide another validation of the EOPs. A full size photographic mock-up of the PBNP control room was created and fourteen scenarios (increased from the typical 5 or 6) were evaluated in an attempt to ensure that every EOP was used. Operators performed walkthroughs of the EOPs during these scenarios, which were also videotaped for later review, and then interviewed for their comments (1985). Again, no concerns were raised regarding the actions to control feed flow or stop feed flow if S/G level increased above the desired level range.
The EOP V&V process was also part of a NUREG-0737 Supplement 1 (GL 82-33) commitment. The EOP procedure generation package (PGP) was submitted to the NRC on 6/1/84. The NRC responded with a draft SER on 5/7/87 that found the PGP to be unacceptable. The PBNP revisions to the draft SER were submitted back to the NRC on 11/10/87, addressing each of the identified concerns. The NRC issued the final SER on 4/9/90 that contained additional programmatic improvements identified by the staff. The SER transmittal letter also referred to the June 1989 NRC Inspection of the EOPs and recommended that PBNP consider both the results of that inspection and the SER discussion and utilize them as appropriate in the next major revision of the EOPs. Current procedures governing the EOP V&V process are OM 4.3.2, EOP Verification Procedure, and OM 4.3.3, EOP Validation.
Evaluation: During the development of the PBNP EOPs from the WOG ERGs, information was to be included to address differences between the reference plant 19
IncreasedCDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir used by WOG and the Point Beach plant. Following development of those procedures, verification and validation reviews were applied to ensure the adequacy of those procedures. Validation is the process of evaluating the EOPs for usability by the operators and operational correctness (e.g., compatibility with plant hardware and control board layout). EOP-0. 1 was operationally incorrect for a loss of IA condition.
Therefore, it was the validation step in the EOP development and implementation process that failed.
EPRI Report TR-100259, An Approach to the Analysis of Operator Actions in Probabilistic Risk Assessment - 6/92: This document is used by the PRA group in evaluating human interactions for the probability of an error. It identifies attributes of certain failure mechanisms that influence the overall probability that the mechanisms will contribute to a human interaction (HI). One mechanism, Relevant Step in Procedure Missed, has four attributes that are considered and evaluated in a decision tree:
"* Obvious vs. Hidden: Is the relevant instruction a separate, stand-alone numbered step, in which case the upper branch is followed, or is it "hidden" in some way that makes it easy to overlook, e.g., one of several statements in a paragraph, in a note or caution, or on the back of a page?
"* Single vs. Multiple: At the time of the HI, is the procedure reader using more than one text procedure or concurrently following more than one column of a flowchart procedure?
"* Graphically Distinct: Is the step governing the HI in some way more conspicuous than surrounding steps?
"* Place Keeping Aids: Are place keeping aids, such as checking off or marking through completed steps and marking pending steps used by all crews?
A hidden step had a 10% probability of being missed, whereas a procedure step exhibiting the best of all four attributes had a probability of only 0.1%, a reduction by a factor of 100. The worst probability for an obvious step is only 1.3%, which is about a factor of 8 lower than a hidden step.
FSAR: The FSAR did not include a description of AFW recirculation line features until updates were made in 6/97 and 6/98. The 1997 update involved the addition of a paragraph describing the diversion of AFW flow via the recirculation line to the CST for a 3-minute period following pump start. This was an original design feature that had never been included in the FSAR description of the AFW system. The 1998 update was an extensive change resulting from the FSAR Review and Upgrade Project that provided a more detailed description of the AFW system and its licensing basis. This change added the wording that each pump had an AOV controlled recirculation line back to the condensate storage tanks to ensure minimum flow to dissipate pump heat. This change also revised the time period for AFW flow diversion during pump start from 3 minutes to 45 seconds.
- Individual Plant Evaluation, Revision 0 dated 6/30/93: The original IPE for Point Beach was developed from a snapshot of the plant and procedures as of 9/5/90. Many 20
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir of the success criteria for systems in the IPE PRA model were based on design basis assumptions. In the original PRA system notebook for Auxiliary Feedwater, it was recognized that the minimum recirculation flow valves failed closed on a loss of instrument air. However, this was not included in the PRA model as a failure mode for AFW because it was assumed that these valves failing to open did not result in pump failure. Assumption 22 in Section 4.6.7.1 of the notebook states:
The discharge lines of the AFW pumps have recirculation lines back to the CSTs.
These lines are normally isolated by AOVs that fail closed on loss of power or instrument air. Although they receive open signals upon a pump start and when pump flow is low, it is assumed that failure to open does not fail the AFW pump.
Failure of one of these AOVs in a full open position is assumed to fail the associated AFW train due to diversion of pump flow.
The potential to damage the AFW pumps with lack of flow was mentioned briefly in the notebook. In Section 4.6.2.2 on Support Systems, the following discussion is found under the "Instrument Air" heading:
The mini-recirculation valves on both the turbine-driven AFW pumps (AF-4002) and the motor-driven AFW pumps (AF-4007 and AF-4014) fail shut on a loss of instrument air. This could cause overheating of these pumps on low flow conditions with no recirculation flow available.
These two sections seem to contradict each other. However, controlling (reducing)
AFW flow was assumed to take place later in the transient so there was plenty of time for the operators to perform this action correctly. Again, there appeared to be an emphasis on ensuring that enough flow was available in the transient initially and it was not recognized how early in the event that AFW flow needed to be reduced to prevent overfilling the Steam Generators. This is evidenced by Assumption 13 where operator actions to control AFW flow later in the transient are discussed. No mention is made of ensuring a minimum flow path is available:
Operator actions to control AFW flow later in an accident sequence are not explicitly modeled in the AFW system fault trees. Operator actions are necessary to prevent the AFW system from overfilling the steam generators as their pressures decrease and AFW flow likewise increases. This was not modeled since there is a long time available and the function would be alarmed.
In addition, the operator would have to successfully supply an alternate source of water to the suction of the AFW pumps (not automatic) and then forget to control flow or check steam generator level.
It seems from these statements in the notebook that some injection flow was always assumed to be required. The need for the operator to shut off flow to the Steam Generators entirely from one or more AFW pumps at some time in the event was apparently not considered.
In Section 4.6.4.2 of the notebook, initiating event impacts on the system are discussed. Under the "Loss of Instrument Air" heading, only the discharge valves for 21
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air the motor driven pumps are considered. The closure of the mini-recirculation valves for the AFW pumps was not documented as a possible effect of the Loss of Instrument Air event:
A loss of instrument air will degrade the operators' ability to throttle the flow rates of that portion of the AFW system associated with the motor-driven AFW pumps.
The discharge pressure control valves, which are intended to limit flow to 200 gpm per pump, (AF-4012, 4019) are air-operated and would fail open on a loss of instrument air. Under this condition the operator is directed to use the turbine-driven pump to supply feed per AOP-5B, "Loss of Instrument Air" (Reference 4.6-12) or use the local gag to control AF-4012 and AF-4019 per 01 62A, "Motor-Driven Auxiliary Feedwater System (P-38A&B)".
The notebook also contains a discussion of potential common cause failures for the AFW system. This review did not identify the closure of each pumps minimum recirculation valve on a loss of instrument air as a potential failure mechanism.
However, this is consistent with the assumption that failure of these valves to open does not fail the AFW pumps.
Updates to the original IPE PRA model (1990) were based on snapshots of the plant taken in 1993 and again in 1996, and implemented a few years later (due to the long time required to perform the model update). The focus of these updates was to incorporate new plant-specific failure data and to incorporate model changes that reflected plant modifications. The PRA model update being completed this year is the first time since the original IPE effort that critical systems were examined from the ground up in a detailed review to ensure all failure modes are captured. This was accomplished in part by use of detailed failure modes and effects fault trees. Adding this detail was considered to be necessary at this point to make the model more flexible for risk-informed applications. It was the use of this approach that identified the concern with operator actions to control AFW flow.
IST Program: In December, 1990 the 3rd interval program (Revision 0) was implemented. There is a line item in the general valve section that states "Due to isolation of S/G by EOPs, it may be necessary for an operating pumps recirc path to be available." The testing to verify the open function was not included in the tabular section of the IST program that identified the actual testing to be done. A valve program relief request (VRR-28) was added to the IST Program under Revision 1 on 5/28/91 that described the recirculation valves function to be "These valves open to ensure minimum recirculation flow from the pumps to prevent pump damage." A cold shutdown test frequency was being sought.
The NRC issued a Technical Evaluation Report (TER) on 4/17/92 that denied the relief request because the valves had a safety function in the closed position and noted that the recirculation valves were not tested by the IST Program in the open position.
The TER referenced the VRR-28 function statement and went on to state "The program should be revised to address these valves' safety function in the open direction." PBNP responded to the NRC on 7/30/92 to clarify that the valves could 22
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir not be stroked except by use of hand wheels until modifications were made that allowed manual stroking using air. The response also stated "Since the AF pumps are capable of delivering feedwater at any steam generator pressure, the minimum flow valves are not required to open to protect the AF pumps under any anticipated accident conditions. The valves will, nevertheless, be stroke time tested in the open direction, as well as in the shut direction, once the modification to permit stroke time testing is completed." A follow-up letter dated 3/2/93, informed the NRC that the modifications would be completed by the completion of the spring 1993 refueling outage and VRR-28 relief request was being withdrawn. Revision 3 to the IST Program was implemented on 3/30/93 deleting relief request VRR-28.
On 10/15/97, CR 97-3363 raised a question about a discrepancy between the open function testing of the AFW recirculation line check valves (not in the IST Program) compared to the recirculation flow control valves (in the IST Program). The evaluation of this concern concluded on 2/5/98 that there was no safety related function for the recirculation valves or check valves to open, and the IST Program would be revised. Revision 5 of the IST Program was issued on 9/30/98 and deleted the open function testing of the recirculation flow control valves.
Interview Results Personnel Statements: Written statements were obtained from key personnel involved in the evolution of this issue covering the period of initial discovery to its reporting to the NRC. The information derived from those statements has been incorporated into the timeline included in Attachment B and involved the following personnel:
- PRA Engineer
- Design Engineer
- Design Engineering Manager
- Regulatory Compliance Engineer
- AFW System Engineer
- Operations Manager
- PRA Supervisor Interviews: Interviews were conducted with the following individuals to obtain additional information:
PRA Engineer: An interview was conducted with the PRA Engineer that identified the concern with operator actions to control AFW flow. That interview identified the following points:
- The PRA group reviewed the effect of the EOP change made (addition of foldout page information) but did not make recommendations on the best method of accomplishing the incorporation of that information. Use of the foldout page resulted in a reduction of the Human Error Probability (HEP) from 0.5 to 0.05.
23
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir
- Use of a foldout page is treated as a continuous step with some additional credit for other control room personnel and training; it does not have as high of a CDF reduction factor as a specific check.
- Credit was given in the recovery factor calculated for use of a procedure reader; it was treated the same as an extra crew.
- The PRA Engineer received information in June or July 2001 that operators stop AFW flow by using valves versus stopping pumps. The information was obtained during discussions with an operating crew. This information was verified later via operator interviews conducted by the HEP expert.
- The PRA group provides feedback to Training, via informal communications, on high-risk accident sequences, but not on specific procedure steps that have high HEPs.
EOP Coordinator: An interview was conducted with the EOP coordinator and identified the following points:
- The direct work item system is a process that allows procedure changes to be made.
Direct work items are changes that are issued by the WOG after review by the appropriate WOG subcommittee. Essentially they are revisions to the ERGs. Any member of WOG can initiate a possible direct work item but it does not become one until issued by the WOG.
- Changes to the EOPs can also be initiated internally without going through the WOG using the procedure feedback process. When this mechanism is used, the EOP Coordinator and an Operations Procedure Writer evaluate the request to decide if it should be processed, and the EOP set changed. There is no procedurally defined process that describes the evaluation methodology. There does not seem to be any guidance on determining specific technical content of a change if it is outside the ERG.
- Foldout page content is expected to be memorized by the operator. Foldout page information is intended to trigger operator memory. The addition of foldout page information to EOP-0 and EOP-0.1 is applicable at all times to continually control AFW flow correctly; this includes transition out of EOP-0 and EOP-0. 1. The EOP Coordinator did not consider the PRA value of foldout page use versus other methods of incorporating the desired actions into procedures when the decision to use a foldout page was made.
- No formal V&V was performed on the foldout page change to the EOPs; a serial review was performed.
- The EOP Coordinator believes that Operations generally keeps Training informed of training needs.
24
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir The EOP Coordinator thinks the changes made to the EOPs are done to streamline the procedures.
Industry and Station Operating Experience Internal Operating Experience CR 97-3363, IST Program Design Basis for AFW Minimum Flow Recirculation Valves: This CR was initiated on 10/15/97 to address a concern with a conflict between the IST Program and the AFW DBD. The IST Program stated that the AFW recirculation line check valves did not have an active safety function to open and that the minimum flow recirculation lines were not needed since there was always adequate flow to the S/Gs under accident conditions. This conflicted with the AFW DBD that did not address the check valves, but had an open safety function for the recirculation valves. The IST Program tests the recirculation valves in the open and close directions. The DBD group performed an evaluation on 2/5/98 that concluded the check valves have no safety related function in either direction and that the recirculation valves only have a safety related function in the closed direction. The basis stated that the main safety related function of AF was to supply water to the S/Gs and that flow to the S/Gs was the most important flow path to maintain. The mini-recirc line was considered a diversion path, and since the AF system was capable of a cold start, a recirculation path was not necessary. The potential to deadhead a pump was considered, but establishment of a flow path through the discharge lines was used to eliminate the concern and the mini-recirc path was deemed to not be needed for pump protection. The evaluation noted that DBD-01 (Rev. 0) was being revised to reflect that there was no open safety function. The evaluation went on further to consider an AFW MD pump scenario where the discharge MOV failed to open or the pressure control valve inadvertently closed along with the recirculation path being blocked. In this event, the recirculation line would be required to prevent pump destruction, but the emergency function to feed the S/Gs is defeated anyway. This active single component failure scenario would only apply to one pump, so it would be acceptable and recirculation flow for MD pumps was not a required safety related function.
QCR 99-0115, Code Testing Conflict With the Aux Feedwater Mini-Flow Recirc Check Valves: This CR was initiated on 5/24/99 and addressed a concern that conflicting information existed about the safety related function of check valves AF 115 and AF-1 17 to OPEN compared to the AFW recirculation valves that have a safety related function to CLOSE. Further, the IST Program did not include these check valves. An evaluation performed on 5/27/99 concluded that the concern identified was in error and had already been addressed by CR 97-3363. Additional evaluation on 6/15/99 concluded that some clarification to the IST Program documentation was needed to address how AFW single failure affected the decision on testing. A new action item was generated to revise the IST Program documentation and closed on 6/19/00 with issuance of Revision 4 of Appendix A of the IST Background Document.
25
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir
- RCE 98-148, P-38A AFW Pump Recirc Valve Found Failed Shut, dated 1/29/99:
This RCE documented an event where an operator was in the process of starting an AFW pump and noted that the recirculation line valve did not open as expected and then quickly secured the pump. This event showed that operators monitor recirculation valve position during AFW system manipulations.
INPO Operating Experience
- SEN 174 - Loss of Nonvital Bus Causes Dual Unit SCRAM and Degraded Auxiliary Feedwater System, dated 11/10/97
Description:
At the McGuire plant, a loss of non-safety related 120V AC instrument and control power caused both units to SCRAM. Also, the recirculation valves for all 3 U-1 AF pumps failed shut. The control board indication for these valves was also lost. As water level in the S/G was recovered, operators eventually shut the pump discharge valves. The pumps were operated for 20 to 60 minutes with their discharge and recirculation valves shut. Valve leakage was adequate to prevent pump damage.
Evaluation: This event is very similar to our case. Our evaluation of the SEN focused only on the power supply failure. AF pump operation without recirculation flow was discussed in the SEN and one question raised was "what procedures require operators to ensure that adequate pump flow is maintained?" This question was not addressed in the evaluation of the SEN.
SOER 88 Instrument Air System Failures, dated 5/18/88
Description:
This document provides a review and evaluation of industry events associated with failures and degradations of instrument air systems.
Recommendations 1, and 2 from this SOER are relevant to this event.
Recommendation 1 (Operations) was to provide procedures to assist operators in the identification, control, and recovery from partial or total loss of instrument air events.
A list of attributes that the operating, abnormal, and emergency procedures should provide included (in part) the following: identification of critical components operated by instrument air and the positions in which they fail, expected system and plant responses to a loss of IA and the consequences of these responses, actions to take if critical components do not fail in their intended position, and manual actions the operator should be expected to take to respond to a loss of IA event. The PBNP response was that AOP-5B, Loss of Instrument Air, contained the necessary instructions and information to assist operators in the identification, control, and recovery from partial or total loss of IA, and fully satisfied that recommendation. At that time, AOP-5B had an appendix for the AFW system that identified the recirculation valves as failing shut and requiring a manual gag override to open.
Recommendation 2 (Training) from the SOER was to provide classroom and simulator training on loss of IA events to operators. The training was to provide the bases for such things as failure modes of critical components and expected operator 26
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir actions, so that the operators would understand the major concerns involved in a loss of IA event. The PBNP response was to initiate Training Needs Analysis (TNA) 88 0425 for the PBNP Training group to evaluate. The result was that classroom training on loss of IA was included in cycle 89-8 of AO, RO, SRO, and DTA continuing training. LP 1782, Revision 0 dated 11/1/89, Instrument and Service Air was developed and approved to address this need. That lesson plan included a section that lists concerns with a loss of IA that focused on four areas: heat removal, auxiliary feedwater, inadvertent safety injection, and containment isolation. For AFW, the lesson plan identified that on the electric driven AFW pumps, the PCV fails open, and on all AFW pumps the recirculation valves fail closed. No simulator training on loss of IA was provided because PBNP was using the KNPP simulator then and loss of IA could not be adequately modeled on it.
Evaluation: The PBNP response to recommendation 1 addressed the need for information in abnormal operating procedures, but did not directly address operating and emergency procedures. The reliance on AOPs for addressing specific plant conditions and using EOPs for general response and mitigation probably influenced the scope of the review. The classroom training specifically identified that the AFW pump recirculation valves failed close on loss of IA, but did not identify concerns with pump damage or the need to gag open the valves, as dictated by AOP-5B.
However, there was a notation relating to the SI recirculation/test line isolation valves failing shut causing pump overheating in a few minutes and reference to an OPS Special Order 85-05 that had the valves currently gagged open. Simulator training was not performed due to modeling difficulties. Overall, the response did address the issue of the AFW recirculation valves failing closed on loss of instrument air, but missed an opportunity to completely integrate the issue into emergency procedures and training.
OE 10727 - PRA Risk Insight to Improve Operator Actions, dated 9/11/00
Description:
This document describes an event at another utility where the NRC identified that they did not effectively use PRA risk insight to improve the timeliness and reliability of mitigating operator actions prior to an actual event resulting in loss of all RCP seal cooling to 2 RCPs. For this event, it was determined that PRA updates were not being used to train operators on plant vulnerabilities to core damage.
Evaluation: At PBNP, procedure ESG 5.1, PRA Maintenance and Update Guidelines, requires the generation of a condition report whenever new vulnerabilities are identified. However, there were no provisions in the ESG that addressed who should be trained. In response to OE 10727, a revision to ESG 5.1 was issued on 12/19/00 that specified what groups should receive training on PRA updates and newly identified vulnerabilities.
Other Operating Experience
- Zion Station LER 90-002, 1A Auxiliary Feedwater Pump Cavitation, dated 2/15/90: This LER describes an event where the 1A turbine-driven AFW pump was 27
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air run in a deadheaded condition resulting in pump damage. Due to a combination of management error and procedural deficiency, the AFW pump was operated with both the discharge valve and recirculation valve shut for a period of about eight minutes until an operator stationed locally at the AFW pump noted an abnormal temperature rise on the pump's thrust bearing, water hammer sounds, and that the oil cooling water relief valve had lifted. This event demonstrates that pump damage can occur in a short period of time when operating a pump in a deadheaded condition. The pump impeller was found to be damaged and required replacement.
NRC Generic Communications Generic Letter 81-14, Seismic Qualification of AFW Systems, dated 2/10/81 The purpose of this GL was for licensees to determine the extent to which their AFWS are seismically qualified and to walk-down the non-seismic portions of the system and identify deficiencies. Our original response was submitted on July 16, 1981, in which we concluded that the PBNP AFWS is adequately protected for a seismic event. No specific mention was made of the AFWS recirculation valves or piping. In a response to the NRC follow-up request for additional information dated May 4, 1982, we specifically noted that the recirculation piping connections to the seismic AFWS piping were inspected and that the recirculation valves close upon receipt of a pump discharge flow signal. The NRC's Technical Evaluation Report (TER) of November 12, 1982, concluded that the PBNP AFWS did not provide reasonable assurance to perform its SR function following a seismic event. In our response dated December 15, 1982, we stated that the recirculation valves fail closed and the discharge AOVs fail open and concluded that the instrument air system that powers these valves is not required for AFWS functioning. Because of the questions concerning the recirculation piping not being well supported, we committed in this letter to independently support each air operated recirculation valve. Finally, in our letter dated April 26, 1985, we responded to the NRC request for comments on their revised TER. In the TER the staff postulated a failure during a seismic event of the non-seismic AFWS piping or a failure of the pump recirculation valves to shut following the switchover of the AFWS supply to service water. In our response we stated that under either condition the operator are trained to recognize off normal condition and that adequate time existed for manual operator actions.
Information Notice 87-28, Air Supply Problems at US Light Water Reactors, dated 6/22/87 The internal evaluation of this IN consisted of a review of all systems that perform safety functions and contain air operated valve operators, for the effect that the loss of air would have on those safety functions. The failure positions of the AFWS valves are identified. The concern for pump damage or failure due to less than minimum pump flow with the recirculation valves failing shut is also discussed. However, the focus of the evaluation was on demonstrating that the AFWS pumps would always feed the S/Gs with sufficiently high flow to protect the pump. This was documented in calculation N 87-041. At that time the discharge AOV for the electric AFW pumps failed open on loss of air; therefore, there was no identified concern with the 28
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir recirculation valves failing shut. This evaluation did not address the consideration that AFWS flow could be secured with the discharge MOV. This was a missed opportunity.
- NRC Bulletin No. 88-04, Potential Safety-Related Pump Loss, dated 5/5/88 This bulletin requested licensees to investigate and correct as appropriate two mini flow design concerns. The first concern was the potential for deadheading one or more pumps that have a common mini-flow line. The second concern is whether or not the installed mini-flow capacity is adequate to prevent damage to safety related pumps. In a response dated June 28, 1988, we acknowledged that each of the pumps in the AFWS have their own recirculation lines with an AOV isolation valve and an orifice upstream of the common return line to the CST. We discussed the logic of the recirculation valves to open or shut dependent on AFWS forward flow but did not address the potential to lose recirculation on an instrument air failure. We also acknowledged that the flow orifice for the pumps will need to be replaced with higher flow orifices to ensure sufficient flow for indefinite pump cooling via the recirculation lines.
10 CFR 50.63 Loss of All Alternating Current Power, effective 7/21/88 The NRC amended its regulations at 10 CFR 50.63 to require all nuclear power plants to be capable of withstanding and recovering from a station blackout (SBO) of a specified duration. Our initial response to this regulation, which addressed the appropriate guidance from Reg. Guide 1.155 and NUMARC 87-00 was submitted on April 17, 1989. In that response we stated that no air-operated valves are required to operate to cope with a SBO for one hour. We also completed an analysis on condensate inventory necessary to cope with the one hour SBO. We concluded that we had sufficient CST inventory, along with the initial S/G fluid inventory to maintain S/G decay heat removal capability. Clearly, for a SBO, only the TDAFW pumps would be available. The concern appeared to be assurance that sufficient water would be fed to the S/Gs until AC power was restored and AFW could be shifted to the safety related service water supply. The first NRC SER on SBO was dated October 3, 1990. The NRC agreed, based on our statement, "that the compressed air is not needed to cope with an SBO for one hour and, after 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />, the Alternate AC power source will supply the compressed air." The Technical Evaluation Report (TER Page 16) also stated agreement that operation of the AFWS is independent of AC and IA for one hour. Indeed the concern identified in the Technical Evaluation Report was that the minimum volume of 10,000 gallons in the CST per unit, was insufficient and ultimately we had to revise our Technical Specifications to change that minimum CST volume to 13,000 gallons. Clearly the focus of AFW was on providing forward flow and this may have been a missed opportunity.
Generic Letter 88-14, Instrument Air Supply System Problems Affecting Safety Related Equipment, dated 8/8/88 In a February 20, 1989, response to this GL we stated that all safety related pneumatic equipment at PBNP is designed to fail to a safe condition with the safety function 29
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air being tested in the PBNP IST Program. The AFWS discharge AOVs were specifically discussed and the concern expressed that the fail open position could potentially lead to over feeding of the S/Gs. There was additional correspondence to the NRC on July 27, 1989, in the form of a supplemental response concerning the potential problem with the discharge valves failing open. We also responded to an inspection report dated January 16, 1991, in which the NRC determined that PBNP had not fully complied with statements in our original GL response regarding testing of safety related AOVs. In this response we clarified that safety related valves with "passive" functions (do not perform a mechanical motion during the course of accomplishing a system safety function) were excluded from IST fail safe testing.
We also noted that since the 1989 submittal the IST program was revised and reissued for the third 10-year interval and that the AFWS mini-recirculation valves were now fail safe tested. This may have been a missed opportunity, but again the focus of the AFWS response was concern with over feeding the S/Gs.
Generic Letter 89-04, Guidance on Developing Acceptable In-service Testing Programs, dated 4/3/89 The attachment to the GL listed eleven specific generic deficiencies related to IST programs and procedures. Item 9 addressed pump testing using minimum flow return line with or with out flow measuring devices. The concern for this item was for those pumps that could only be IST tested using minimum flow return. In our response dated October 3, 1989, we confirmed that SI, RHR and AFW are tested in compliance with the GL position 9. The GL advised licensees that meeting the guidelines for Code testing does not supercede the thrust of Bulletin 88-04 (See discussion above).
This issue does not appear to be a missed opportunity for evaluation of the AFWS minimum recirculation valve failure modes.
Evaluation Methodology & Analysis Techniques The analytical techniques used in this root cause evaluation were:
- Document Review
- Interviewing
- Event and Causal Factor Charting (Attachment D)
- Timeline Development (Attachment B)
- Why Staircase Development (Attachment C)
Data Analysis Summary Identification of Causal Factors A "Why Staircase" was constructed based on the information obtained in the Information
& Facts Sources section of this report. This technique results in a repetitive asking of the question "why" until a detailed understanding of the problem is obtained. The "Why Staircase" for this event is provided in Attachment C. This approach identified four main causal factors that contributed to this event.
30
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air EOP-0. 1 contains a step (step 1) to CONTROL feed flow because of RCS cool down considerations and another step (step 4) to STOP feed flow to a steam generator if an increasing level cannot be maintained below the desired setpoint - these steps do not specify the method to be used to CONTROL or STOP flow. (It is postulated that an operator could throttle the AFW discharge valves closed and with a loss of instrument air when the recirculation valves are failed closed, the running pumps would dead-head and destroy themselves in short period of time; a potential common mode failure.) There were two reasons influencing why specific information was not provided in the EOP.
First, reliance had been placed on AOP-5B for providing specific operator actions for a loss of instrument air scenario, and second, closure of the AFW discharge valves due to operator action was not previously considered as a possible failure mechanism.
Reliance on AOP-5B:
Reliance on AOP-5B was faulty because operator action to control AFW flow (under loss of instrument air conditions) was needed in the early steps of EOP-0. 1. This need had not been identified prior to this event. A key opportunity to have identified this need was via the EOP validation process. The original validation of EOP-0.1 steps was done in 1985 using a Reactor Trip w/o SI scenario. This scenario did not include a concurrent loss of instrument air condition. Consequently, it would not matter what method an operator used to control flow since either throttling flow or shutting off pumps would be successful. These steps have not changed since Revision 0, so additional validation would not have been required. The original validation of EOP-0.1 was less than adequate.
Another key opportunity to identify the need for operator action while in EOP-0. 1 was when the initial PRA model was developed to support the IPE submittal in 1993. The original PRA model did not model operator actions to control AFW flow in the system fault trees because it was assumed that there was a long time available and the function (S/G overfill) would be alarmed (assumption 13). The flaw in this assumption was not identified during the PRA model review because the fault trees were based primarily on functions described in design documents. The selection of the evaluation method using fault trees focused on design functions over other FMEA methods was based on an assumption that the design function approach was more conservative. The current PRA model review uses a methodology that integrates system performance with potential human actions to obtain a spectrum of plant responses. The original PRA Model was less than adequate because of a wrong assumption.
Finally, routine performance of accident scenarios on the PBNP simulator should also have provided an opportunity to identify this need for operator action. Simulator Guides are presented in outline form and do not contain detailed information on evaluation of all actions performed during the scenario. PRA information has been used to identify which scenarios are important to teach from a risk perspective, but information on which steps in emergency procedures are risk-significant has not been incorporated into scenario evaluation criteria. Consequently, scenarios often went quickly through the loss of air condition to other conditions such as loss of secondary heat sink without evaluating the 31
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air intermediate steps such as S/G level control. The interface between the PRA and Training programs is less than adequate.
Operator Action was not Previously Considered as a Possible Failure Mechanism Previous evaluations of the effects of the AFW recirculation valves failing closed on loss of IA concluded that the AFW pumps would not be damaged because forward flow was always available. Closure of a single discharge valve due to component failure concurrent with the AFW recirculation valve failing closed was evaluated and considered to be outside the design and licensing basis. (This used NUREG-0800 assumptions and PBNP was not committed to that NUREG.) Closure of all the discharge valves due to operator action was not considered. The two reasons identified for not considering operator actions were the lack of integrating human actions into failure mode analyses and the lack of insight that a specific operator action could result in pump damage.
Although the concept of determining the potential failures that could result from human errors has been around since at least the TMI accident, it is most often utilized in the PRA area. The current Design Input Checklist (PBF-1584) does not prompt an evaluation of the creation of a new failure mode from a human action perspective. When the MDAFW pump discharge AOVs were modified with a nitrogen back-up system, a throttling capability was created that did not exist on that valve before (under a loss of instrument air condition). Throttling of the MOVs that direct AFW flow to the respective steam generators had already existed, so this was an additional opportunity to perform that same action on another component. Only recent use of failure mode fault tree tables in the PRA program allowed identification of the concern on AFW control. The knowledge learned from evaluating human interactions in the PRA program has not been transferred into the failure modes and effects analysis element of the design control program. The interface between the PRA and Design Control programs is less than adequate.
Insight was needed to understand that the actual operator response to a "CONTROL or STOP feed flow" command under a loss of instrument air scenario would be closure of the discharge valves instead of stopping the AFW pumps. The expected operator response to the "CONTROL or STOP feed flow" command under a loss of instrument air scenario was not clearly stated in training documents. Knowledge that operation of the AFW discharge valves had a human error probability associated with it could have resulted in focused training on that evolution that may have identified the potential for pump damage. However, the information on risk-significant human interactions was not effectively incorporated into the training program. The interface between the PRA and Training programs is less than adequate.
Other Conclusions The assumptions used by the PRA group in evaluating human interactions are based on industry guidelines that determine how the effectiveness of procedures is established.
These same rules have not been applied to our process for procedure writing. One example is the use of action steps in notes. The industry guidance is clearly not to include actions in notes. However, the AOP and EOP Writers' Guide (and WOG ERG 32
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir Writers Guide) allows the use of condition monitoring that initiates an action in a note.
Under PRA rules, little credit is given for an action embedded in a note. Procedure effectiveness can be improved by incorporating PRA rules into our procedure development process. The interface between the PRA and procedure development processes is less than adequate.
ESG 5.1, PRA Maintenance and Update Guideline, is the governing document for administration of PRA updates. That procedure contains interfaces with departments outside of Engineering. The use of a higher tier document may be more appropriate for this process. Organizational interfaces for the PRA update process lack formality.
There was a lack of consistency between different design basis and licensing documents regarding the description and function of the AFW recirculation valves. The predominant position taken in various licensing correspondence was that AFW flow could always be provided to the S/Gs and the recirculation valves were not required to provide an open safety function. However, the initial AFW DBD (1994) contained a statement that the valves had an open safety function, and the basis was not clear. The open function was removed from the AFW DBD in 2000. The IST program did not include an open safety function, but did test the valves in the open direction based on prior NRC correspondence (1992). That testing was removed from the IST program in 1998. The FSAR did not include any discussion of the recirculation line function until updates made in 1997 and 1998. Consistency between AFW licensing and design basis documents is less than adequate.
FailureMode Identification PP-1 ILack of Interface Requirements - Actions required by one program not belonging to any program, which is needed to ensure consistency. I
"* Information on risk-significant human interactions was not effectively incorporated into the operations training program, including scenario development
"* Knowledge learned from evaluating human interactions in the PRA program has not been transferred into the failure modes and effects analysis element of the design control program
"* PRA concepts are not included in the emergency procedure development process
"* Consistency in the licensing and design basis for the AFW system was not maintained between the FSAR, AFW DBD and IST program P-1 Insufficient Detail - Inadequate program desi n
" The original validation of EOP-0. 1 steps done in 1985 using a Reactor Trip w/o SI scenario did not include a concurrent loss of instrument air condition 33
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air 00-1 Inadequate Interface Among Organizations - Lack of interface formality
"* The PRA update interface requirements with other organizations are contained in an Engineering Supplemental Guideline, and lack formality.
MJ-5 I Wrong Assumptions - Erroneous assumptions used in decision making
"* Forward flow was assumed as the only necessary requirement to ensure AFW system availability. It was assumed that the operators would maintain flow necessary to ensure pump cooling.
"* The selection of the original PRA model evaluation method using fault trees focused on design functions over other FMEA methods was based on an assumption that the design function approach was more conservative.
VII. Root Causes & Contributing Factors Conclusions EOP procedural weaknesses existed because the actions required for AFW control under loss of instrument air conditions were not identified during the original EOP validation process. The recognition of the importance of specific AFW control steps occurring earlier in the EOPs prior to the transition to AOP-5B came only when human error probability methods were applied during the PRA model update process in 2001. These same human error probability methods have not been integrated into procedure development, training, and design control processes.
Previous opportunities to identify this issue were missed due to a faulty assumption in the original IPE, an assumption that AOP-5B adequately addressed a loss of IA condition, and the failure to consider the effects of operator actions on system performance.
Root Cause The root cause of the EOP procedural weaknesses was the failure of the original EOP validation process to identify that specific operator actions were needed to properly control or stop AFW flow under a loss of instrument air condition. This resulted in a mismatch between plant design and procedural guidance.
Contributing Causes Significant contributing causes to this condition continuing to exist were:
- the original PRA model fault trees evaluated system performance primarily on functions described in design documents and did not adequately consider human actions, 34
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air
- the lack of integration of human error reduction methods into the operations training process,
- the failure to consider human actions during FMEA reviews in the design control processes, and
- the assumption that forward flow was the only necessary requirement to ensure AFW system availability without consideration of intervening operator actions.
Other causes that were not significant contributors were:
- the lack of integration of human error reduction methods into the emergency procedure development process,
- the lack of formality of organizational interfaces in the PRA update process, and
- the inconsistencies between the FSAR, AFW DBD, and the IST program concerning the description and function of the AFW recirculation valves.
VIII. Corrective Actions Interim Corrective Actions (mitigation)
- CA #1 Responsible Group: Operations, Completion Due Date: Complete Revise EOP-0, EOP-0. 1 and ECA-0.0 to address AFW control under loss of instrument air conditions.
Corrective Actions to Prevent Recurrence (CATPRs)
"* CA #1 Responsible Group: Engineering (PRA), Priority: 2, Completion Due Date:
5/6/2002 [CA003691]
Assist Operations in determining what initiating events should be included in the EOP validation process by formally providing information on which initiating events considered risk-significant for each EOP.
"* CA #2 Responsible Group: Operations, Priority: 2, Completion Due Date: 8/5/2002 (90 days after CATPR #1 is completed) [CA003692]
Revise the EOP validation process to ensure that appropriate initiating events are included. Utilize PRA input in determining what initiating events are applicable.
35
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Corrective Actions to Restore (broke - fix)
"* CA #1 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:
10/4/2002 [CA003693]
Complete the analysis portion of the PRA model review to identify any other risk significant vulnerabilities in the current EOPs.
"* CA #2 Responsible Group: Operations, Priority: 3, Completion Due Date: 6/5/2002
[CA003694]
Review the operator actions specified in AOP-5B to determine if they should be included in applicable EOPs to ensure timeliness of the actions, and initiate revisions as required.
"* CA #3 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:
6/5/2002 [CA003695]
Formally provide Operations and Training with an updated list of high-risk human error events based on the PRA model.
" CA #4 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:
6/5/2002 [CA003696]
Formally provide Operations and Training with a description of the human error reduction methods used in evaluating operator actions in the PRA model.
" CA #5 Responsible Group: Operations, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA #2 and CA # 3 are completed) [CA003697]
Review EOPs and AOPs containing high-risk human error events against human error reduction methods used in the PRA model and revise where appropriate to achieve significant CDF risk reduction.
" CA #6 Responsible Group: Operations, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA # 3 is completed) [CA003698]
Revise OM 4.3.1, AOP and EOP Writers' Guide, to incorporate human error reduction methods used in the PRA model that can significantly reduce CDF risk.
" CA #7 Responsible Group: Training, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA #2 and CA # 3 are completed) [CA003699]
Review initial operator training materials and methods associated with high-risk human error-events against human error reduction methods used in the PRA model and revise where appropriate to achieve significant CDF risk reduction.
36
Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air "CA#8 Responsible Group: Training, Priority: 3, Completion Due Date: 10/4/2002 (120 days after CA # 3 is completed) [CA003700]
Revise operator training procedures to incorporate human error reduction methods used in the PRA model that can significantly reduce CDF risk.
" CA #9 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:
6/5/2002 [CA003701]
Revise the AFW PRA model to accurately reflect system performance.
" CA #10 Responsible Group: Engineering (Systems), Priority: 3, Completion Due Date: 6/5/2002 [CA003702]
Review the description of the AFW recirculation line function in the FSAR, DBD-01, and the IST Program for consistency and accuracy, and initiate revisions as required.
"* CA #11 Responsible Group: Engineering (Design), Priority: 3, Completion Due Date:
6/5/2002 [CA003703]
Revise the design input checklist to include consideration of human action induced failure modes.
" CA #12 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:
6/5/2002 days [CA003704]
Evaluate if an Engineering Supplemental Guideline is the appropriate procedural method for controlling PRA updates, or if a higher tier document such as a Nuclear Procedure (NP) should be used considering the interfaces involving other departments. Initiate any procedure changes resulting from that evaluation.
" CA #13 Responsible Group: Engineering (PRA), Priority: 3, Completion Due Date:
6/5/2002 [CA003705]
Revise the procedure governing PRA updates to include identification of the formal methods to be used for providing information to other groups. Use of existing processes, such as training work requests and procedure feedback forms, should be used whenever possible.
37
Increased CDF in AFWPRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air IX. References AOP-5B, various revisions, Loss of Instrument Air CR 97-3363, dated 10/15/97, IST Program Design Basis for AFW Minimum Flow Recirculation Valves CR 01-2278, dated 7/6/01, AFW PRA Model for Loss of Instrument Air CR 01-3595, dated 11/29/01, PRA for AFW System CR 01-3633, dated 12/4/01, Response of MDAFWPs to an Appendix R Fire CR 01-3641, dated 12/4/01, AFW Pumps Common Mode Failure Information for CR 01-3595 RCE CR 01-3648, dated 12/5/01, Response of MDAFWPs to an Appendix R Fire CR 01-3654, dated 12/6/01, AFW System DBD Missed Opportunity DBD-01, Revision 0, dated 4/4/94, Auxiliary Feedwater System DBD-0 1, Revision 1, dated 3/21/00, Auxiliary Feedwater System DD-EOP-0, various revisions, Deviation Documents - Reactor Trip or Safety Injection EOP-0, various revisions, Reactor Trip or Safety Injection EOP-0. 1, various revisions, Reactor Trip Response EPRI Report TR-100259, dated 6/92, An Approach to the Analysis of Operator Actions in Probabilistic Risk Assessment Event Notification Worksheet EN#38525, dated 11/29/01 Event Notification Worksheet EN#38525 Supplemental, dated 11/30/01 FSAR, Chapter 10, various revisions, Auxiliary Feedwater System (AF)
Individual Plant Evaluation, Revision 0, dated 6/30/93 IST Background Document -Appendix A, dated 5/17/00 IST Program - 3 d Interval, various revisions Internal Memorandum, dated 12/3/01, CR 01-3595 Reportability Recommendations NUREG-0899, dated 8/82, Guidelines for the Preparation of Emergency Operating Procedures NUREG-1358, dated 4/89, Lessons Learned From the Special Inspection Program for Emergency Operation Procedures NUREG-1358 Supplement 1, dated 10/92, Lessons Learned From the Special Inspection Program for Emergency Operation Procedures NUREG/CR-2005, dated 4/83, Checklist for Evaluating Emergency Operating Procedures Used in Nuclear Power Plants OD 01-3595 Rev. 0 dated 11/30/01, and Rev. 1 dated 12/1/01 OD 01-3648 Rev. 0 dated 12/7/01 OM 4.3.1, Revision 1, dated 6/4/99, AOP and EOP Writers' Guide OM 4.3.2, Revision 1, dated 6/14/95, EOP Verification Procedure OM 4.3.3, Revision 0, dated 7/30/93, EOP Validation PRA System Notebook - AFW, Revision 0, dated 1991 QCR 99-0115, dated 5/24/99, Code Testing Conflict With the AFW Mini-Flow Recirc Check Valves RCE 98-148, dated 1/29/99, P-38A AFW Pump Recirc Valve Found Failed Shut S-A-ENG-01-03, PBNP PRA Peer Review Report (Draft Report - 7/01) 38
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air SEN 174, dated 11/10/97, Loss of Nonvital Bus Causes Dual Unit Scram and Degraded Auxiliary Feedwater System WOG ERG Executive Manual WOG ERG Writers Guide, dated 7/1/1987 WOG LP-ERGs Zion Station LER 050-295/90-002-00, dated 2/15/90, 1A Auxiliary Feedwater Pump Cavitation X. Attachments Attachment A: Team Charter Attachment B: Timeline Attachment C: Why Staircase Attachment D: Event & Causal Factor Chart 39
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Attachment A: Team Charter Root Cause Investigation Charter CR 01-3595 RCE 01-069 Issue Manager:
Rick Mende Problem Statement:
Discovery during the review of the AFW PRA model for transients involving loss of instrument air that emergency and abnormal operating procedures may not adequately address maintaining minimum AFW pump recirculation flow to prevent AFW pump failure.
Investigation Scope:
Determine the following:
"* the root cause of why the condition exists
"* why the problem was not identified previously Make recommendations for:
"* correcting the problem
"* preventing recurrence of the problem
"* applicability of the root cause to other areas (extent of condition)
Team Members:
Team Leader - Richard Flessner, Engineering Processes Team Member - R. Wood, PRA Team Member - J.P. Schroeder, System Engineering Team Member - T. Staskal, Site Assessment Team Member - C. Krause, Licensing Milestones:
Status Update - 12/11/01 Draft Report - 12/20/01 Final Report - 1/10/02 Approved: (Original signed by F. Cavia) Date: 12/412001 Fred Cayia, PBNP Plant Manager 40
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir Attachment B: Event Timeline DATE / TIME DESCRIPTION 9/1/9 M-623/624 TDAFP alternate bearing cooling supply modification issued 2/1/80 IC-274 AFW recirculation valve logic (keep open) modification issued 2/10/81 GL 81-14 issued on Seismic Qualification of AFW System (response is dated 7/16/81) 5/4/82 Additional response to GL 81-14 due to NRC RAI - response says that AFW recirc valves close on receipt of AFW pump discharge flow signal 6/82 WOG Basic ERGs validated on Calloway Simulator 8/82 NUREG-0899, Guidelines for the Preparation of EOPs, is issued 8/31/82 IC-274 AFW recirculation valve logic (keep open) modification cancelled 11/12/82 NRC issues TER concluding that PBNP AFW system did not provide reasonable assurance to perform its SR function following a seismic event 12/15/82 PBNP response to NRC TER on AFW - concluded that IA is not required for AFW system functioning (based on recirc valves FC and discharge valves FO);
commit to independently supporting each recirc valve 4/83 NUREG/CR-2005, Checklist for Evaluating EOPs, is issued 8/1/83 MR 83-104 AFW system discharge MOV controls modification issued 4/26/85 PBNP response to revised NRC TER on AFW - conclude that AFW piping failure or failure of AFW recirc valves to close will be handed by operators trained to recognize off normal condition that adequate time exists for manual action 7/1/85 Revision 0 of the EOPs issued 5/2/86 AOP-5B, Loss of Instrument Air, Revision 0 issued 6/22/87 IN 87-28 issued on Air Supply Problems at US Light Water Reactors 7/1/87 WOG ERG Writers Guide issued 12/20/87 IN 87-28 Supplement 1 issued on Air Supply Problems at US Light Water Reactors 3/23/88 NPERS evaluation of IN 87-28 issued via NEPB 88-090 5/5/88 IEB 88-04 issued on Potential SR Pump Loss (response is dated 6/28/88) 5/18/88 INPO issues SOER 88-01 on Instrument Air Failures 7/7/88 MR 88-099 AFW pump mini-recirculation line improvements modification issued 7/21/88 SBO Rule (10CFR50.63) became effective (response is dated 4/17/89) 8/8/88 GL 88-14 issued on Instrument Air Supply System Problems Affecting SR Equipment (response is dated 2/20/89) 4/89 NUREG-1358, Lessons Learned From the Special Inspections Program for EOPs, is issued 4/3/89 GL 89-04 issued on Guidance on Developing Acceptable IST Programs (response is dated 10/3/89) 5/8/89 MSS approves response to SOER 88-01 2/15/90 Zion Unit 1 LER issued on AFW Pump Cavitation 12/90 3 rd interval IST Program is implemented
-1991 Original IPE Notebooks developed 41
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 5/28/91 Revision 1 to IST Program adding VRR-28 on recirc valves 4/17/92 NRC issues TER on IST Program denying VRR-28 and requesting OPEN safety function be added for recirc valves 6/92 EPRI Report TR-100259, An Approach to the Analysis of Operator Actions in PRA, is issued 6/19/92 MR 92-091/092/093 IST testability of AFW recirculation line AOVs modifications issued 7/30/92 PBNP response to NRC TER clarifying that recirc valves are not required to OPEN to protect AFW pumps 10/92 NUREG-1358 Supplement 1, Lessons Learned From the Special Inspections Program for EOPs, is issued 3/2/93 PBNP informs NRC that mods will be completed for testing recirc valves and withdraws VRR-28 3/30/93 Rev. 3 of IST deletes VRR-28 4/93 DBD-01 validation considers worst-case flow (discharge and recirc valves closed) outside design and licensing basis 6/30/93 Revision 0 of IPE PRA model is issued 4/4/94 DBD-01, AFW System, Revision 0 is issued
-1995 Affects of excessive AFW flow introduced into EOPs 4/15/97 MR 97-038*A/B MDAFP discharge pressure control valve backup nitrogen supply and cable separation modifications issued 6/97 Update to FSAR adding AFW recirc feature for 3 minute closure on pump start 9/26/97 AOP-5B, Revision 11 issued that moved time critical steps from appendices to main body of the procedure 9/30/97 Revision IC of WOG ERGs issued 10/15/97 CR 97-3363 initiated on IST Program Design Basis for AFW Minimum Flow Recirculation Valves (closed 10/5/98) 11/10/97 INPO issues SEN 174 on Loss of Nonvital Bus Causes Dual Unit Scram and Degraded AFW System (McGuire Units) 1998 Update to IPE PRA model is issued 1/6/98 Evaluation of SEN 174 completed - focus was on power supplies and did not address degradation of AFW recirculation valves 6/98 Update to FSAR adding detailed description of recirculation line function 6/29/98 CR 98-2575 (RCE 98-148) initiated on P-38A AFW Pump Recirc Valve Found Failed Shut 9/30/98 Rev. 5 of IST Program issued deleting testing of AFW recirc valves in the open direction 5/24/99 QCR 99-0115 initiated on Code Testing Conflict With the AFW Mini-flow Recirc Check Valves 3/31/00 DBD-01, AFW System, Revision 1 is issued 9/11/00 OE 10727 initiated on industry event involving PRA 42
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 7/6/01 While revising the Probabilistic Risk Assessment (PRA) model for the Auxiliary Feedwater system, a potential procedural shortcoming was identified in AOP-5B, Loss of Instrument Air. Condition Report 01-2278 was originated to document the above finding 7/10/01 A CR action item #1 was created for Operations to move the step in AOP-5B, "Loss of Instrument Air," for gagging open the AFW minimum recirculation valves to an earlier location in the body of the procedure. (CR 01-2278) 7/30/01 Operations discussed issue with PRA group. PRA to run an evaluation to determine the significance of the issue. Analysis was expected to be completed by 8/20/01 (CR 01-2278) 8/20/01 The analysis is not ready yet. The evaluation is expected to determine the actual risk significance of the condition and address the type of actions that may be recommended. (CR 01-2278) 10/19/01 Per discussion with the PRA group, the PRA model is showing a higher risk and the recirculation valve should be procedurally addressed. The AOP is sequenced properly to address the loss of instrument air. PRA Group is requesting that the ARP for low instrument air pressure be changed to address this concern. This should be adequate rather than changing the sequence of the AOP. PRA will follow up with a procedure feedback. (CR 0 1-2278) 10/24/01 CR 01-2278 Action #1 was completed with direction to create a new action item to track issuance of a change to ARP C01 A 1-9 for low instrument air pressure. (CR 01-2278)
Early Operations had discussions with PRA Group regarding whether procedure November, changes were adequate.
2001 Week of Nov PRA Group went to work to adjust the PRA model to evaluate the risk if the 13 th" 2001 procedure change was not complete or would not be adequate.
11/26/01 Modeling adjustments were completed. A risk evaluation was done for the minimum recirculation valves. A factor of 2.3 risk increase was identified.
This was considered high-risk significance. A discussion was held with Operations and Engineering. Decided we needed to determine what the scope of this was and what further actions may be appropriate.
11/28/01 - A meeting was held with Operations, Engineering and PRA personnel to 1300 discuss the significance and appropriate actions. The mechanistic details of the issue were well understood and developed by all present. The consensus was that this item represented a real possibility, and that it required further attention. Various possible actions were discussed, focusing primarily on enhancing Operator awareness of the system design, as well as modifications or procedural changes that may be desirable to eliminate it.
The subject of Operability was discussed during the meeting, and it was agreed that there was no operability concern because no equipment degradation, failure, or non-conformance had been identified. Regardless, the level of concern was great enough that further prompt action was felt justified.
43
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 11/28/01 - The Operations manager had discussions with Engineering about this potential Late afternoon concern regarding significantly increased CDF risk resulting from an event where instrument air was lost and during the subsequent EOP actions, operators may take inappropriate action which could cause one or more AFW pumps to fail.
11/29/01 - AM Operations manager briefed the resident inspectors on the concerns of the issue and that we were evaluating the condition and risk.
11/29/01 - Following discussions with the staff SRO, operations concluded that use of Late AM temporary information tags and a briefing of all watch standers, would be an important step to reduce the risk of the event. We also started evaluating procedure changes that might help improve the safety of the plant and reduce the risk profile.
11/29/01 - PRA briefed the STA and Shift Manager on the issue and discussed potential 10:00 wording for control board placards.
11/29/01 - PRA discussed potential reportability concerns with licensing.
11:00 11/29/01 - PRA briefed the RI and provided estimated risk impact values.
11:30 11/29/01 - CR 01-3595 documenting the increased risk was written. The CR was brought 14:45 to the WCC and screened by an SRO. At that time, extensive discussion regarding whether an OD was required had already occurred, and extensive discussion on operability had occurred. My discussions with engineering and others focused on the fact that there was not an equipment problem, no equipment is degraded such that operability is in question, that this is a risk issue upon which we are relying on operator action to mitigate, and therefore, use of the OD was not appropriate. Those discussions were not captured in either the CR, or the associated screening.
11/29/01 - The oncoming crew was briefed and temporary information tags placed 1520 adjacent to the controls for 1/2P-29 and P-38A/B. This briefing summarized the concerns of this potential event. The temporary information tags provided a reminder that the minimum flow requirements for the AFW pumps are 50 GPM for the motor driven pumps and 75 GPM for the steam driven pumps.
11/29/01 - CR 01-3595 was screened by the WCC SRO (CR 01-3595) 1553 11/29/01 - Operations Manager briefed Plant Manager on this issue.
1700 11/29/01 - Event Notification 38525 made to NRC via ENS phone.
1705 11/30/01 - AM Licensing manager received a call from the NRC-NRR backup PM concerning confusion over the event notification. A return conference call was made with engineering to address NRR questions.
44
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 11/30/01 - AM Friday morning, after discussing this with the residents, Operations Manager concluded that to properly document the operability of the AFW system, we should initiate an operability determination to ensure the discussions we had the previous 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> regarding operability were properly documented.
Engineering was requested to start on the OD. The Shift Manager was informed that an OD on the issue was being performing it and that it was expected to be completed mid to late afternoon.
11/30/01 - Operations Manager met with Sr. Resident, Resident, and their supervisor to Noon discuss situation. At that point NRC brought forward their concerns regarding whether AFW was operable in the condition that existed prior to Thursday afternoon and whether it was currently operable. The Plant Manager called NRC Region III along with the Operations Manager and had a discussion regarding operability of the system.
11/30/01 - Ran a simulator scenario to get information on plant response to a loss of 1400 offsite power coincident with a rapid loss of instrument air pressure.
NOTE: Additional simulator scenarios were run on 11/30 and 12/1.
11/30/01 - Temporary procedure changes were completed to EOP-0 and EOP-0.l to 1645 reflect the guidance provided earlier to operators on the temp info cards.
11/30/01 - Plant Manager informed that a five-man incident investigation team would
-1700 arrive on 12/3.
11/30/01 - A supplement to the Event Notification was provided to the NRC to clarify the 1746 discussion of the potential for an AFW failure as described in the original event notification 38525 11/30/01 - The OD was approved. This OD evaluated the current operability of the AFW
-1830 system and included a discussion of the compensatory measures already taken to assure compliance with our licensing basis.
12/1/01 - 0930 Staff meeting to prepare for NRC inspection team.
to 1200 12/1/01 - 1515 Revision 1 to the Operability Determination was approved. The discussion of the AFW pump motor duty cycle was revised.
12/3/01 - 0830 CR 01-3595 screened as requiring an ACE.
12/3/01 - 1000 Inspection Team meeting to prepare presentation for NRC entrance meeting.
to 1200 12/3/01 - 1200 SVP and Plant Manager agree that CR 01-3595 requires a RCE.
12/3/01 - 1400 NRC Inspection Team has entrance meeting.
12/4/01 HEP expert onsite 12/4/01 - 0700 Initial RCE Team meeting held.
12/4/01 - 1200 Plant Manger approves RCE Charter.
12/4/01 - 1620 CR 01-3633 initiated on Appendix R concerns associated with MDAFW pump and LOOP and loss of IA and coincident fire. (CR 01-3633) 12/5/01 - 1545 CR 01-3648 initiated on response of MDAFW Pump to an Appendix R fire coincident with a LOOP and loss of IA. Potential existed for auto-start with discharge and recirc valves failed closed causing pump damage. (CR 01-3648) 12/7/01 - 0900 NRC Inspection Team has technical debrief.
45
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air DATE / TIME DESCRIPTION 12/13/01 - NRC Inspection Team has exit meeting.
1400 12/14/01 Permanent Revision to EOP-0 and EOP-0.1 implemented.
12/20/01 Additional revision made to EOP-0, EOP-0.1, and ECA-0.0 46
Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Attachment C: Why Staircase Problem: There is an increased CDF during a loss of instrument air scenario due to a common mode failure of all AFW pumps.
Why?: EOP-0. 1 contains a step (step 1) to CONTROL feed flow because of RCS cool down considerations and another step (step 4) to STOP feed flow to a steam generator if an increasing level cannot be maintained below the desired setpoint - these steps do not specify the method to be used to CONTROL or STOP flow. (It is postulated that an operator could throttle the AFW discharge valves closed and with a loss of instrument air when the recirculation valves are failed closed, the running pumps would dead-head and destroy themselves in about 30 seconds; a common mode failure.)
Problem: EOP-0. 1 contains insufficient information to direct operators to take the correct actions for controlling AFW flow or stopping AFW flow to S/Gs under a loss of instrument air scenario.
Whyl?: Reliance had previously been placed on AOP-5B for directing operator response to a loss of instrument air scenario; however, it was just recently recognized by the PRA group that action by operators would be required earlier in the scenario while still in EOP-0.1 (e.g., controlling S/G level without the availability of the AFW recirculation valves).
Probleml: The need for specific operator response actions for AFW flow control due to a loss of instrument air scenario while in EOP-0. 1 was not previously identified.
Whyl-l?: The original validation of EOP-0.1 was done with a Reactor Trip w/o SI scenario that did not include loss of instrument air. The actions to stop feed flow on high S/G level had not changed since Revision 0 so additional validation would not have been required. A similar step in ECA-0.0 exists and was validated originally in a Loss of All AC scenario. The specified actions (only applicable to the TDAFW pump) were to isolate AFW flow and then close the steam supply valve to the steam-driven AFW pump. No problems were identified with that step during the validation. (EOP Validation LTA)
Whyl-2?: The original PRA model did not model operator actions to control AFW flow in the system fault trees because it was assumed that there was a long time available and the function (S/G overfill) would be alarmed (assumption 13). The flaw in this assumption was not identified during the PRA model review because the fault trees were based primarily on functions described in design documents. The selection of the evaluation method using fault trees focused on design functions over other FMEA methods was based on an assumption that the design function 47
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir approach was more conservative. The current PRA model review uses a methodology that integrates system performance with potential human actions to obtain a spectrum of plant responses. (Wrong Assumption)
Whyl-3?: Routine accident scenarios run on the simulator did not specifically evaluate performance of human interactions having PRA significant error probabilities. (Program Interface LTA)
Why2?: Previous evaluations of the effects of the AFW recirculation valves failing closed on loss of IA concluded that the AFW pumps would not be damaged because forward flow was always available. Closure of a single discharge valve due to component failure concurrent with the AFW recirculation valve failing closed was evaluated and considered to be outside the design and licensing basis. (This used NUREG-0800 assumptions and PBNP was not committed to that NUREG.) Closure of all the discharge valves due to operator action was not considered.
Problem2: Closure of the AFW discharge valves due to operator action was not previously considered as a possible failure mechanism.
Why2-1?: The consideration of human actions in failure modes and effects analyses has occurred primarily only in the PRA area. (Program Interface LTA)
Why 2-2?: Insight was needed to understand that the actual operator response to a "CONTROL or STOP feed flow" command under a loss of instrument air scenario would be closure of the discharge valves instead of stopping the AFW pumps.
Problem: The expected operator response to the "CONTROL or STOP feed flow" command under a loss of instrument air scenario was not clear.
Why?: Training materials did not contain specific information on operator actions for controlling steam generator level (and AFW flow) under a loss of instrument air condition.
Problem: Training materials did not specify the actions required for successful control of AFW flow under loss of instrument air conditions.
Why?: The importance of the AFW control evolution was not previously emphasized. (Program Interface LTA) 48
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air Attachment D: Event & CausalFactorChart KEY
- - -- - ----- 1 UNVERIFIED ENAPPROPRIAT
.1 L-------------
EVENT CTIO CONTRIRUliNO CAUSAL
'~,,CONDITION.,, FACTOR aInelI.
(faor 49
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED EVENT E INAPPROPRIATE CTION CONTRIBUTING CAUSAL. Inves FACTOR "COND ON -N 50
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED EET (F.1.1.1~ INAPPROPRIATE EVENT I CTIO CONTRIBUTING R T
'...~CONDITION CAUSAL .1 \I rst-rea'so FACTOR Statements directing performance of actions must not be In notes .
Do explanatory notes avoid the use of action statement? ) (Noon AFW change needed recirc vlvs 3/10/83 - 4/18/83 Early drafts of PBNP EOPs validated on Zion Simulator "Scopeis MDAFP discharge valves auto actuation 51
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir 4/26/85 PBNP responded to revised NRC AFW TER NRC NRC postulated postulated failure failure of AFW of non-seismic recirc vlvs to shut KEY AFW piping UNVERIFIED? -aI INAPPROPRIATE EVENT ! CTIO CONTRIBUTING CAUSAL Invesh.
"CAUSE CONDITIt.i" o FACTOR 52
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY UNVERIFIED E INAPPROPRIATE EVENT 'conditioin CTION CAUSAL (I or\
""CONDITION
, \veti-J "Scopeis a walk through on full scale PBNP CR mock-up Performed as part of CRDR project 53
hzcreased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED EVENT LZoi C'.-.I.
ndiIIo)0 CNAPRPITIO NAPPOPRAT fr CONTRIBUTING ROO (CAUSAL (Investl- FACTOR CASE .CONDITION.," alo note may be Passive action used to provide a statements Innotes contingent may be appropriate transition Discrepancy 54
Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY UNVERIFIED I INAPPROPRIATE EVENT on~Ion) CTION for CONTRIBUTING OO CAUSAL 1 Invesui- FACTOR
'Design review dide not identify concern with FC recirc vlvs, 55
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY A
UNVERIIFIED Fare INAPPROPRIATE EVENT j ~ondiffon. CTIO N. ANVEIFI~N~ for~ CONTRIBUTING (ROO\ (CAUSAL UnesIJ FTACTOR
~CUE~~CNDITIO hvsf-IN 2/9/89 2/20/89 NPERS evaluation of PBNP response to SOER 88-01 issued GL 88-14 issued 56
Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED E (FactlrI/)
EVENT I INAPPROPRIAT CTION CONTRIBUTING R.OO.T "( CAUSAL TIn FACTOR
'. CONITION_./
'AFW disch AOVsM FO could overfeed S/Gs 57
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED EVN Factors? INAPPROPRIATE EVENT i iito CTION
~1i~d\ easo~ CONTRIBUTING (ROO~\ (CAUSAL (Investi- FACTOR VRR-28 describes recirc line function occurred In damage Pump 8 minutes 58
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY Conservative UNVERIFIED EVENT E Faclorsf INAPPROPRIATE CTIO decision ondilion 1JNERII~d for CONTRIBUTING R CAUSAL
",.CONDITION :,: i FACTOR
,'-IST program has a safety function to close for AFW recirc Svalves ,
59
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY A
UNVEIFID EVNT F~do~/) INAPPROPRIATE EVENT o.Cnd'itonZ CTION Wrong Assumption Based on NU!REG- NUREG-0800 not FMEA based on design L - -- -- - -
0800 a~ssumptio~ns part of PBNP CLB funcion thuhIob iJNERFI~N~ for CONTRIBUTING conservative CASE CONDITAO C NDITIN, Investi-110io FACTOR
" Fault tree based "
primarily on design functions
-, Evaluated AFW ,- Assumption on -"
pump operations with long time available recirc & disch AOVs for AFW flow control N- closed . 11 actions I
-1995 Affects of excessive AFW flow put into EOPs
' Scope is original '
PRA based on 9/90
. plant snapshot _
60
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY UNVERIFIEDEVENT INAPPROPRATE EVENT i (Fondtions CTIO CONTRIBUTING OOTr "( CAUSAL FACTOR
"\~CA~J~"-.c2N*,oIT-" fbor Scope is movement of time critical steps from Appx. to main 61
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated - T -- S of InstrumentAir KEY UNVERIFIED Vactorsa) INAPPROPRIATE EVENT (a ) CTION CONTRIBUTING
( CAUSAL Inves"J FACTOR
""CAUSE C.NDITION.,
Loss of Nonvital bus causes 2 unit scram and degraded AFW 62
Increased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY jI UNVERIFIED VN Factorst INAPPROPRIATE EVENT CTIO CAUSAL nve
,E CONDITION ./
63
hicreased CDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY 1UNVERIFIED 1 < INAPPROPRIATE 1EVEN4T EVENTtio CTION ROO ( CAUSAL ) Investi-) FACTOR CAS CON DIT ON ~ \ationV 6/01 Late June or Early PRA group revising July '01 PRA group AFW portion of PRA identifies concern with model AOP-SB 64
Increased CDFin AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of Instrument Air KEY UNVERIFIED Factosa)
EVENT INAPPROPRIATE EVENT CTION (I'Or
\Investio R" ( CAUSAL CONTRIBUTING FACTOR -
\,CLJS,.~CONDITONI,,
11/26/01 1520 Risk evaluation 11/29/01 performed on AFW Interim corrective recirculatlon valves actions Implemented 65
IncreasedCDF in AFW PRA Model Due to Procedural RCE 01-069 InadequaciesRelated to Loss of InstrumentAir KEY UNVERIFIED I i EVENT EN (F tor INAPPROPRIATE CTION CONTRIBUTING CAUSAL ) nvesti FACTOR
',CONDITION, 66