Text

RCE 254, Potential Loss of Hot Leg Vent Path During Nozzle Dam Installation.
	ML051050045
Person / Time
Site:	Point Beach
Issue date:	04/09/2004
From:	Hettick D, Holmes J, Smoker C; Nuclear Management Co
To:	; Office of Nuclear Reactor Regulation
References
	CAP 055538, FOIA/PA-2004-0282 RCE 254
	Download: ML051050045 (66)
	v • d • e

i h A5 f

NMC_

Committed to NudurExceirnc PointBeach .I Rp'--

PotentialLoss ofHotLeg Vent Path duing Nozzle Dam Installation RCE 254 CAP 055538 Event Date: 4/09/2004 Principle Investigators:

Dennis Hettick - Team Lead, Point Beach Assessment Chuck Smoker - Hudson Assessment Jim Holmes - Point Beach Training Approvals:

Team Leader - Dennis Hettick Management Sponsor - Pat Russell 1+1-I

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Table of Contents I. Executive Summary .......................................................... 3 II. Event Narrative ............................................................

III. Extent of Condition Assessment .......................................................... g 9 IV. Previous Similar Events .. .................................... lo V. Nuclear Safety Significance ......................................................... 11 VI. Reports to External Agencies ........................................... 11 VII. Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Evaluation Methodology & Analysis Techniques .......................................... 12 Findings from the NMC Incidence Response Team ........................................ 12 Procedure adequacy - Operations Procedures .......................................................... 14 Procedure adequacy - Maintenance Procedures ......................................................... 17 Training adequacy..................................................................................................................... 19 Operational Decision Making ......................................................... 21 Outage Preparation ........................................................... 22 Generic Letter 88-17 Response, Implementation, and Tracking . .............................................

23 Response to Westinghouse Midloop Calculations ......................................................... 26 IPTE Briefs .......................................................... 27 RMP 9030, Unit 1 and 2 Pressurizer Manway Cover Removal and Installation ..... ................. 28 Station Log Keeping ......................................................... 28 Response to SOER 88-3 ......................................................... 28 Data Analysis Summary ......................................................... 29 Failure Mode Identification....................................................................................................... 30 VIII. Root Causes & Contributing Factors . .................................................

31 IX. Corrective Actions ......................................................... 34 X. References .............................................. 37 XI. List of Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

1. Engineering Evaluation of Hot Leg Vent Path .. 40

2. Graphic Sequence of Nozzle Dam Installation on SG Hot Legs . .40

3. RCE Charter .. 40

4. Event Time Line .. 40

5. Nozzle Dam Photograph .. 40

6. Event and Causal Factor Chart .. 40 2

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation I. Executive Summary

Purpose:

This investigation was performed to determine the root and contributing causes surrounding a significant near miss that occurred during the steam generator nozzle dam installation.

Specifically, workers were allowed to proceed with the installation of hot leg nozzle dams prior to establishing a hot leg vent path.

Event Synopsis:

Unit 1 refueling outage U1R28 was in progress on 4/09/2004 with the RCS drained to midloop in preparation of installation of steam generator nozzle dams. Nozzle dams had not been used at the station since October 1992.

Installation of cold leg nozzle dams in both steam generators was reported complete at 0537.

Removal of the pressurizer manway (the hot leg vent path) was working at the same time, and difficulty was encountered due to a stuck bolt. Removal of the pressurizer manway was a predecessor to the installation of hot leg manways. The unit was in a high risk condition (orange path) due to the RCS being drained to midloop. The shift outage manager questioned if the hot leg nozzle dams could proceed prior to removal of the pressurizer manway. The shift outage manager enlisted the help of other SROs to determine if this was an acceptable course of action.

After review of operating procedures, the SROs determined that it would be acceptable to install hot leg nozzle dams and the decision was made to proceed with this course of action.

Installation of SG A hot leg nozzle dam was started at 0536 after receiving approval to proceed.

Installation of the A hot leg nozzle dam could not be completed because of difficulty installing one bolt on the upper segment of the nozzle dam. Installation of the hot leg nozzle dam in the B SG began at 0643 and was completed shortly after without incident.

During the course of morning turnover activities, the shift outage manager was questioned about the need for a hot leg vent. The response was that the issue had been researched, and the current course of action was acceptable. Further research indicated that installation of the hot leg nozzle dam without a hot leg vent was not acceptable. The Outage Control Center ordered all of the nozzle dam segments be removed from SG A.

The pressurizer manway was removed at 0847, establishing a hot leg vent path.

The plant manager had placed nozzle dam installation activities on hold due to personnel safety issues related to bubble hood air supply problems. After the hold was released nozzle dam installation of the A SG hot leg began and was installed, verified, and tested by 1420.

Operations subsequently raised RCS water level and exited midloop operations.

==

Conclusions:==

The station's inadequate response, identification, tracking, and maintenance of committed actions to GL 88-17, resulted in challenges to many organizations at the station that were not overcome.

3

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Nuclear Safety Significance:

Actual nuclear safety significance of having hot leg dams installed without a hot leg vent path was very close to base line for midloop operations. The hot leg nozzle dam for steam generator A was not completely installed because of a damaged bolt hole. An engineering analysis was performed that showed that an adequate hot leg vent path is available as long as one of the side pieces is not installed. The PRA was conservatively analyzed for having all nozzle dam segments installed for one minute. The PRA group indicated a risk of 1.9E-08 (green, very low risk significance).

Potential nuclear safety risk is high. Actual risk would have been higher had there not been delays is installing the dam due to the damaged bolt hole. More troubling is the lack of understanding of the nuclear safety significance of using nozzle dams, and potential damage that can occur if a loss of shutdown cooling occurred when in a vulnerable plant condition. For this reason, this event is being treated as a very serious near miss.

Root Cause:

The root cause of this event was Point Beach's inadequate response, identification, tracking, and maintenance of the actions taken in response to the expeditious actions in Generic Letter 88-17.

A significant contributing cause was the failure to apply the lessons learned from the events described in the generic letter to continuing training.

Contributing causes were the lack of a maintenance procedure to control the prerequisites for hot leg nozzle dam installation, and operational decision making.

Corrective Action Synopsis:

Corrective actions include:

Monitor improvements made to committed action tracking (both licensing basis, and non-licensing basis commitments), and report results to CARB.

Develop an operations procedure to provide operational control of nozzle dam installation.

Improve training on bases/fundamentals for high risk activities/evolutions.

Develop a maintenance procedure to for nozzle dam installation.

Improve operational decision making.

Reports to external agencies OE 18191 was provided to MNPO as a preliminary notification.

No reports to the NRC are required.

4

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation II. Event Narrative This event has roots in areas that are latent, as well as active. Thus this event narrative begins in 1987, and terminates in 2004, a span of nearly 17 years.

The Nuclear Regulatory Commission (NRC) became concerned with challenges to nuclear safety related to loss of cooling to shutdown reactors. This concern culminated in the issue of Generic Letter (GL) 87-12, "Loss Of Residual Heat Removal (RHR) While The Reactor Coolant System (RCS) Is Partially Filled", on July 9, 1987. Point Beach responded to GL 87-12 in September 1987 addressing each issue and citing several improvements made to improve nuclear safety.

In the spring refueling outage of 1988, Diablo Canyon lost RHR for a sufficient time to result in boiling in the core. The NRC responded with an Augmented Inspection Team (AIT). The AlT reported their findings in NUREG 1269 and noted that there were indications that water, which would ordinarily be available to cool the core, was forced out of the reactor vessel through several paths. The understanding the NRC gained from the review of this event provided the basis for Information Notice (IN) 88-36, "Possible Sudden Loss Of RCS Inventory During Low Coolant Level Operation", which was issued in June of 1988. The purpose of the IN was to alert the industry to the potential for sudden loss of RCS inventory when nozzle dams were installed in the hot legs of the steam generators. A loss of RHR flow while in reduced inventory would quickly result in boiling in the reactor core, pressurizing the RCS if no vent path was available.

A pressure of a little over 2 psi would lower the coolant to the top of the fuel if a hot leg nozzle dam was installed, and the cold leg was open. If hot and cold leg nozzle dams were installed, with no hot leg vent path, pressure would increase rapidly after the onset of boiling, ultimately challenging the integrity of the nozzle dams. If a cold leg nozzle dam was expelled, coolant would be ejected through the open manway and the core would rapidly uncover. The IN recommended that rapid uncovery of the core through this mechanism could be avoided by having an adequate hot leg vent. In addition, the IN stated that 'not installing the last hot leg nozzle dam until a sufficient vent path is established in the reactor vessel or pressurizer will reduce the possibility of developing a pressure differential which could eject a dam.'

In the period of July 1988 through October 1988, Managers Supervisory Staff (MSS) discussed IN 88-36, including Westinghouse Owners' Group (WOG) efforts in the area, in three separate meetings. MSSM 88-16 in July 1988 briefly discussed the potential impacts to the upcoming outage. MSSM 88-17 in August of 1988 included extensive discussion of the IN, and several short term follow up actions were identified in the meetings minutes. The minutes indicated that MSS discussed that all hot legs cannot be blocked unless a vent path is provided from the reactor vessel which is large enough to prevent RCS pressurization. There was no action assigned to ensure this requirement was implemented.

NEPB 88-366 issued in September 1988 reviewed IN 88-36 and made three recommendations, noting that the WOG and the NRC were continuing to study the issue, and would likely provide recommendations in the future. The three recommendations were to procedurally control the sequence of nozzle dam installation and removal, inform the operating staff of the potential need for injection to the core, and maintain two thermocouples connected as long as possible.

Establishment of a hot leg vent path is not addressed in the memo.

5

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation MSS reviewed Safety Evaluation Report (SER) 86-047-02, Steam Generator Nozzle Dam installation, as documented in MSSM 88-22 Addendum B in minutes dated October 18, 1988.

Addendum 2 to the safety evaluation was added to reflect changes in plant operation during the use of steam generator nozzle dams. The MSS listed 6 items in the meeting minutes: 1.

installation and removal sequence; 2. one safety injection pump available; 3. containment closure in effect when at mid loop; 4. reactor head detensioned or removed (or one of the following); 5.

establish the following if the head is tensioned - reactor coolant pumps disabled, RHR administratively controlled to prevent pressurization of the RCS when nozzle dams are installed, both PORVs and their associated block valves open; 6. all cold leg openings with exception of steam generator manways are closed when nozzle dams are installed.

Special Maintenance Procedure SMP #907 was issued on 10/11/1988. This procedure addressed the installation and removal of steam generator nozzle dams. In the prerequisite section, step 2.1.8 stated that one of the following conditions must be met: 1. The reactor vessel head is detensioned or removed. 2. Both PORVs must be blocked open with their associated block valves open. The procedure also cautioned about the correct sequencing of installation and removal.

IN 88-36 was closed out as documented in MSSM 88-23 in November 1988. The actions documented in the minutes were: 1, Ensure correct sequence of installation and removal of nozzle dams (Special Maintenance Procedure, SMP #907), 2. Prepare standing orders for operations to supplement training performed under lesson plan (LP) 1663, and 3. Maintain at least two core thermocouples connected as long as possible.

The NRC issued Generic Letter GL 88-17 in October 1988 due to concerns about continued events in the industry related to the loss of shutdown cooling. The Diablo Canyon event in the spring of 1988 was prominently included in the generic letter. The generic letter listed eight recommended expeditious actions and six programmed enhancement recommendations.

Two of the recommended expeditious actions are of particular interest to this investigation:

(1) Discuss the Diablo Canyon event, related events, lessons learned, and implications with appropriate plant personnel. Provide training shortly before entering a reduced inventory condition.

(7) Implement procedures and administrative controls that reasonably assure that all hot legs are not blocked simultaneously by nozzle dams unless a vent path is provided that is large enough to prevent pressurization of the upper plenum of the RV.

Special Maintenance Procedure SMP #907, Installation/Removal of Steam Generator Nozzle Dams Unit 2, was issued on October 11, 1988. This procedure contained prerequisites, among others, to establish a hot leg vent path (detensioned or removed reactor vessel head or PORVs);

control cold leg openings; and establish containment integrity. The SMP also contained precautions for the proper sequencing of nozzle dam installation and removal.

Point Beach replied to the recommended expeditious actions in December 1998.

Item (1) was implemented via training Lesson Plan #1663. The response indicated this was completed in cycle 88-6 on October 8, 1988 before entering reduced inventory operation, and before GL 88-17 was received on November 3, 1988. The response detailed what was covered in the lesson plan, and indicated that a separate Training Needs Analysis (TNA) covers the addition of industry events, in general, to initial 6

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation operator lesson plans. Future procedure changes, standing orders, etc., that pertain to mid-loop operations will be placed into the TNA system for inclusion in the training programs.

Item (7) was implemented by plant procedure SMP-907. The response indicated this procedure required that the cold leg nozzle dams be installed prior to the hot leg nozzle dams and not removed until after the hot leg nozzle dams are removed. The response continues by stating that the procedure was implemented during the refueling outage that started on October 8, 1998 and similar procedures for nozzle dam installation will be used during all future refueling outages.

Revision 0 of procedure OP 4F, Reactor Coolant System Reduced Inventory Requirements, was issued on March 27, 1989. The stated purpose of the procedure is to define reduced inventory conditions and requirements and included a statement that the procedure implements commitments made per Generic Letter 88-17. Requirements for sequencing of SG nozzle dam installation and removal, and requirement for a hot leg vent when nozzle dams are installed are not included in the procedure, although there is a step for the condition of an opening in the cold leg greater that 1 square inch without a hot leg vent. This condition requires MSS Concurrence.

Westinghouse transmitted their Midloop Calculations to Point Beach under WEP-89-143 on June 30, 1989. These calculations addressed hot leg vent size and time since shutdown to keep pressurization of the reactor vessel less than 2 psi. The calculations indicated PORVs and smaller vent paths could not be used until much later than pressurizer safeties, which required 225 hours0.0026 days 0.0625 hours 3.720238e-4 weeks 8.56125e-5 months after shutdown. The maintenance procedures were never updated to remove the PORVs as an allowed vent path.

In October 1989, MSS approved RMP-141 and RMP-142, routine maintenance procedures for nozzle dam removal and installation. These procedures had DSS signoffs for a reactor vessel vent, head detensioned or removed, both PORVs blocked open, or pressurizer manway removed.

The procedure did not incorporate the information from the Westinghouse Calculation.

On October 2, 1989, DSS granted permission to start work (install nozzle dams) per RMP 142, utilizing the PORVs as the reactor vessel vent path.

Nozzle dams were installed in Unit 2 outages in October 1990, 1991, and 1992. The reactor vessel vent paths were pressurizer manway, pressurizer safety valves (added as a vent via a temporary change in October, 1991), and pressurizer manway in these outages, respectively.

The temporary change cited safety evaluation SER 86-047 and did not incorporate the time after shutdown calculations from the Westinghouse Midloop Calculations (two safety valves removed requires about 225 hours0.0026 days 0.0625 hours 3.720238e-4 weeks 8.56125e-5 months shutdown).

INPO evaluated the implementation of recommendations from SOER 88-3, Losses of Residual Heat Removal with Reduced Reactor Vessel Water Level at PWRs, in September 1990.

Recommendations 2, 3, and 4 of the SOER were re-opened. Recommendation 2.b. addressed methods to establish and maintain a hot-leg vent path to prevent pressure build-up in the reactor vessel. Point Beach response was "OP 4F Step 6.0 specifies acceptable hot-leg vent paths." At this point in time (January, 1991) Revision 4 of OP 4f was in effect, and listed acceptable vent paths in section 6. However, the procedure referenced figure 1 to determine closure requirements. Following this flow chart, if the hot leg opening was NOT OK, a 2 hour2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months closure time was allowed if cold leg openings were less than 1 square inch.

7

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation RMPs 141 and 142, removal and installation of SG nozzle dams, were cancelled in November 1993. This left no maintenance procedures to control the installation and removal of SG nozzle dams.

The Unit 2 fall outage of 2003 planned to use nozzle dams. During planning in the spring of 2003, the outage safety review identified requirements for hot leg vent path and the need for JIT for operations as nozzle dams had not been used for several years.

In March 2004, PORC reviewed the Unit 1 outage risk plan, noting that the plan is very similar to the plan for the Unit 2 outage.

In April 2004 Operations began establishing plant conditions for entering mid loop operations to allow the installation of SG nozzle dams. Draindown commenced at 0245 on 4/07/04 using procedure OP 4D Part 1. Drain down was completed with reactor vessel level between 22% and 25% on 4/08/04 at 0338. Protected equipment and signage was established as required by the drain down contingency plan.

After draining to mid loop, conditions allowed removal of SG primary manways. SG A manways were removed about 1600 hours0.0185 days 0.444 hours 0.00265 weeks 6.088e-4 months , followed by SG B manways about 2300 hours0.0266 days 0.639 hours 0.0038 weeks 8.7515e-4 months .

An IPTE brief was conducted for nozzle dam installation about 0100 hours0.00116 days 0.0278 hours 1.653439e-4 weeks 3.805e-5 months on 4/09/04. Nuclear Oversight observed this brief, and indicated in their rapid trending report that the brief was unacceptable. Additionally, senior management did not attend the brief as required by the IPTE procedure.

Maintenance reported to the OCC at 0400 hours0.00463 days 0.111 hours 6.613757e-4 weeks 1.522e-4 months that they were having difficulty removing the last bolt for the pressurizer manway.

Sometime after the IPTE brief and the report of difficulty in removing the pressurizer manway, the OCC Shift Outage Manager (SOM) discussed the potential of continuing with installation of nozzle dams without a hot leg vent path. The discussion focused on sequencing the installation, and whether a cold leg opening would be present. The SOM enlisted the assistance of other operations personnel to review the situation and determine if nozzle dam installation could proceed. After reviewing OP 4F and other document(s) the conclusion was reached that it would be permissible not to have a hot leg vent, as long as the cold leg openings were closed.

Installation of SG B cold leg nozzle dams began at 0434. During installation of this Dam, the contract installer experienced low air pressure requiring him to exit the SG bowls. Issues surrounding steam generator bowl jumps are covered in RCE 253. Of interest to this investigation, is the fact that problems doing the steam generator jumps resulted in work delays, which precluded hot leg nozzle dams from being installed on both SGs simultaneously.

Installation of SG A cold leg nozzle dams began at 0452, in parallel with work on SG B. The station log indicated that nozzle dams were installed in the cold legs of both the A and B SG at 0537.

At 0536, the contractor nozzle dam procedure was signed by the plant representative (nozzle dam installation engineer) indicating plant approval to proceed with installation of the hot leg dam in A SG. There is no sign-off for the operations authority, or allowance to identify the operations authority granting permission in the contractor procedure.

8

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation By 0555, the nozzle dam center and both sides had been placed in SG A bowl. One of the left-hand bolts on the upper side could not be installed. This resulted in removal of the nozzle dam sides from the bowl at 0633 to allow the center dam to be rotated.

Simultaneous with the installation effort on SG A hot leg nozzle dam, the Site Director questioned the OCC on the need for vent path sequencing during installation of nozzle dams.

Based on earlier efforts looking at the schedule sequence, the Site Directors concern was answered indicating that the current path was permissible.

Installation of the hot leg nozzle dam in the B SG began at 0643 and was completed shortly after without incident.

During the period of time after shift turnover when the Site Director asked about sequencing nozzle dam work and the installation of the hot leg nozzle dam in B SG, the Outage Director brought a copy of NUREG 1449 to the OCC, stating the requirement for a hot leg vent path.

Fortuitously, the problems with the bolthole in SG A hot leg precluded full installation of this nozzle dam simultaneously with the SG B hot leg nozzle dam, while the pressurizer manway was yet removed. Details of nozzle dam installation, and resultant nuclear safety risk, are addressed in the Safety Significance section of this report.

OCC logs indicate that removal of the nozzle dam from SG A bowl was ordered at 0652. The FME logs indicate that all nozzle dam pieces were removed by 0656. Interviews with the steam generator jumpers indicated that they installed a "top hat" (FME cover) after removal of the nozzle dam center section, which included the flexible diaphragm. Installation of the hot leg nozzle dam in SG B was allowed to continue.

Installation of the B SG hot leg nozzle dam was completed and logged in the OCC log at 0734.

The pressurizer manway and diaphragm were removed at 0847, establishing a hot leg vent path.

The plant manager put nozzle dam installation on hold at about 1100. Protocol was developed to address personnel safety issues related to nozzle dam installation. All protocol items were complete, and nozzle dam installation of the A SG hot leg began about 1212 hours0.014 days 0.337 hours 0.002 weeks 4.61166e-4 months . All nozzle dams had been installed, verified, and tested by 1420.

Operations began raising Reactor Vessel level at 1508, and exited reduced inventory at 1554.

Level was stabilized at 70% at 1611.

III. Extent of Condition Assessment Procedures related to nozzle dams Station procedures were searched for procedures invoking hot leg vent paths. The following procedures were found, which reference a hot leg vent path, and require review for consistency and accuracy. Procedure changes are to be made, if appropriate.

1. IRMP 9007 Unit 1 Steam Generator Primary Manway Cover Removal and Installation 9

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation

2. 2RMP 9007 Unit 2 Steam Generator Primary Manway Cover Removal and Installation

3. RMP 9030, Unit I and 2 Pressurizer Manway Cover Removal and Installation

4. RMP 9002-7, Reactor Coolant Pump Strongback Installation and Removal

5. RMP 9002-4, Reactor Coolant Pump Uncoupling and Coupling

6. RMP 9004, Power Operated Relief Valve Maintenance

7. IRMP 9096, Reactor Vessel Head Removal and Installation

8. 2RMP 9096, Reactor Vessel Head Removal and Installation

9. RMP 9054-1, Pressurizer Safety Valve Removal and Installation

10. RP 1B, Recovery from Refueling Identification, Implementation, and maintenance of responses to the NRC Based on the information provided in the following section, maintenance of commitments is a broad problem at Point Beach. The large majority of the issues cited below are not licensing basis commitments, but nonetheless, reflect actions that we said we were going to do, but have not implemented and/or maintained. As this issue is related to the root cause, corrective actions to prevent recurrence have been developed to address this extent of condition.

IV. Previous Similar Events Commitment Identification and Tracking A search of commitment issues was made using the keyword *commitment

as search criteria in tTrack. This search yielded 823 hits. Review of the subjects indicates that the station had considerable problems related to missed commitments, potential to miss commitments, commitment tracking, commitment timeliness, and database issues related to commitment tracking. Quality Assurance identified QA significant issue 98-04 in 1998 based on a number of Condition Reports and Quality Condition Reports. The record of the disposition and results of this QA significant issue is tracked in tTrack under CAP 1689. This resulted in a several year project to upgrade the stations commitment tracking program. The QA significant issue was closed out in 2002.

The goal of the project was to ensure that all licensing basis commitments were included in the program in accordance with NEI guidance contained in NEI 98-05 and 99-04 regarding commitment management, as well as similar NRC guidance contained in SECY 95-300 and SECY 98-224. These Generic Letter commitments, however, are not germane to the QA Significant Issue regarding the "commitment management system" QA Significant Finding. The actions resulting from the response to GL 88-17 were maintained in NUTRK since they are not "licensing basis" commitments as defined by the NEI and NRC guidance documents. When the NUTRK to tTrack conversion was performed in December 2003 for regulatory-related items, Generic Letter response actions were not downloaded into either tTrack or the Regulatory Information System. CAP 54188 was initiated to effect corrective actions, and a non-licensing 10

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation database is being developed to aid tracking these type of items, such as GL responses (CA 56084).

Lack of knowledge, or appreciation for nuclear safety The tTrack database was searched for significance level A events, with results going back to mid- 1996. The list of events was reviewed, looking for instances where a significant event occurred, and was related to an operational misunderstanding, lack of knowledge, or lack of appreciation for nuclear safety. This review yielded three potential events: The freezing of a safety injection recirculation line in the facade; a manual reactor trip in response to a lowering forebay level due to formation of frazil ice in the intake crib; and gas binding of a safety injection pump due to leakage from a safety injection accumulator.

A nuclear safety culture assessment is being performed under CAP 56175 which addresses this issue.

V. Nuclear Safety Significance Actual nuclear safety significance of having hot leg dams installed without a hot leg vent path was very close to base line for midloop operations. A graphic of nozzle dam pieces installed and removed from the hot legs of the steam generators is included in attachment 2. This graph was constructed from the FME logs. An engineering analysis was performed that showed that an adequate hot leg vent path is available as long as one of the side pieces is not installed. The FME logs showed a potential for all the nozzle dam sections to be installed in the bowl of SG A for one minute, at 0647. Interviews with the SG A nozzle dam installers indicated that the occurrence at 0647 was a timing issue, as the dam was handed out, and handed back in after trying to tap the bolt hole. As the FME log is the only documented record, 1 minute with all 3 dam sections in the bowl was conservatively used in determining risk. The PRA group indicated a risk of 1.9E-08 (green, very low risk significance).

Potential nuclear safety risk is high. Actual risk would have been higher had there not been delays is installing the dam due to the damaged bolt hole. More troubling is the lack of understanding of the nuclear safety significance of using nozzle dams, and potential damage that can occur if a loss of shutdown cooling occurred when in a vulnerable plant condition. For this reason, this event is being treated as a very serious near miss.

VI. Reports to External Agencies This event did not require a verbal or written notification of the NRC under 10CFR50.72 or 73.

The NRC resident inspector was monitoring nozzle dam installation, and was aware of the issue.

A preliminary report was made to INPO under OE 18191 An internal OE was shared with the NMC on 4/15/2004 11

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation An action is being tracked under CAP 055538 to post a final OE to the industry.

VII. Data Analysis Evaluation Methodology & Analysis Techniques Event and Causal Factor Charting was the primary investigation technique used in the root cause investigation.

Barrier analysis was used by the Incident Response Team, and communicated barrier weakness based on the "Barriers for Excellence" from the site picture of excellence.

Both the Incident Response Team and the Root Cause Team used interviewing. The RCE team interviewed over 25 individuals during the course of the investigation.

Industry Operating Experience The INPO database was searched and 8 hits related to nozzle dam events were found, one being the OE submitted by Point Beach. The external events were related to nozzle dam leaks, radiation exposure, and FME related events. The INPO website also contained Just In Time (JIT) Operating experience. The JIT contains links to additional operating experience. None of the linked OE was related to failure to establish a hot leg vent path. The JIT OE does address the need for a hot leg vent path in the Engineering section of the JIT.

The INPO JIT was used in the briefings that were performed prior to nozzle dam removal.

There is no record that the JIT was used in the briefings, for nozzle dam installation, although review of the briefing material and interview comments indicate that the need for a hot leg vent was covered in the pre-installation brief.

Findings from the NMC Incidence Response Team A team of NMC personnel responded to Point Beach shortly after the events of 4/09/2004 to review the activities that led up to two events: the decision to proceed with installation of steam generator nozzle dams before the required vent path (pressurizer manway) was removed; and several problems associated with use of forced air bubble hoods.

The team was composed of the following individuals:

Paul A. Harden (Team Lead) Palisades Director, Site Operations Craig Chovan NMC Fleet Outage Director Dave Geisen Kewaunee Nuclear Oversight Manager Tom Taylor Prairie Island Performance Assessment Manager Kari Den Herder Prairie Island Program Engineer Joe Maurer Prairie Island Outage Manager Al Capristo NMC Employee Concerns Program Manager The Incidence Response Team performed numerous interviews and document reviews, and compiled a list of issues that had occurred during these events. These issues were then characterized by failed barriers using the six Barriers for Excellence associated with the Point 12

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Beach Picture of Excellence. The results from the barrier analysis identified the primary issues that contributed to these events:

Failure to recognize the significance and potential consequences of activities related to reduced inventory and steam generator nozzle dam installation

Lack of clarity of specific vent paths and requirements in procedure OP 4F and lack of similar specific prerequisites in the nozzle dam installation procedure

Lack of a clearly understood process for decision making and resource involvement in the OCC for changes to outage schedule logic

Lack of questioning attitude, advocacy of questions and self-critical nature

Roles, responsibilities and expectations were not clearly defined, communicated and/or adhered to (including adherence to site procedures)

Failure to establish clear ownership and accountability at all levels from project/job leads to management oversight

Inadequate threshold for raising issues to management and use of the CAP 13

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Procedure adequacy - Operations Procedures

==

Conclusion:==

Procedure OP 4F was not intended to control or verify hot leg vent path during the installation of steam generator nozzle dams. For this reason, the procedure was not effective in requiring establishment of a hot leg vent prior to installation of the hot leg nozzle dam.

Hot leg vent paths are correctly specified based on the Westinghouse Midloop Calculations.

Hot leg vent paths are provided to allow determination of containment closure time in accordance with the guidance for containment closure in enclosure 2 of the GL.

A procedure change request to add verification of hot leg vent paths prior to nozzle dam installation was initiated and requested incorporation prior to next revision. This request was not completed prior to use.

OP 4D Part 1 does not reference GL 88-17 for Water Level Indications Revision 0 of procedure OP 4F, Reactor Coolant System Reduced Inventory Requirements, was issued on March 27, 1989. The stated purpose of the procedure is to define reduced inventory conditions and requirements and included a statement that the procedure implements commitments made per Generic Letter 88-17.

Specific items from the generic letter which were addressed by the procedure were not identified.

Requirements for sequencing of SG nozzle dam installation and removal, and requirement for a hot leg vent when nozzle dams are installed are not included in the procedure.

This procedure does address, in part, some of the items from GL 88-17:

Item 1 - Diablo Canyon Training - not addressed by this procedure

Item 2 - Containment Closure

Item 3 - RCS temperature indications

Item 4 - RCS Water Level indications - this item addressed by OP 4D Part 1, Draining the Reactor Coolant System. Note that GL 88-17 is not addressed in the purpose section of this procedure OP 4d Part 1.

Item 5 - RCS perturbation

Item 6 - RCS Inventory Addition

Item 7 - Nozzle Dams - Not addressed by this procedure, except in Appendix where credit is given for an open manway as a vent, if the hot leg nozzle dam is not installed.

There are no vent requirements identified for the condition where a hot leg nozzle dam is installed.

Item 8 - Loop Stop Valves - Not applicable to Point Beach INPO evaluated the implementation of recommendations from SOER 88-3, Losses of Residual Heat Removal with Reduced Reactor Vessel Water Level at PWRs, in September 1990.

Recommendations 2, 3, and 4 of the SOER were re-opened. Recommendation 2.b. addressed 14

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation methods to establish and maintain a hot-leg vent path to prevent pressure build-up in the reactor vessel. Point Beach response was "OP 4F Step 6.0 specifies acceptable hot-leg vent paths." At this point in time (January, 1991) Revision 4 of OP 4F was in effect, and listed acceptable vent paths in section 6.

6.1 A steam generator hot leg manway opening is acceptable if its hot leg nozzle dam is not installed.

6.2 A steam generator cold leg manway opening is acceptable if its hot leg nozzle dam is not installed and its tube bundle is drained.

6.3 The reactor vessel head raised to greater than or equal to 1 inch above the reactor vessel flange is acceptable.

6.4 The pressurizer manway open is acceptable if the reactor has been shut down for greater than or equal to 110 hours0.00127 days 0.0306 hours 1.818783e-4 weeks 4.1855e-5 months 6.5 Both pressurizer code safeties removed is acceptable if the reactor has been shut down for greater than or equal to 240 hours0.00278 days 0.0667 hours 3.968254e-4 weeks 9.132e-5 months .

Revision 4 of the procedure referenced figure 1 to determine closure requirements. Following this flow chart, if the hot leg opening was NOT OK, a 2 hour2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months closure time was allowed if cold leg openings were less than 1 square inch.

The vent paths listed in section 6 of OP 4F were acceptable based on Westinghouse Midloop Calculations, which identified 100 hours0.00116 days 0.0278 hours 1.653439e-4 weeks 3.805e-5 months post shutdown for the pressurizer manway, and 225 hours0.0026 days 0.0625 hours 3.720238e-4 weeks 8.56125e-5 months post shutdown for the 2 pressurizer safety valves.

The vent paths listed in section 6 were introduced in revision 2 (10/30/1990) and remained in section six through revision 10, until the vent paths were moved to Figure 1 in revision 11 (11/03/1995). The list of vent paths remained in Figure 1 through revision 19, which was performed on 4/09/2004.

Guidance for meeting GL 88-17 is contained in Enclosure 2 to the GL. The expeditious action of GL 88-17 addressing containment closure allowed that a containment closure of 2.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months after loss of DHR was allowable, but required modification under certain conditions. The 2.5 hour5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months requirement was to be replaced with a 30 minute requirement if openings totaling greater than one square inch exist in the cold legs. This time could be increased to two hours if a vent path from the upper RV is provided which is sufficiently large that core uncovery cannot occur due to pressurization resulting from boiling in the core.

A procedure change request was written by a member of Production Planning Outage against OP 4F indicating that a required hot leg vent path for nozzle dam installation was not addressed in the procedure. The initiator identified recommended incorporation into the next revision. The PCR was written on 3/16/04. A CAP was not initiated, and the procedure was not changed prior to use.

Interviews with several SROs and Operations Procedures personnel indicated that OP 4F was never intended to address the installation, control, or removal of nozzle dams. The vent paths are specified to be used in determining if MSS approval is required based on the size of openings in the cold legs of the RCS.

15

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation A review of the procedure revision history of OP 4F, indicates that the procedure never addressed verification of a vent path for installation of nozzle dams from the initial revision 0 (3/27/89) through revision 19 (9/04/2003), which was in effect during the unit 1 outage.

The stated purpose of OP 4F Revision 0 (MSS Review Summary) indicated: "Original Procedure. This Instruction is to establish the requirements for containment closure and core make-up during reduced inventory condition of the Reactor Coolant system."

16

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Procedure adequacy - Maintenance Procedures

==

Conclusions:==

Maintenance procedure RMP 142 for installation of nozzle dams in Unit 2, provided explicit DSS signoffs, specifying which hot leg vent path was being used. This was a strong administrative control.

RMP 142 incorrectly allowed the use of PORVs as a vent path. The procedure did not incorporate the analysis provided by the Westinghouse Midloop Calculations due to informality in receipt and evaluation of the Westinghouse letter.

The RMP did not identify that the procedure implemented a response to GL 88-17 expeditious item 7, and was subsequently cancelled, leaving no procedure in place to incorporate the response to the generic letter.

The Scientech procedure did not maintain positive administrative control to establish a hot leg vent prior to installation of the hot leg nozzle dam.

Item (7), expeditious actions, of GL 88-17 stated: "(7) (applicable to Westinghouse and Combustion Engineering nuclear steam supply system (NSSS) designs) Implement procedures and administrative controls that reasonably assure that all hot legs are not blocked simultaneously by nozzle dams unless a vent path is provided that is large enough to prevent pressurization of the upper plenum of the RV. See references 1 and 2."

PBNP responded to the Generic Letter request via VNPD-88-635 on December 30, 1988, indicating that expeditious item 7 - Nozzle Dams was implemented by plant procedure SMP-907. The response indicated this procedure required that the cold leg nozzle dams be installed prior to the hot leg nozzle dams and not removed until after the hot leg nozzle dams are removed.

The response continues by stating that the procedure was implemented during the refueling outage that started on October 8, 1998 and similar procedures for nozzle dam installation will be used during all future refueling outages.

SMP # 907, Installation/Removal of Steam Generator Nozzle Dams, was approved 10/7/88. This procedure addressed the installation and removal of nozzle dams in Unit 2. Requirement for a hot leg vent path was met by the following prerequisite:

2.1.8 One of the following conditions must be met:

1. The reactor vessel head is detensioned or removed.

2. Both PORVs must be blocked open with their associated block valves open.

RMP 142, Installation of Steam Generator Nozzle Dams Unit 2 was issued 10/2/89. RMP 142 effectively superseded the SMP for installation of nozzle dams.

The MSS review summary of the initial issue of the procedure stated: " This is the initial issue of this procedure which outline the necessary steps to install nozzle dams in the Unit 2 steam generators. It was previously controlled via a SMP and was revised to reflect midloop hot leg vent path concerns (IEN 88-36)"

The RMP does not reflect that the procedure was developed in response to GL 88-17.

17

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation The RMP controlled the hot leg vent path the through DSS signoff prerequisites:

2.1.7 One of the following conditions must be met:

a. The reactor vessel head is detensioned or removed (DSS)

b. Both PORVs must be blocked open with their associated block valves red tagged open. (DSS)

c. Pressurizer manway cover is removed.

(DSS)

RMP 142 was utilized in October 1989 using the PORVs as the hot leg vent path.

The RMP was used again in October 1990 using the pressurizer manway as the hot leg vent path.

A one time temporary change was made in October 1991 adding to condition b. (PORVs) "or the pressurizer safety valves removed and the upstream piping vented." Condition b. was used as the hot leg vent path in October 1991.

The last performance under RMP 142 occurred in October 1992 and used the pressurizer manway cover removed as the hot leg vent path.

RMP 141 and 142, removal and installation of SG nozzle dams, were cancelled in November 1993. Interviewees indicated the procedures were cancelled due to on-going industry concerns and concerns at Point Beach regarding shutdown safety. This was coupled with the stated beliefs that nozzle dam installation was saving little time, and pressure for short outage durations was not great.

Cancellation of the RMPs left no Point Beach maintenance procedures to control the installation and removal of SG nozzle dams.

Interviews indicated that the Scientech procedure for installation and removal of nozzle dams had no reference to a hot leg vent prior to the Point Beach fall 2003 outage. Subsequent interviews indicated that the Scientech procedure did address the fact that a vent path may be required, and found some lesser problems with the procedure, that were corrected by Scientech.

The nozzle dam HIT team deliberated over creation of a Point Beach procedure or use of the vendor procedure. Interviews indicated that maintenance resources precluded creation of a maintenance procedure. The HIT team management sponsor indicated that he had persuaded Scientech to add a caution about the necessity of a hot leg vent. This precaution was generic, and without an Operations' signoff. Scientech procedure revision 5 dated 12/12/2001 was found to contain the same caution as Revision 6, which is shown below.

18

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation The following caution is in Scientech procedure 83A7564, Revision 6, Appendix F, Installation Procedure, page Fl:

CAUTION Plant procedures may require one hot leg nozzle to be open as a vent path for the Reactor Coolant System. If the above dam is to be installed in the designated vent path nozzle, ensure this is the last dam to be installed and obtain Plant authorization to proceed with the dam installation.

Approval to Proceed:

Plant Representative Date/Time At 0536, the contractor nozzle dam procedure was signed by the plant representative (nozzle dam installation engineer) indicating plant approval to proceed with installation of the hot leg dam in A SG. There is no sign-off for the Operations authority, or allowance to identify the Operations authority granting permission in the contractor procedure.

Training adequacv

==

Conclusions:==

The training provided under LP 1663 in anticipation of GL 88-17 was adequate based on review of the lesson plan.

Based on the training documents reviewed, the training provided on the Diablo Canyon Event as specified in GL 88-17 was a one-time-teach, and was not carried over into continuing training, or Initial Licensed Operator Training.

Training provided prior to U2R26 was timely, but did not cover lessons learned from the Diablo Canyon event.

Much of the training is narrowly focused, such as training on nozzle dam control panel operations, loss of indicted level, loss of core cooling, and indications of pump cavitation and does not address or readdress knowledge fundamentals.

Generic Letter 88-17 Expeditious Action Item 1 - Diablo Canyon Event, had the following NRC Recommendation: "Discuss the Diablo Canyon event, related events, lessons learned, and implications with appropriate plant personnel. Provide training shortly before entering a reduced inventory condition."

Point Beach responded by performing a training needs analysis, and creating Lesson Plan #1663.

The response to the GL indicated that training was provided to licensed operators and Duty Technical Advisors during Cycle 88-6 of continuing training. The response indicated this training was completed on October 8, 1988, before entering reduced inventory operation on October 13, 1988 and before GL 88-17 was received on November 3, 1988.

LP 1663 was developed as a Plant Status Update, and presented in continuing training cycle 88-

6. The lesson plan covered various operations related items, but focused on Instructional Objective 1.2.6, Discuss NRC concerns with the loss of decay heat removal during mid-loop 19

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation operation. Transparencies TP-4 through TP-29 covered the training material related to GL 87-12, and advance information related to GL 88-17. The lesson plan focused on the following areas:

The Issue (concern of NRC), and background (occurrences, risk), and industry response to GL 87-12

Serious deficiencies and concerns identified by the NRC

Corrective actions identified by the NRC (Expeditious actions, programmed enhancements) o Control of Hot and Cold leg closures was addressed as part of this section, and specified that a vent path of sufficient size to prevent pressurization of the RCS must be provided if all hot legs were simultaneously blocked.

Discussion of the Diablo Canyon Event.

Training personnel reviewed Licensed Operator Requal Training and provided the following:

LOR 04-02, training on Reduced Inventory, Nozzle Dams and Hot Leg Vent Paths was held in March 2004, prior to U1R28. This training reviewed the construction and use of nozzle dams and the alarms available. The draft Orange Path Contingency Plan was reviewed. No direct tie to a nozzle dam failure scenario was made in this training.

LOR 03-05 (prior to U2R26) September, 2003 contained training on Shutdown Emergency Procedures on loss of RHR, and a loss of 480 VAC to Unit 2 while in mid-loop configuration. The scenario also included a brief discussion on how the required actions would have differed if nozzle dams were installed. LP 3791 provided a thorough discussion on the use of nozzle dams, and had a short video of Scientech nozzle dam demonstration. OI-11 was discussed, covering nozzle dam alarms, indications, and controls at the nozzle dam control panel.Section II, B.2.a.2), "Cold leg dam installed first and last to be removed for vent path considerations per OP 4F, Reactor Coolant System Reduced Inventory Requirements." The vast majority of the Nozzle Dam related training centered on design, installation and monitoring, which is what Operations is tasked with in regard to their use at PBNP. Other than the item mentioned above, nothing related to PZR vent path was included. The discussion was limited to complications and response, and did not address reasons for the actions, or the commitments.

U1R27 Mid-Loop Briefing, October 2002 covered anticipated normal procedure flowpath in OP 4F and decisions based on cold leg openings and hot leg vent paths. OE and Shutdown Emergency Procedures were also discussed.

LOR 98-06, October 1998 provided training on the Shutdown Emergence Procedures.

The basis for checking the cold leg openings and hot leg vent status was also covered.

Training personnel reviewed Initial Licensing Training and provided the following:

LP1308, TRPR 18.0 Control Operator - RCS Drain down / Reduced Inventory, Section 3.1.2.d.5 - Briefly discusses Figure 1 of OP 4F, and the acceptable hot leg and cold leg vents. These same items are contained within SEP-1.1, "Alternate Core Cooling", step 34.

20

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation LP2189, TRPR 18.0, 19.0, 20.0, CO, SRO, STA, "Loss of or Degraded RHR (SEP-I Series: Found no specific mention to a precautionary concern related to installed hot leg nozzle dams and an alternate RCS vent path within this LP. This LP seemed to deal with the RHR issues addressed in SOER 88-3 /GL 88-17. SEP-1.1, step 34 does list the hot leg vent paths. SEP-1.1 34 directs the operator to check for an adequate RCS vent path following verification that a clod leg opening greater than one square inch exists. To prevent steam voids forming in the reactor vessel head from forcing RCS inventory out of the cold leg opening and uncovering the core, an adequate hot leg vent path must be verified. The acceptable hot leg vent paths are identified. Subsequent actions are dependent on the existence of an adequate RCS vent path.

The fundamental lessons learned from the Diablo Canyon event were included in the training provided in 1988. Neither this lesson plan, nor the subject matter of core uncovery when in midloop was carried on into continued training.

The response detailed what was covered in the lesson plan, and indicated that a separate Training Needs Analysis (TNA) covers the addition of industry events, in general, to initial operator lesson plans. Future procedure changes, standing orders, etc., that pertain to mid-loop operations will be placed into the TNA system for inclusion in the training programs.

The nozzle dam hit team for the Unit 2 fall outage U2R26 identified through a benchmarking effort that training was required. This was incorporated into the U2R26 SG nozzle dam project plan, and was subsequently provided prior to the unit 2 outage.

In March 2004, PORC reviewed the Unit 1 outage risk plan, noting that the plan is very similar to the plan for the Unit 2 outage.

The Incidence Response Team found that knowledge of the requirement for a hot leg vent path was not well understood by all affected parties, including SROs, the Shift Outage Manager, and the OCC Engineering representative. Other individuals such as outage planning personnel and HIT team personnel had some level of knowledge of the requirements.

Operational Decision Making

==

Conclusions:==

The shift outage manager and SROs incorrectly used procedure OP 4F as the basis for determination that nozzle dam installation could proceed without installation of a hot leg vent.

At that point in time, there was no active procedural requirement at Point Beach for having a hot leg vent when nozzle dams are installed. This requirement was previously contained in maintenance procedures, which had been cancelled.

The shift outage manager did not engage the resources available during the decision making process.

The shift outage manager and operations personnel were narrowly focused on review of Operations Procedures, as opposed to finding and understanding the basis for the outage activity logic ties. Based on the actions taken, Generic Letter 88-17 was not reviewed during the course of researching the option of not requiring a hot leg vent path.

21

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation The SOM enlisted the assistance of other operations personnel to review the situation and determine if nozzle dam installation could proceed. After reviewing OP 4F and other document(s) the conclusion was reached that it would be permissible not to have a hot leg vent, as long as the cold leg openings were closed.

The Shift Outage Manager confirmed that a hot leg vent path through the pressurizer manway was not required when asked by the site director to verify that venting requirements were met.

Based on interviews, the NSA representative was requested to provide input to the question of whether the safety review would allow installation of the hot leg nozzle dam without the prior removal of the pressurizer manway. This review indicated no change to safety (remains orange) based on NP 10.3.6. The individual pointed out that there is no requirement in the core cooling section of the checklist for a hot leg vent path if nozzle dams are installed.

The Incidence Response Team determined that there were other personnel on shift that were aware of the requirement for a hot leg vent path, but these individuals were not consulted.

Interviews during the RCE investigation indicated that maintenance planning personnel, and production planning personnel were on shift, were knowledgeable of the requirement, but were not consulted.

The Incident Response Team concluded that the lack of recognition of the need or benefit in obtaining additional verification or validation contributed to not establishing an adequate RCE vent path.

The Incident Response Team indicated that it appeared that the SROs asked to determine if installation of the nozzle dam could proceed without the hot leg vent reviewed OP 4F, to see if this was allowed. The section of the RCE report, Procedure Adequacy - Operations Procedures, shows that OP 4F did not intend, and never had, contained requirements for a hot leg vent path for installation of nozzle dams.

Maintenance Procedure RMP 142, which controlled the hot leg vent during nozzle dam installation, had been previously cancelled.

OP 4F, revision 19, in the Purpose section, step 1.5 indicated: "Implement commitments made per Generic Letter 88-17.

Outage Preparation

==

Conclusions:==

Hit teams, an Outage Planning Integration Team, and Nuclear Safety Assessment reviews were used, among other means, to ensure an adequate, safe schedule was developed for the outage.

Nuclear Safety Assessment reviewed the outage schedule per NP 10.3.6, Outage Safety Review and Safety Assessment. The results of this review were incorporated into the "U1R28 Outrage Risk Plan. This is a Production Planning document that was stored on the Production Planning Shared Drive. The U1R28 Outage risk plan was presented to PORC on March 22, 2004 and again on March 22, 2004, when it was approved. This plan identified the requirement for a hot leg vent in the report section titled Reduced Inventory and Mid-Loop, under Work that may start while the RCS is at Reduced Inventory 22

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation

"RCXOO - RCS Boundary - During this time the pressurizer manway will be removed to establish a large hot leg vent. This is required to be complete prior to installing any hot leg nozzle dam."

A contingency plan for the identified high risk portion of the outage was prepared and approved.

This document was titled "U1R28 Reduced-Inventory Orange Path Contingency Plan" and was revised to incorporate comments regarding additions to the protected equipment list. Copies of this document were required to be in the possession of the Shift Manager, Outage Control Center, and Control Room. This contingency plan stated that a loss of DHR could lead to boiling in the core, and core uncovery if DHR is not restored or alternate cooling methods are not initiated. The contingency plan did not identify the requirement for a hot leg vent path prior to installation of the last hot leg nozzle dam.

HIT teams were established for the prior U2 outage (U2R26) and the Unit one outage. The HIT team for U1R28 Steam Generators (includes manways, nozzle dams, eddy current, sludge lancing, and Orange Path) was led by the Engineering Programs manager, and had members from Program Engineering, Operations, Production Planning, Maintenance, Maintenance Planning, and RP. Some optional invitees, who attended meetings occasionally, or on an as needed basis, were QA, Licensing, Procurement, Chemistry, and Safety.

A benchmarking effort was made prior to the U2R26. This effort involved a visit to Comanche Peak. The result of the benchmarking effort was a project plan incorporating, in part, lessons learned from the benchmarking effort. Noteworthy in the project plan were some items that were not done, and some items that had a positive impact. One item was assigned to the Operations Manager to update operations procedures to accommodate use of nozzle dams. No changes were made to Operations procedures to complete this task. Another noteworthy item that was not completed was for the Maintenance Manager to update Maintenance procedures as required to support use of nozzle dams. Another item that was completed and had a positive was assigned to the Training Manager to provide training.

The Incidence Response Team found through interviews, that a number of personnel felt there was insufficient preparation for the outage, citing inadequate time for some personnel to participate on vertical slice reviews.

Generic Letter 88-17 Response, Implementation, and Tracking

==

Conclusions:==

Training on the Diablo Canyon event (expeditious action item 1) was a one time teach, and has not carried over in subsequent training to the depth intended in the Point Beach Response.

Westinghouse Midloop Calculations were not properly addressed in maintenance procedure RMP 142, and allowed an inadequate sized vent path (PORV) to be used to satisfy the hot leg vent requirement.

Maintenance procedures RMP 141 and 142 were not identified as fulfilling Point Beach's response to GL 88-17, and were subsequently cancelled, contrary to Point Beach's response to the GL.

23

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation The vendor procedure for nozzle dam lacks positive control to establish a hot leg vent and does not does not meet the requirement of the GL for positive control.

The actions contained in the WE response to the NRC regarding GL 88-17 were not appropriately translated into plant procedures in that the Generic Letter was not referenced in the procedures. Thus, the maintenance-related procedures were canceled.

Generic Letter (GL) 88-17 was issued on October 17, 1988. The GL requested licensees to provide two responses, one response describing actions taken to implement the 8 expeditious actions identified in the attachment, and another describing plans and schedules for implementation of programmed enhancements identified in the attachment.

Point Beach responded to the eight expeditious actions identified in the attachment to generic letter 88-17 on December 30, 1998. This is documented in Point Beach letter VPNPD-88-635.

Two of the expeditious actions are crucial to this event:

Expeditious Action Item 1 - Diablo Canyon Event NRC Recommendation: "Discuss the Diablo Canyon event, related events, lessons learned, and implications with appropriate plant personnel. Provide training shortly before entering a reduced inventory condition."

Point Beach responded indicating a Training Needs Analysis (TNA) #88-0493 was issued to address concerns brought about by the GL 87-12. This TNA identified a need for operator training on the generic implication of mid-loop operation, possible problems with loss of core cooling, RHR pump vortexing, and Point Beach Specifics for this operating mode. Lesson Plan

1663 was developed.

Training related to the Diablo Canyon event was provided to licensed operators and Duty Technical Advisor during Cycle 88-6 of continuing training. The response indicated that this training was completed on October 13, 1988, before entering reduced inventory operation, and before GL 88-17 was received on November 3, 1988. The response also stated that a separate TNA covers the addition if industry events, in general, to initial operator lesson plans. Future procedure changes, standing orders, etc., that pertain to mid-loop operations will be placed into the TNA system for inclusion in the training programs.

The RCE section, Training Adequacy, concludes that based on the training documents reviewed, the training provided on the Diablo Canyon Event as specified in GL 88-17 was a one-time-teach, and was not carried over into continuing training, or Initial Licensed Operator Training.

Subsequent training was narrow in scope and did not address the knowledge fundamentals to be learned from the Diablo Canyon event.

Expeditious Action Item 7 - Nozzle Dams NRC Recommendation: "Applicable to Westinghouse and Combustion Engineering nuclear steam supply system (NSSS) designs). Implement procedures and administrative controls that reasonably assure that all hot legs are not blocked simultaneously by nozzle dams unless a vent path is provided that is large enough to prevent pressurization of the upper plenum of the RV."

24

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Point Beach responded stating that Plant Procedure SMP-907 addressed the installation and removal of the steam generator nozzle dams. This procedure required that the cold leg nozzle dams be installed prior to the hot leg nozzle dams, and not removed until after the hot leg nozzle dams are removed. This procedure was implemented during the refueling outage which started on October 8, 1988 and similar procedures for nozzle dam installation will be used during all future refueling outages.

The actual response provided no commitment to maintain a large vent path for the RV plenum.

Once the last nozzle dam (hot leg) is installed there is no committed action to provide or maintain a vent path. SMP #90a7, Minor, Installation/Removal of Steam Generator Nozzle Dams Unit 2, in addition to the nozzle dam sequencing described in the response to the GL, contained a prerequisite intended to prevent pressurization of the upper plenum of the RV:

Prerequisite 2.1.8 1: The reactor vessel head is detensioned or removed.

2: Both PORVs must be blocked open with their associated block valves open.

Point Beach provided additional information related to expeditious action item 7 in letter VPNPD-89-061 dated February 2, 1989. In this letter Point Beach identified that steam generator nozzle dams are not expected to be installed during the Unit 1 outage.

RCE report section Procedure adequacy - Maintenance Procedures describes the use of new maintenance procedures, RMP 142, and RMP 141, to install and remove nozzle dams in unit 2, starting in the fall, 1989.

By letter dated July 12, 1989, the NRC identified that the NRC had reviewed the December 30, 1998 response to GL 88-17 expeditious actions. The NRC found that the response appears to meet the intent of the generic letter with respect to expeditious actions, however the response was brief and sufficiently vague that the NRC could not fully understand actions taken in response to GL 88-17.

The letter identified questions about training for personnel other that licensed personnel and Duty Technical Advisors (i.e. the response does not specifically state that maintenance personnel would be trained). The NRC indicated that the intent was that all personnel who can effect reduced inventory operation be included.

The letter also identified that Point Beach did not state the use of any specific vent openings on the hot side of the RCE to relieve RCS pressurization, and that calculations need to be performed to verify the effectiveness of RCS openings.

No response to the July 12, 1989 letter was required.

Point Beach responded to the July 12, 1989 letter on October 16, 1989 (VNPD-89-542) stating that a training needs analysis has been performed and it was determined that such training should be provided to maintenance superintendents and planners.

The Point Beach response also indicated that Point Beach did not provide specific vent openings on the hot side of the RCS, as calculations needed to be performed to verify the effectiveness of the RCS openings. Westinghouse provided calculations, which verified the effectiveness of RCS openings, including the removal of the pressurizer manway. This calculation was reviewed and approved by the Point Beach staff.

25

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Contrary to Westinghouse Transmittal of Midloop Calculations dated June 30, 1989 (WEP 143), the PORV vent path was retained in RMP 142 as an acceptable vent path. The Westinghouse letter stated, in part, after discussion that 2 safeties could not be used until around 225 hours0.0026 days 0.0625 hours 3.720238e-4 weeks 8.56125e-5 months after shutdown; "Note, the PORVs and other smaller vent paths cannot be used to keep RCS pressure less that 2 psig until much later." See RCE report section Response to Westinghouse Midloop Calculations.

The minutes indicated that MSS discussed that all hot legs cannot be blocked unless a vent path is provided from the reactor vessel which is large enough to prevent RCS pressurization. There was no action assigned to ensure this requirement was implemented NEPB 88-366 issued in September 1988 reviewed IN 88-36 and made three recommendations, noting that the owners group and the NRC were continuing to study the issue, and would likely provide recommendations in the future. The three recommendations were to procedurally control the sequence of nozzle dam installation and removal, inform the operating staff of the potential need for injection to the core, and maintain two thermocouples connected as long as possible. Establishment of a hot leg vent path is not addressed in the memo.

In October 1989, MSS approved RMP-141 and RMP-142, routine maintenance procedures for nozzle dam removal and installation. These procedures had DSS signoffs for a reactor vessel vent, head detensioned or removed, both PORVs blocked open, or pressurizer manway removed.

The procedure did not incorporate the information from the Westinghouse Calculation RMP 141 and 142, removal and installation of SG nozzle dams, was cancelled in November 1993. This left no maintenance procedures to control the installation and removal of SG nozzle dams COMTRAC tracking numbers were assigned to expeditious actions on the records copy of Point Beach's response. COMTRAC was the acronym that described an electronic recordkeeping system used by Regulatory Affairs until creation of NUTRK in about 1990. The items contained in NUTRK were not appropriately downloaded into the web-based Regulatory Information System prior to abandonment of NUTRK in late 2003. CAP 54188 was initiated for IT to resolve this issue. Licensing has been unable to locate these specific actions because of the noted issue.

Response to Westin2house Midloop Calculations

==

Conclusions:==

Operations Procedure OP 4F was appropriately updated to specify acceptable hot leg vent paths based on time after shutdown.

Maintenance Procedure for installation of nozzle dams was not updated based on Westinghouse midloop calculations, and continued to allow use of PORVs as a hot leg vent path.

MSS review of mid-loop calculations failed to identify a need to update maintenance procedures.

The routing process used at the time lacked rigor to ensure than need for change was evaluated and needed changes tracked to completion.

26

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Westinghouse performed analysis of the effectiveness of RCS vent paths under letter WEP 143 dated June 30, 1989. These calculations provided best estimate calculations of the time to reach saturation, boil-off rate, minimum vent size to prevent pressurization and estimated time to core uncovery as a function of time after shutdown following a loss of RHR cooling while operating at mid-loop. Simply stated, these calculations resulted in identifying hot leg vent size and time since shutdown to keep pressurization of the reactor vessel less than 2 psi.

The results of the calculations indicate that, assuming nozzle dams are in place:

A removed pressurizer manway will provide an adequate vent approximately 100 hours0.00116 days 0.0278 hours 1.653439e-4 weeks 3.805e-5 months after shutdown.

Removal of 2 safety valves will provide an adequate vent approximately 225 hours0.0026 days 0.0625 hours 3.720238e-4 weeks 8.56125e-5 months after shutdown.

The calculations indicated PORVs and smaller vent paths could not be used until much later than pressurizer safeties, which required 225 hours0.0026 days 0.0625 hours 3.720238e-4 weeks 8.56125e-5 months after shutdown. The maintenance procedures were never updated to remove the PORVs as an allowed vent path On October 2, 1989, DSS granted permission to start work (install nozzle dams) per RMP 142, utilizing the PORVs as the reactor vessel vent path.

In October 1989, MSS approved RMP-141 and RMP-142, routine maintenance procedures for nozzle dam removal and installation. These procedures had DSS signoffs for a reactor vessel vent, head detensioned or removed, both PORVs blocked open, or pressurizer manway removed.

The procedure did not incorporate the information from the Westinghouse Calculation The MSS review attached to the copy of Westinghouse's Transmittal of Midloop Calculations failed to identify that a procedure change to the maintenance procedure for installation of nozzle dams was required.

MSS discussed the Westinghouse Midloop Calculations in relation to their review of RMP #30, Opening of Pressurizer Manway 1 (2) T-001 Revision 2 (MSSM 89-19). This review was focused on analysis of the temporary cover for the pressurizer manway, and failed to identify a need to update the nozzle dam installation procedure.

OP 4F was revised to indicate hot leg vent paths based on time after shutdown based on Westinghouse Midloop Calculations in Revision 2 (October 3, 1990)

IPTE Briefs

==

Conclusions:==

IPTE briefs were not attended by senior managers as required by procedure An IPTE brief was conducted for nozzle dam installation about 0100 hours0.00116 days 0.0278 hours 1.653439e-4 weeks 3.805e-5 months on 4/09/04. Nuclear Oversight observed this brief, and indicated in their rapid trending report that the brief was unacceptable. Additionally, senior management did not attend the brief as required by the IPTE procedure.

NP 1.2.6, Infrequently Performed Tests of Evolutions (IPTEs), Revision 8, Step 3.5, Plant Manager, Step 3.5.2 states: "Determine level of importance of IPTE and either perform pre-evolution briefing and pre-shift briefings or designate the appropriate level of management to perform the briefings.

27

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Section 4.5, Briefings, states: "The Sr. Line Manager or designee shall brief personnel assigned to the activity on the expectations of management. The briefings shall be conducted prior to ---

Copies of other IPTE briefs were obtained from Operations clerical staff. Copies reviewed were:

U1R28 OP-3B (Reactor Shutdown) 4/02/2004; Drain Ul RCS 4/07/2004; and, OP-SA (Fill RCS) 4/09/2004. In all of these examples, the senior person identified was the supervisor performing the brief.

The Incident Response Team indicated that the plant manager did not realize that IPTE briefs were being delegated to lower levels in the organization.

RMP 9030, Unit 1 and 2 Pressurizer Manway Cover Removal and Installation

==

Conclusion:==

Addition of instructions to remove, dress, and reinstall a specified number of bolts prior to cover removal would preclude last minute delays in the removal of the manway cover.

The pressurizer manway removal procedure did not remove, clean, and reinstall bolts prior to manway removal to ensure manway removal would not be delayed. Maintenance reported to the OCC at 0400 hours0.00463 days 0.111 hours 6.613757e-4 weeks 1.522e-4 months that they were having difficulty removing the last bolt for the pressurizer manway.

Station LIog Keeping

==

Conclusion:==

Station log keeping requires improvement.

There were several instances of weak or inadequate log keeping. For example, the Station Log had no entries related to nozzle dam installation between 0452 when nozzle dam work was beginning on the SG cold legs, and 1540 when reactor vessel water level was raised after installation, verification, and testing of the SG nozzle dams.

Response to SOER 88-3

==

Conclusions:==

The response to Item 2. b. related to control of a hot leg vent was not correct. OP-4F did not control the installation or removal of a hot leg vent. The hot leg vent was controlled in Maintenance procedures, which were ultimately cancelled.

INPO issued SOER 88-3, Losses of Residual Heat Removal with Reduced Reactor Vessel Water Level at PWRs, on October 19, 1988. This SOER was issued because of continuing industry problems related to reduced inventory operations after issuance of SOER 85-4, Loss or Degradation of Residual Heat Removal Capability in PWRs in August 1985.

Section 2, Operations, step 2.b.contained the following specific direction related to use of nozzle dams:

28

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation

2. "Review the procedures supporting residual heat removal system operation to ensure that procedure improvements necessary to so support plant actions in response to SOER 85-4 have been incorporated. Ensure that the following specific items are included:

b. Methods to establish and maintain a hot-leg vent path to prevent pressure buildup in the reactor vessel" Point Beach responded with evaluation of the SOER under letter NEMN 440. The response to item 2.b. is as follows:

"OP-4F provides methods to establish and maintain hot leg vent paths which will prevent pressure build-up in the reactor vessel" During the INPO evaluation of PBNP in 1990, Recommendations 2,3, and 4 of the SOER were reopened because the initial evaluation did not address actions to be taken if core temperature instrumentation is unavailable and RHR flow is lost. Point Beach responded in NPM 91-0056.

The response to Recommendation 2.b. stated, "OP 4F Stem 6.0 specified acceptable hot -leg vent paths."

The information provided in the RCE section Procedure Adequacy - Operations Procedures indicate that this is an inappropriate response. Operations procedures did not control establishment of a hot leg vent, and only listed the permissible vent paths to be used as a decision point to determine if MSS approval was required to continue when a cold leg opening greater than one square inch was present. The actual control of the hot leg vent was contained in maintenance procedures, which were ultimately cancelled. See the RCE section Procedure Adequacy - Maintenance Procedures.

Data Analysis Summary The following issues are described in the analysis section of the report, and are summarized below:

1. Operations Procedures: Procedure OP 4F was not intended to control or verify hot leg vent path during the installation of steam generator nozzle dams. For this reason, the procedure was not effective in requiring establishment of a hot leg vent prior to installation of the hot leg nozzle dam. The procedure performed its intended function, to control containment closure.

2. Maintenance procedure RMP 142 for installation of nozzle dams in Unit 2 provided explicit DSS signoffs, specifying which hot leg vent path was being used. This was a strong administrative control, but incorrectly allowed the use of PORVs as a vent path.

The procedure was inappropriately cancelled, not realizing that it constituted the response to GL 88-17

3. The training provided under LP 1663 in anticipation of GL 88-17 was adequate based on review of the lesson plan. Based on other material reviewed, the training was a one-time-teach, and was not carried over into continuing training, or Initial Licensed Operator Training. Much of the training is narrowly focused, such as training on nozzle dam control panel operations, loss of indicted level, loss of core cooling, and indications of pump cavitation and does not address or readdress knowledge fundamentals.

29

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation

4. Operational decision making was weak. The shift outage manager and SROs incorrectly used procedure OP 4F as the basis for determination that nozzle dam installation could proceed without installation of a hot leg vent. The shift outage manager did not engage the resources available during the decision making process. The shift outage manager and operations personnel were narrowly focused on review of Operations Procedures, as opposed to finding and understanding the basis for the outage activity logic ties.

Improvements in operational decision making have been previously identified and are incorporated into the excellence plan.

5. Outage preparation was adequate. Hit teams, an Outage Planning Integration Team, and Nuclear Safety Assessment reviews were used, among other means, to ensure an adequate, safe schedule was developed for the outage.

6. Implementation of GL 88-17 in plant procedures was ineffective to ensure that responses to the GL remained active. Training on the Diablo Canyon event (expeditious action item 1) was a one time teach, and has not carried over in subsequent training to the depth intended in the Point Beach Response. Westinghouse Midloop Calculations were not properly addressed in maintenance procedure RMP 142, and allowed an inadequate sized vent path (PORV) to be used to satisfy the hot leg vent requirement. Maintenance procedures RMP 141 and 142 were not identified as fulfilling Point Beach's response to GL 88-17, and were subsequently cancelled, contrary to Point Beach's response to the GL. The vendor procedure for nozzle dam lacks positive control to establish a hot leg vent and does not meet the requirement of the GL for positive control.

7. Response to Westinghouse Midloop Calculations was inadequate. Maintenance Procedure for installation of nozzle dams was not updated based on Westinghouse midloop calculations, and continued to allow use of PORVs as a hot leg vent path.

However, Operations Procedure OP 4F was appropriately updated to specify acceptable hot leg vent paths based on time after shutdown.

8. IPTE briefs were not attended by senior managers as required by procedure

9. RMP 9030, Unit 1 and 2 Pressurizer Manway Cover Removal and Installation does not contain instructions to remove, dress, and reinstall a specified number of bolts prior to cover removal to preclude last minute delays in the removal of the manway cover.

10. Station log keeping requires improvement.

11. The response to SOER 88-3Item 2. b. related to control of a hot leg vent was not correct.

OP-4F did not control the installation or removal of a hot leg vent. The hot leg vent was controlled in Maintenance procedures, which were ultimately cancelled.

Failure Mode Identification IRR5 IActions not tied to another process when necessary.

30

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation This failure mode existed because of the weak tracking of committed actions from GL 88-17 that resulted in cancellation of maintenance procedures, and failure to incorporate lessons learned into continuing training.

KS Inadequate knowledge of fundamentals l Operations personnel did not possess the knowledge fundamentals from the lessons learned from the Diablo Canyon event. This resulted in inadequate operational decision making.

AR3 INo Process Monitoring l Tracking of committed actions failed to ensure that the responses to GL 88-17 remained in effect. A long history of tracking failures ultimately resulted in a QA significant issue. Process monitoring failed to effect change until prompted by QA.

RR6 IMethods not clear. l Tracking of incoming correspondence from outside sources was informal in the late 1980's. Recent initiatives have improved the VTIP process. This is a historical failure mode.

RR8 Wrong Information l The response to SOER 88-3 is incorrect. The response incorrectly indicated that operations procedure OP 4F controlled the establishment of a hot leg vent path for installation of nozzle dams.

VIII. Root Causes & Contributing Factors The issues identified during the course of the investigation were tested for root cause. Root Cause is defined in the NMC Root Cause Evaluation Manual as "Identified cause(s) that, if corrected, will prevent recurrence of a condition adverse to quality."

The following is a brief review of the issues, and their identification as a barrier, contributing cause, or root cause.

Operations Procedure Adequacy, OP 4F: Procedure OP 4F was correct as written, and was not causal. Improper use of the procedure was a result of two factors, knowledge of 31

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation the fundamental lessons learned from the Diablo Canyon event, and a narrow focus during the decision making process.

Lack of a maintenance procedure for installation of nozzle dams, cancellation of RMP 142: This constituted a barrier that was not in place, which should have been. This was causal, but root cause lies deeper.

Operations Training: Lack of knowledge of the fundamental lessons learned from the Diablo Canyon event was causal, significantly contributing to the narrowly focused decision making.

Operational Decision Making: This was a failed barrier that resulted from operating in knowledge-based performance and not taking the required prevention measures - - make the brain bigger by networking.

GL 88-17 response, tracking, and implementation: This is the root cause, and was manifested by cancellation of the maintenance procedure, inadequate identification of specific steps of the generic letter addressed by the operations procedure, lack of reinforcement of knowledge fundamentals by a narrow implementation of the training for the Diablo Canyon event.

Failure to incorporate Westinghouse Midloop Calculations: This was not causal. The results were incorporated appropriately into Operations procedures, but were not incorporated into maintenance procedures. Since the maintenance procedures were cancelled, this was not causal.

SOER 88-3 response: The incorrect response to the SOER is encompassed by the root cause.

IPTE briefs: This was not causal, and at most weakened a barrier.

Remaining issues; RMP 9030, Log keeping, outage preparations, were not causal.

Root Cause: The actions contained in the Point Beach response to the NRC regarding GL 88-17 were not appropriately translated into plant procedures. Specific items were not identified in operations procedures, and maintenance procedures did not identify the GL as a source document. The maintenance procedures were subsequently cancelled. Training was implemented as a 'one-time-teach.

Significant Contributing Cause: Training on the lessons learned from the Diablo Canyon event was a one-time teach, not carried forward into continuing training. As such, knowledge of fundamentals was weak throughout much of the organization. This had a significant impact on operational decision making.

Contributing Cause #1: Lack of a maintenance procedure to control prerequisites for installation of the hot leg nozzle dam was a contributing cause.

Contributing Cause #2: Operational decision making was weak and narrowly focused.

Operating procedures was the focus of research, although installation of nozzle dams is not an 32

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation operations activity. Source documents, although referenced in the operating procedure, were not reviewed as part of the decision making process.

33

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation IX. Corrective Actions Corrective Actions to Restore (broke - fix)

CA #1. Revise NP 10.3.6, Outage Safety Review and Safety Assessment, to incorporate the requirement for a hot leg vent when RCS water level is at midloop, in the Core Cooling assessment section. (Engineering Safety and Design Review Group. Priority 4. Due 11/18/2004)

CA #2. Review the following maintenance procedures and correct references to a hot leg vent if appropriate. Reference the GL 88-17 requirement at the step, section, or procedure level as appropriate to ensure the commitment is maintained. This item should be completed prior to the next use of nozzle dams. This item addresses extent of condition. (Business Procedures.

Priority 4. Due Date October 21, 2004)

a. 1RMP 9007 Unit 1 Steam Generator Primary Manway Cover Removal and Installation

b. 2RMP 9007 Unit 2 Steam Generator Primary Manway Cover Removal and Installation

c. RMP 9030, Unit 1 and 2 Pressurizer Manway Cover Removal and Installation

d. RMP 9002-7, Reactor Coolant Pump Strongback Installation and Removal

e. RMP 9002-4, Reactor Coolant Pump Uncoupling and Coupling

f. RMP 9004, Power Operated Relief Valve Maintenance

g. IRMP 9096, Reactor Vessel Head Removal and Installation

h. 2RMP 9096, Reactor Vessel Head Removal and Installation

i. RMP 9054-1, Pressurizer Safety Valve Removal and Installation CA #3. Review operations procedure RP IB, Recovery from Refueling, and correct references to a hot leg vent if appropriate. Reference the GL 88-17 requirement at the step, section, or procedure level as appropriate to ensure the commitment is maintained. This item should be completed prior to the next use of nozzle dams. This item addresses one procedure in the extent of condition. (Operations. Priority 4. Due date October 21, 2004)

CA #4. Develop a maintenance procedure to install and remove nozzle dams. Incorporate the requirements of GL 88-17 in the procedure. Reference the GL 88-17 requirement at the step, section, or procedure level as appropriate to ensure the commitment is maintained. This item should be completed prior the next use of nozzle dams. This item addresses contributing cause

1 (CA 57750, CATPR #2 from RCE253, Develop Procedure for Nozzle Dam Installation &

Removal, Priority 2, Assigned to Engineering Programs Testing, due 11/15/04 )

CA # 5. Develop operations procedures to establish positive control over establishment of a hot leg vent prior to installation of the last hot leg nozzle dam and control reduced inventory operations. Reference the GL 88-17 requirement at the step, section, or procedure level as appropriate to ensure the commitment is maintained. This action will, in part, address the 34

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation significant contributing cause by providing a forum for operations continuing training.

(Operations. Priority 3. Due date January 6, 2005)

CA #6. Take the following to the training procedures review group and evaluate the addition of the statement to FP-T-SAT-30, 'Development Phase': "When providing training, ensure the basis (knowledge fundamentals) for the training is appropriately developed, contained in the lesson plan, and presented." (Training. Priority 4. Due Octoberr 21, 2004)

CA #7. Evaluate, using the Systematic Approach to Training, the addition of the lessons learned from the Diablo Canyon event referenced in GL 88-17 as a reoccurring topic for Licensed Operator Requalification. The content of the material should reference the GL 88-17 requirement and the basis of the expeditious actions specified in the GL, such as behavior of the RCS during drained conditions, blowdown of the reactor vessel after loss of SDC with nozzle dams installed, the need for capability of hot leg injection, and the basis for differences in containment closure requirements. This addresses the significant contributing cause. (Training. Priority 3. Due October 21, 2004)

CA #8. Update procedure OP 4D Part 1, Draining the Reactor Coolant System, identifying the steps that address specific items in GL 88-17. Reference the GL 88-17 requirement at the step, section, or procedure level as appropriate to ensure the commitment is maintained. (Operations.

Priority 4. Due October 21, 2004)

CA # 9. Review the response to SOER 88-3, item 2b, and correct if necessary. (Operations.

Priority 4. Due October 21, 2004)

CA #10. Review RMP 9030, Unit 1 and 2 Pressurizer Manway Cover Removal and Installation.

The procedure does not contain instructions to remove, dress, and reinstall a specified number of bolts prior to cover removal to preclude last minute delays in the removal of the manway cover.

Change the procedure if deemed appropriate. (Business Procedures. Priority 4. Due October 21,2004)

CA #11. Incorporate on-going efforts to improve operational decision making into the expectations for Shift Managers. This is being tracked by Excellence Plan item OR-08-017.1, entitled "Implement an Operational/Decision Making Procedure" This is a priority 3 item assigned to operations and due 8/13/04. (CA 56045)

Immediate Corrective Actions The Incident Response Team provided senior management with recommendations for continuation of the outage. Recommendations included review of schedule logic ties, control of schedule changes, safety assessments of changes, communications improvements, improving schedule use, IPTE brief improvements, and specific recommendations related to midloop operations and nozzle dam removal.

These recommendations were implemented through a comprehensive series of actions including an OCC brief, NMC Today articles, implementation of an interactive turnover process, a human performance standdown, management observation improvements, and an enhanced brief and preparation process that required verification that outage activity predecessors were complete.

35

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Operations Procedure OP 4F was revised to incorporate requirements for nozzle dam installation in Unit 1.

Corrective Actions to Prevent Recurrence (CATPRs)

CATPR #1. Develop and implement a process to sample Licensing Basis Commitments, and non-licensing basis commitments to provide assurance that commitments are being implemented and maintained appropriately. NMC Policy OP 0001 may be used as guidance for this effort.

This addresses, in part, the root cause and extent of condition. (Responsible organization -

Regulatory Affairs. Priority 2. Due Date September 23, 2004)

CATPR #2. Provide for a periodic on-going sampling process described in CATPR #1 by incorporating a requirement for a periodic sample in a licensing organization procedure. This addresses, in part, the root cause and extent of condition by providing for a periodic sample to ensure continued performance. (Responsible organization - Regulatory Affairs. Priority 2. Due Date September 23, 2004)

Effectiveness Reviews Present the results of the initial sample review of commitment implementation to the CARB.

This presentation should include the sample plan, including sample size and its basis, threshold for sample expansion, sample distribution between licensing basis commitments and other commitments, and sample results. (Responsible Organization - Regulatory Affairs. Priority 4.

Due Date November 17, 2004) 36

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation X. References The following documents were reviewed during the course of the investigation.

1. VPNP-88-635 / NRC 88-131 Response to GL 88-17

2. NUREG 1449 Shutdown And Low - Power Operation At Commercial Nuclear Power Plants in the United States

3. OP 4F Reactor Coolant System Reduced Inventory Requirements Rev 19

4. Outage Roles & Responsibilities - word document on Production Planning Shared Drive

5. Vent Path Timeline from Incident Response Team

6. Shift Outage Director Turnover Log U1R28

7. OCC Logs

8. Exposure Logs (SG Jumpers)

9. OCC Org Chart

10. U1R28 Outage Risk Plan

11. U1R28 Hit Teams

12. IRMP 9391, Connection of Unit I Nozzle Dam Control Console Remote Alarms to 1C20

13. Scientech Procedure 83A7564, Revision 5, Steam Generator Nozzle Dam Installation And Removal, Test, Operation and Maintenance Manual

14. CAP 031950 Nozzle Dam Benchmarking

15. NP 1.2.6, Infrequently Performed Tests or Evolutions (IPTE)

16. Control Room Logs

17. PORC 2004-015, PORC 2004-016 DRAFT (includes review of U1R28 Outage Risk Plan)

18. U1R28 Reduced-Inventory Orange Path Contingency Plan Rev 1

19. RMP 9030, Unit 1 and 2 Pressurizer Manway Cover Removal And Installation

20. SCR2003-0329 50.59 SG Nozzle Dams

21. Jim Connolly material on Mid Loop (GL 88-17 and responses)

22. CAPs:

CAP056719 - OP 4F Feedback from final training sessions, CAP056567 - PBF-2133 Shift Logs rev. 4 do not match new OP 4F Unit I rev. 1 requirements, CAP056609 - Loss of RHR procedure(s) do not recognize potentially unrecoverable condition, CAP056605 - Questions on OP 4F Unit 1 Rev. I from training session, CAP056574 - Procedure concerns with newly revised OP-4F "RCS Reduced Inventory Req'ments...",

37

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation CAP055505 - Potential error trap associated with reduced inventory procedure OP4F, CAP055510 - Guidance Needed For Use of Pre-job Brief Checklists, CAP055538 - Potential for No Hot Leg Vent Path during Unit I SG Nozzle Dam Installation.,

CAP055547 - Procedure controls for nozzle dam installation are weak, CAP055548 - Removal of SG primary manway bolting may require further evaluation, CAP055576 - SG manway removal/installation RMP could generate CL opening without HL vent, CAP055585 - Improvement to Hot Leg Vent Path Controls Required, CAP055587 - S/G Nozzle Dam Installation Dose Exceeded Estimate, CAP055834 - Insufficient training on nuclear safety issues, CAP056551 - Inconsistencies in Nozzle Dam protected equipment expectations

23. Training Materials (specific references are in the Training Section of the RCE report)

24. NOS Rapid Trending 4/8/2004 Nights

25. SOER 88-3, Losses of Residual Heat Removal with Reduced Reactor Vessel Water Level at PWRs.

26. Outage Schedule (specific prints provided by Production Planning)

27. NP 10.3.6 Outage Safety Review and Safety Assessment

28. NP 2.1.8 Protected Equipment

29. IPTE Nozzle Dam Installation Briefing Documentation of Information Sharing Sheets

30. FME Material Control Log

31. PRA Risk - e mail

32. Jumper Statements IA mL Dam Installation Problems

33. U1R28 OCC Brief (post event)

34. SEM 7.11.9 Rev 0, June 2,1994,Installation of SG Nozzle Dams Unit 2

35. NMC Net - "Time out" article

36. U1R28 OCC Interactive Turnover (Post event)

37. OP 4D Part 1, Draining the Reactor Coolant System, TCN 2004-0170

38. JIT OE, SG Nozzle Dam and Cover Installation and Removal

39. IRMP 9007. Unit 1 Steam Generator Primary Manwvay Cover Removal and Installation

40. SMP 907, Minor, Installation/Removal of Steam Generator Nozzle Dams Unit 2 TCN2004-0109

41. Incident Response Team Report

42. Work Orders 9945861, 0407707, 0311945

43. NMC Today article - Point Beach Event-Free Clock Reset, Friday, March 16, 2004

44. Recommendations from Incident Response Team 38

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation

45. OP 4D Part 1, Draining the Reactor Coolant System (Performed 4/07 -) TCN 2004-237

46. IN 88-36 Possible Loss of RCS Inventory during Low Coolant Level Operation

47. MSSM 88-16, -17, -22, -23

48. NEPB 88-366 NRC Information Notice No. 88-36 ---- from A. La Plante to R.J. Bruno and R.A. Newton

49. MSSM 89-07 (OP 4F Approval)

50. Procedures addressing venting of the RCS (RP 1B; RMP 9002-7; RMP 90024; IRMP 9007; RMP 9004; IRMP 9096; RMP 9054-1; RMP 9030

51. GL 87-12 Loss of Residual Heat Removal (RHR) while the reactor coolant system (RCS) is partially filled. And VPNPD-87-396 / NRC 87-95 Response to GL 87-12.

52. NPM 2003-0530, U2R26 Outage Safety Review Results

53. SOER 85-4, Loss or Degradation of Residual Heat Removal Capability in PWRs

54. NPM 91-0056, SOER No. 88-3 Losses of Residual Heat Removal with Reduced Reactor Vessel Water Level at PWRs

55. RMP 142, unit 2, Minor, Installation of Steam Generator Nozzle Dams (completed procedures)

56. Document Feedback to OP 4F (Ops 2004-0389)

57. RMP 141/142, Removal / Installation of Steam Generator Nozzle Dams (Staff Document Reviews)

58. WEP 89-143, Westinghouse Midloop Calculations

59. KNPP Mid loop Operating and Maintenance procedures: GMP-126, Installation and Removal of Steam Generator Nozzle Dams, Rev M; N-RHR-34C-CL, Requirements for entering reduced inventory checklist, Rev H; N-RHR-34C, RHR Operation at a reduced inventory condition, Temp Change; GMP-125, Nozzle Dam Support, Rev H.

60. Nozzle Dam hit team meeting minutes January 28, 2003

61. KNPP Response to GL 88-17. KNPP NRC 89-1

62. RMP Canceled Index, Rev 75

63. MSSM 89-19, RMP 141,142 approval.

64. SER 86-047-04 U2 Steam Generator Nozzle Dam Installation and Removal

65. LP 1663

66. NP 5.1.7 Revision 7, Regulatory Commitment Management

67. OP 0001 Revision 0, NRC Commitment Management (NMC Policy) 39

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation XI. List of Attachments

1. Engineering Evaluation of Hot Leg Vent Path

2. Graphic Sequence of Nozzle Dam Installation on SG Hot Legs

3. RCE Charter

4. Event Time Line

5. Nozzle Dam Photograph

6. Event and Causal Factor Chart 40

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation Attachment 1 Engineering Evaluation of Hot Leg Vent Path 41

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation PURPOSE Assess the risk posed by a partially installed nozzle dam.

It has been reported that during the incident of nozzle dam installation without an analyzed hot leg vent path on 4/9/2004, a very brief period existed in which all four nozzle dams were installed. At all other times, at least one segment of one of the hot leg nozzle dams was not installed. In such a configuration, only the rubber diaphragm occludes the hot leg where the strong back segment is not installed. The question posed is whether the missing segment and diaphragm provided a sufficient vent path to prevent pressurization and/or core blow-down.

TIME AFTER SHUTDOWN The refueling outage started on 4/3/04, with the reactor shutdown at approximately 0200 per the outage narrative logs. The nozzle dam event with simultaneous installation occurred at approximately 0647 on 4/9/2004. This was -196 hours post shutdown.

MINIMUM REQUIRED VENT PATH WEP-89-143 ("Transmittal of Midloop Calculations" dated 6/30/1989) provides several key pieces of information, including the maximum permissible resistance to flow as a function of time after shutdown. Resistance is a function of both characteristic flow resistance ("K") and flow area ("A"). Figure 7 of that document shows that at 196 hours0.00227 days 0.0544 hours 3.240741e-4 weeks 7.4578e-5 months post-shutdown the maximum permissible resistance is -39 ft4. For comparison, figure 8 of the document illustrates that this is slightly less than the resistance of two pressurizer safety valves removed (45 ft 4 per the text).

For practical purposes, the "K" value of the missing segment will be dominated by only the flow resistance of the missing segment and the installed rubber diaphragm. The pipe large diameter hot leg piping (29" ID per FSAR table 4.1-6) presents a negligible resistance to the steam flow when compared to the relatively small area of the missing nozzle dam restriction.

DIMENSIONS & PROPERTIES OF DIAPHRAGM By inspection, the nozzle dam is divided into three approximately equal width segments to facilitate passage through the primary manways. With a 29" ID, this limits the width of each piece to slightly less than 9.67" to allow for the thickness of the sealing diaphragm.

Conservatively assuming a smaller width of 9", the area of a 29" diameter circular segment subtended by a chord located 9" from the circumference at the midpoint is calculated:

The chord and radii to the two ends of the chord form an equilateral triangle with equal sides of length R (29"/2 = 14.5"). The height of this triangle is R - 9" = 5.5". The base angles (a) are therefore sin-'(h/R) = sin-'(5.5/14.5) = 22.3 deg or 0.389 radians.

42

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation The base of the triangle is therefore 2Rcos(oc) = 26.8". This establishes the area of the triangle as 1/2 bh = 0.5 (26.8)(5.5)=73.7 in 2 The apex angle (P3) of the triangle is twice the complement of ot, or 2.36 radians (135.4 deg).

The area of the circular segment subtended by the apex angle D is R2 P/2 = 248.1 in2 .

Subtracting the area of the triangle gives a net area for the segment of the removed edge piece of 248.1 in 2 - 73.7 in2 = 174.4 in2 (1.21 ft2 ).

While a rigorous evaluation of the condition would require establishing the centroid of this area, the following results will show such an effort is unnecessary due to the very large margin for accommodating flow.

FORCE TO DEFLECT DIAPHRAGM A training video on the installation of nozzle dams was viewed, and this showed that with the side segments removed, the diaphragm was easily rolled around the section and secured in place by a few Velcro© straps.

On 5/12/04, the training mock-up of the steam generator channel heads and the training nozzle dams were inspected. The installed nozzle dam had one side segment removed, consistent with the most limiting condition being considered. Reaching up into the nozzle from the reactor side, a balled fist was placed at the approximate centroid of the un-backed diaphragm, and enough force exerted to fully deflect the diaphragm 90 degrees from the "closed" position.

This was easily achieved with 20 lbs of force. In no case did the force applied exceed 40 lbf, although this will be used as a conservative assumption. 40 lbf distributed over the 174 in area is 0.23 psi.

MAXIMUM ALLOWABLE PRESSURE Per the previously cited 1989 correspondence on midloop calculations, it is necessary to limit the pressurization of the RCS to less than 2 psig to prevent rapid core uncovery. This is 8.7 times as high as the force conservatively estimated above to fully deflect the un-supported diaphragm.

Clearly, the diaphragm does not present a significant impediment to passing steam at low pressure. The only question remaining is whether the resulting opening would be sufficiently large to pass the needed volume of steam at that time after shutdown.

43 .f

RCE 254 Potential Loss of Hot Leg Vent Path during Nozzle Dam Installation RESISTANCE TO STEAM PASSAGE Estimating the flow resistance of the "open" diaphragm is complicated, but can be bounded by observation of some standard valve configurations. The deflected diaphragm results in a flow passage roughly equivalent to half of an open butterfly valve, or half of an open tilting disk check valve. Halving a symmetric flow area results in doubling the resistance. The Crane #410 manual cites a value for "K" for each of these configurations:

For an open butterfly valve with a diameter of 16"-24" diameter, the value for "K" is tabulated as 25f,.

For a tilting disk check valve with a diameter of 16"-48", the "K" value is tabulated as 20ft (5 degrees from neutral) to 60ft (15 degrees from neutral).

Taking the most conservative of these values of 60ft and doubling it results in a K of 120ft. In the size range of interest (18"-24"), ft is listed as 0.012. This gives an overall K of 120 x 0.012 =

1.440.

The flow resistance in equivalent units used in the 1989 correspondence is K/A2 = 1.44/

(1.21ft 2 )2 = 0.98 ft 4 .

As previously established, the maximum resistance to flow that could be tolerated at that time was -39 ft4 . Clearly, the missing segment would have provided ample flow area to prevent significantly pressurizing the reactor outlet plenum.

CONCLUSIONS With just a single segment of a single hot leg nozzle dam removed, the diaphragm does not present a significant impediment to steam flow. Additionally, the resulting open area would be quite adequate to preclude any pressurization of the reactor outlet plenum.

This conclusion was drawn from imprecise observations and assumptions. To account for this, some very conservative assumptions were used. Yet the results show such a large margin of safety that more precise calculations and/or testing are judged to be unnecessary.

44

RCE 254 I

Attachment 2 Graphic Sequence of Nozzle Dam Installation on SG Hot Legs 45

RCE 254 I

Nozzle Dam Installation 4/09/2004 from 0552 to 0656 Time 0552 0553 0554 0555 Cont 0633 0643 0644 0645 0646 0647 0648 0649 0650 0651 0652 0653 0654 0655 0656 A Side X X out X X X out A__

Cntcr X X X X X X X X X X X X X X X X X X X X out A Side X X out X X X X X X X X out Tap x out B Side X X X X X X X X X X B x x x x x x x x x x x x x x Center = ===== X X X X X X X X XXXXX B Side X X X X X X X X X X FME Log Data For SG A Hot Legs Nozzle Dam Pieces SG A In: Center 0552 Out: 0656 Side 0555 0633 Side 0555 0633 Side 0644 0647 Side 0647 0655 Tap 0647 0648 46

RCE 254 I

RCE Charter 48

RCE 254 Root Cause Evaluation Charter CAP# 55538, Potential Loss of Vent Path During Nozzle Dam Installation, RCE# 254 Issue Manager/Sponsor:

Pat Russell - Manager Performance Assessment Problem Statement:

A significant near miss occurred during the steam generator nozzle dam installation sequence.

Specifically, workers were allowed to proceed with the installation of hot leg nozzle dams prior to completion of removal of the pressurizer manway. This was contrary to the established outage schedule. (Removal of the pressurizer manway is required to provide a Hot Leg vent path during reduced inventory operations.) The issue was identified in Point Beach CAP055538.

Investigation Scope:

An Incident Response Team (IRT), composed of NMC staff, will perform a prompt investigation of issues associated with the event. They will provide a summary report to site management and the RCE team. Additionally, they will provide recommended corrective actions to site management for safe continuation of the outage.

The RCE investigation scope will include Organizational and/or Programmatic issues as well as human performance issues leading up to this event.

The RCE team will make recommendations for:

Correcting the problem

Preventing recurrence of the problem

Applicability of the root cause to other areas (extent of condition)

Investigation Methodology:

The RCE team will perform document reviews and consolidate identified examples into failure modes/problem statements. Interviews may be used to gain additional understanding or validate identified problems. Investigations methodologies will include Event and Causal Factor Charting and Failure Mode Analysis.

The following sources of information have been identified for the document review phase:

Interview notes

Procedures and work instructions

NUREG 1449/GL 88-17

INPO SOER 88-03

Outage schedule 49

RCE 254 I

Training lesson plans Incident Response Team Members:

Team Lead Paul Harden Palisades Site Director Team Member Dave Geisen KNPP NOS Manager Team Member Tom Taylor PINGP Assessment Manager Team Member Aldo Capristo NMC ECP Fleet Manager Team Member Joe Maurer PINGP Outage Manager Team Member Kari Den Herder PINGP Engineer (Nozzle Dam SME)

Team Member Craig Chovan NMC Fleet Outage Director RCE Team Members:

This will be a team RCE utilizing Point Beach Personnel and NMC personnel.

Team Leader Dennis Hettick Point Beach Assessment Team Member Jim Holmes (Part Time) Point Beach Training Team Member Chuck Smoker (Part Time) Hudson Assessment Milestones:

Date Assigned Date 4/12/2004 IRT Update Sr. Management Date 4/13/2004 (complete)

Status Update CARB or Screen Team Date 4/22/2004 Draft Report Date 5/06/2004 Final Report Date 5/13/2004 Communications Plan:

Initial communication to the station - Broadcast voicemail and stand-downs (complete)

IRT debrief to NRC - 4/13/2004 (complete)

Communication to INPO - OE submittal Follow up to the station - Leadership Meeting (Date TBD)

Approved: Date:

Management Sponsor Reviewed by: Screen Team / CARB on (circle one) Date 50

RCE 254 I

Attachment 4 Event Timeline 51

RCE 254 I

Event Timeline Date Description Source Time 7/09/87 NRC issues Generic Letter 87-12, Loss of Residual Heat GL 87-17 Removal (RHR) while the Reactor Coolant System (RCS) is Partially Filled 9/18/87 Point Beach responds to GL 87-12 VPNPD-a87-396 6/08/88 Information Notice IN 88-36 issued to alert the industry of the IN 88-36 potential sudden loss of RCE inventory when operating with nozzle dams in the SG hot legs. (Diablo Canyon Event of spring 1988) 7/19/88 MSS briefly discusses IN 88-36 after serial review of the IN, MSSM 88-16 recognizing that a Generic Letter may be issue, and there may be an impact to outage U2R14 8/02/88 MSS holds extensive discussions regarding IN 88-36, MSSM 88-17 Westinghouse owners group concerns, and NRC concerns.

Several MSS actions are identified and assigned. No action is assigned to address hot and cold leg closure, and hot leg vent.

9/08/88 Internal memo from J. LaPlante to J. Zach discusses IN 88-36 NEPB-88-366 and recommends procedurally controlling the sequence of nozzle dam installation and removal, and informing the operating staff of the potential need for injection to the core.

Establishment of a hot leg vent is not addressed 10/7/88 MSS reviews SER 86-045-02. Addendum 2 was prepared to MSSM 88-22 reflect changes in mid-loop operation and documented 6 guidelines to be used, including nozzle dam installation and removal sequence, and providing a hot leg vent path through the PORV and blocs valves if the head is tensioned 10/11/88 SMP #907, Installation/Removal of Steam Generator Nozzle SMP #907 Dams Unit 2 is issued 10/17/88 Generic Letter GL 88-17 issued due to concerns about continued GL 88-17 loss of decay heat removal in the industry.

11/01/88 MSS serially reviewed IEN (sic)88-036 and stated the MSSM 88-23 following actions were taken to address the recommendations:

1. Ensure procedures specify installation and removal sequence of nozzle dams. This is accomplished by SMP #907.

2. Prepare special operations or standing order to address 52

RCE 254 I

l cooling concerns. This is accomplished by a night order book entry.

Maintain two core thermocouples connected as long as possible.

This is accomplished as line items in the outage work list.

12/30/88 PBNP responds to GL 88-17, expeditious item 7 - Nozzle Dams VPNPD-88-635 stating SMP-907 addressed installation of nozzle dams during (NRC 88-131) the October 1988 refueling outage, and that similar procedures for nozzle dam installation will be used in the future.

3/23/89 MSS serially reviews OP 4F Revision 0 dated March 27, 1989 MSSM 89-07 (New Procedure Attachment C 6/30/89 Westinghouse Transmitted Midloop Calculations to Point Beach WEP-89-143 WE NEVER CHANGED OUR PROCEDURES!!! Roger Newton!

10/03/89 MSS serially reviews and approves RMP 141 (minor), Removal MSSM 89-19 of Steam Generator Nozzle Dams, and RMP 142(minor),

Installation of Steam Generator Nozzle Dams. Both were new procedures dated October 2, 1989. The procedures had embedded errors (PORV allowance) that were not detected.

The Same MSSM discussed the pressurizer manway in regards to 88-17 and WEP 89-143, Mid loop calculations.

10/02/89 DSS grants permission to start work on U2 Nozzle Dams per Completed RMP RMP 142. PORVs are hot leg vent path. 142 10/05/89 DSS grants permission 0555 to begin removal of U2 Nozzle Completed RMP Damp per RMP 141. Work complete 1420 same day. 141 10/11/90 DSS grants permission to start work on U2 Nozzle Dams per Completed RMP RMP 142. Pressurizer manway is the hot leg vent path. 142 10/27/90 DSS grants permission 1850 to start removal of U2 Nozzle Completed RMP Dams per RMP 141. Work complete 0200 10/28/90. 141 1/28/91 PBNP response to recommendations 2,3,and 4 which were NPM 91-0056 reopened, iterates that OP 4F Step 6.0 specifies acceptable hot-leg vent paths.

10/03/91 Temp change (one time use) to RMP step 2.1.7 b. added: "or Completed RMP the pressurizer safety valves removed and the upstream piping 142.

vented." To the step indicating PORVs blocked open for a vent path. Step 2.1.7 b. was the designated hot leg vent.

10/03/91 DSS grants permission to start work on U2 Nozzle Dams per Completed RMP RMP 142. PORVs/Pressurizer Safety valves are the vent path. 142 10/31/91 DSS grants permission 0700 to start removal of U2 SG nozzle Completed RMP dams perRMP 141. Work complete 11/02/91 at 1621 141 53

RCE 254 I

10/05/92 DSS grants permission to start work on U2 Nozzle Dams per Completed RMP RMP 142. Pressurizer manway is the hot leg vent path. 142 10/26/92 DSS grants permission 0601 to start removal of U2 Nozzle Completed RMP Dams per RMP 141. Work complete 10/28/92 at 2037 141 11/18/93 RMP 141 andRMP 142 are cancelled. RMP Cancelled Index 3/24/03 U2 R26 Outage Safety review identifies requirements for hot leg NPM 2003-0530 vent path and need for JIT for operations as nozzle dams have not been used for several years. This review also noted an incorrect sequence for installation on nozzle dams, and notes that the schedule should be corrected, and logic ties incorporated to ensure establishment of a hot leg vent path to minimize the risk of a rapid loss of inventory 3/22/04 PORC reviews the Unit 1 outage risk plan, noting the plan is PORC 2004-016 very similar to the plan for the unit 2 outage.

4/07/04 Commenced drain down to 70% reactor vessel level IAW OP Station Log 0254 4D Part 1 4/07/04 Secured drain down of RCS. Reactor vessel level 73% Station Log 0629 4/07/04 Commenced drain down of the A SG per OP 4D Part 1 Station Log 0951 4/07/04 Completed drain down of the A SG per OP 4D Part 1 Station Log 1308 4/07/04 Commenced drain down of the B SG per OP 4D Part 1 Station Log 1314 4/07/04 Completed drain down of the B SG per OP 4D Part 1 Station Log 1632 4/07/04 Completed walkdown of all protected equipment, and verified Station Log 2123 all signs specified by U1R28 Draindown Contingency Plan are in place.

4/07/04 Commenced drain down of RCS to a target of 40%, IAW OP Station Log 2307 4D Part 1, section 5.28. OP 4F Reduced Inventory in effect 4/08/04 Holding level at 40% in the RCS per OP 4D Part 1 Station Log 0015 4/08/04 Commenced draining the RCS to a target level of 22-25% per Station Log 0123 OP-4D Part 1.

4/08/04 Secured drain down of RCS per OP 4D Part 1. Maintaining Station Log 0338 level between 22% and 25%

54

RCE 254 I

4/08/04 Notified by maintenance that the A SG Primary Manways are Station Log 1601 removed 4/08/04 Informed by WCC that B SG manways have been removed Station Log 2254 4/09/04 Unit I in Mode 5 RCS temp 110 degrees F RCS is in reduced Station Log 0000 inventory Reactor Vessel Level 24%.

4/09/04 IPTE brief started 0100 4/09/04 IPTE brief completed (approx.)

0200 4/09/04 Enter Mode 6 - Started head detensioning Station Log 0301 OCC Log 4/09/04 Maintenance reports that they are having problems removing the OCC Log 0400 last bolt for the pressurizer manway. They are planning to work through turnover to get it removed.

4/09/04 Commence installation of Unit 1 B SIG Cold Leg Nozzle Dam Station Log 0430 4/09/04 SIG Jumpers arrive at B S/G platform (approx.)

0430 4/09/04 Commenced installation of B SIG Cold Leg nozzle dam OCC Log 0434 4/09/04 B S/G Cold Leg Nozzle Dam center in bowl FME Log 0434 4/09/04 B S/G Cold Leg Nozzle Dam sides in bowl FME Log 0440 4/09/04 Commenced A S/G Cold Leg Nozzle Dam installation Station Log 0452 4/09/04 RPM reports to OCC that 1 S/G jumper lost breathing air and OCC Log 0452 had collapsed hood 4/09/04 Received word that the S/G jumper is OK and he believes he 0459 pinched his hose, Also noted that air pressure may have been a little low - adjusted up. Work has resumed.

4/09/04 A SIG Cold Leg Nozzle Dam center in bowl (FME Material FME Log 0515 4/09/04 B S/G Cold Leg Nozzle Dam installed, moving to A S/G cold OCC Log 0521 leg 55

RCE 254 I

4/09/04 RP reports that both cold leg nozzle dams are installed and they OCC Log 0525 are moving to hot legs 4/09/04 A S/G Cold Leg Nozzle Dam sides in bowl FME Log 0526 4/09/04 Approval to proceed with installation A SIG Hot Leg Nozzle Scientech 0536 Dam signed procedure 83A7564 4/09/04 Installation of A&B S/G Cold Leg Nozzle Dams is complete Station Log 0537 and verified installed. Commence installation of A S/G Hot Leg Nozzle Dam 4/09/04 Installation of Unit I A and B S/G Cold Leg Nozzle Dams are OCC Log 0544 complete and verified installed. Commence installation of Unit 1 A S/G Hot Leg Nozzle Dam 4/09/04 A S/G Hot Leg Nozzle Dam center in bowl FME Log 0552 4/09/04 A S/G Hot Leg Nozzle Dam sides in bowl FME Log 0555 4/09/04 Start of turnover meeting (start of dayshift) 0600 4/09/04 Site Director question's OCC on importance of vent path Interview 0620 sequence, Ops claims it is ok per procedure. (approx. time) 4/09/04 A S/G Hot Leg Nozzle Dam sides out of bowl for re-alignment FME Log 0633 of center 4/09/04 Outage Director brings copy of NUREG to OCC stating vent Interview 0635 path requirement (approx.)

4/09/04 B S/G Hot Leg Nozzle Dam center in bowl FMIE Log 0643 4/09/04 One A S/G Hot Leg Nozzle Dam side out of bowl FME Log 0647 4/09/04 B S/G Hot Leg Nozzle Dam sides in bowl FME Log 0647 THESE TWO 0647 TIME ENTRIES IN THE FME LOG ARE CRITICAL FROM THE STANDPOINT THAT THEY INDICATE, ALTHOUGH PURE LY BY COINCIDENCE AND NOT INTENTION, THAT THE "A" S/G HOT LEG NOZZLE DAM WAS BEING DISASSEMBLED (ONE SIDE OUT) JUST PRIOR OR NEARLY SIMULTANEOUSLY AS THE "B" S/G HOT 56

RCE 254 I

LEG NOZZLE DAM FINAL SIDES WERE GETTING INSTALLED. THESE TIMES ARE THE TIMES THAT THE NOZZLE DAM PIECES WERE PASSED INTO THE BOWL, AND ALLOWING FOR THE FEW SECONDS IT TAKES FOR THE TECH IN THE BOWL TO POSITION THE PIECES IN PLACE, WE CAN SUMMIZE THAT THERE WAS, ALTHOUGH BY ACCIDENT, A HOT LEG VENT AT ALL TIME.

4/09/04 Concern raised about the validity of installing SG hot leg nozzle OCC Log 0650 dams without an established hot leg vent path. Cold leg nozzle dams are installed and tested. OCC Investigating 4/09/04 OCC contacted containment 8 foot and learned that some pieces OCC Log 0652 of the hot leg nozzle dams are located inside the hot leg opening but the hot leg nozzle dams are not completely installed. SOM directed that all pieces of A hot leg nozzle dam be removed from A SG hot leg opening. Installation of the B hot leg nozzle dam can continue 4/09/04 One A S/G Hot Leg Nozzle Dam side out of bowl FME Log 0655 4/09/04 A S/G Hot Leg Nozzle Dam center out of bowl for vent path FME Log 0656 4/09/04 Phone call to SOM informing OCC that A hot leg is clear of all OCC Log 0715 nozzle dam pieces, there is a problem with one set of threads on the A SG that may need to be retapped, B hot leg nozzle dam installation is in progress. OCC also informed that RP crew will be reaching work hour duration limits and will need to leave site following completion of B hot leg nozzle dam. Day shift RP crew will need to be briefed to allow completion of the A hot leg nozzle dam 4/09/04 OCC informed that B hot legnozzle dam installation is OCC Log 0734 complete. IPTE brief for day shift RP techs supporting nozzle dam job will be led by Gary Sherwood. Tentatively set for 0830 4/09/04 IPTE brief for nozzle dam installation confirmed for 0830 in the OCC Log 0807 RP briefing room. (OCC LOG) 4/09/04 Clay Hill assigned as management representative to discuss the OCC Log 0821 orange path contingency plan for IPTE brief 4/09/04 Update from Gary Sherwood is that we will need to chase OCC Log 0822 threads for one penetration in the A SG, procedure change may be required to allow for this, and estimated installation is now 57

RCE 254 1000 4/09/04 OCC informed that pressurizer manway has been removed with OCC Log 0824 the exception of the diaphragm 4/09/04 OCC informed that pressurizer manway and diaphragm are OCC Log 0847 removed 4/09/04 Nozzle dam installation put on hold. OCC informed that we had OCC Log 1024 more than the one personnel safety incident discussed at the 0600 turnover 4/09/04 Following a meeting to discuss personnel safety on the nozzle OCC Log 1059 dam installation activity, the Plant Manager informed the OCC of the protocol to follow prior to recommencing the nozzle dam job. Engineering will conduct an inspection of the hoses and fittings for breathing air, Safety will brief the four personnel safety issues with the affected individuals, Safety and Nuclear Oversight will observe the observation via camera, Safety has authority to stop the job at any point they desire. Once these measures are in place, OCC will be informed. Subsequently, OCC will update the WCC. WCC will control release of the nozzle dam job 4/09/04 OCC informed that nozzle dam predecessors to work have been OCC Log 1152 completed 4/09/04 WCC informed that nozzle dam work predecessors have been OCC Log 1153 completed 4/09/04 OCC informed that A SG hot leg nozzle dam installation is OCC Log 1212 commencing 4/09/04 OCC informed that A hot leg nozzle dam is installed. OCC Log 1255 Verification of proper installation is in progress 4/09/04 WCC informed that A SG hot leg nozzle dam is installed and OCC Log 1301 verification is in progress 4/09/04 OCC informed that all nozzle dams inflated and tested OCC Log 1420 4/09/04 Commenced raising Rx Vessel Level with a target of 70% Station Log 1508 vessel level per OP5A 4/09/04 Unit 1 exits reduced inventory, vessel level greater than 55%. Station Log 1553 Continuing to fill with a target of 70%

4/09/04 Unit 1 exited reduced inventory OCC Log 1554 4/09/04 Rx Vessel Level stabilized at 70% per OP5A Station Log 1611 58

RCE 254 I

59

RCE 254 Attachment 5 Photo of Steam Generator Nozzle Dam 2 Dam Segments Installed in the Training Mockup Flexible Diaphragm being raised from outside the SG Bowl 60

RCE 254 2 Dam Segments Installed in the Training Mockup Flexible Diaphragm being raised from outside the SG Bowl 61

RCE 254 I

Attachment 6 Event and Causal Factor Chart 62

RCE 254 Events Occurrences Inappropr a e Causal Factor Actions Happenings 64

RCE 254 Informal control of VE1IP Falure to translate and information maintalnNRC commitments Vnto procedures and training 10/17/88 12130/88 10/89 GL 88-17 Issued re concems PBNP response to GL 88-17, IVMSS approves RMP 142 for 7/ \ 11/18 about loss of decay heat j item 7, commits to procedural controls for ND J ND installation, and RMP 141 for ND removal, for W midloop calc's not incorporated in RMPI41 and RMP 142 removal events installation/removal; SMP- 10/89 RFO RMP 142 907 for 10/88 RFO

\/\

65

RCE 254 I

0452 4/9/04 Ul A S/G CL ND Installation begins 05374/9/04 Ul AS/GCLND installation complete 66

RCE 254 0630 419/04 Put stop on HL ND Installation due to personnel safety issues with loss of breathing air 67

RCE 254 I

0635 4/9/04 0652 4/9/04 0656 4/9104 1152 4/9/04 1554 4/9/04 Raised RCS level after SOM reviews NUREG 1449 SOM directs all ND All ND pieces are ND work stoppage lifted U1 A S/G HL ND installation which states vent path pieces be removed from removed from Ul A SIG completed.

requirements Ul A S/G HL HL Exited reduced RCS inventory.

68

ML051050045

Text

Purpose:

Navigation menu

Search