Text

NMC Incident Response Team for Issues Encountered During Unit 1 Reduced Inventory and Nozzle Dam Installation at Pbnp - Final Report
	ML050840509
Person / Time
Site:	Point Beach
Issue date:	04/30/2004
From:	Harden P; Nuclear Management Co
To:	; Office of Nuclear Reactor Regulation
References
	FOIA/PA-2004-0282
	Download: ML050840509 (18)
	v • d • e

NMC Incident Response Team For Issues Encountered During Unit 1 Reduced Inventory and Nozzle Dam Installation at PBNP April 9- 14, 2004 Final Report Team Members:

Paul A. Harden (Team Lead), Palisades Director, Site Operations Craig Chovar NMC Fleet Outage Director Dave Geisen, Kewaunee Nuclear Oversight Manager Tom Taylor, Prairie Island Performance Assessment Manager Kari Den Herder, Prairie Island Program Engineer Joe Maurer, Prairie Island Outage Manager Al Capristo, NMC Employee Concerns Program Manager Signed by Team Leader, Paul A. Harden 4/30/12004

_"t vv

Executive Summary During the morning of 4/9/2004, two events occurred related to a series of activities in the Point Beach Unit 1 refueling outage. First, due to difficulties in removing the pressurizer manway, an erroneous decision was made to deviate from the pre-established outage schedule sequence logic. This decision was to proceed with installation of steam generator nozzle dams in parallel with completion of pressurizer manway removal. This decision conflicted with requirements and recommendations of Generic Letter 88-17, NUREG- 1449 and SOER 88-03 to have an established adequate hot leg vent path prior to installation ofthe hot leg nozzle dams. The decision was made by the Shift Outage Manager in consultation with several Senior Reactor Operators on shift at the time, without full engagement of other Outage Control Center or available on site knowledgeable resources.

The second event or issue concerned the actual steam generator nozzle dam installation activities. Nozzle dams were being installed on the two steam generators in parallel, and on the first two jumps into each steam generator there was either degradation or loss of breathing air to the contractors entering the steam generator bowls. In two of the cases, the bubble hoods had to be cut off of the individuals who had lost air, with at least one of those cases having the hood collapsed and fogged by the time the hood was cut off.

During this time, only one of the breathing air supply issues was reported to the Outage Control Center, and this report was some time (approximately one hour) after the issue had occurred. In addition, it was discovered that due to the physical size of the contractors making the steam generator nozzle dam jumps, a lubricant and physical force were needed to help get one or more of the individuals through the manways into the confined space. The fact that multiple breathing air supply issues had occurred was brought to the attention of senior site management by an NRC inspector, who had been observing the activities with the site's contractor liaison via remote video monitors.

Through numerous interviews and document reviews, an incident response team compiled a list of issues that had occurred during these events. Those issues were then characterized by failed barrier using the six Barriers for Excellence associated with the site's Picture of Excellence. The results from the barnier analysis identified the primary issues that contributed to these events, as listed below.

Inadequate understanding of adherence to schedule and schedule logic ties by personnel,

Failure to recognize the significance and potential consequences of activities related to reduced inventory and steam generator nozzle dam installation,

Lack of clarity of specific vent paths and requirements in procedure OP-4F and lack of similar specific prerequisites in the nozzle dam installation procedure,

Lack of a clearly understood process for decision making and resource involvement in the OCC for changes to outage schedule logic,

Lack of questioning attitude, advocacy of questions and self-critical nature,

Roles, responsibilities and expectations were not clearly defined, communicated and/or adhered to (including adherence to site procedures),

Failure to establish clear ownership and accountability at all levels from project/job leads to management oversight,

Inadequate threshold for raising issues to management and use of the CAP.

2

==

Introduction:==

On the morning of April 9, 2004, an Incident Response Team was requested from the NMC fleet to provide initial investigation into issues that had occurred that morning at Point Beach Nuclear Plant (PBNP) during the sequence of activities for steam generator nozzle dam installation on Unit 1. Two separate issues had occurred which were requested to be in the charter of the team: a schedule sequence change that permitted steam generator nozzle dam installation to begin without an adequate reactor coolant system (RCS) vent path and multiple steam generator bowl entries to install nozzle dams which experienced diminished or loss of breathing air.

The Incident Response Team began assembly and held an initial team meeting at approximately 2100 hours0.0243 days 0.583 hours 0.00347 weeks 7.9905e-4 months on April 9, 2004 to discuss the team charter and direction.

Following the kickoff meeting, the investigation began that evening and continued through April 13, 2004. The investigation included interviews of personnel involved in the issues as well as review of station logs, plant and vendor procedures, action requests, operating experience evaluations, regulatory guidance documents and associated correspondence, outage risk assessment, PORC review of outage risk assessment and training lesson plans.

A barrier analysis approach, using the six Barriers for Excellence associated with the site's Picture of Excellence and ACEMAN for Individual Excellence, was used to identify all of the individual issues that occurred and associated barriers that failed. From analysis of the failed barriers, the team and site were able to evaluate appropriate actions to implement during the remainder of this refueling outage to prevent similar events as those that initiated this investigation. In addition, analysis of the failed barriers also facilitated determination of the common underlying issues that contributed to the failure of multiple barriers. In an effort to retain the focus of the incident response team, action requests were generated for items tint require further evaluation outside of the charter of the team.

Incident Response Team Charter:

1. Determine the process, procedure and organizational failures that enabled the following twvo events:

a. Nozzle dam air supply issues on multiple steam generator jump attempts

b. Violation of RCS vent path requirements when nozzle dams were installed, considering the following areas: work control, communication, oversight, procedural control, control of contractors, operations interface, scheduling logic and human performance.

2. Recommend the immediate actions required to implement barriers that would prevent such events, such that activities can recommence to return risk status to green for core heat removal and inventory (i.e. cavity flood up), as well as restart work on non-risk significant projects of turbine work and steam generator work.

3. Recommend additional actions necessary to restart all outage work.

3

Background & Description of Events:

There were two separate issues that occurred in the early morning of 4/9/2004 during the evolutions to prepare for and install steam generator nozzle dams on Unit I at PBNP.

The first issue concerned potential violation of RCS vent path requirements with steam generator nozzle dams installed. The second issue concerned multiple difficulties encountered with breathing air supplies while attempting to install steam generator nozzle dams.

RCS Vent Path Issue The first issue was that a decision was made by the Shift Outage Manager to deviate from the pre-established outage schedule logic to allow steam generator hot and cold leg nozzle dam installation in parallel with removal of the pressurizer manway. At approximately 0400 hours0.00463 days 0.111 hours 6.613757e-4 weeks 1.522e-4 months on 4/9/2004, the PBNP Shift Outage Manager was notified of difficulties in removing one of the bolts on the pressurizer manway, which was a critical path activity that would delay exit from a reduced inventory condition (orange path for core heat removal and RCS inventory). The next scheduled activity was to install steam generator nozzle dams, and the crews were standing by ready to install them. The outage schedule had the pressurizer manway removal logic tied as a predecessor to steam generator nozzle dam-installation. The Shift Outage Manager consulted with the Outage Control Center (OCC) Operations representative, who in turn consulted with the Work Control Center (WCC) Supervisor and the Shift Manager on shift at the time to determine if steam generator nozzle dam installation could proceed without having the pressurizer manway removed. After reviewing procedure OP-4F, the SRO's and Shift Outage Manager erroneously concluded that there was not anything in the procedure that prohibited steam generator nozzle dam installation in parallel with completion of the pressurizer manway removal as long as the cold leg nozzle dams were installed prior to the hot leg nozzle dams. Authorization was then given to commence installation of cold leg nozzle dams followed by hot leg nozzle dams. The entire resources in and available to the OCC were not consulted prior to making the decision By 0537 hours0.00622 days 0.149 hours 8.878968e-4 weeks 2.043285e-4 months , the cold leg nozzle dams were installed and installation of the hot leg nozzle dams commenced. During the 0600 morning briefing when plant status was given, the Site Director questioned the sequence of activities relative to RCS vent path requirements. The Shift Outage Manager and WCC Supervisor replied that the sequence was in accordance with procedure. Upon hearing this question, the dayshift Outage Director looked at NUREG- 1449 which had been referenced in determining the outage schedule logic for nozzle dam installation and reported to the OCC that both hot leg nozzle dams should not be installed without a RCS vent path available. During this time, difficulties were encountered installing the IA Steam Generator hot leg nozzle dam, but installation of the nozzle dams for both hot legs were proceeding in parallel. At 0652, logs indicate that the Shift Outage Manager gave direction to stop installation of the IA 4

Steam Generator hot leg nozzle dam and remove all sections of that nozzle dam based upon discussion with the Outage Director concerning to need to establish a hot leg vent path prior to installation of both hot leg nozzle dams. At 0847, the OCC was informed that the pressurizer manway and diaphragm were removed. Later that morning, the IA Steam Generator hot leg nozzle dam installation was authorized allowing RCS water level to be increased up to the reactor vessel flange.

S/G Nozzle Dam Installation Issues The second issue that occurred on the morning of 4/9/2004 was that multiple steam generator bowl jumps were conducted to install nozzle dams in which air supplies were either diminished or lost completely. During the nozzle dam installation evolution, four separate incidents of personnel breathing air issues occurred. Of those four events, only one occurrence was reported to the OCC and none were reported to the on duty Shift Manager. An NRC inspector alerted the Site Director that multiple air supply issues had occurred. The inspector had been observing the activities via the remote video monitors along side of the site contractor liaison assigned to oversee the work. The following is a brief summary of those events:

1) An individual experiences low air pressure to bubble hood in the lB Steam Generator.

When authorized to install the cold leg nozzle dams, the Radiation Protection (RP)

Technician sprayed down the contractor who was to enter the lB Steam Generator cold leg with an approved cleaner (409) to lubricate him due to his size relative to the manway opening. While in the steam generator bowl, the worker reports low air pressure to bubble hood. The individual exited the steam generator prior to completing nozzle dam installation due to the low air pressure. The worker was cut out of the bubble hood by the RP Technician upon exit from the steam generator, resulting in personal contamination. Work on both steam generator platforms was stopped while the RP technicians investigated the event, and the RP Manager notified the OCC of air supply issues. The airline in question was inspected and no problems (i.e. kinks, disconnected fitting, leaks, etc.) were noted. In an effort to restore air pressure, RP personnel increased the manifold air pressure above the procedurally permitted range to the IB Steam Generator platform workers to a setting of approximately 64 psig.

The decision to increase the air pressure was made by the RP Technician in consultation with his immediate supervisor per interviews of the two. When the workers had requested more air, the RP Technician asked his supervisor if he could increase air pressure to which the supervisor responded to give them as much as they need. Site procedures HPIP 4.51.3 and HPIP 4.58 limit manifold air pressure to a setting of 20-28 psig when using bubble hood respirators. In interviews, the RP supervisor indicated that he did not know of the procedurally directed pressure range at the time, nor did he ask. The RP Technician stated that he was aware of the setting 5

directed by the procedure, but felt that raising the pressure was acceptable because the workers requested more air and his supervisor authorized him to increase it.

2) Air supply line becomes disconnected while in the lB Steam Generator bowl.

A second individual was tasked to finish lB Steam Generator cold leg nozzle dam installation. This individual was physically larger than the first and required a total of three attempts to successfully enter the steam generator through the manway. On the first attempt, the individual got stuck at the shoulders. After exiting the manway and being sprayed with additional 409 cleaner for lubrication, he got stuck at the waist on his second attempt. After exiting the manway and being sprayed down with additional 409 cleaner, the third attempt was successful with some physical aid from personnel outside of the steam generator.

While inside the lB Steam Generator bowl, the individual realized his air supply line must have disconnected. The individual continued the nozzle dam installation until he estimated he had "three to four good breaths" left. At this time, the individual exited the steam generator, with his bubble hood was deflated, sucked against his face, and fogged. Upon exit, the RP Technician attempted to reconnect the air supply line but was unable due to interference of tape on the fittings. The RP Technician then immediately cut the worker out of the bubble hood. The tape had been applied in a failed attempt to keep the airline fittings connected during this evolution.

3) Air supply line becomes disconnected upon entry into the IA Steam Generator bowl.

While attempting to enter the IA Steam Generator manway, the worker's airline disconnected. RP Technician stationed on the IA Steam Generator platform reported that the fitting hit the manway on entry and disconnected. The worker, who was only halfway into the steam generator, immediately exited the manway and the RP Technician reconnected his air supply line. At this point, the fittings on the individual's air supply line were taped and reconfigured. Fittings were taped in the horizontal direction on the jumper's back to prevent inadvertent disconnections caused by contact with the manway.

4) Air supply line is damaged/cut while entering the A steam generator bowl.

While working on the IA Steam Generator platform, a strongback inadvertently cut an individual's air supply line. The strongback was in use to aid nozzle dam jumpers when entering the steam generator manways. Individuals at the job site reported that the strongback rocked and pinched one worker's air supply line. The air supply line was leaking air and was taped to control potential spread of contamination on the platform. The RP Technician asked the worker if the air pressure/supply was adequate and the individual replied that it was "okay" several times. Upon completion of his jump, the damaged hose was only used on the platform with an extra emergency hose on standby. As additional entries in the steam generator were completed, the hoses were switched between the worker entering the steam generator and those individuals remaining on the platform.

6

Event Significance & Implications:

The actual nuclear safety significance of the event will be explored further in the root cause evaluation for the RCS vent path issues to be performed by the site. From the timeline developed using available logs, it appears that there may have been no time when both steam generator hot leg nozzle dams were completely installed prior to having the pressurizer manway removed. This observatbn is based primarily on the FME Material Control Log that indicates one side of the IA Steam Generator hot leg nozzle dam was not in the bowl when the OCC called to direct removal of that nozzle dam.

However, this will need to be evaluated definitively in the site's root cause evaluation as well as evaluating if having one side piece of the nozzle dam would constitute an adequate hot leg vent path.

The decision-making and circumstances that led to authorization for installation of the steam generator nozzle dams prior to having the pressurizer manway removed is of great significance to the PBNP site and to NMC.

The breathing air supply and actual nozzle dam installation issues are a matter of industrial safety, for which no one was injured during this event (but personnel contamination did occur). The decision-making and failure to raise all issues to senior management, however, do imply that underlying cultural issues may exist that inhibit identification and raising issues. Again, this adds to additional concern with the credibility and regulatory margin of the PBNP site.

Analysis Against The Barriers For Excellence:

Although the issues regarding failure to adhere to the pre-established schedule logic for removal of the pressurizer manway prior to installing steam generator nozzle dams and the issues related to air supply continuity during steam generator nozzle dam jumps are completely separate issues, the incident response team did identify similar failed barriers.

Therefore, relative to the discussion of the six Barriers For Excellence, the broken barriers and issues from the two issues are discussed together in the tables that follow.

From these tables, common underlying themes have then been identified.

7

S

=Wv i s-I 8

Qualified Workers ssu i;saxil '.' l-'T'i:

ar;

.-, -_IndIng F

4t- 'e'4A id -.I :. . t sessmentoinhg Knowledge of hot leg

Interviews of several SRO License holders indicated lack of knowledge concerning

All personnel involved in the decision venting requirements hot leg vent requirements. to proceed with nozzle dam

Interview of the Shift Outage Manager on the 4/9/04 nightshift indicated a lack of installation prior to pressurizer awareness of hot leg vent requirements. manway removal had been previously

Interview of the OCC Engineering representative on 4/9/04 nightshift indicated trained on RCS vent path lack of knowledge concerning hot leg vent requirements. requirements.

Interviews of schedulers indicated general knowledge of the nozzle dam

Pre-outage training for the Spring installation prerequisite for pressurizer manway to be removed, but some 2004 UI Outage did not cover the uncertainty as to the basis. same level of detail regarding venting

Interviews indicated that outage planning personnel and steam generator hit team requirements as the Fall 2003 U2 personnel were aware of RCS vent path requirements and basis. Outage.

ILT and LOR training lesson plans clearly contain discussion on the requirement to

ILT & LOR lesson plans do not cover establish an adequate hot leg vent path, but do not contain detailed basis for that the detailed basis for RCS vent path requirement to explain what could happen without it. requirements, to ensure understanding

LOR training prior to the Fall 2003 U2 Refueling Outage contained details of of the potential consequences of not nozzle dam installation and associated requirements, including vent paths. LOR having an adequate vent path.

training prior to the Spring 2004 Ul Outage covered big picture overview, but did not cover details of requirements related to nozzle dams.

Orange path planning including contingency actions were trained/briefed, but did Although, areas for improvement exist, not cover venting requirements for nozzle dam installation, training on RCS vent paths has been

Fall 2003 U2 Refueling Outage was the first outage in more than a decade in provided and adequate knowledge level of which steam generator nozzle dams were used. the requirements existed on site and in the Schedule knowledge

Most of the SRO License Holders interviewed did not look at nor understand the Not all personnel appear to have importance of logic ties in the outage schedule. Some indicated that they relied appropriate knowledge of the basis and upon their procedures, rather than schedule logic. importance of outage schedule logic ties,

The basis for pressurizer manway removal as a predecessor to steam generator or the evaluation that must be performed manway installation was specified in the schedule, but was not known by the Shift to break the schedule logic. In addition, Outage Manager, Operations Coordinator or the Engineering representative in the some SRO's and OCC personnel OCC on the 4/9/2004 night shift. demonstrated a lack of

Some members of the OCC who were interviewed, indicated that they had not familiarity/understanding of the schedule.

taken part in vertical slice schedule reviews prior to the outage.

9

Job Planning/Preparation ISSUesA - 1 -;. Findings int fiFindin s General pre-outage

Interviews with a number of station personnel indicated that they felt that there

The level of preparation for the Ul preparation were insufficient preparations for this outage. outage did not meet site/fleet

Interviews indicated that the OCC staff was selected/identified shortly before the expectations and milestones.

outage commenced, and that had impacted their time to prepare and participate in

Although the proper schedule reviews vertical slice schedule reviews. were performed, it appears that not all

Interviews indicated that detailed discussions were held concerning the hot leg appropriate personnel were involved vent path during pre-outage vertical slice reviews of the schedule. However, it in those reviews.

appears from interviews that Operations personnel in attendance were limited and

Not all personnel prepared to the level that not all OCC personnel participated to the extent needed for their assigned or performed the preparatory actions OCC positions. expected for the outage positions they

There were two individuals assigned as mid-loop coordinators for the outage. were assigned.

However, interview indicated that their responsibilities were not explained to them and no action was taken associated with the assignment. The level of preparation for the outage

A special tool was procured for pressurizer manway removal but was not used. among personnel in key positions Use of this tool may have avoided the difficulties encountered in bolt removal. contributed to lack of understanding of schedule knowledge and requirements associated with RCS vent paths.

Steam generator nozzle

The steam generator HIT team did not have a senior Operations (SRO)

It does not appear that the operational dam installation representative to focus on nuclear safety and risk aspects of the evolutions significance of the nozzle dam preparations associated with nozzle dam installation. installation evolution was recognized

The Work Control Center SRO was assigned to attend the nozzle dam installation given the low level of Operations infrequently performed evolution (IPTE) briefing only shortly before it was involvement in the preparation conducted, and was not previously aware that he would need to attend the briefing. activities.

This SRO also left the briefing prior to its completion.

NOS missed an opportunity to

The IPTE brief for the nozzle dam installation contained information relative to the identify and communicate the lack of need to have an RCS vent path, but did not stress or focus on that aspect of focus and clarity on a significant evolution preparation. activity related to nuclear safety.

A Nuclear Oversight (NOS)observation of the IPTE brief characterized it as less than adequate. The observation was not shared with the IPTE brief leader or with The significance & potential on site management at the time. The practice for NOS observations was to brief at consequences of the evolution does not the end of shift, which was after nozzle dam installation in this case. (Te appear to have been recognized by many l shortcomings noted did not inclbde vent path discussion.) on site during preparation.

10

I Procedures/Work Instructions Issues ; ;l 2 '~:.~zi.

. -;-;n'. .Findins;.

Reduced Inventory

OP-4F does not address specific hot leg vent path requirements. A change to OP4F

Although, OP-4F had been reviewed Requirements was submitted March 16, 2004 to clarify NUREG 1449 & GL 88-17 requirements, several times (GL 88-17 & SOER 88-Procedure (OP4F) - but was not incorporated into the procedure before the outage. (PCR #2004-00389) 03), inadequacies were not recognized

PBNP's responses to GL 88-17 and SOER 88-03 did not result in ensuring & it was not revised to the specificity appropriate vent paths were established for nozzle dam installation. (CAP 055538) of current industry standards.

10 CFR 50.59 performed for nozzle dam installation for the Fall 2003 U2 Outage

The basis for the 50.59 evaluation for and applicable to the current Ul Outage was based on hot leg venting via pressurizer use of the nozzle dams included manway and was not specifically contained in OP-4F. This was not specifics of the pressurizer manway reviewed/evaluated by OCC personnel before making the decision to leave the being removed, but did not ensure site pressurizer manway installed during nozzle dam installation. (CAP 055547) procedures were similarly specific.

OP-4F has several inconsistencies and conflicting statements in it, as well as being vague/interpretive for significant requirements. However, in the judgment of the Lack of clarity of specific vent paths &

incident response team, interpretation of the current OP-4F flowchart should requirements in OP-4F was a failed conclude that to proceed with nozzle dam installation without the pressurizer barrier that significantly contributed to manway removed should have required MSS review (similar to PORC). this event.

Steam generator

The vendor procedure used to control nozzle dam installation is not specific on The procedure currently used to control nozzle dam installation venting requirements or the sequencing of nozzle dam installation & removal. It nozzle dam installation and removal procedure references similar non-specific requirements as contained in OP4F. constitutes a missed opportunity to

A current site procedure for nozzle dam installation does not exist. Site personnel provide an additional barrier to such an indicated they did not have sufficient time/resources to develop a site-specific event.

procedure. The team did identify an existing Site Engineering Manual Procedure (SEM 7.11) for U2 nozzle dams no longer used that specifically requires pressurizer manway removal prior to nozzle dam installation.

Processes for changing

No clear procedure or process exists for breaking logic ties in the schedule and the Use of many resources available for such outage schedule or understanding among individuals is not consistent. The scheduling group expected changes in schedule logic (such as Outage logic that the responsible scheduler input is obtained before changing logic ties. Schedulers), which is the purpose of the

A shutdown safety analysis was not done when deciding that the pressurizer manway multi-disciplined team in the OCC, would would not be removed before the nozzle dams were installed. It was the belief of the have prevent the decision to proceed with safety analysis group that they be contacted for such changes. nozzle dam installation. Lack of a clear

From interviews, it appears that the Shift Outage Manager only consulted several process understood by all directly Operations SRO's prior to making the decision to proceed with nozzle dams. represents a failed barrier to this event.

Other procedural

S/G manway removal procedure allows removal in any sequence rather than The schedule is used to control S/G opportunities requiring proper hot & cold leg sequence (error-likely situation). (CAP 055576) manway removal sequence rather than via

Work instructions had all but four bolts removed from S/G manways before drain procedure & lessons learned on S/G's down to limit time in reduced inventory. Similar guidance on the pressurizer was not have not been applied to the pressurizer provided, which would have identified the problems in removing the manway earlier. manway.

11

Verification/Validation ISSues : :; Findin s : AI 1e--' - esn F g Decision making

When making the decision to proceed with nozzle dam installation prior to

Those involved did not recognize the pressurizer manway removal, all available resources were not used to help make the potential significance of the decision decision. Although much Operator input was obtained, other personnel in or being made to deviate from pre-available to the OCC such as outage manager, schedulers, and safety analysis established schedule logic ties.

personnel were not consulted.

Additional verification/validation

During the decision to not remove the pressurizer manway, the outage risk plan was from other knowledgeable outage not referenced. This plan clearly states that the pressurizer manway needs to be scheduling & management personnel removed before installing any hot leg nozzles. was not obtained.

The Shift Outage Manager did not consider that the decision being made to proceed with nozzle dam installation without pressurizer manway removed warranted Lack of recognition of the need or benefit notification to the Plant Manager. in obtaining additional verification or validation contributed to not establishing an adequate RCS vent path.

Lack of questioning

Although many personnel understood the need to install cold leg nozzle dams prior The involved staff did not appear to attitude to hot leg nozzle dams, they did not further question this sequence to understand the exhibit an inquisitive nature, natural basis, which may have led to recognition of the need for a hot leg vent path when all questioning attitude or respect for human nozzle dams were installed. fallibility approach common in high

Inadequate or imprecise communications were noted in a number of cases while performing organizations.

investigating and during activities while the team was on site. One example includes the WCC SRO giving authorization to install nozzle dams to an individual who signed off the procedural step that venting prerequisites were met without specifically asking. Several others were observed.

Confined Space Rescue Team (CSRT) members stated that they wanted to be at the job site. This request was denied by the Radiation Protection department due to dose concerns. CSRT did not question further, even though they later indicated difficulties with accepting this decision..

Two nozzle dam jumpers had to be "lubricated" to get through the manway opening.

In addition to being "lubricated" to aid in entry, one of the two required three attempts to successfully enter the steam generator with physical force from others.

Integrated training session involving Scientech, RP and the S/G hit team lead was not completed. The station's contractor liaison stated that it was desired to complete integrated training, however did not actively advocate such.

Radiation Protection personnel did not question appropriateness of air supply regulator adjustments. Personnel interviews indicated that individuals were aware of procedural requirements, but felt it more important to give the air needed to individuals in the bubble hoods.

12

Supervisor Oversight

Issues - - ' : i Findins l-- - sesmt of Findings Roles &

Roles/responsibilities were not clar in several instances:

Although, roles & responsibilities for Responsibilities o Work Control Center SRO stated that they do not typically review outage the OCC were developed and schedule - focuses on work packages and tagouts. This is contrary to Operations communicated, it does not appear that Manager's expectations as indicated in an interview. all OCC personnel clearly understood o Assigned mid-loop coordinators indicated that they were not clear on their roles practical application of those.

and responsibilities and therefore did not perform this function.

It appears that some personnel are o Confusion existed as to who was the lead for nozzle dams. During interviews, assuming, or simply not asking roles various names mentioned as lead included Erdman, Klesper, and Sherwood. & responsibilities when in question o The assigned contractor liaison, the only site person overseeing the Scientech (mid-loop coordinators, WCC SRO).

personnel for nozzle dam installation, only viewed his role as one of a technical

Some roles & responsibilities are oversight, which differed from the role identified in discussions with other clearly defined in site documents but engineering management personnel. not well understood or adhered to o Lack of ownership for and knowledge of outage schedule by SRO's. Several (contractor liaison, IPTE brief lead).

Operations SRO's indicated they do not typically look at the outage schedule;

There was not a coordinated effort to they rely primarily on procedures and believe it is the OCC's responsibility to ensure management oversight for monitor the schedule. critical activities or evolutions.

o The Shift Outage Manager did not engage the entire OCC in making a safety significant decision, which defeated part of the purpose of staffing the OCC with Lack of clearly defined, communicated a multi-disciplined team. and adhered to roles and responsibilities o An individual deep in the organization conducted the IPTE brief for the nozzle significantly contributed to all issues dam evolution. Per station procedure, the plant manager or designee is surrounding nozzle dam installation. In responsible for IPTE briefs. The plant manager did not realize that this some cases, it does not appear that they important briefing had been delegated to this level. In addition, the assigned pre- were well enough defined and/orl outage S/G HIT Team sponsor and leader were not present for the briefing. communicated. While in other cases it

The management sponsor for the SIG HIT Team was assigned a day shift position simply does not appear that they were and did not provide any oversight to the nozzle dam installation activities that night. internalized or adhered to. In addition, management oversight of critical activities does not currently support what is needed to drive the site towards Excellence.

Expectations

Expectations were not clearly set nor enforced in several instances, including: It appears that some did not clearly o The schedule for pressurizer manway sequence and nozzle dam activities was understand expectations. In other cases, not followed and inappropriately changed resulting in the event. there was simple disregard to adhere to o The outage risk plan was violated by the decision to change the schedule them or to ask what the expectations and sequence without looking at the risk plan or discussing with the safety analysis procedure requirements were.

person on shift at the time.

13

o The IPTE brief was delegated to a line level individual without knowledge of the Plant Manager, violating site procedures.

o Mid loop coordinators were assigned pre-outage, but took no action since responsibilities were not explained to them.

o The breathing air supply regulator was adjusted outside of procedural requirements following a discussion between a technician and his supervisor. (CAP055595) o The air supply to the nozzle dam seals and eddy current test equipment did not meet the requirements of NP 8.4.9. Vendor supplied red hoses were used, whereas the site procedure requires a yellow hose or a yellow stripe. Yellow tape was subsequently added for a stripe. (CAP055560) o The investigation revealed that the bubble hoods used for nozzle dam installation were not issued in accordance with HPIP 4.58, in that issue forms PBF-4234 were not completed. (CAP055645) o Log keeping in the various site logs (control room, OCC, FME logs, etc.)

were not adequate to readily construct timelines of important site evolutions such as the actual hot and cold leg nozzle dam installations, breathing air issues to the nozzle dam installation crew, etc.

14

Worker Practices Issues Findings : -7 :Assessm of Findings Ownership &

Lack of ownership, acceptance, and accountability were exhibited in a number of

The issues of ownership and accountability areas including: accountability appear to exist at all o Nozzle dam lead and IPTE brief delegated to a low-level individual. levels, from the individual given job o Mid-loop coordinators did not understand responsibilities or follow up to assignments to the management determine what they should be doing. sponsors who should have been o The OCC team lost positive control of mid-loop operation and did not fully present for oversight of critical understand venting requirements for nozzle dam installation. As a result, they evolutions. It appears that individuals lost the "big picture." (at all levels) have not internalized the o Unclear understanding of nozzle dam installation sequence, basis and vent path ownership & accountability requirements by those who were responsible for conduct of the outage schedule.

The concept of HIT teams is typically o The S/G Hit team leader and management sponsor were both on day shift, to ensure a high level of ownership, leaving the nozzle dam evolution to a less knowledgeable individual on night but do not appear to have been shift. implemented successfully in that o The contractor liaison, providing the only site oversight of the nozzle dam respect given that the S/G HIT team associated activities and contractor for those activities, only viewed his role as leader and management sponsor were one of technical oversight. both assigned day shift positions and o No single person was providing the oversight, stepping back, to see the "big did not cover the critical evolutions.

picture" regarding prerequisites for nozzle dam installation or overall reduced inventory success paths. The same was true for nozzle dam installation activities The concept of "who owns this" from o It appears that the three SRO's and Shift Outage Manager involved in the start to finish on all aspects of a given decision to deviate from schedule logic, looked at site procedures to figure out evolution do not appear to exist on a how to make them work, as opposed to understanding the basis of the schedule consistent basis. This is a barrier that is and procedures. necessary for the organization to achieve their picture of excellence.

Questioning attitude,

Lack of questioning attitude and advocacy were apparent in several instances:

Many examples indicate a lack of advocacy and self o The Operations scheduling SRO remembered seeing a 10CFR50.59 inquisitive nature or natural critical nature screening regarding pressurizer manway removal, but could not locate it questioning that exists in high so he concurred with the decision to proceed with nozzle dam installation, performing nuclear organizations.

without raising the issue that he thought he had once seen. the "watite ttiouddoet ape o A devil's advocate was not included in the decision making process to be consistent throughout the regarding the pressurizer manway, as only SRO's were consulted vs. the entire OCC on the decision to deviate from the schedule logic. The lack of questioning attitude and lack o The SRO's appeared to have looked at OP-4F to see how they could make of advocacy of the "what if" question it work, as opposed to whether the proposed action was consistent with contributed to the group think that the procedure basis. resulted in the decision to proceed with o Procedure change request to OP-4F was treated as an enhancement vs. a nozzle dam installation prior to 15

I significant procedural deficiency. pressurizer manway removal, and to the o The individuals assigned as mid-loop coordinators did not ask for those issues surrounding actual nozzle dam responsibilities when given the assignment without any expectations. installation. This is also true of the o When questioned on the sequence of activities regarding nozzle dam evaluation and actions taken in response to the GL 88-17, SOER 88-03, NUREG-installation and vent paths by the Site Director at the 0600 hour0.00694 days 0.167 hours 9.920635e-4 weeks 2.283e-4 months meeting, 1449 as well as the Operating Experience the Shift Outage Manager and OCC Operations Coordinator responded regarding breathing air supply issues.

that it was per procedure, with others in the outage organization then following up to identify that the sequence of activities was an issue.

o The Confined Space Rescue Team members and the contractor liaison interviewed indicated that they wanted to be near the job site to provide proper response time during steam generator entries, but were stationed outside the containment hatch by Radiation Protection department to reduce dose. This decision was not questioned further or raised to management by the team members, yet some were still expressing disagreement with the decision to the incident response team.*

o When the nozzle dam installers had to be lubricated with 409 cleaner to fit through the steam generator manway, and one required three attempts to get through, other personnel at the job site did not question the appropriateness of having the individual enter that confined space.

o Integrated training and mockup for steam generator nozzle dam installation was not performed. The site's contractor liaison stated that he had desired that to occur but did not advocate it as he felt he would not get budget approval.

o RP Technicians covering the nozzle dam installation did not question why the air pressure regulator had to be adjusted outside of the initial procedural setting.

o The site's evaluation/response to previous operating experience (OE21454) on breathing airlines becoming disconnected was to tape the fittings, whereas many plants changed the fittings.

o The contractor liaison overseeing nozzle dam installation did not question the understanding or resolution of the air supply issues, even after being prompted by an NRC inspector, but simply deferred to the judgment of the contractors.

Threshold for raising

The threshold for raising and documenting issues does not appear to meet industry

The corrective action process and issues standards for excellence, as demonstrated by the following examples: raising issues to management do not o CAP's were not written on some issues until prompted (e.g., OP4F procedure appear to be at the forefront of deficiency). thinking versus just attempting to o The procedure change request to OP-4F for specifically identifying GL 88-17 and solve the issue. No evidence was SOER 88-03 recommendations was thought to be just an enhancement. observed of fear for retaliation, but o NOS observations of the IPTE brief for the nozzle dam installation characterized just a lack of sensitivity to raising the 16

the brief as less than adequate but did not stop the brief or immediately raise the issues vs. solving them.

l i, issue to senior management.

o The 30 minute rule was not followed or reinforced for the pressurizer manway Lack of an adequately low threshold for delays. raising issues contributes to impeding o Senior plant management was not contacted on pressurizer manway issues. senior site management's ability to o Although four separate issues regarding degraded or lost breathing air occurred identify and address overall issues that during nozzle dam installation, only one was reported to the OCC and none were exist within the organization.

reported to the on duty Shift Manager in the Control Room.

o The contractor liaison overseeing the nozzle dam installation did not elevate the issues observed. Rather, the NRC inspector who was watching the video monitors with the contractor liaison alerted site management.

o CAP055527 written to summarize the breathing air supply issues for the nozzle dam installers had to be amended twice to include the details of all four issues and provide clarification, each time after being prompted by the incident response team.

o Many CAP's referenced in this report were only written after being prompted by the incident response team.

o The threshold for log keeping requirements for various site logs appears to be inconsistent with industry norms based upon the difficulty in reconstructing an accurate timeline associated with the actual hot and cold leg nozzle dam installation times.

17

==

Conclusion:==

To summarize the results from the barrier analysis, the primary issues that contributed to these events are listed below.

Inadequate understanding of adherence to schedule and schedule logic ties by personnel,

Failure to recognize the significance and potential consequences of activities related to reduced inventory and steam generator nozzle dam installation,

Lack of clarity of specific vent paths and requirements in procedure OP-4F and lack of similar specific prerequisites in the nozzle dam installation procedure,

Lack of a clearly understood process for decision making and resource involvement in the OCC for changes to outage schedule logic,

Lack of questioning attitude, advocacy of questions and self critical nature,

Roles, responsibilities and expectations were not clearly defined, communicated and/or adhered to (including adherence to site procedures),

Failure to establish clear ownership and accountability at all levels from project/job leads to management oversight,

Inadequate threshold for raising issues to management and use of the CAP.

This summary indicates that many of the principles provided in INPO's Principles for Effective Operational Decision-Making were not present during these events. When comparing these contributing factors to previous incidents and assessments at PBNP, several of them appear to be similar or related to previous underlying causes. Those that appear to be the same or related are listed below.

Lack of questioning attitude, advocacy of questions and self critical nature,

Roles, responsibilities and expectations were not clearly defined, communicated and/or adhered to,

Failure to establish clear ownership and accountability at all levels from project/job leads to management oversight,

Inadequate threshold for raising issues to management and use of the CAP.

The most significant of these may be the ownership and accountability issues.

Ownership and accountability at the individual level should result in the behaviors necessary to establish a healthy questioning attitude and ensuring that issues are raised to management in a timely manner. Ownership and accountability at the manager/supervisor level should ensure oversight is provided for critical activities and evolutions, and should reinforce the expectations for questioning attitude and raisings issues. Ownership at both levels should ensure that roles are clearly defined, or asked for when they are not, as well as strictly enforced or reconciled.

The compilation of issues identified by these events and through this investigation also raises some implications in relation to INPO's Principles for a Strong Nuclear Safety Culture. The NMC Fleet ECP Manager explored these implications in a discussion separate from this incident.

Recommended actions in the form of an action plan to address the failed barriers during the remainder of the refueling outage were developed and provided to site management for review and implementation separate from this report. CAP055863 was written to document and track implementation of that action plan.

18

ML050840509

Text

Navigation menu

Search