ML060590430

From kanterella
Jump to navigation Jump to search
Executive Summary, Titled Reactivity Events During Performance of an Infrequently Performed Test or Evolution (IPTE)
ML060590430
Person / Time
Site: Hope Creek 
Issue date: 07/16/2003
From: Schweitzer G
- No Known Affiliation
To:
Office of Nuclear Reactor Regulation
References
FOIA/PA-2005-0194
Download: ML060590430 (11)
v • d • e


Text

144

E Reactivity Events During Per. ormance of an Infrequently Performed Test or Evolution (II1TE)

Limited Distribution July 16, 2003 Executive Summary C) n March 17, 2003, during an infrequently performed test or evolution (IPTE) for a controlled shutdown at Hope Creek, two separate, unexpected power transients occurred that met the termination and scram criteria in the IPTE implementing procedure. In neither case did the operators scram the reactor as required.

The first transient involved a power change of less than I perce: t s.,

1mo three turbine bypass valves (BPVs) opened and closed quickly. The crew discussed that plant's response and elected to continue the IPTE shutdown. The second transient occurred -4b)ont ' 5 minutes later. The number three BPV suddenly opened, causing reactor pressure t decrcase 50 psig and reactor water level to decrease 8 inches. The cold water addition to res! ore ard maintain reactor water level added positive reactivity, increasing reactor power from 6 5 percent to 13.5 percent for about 25 seconds. The operating crew stabilized the plant and did not scram the reactor. After discussing the plant response with the dui:y assistant operations.manager, the crew continued the shutdown and removed the unit from service without further incidcnt.

The unit was being shut down because the number two BPV wo-ild not close fully after the unit was synchronized and loaded to the grid following an unrelated outage. As part of the repair plan for the partially open BPV, personnel elected to perform a ontrolled shutdown versus scramming the reactor because of a concern that the reactor vessel cooldown rate limit would be exceeded. It was later determined that a loose bolt from the nui. Ser five BPV had become lodged in the seat of the number two BPV, preventing full closL -.

Although the impact to equipment and safety was minimal, the (-vent revealed organizational deficiencies relating to the preparation and implementation of tC - IPTE process, as follows:

  • Operating crew performance did not meet station expect dons. Failed protective barriers included shift management oversight, shift technical au. Nor (STA) independence, conservative reactivity management, and procedure ca -

liance. The crew, including the control room supervisor, became overly attentive to 1i,.

chnical aspects of the shutdown, instead of maintaining oversight of plant cu:

tions. Similarly, the shift manager and STA became overly engaged in crew act

. during the evolution. In particular, the crew did not exhibit a sufficient question g attitude during both the preparation and the performance of the IPTE implement..g procedure.

  • Station personnel did not follow operating and administr..clve procedures when preparing for and implementing the IPTE. The termination and scr ~m criteria contained in the integrated operating procedure (LOP) relied on operator i terpretation and decision-making at the time of event. However, the crew was no, amiliar with the criteria because they were not discussed during pre-evolution ex Rcises in the simulator or during the prejob briefing as required by station procedures.

SER 3-03, Page 1 of I lIpV

Operations personnel involved in the IOP revision and review process used for the IPTE did not recognize the use of a manual mode of pressure control as an undesired method with the reactor critical and at elevated pressure. Operations management had expected that manual control would only be used when the reactor was subcritical, but this requirement was neither properly stated in the newly revised IOP nor reinforced through training. This approach placed two different board operators in the position of controlling reactivity. Because this problem had not been identified, the newly revised IOP was not forwarded to the Station Operations Review Committee (SORC) for review.

  • Operations management had directed that just-in-time simulator training be conducted for the evolution. Contrary to that expectation, the simulator time was actually used to develop, verify, and validate the new IOP. The operating crew did not receive training on the final product; consequently, operating experience, contingencies, termination criteria, and expected crew actions for termination were not discussed. Training personnel who observed the simulator activity did not ensure that appropriate just-in-time training was conducted prior to implementation of the new procedure.
  • The governing IPTE process procedure contained several deficiencies when compared to INPO SOER 91-01, "Conduct of Infrequently Performed Tests or Evolutions." The process procedure did not require the appointment of a line manager, senior to the shift supervisor, to oversee the IPTE, did not require an engineering review of the IPTE procedure, and did not identify training department responsibilities.

The Event approximately 10:00 p.m. on Friday, March 14, 2003, the Hope Creek main generator i

was synchronized to the grid following completion of a planned maintenance outage.

I Upon synchronization, it was identified that the number two BPV failed to close fully as expected.

Personnel determined that the BPV problem was likely mechanical and could not be repaired with the turbine and associated steam piping pressurized. With the plant stabilized at 20 percent reactor power, two shutdown strategies to repair the BPV were reviewed, as follows.

Strategy 1:

Scram the reactor from normal operating pressure, and conduct a plant cooldown to cold shutdown.

Strategy 2:

Perform a controlled shutdown of the reactor and cooldown of the plant to achieve reactor power less than 5 percent, while reducing reactor pressure to less than 650 psig. Reactor pressure would be controlled using the bypass valves and the main condenser.

The second strategy was selected, as it was believed to have the lowest operational risk and was similar to that used during prior plant startups and shutdowns. It was recognized as an infrequent evolution, and new procedure guidance for conducting the evolution was developed, which involved modifying sections of the lOP normally used at low power. The first strategy was not selected because the amount of steam required after the scram would exceed the expected decay SER 3-03, Page 2 of 11

Unit shutdown began during the day shift on March 16, using the normal shutdown procedure.

At about midnight, the night-shift crew commenced the IPTE. At 12:07 a.m. on March 17, with the reactor at approximately 6.5 percent power and reactor pressure at 920 psig, the crew began using the automatic (pressure set) and manual (BPV jack) modes of pressure control alternately to reduce reactor pressure. During preparation of the procedure, operations management had intended that automatic pressure control would be used when the reactor was critical and that manual control would only be used when the reactor was subcritical. However, this was not clearly stated in the procedure. The operating crew interpreted the procedure as allowing use of manual and automatic control individually or together.

At 12:15 a.m. with reactor power at 6.5 percent and reactor pressure at 820 psig, reactor pressure control was being transitioned from manual to automatic. During this changeover, the electrohydraulic control (EHC) system experienced a transient that caused the numbers one, two, and three BPVs to shut quickly (45 percent for number two BPV). then reopen to their original positions. Reactor power increased less than 1 percent, reactor prcssure increased about 7 psig, and reactor water level decreased about 3 inches. The operating crew-stopped the evolution and discussed the response. Per the IOP, this transient met the termination and scram criteria (unexpected plant or system response and addition of unexpected positive reactivity); however, the test manager, test engineer, and operating crew did not terminate the IPTE. This decision was made because the managers and crew were not cognizant of the specific criteria in the IOP.

The crew subsequently decided, incorrectly, that the minor transient was the result of shifting pressure control from manual to automatic and that it was acceptable to continue using manual control to reduce pressure. After the event, it was determined that a rheostat associated with the BPV jack push-button was defective and caused the BPVs to open and close suddenly.

The ]3PV jack controller was increased to take manual control of pressure to continue the reactor cooldown. At 12:30 a.m., when the BPV jack push-button was depressed, the number three BPV immediately opened from 0 percent to 75 percent. Reactor pressure decreased approximately 50 psig, and reactor water level decreased by 8 inches.

The ]BPV jack demand was reduced to zero and all of the BPVs reclosed, with the exception of number two BPV, which reclosed to its original 45 percent open position. The reactor water level control system responded to the decreasing water level and increased feedwater flow to the reactor.

The influx of cold feedwater along with increasing reactor pressure caused reactor power to rise at a rate of a 70 second period. Reactor power peaked at 13.5 percent on the average power range monitors (APRMs), exceeding the APRM rod block alarm setpoint of 12.5 percent for 25 seconds. Reactor water level was restored to 30 inches, and EHC automatically stabilized and controlled reactor pressure at 800 psig. The transient lasted approximately one minute.

SER 3-03, Page 3 of 11

M I

The second transient also met 1OP termination and scram criteria. However, the operating crew did rot scram the reactor because of a deficient understanding of the criteria. The shift manager subsequently provided direction to terminate the use of manual pressure control while the reactor was critical and to use automatic pressur control for the remainder of the shutdown. The cooldown and shutdown evolution was then completed with no other operational challenges.

Preparation aiid Implemenintation of the IPTE Iniplementing Procedure was Inadequate Implementation of the IPTE process was not sufficient to ensure that desired outcomes were met, and management did not detect breakdowns in administrative barriers. These resulted in control room personnel not meeting station expectations when faced with adverse plant conditions.

The following areas that contributed to the event are discussed in subsequent sections:

  • Crew Performance and Management Oversight
  • Procedure Adherence
  • Procedure Revision Process
  • Training
  • IPTE Process Procedure
  • Problem Identification and Resolution Crew Performance and Managemnent Oversight Operating crew performance did not meet station expectations. Weaknesses included nonconservative reactivity management, ineffective management oversight and independence, and procedure noncompliance.

From the beginning of the process, the operating crew became overly focused on the technical aspects of the shutdown. The reactor operator was focused on power monitoring, while the control room supervisor was concerned about the possibility of an uncontrolled plant depressurization that could cause the reactor cooldown rate to be exceeded. The STA was located at the EHC control, directly behind the plant operator who was manipulating the EHC controls. This prevented the STA from maintaining oversight of plant response, and overly engaged him in the crews' decisions.

When presented with an unexpected increase in reactor power, the operating crew did not scram the reactor as required by the IOP termination criteria. Crewmembers did not recognize that the criteria had been exceeded because they had not reviewed them or the expected crew actions in the procedure prior to the evolution.

Shift management did not ensure that appropriate training was conducted, that all termination criteria were fully understood, and that the crew scrammed the reactor when appropriate. Shift SER 3-03, Page 4 of 11

M

_l Il 1

11l I

manager and STA oversight and independence were ineffective, in part because they were overly engaged with crew actions during the development and implementation of the IOP.

The test manager and test engineer did not maintain an appropriate independence, which adversely affected their abilities to maintain oversight of the evolution. In particular, the test manager assignment was made at the beginning of the shift, which provided insufficient time for the individual to thoroughly review the IOP and ensure that all the requirements of the IOP and crew preparation were met. The test manager did not establish a communication link to senior management, as required by procedures, when the crew encountered the unexpected transient conditions.

Procedure Adherence Station personnel did notfollow operating and administrative procedures during the preparation and implementation of the LPTE.

The termination and scram criteria contained in the LOP relied on operator interpretation and decision-making at the time of the event. In addition, prior startup training involved small power changes induced by feedwater transients, and the associated procedures did not require a scram.

This conditioning may have influenced the crew's decision not to scram the reactor during the power excursions. The LOP required that the mode switch be placed in shutdown if an unexpected positive reactivity addition occurred during low power. Neither the test manager nor the shift manager had thoroughly reviewed the IOP. Consequently, the crew and test personnel did not recognize that termination criteria had been exceeded twice during the evolution.

The reactor operator controlling reactor pressure with EHC did not follow IOP requirements to keel) the automatic pressure setpoint between 50 to 100 psig as pressure control was being swapped between automatic and manual to lower reactor pressure. Crew supervision and the STA were cognizant of but did not question these actions.

Procedure Revision Process Participants in the operations procedure revision review process for the integrated operating procedure usedfor the IPTE did not recognize that the use of manual pressure control was as an undesired method with the reactor critical and at elevated pressure.

Operations personnel reviewing the revised IOP did not recognize that the use of a manual reactor pressure control when the reactor was critical was undesirable because it resulted in two different board operators controlling reactivity at the same time. The individual performing the 10CFR50.59 safety evaluation review of the IOP, who had been involved in developing and writing it, did not identify the adverse impact of including manual pressure control and overrelied on the department reviewers. Conversely, the department reviewer, who had also been involved in preparing the 1OP, only focused on usability, clarity, and impact on associated procedures. Consequently, the 50.59 review did not identify the potential impact of manual pressure control mode during reactor critical operation. The individual performing the station qualified reviewer (SQR) review did not use a required form, but a more narrowly defined worksheet that focused on a mechanistic review of the procedure. Consequently, the SQR did SER 3-03, Page 5 of 11

not identify the potential impact of using the manual pressure control mode when the reactor was critical.

Because the problem was not identified, the completed 1OP, which included the use of manual pressure control, was not presented to the SORC for review. As a result, SORC members were unaware that manual pressure control was to be used to reduce reactor pressure when the reactor was critical. The SORC had earlier reviewed a preliminary plan, but it proposed only using automatic pressure control to lower reactor pressure. The SORC governing procedures required the IPTE procedure to be reviewed by the SORC if a potential problem or a significant change was been identified during the initial review process.

Training Operator just-in-time training was not sufficient to ensure the operating crew was prepared to perform the evolution.

Station management did not ensure the crew received the training necessary to perform the test.

Because the implementing procedure for the IPTE had not yet been completed, it could not be practiced or discussed during the simulator training conducted on Saturday evening, March 15.

The training did not contain industry and station operating experience, a discussion of contingencies for potential problems, specific termination criteria, and expected crew actions for termination. Furthermore, one of the reactor operators, who was added to the crew for the evolution, had not attended the training exercise or the prejob briefing.

Simulator training was subsequently characterized as a procedure verification and validation.

Training department personnel recognized that the activity was not just-in-time training, yet they did not follow up with operations management to ensure that just-in-time training was conducted prior to the evolution.

IPTE Process Procedure Deficiencies The governing IPTE process procedure contained several fundamentalflaws that conflicted with the intent of SOER 91-01, "Conduct of infrequently Performed Tests or Evolutions. "

  • The procedure did not require the appointment of a line manager, senior to the shift supervisor, with the authority and experience to exercise continuous responsibility for the oversight of an IPTE. The assigned test manager, who was expected to fulfill this role, instead had a position equal to the shift manager. It was expected that the test manager would consult other senior line managers if problems were encountered during the IPTE.

However, this was not done because the test manager did not recognize that the transients met that expectation.

  • The procedure did not require an engineering review for technical accuracy. Although engineering personnel were initially approached for input, a formal engineering review of the IOP was not performed.
  • The procedure did not provide requirements for training department responsibilities and expectations. Although training personnel observed and recognized that the training SER 3-03, Page 6 of It

MMMK exercise was not just-in-time training, no one questioner the operations department regarding when appropriate just-in-time training would conducted.

Problem Identification and Resolution Ineffective prior problem identification and resolution influenced the crew to take inappropriate actions when confronted with unexpected problems.

During an earlier plant shutdown on March 7, 2003 to repair the B recirculation pump seal, several reactor water level swings occurred when automatic pressure control was used.

Consequently, the operating crew wrote a notification report describing the water level swings, requesting that the shutdown procedure be changed to allow use of manual pressure control when critical to ensure smooth plant control. This influenced the operating crew to include manual pressure control when the IOP was revised for the IPTE.

Prior experience with reactor water level swings also influenced how the operating crew responded o tac first transient. Although the minor transient indicated an abnormal condition, which was later determined to be the result of a bad rheostat, the crew rationalized that the condition was tb-rcsuL! of using automatic pressure control and that manual pressure control was acceptabl for zontinuing the shutdown as planned.

In addition, short-and long-term plant equipment degradation has presented challenges to plant operation. Exposure to these conditions over time may have reduced operator sensitivity to the equipment degradation. For example, the station auxiliary boiler is currently configured to run at a reduced capacity because of Environmental Protection Agercy concerns. The reduced capacity provides an insufficient amount of nonnuclear steam sL ply to operate the condenser steam jet air ejectors after the scram. This contributed to the de-ision by station personnel to pursue a controlled shutdown versus scramming the reactor and -losing the main steam isolation valves when the stuck-open BPV was originally discovered. Thl boiler had been operated in this condition for several years. The operations department has -ot been successful in prioritizing and improving performance of the auxiliary boilers l irough the corrective action program.

The unexpected opening of the BPVs during both transients war caused by a bad rheostat within the manual pressure control system. The first transient present indications of a problem, but no immediate follow-up actions were taken to determine if a "o ponent problem existed.

Conclusions INPO 02-005, Analysis of Significant Events, identified the mu.

frequent cause of the 20 most significant events as "Personnel did not have an appreciation oi ie risks associated with their actions or did not exhibit a sufficient questioning attitude toward these risks." The report also concluded that "Problems with procedure guidance and inappror' iate use of procedures have been and continue to be problem areas," and "Shift managers no} remaining in an oversight role in a transient continue to be contributors." These conditions we evident in the event described in this report.

SER 3-03, Page 7 of 11

I The decision by the operating crew to continue with a normal plant cooldown after IPTE termination and scram criteria had been met (not recognized by the crew) twice during the evolution indicates an insufficient questioning attitude, inadequate procedure preparation and implementation, and insufficient management oversight.

In addition, the crew decision to reduce reactor pressure using manual pressure control was made without an appreciation of the possible consequences of having two different reactor operators controlling reactivity at the same time with the reactor critical. Without this appreciation, and reviews crew and shift management decisions and reviews regarding the shutdown strategy were made without outside input.

Awareness of the potential risks involved, or an approach whereby station personnel were able to better discern that additional risks existed, may have helped prevent the problems noted.

References

1. Root Cause Analysis Report 70030270, "Hope Creek Reactivity Event," Salem/Hope Creek Generating Station, June 10, 2003.
2. Nuclear Networks Operating Experience message OE-15852, "Power Transient during Shutdown to Repair a Bypass Valve (Preliminary OE)," March 31, 2003.
3. Nuclear Network Operating Experience message OE-15683, "Plant Outage to Resolve Increasing Drywell Floor Drain Leakage," March 10, 2003.
4. INPO 02-005, Analysis of Significant Events, September 2002.
5. INPO SOER 91-1, "Conduct of Infrequently Performed Tests or Evolutions," June 1991.

Plant Information Unit:

Hope Creek Year Commercial:

1986 Reactor Type (Size):

BWR (1,118 MWe)

Reactor Manufacturer:

General Electric Turbine Manufacturer:

General Electric Plant Designer:

Bechtel Event Date:

March 17, 2003 Event Criteria Unusual Plant Transient Involving Nuclear Safety (organization shortfalls that contributed to inadequate preparation and implementation of an infrequently performed test or evolution procedure, which resulted in the reactor not being scrammed as required)

Cause Categories Written Procedure (insufficient guidance in LOP)

Training/qualification (insufficient knowledge and just-in-time training)

SER 3-03, Page 8 of 1

i I

111l11111ll0ll Managerial methods (insufficient oversight of procedure preparation and implementation, insufficient prejob briefing, and insufficient use of operating experience)

Work practices (inappropriate actions by operating crew members)

This document is based on technical information provided by PSEG (Salem/Hope Croek Generating Station). Utilities and participants are requested to provide feedback on similar occurrences and solutions at their plants to the information contact listed below.

Keywords HUMAN PERFORMANCE, INFREQUENTLY PERFORMED TEST OR EVOLUTION, MANAGEMENT OVERSIGHT, OPERATOR TRAINING, PREJOB BRIEFING, PROCEDURE ADHERENCE, SUPERVISORY OVERSIGHT Telecopy No.: (770) 644-8121 Information

Contact:

Gary Schweitzer, (770) 644-8339; e-mail: SchweitzerGJ@inponn.org Limited Distribution: Copyright 2003 by the Institute of Nuclear Power Operations. Not for sale nor for commercial use. Unauthorized reproduction is a violation of applicable law. Each INPO member and participant may reproduce this document for its business use. This document should not be otherwise transferred or delivered to any third party, and its contents should not be made public. without the prior agreement of INPO. All other rights reserved.

Notice: This information was prepared in connection with work sponsored by the Institute of Nuclear Power Operations (INPO). Neither INPO, INPO members, INPO participants, nor any person acting on behalf of them: (a) makes any warranty or representation, expressed or implied, with respect to the accuracy, completeness, or usefulness of the information contained in this document, or that the use of any information, apparatus, method, or process disclosed in this document may not infringe on privately owned rights; or (b) assumes any liabilities with respect to the use of, or for damages resulting from the use of any information, apparatus, method, or process disclosed in this document.

SER 3-03, Page 9 of 11

I MM PREVENT EVENTS Learning from Industry Experience PREVENT EVENTS is intended for use by personnel during morning meetings, prejob briefings, and work unit meetings to communicate key industry experience.

On March 17, 2003, during an infrequently performed test or evolution (IPTE) for a controlled shutdown at Hope Creek, two separate, unexpected power transients occurred that met the termination and scram criteria in the IOP. In both cases, the operators did not scram the reactor as required.

The first transient involved a power change of less than 1 percent when three turbine bypass valves (BPVs) opened and closed quickly. The crew discussed that plant's response and elected to continue the IPTE shutdown. The second transient, which occurred about 15 minutes later, was more significant. The number three 13PV opened suddenly, causing reactor pressure to decrease 50 psig and reactor water level to decrease 8 inches. The cold water addition to restore and maintain reactor water level added positive reactivity, increasing reactor power from 6.5 percent to 13.5 percent for about 25 seconds. The operating crew stabilized the plant and did not scram the reactor. After discussing the plant response with the duty assistant operations manager, the crew continued the shutdown and removed the unit from service without further incident.

Management

1. How can we be sure our staff appreciates the risks associated with the preparation and implementation of an IPTE procedure?
2. Is the level of management involvement in monitoring an IPTE appropriate?

Operations Supervisors

1. How can we ensure that appropriate oversight is maintained during the preparation and implementation of an IPTE procedure?
2. What can be done to ensure that test termination criteria are understood and followed during implementation?
3. What are the oversight roles of the shift manager and STA during the preparation and implementation of an IPTE?

Shift Technical Advisors

1. What are the consequences of becoming too involved with the actions of an operating crew?

Test Manager/Engineer

1. What actions need to be taken by a test manager and/or engineer to ensure that an IPTE procedure is properly developed and implemented?

SER 3-03, Page l0 of 1 1

I 11 i

=

0

--- I Training

1. How can we ensure that appropriate just-in-time training is conducted prior to implementation of a newly prepared 'PTE procedure?

Please provide feedback on the usefulness of PREVENT EVENTS to Bob Heublein, INPO Events Analysis Department manager, at (770) 644-8671 or Internet address:

HeubleinRM@inponn.org.

Information

Contact:

Gary Schweitzer, (770) 644-8339, or Internet address:

SchweitzerGJ @ inponn.org SER 3-03, Page 11 of 11

Retrieved from "https://kanterella.com/w/index.php?title=ML060590430&oldid=5369208"