IR 05000440/1997022
| ML20202F047 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 12/31/1997 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML20202F033 | List: |
| References | |
| 50-440-97-22, NUDOCS 9802190118 | |
| Download: ML20202F047 (12) | |
Text
.
'
.
,
U 8. NUCLEAR REGULATORY COMMISSION REGIONlli Docket No:
50440 Ucense No:
NPF-58 Report No:
50440/97022(DRP)
Licensee:
Centerior Service Company Facility:
Perry Nuclear Power Plant Location:
P. O. Box 97, A200 Perry, OH 44081 Dates:
October 15 - December 1,1997 Inspectors:
D. Muller, Reactor Engineer D. Kosloff, Senior Resident inspector Approved by:
Thomas J. Kozak, Chief Reactor Projects Branch 4 i
9002190118 971231 PDR ADOCK 0$000440
.
-
.
EXECUTIVE SUMMARY Pony Nuclear Plant, Unit 1 NRC Inspection Repori No. 60440/g7022(DRP)
This special inspection reviewed the circumstances which led to an inadvertent loss of reactor pressure vessel water inventory that occurred on October 11,1997, in addition, the licensee's root cause i*westigation, short term corrective actions, and long term correc6e actions in response to this event were reviewed.
Operations Personnel involved with a tag-out generated for scram air (SA) header valve work e
inoonectly believed that the scram discharge volume (SDV) vont and drain valves would close before the scram inlet and outlet valves started opening as the SA header slowly depressurtred. However, the scram inlet and outlet valves swried opening before the SDV vent and drain valves closed which created a water flowpath from the reactor pressure vessel (RPV) to the suppression pool. -The inadequate evaluation of the effects that the tag-out would have on plant conditions was a violation (Section 01.1).
A lack of a questioning attitude by those involved with the SA header work tag-out, poor
.
work planning practices, and poor communications contributed to the creation of the water flowpath from the RPV to the suppression pool (Section 01.1).
The response by control room personnel to the inadvertent loss of reactor pressure
.
vessel water inventory was satisfactory overall. The diagnosis of the event and the determination to insert a manual scram to terminate the event were well thought-out.
However, a weakness in the response was a delay by the operator at the controls ir, reporting th p@lem to the Control Room Unit Supervisor (Section 01.1).
The actual safety significance of this event was low. This event, however, is of significant
conoom due to the multiple failures of baniers designed specifically to ensure evolutions are conducted in a safe and controlled manner and the potential for similar events to continue to challenge plant operators (Section 01.1).
The investigation that followed this event wss conducted in a methodk al, thorough and
timely fashion. The investigation team's findings accurately represented the issues surroundir.g this event (Section 07.1).
The short term corrective actions that were taken as a result of this event were timely and
appropri#e. The long term correctivs actions also appear appropriate, however, at the time of this inspection, corrective actions for generic tag-out problems were still being
~ developed (Section 07.2).
-
.
_
_
__
i
_
.
_ _ _ _ _ _. _ _ _ _. _
_.
_.... _ - - _ _. _.... _. _ _. _ _. _ _ _ _. _ _. _. _ - _ _ _ _.
.
.
.
!
Ranort Detalis Summarv of Plant Status i
.
Unit 1 was in cold shutdown following refueling with the 'B' residual host removal system aligned for shutdown cooling at the time of the event.
i
[ Operations
!
01-Conduet of Operations
!
01,1 Unexpected Loss of Reactor Pressure Vessel (RPV) Water inventory a.
Inanection Soone f71707)
'
The inspectors reviewed the unexpected loss of approximately 2g00 gallons of RPV f
water inventory that occurred following the placement of a tag-out on the scram air (SA) -
l header, inspection activities included; l
interviews were conducted with the licensee's investigation team members, work e
planners involved with the tag-out, and the operator at the controls (OAC) during the event.
Reviews were conducted of PAP.1401, " Safety Tagging," Rev. 8 and the tag-out
'
- -
that was involved in the event.
il
' b.
Observations and Findinos On October 10, igg 7, several valves and fittings in tr,e SA header were it,entified as -
requiring rework, due to leaks discovered during post ma'ntenance testing. During the y
night shift on October 10, a tag-out (tag-out number 2g4g1) was developed to support the rework activities. This tag-out consisted of, in part, isolating the SA header and allowing -
the header to slowly depressertze througli the leaks in the system.- There _was some conoom by a work planner that slowly depressurizing the SA header might not ensure the r
- positive and rapid repositioning of the scram inlet and outlet valves (on each hydrculic l
control unit) and the scram discharge volume (SDV) vont and drain valves. A separate
-
document was thus developed which required inserting a manual scram in conjunction
,
with the tag out. This soprate document, however, was never approved for use and no
'
reference to this separate document requiring a scram appeared on the tag-out sheet.
During the day shift on October 11, the Project Manager (PM di) who_had everall.
L responsibility for planning the work was replaced with a different Project Manager (PM
- 2). Also, during the night shift of October 10 through the day shift ci October 11, the
,
scope of the work changed twice. The first change involved depressurizing only that
'
L portion of the SA header necessary S rework the leaks.:The remainder of the SA header was to remain pressurized via " sir jumpers' and temporary power supplies to allow for restoration of scram accumulators in parallel with the SA header work. This first change i--
in scope added a significaat amount of complexity, and so a responsible system engineer
,
l was involved in the planning of the work. The second and final change in the scope of
l a....
.
-
~.
.
-
- -.
.
...
.
. - - - - -
..
-
.
-
.
.
the work occurred after a walkdown was performed on the SA header by PM #2. This walkdown determined that only one valve in the SA header required reworn at that time (the 'A' backup scram valve, C11 F110A); the other leaks were small enough that they could be worked at a later date. This second end final change in scope resulted in the decision to depressurtze the entire SA header, and thus avoided using the complicated plan involved with maintaining most of the header pressurized. 98nce this second change in scope was apparently of much less complexity than using " air jumpers" and temporary =
power supplies, the responsible system engineer was removed from the planning process. - A complicating factor in the planning process was that while the SA header work was being planned, work was also being planned to fill and vent the control rod drive hydraulic system and conduct other testing involving multiple scrams and resets.
Project Manager #2 proceeded with the plan (i.e., the secc.:':1 change in scope plan)
during the day shift of October 11. This plan consisted, in part, of developing a tag-out that would isolate and slowly depressurize the entire SA header to allow for work on the C11 F110A valve. The tag-out that was developed was a revision of the original tag-out
.g (tag-out number 2g4g1) drafted during the night shift of October 10. This plan included
bypassing the SDV high level scram function to prevent an inadvertent scram that would result as the SDV filled with RPV water. This tag-out, however, did not include the separate document that required that a mar.ual scram be inserted. All personnel involved with the tag-out (preparar, reviewer, approver and control room personnel) incorrectly believed that the SDV vont and drain valves would close before the screm inlet and outlet valves started opening as the SA header slowly depressurized. However, as noted below, the scram inlet and outlet valves started opening before the SDV vent and drain valves closed which created a water flowpath from the reactor pressure vessel to the suppression pool.
Technical Specification 5.4.1.a requires that written procedures be established, implemented, and maintained covering activities recommended in Regulatory Guide 1.33, Rev. 2, Appendix A. Technical Specification 5.4.1.a applies to PAP 1401. The failure of the preparer, reviewer, and approver of tag-out 2g4g1 to identify that a water flowpath from the RPV to the suppression pool would be created by using the tag-out was a violation of TS 5.4.1.a in that PAP 1401 required an adequate evaluation of the tag-out's effect on plant status prior to authorization (VIO 50-440/g7022 01).
At about 1:30 p.m. on October 11, the tag-out was delivered to the control room by the Work Support Center Unit Supervisor (WSC US). The WSC US informed the Control Room Unit Supervisor (CR US) of the effects of the tag-out. The effects discussed included: (1) the SA header was being isolated and depressurized, (2) the crew should receive expected alarms (SA header low pressure, SDV not drained, SDV high level rod block, and SDV high level scram) and (3) the crew should bypass the SDV high level scram to prevent an inadvertent reactor protection system actuation. The WSC US did not discuss the potential for establishing a flowpath that would reduce RPV water inventory.- A similar briefing was also conducted separately with the OAC.
,
dated on direction received from the CR US, an operator placed the SDV high level bypass switches in the bypass position. At about the same time as the SA header tags were being placed, the OAC began a routine RPV water inventory reduction, so as to maintain RPV water level within the assigned band of 360" to 365", _ Reactor pressure
-.
-
veesel water inventory had to be periodically reduced due to flow into the RPV from
_ _ _ _ _ - _ - _ _ _ - _ - _ _ _ - _ _
.- _ - -
.--- - - -
.
.
.
recirculation pump seal purge. This routine reduction of RPV wrtr inventory was accomplished via the reactor water cleanup (RWCU) dump valve to radwaste; it was a fairiy slow process, taking 1 to 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to reduce RPV level 5".
An approximate time line of the events of October 11 after the SA header tags were in place is as follows:
Time (o m3 Event Description SA header began to slowly depressurize
-
2:35 At 72 p.ounds per square inch-gauge (psig) in the SA header, the "SA headef pressure low" alarm was received, and was treated as an expected alarm.
_
2:53 SA header pressure reached 38 psig. Scram inlet and outlet valves started to drift open, and the "SDV not drained" alarm was received. This alarm was treated as an expected alarm. At this point in the event, the SDV vent and drain valves were still open, and a flowpath from the RPV to the suppression pool was established via the SDV and the open SDV vent and drain valves. The OAC did not notice that the SDV vent and drain valves were open. As SA header pressure continued to slowly decrease, more scram inlet and outlet valves began to open, which resulted in a higher flowrate from the RPV to the SDV. This higher flowrate from the RPV exceeded the drain capacity of the SDV drain lines, and SDV level began to increase.
2:56 GDV high level rod block and scram alarms were received. No scram occurred due to the SDV high level scram function being bypassed.
These alarms were treated as expected alarms.
3:02 RPV level reached 360", the low-end of the specified band. Shortly thereafter, the OAC observed ' hat RPV level was low out of the specified band. The OAC also noted that it took much less time than usual to lower RPV level. The OAC closed the RWCU blowdown flow control valve, by adjusting its flow controller to zero, and looked for a response in RPV level. RPV level continued to lower. At this point in the event, the OAC began to look for a possible cause for the loss of RPV water inventory. The OAC also closed a second valve in the RWCU to radwaste blowdown path, but RPV level continued to lower.
3:09 A second operator obse:ved that the OAC was having difficulties with some evolution at the controi panel. The second operator approached the control panel to see what the problem was. The *econd operator noticed that RPV level was approximately 353". The second operator mentioned to the OAC that the CR US should be informed. The second operator also mentioned that the SA header tag-out may be a possible cause.
3:11 The CR US was informed that RPV level was low out of the band. The CR US entered off normal inst:uction (ONI)-E12-2, " Loss t Decay Heat Removal," which was applicable for a loss of RPV level while in cold shutdown.
_ _. _
..
....
-_
.
'
.
i 3:13 The CR US informed the Shift Supervisor (SS) of the event and the SS assumed command in the control room. The SS directed that PM #2 report to the control room with the SA header tag-out.
3:16 PM #2 arrived in the control room with the tag-out. The SS directed the OAC to determine scram inlet and outlet valve and SDV vent and drain valve positions. The SA header tag-out was evaluated as a possible cause for the loss of RPV water inventory, 3:20 --
As directed by the SS,' the OAC placed the SDV high level scram bypass -
switches to normal. This action caused a scram and closed the SDV vent and drain valves, terminating the event at approximately 347" RPV level.
Also, as directed by the SS, the OAC deprersed the manual scram pushbuttons. RPi' level was slowly restored to the normal band (360" -365").
4:33 The licensee informed the NRC of the problem.
'
Reactor pressure vessel level decreased at a rate (on average) of approximately one-half inch per minute during this event. The total change in RPV level was approximately
--18 inches (365" to 347") or about 3600 gallons. Of this 3600 gallons,' approximately 2900 gallons went to the suppression pool (not expected by the licensee),500 gallons went toward filling the SDV (expected by the licensee), and approximately 200 gallons --
- went to radwests from the routine RPV water inventory reduction evolution.
_
Shutdown Cooling (SDC) was not affected by this event. However, had this event not
- been terminated, the OAC.would have received an RPV level low alarm at 197".
Shutdown cooling would hue been Icst if RPV level reached 177" (Iow RPV level SDC
-
isolation)._ At a rate of decrease of one-half inch per minute, this event would have had to I:
continue, with;ut operator action, for another 5 to 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> befom SDC would have been
<
lost.
As can be seen from the time line above, some delay occurred between when the OAC first noted that RPV level was low out of the prescribed band (and still decreasing) and when the CR US was notified. This delay was due to an attempt by the OAC to diagnose the cause of the RPV level drop. During an interview, the OAC stated that he wanted to present a solution to the CR US, vice presenting only the problem.-
Also,' as can be seen from the time line, approximately 9 minutes elapsed between when the CR US was notified and the scram was inserted which terminated the event. This time was used by control room personnel to ensure that a correct diagnosis was made
and that the correct actions were taken. Based on the relatively slow rate of RPV water invento'y loss, control roorn personnel believed that there was no need to take rapid action; there was ample time to carefu,y assess the situation and develop the correct plan of action.
'
_
_ _ _ _ _ _
.
c.
Conclusions Evaluation of the Licensee's Work Plannina i
The overall cause of this event was an inadequate tag-out, due to a common misunderstanding of the effects of a slow depressurization of the SA header. Examples of poor work planning and inadequate tag-out practices were observed. Changes in the work scope and reassigning PMs coupled with poor communicailons between the work planners centributed to the tag-out being P aced without a manual scram being inserted.
l The work planning was additionally complicated by the perceived need to perform several jobs on the control rod drive hydraulic system in parallel. This perceived need it. conduct these jobs in parallel was somewhat driven by the desire to rapidly complete the refueling outage. In addition, the work planners did not ensure that the plant was in the desired con 5guration prior to placing the tag-out, and the work planners demonstrated a lack of a questioning attitude conceming the tag-out. All of the personnelinvolved with the final tag-out incorrectly believed that as the SA header slowly depressurized, the SDV vent and drain valves would close before the scram valves would begin to open. Finally, the original tag-out sheet did not reference the separate document which required inserting a manual scram as part of the tag-out.
Evaluation of the Response in the Control Room The control room personnel response was satisfactory overall. The diagnosis of the event and the determination to insert a manual scram were well thought-out and executed. The control room personnel were correct in their assessment that there was no need for a rapid response; there was in fact ample time for diagnosis and plan of action determination. However, a weakness in the response was the delay between problem discovery by the OAC and reporting this to the CR US.
Safety Sionificance The actual safety significance of this event was low. Shutdown cooling was never significantlyjeopardized. This event, however, is of significant concem due tu the multiple failures of barriers designed specifically to ensure evolutions are conducted in a safe and controlled manner and the potentia' for similar events to continue to challenge plant operators.
Quality Assurance in Operations 07.1 Licensee's Root Cause investiaation a.
Inspection Scope (71707 and 40500)
The inspectors interviewed members of the licensee's investigation team and reviewed their investigation report conceming the inadvertent loss of RPV water inventory event.
b.
Observations and Findinas On October 11, the licensee initiated an investigation into the cause of the inadvertent loss of RPV water inventory. The invastigation team conducted interviews with the work
.,
.
.
.
.
.. t
-
.
.
planners and operators involved, reviewed the associated work documents, and reviewed plant data and iogs during the event. From this data, the investigation team constructed -
a time line of events and determined where inappropriate actions were taken. The
'.nvestigation team then reviewed plant procedures goveming the control of work and determined what barriers were in placw that should have prevented this event. In addition, the team also assembled a database of past similar events that have occurred at Perry and industry-wide.
The investigation team identified three inappropriate acticns that occurred during this event: (1) An inadequate tag-out was prepared, (2) a PM did not consider concoms on a potential drain path when developing the work plan, and (3) the final work scope was not evaluated for risk significance. The team further identified root-causes and contributing factors for each of the ineppropriate actions, including-The process for adding emergent outage work was not well developed.
.
Indoctrination program for tag-out personnel did not include lessons leamed from
industry and Perry events; emphasis on management expectations needed to be -
strengthened.
Previous corrective actions regarding tagging issues have been ineffective.
- PM #2 did not fully understand or question the starting point for the work.
- The tag-out did not reference the separate document which required a manual
.
scram be inserted as part of the tag-out.
inadequate communications occurred between PM #1 and PM #2.
.
Emergent work was not correctly evaluated for risk significance in accordance
.
with existing policy, in addition, the existing policy was not entirely clear on the conduct of risk reviews for emergent work.
The investigation team completed its report, including recommended corrective actions on October 27,1997. Management reviews of this report were completed on October 30.
'
c.
Conclusions The invest.'gation was conducted in a methodical, thorough and timely fashion. The investigation team's findings accurately represented the issues surrounding this event.
07.2 Licensee's Corrective Actions a.
Lrig!qction Scope (71707 and 40500)
The inspectors observed two lessons leamed review sessions of this event, which were a portion of the licensee's short term corrective actions. In addition, the irispectors reviewed licensee documents which described both the short term and long term corrective actions for this event.
b.
Observations and Findinas Short term corrective actions for this event consisted of counseling the key personnel who were involved (work planners, tag-out personnel, the OAC) and conducting a lessons leamed review of this event with all of the operations department, including work planners. These short term corrective actions were completed by October 16.
,
-
__A
._
-_-
-
--.
---
--
.
The lessons leamed reviews were conducted by operations management personnel prior to each crew (including the work support center staff) assuming shift duties. Each review
session lasted approximately 40 minutes. The review sessions consisted of: (1) a presentation of the time line of events, (2) discussions between crew members and operations management personnel to determine the inappropriate actions end root causes for the event and (3) further discussions to determine how to prevent similar errors in the future. In addition, these review sessions discussed other events, management expectations, and the performance of self and peer checks.
Based on the identified inappropriate actions and barrier breakdowns, the licensee developed long term corrective actions for this event. The long term corrective actions identified and the scheduled completion dates included:
Develop and implement a method so that separate documents conceming a
.
E tag-out become connected to the tag-out (5/30/98).
O Revise PAP-1401, " Safety Tagging," so that an additional technical review is
+
required for all tag-outs involving the SA header (4/30/98).
Coach / counsel personnelinvolvuiin preparing and reviewing tagouts to
.
emphasize the importance of producing tagouts based on verifiable, factual information (11/30/97).
Modify existing plant policies to include:
.
treating work on the SA header as risk significant during an outage
-
(5/30/98);
processes for emergent and outage work, including risk reviews and
-
shutdown safety reviews (5/30/98);
an expectation that reinforces employing positivo measures to obtain the
-
desired plant configuration during tag-outs (5/30/98);
Consider activities which create the potential for RPV water inventory loss es risk
significant (5/30/98);
Perform an evaluation to determine the activities which create the potential for
.
RPV water inventory loss (5/30/98);
Communicate the lessons teamed from thls event to other managers, and
.
incorporate this event into the refueling outage 6 lessons leamed package (11/22/97);
Strengthen th6 indoctrination program for new personnel assigned to tag-out
preparation and review. Specific items to be added willinclude reviews of Perry and industry events, PAP-1401, " Safety Tagging," and management expectations (5/30/98).
In addition, the licensee indicated it will continue to stress the reporting of problems, ownership of problems, and teamwork during operator training. Also, a separate potential issue form (No. 97-2245) has been written to document this event and other recent tag-out problems at the facility.
c.
Conclusions The short term corrective actions were timely and appeared to be adequate. The long term corrective actions are appropriately scheduled and appear to be satisfactory. At the
_ _ _ _ _...
'
'
.'
e
,
-
.
time of this inspection, corrective actions for generic tag-out problems (as documented in potential issue form No. 97 2245), were still being developed.
OS-Miscellaneous Operations issues 08.1 (Closed) LER 50-440/97013: Control rod drive hydraulic system maintenance activities result in reactor protection system actuations. This issue is the subject of this inspection import. No new information was provided in the LER. The LER is closed.
V. Mananoment Meetinas X1'
Exit Meeting Summary i
The inspectors presented the inspection results to members of licensee management on
'
Decamber 1,1997. The licensee acknowledged the findings presented. The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary.- No proprietary information was identified.
!
.
.
.
.
..
.
..
.
r
.
.
<
.
.
PARTIAL UST OF PERSONS CONTACTED Ucensee L Myers, Vice President - Nuclear H. Hegrat, Regulatory Affairs Manager W. Kanda, General Manager Nuclear Power Plant Department J. Messina, Operations Manager T. Rausch, Director, Quality and Personnel Developi.1ent Department R. ',chrauder Director, Nuclear Engineering Department J. Wright, instrumentation and Controls Superintendent i
i
, _.
_ _ _ _ _ _ _ _ _ _ _
.
..
..
.
,.
.
...
ITEMS OPENED, CLOSED, AND DISCUSSED Opened 50-440/97022-01 VIO Failure to adhere to the safety tagging procedure results in an inadequate tag-out 99AtY.
50-440/97013 LER Control rod drive hydraulic system maintenance activities result in reactor protection system actuations Dscussed NONE
- -
-
____ _ __j
,
.. _
.
...
..
.