ML20080A913

Forwards Investigative Committee Final Rept Re 840109 out-of-sequence Shutdown Event.Rev to Dgp 3-4 Providing Consistent Method of Designating Sequence Steps Recommended

ML20080A913
Person / Time
Site:	Dresden
Issue date:	01/30/1984
From:	Rybak B COMMONWEALTH EDISON CO.
To:	James Keppler NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
Shared Package
ML20080A916	List: ML20080A913 ML20080A921
References
7971N, NUDOCS 8402060396
Download: ML20080A913 (10)

Text

-~

n) - .. n /nb

/ One First Nitional Plaza, Chicago Illinois O ' Address Reply to: Post Office Box 767 Chicago. Illinois 60690 January 30, 1984 Mr. James G. Keppler Regional Administrator U.S. Nuclear Regulatory Commission Region III 799 Roosevelt Road Glen Ellyn, IL 60137

Subject:

Dresden Station Unit 2 Out-o f-Sequence Shutdown Event on January 9, 1984 NRC Docket No. 50-237 Reference (a): J. G. Keppler letter to Cordell Reed dated January 10, 1984.

Dear Mr. Keppler As required by Commonwealth Edison Cdmpany procedure, the final report on the subject event has been completed by the Investigating Committee. In accordance with Item 4 of the referenced letter, we are onclosing the final report in the form of an attachment to this letter.

If you have any questions on this matter, please direct them to this office.

Very truly yours, h, '

B.Qyb Administrator Nuclear Licens im Attachment cc: NRC Senior Resident Inspector - Dresden R. Walker - NRC Region III 7971N M 8402060396 840130 T,gS

/

PDR ADOCK 05000237 PDR S g Gil-g-ei---

- A *g1*- * -ew 4:eyw- p wp- ,,memr ye - - - - - - - - - gag--

e

i. -

Dresc+n 2 Out-of-Sequence Shutdown January 9, 1984 Investigative Committee - Final Report INTRO, DUCTION On January 9, 1984, Dresden Unit 2 was being shut down to the HOT STANDBY condition to repair 2B EHC pressure regulator. During control rod insertions to reduce reactor power, a number of steps in the control rod sequence were skipped, resulting in an out-of-sequence shutdown. As described below, on-site evaluation of the sequencing errors resulted in declaring a potentially significant event based on the guidelines established in Nuclear Stations Division Directive NSDD-A07. As a result, on 1/10/84 an investigative committee was established and an investigation initiated. This repart presents the findings of the investigation and specifi2s long tem corrective actions to be taken.

INVESTIGATION COMMITTEE The committee was appointed by Mr. B. B. Stephenson, Operations Manager NSD and Mr. D. J. Scott, Dresden Station Superintendent on January 10, 1984. The committee consisted of the following personnel:

R. M. Ragan Dresden Assistant Superintendent - Operating J. R. Wojnarowski BWR Plant Support Group Leader, NFS D. J. Nel son Dresden Assistant Lead Nuclear Engineer T. E. Burns Dresden Training Instructor J. G. Uremovic Nuclear Station Operator - Dresden (Operating Union Steward - local 1460)

Investigations were initiated by individual members of the committee on January 9 and 10. The full committee met on January 11 through 13 to continue and complete the investigation.

INITIAL CONDITIONS PRIOR TO EVENT Dresden Unit 2 had been operating at full power when the determination was made by plant management that a shutdown to HOT STANDB) was required to repair 2B EHC pressure regulator. At 1900 hours0.022 days <br />0.528 hours <br />0.00314 weeks <br />7.2295e-4 months <br /> on 1-9-84, a 200 MWe/hr load decrease on recirculation flow was begun in accordance with DGP 2-4.

At 2100 hours0.0243 days <br />0.583 hours <br />0.00347 weeks <br />7.9905e-4 months <br /> minimum recirculation flow was reached with reactor power _

level at approximately 415 MWe. Control rod insertion then began in accordance with control rod sequence A2 revision 4. Rods were inserted beginning at step 149, in reverse order as required for shutdown.

EVENT DESCRIPTION (Background - On 12-27-83, a new copy of control rod sequence A-2 rev. 4 was issued by the Nuclear Engineers. This was to replace the previous copy which had run out of space for NSO signoffs due to 1

previous control rod maneuvers. On the new copy of the sequence, steps which were currently withdrawn were indicated by a diagonal -

line drawn through the "0UT" box for the appropriate steps on the sequence [see attached]).

Subsequent to its issuance, the new copy of the A-2 rev. 4 sequence had been used for additional control rod maneuvers and therefore contained numerous steps which contained the NSO initials, date and time in the "OUT" boxes for the appropriate steps. Therefore the sequence contained a combination of initials (primarily towards the end of the sequence) and diagonal lines (primadly toward the begin-ning) to indicate the current status of sequer .e steps .

Control rod insertion was in progress, beginning at step 149. Step ,

123 of the sequence was the first step where the NSO encountered the diagonal line in the "OUT" box intended to mean the step had been withdrawn. However, the NS0 interpreted the diagonal line to mean the step should be skipped and he proceeded to the next step which had been initialed and dated as withdrawn, step 121. He then inserted that step, and continued in this fashion, inserting steps which were initialed and skipping steps containing diagonal lines.

At step 112, a trainee began performing the rod insertions, under the direction of a new NS0 who had came in early to assist in the shutdown (extra NSO in addition to Unit NS0). Control rod insertions continued, with steps containing diagonal lines continuing to be skipped until step 98 was reached. At this point (40% power) the extra NSO attempted to latch in the Rod Worth Minimizer (RWM). It would not latch in, with RWM error messages "97-NOT LOADED" and "81-INCOMPLETE SEQUENCE" being displayed. They then notified the SCRE who printed out an 00-7, Rod Position Display, off of the process computer to verify symmetrical rod pattern. It was detennined that the rod pattern was symmetric and contained no unusual rod configurations (adjacent rod tips). It was then suspected that the RWM would not latch due to core power being at 40%, above the RUM Low Power Alarm Point of 35%. (Apparently error messages 81 and 97, indicating the sequence was not loaded, were not fully understood at this point.)

Since RWM insertion operability w(as not required above 20% power, control rodby train was resumed NS0) to further reduce core power and re-attempt RWM initialization. ,

Again, all arrays indicating a diagonal line through the corresponding steps were bypassed.

(Subsequent investigation by this committee determined that System m Operational Analysis Department (SOAD) had performed maintenance on the RWM earlier on 1/9/84, culminating in a parity error condition and inability to load the RW'i sequence. SOAD then departed from the site. Shift personnel were not infonned of the RWM status.)

At step 88 (2145 hours0.0248 days <br />0.596 hours <br />0.00355 weeks <br />8.161725e-4 months <br />), the NS0's noticed that all steps preceeding step 88 were also " lined-out", and that an apparent error had been made. This was verified, the Shift Engineer informed and a Qualified Nuclear Engineer called in. Reactor power was at 37% at this point, and further control rod manipulations were terminated (2150 hours0.0249 days <br />0.597 hours <br />0.00355 weeks <br />8.18075e-4 months <br />) 2

, a~. ., , ,- -

'o 4

until the Nuclear Engineer arrived on site (2230 hours0.0258 days <br />0.619 hours <br />0.00369 weeks <br />8.48515e-4 months <br />). The Station Superintendent (D. J. Scott) was notified and a potentially significant ,

event was classified (NSDD-A07), due to the potential for the Banked Position Withdrawal Sequence (BPWS) being violated if the error would have continued below 20% (when the RWM would have been required operable). Assistant Superintendent for Operations, Ron Ragan, was called in and initiated an independent investigation. D. P. Galle, Division V.P. Nuclear Stations, was immediately contacted for concur-rence with the actions in progress.

The Qualified Nuclear Engineer evaluated BPWS concerns, precondi.tioning, rod pattern symmetry. He then proceeded to issue Change Log Instructicns per DGP 3-4 to correct the control rod pattern and bring it into com-pliance with the BPWS rules. Refer to the Unit Nuclear Engineer's Log for a detailed description of the rod pattern recovery.

Attempts were then made to initialize the Rod Worth itinimizer. The ~

RWM would not latch, with error message 61 displayed. The computer sof tware technician was contacted and it was determined that an RWM hardware problem existed (parity error). At 0200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />, control rod insertions were resumed to attain HOT STANDBY using an independent verifier. Sequence steps indicating a diagonal line were inserted.

SUMMARY

OF INVESTIGATION On 1/9/84, upon determination of the potentially significant event, Ron Ragan, Assistant Superintendent of Operations was called in to the site and conducted an initial investigation of the event. All personnel directly involved in the event were interviewed and log books, canputer edits, the DVR and the control rod sequence package were reviewed. His findings concluded that a procedure violation had occurred in that steps in the control rod sequence were skipped without Qualified Nuclear Engineer approval. The steps were skipped when the NSO misinterpreted the meaning of the diagonal line drawn through the "out" box on the sequence steps. No guidance was provided in the pro-cedure or sequence package denoting the intended meaning of the diagonal line.

In addition to the interviews conducted by Mr. Ragan, the following personnel were again interviewed by the full committee on 1/11/84 through 1/13/84 (in some cases by telephone):

Unit NS0 Extra NSO . _ .

Shift Engineer SCRE Computer Hardware Technician (SOAD)

The EA in training involved was unavailable for interviewing but had been interviewed by R. Ragan on 1/9/84. His role in the event was considered secondary due to his trainee status and the fact that he was acting under the supervision of a licensed NSO.

Also, the committee reviewed the NSO, Shift Engineer and Nuclear Engineer log books, the sequence package in use and relevant operating and general procedures (DGP 3-j, DGP 2-4, D0P 400-2). Computer output

was reviewed to confinn that no Techn' cal Specifications had been violated. The circumstances surrounding the RWM being left inoperable -

without shift personnel being notified were also investigated (see

- event description).

As described in the event description, the error occurred when the Unit NSO was inserting control rod steps on the sequence in reverse order. The first 17 steps inserted, spanning over 2 pages in the sequence package had been indicated as withdrawn by NSO initials, date and time. Based on interviews with the Unit NSO, this developed a pattern of inserting steps with initials and skipping steps that were blank (i.e. had not been withdrawn). When the first step containing a diagonal line through the out box was encountered (step 123) the NSO said he immediately associated the diagonal line with " omit" or "not applicable" since that designation is frequently used on other pro-cedures and surveillances to indicate a step was not performed or not required. In addition, he indicated he could not recall previous situations where diagonal lines had been used on the sequence. He proceeded to insert subsequent steps marked with initials and to skip steps marked with a diagonal line. When the EA in training took over rod insertions under the supervision of the extra NSO, the extra NS0 reviewed the portion of the sequence that had been inserted, noting that steps with the diagonal line had been skipped. Discussions with the extra NSO on 1/12/84 revealed that he was not experienced with control rod insertion for reactor shutdown and therefore assumed the actions taken by the Unit NSO were appropriate (the Unit NS0 is an experienced operator considered conscientious and reliable by others).

As a result, he did not question the interpretation of the diagonal lines and rod insertions continued in the same manner as before.

This committee has concluded that use of th'e diagonal line for indicating the' status of sequence steps was ambiguous in that its meaning was not defined in the procedure or sequence package, other designations had been used in the past to indicate steps were withdrawn ("X" or "OUT" written in) and the marking was more commonly used to indicate "not applicable" in other applications. This, in combination with the pattern established of inserting steps with initials and skipping steps without initials, led to the misinterpretation of the diagonal line and the resultant procedure violation when the corresponding steps were skipped.

Both the Unit NS0 and the extra NSO indicated that there was no question in their mind that they were proceeding in accordance with the sequence and therefore there was no reason to seek assistance or guidance. It wasn't until essentially all remaining steps contained diagonal lines _

that it became apparent they had made an error. Immediately upon recognition of the error, rod motion was terminated and the proper management personnel informed. During the interviews with the operators, we questioned them with respect to the intent, purpose and requirements regarding control rod sequences and their familiarity wth the recent out-of-sequence shutdown at Quad Cities. It is our opinion that the operators (particularly the more experienced Unit NS0) are knowledgeable 4

~

- - . , , . - .-_.m _y,____,,___.__-.,__.,v _

E t

),

l l

regarding sequencing requirements and the reasons for skipping certain steps and were familiar with the Quad Cities event and its ramifications. -

We therefore determine that the cause of the procedure violation was human error and that at no time did the personnel involved knowingly ,

violate a procedure or demonstrate any disregard or lack of concern for j procedure requirements. Upon discovery of the error, the operators reacted promptly and in a responsible manner.

During discussions with the computer hardware technician, it became evident that maintenance and testing on the RWM is routinely perfomed without femal control and documentation. Although shift personnel are normally infomed when process computer maintenance is performed, they typically are not infomed of RWM maintenance. This issue is addressed further under EVENT CAUSES and CORRECTIVE ACTIONS.

The committee also noted that upon discovery of the error, the sub-sequent actions by station personnel in correcting and documenting the event were well organized, effective and promptly implemented.

Nuclear Station Division Directive NSDD-A07 was effectively implemented and resulted in prompt definition of the event as potentially significant and fostered an environment of immediate investigation, general office involvement and establishment of interim corrective actions within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of the event.

The following sections specify the principal causes and contributing factors in the event in more detail and provide our recommendations for long term corrective actions. During the course of our investigation, a number of procedural shortcomings were identified which were unrelated to this specific event, but could potentially result in similiar mis-interpretations or oversights in the future. Corrective actions in these areas have also been specified.

~

EVENT CAUSES The investigatio1 committee has identified two principal causes of the event described above.

1. The Control Rod Sequence Package lacked guidance with respect to the symbols and notes contained within it. Although these no-tations were properly recorded, there weren't sufficient in-structions as to their meaning and significance.

2. On shift personnel failed to undergo a thorough review of the l sequence prior to its use. This action is not required by station _

procedures, yet it is more than likely that the event could have been prevented had it been done. Lack of this review did not constitute good operator practice.

In addition to the root causes described above, the following additional l

factors contributed to the occurrence:

l 5

1. DGP 3-4 did oot provide sufficient control over the issuance of new copies of the sequence package. As a result of this, symbols ,

and other ways of denoting that a step in the sequence had been performed were not consistent. Also, the diagonal line through the "OUT" box that was recently being used was also used to mean "not applicable" or an omission in other applications.

2. The Control Rod Sequence Package did not provide a convenient or expedient way of verifying that the actual control rod pattern complied with the desired rod pattern at key points in the sequence.

3. The operating personnel failed to realize the significance of the BPWS break point and Rod Worth Minimizer (RWM) hold point in the context of ensuring they had the proper rod pattern. This is particularly significant in light of the fact that the RWM failed to latch at the proper point during the shutdown.

4. There was insufficient procedural guidance as to how to perform an independent verification of control rod movements in the event of a failure of the RWM. As a result, the independent verifier began without first establishing the proper initial conditions (i .e. , correct rod pattern).

5. The unavailability of the RWM was also a contributing factor.

The RWM was lef t in a degraded mode (no sequence loaded) following testing earlier in the day without the shift being informed. Had shift personnel been informed, the RWM could have been made available for the shutdown and would have resulted in the earlier detection of the event.

CORRECTIVE ACTIONS - SHORT TERM The following items contain the short term corrective actions that were taken prior to Unit 2 startup from hot standby on 1/10/84:

1. Operating Order 23-84 was issued. It requires that : Qualified Nuclear Engineer be present in the Control Room during all control rod evolutions except CRD exercising, emergency conditions, CRAM arrays, or Flow Control Rod maintenance.

2. " Tailgate" training sessions were conducted with the licensed operating personnel from each shift prior to the assumption of their operating responsibilities. This session covered:

~~

f a. the description and communication problems associated with the . improper control rod insertion event

b. the expected operator performance

c. the expected response from the Nuclear Engineering Department l

i 1

l l 6

, , - - - , - - . - - -----------c -

F-

d. the expected actions of persons or groups working on station computers (particularly the RWM), including their responsibi- '

lities to inform the proper operating personnel of equipment status and to conduct their work under proper procedural control s.

3. . Training sessions were_ conducted with the on-site Nuclear Engineers, the Operating Analysis Department, Technical Staff, Computer-

. Engineering personnel and other appropriate personnel on the subject of item 2.d . above.

4. An investigation was begun in accordance with NSDD-A07 as described earlier in this report, and this report is a result of that investigation.

CORRECTIVE ACTIONS - LONG TERM The investigation of this event, and the similiarities observed in the Quad Cities event indicate that even the most experienced and conscientious operators are susceptable to human error, particularly in situations where procedural guidance or instructions are unclear or ambiguous. Since regardless of administrative effort, human error

-cannot be totally precluded, the corrective actions for this event have been developed along two general philosophies. Number one, actions are recommended to minimize the occurrence of misinterpretation and resultant procedure violations. Secondly, actions are recommended to provide for the early detection of errors when they occur.

A third category of corrective actions is provided, relating to pro-

.cedural shortcomings identified during the event. Although these items were not directly related to this event, they represent the potential for similiar misinterpretations or oversights in the future and therefore should be corrected.

Minimize Occurrence of Errors

1. DGP 3-4 should be revised to provide:

- A consistent method of designating that a sequence step has been performed when clean copies or revised sequences are issued. An accompanying legend should be provided, if neces-sa ry. The symbol "X" or a diagonal line are not to be used.

- A caution sheet in the Control Rod Sequence Package requiring .-.

a signoff by an NS0 prior to use of its contents. This sheet would infonn the operator to review and familiarize himself with the entire sequence package prior to use. It would also require that, while performing in sequence control rod ma-neuvers, the status of steps to be skipped should be confirmed by selecting the appropriate rod (s) and verifying their position.

- An index in the sequence package describing its contents.

- See items 1 and 2 of Early Detection of Errors.

See items 1 and 2 of Additional Corrective Actions.

7

2. Station procedure DAP 9-1 should be revised to caution against any unexplained or ambiguous use of symbols in any procedures. ,

Early Detection of Errors

1. Revise DGP 3-4 to provide in the sequence package a convenient means of verifying the proper control rod pattern at key points in the sequence. For example, quarter core maps showing the BPWS breakpoint and the step 98 - RWM loaded rod patterns should be included with a requirement to verify the current rod pattern against them before proceeding. . _.

2. Although 00P 400-2 provides some guidance as to when to bypass the RWM, a procedure review should be initiated to ensure that there is adequate guidance regarding RWM operability. Consider-ation should be given to require contacting a computer technician prior to bypassing the RWM. In addition, the procedure should describe the proper method for indepenslent verification of control rod movement in the event of failure of the RWM. The independent verifier should be required to varify the current rod pattern is in compliance with the sequence prior to resumption of control rod movement. DGP 3-4 should also be revised to reference DOP 400-2 in the sequence package in order to . determine RWM operability and for instructions on independent verification.

3. Establish measures to administratively control work on the RWM or process computer which could render them inoperable or degrade their perfomance. As a minimum, a procedure should be instituted which directs RWM/ computer maintenance and establishes controls such as the caution card system to ensure shif t personnel are kept informed of RWM computer status.

Additional Corrective Actions

1. DGP 3-4 should be revised to require that an independent Nuclear Engineer verify that the completed sequence steps have been properly) transcribed sequence onto a new copy of the sequence (or a revised prior to its issuance.

2. Before a new sequence is distributed, it should be verified that

, this sequence is loaded into the RWM. DGP 3-4 should be revised l' to reflect this.

DGP 3-4 should be revised to require a Qualified Nuclear Engineer

3. ~

i to complete a Control Rod Maneuver Request Form (with the Shift Supervisor's approval) prior to any unplanned changes to the Control Rod Sequence Package.

4. The committee recommends that a task force shall be established

, to investigate control of computer maintenance at all company l s tations .

Finally, all licensed personnel, nuclear engineers, and other affected

departments should receive training on the applicable procedures de--

l scribed above.

t

- - , _ , - _ , , ,e -m - - _ . . - -

CONCLUSIONS This report represents our final findings and recommendations regarding the Dresden 2 out-of-sequence shutdown on January 9,1084. With the exception of the procedure violation resulting from inadvertently skipping steps in the sequence, all other aspects of the control rod manipulations and shutdown evolution were in complicnce with station procedures and good operating practices. The NS0's involved were found to be knowledgeable and sufficiently trained on control rod sequencing and were well aware of the recent Quad Cities event. No evidence of negligence was found, the operators were guilty only of human error under circumstances which fostered misinterpretation. Procedural deficiencies have been identified and addressed under corrective actions.

We recommend that all corrective actions be implemented by April 1, 1984. Operating Order #23-84 should remain in affect until the recommended revision to DGP 3-4 are completed. At that time the Operating Order should be cancelled.

N - .\ %N 6. =-~

R. M. Ragan, C nnan 1. E. Burns i mA- J. R. Wbjnarowski dl1 L:

J. G.~Uremovic h.\ h DQJ.f)elson Approved: / (.h t 1 p.J. Scott GlidW DR O. P.~Galle t

9

,