ML13322B006
| ML13322B006 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 06/30/1989 |
| From: | Schumacher R Southern California Edison Co |
| To: | |
| Shared Package | |
| ML13322B005 | List: |
| References | |
| NUDOCS 8907060193 | |
| Download: ML13322B006 (18) | |
Text
CASE STUDY INITIAL STARTUP WITH NEW NUCLEAR INSTRUMENTATION SYSTEM SONGS UNIT 1 JUNE 1989 Prepared by Robert F. Schumacher June 20, 1989 890706.0193 89062:3 PDR ADOCK 05000206 P
EXECUTIVE
SUMMARY
On May 16, 1989, the initial Cycle 10 startup of SONGS Unit 1 was halted when electrical noise ("noise") was encountered on the new Nuclear Instrumentation System (NIS).
The new NIS had been installed during the refueling outage, and noise had occurred on the system during various test phases. When noise was encountered during the
- startup, there were no design engineering representatives on site, and the operations personnel were unsure of the acceptability of the noise or the likelihood of it leading to an unanticipated reactor trip. A decision ultimately was made to insert control rods and delay the startup until engineering representatives could review and evaluate the noise and recommend a course of action.
A case study was commissioned to review the events surrounding the attempted startup of May 16.
The results of the case study indicate that:
(1)
Preceding May 16 there was a lack of thorough feedback from the design organization to operations personnel regarding determination of acceptable levels of noise observed on the NIS and the magnitude of noise which would be necessary to cause an unwarranted High Startup Rate trip; (2) There was a lack of awareness outside of the design engineering organization that design engineers intended to be present for the startup, and that they expected to be notified when the startup began; (3) There is a lack of a uniform understanding in the NE&C and NGS organizations of the roles and responsibilities of the various groups within those organizations.
Recommendations resulting from the case study are:
(1) Increased procedural emphasis of the expanded role and responsibilities of the design organization (NEDO) to assure appropriate degrees of involvement; (2) Training and indoctrination of personnel within the NES&L and NGS organizations to improve awareness of responsibilities and improve communication between organizations; (3) Review of post-modification testing associated with Unit 1 Cycle 10 design modifications to assure thorough testing and feedback of results to operations personnel as appropriate.
(4) Continued development of integrated jurisdiction statements of the various NES&L and NGS organizations; (5) Training in and continued reinforcement of personnel of the NES&L and NGS organizations regarding the jurisdictions and responsibilities of the various organizations.
CASE STUDY INITIAL STARTUP WITH NEW NUCLEAR INSTRUMENTATION SYSTEM SONGS UNIT 1 JUNE 1989 Introduction Case Study Approach Description of Organization Summary of Events Problems and Contributing Factors Conclusions Recommendations Appendices A
Personnel Contacted During Case Study
CASE STUDY INITIAL STARTUP WITH NEW NUCLEAR INSTRUMENTATION SYSTEM SONGS UNIT 1 JUNE 1989 INTRODUCTION One of the major activities during the SONGS Unit 1 Cycle 10 refueling outage was a
full replacement of the Nuclear Instrumentation System (NIS).
Electrical noise (referred to as noise) had been encountered during testing of the system, and an extensive noise monitoring and reduction effort. had been put forth.
During control rod withdrawals associated with rod drop testing, an unanticipated reactor trip had occurred as a result of noise on the system.
Subsequent noise suppression efforts were felt to be effective as verified by additional testing.
On May 16, 1989, preparations for Mode 2 were complete and the approach to criticality was begun.
During the control rod withdrawals noise was again observed.
There was concern among those present in the control room regarding the magnitude of the noise and the recognized potential for an unanticipated reactor trip.
There was no design engineer support on-site, nd no specific guidance regarding "acceptable" levels of noise had been provided.
The critical approach was terminated (control banks inserted), and additional personnel (including design engineers) were notified.
A case study was' commissioned to review the events and cir cumstances surrounding the approach to criticality, and to identify problem areas.
This report presents the findings of that study and offers recommendations to preclude a future similar occurrence.
CASE STUDY APPROACH During the performance of this case study the organizational structure and the responsibilities and interfaces of the various departments were reviewed.
Emphasis was placed upon how these departments interact and communicate, particularly for the case of the NIS modifications.
The majority of the case study was performed via interviews and discussions with persons from the various organizations who were directly involved in the NIS related design, installation, and testing activities, and from those involved in the critical approach which occurred on May 16.
Log books and records were also reviewed to establish event sequences, and meetings and
working relationships were observed.
Appendix A includes a listing of persons who were contacted during this case study and from whom input was obtained.
DESCRIPTION OF ORGANIZATION Figure 1 shows the Nuclear Engineering, Safety & Licensing (NES&L) organization which includes the Nuclear Engineering and Construction division. Figure 2 shows a further breakdown of the Nuclear Engineering and Construction (NE&C) division which includes the Design Engineering (NEDO),
Site Engineering (SNE),
and Nuclear Construction (NC) departments.
Within the Nuclear Engineering and Construction (NE&C) division, responsibilities are generally divided as follows.
(1) NEDO is responsible for maintenance of the design and design bases of the plants and for development of design changes. This organization is currently located at the General Offices at Rosemead.
(2) Nuclear Construction is responsible for implementation of design modifications into the plant, performing any required testing, and turnover of the modifications to the Station organization.
Nuclear Construction is located at the San Onofre site.
(3) Site Engineering (located at the San Onofre site) serves as the focal point for resolution of engineering problems which arise during the implementation and testing phase.
Site Engineering normally serves as an on-site link between NEDO and Nuclear Contruction.
Within the NGS (Station) organization the Station Technical department is a counterpart of NEDO and Site Engineering, and technical cognizance of the plant design resides within this department.
The Outage Management Department (OMD) fulfills a
planning and scheduling function during plant outages to coordinate activities of the various organizations (both Station and NES&L).
OMD is the focal point for outage related ac tivities, and obtains schedule and activity input from the other organizations.
During an outage, daily OMD meetings are held (generally morning and afternoon) at which the current schedule, activities, and critical path are identified.
Included among the participants at these meetings are Station Operations, Station Technical, and Nuclear Construction. Site Engineering (NE&C) and NEDO are generally not directly represented at these meetings, Nuclear Contruction serves as the NE&C representative.
The Nuclear Construction department also holds daily meetings during an outage.
These meetings are attended by representa-
50 tives of Nuclear Construction and Site Engineering, and are a means by which these two departments status and coordinate their activities.
Nuclear Construction is generally responsible for assuring proper feedback of information between these meetings and the OMD meetings.
With regard to the NIS design modification activities, the interactions between the various NE&C departments differed somewhat from the above description.
Due to the magnitude and complexity of the design change and the amount of design engineering involvement anticipated throughout the installation and testing phase, a NIS design team was established at the San Onofre site. This team was headed by the Unit 1 Project Engineer and included design engineers (all from NEDO).
The NIS team interacted directly with Nuclear Construction to resolve problems encountered in the field.
Subsequent involvement of Site Engineering in the NIS design modification work was minimal.
Separate daily meetings (in addition to the daily OMD meetings and the daily Nuclear Construction meetings) were held to discuss
- status, schedule, and activities associated with the NIS work.
Attendance at these meetings generally included representatives of the NIS team, Nuclear Construction, and Station Technical.
With the creation of these daily NIS meetings, the NIS work was not covered in detail at the daily Nuclear Construction meetings.
Nuclear Construction was still the link between NE&C and Station for NIS related activities since they participated in both the daily NIS meetings and the daily OMD meetings.
SUMMARY
OF EVENTS The following is a summary of events and situations preceding and during the attempted approach to criticality on May 16.
(1) Due to the anticipated level of design engineering involve ment during installation and testing of the NIS modifica tion, design personnel were placed on-site for the duration of the outage.
These design engineers worked closely with the retrofit engineers from the Nuclear Construction department to investigate and resolve problems, a number of which were related to noise. The noise was most apparent on the Intermediate Range power channels (wide range log channels).
(2) The Intermediate Range (IR) power channels include a High Startup Rate (Hi SUR) trip. To reduce the susceptibility of the system to unwarranted SUR trips caused by noise, the Hi SUR circuit includes a feature to disable the trip below a pre-established lower limit of 1.OE-4 % power (as indicated by the IR channels).
If a SUR of 5 DPM is calculated by an IR channel while that IR channel indicates greater than 1.OE-4 % power, a Hi SUR trip will occur.
(3)
A noise monitoring program had been established as part of the NIS testing. This program was to remain in effect until the plant had attained full power and the NIS had been verified to be operating satisfactorily with no adverse effects from the presence of noise.
The Nuclear Construc tion department is responsibile for post-modification
- testing, and full shift coverage by retrofit engineers and technicians was in effect.
The design engineers were involved in evaluating the noise and developing solutions.
(4) Arrangements had been made by the design organization to have a vendor representative here for the initial startup of the unit.
(5) Several days prior to the approach to criticality, while Control Rods were being withdrawn for rod drop testing, noise was observed on the IR channels.
The noise spikes observed on the two IR channels were l.OE-3 and 5.OE-4 %
- power, respectively, and were sufficiently large to result in generation of a High Startup Rate trip, leading to an unanticipated reactor trip.
A Noncomformance Report (NCR) was generated, potential sources of noise were investigated and identified, and noise suppression modifications were implemented.
The disposition was documented on the NCR.
(6) Preceeding May 16, there was a general impression among people on site that the critical approach would be started on a day shift or early on swing shift (start by 5:30 or 6:00 pm or wait until the next day).
This is consistent with information which was provided at the daily OND meetings.
(7) By May 16, all major Mode 2 restrictions had been cleared, including concurrence from the NRC on several items, and preparations for the approach to criticality were being finalized.
While no specific time for criticality was
- provided, it was indicated at the afternoon OMD meeting that a tailboard meeting for operations would occur at 4:30 pm in the Control Room and that criticality was expected to follow sometime thereafter on swing shift.
(A similar tailboard meeting had been held at 9:30 am for the day-shift operations personnel.)
The plan to start the critical approach on swing shift was reconfirmed at the 4:30 pm tailboard meeting.
NE&C was represented at the tailboard meetings by the day-shift duty Retrofit engineers who were recognized as being on shift to monitor for noise on the NIS during the startup and provide related support as required.
Neither design engineers nor the vendor representative were present at the tailboard meeting.
(8) Management personnel from Nuclear Construction and design engineering personnel had. left site for the day by ap proximately 5:30 pm.
The vendor representative returned to his hotel room after the 4:30 pm tailboard meeting, thinking
that he would be called by the shift Retrofit engineer if and when the critical approach started.
This was contrary to the understanding of the Retrofit engineer.
(9) The approach to criticality began at approximately 8:00 pm on May 16 with Control Bank 1 withdrawal.
Among those present in the control room were Mr. McCarthy (Vice President & Site Manager), Mr. Krieger (Operations Manager),
and Mr. Waldo (Technical Manager).
This was in addition to the normal operations staff and the Core Analysis engineers who were in charge of the physics testing.
The only representative from NE&C was the shift retrofit engineer who was present primarily for the noise monitoring program.
No significant noise was observed during withdrawal of Control Bank 1, and withdrawal of Control Bank 2 was started. During withdrawal of Control Bank 2, noise (in the range of 3.E-5
% power) was observed on the intermediate range power indicators.
Rod withdrawal was temporarily suspended.
The rod withdrawal was subsequently continued one step at a time.
After several steps, additional noise spikes (in the range of 3.E-5 to 5.E-5 % power) were observed. Rod withdrawals were again stopped due to concern over the magnitude of the noise spikes, the associated fluctuations in the SUR meters, and the recognized potential for a Hi SUR trip.
There was no pre-established criteria for judging "accep tability" of the noise, and since there was not a design engineer present, a telephone call was placed to Mr.
Nunn (approximately 10:30 to 11:00 pm).
After speaking with the Retrofit engineer, Mr.
Nunn indicated he would call back with additional information.
Mr. Nunn then initiated a conference call with personnel in his
- division, and subsequently called the Control Room with a recommendation that the startup not proceed until additional personnel arrived on site.
A concurrent call had been placed from the Control Room to Mr. Morgan informing him of the situation and that a recommendation from engineering was forthcoming.
During this call, it was decided to insert the Control Banks. When Mr. Nunn called back to present the recommendation of NE&C, the startup had been terminated.
(10) The noise observed during the aborted startup of May 16 was documented by NCR.
Testing resulted in additional noise suppression modification, and the NCR included quantitative criteria for "acceptable" levels of noise during retesting.
The NCR also included a memorandum issued by the design organization on May 20 specifying "acceptable" noise spike levels on the IR channels during subsequent startups.
(11) Initial criticality of the Unit occurred on May 21.
Some
noise was observed on the IR channels, but it was below the criteria provided in the memorandum.
Design engineers and the vendor representative were present for the critical approach.
PROBLEMS AND CONTRIBUTING FACTORS Two specific problems have been identified in connection with the NIS related circumstances and events leading up to the aborted critical approach of May 16.
A number of factors contributed to those problems.
(1) Although this was the initial startup of the unit following major modifications to a system with safety and operational implications, and although significant noise problems had been encountered, there was no design engineering support in the control room or on-site.
The only representative from the NE&C organization was a shift retrofit engineer, who was present to monitor for noise.
(a) While a decision had been made within the design engineering organization to provide coverage during the approach to criticality and the subsequent initial power increases, this intent was not effectively communicated to other organizations.
This was despite the fact that:
The design engineers worked with Retrofit closely on a daily basis and a Retrofit engineer was on shift; The design engineers and management representa tives from Nuclear Construction met daily to discuss the NIS work; Nuclear Construction attended the daily OMD meetings at which activities and schedule were discussed.
(b) There was a mindset about criticality occurring on day shift.
On May 16, awareness of the plan for critical ity later on swing shift apparently did not reach the design engineers or Nuclear Construction management.
This was despite the fact that:
It was at least raised as a possibility at the afternoon OMD meeting; It was discussed at the 4:30 pm tailboard meeting at which Nuclear Construction was represented.
(c) The Station cognizant NIS engineer (counterpart of design engineer) was not present for the critical
approach and was not contacted when "significant" noise was first observed.
(d) There was apparently no specific awareness, by the people present at the critical approach that design engineering support was not present.
If there was such awareness, apparently no special significance was attached to it.
(e) There was no attempt to contact design engineering support when the initial "significant" noise was encountered, although Nuclear Construction was represented in the control room.
The first call was not made until the second suspension of rod withdrawal, and then it was made at a high level of management rather than by technical personnel.
(f)
There was no established criteria of whether design engineering support should be available during critical plant evolutions following major design change activities and/or what role design engineering should play at such times.
(2) Noise had been observed during testing of the new NIS and during heatup of the Unit to Mode 3 in preparation for criticality.
Noise sufficient to cause an unanticipated reactor trip had occurred several days before the attempted startup during rod withdrawals associated with rod drop testing.
During the May 16 startup,
- however, operations personnel were not equipped to confidently evaluate the significance of the noise which was observed.
While the engineering organization was familiar with the noise which had been seen throughout the test
- program, detailed information in this area and criteria for evaluating the significance of noise had not been provided to operations personnel.
Information was later provided by the May 20 memorandum.
The above factors contributed directly to the terminated critical approach on May 16.
On an individual basis, none of the factors should have led to the necessity to terminate the critical approach.
- However, the aggregation of the items was sufficient to do so, and seems to indicate a more general problem.
There are three specific observations which support this.
The design engineering organization (NEDO) has recently undergone a
significant reorganization, and great emphasis has been placed upon what is expected from NEDO personnel in the areas of quality of work, ownership of design, and interaction with Station personnel.
However, there is a lack of uniform understanding among other organizations, particularly
by working level personnel, of the NEDO organization or of the charter which they have been presented.
Most of the persons interviewed were asked whether or not they would have expected the design organization to have been present in the control room during the initial startup with the new NIS system, particularly considering the noise problems which had been en countered.
Almost all of the responses were negative, including those from members of the Nuclear Construc tion department.
All of the responses from members of the design organization (NEDO) were of a positive nature.
During the attempted startup on May 16, the absence of design engineering support (again particularly in light of the noise which had previously been observed) was not noted or not considered significant. When problems were first encountered, no attempt was made to contact the design engineers or the Station cognizant engineer.
These observations, particularly when combined with the aggregate of specific factors, indicate lack of a uniform perception of roles throughout various organizations.
CONCLUSIONS The findings of this case study can be summarized as follows.
(1) Preceding May 16 there was a lack of thorough feedback from the design organization to operations personnel regarding evaluation of acceptable noise on the NIS and the magnitude of noise which would be necessary to cause an unwarranted High Startup Rate trip.
This is reflected in problem item (2).
Problem area 1(f) also falls somewhat into this category; thorough and effective feedback between the design organization and operations personnel throughout and following design changes should be recognized and expected.
(2) There was a lack of awareness outside of the design engineering organization that design engineers intended to be present for the startup, and that they expected to be notified when the startup began.
This is a case of lack of effective communication and is reflected in problem items 1(a) and 1(b).
It is especially difficult to understand how the intent to have design engineering coverage and provisions for them to be called was never clearly established between
Design Engineering and Nuclear Construction. These two groups are in the same organization, worked together on a daily basis for months, and jointly attended daily NIS meetings with management personnel from both groups present.
(3) There is a lack of a uniform understanding throughout the NE&C and NGS organizations of the roles and responsibilities of the various groups within those organizations.
This is reflected in problem items 1(c),
1(d),
and 1(e), and, to some extent, item 1(f).
It should be noted that despite the difficulties encountered during the critical approach, the significance of the initial startup with a new NIS was recognized throughout the organization and emphasis was placed upon adequate planning and preparation.
This included planning to assure the best possible initial alignment of the system, including compensation for the sen sitivity differences between the old and new detectors (which are not of the same type).
Thorough monitoring of the power instrumentation was proceduralized and performed during the startup and at low power operation to assure that the indications did not result in non-conservative operation of the plant.
The presence of the design engineers on site throughout the installa tion and testing of the system was a definite benefit.
These things should be recognized and commended.
RECOMMENDATIONS It is apparent that problems existed which led to the difficul ties encountered during the attempted startup on May 16.
Until corrected, these problems will continue to exist and a similar occurrence could happen again. The following recommendations are intended to help resolve those problems and prevent such a reoccurence.
- 1.
The new roles and responsibilities which have recently been established for the design engineering organization (NEDO) need to be reinforced by reflection in the appropriate procedures.
- a.
Explicit emphasis should be placed upon increased awareness by the System Design Engineer (SDE) through out the implementation and testing phases of the modification process, including, for appropriate cases, direct involvement in such evolutions.
- b.
More emphasis should be placed on post-modification
system testing which is described in the DCP procedur es.
The procedures should address thorough and complete identification of all testing required to assure that the modified system responds as expected under all applicable operating configurations and conditions and through all operating modes up to and including full power operation.
C.
Explicit provisions should be included to assure adequate feedback to operations personnel regarding design modifications.
This is particularly true for problems encountered during installation and testing activities which may occur after formal training has been completed.
- d.
The role of the SDE should be expanded in the areas of review and approval of test procedures and review and approval of test results.
There is presently a provision for engineering involvement, but no require ment for the SDE specifically to be involved.
Since the SDE provides the original test guidelines and acceptance criteria, his participation in the review and approval of the procedures and results is logical.
- 2.
Personnel outside of NEDO need to receive training and indoctrination regarding the roles and responsibilities which have been established for NEDO (and the SDE) and how it potentially affects them and/or the job they do.
While all personnel in the NES&L and NGS should ultimately receive such indoctrination, the following groups should receive priority:
- a.
Nuclear Construction,
- b.
Station Outage Management,
- c.
Station Technical,
- d.
Station operations.
- 3.
Personnel within NEDO and Nuclear Construction should be indoctrinated on the responsibilities (shared and unshared) of the two groups in the design modification process and the importance of effective communication with each other and with the Station organization, particularly with OMD, Opera tions, and Technical.
- 4.
Considering the problems recently encountered with the wide range steam generator level modification in addition to the NIS related problems, strong consideration should be given to a review of testing associated with all design modifica tions for Unit 1, Cycle 10.
Such review should verify that the specified testing covered all anticipated operating conditions and configurations through all operating modes as appropriate, that all results are as expected, and that any necessary feedback to operations personnel occurred.
As a
- minimum, modifications on safety related systems or those
which may impact performance of safety related systems should be reviewed.
- 5.
The responsibilities of the various groups within the NES&L and NGS organizations should be more completely integrated and jurisdictions established.
These jurisdictions should allow for and require a level of involvement by the design organization in continued operation of the units beyond turnover of system design modifications.
- 6.
Following establishment of the above jurisdictions and responsibilities, this information should be disseminated to personnel in both organizations via training. This training must be recognized as part of a culture change, and, to be effective, must be continually reinforced by the actions of management to support the established jurisdictions and responsibilities.
APPENDIX A PERSONNEL CONTACTED DURING THE INTERVIEW PROCESS Person Contacted Organization D.E. Nunn, Jr.
Manager, NE&C J.J. Wambold Project Manager M.L. Merlo Manager, NEDO K.C. O'Conner Manager, Nuclear Construction C.K. Balog Manager, Site Engineering G.J. Stawniczy Project Engineer O.A. Hollaway Supervisor, Retrofit T.R. Elkins Supervisor, Electrical and I&C D.E. Frey Supervisor, NEDO Planning
& Scheduling A.A. Hernandez Lead NIS Design Engineer R.
Farias Retrofit Engineer J.
Murray Retrofit Engineer L.
Porter Retrofit Engineer L.
Greenberg Westinghouse Representative C.B '. McCarthy, Jr.
Vice President & Site Manager H.E. Morgan, Jr.
Plant Manager R.W. Krieger, Jr.
Operations Manager R.W. Waldo Station Technical Manager A.J. Schramm Superintendent, Unit r Operations K.L. Johnson Supervisor, NSSS Engineering D.A. Niebrugge Supervisor, NSSS Electrical J.M. Joy Supervisor, Unit 1 Outage Mgmt.
S.J. Hetrick Supervisor, Computer Engineering M.J. McDevitt' Supervisor, Core Analysis Engr.
A.J. Eckart Core Analysis Engineer S.C. Swoope Core Analysis Engineer D.J. Ramendick Core Analysis Engineer J.S. Iyer Shift Technical Advisor
Nuclear Engineering, Safety and Licensing Kenneth P. Baskin Vice President Nuclear Engineering, Safety and Licensing M. McCreery R. M. Rosenblum D. E. Shull D. E. Nunn J. J. Wambold Manager of Manager Nuclear Manager of Manager of Project NES&L Regulatory Nuclear Nuclear Eng. &
Manager Services Affairs Oversight Construction Figure 1
NUCLEAR ENGINEERING AND CONSTRUCTION MANAGER D. E. NUNN TECHNICAL CONSULTANT CETARY D. F. PILMER S. MIYASATO SNUCLEAR NUCLA NUCLEAR ENGRG.
DESIGN BASES ENGINEERING CONSTRUCT)N FUEL DESIGN DOCUMENTATION C. K. BALOG K. E. C CONNOR P. D. MYERS M. L. MERLO M. P. SHORT Figure 2