ML20059N019
| ML20059N019 | |
| Person / Time | |
|---|---|
| Site: | Quad Cities  |
| Issue date: | 11/12/1993 |
| From: | Beach A, Jordan E NRC OFFICE FOR ANALYSIS & EVALUATION OF OPERATIONAL DATA (AEOD) |
| To: | |
| Shared Package | |
| ML20059M955 | List: |
| References | |
| NUDOCS 9311300059 | |
| Download: ML20059N019 (77) | |
Text
. . . . - . ._ .
-]
j l
j b
1 y DIAGNOSTIC EVALUATION :
TEAM REPORT
'l ON s
QUAD CITIES NUCLEAR POWER- ,
STATION.
1 3
(August 23 - September 3,1993 o
.and September 20-30,1993)l 1
.\
U.S. -Nuclear Regulatory Comission . !
Office for Analysis and Evaluation of Operational Data Division of Operatior.al Assessment :
. ?
Diagnostic Evaluation and Incident Investigation Branch -
r 9311300059 931117 i PDR ADDCK 05000254 p PD8; ,
r
OFFICE FOR ANALYSIS AND EVALUATION OF OPERATIONAL DATA DIVISION 3F OPERATIONAL ASSESSMENT Licensee: Commonwealth Edison Company Facility: Quad Cities Nuclear Power Station, Units 1 and 2 Location: Quad Cities Site, County of Rock Island State of Illinois Docket Nos: 50-254, Unit 1 i 50-265, Unit 2 Evaluation Period: August 23 - September 3, 1993, and September 20-30, 1993 Team Manager: A. Bill Beach Administrative Assistant: Michelle Smith Team Leaders: Henry Bailey Elmo Collins Thomas Foley William Raughley Team Members: Ronald Lloyd Alan Madison Anthony Mendiola Melanie Miller (September 20-24 1993 only)
Peter Prescott Melvin Shannon John Thompson Leonard Wert Contractors: Russell Brown John Darby Whitney Hansen David Schult :
1 Submitted By: ; ,al J ll/M%
A.fillBeach',~TeamManager Date Quad Cities Diagnostic Evaluation Team Approved By:
Dordan, Director
///I f.3 Dhte' Edward Office r Analysis and Evaluation Operational Data
CONTENTS PAGE Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Residual Heat Removal System Figure . . . . . . . . . . . . . . . . . . . vii 1 Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii j
1.0 INTRODUCTION
............................ I 1.1 Background .......................... I 1.2 Scope and Objectives ..................... 2 1.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 F:cility Description ..................... 2 1.5 Organization ......................... 2 2.0 EVALUATION RESULTS ......................... -5 2.1 Operations and Training . . . . . . . . . . . . . . . . . . . . 5 l
2.1.1 Acceptance and Limited Awareness of Equipment Degradation ..................... 5 2.1.2 Operability of Degraded Equipment Frequently Not Evaluated .................... 7 2.1.3 Observed Procedural Deficiencies . . . . . . . . . . . . 9 2.1.4 Decreased Sensitivity to Control Room Annunciators . . 12 2.1.5 Limited Oversight of Control Room Activities During Busy Periods . . . . . . . . . . . . . . . . . . . .
112 2.1.6 Strong Overall Operator Training . . . . . .. . . . . . 13 2.1.7 Positive Observations ................14 2.2 Maintenance and Testing . . . . . . . . . . . . . . . . . . . .
14 2.2.1 Failure to Fix the Root Cause of Known Testing Deficiencies . . . . . . . . . . . . . . . . . . . . . 15 2.2.2 Observed Weaknesses in Maintenance Implementation . . 18 2.2.3 Failure to Fix the Root Cause of Motor-Operated Valve Deficiencies . . . . . . . . . . . . . . . . . 20 2.2.4 Ineffective Maintenance Work Process Resulted in the f ailure to Get Equipment Fixed . . . . . . . . -. . . 22 2.2.5 Support to Maintenance Was Not Sufficient to Maintain an Effective Program . . . . . . . . . .-. . . . . . 24' 2.2.6 Preventive Maintenance Program Not Effectively Impl emented . . . . . . . . . . . . . . . . . . . . . 25 2.2.7 Positive Observations . . . . . . . . . . . . . . . . 25 f
2.3 Engineering and . Technical Support . . . . . . . . . . . . . . 26 2.3.1 RHR Equipment Problems Resulted in System Degradation . 26 2.3.2 Engineering Failure to Effectively Address Plant Vibration Problems . . . . . . . . . . . . . . . . . 27 iii
4 . 1 a
2.3.3 Engineering Failure to Fully Evaluate Equipment [
Design Issue . . . . . . . . . . . . . . . . . . . . 28-2.3.4 Site Engineering Did Not Always Support the Plant . . 30 2.3.5 Corporate Engineering Was Not Sufficiently Involved in Plant Issues . . . . . . . . . . . ... . . . . . 33 2.3.6 Modification Implementation Was Often Untimely .
and Incompl ete . . . . . . . . . . . . . . . . . . . 36 2.3.7 Ceramic Fill and Coating Compounds Not Controlled. . . 37 2.3.8 Positive Observations ................38 ;
2.4 Management and Organization . . . . . . . . . . . . . . . . . 38 2.4.1 Untimely and Ineffective Corrective Action Processes . 39 2.4.2 Ineffective Quality Oversight . . . . . . . . . . . . 41-2.4.3 Corporate Management Failed to Ensure that lessons Learned from Dresden and Zion Were Implemented . . . 42 2.4.4 Corporate Improvement Programs Did Not Correct' Declining Performance ....................43 2.4.5 Site Management Was Not Effective ..........45 1 2.4.6 A Strong Staff . . . . . . . . . . . . . . . . . . . . 48 s 2.4.7 Positive Observations . . . . . . . . . . . . . . . . . 49 ,
3.0 ROOT CAUSES . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.1 Ineffective Corporate Leadership, Oversight,: Involvement, and Foll ow-Through . . . . . . . . . . . . . . . . . . . . . 50 3.2 Site Management's Failure to Resolve Identified Safety ,
P robl ems . . . . . . . . . . . . . . . . . . . . . . . . . . 50 ;
3.3 Low Standards of Performance . . . . . . . . . . . . . . . . . 51 ;
3.4 Site Management's Failure to Exercise Effective Leadership . . 51 4.0 EXIT MEETING . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 -
Appendix A - Exit Presentation f
i iv
ABBREVIATIONS f AC alternating current ADS automatic depressurization system .
AEOD Office for Analysis and Evaluation of Operational Data ASME American Society of Mechanical Engineers j BWR boiling-water reactor ,
CAMS containment atmosphere monitoring system CAR Corrective Action Record l CCST closed cooling water storage tank Ceco Commonwealth Edison Company CFR Code of Federal Regulations CM corrective maintenance CR control room CS core spray CWP cooling water pump DBD design-basis documentation DC direct current DE diagnostic evaluation DET Diagnostic Evaluation Team DGCW diesel generator cooling water DP differential pressure DR discrepancy record DVR deviation report EA enforcement action EAT Event Assessment Team EDG emergency diesel generator EDO Executive Director for Operations EDSFI electrical distribution system functional inspection ERV electromatic relief valve GE General Electric GL generic letter HPCI high-pressure coolant injection IN information notice IPE individual plant examination ISEG independent safety. engineering group ISI inservice inspection IST inservice testing >
.LER licensee event. report LLRT local leak rate testing LOCA loss-of-coolant accident- 4 LPCI low-pressure coolant injection MHE maintenance history evaluation MIS management information system-MOV motor-operated valve i MSL main steam line v
I
'NOC Nuclear Oversight Committee :
N0D Nuclear 0perating Division NRC Nuclear Regulatory Commission NRR Office of Nuclear Reactor Regulation i NSRB Nuclear Safety Review Board ;
NTS nuclear tracking system NWR nuclear work request ,
OER operating experience review ONS onsite nuclear safety PEG Process Experts Group PEP Performance Enhancement Program t PIF problem identification form PM preventive maintenance '
PRA probabilistic risk assessment QAP Quality Assurance Program QCAP Quad Cities Administrative Procedure .;
RB reactor building RCIC reactor core isolation cooling RCM reliability-centered maintenance REP recurrent equipment problem RHR residual heat removal RHRSW residual heat removal service water i RWCU reactor water cleanup SALP Systematic Assessment'of Licensee Performance SBLC standby liquid control 'i SCRE shift control room engineer
-SE shift engineer SEP Systematic Evaluation Program SESR site engineering service request SIL services information letter
, SPC suppression pool cooling SQV site quality. verification i TS Technical Specifications UFSAR Updated Final Safety Analysis Report VAT Vulnerability Assessment Team VOTES valve operation test evaluation system VP vice president Vice President and Chief Nuclear Officer VPCN0 VPN0 Vice President of Nuclear Operations vi
U 4
FROM FIRE s La La PR OTECTION ' r ,
,rg ,,
@e- oes.a -
! ILMT'IEC '
E E a __d, *m v
, TETUI I
rL.,J Ll2 ,
r, s tl2 tl; 288 235 FROM ECCS l [3} {,' I IC E E P FILL 3Q d TO R E CIR C QQ ! ot,,
1137A k>
13781
~ '
{ i 28A 29Af \293 288 , ,,
3'A SHUTDOWN 348 $ l
, HEADER , 333 hh1 h a 37A V pg..
375 hh CZ n 284 _-
I' i k '
,h k
.a pg, mm
= c E cHL.A l ECCS SUCTION HEADER i .. p3 E;
}
E l E
- inA tk I' J_ ;O?t"
- k Tl T-1 FROM CCST*3
- p. I-g
!2i I-E Q re.t.:.t.
>>=
TO FU L POOL 41 7A 73 7C 7D 4 2,A ya ;' 4,2,0
- 42 El 42C 3 T 43A 438143C 4 oT
r, a,
r U
S U S V S -
U N
A R C 0 m _ _ _ _
n
& NM-
-MP3- -NM-2* Z
~
E~ Z~
Nai GEB Z Z Z Z8 . 8 l
17Al l iga 18A 133155 l l 175 1
FIGURE 1000 1 R EY. 2 RHR SYSTEM vii
1 EXECUTIVE
SUMMARY
From August 23 - September 30, 1993, a diagnostic evaluation team from the U.S. Nuclear Regulatory Commission (NRC) evaluated the performance of Commonwealth Edison Company in ensuring safe operations of the Quad Cities Nuclear Power Station. The evaluation was requested by the NRC Executive Director for Operations in order to obtain information needed to make an adequately informed decision on overall performance at Quad Cities. The team of 15 evaluators (plus an additional evaluator for the last week onsite only) was led by a NRC inanF.ger during the 6-week evaluation. Areas evaluated included operations sad training, maintenance and testing, engineering and technical support, and management and organization. Both units were_ operating throughout most of the evaluation period.
In the area of operations, personnel were challenged by the large number of equipment problems. Although the Operations staff had repeatedly expressed frustration with equipment problems and the efforts needed to get equipment fixed, this situation eventually fostered an attitude of accepting or "living with" equipment problems. Repetitive equipment failures continued to contribute to a neavy operational workload. Site management had little central focus or comprehensive effort to address equipment performance problems. The team identified seven1 problems that had gone undetected by operations staff. These included Class 1E electrical cabinets associated with the Unit 2 emergency diesel generator that were not anchored and a reactor core isolation cooling steam admission valve that had sheared bolts. _ The licensee later declared this equipment inoperable until it was repaired. The team noted that the' licensee had recently made repairs to some equipment, and had scheduled maintenance outages to repair, test, and further evaluate existing equipment problems.
The Operations Department did not show ownership in the. plant and was not always involved in evaluating the operability of degraded equipment. Although a process had been established to evaluate degraded conditions, it was _ not appropriately implemented. The team identified several significant degraded conditions that, although known by the licensee for some period of time, had not received a rigorous operability review. Feedwater flow nozzle inaccuracies, identified in November 1992, leading to the possibility that the reactor was operating above the licensed power level, had not been evaluated.
(After identification by the team, the licensee restricted power operation as a compensatory measure for this deficiency.) Reactor coolant pressure bouridary valves had failed inservice testing in April 1993, yet the licensee had not evaluated the operability of the valves.
Operations management had not established appropriate standards'for control room annunciator operating procedures. Operators did not vigorously pursue procedure revisions that were necessary to reflect the attual operation of the systems. Some weaknesses in the implementation of Technical Specification requirements were identified. The team observed operators not following-procedures on several occasions, such as; when entering shutdown cool.ing, when responding to hydrogen / oxygen analyzer alarms, and when performing standby-liquid control system surveillance testing. The licensee's Event Assessment Team, and NRC Inspection Reports had historically identified problems regarding procedure adherence, and corrective actions had not been effective.
viii
Even through weaknesses in Operations management were identified, the team found that, except for procedural deficiencies, the operators performed well and that operator training was strong. While oversight by the Shift Control Room Engineer was sometimes limited, the control room operators were confident, had a high level of knowledge, and worked well as a team. The use of a site specific simulator had made significant improvements to overall operator performance.
In the area of maintenance and testing, several safety-related pump and motor inservice test results were above industry acceptance criteria for years, and some pump capacities were inconsistent with design requirements. Some safety-related check valves were not tested in accordance with ASME Section XI requirements, other check valves failed during operation, and appropriate action was not taken. Additionally, the failure rate of residual heat removal '
(RHR) relief valves was unusually high. Maintenance implementation weaknesses, observed by the team, included the unsuccessful repair of a Unit 2 feedwater check valve and the at-power replacement of the wrong drywell sump pump. The causes for those weaknesses included: limited pre-job briefings, limited engineering support, poor communications, and inaccurate drawings. ,
The failure to identify and correct root causes and the acceptance of long.
standing motor-operated valve deficiencies led to a large number of repetitive valve problems including; stretched fasteners, bent stems, motor failures, actuator component failures, and damaged valve seats; many of which were -
associated with containment isolation valves that were not always promptly evaluated or corrected. Some progress had been made in correcting the historically high number of valve deficiencies, but management's failure to correct the root cause of the problems hindered significant improvement.
Recommendations to correcttthe valve deficiencies were provided by internal audits or studies as early'as 1988, but sufficient corrective action. were not taken.
Maintenance management seemed more focused on industry. performance indicator goals, rather than focusing on the level of effort required to reduce the backlog. There were approximately 5,000 open work requests, of which over
, 3,300 were awaiting planning by the maintenance analysts. Only high priority corrective maintenance that directly affected plant operation could be assured of being worked. Many work requests were combined, making the corrective maintenance indicator appear smaller, and many work activities were categorized as preventive maintenance were ectually corrective maintenance.
There were'a number of barriers for the maintenance technicians to overcome.
The maintenance work process was cumbersome and difficult to implement. Work history documentation was confusing, incomplete, and difficult to track. The standardized work packages were almost equivalent to modification packages in size and detail for all jobs, which included as many as 15 signatures before work could begin.
Failure of maintenance management to recognize the need for engineering involvement in the root cause process resulted in a number of inadequate root cause evaluations and repetative equipment failures. Insufficient support was provided to maintenance from other organizations. System engineers were not significantly involved with maintenance activities, and Plant Support engine 1 ring involvement in the maintenance process was not effective.
ix '
P The preventive maintenance! program did not include several recommendations of the (cancelled) reliability centered maintenance program. One recommendation involved the high-pressure soolant injection level switches which, if implemented, may have prevented the June 1993 rupture disk event and -
associated personnel injuries.
Outward plant appearance and housekeeping was good due, in part, to initiatives such as the residual heat removal service water pump room restorations, mechanical pump seal installations, and valve packing improvements. Improvements in maintenance training had also been implemented.
Radiological control practices were also good.
In the area of engineering and technical support, the team evaluated, in-depth, the residual heat removal and residual heat removal service water systems and other significant equipment issues. The residual heat removal system was degraded in several areas largely due to motor-operated valve and vibration problems. A high number of equipment problems were identified but had not been repaired. Similar conditions existed in other systems.
Engineering had failed to effectively address plant-wide vibration problems.
Engineering also had failed to fully evaluate equipment design issues involving potential leakage onto safety-related electrical switchgear; an emergency diesel generator calculated loading deficiency; and non-functioning -
standby liquid control system heat tracing.
Site engineering did not always support the plant, resulting in equipment operability issues not being promptly addressed. Operability assessments and root cause determinations were often weak or nonexistent. Operating experience reviews were not comprehensive. Many corporate and site engineering managers were not sufficiently aware of Quad Cities' problems and showed a lack of ownership and accountability for these problems. Ineic was a lack of aggressive actions on issues identified in the Vulnerability-Assessment Team Report. The design basis documentation program was not-comprehensive, and the draft Individual Plant Examination was not specific to Quad Cities in some cases.
Several safety significant modifications were not implemented in a timely manner. In some cases, some safety reviews were not performed as required.
The quality of some modification and safety reviews were poor. Several work requests included application of a ceramic fill (Belzona ceramic-R) and a ceramic coating (Belzona Ceramic-S) to the interior of pumps and valves.
Safety-related applications of the material received little site engineering evaluation, and the work process was not controlled or monitored.
The licensee was effectively addressing issues identified during residual heat removal heat exchanger testing and the electrical distribution system functional inspection. Effective response to these issues indicated that the necessary technical resources were available to address plant problems.
However, these resources were applied only after the issue had received significant attention from the NRC and/or industry groups.
In the area of management and organization, site management's failure to implement an effective corrective action program in a timely manner significantly contributed to the degraded equipment conditions identified x
A throughout the plant. Corporate improvement programs had not been effective.
Just as corporate management failed to transform lessons learned during the Dresden Diagnostic Evaluation (DE) to the Zion plant, the team found that approximately 25 percent of the lessons learned from the Dresden and Zion des-were still uncorrected at Quad Cities.
The Site Quality Verification (SQV) group had not assessed the collective significance of its findings and was ineffective in elevating identified problems and concerns to the appropriate management levels to ensure adequate resolution. As a result, some of the issues identified by the team had been previously identified by SQV but had not been fully addressed. In addition, corporate munagement had weakened SQV by staffing reductions and redirection of efforts. ,
Organizational instability over the last six years had caused losses of consistency of purpose and focus at Quad Cities that impeded progress on many improvement initiatives, including degraded equipment fixes. Because corporate management failed to recognize the scope and extent of problems, their past actions demonstrated a lack of focus and commitment to provide the' i necessary resources for needed improvement.
Quad Cities management often did not demonstrate an ability to organize, plan, execute, evaluate and resolve issues. The lack of leadership fostered an approach to safety such that equipment was usually repaired only when it failed to function. Site management exhibited little sense of urgency to solve equipment problems and generally accepted a low level of performance.
Site management had abrogated its oversight function and deferred resolution of problems to a committee, group, or program. These programs were usually detailed, elaborate, and comprehensive, and took substantial time to-develop.
The team observed that site management was absent from most problem-solving sessions and other critical activities. Throughout the evaluation, most department managers rarely exhibited awareness _of the plant' problems or assessments affecting safety performance.
The team found a very comprehensive and well thought out plan to improve
, corporate and site performance. However, in the past, the NRC has placed too much confidence in corporate and site initiatives and plans for improvement.
Although, the Vice President and Chief Nuclear Officer's commitment to the plan was forthright and commendable, improvements are needed in plant hardware and existing site management performance. Some equipment problems had been recently fixed, and two short maintenance outages were planned to test, repair, and further evaluate degraded equipment. A new plant manager had been appointed, and assumed responsibilities since the diagnostic evaluation.
The team found the root causes of Quad Cities' performance to be:
(1) ineffective corporate leadership, oversight, involvement, and follow t through; (2) site management's failure to resolve identified safety problems; j (3) low standards of performance; and (4) site management's failure to l exercise effective leadership. !
l xi
1.0 INTRODUCTION
1.1 Background
In. late 1990, because of declining performance, Quad Cities formulated and implemented the Performance Enhancement Program (PEP). In 1991, however,.
performance at Quad Cities continued to decline as evidenced by the large number of significant operational events. In response to these operational events, the licensee conducted a critical assessment. Corrective actions were implemented to address management expectations, administrative controls, and human performance issues. These corrective actions were incorporated into the PEP (later the Management Plan). Systematic Assessment of Licensee ,
Performance (SALP) Report 9 (December 1, 1989, th-ough February 28,1991) identified declining performance in operations and safety assessment / quality verification. In June 1991, Quad Cities was <!1scussed at a meeting of NRC Senior Managers because of the declining perf ormance.
Throughout the later part of 1991, the license 9 believed that performance had improved; however, by 1992, it became evident to the licensee that the frequency of events continued to be inconsistent with the PEP and Management Pl an. The licensee formed the Event Assessment Team (EAT) to look at events that had occurred (January 1991 through April 1992), identify the common causes of the events, and ensure that the Management Plan was appropriately focused. The EAT Report was issued in May 1992 and identified corrective action and the coaching of personnel as significant areas to.be addressed.
The Management Plan was revised to include EAT recommendations. ,
In June 1992, the declining performance trend at Quad Cities was again #
discussed at a meeting of NRC Senior Managers. The licensee's corrective action program was considered to be in its early ' stages of implementation. In August 1992, an NRC management team visited Quad Cities to evaluate the long-term effectiveness of the improvement programs. NRC concluded that '
improvements were being made at Quad Cities. Quad Cities was not discussed-at the January 1993 meeting of NRC Senior Managers because it appeared that the corrective action program was being implemented at an acceptable rate and that the completed improvement program action plans were effective in improving performance.
The performance of licensed activities at selected reactor facilities was discussed at a meeting of NRC Senior Managers in June 1993, and performance at Quad Cities was again discussed. Equipment performance problems of concern included: the high-pressure coolant injection (HPCI)'and reactor core isolation cooling (RCIC) systems' high rates of failure, residual heat removal system heat exchanger fouling, HPCI/RCIC suction check valves' failure to seat, diesel generator cooling water (DGCW) pump failure to start, and degraded bearings in DGCW pumps. Major organizational and personnel changes at the site and within the Commonwealth Edison Company may also have impacted the ability to focus attention on sustaining improvements. From these discussions, the need for additional information to make an adequately informed decision on overall performance at Quad Cities was apparent. The Executive Director for Operations (FD0) directed the staff to obtain this information by conducting a diagrtostic evaluation at Quad Cities.
I
~
e
- 1.2 Scope and Objectives The EDO directed the staff to perform a broadly structured evaluation to assess overall plant operations and the adequacy of the licensee's major programs for supporting safe plant operation. The following goals were set for the diagnostic evaluation: (1) provide information to NRC Senior Management to make a more informed assessment of plant safety performance, ;
(2) determine causes for the significant number of safety system failures, (3) evaluate the effectiveness of engineering, and (4) assess the impact of corporate management on operational safety at the station. ,
1.3 Methodology ,
The diagnostic evaluation team (the team) consisted of 15 technical members _
(plus an additional evaluator for the. final week onsite) and an administrative assistant and was organized with four teain leaders reporting to a team manager. The team devoted several weeks to preparation that included team meetings and briefings by representatives from Region III, the Office of Nuclear Reactor Regulation (NRR), and the Office for Analysis and Evaluation of Operational Data (AE0D). On August 23, 1993, the team began a 2-week evaluation at the facility, including the corporate office. The team returned to the plant on September 20, 1993 for an additional week of evaluation.
During the week of September 27, 1993, several team members conducted interviews in the corporate office. The NRC Resident Inspectors frequently attended team meetings at the site and provided technical advice to the team.
Representatives from the team met daily with their licensee counterparts to discuss team activities and findings.
An in-depth assessment of the residual heat removal system was performed to gain insight into licensee performance of activities such as maintenance, testing, operation, and design control. Special emphasis was placed on identifying the causes of performance problems. The licensee's performance in identifying and correcting its own problems was also assessed.
1.4 Facility Description The Quad Cities Nuclear Power Station is located in the County of Rock Island, State of Illinois. The plant features two General Electric boiling-water reactors and was built by General Electric as the prime contractor. Sargent and Lundy was the architectural engineer. The units were " turn-key" in that they were turned over to the owners after completion of a demonstration of unit operational capacity. The initial operating licenses were issued for Units 1 and 2 on December 14, 1972.
1.5 Organization The Quad Cities Nuclear Power Station is owned by Commonwealth Edison Company and Iowa-Illinois Gas and Electric Company. The following chart illustrates the Commonwealth Edison Company organizational structure for management and support of Quad Cities Nuclear Power Station.
2
Quad Cities Station Site Vice President -
Site Vice President R. Pleniewicz inummunesammu Executive Assitant b Confidential Secretary T R. Keister Performance improvement Supervisor A. Chernick u-Staff Management Loan Servant
- A. Beveroth D. Holesinger C. Parsons l l l l l l l Support Services Site Financial Regulatory Assuranc Site Engineer & Long Range Work -
Station Manager Supervisor Director Services Director H.g.R'simpsm Supervisor Construction Manager ControlSuperintendent R Bax J. Sirovy w J. Kudalis - A. Misak R. Moravec R.Zurbek m '
_. Operations Miinager B. Bis e H.Hentschel J.Lopatkiewicz h C. Wilkerson NRC l INP0/QV/IDNS D. Kanakares g W. Krause Maintenance Superintendent D. Craddick - Confidential R. Barnett Licensing Group Compliance Group Leader Leader CW T. Fuhs T. Schares Superintendent R. Frohock P. Franklin T. Barber .
W. McGaffigan J. Stepp L Hamilton R. Baumer.
W. Stoermer M. Miller D.Thayer A.Zernich M.Wiesneth TechnicalServices
- Superintendent J. Leider
Nuclear Operations Organization (June 1993)
Vice President andChief NuclearOfficer Nuclear Operations Manager Nuclear Engineer &
W W alS c h t
N lear r l eWeResidem l Manager -
i I
Human Resources i
Nuc 8 Ppt -
i nager l Station Manager l Serv' r or Manager Performance Monitor Business Services
- H. R
- & Improvement Manager Supervisor g
"# # Maintenance Operations Manager Superintendent Regulat Ass rance Di orof dvance Nuct ersight -
- Work Control TechnicalServices
"# * # '# Site Engineer &
E -
NuclearRegulatory Services Manager 1 Construction Manager Long Range Work '
ControlSuperintendent Support Services Director T
, c +u..- e w - --_ -----~u.__..__.-- .--s -. _ _ ---_-- - - - _ - . _ . - _ _ - - . _ . - _ . - - . - - - . _ - . - - . - - - . _
2.0 EVALUATION RESULTS 2.1 Operations and Training The team evaluated the OperC ions Department in the areas of awareness and resolution of equipment problems, evaluating degraded equipment for operability, control room organization and staffing, and control room activities. In addition, the team evaluated the effectiveness of licensed operator training.
To perform these evaluations, the team observed plant and control room activities on all shifts. A downpower evolution to enter the drywell and a plant shutdown and cooldown to repair a leaking feedwater check valve were observed. Surveillance testing of the high pressure coolant injection (HPCI) and standby liquid control systems was observed. Startup of the residual heat removal (RHR) system in shutdown cooling, and startup of the RHR service water system were also observed. Preparations for the annual licensed operator requalification examinations were also observed in the simulator.
The team determined that Operations Department personnel were challenged by the number of problems involving degraded equipment. Several potentially.
significant degraded conditions had not received a rigorous operability review, and the Quad Cities staff frequently failed to use its operability '
determination process. The team observed examples of failure to adhere to procedures involving safety-related equipment. Operations management had not established appropriate standards or expectations regarding overall control and operator response to annunciators.
The team found, with the notable exception of procedural issues, that the operators performed well and that operator training was strong. The control rocm operators were confident, had a high level of knowledge, and worked well as a team.
2.1.1 Acceptance and Limited Awareness of Equipment Degradation The team determined that Operations Department personnel were challenged by the number of problems that existed involving degraded equipment. Equipment problems (1) caused operator workarounds, (2) challenged the operators after reactor scrims, and (3) added to the operational workload. The large number of unresolved equipment problems fostered an attitude of accepting, or. "living with," equipment problems.
The staff at Quad Cities had a Recurrent Equipment Problem (REP) Program that gave station management the latest status of the higher priority equipment issues-(REP Top Ten). System Engineering, the Operations staff, and other organizational units had separate, fragmented programs-(lists) to prioritize plant hardware / equipment problems that needed to be fixed. Examples included the System Engineer Safety System Problem Identification and Tracking System, Operator Work Around List, Issues Management System, and Management Action Item List. There was no central focus at the site on what needed to be fixed i nor was there'a comprehensive effort to address equipment performance problems. ;
)
5 l l
l
- p The Operations staff repeatedly expressed frustration with equipment problems .1 and the efforts needed to get equipment fixed. Operations. management was not l always successful in having equipment repaired. in a timely manner; sometimes equipment was not fixed until it had failed. For example, the team observed ,
during surveillance testing of the standby liquid control system that Manual "
Valve 2-1101-33 was very difficult to open. Although a work request (Tag Number B9104)'had been written on December 4,1991, the problem had not been corrected. Later, while the team was on site, this valve could not be opened.
This was an example of a low-priority work request that did not immediately affect operability and that was not repaired until the component actually failed.
In addition, the Operations Department was not always aware of the degraded status of some important equipment. For example, Class IE electrical cabinets ,
associated with the Unit 2 emergency diesel generator (EDG) were not anchored .
and a reactor core isolation cooling (RCIC) steam admission valve had sheared bolts. After the team identified these two problems, the licensee declared the Unit 2 EDG and the RCIC system inoperable. Some individuals, including managers, indicated that equipment was sometimes declared inoperable to get it' fixed. 1 While the team was on site, several equipment problems occurred. Personnel reduced power at Unit 2 to' replace a drywell sump pump and add oil to a=
recirculation pump motor i.Later, Unit 2 was shut down so that a leaking.
feedwater check valve could be repaired. The reactor building (RB) ,
ventilation system tripped', and when the operator attempted to start the standby gas treatment system to. maintain negative pressure in the RB, that ,
system also tripped. l Operators have _ worked around a number of longstanding equipment problems. For example, the B pumpback compressors in-both units.were not. fully installed ;
after a modification performed about 9 years ago-(the Updated. Final Safety Analysis Report (UFSAR) indicated that'two compressors were installed).
Whenever the remaining compressor failed, the operators maintained the-drywell-to-torus differential pressure by feeding nitrogen 'into the 'drywell ;
, and continuously venting the; torus by opening containment isolation valves i which were listed as normally closed in. the Technical Specifications'(TS). ,
UFSAR Section 6.2.1.2.4.5 states that containment' venting is kept'to a minimum- ,
during reactor power operation. In addition, the-site. operated in single-element reactor level control since-initial plant startup because three-element' control was unstable, and the problems were not resolved. -The .
Vulnerability Assessment Team (VAT) identified this as a vulnerability and~ $
stated that three-element control was an assumption used in the UFSAR. .The :
team also noted that the A feed-regulating valves in both units were operated 8 in the manual mode because they' were unstable in the automatic mode. '
Longstanding equipment problems gave operators problems after reactor scrams.
Unit 2 experienced spurious Group'1 isolations, the most recent _on June 13, 1993. The isolations complicated operator ' control of plant conditions and.
challenged equipment. In January 1992, testing to support a modification to l resolve this problem was performed, but the modification'was not implemented.
The 2B feed-regulating valve " locked up" on a scram and forced the operator to take additional manual action to prevent overfeeding the reactor. The problem had existed for many years. (After _the most recent occurrence of lockup in June 1993, the replacement 1of defective runout flow control relays may have 6
I rg ~
solved the problem.) The Electromatic relief valves (ERV) have failed to open !
on demand; the last occurrence was in February 1992 (Licensee Event Report. ~'
(LER)92-004, DVR 4-1-91-131 and Deviation Report (DVR) 4-1-90-073). The licensee concluded (LER 92-004) that a more thorough root cause analysis of DVR 4-1-91-131 could have prevented the failure. The VAT report also identified the Electromatic relief valve failures as a vulnerability.
Repetitive equipment problems added to the operational workload. Because the toxic gas analyzers failed repeatedly, the control room ventilation system was operated in the recirculating mode for the majority of the time (the licensee was actively pursuing resolution of these toxic gas analyzer problems while the team was on site). More than 40 hard de grounds had occurred in 1993.
Although each ground was individually evaluated and corrected, the licensee had not determined and corrected the root causes for the recurring grounds.
The Unit 2 mechanical vacuum pump recently tripped during plant startup. The ventilation systems tripped frequently (in addition to the observed RB ventilation trip, the RB ventilation system tripped twice on September 13, 1993). Numerous condenser tube leaks had occurred. Other equipment failures also caused significant plant events. Spurious main steam isolation valve closures caused plant scrams in February 1992 and January 1993.
The licensee had made some repairs to fix degraded equipment. Among equipment problems corrected recently were installation of new battery chargers to replace original equipment that proved unreliable, upgrade of the service water and instrument air systems, and modification to the main transformer for backfeed to provide a second source of offsite power. Licensee management had also announced plans to enter short maintenance outages in the near future for both units to repair, test, and further evaluate existing equipment problems.
2.1.2 Operability of Degraded Equipment Frequently Not Evaluated Minimal Operations Department involvement and the absence of operability determinations showed a lack of responsibility and accountability by Operations Department management for the evaluation of degraded plant equipment. Station management had established an operability determination process for degraded and nonconforming conditions but had no clear expectations regarding the use of this process. The licensee rarely used Quad Cities Administrative Procedure (QCAP) 300-39, " Operability Determinations,"
for documenting operability determinations. This procedure was put in place to respond to Generic Letter 91-18, which addressed resolution of degraded and nonconforming conditions and operability. The licensee had documented only three operability determinations using QCAP 300-39 in 1993 before the team's arrival on site.
When the Operations staff questioned operability, the Plant Support staff would provide assistance. However, the Plant Support staff did not use the-engineering procedure ENC-QE-40.1, " Operability Evaluation," to document the evaluat .on unless the " issue" rose to the level of a " concern," as defined in that procedure. The main criterion for defining a concern appeared to be if the issue was outside design limits. The criterion did not address degraded, deteriorating, or indeterminate conditions. If the issue did not rise to the level of concern, or was indeterminate, most operability assessments were informally completed. The absence of a clearly defined design basis, in combination with the high. threshold for performing operability evaluations, 7
1 resulted in a lack of rigorous evaluations and formal documentation of operability determinations. The failure to use QCAP 300-39; the use of a high- -
threshold in engineering procedure ENC-QE-40.1; and the failure to evaluate >
degrading, deteriorating, or indeterminate conditions led to an approach of '
proving _inoperability instead of assuring operability.
While it was appropriate for the technical staff to perform operability evaluations, the Operations Department was often not involved in the operability decision. This was noticeable in the issues documented in the VAT report, completed in November 1992. For these issues, Operations management indicated it " assumed that the engineers" had performed the operability ,
evaluations rather than taking a leadership role regarding the status of plant equipment. Of the 53 vulnerabilities identified by the VAT report, at the time of the diagnostic evaluation, only one operability evaluation had been performed to address that the feedwater system was-potentially operated above the design temperature. Several operability evaluations for VAT issues using ENC-QE-40.1 were initiated when the team raised questions.
The team identified that several potentially significant degraded conditions had not received a rigorous operability review.
(1) No operability evaluation had been performed for a feedwater flow nozzle vulnerability identified in the VAT report. The VAT report indicated "the reactor may be operating above its licensed power level." A t modification had been performed (about 1974) changing the flow element configuration, but the new configuration was not calibrated. The licensee had been using data from a calibration of similar flow nozzles at another facility and had applied those data to Quad Cities. Also, '
the VAT report noted that an uncertainty of 4.1 percent existed for the flow nozzles. Although the uncertainty associated with feedwater flow used in the core operating limits report was 1.76 percent, the licensee had not resolved the differences in accuracy.
On September 24, 1993, after questioning by the team, the licensee implemented administrative limits of 97 percent on reactor power given the uncertainties associated with the.feedwater flow accuracy which the licensee had known about since November 1992. The licensee also prepared an operability evaluation using ENC-QE-40.1 and concluded that the feedwater flow elements were potentially degraded, but operable.
The licensee planned to calibrate the nozzles after the planned maintenance outage.
(2) In April 1993, both core spray system testable check valves (CS 9A and 98) in Unit 1 failed in-service testing performed to verify that the check valves were closed. No operability determination was performed.
These valves were not identified in the TS as reactor coolant pressure boundary valves, and no leak rate had been specified or measured for this function. In the initial operability determination performed after the team questioned the Quad Cities staff, the licensee did not address the pressure isolation function of the valves since this function was i not identified in the TS. The licensee subsequently performed another i operability determination using ENC-QE-40.1 which implemented compensatory actions to verify operability of an interlock prior to I opening the remaining pressure isolation valve for testing.
l 8
i I
(3) No operability evaluation had been performed for the torus cooling .
return motor-operated valves (MOV), RHR MOV 36A and 36B, affected by ,
severe cavitation and vibration until the team raised questions about '
RHR operability. This issue was the subject of Information Notice 83-70 and was also identified as a vulnerability in the VAT report. The i minimum flow valves RHR MOV 18A and 188, the torus spray valves RHR MOV 37A and 378, and the full-flow test return valves RHR MOV 34A and 34B were also affected. Vibration was so extensive that a large valve handwheel was shaken off the Unit 1 torus cooling return valve RHR MOV 36A. The support for the cables to the motor operator had separated from the ceiling for Unit 1 minimum flow valve RHR MOV 18A. The
" operability assessment" provided to the team late in the evaluation was not done using ENC-QE-40.1 and did not clearly state that the time frames for the vibration analysis enveloped those required for the long-term containment cooling function. The evaluation also failed to include verification of the assumptions used in the vendor analysis.
(4) No operability evaluation had been performed on another vulnerability raised in the VAT report. Non-safety-related auxiliary steam lines, fire-protection water-distribution lines, and other water piping ran over or near both divisions of safety-related switchgear. The VAT report indicated that a leak in a heating-coil line or fire-water line could wet the switchgear/ motor control centers and, in the worst case, lead to a failure of safety-related systems. After the team asked why no operability evaluation had been performed, the licensee walked down the piping, identified possible support deficiencies in the piping, and initiated an operability evaluation using ENC-QE-40.1.
2.1.3 Observed Procedural Deficiencies The procedural problems observed by the team were excessive. The team observed examples of failure to adhere to procedures involving safety-related equipment. Appropriate standards regarding procedural compliance had not been enforced. Historically, procedural adherence has been a problem at Quad
, Cities. The failure to follow procedures and inadequate procedures led to a ,
HPCI rupture diaphragm failure in June 1993. Further, operators did not vigorously pursue procedure revisions which were necessary so that the procedures would reflect the actual operation of the systems. Some TS '
weaknesses were also identified.
2.1.3.1 Procedural Adherence Problems The team concluded that Operations management and supervisors had not been-enforcing high standards of procedural adterence and that corrective actions-had not effectively addressed historical ;:rocedure performance problems. The-Event Assessment Team determined that this area needed improvement. NRC had ,
issued several. Notices of Violation for failure to follow procedures. The licensee's 1993 Administrative Discrepancy record noted that the number of procedure violations had increased significantly in 1992.
The team observed several examples of failure to adhere to procedures involving safety-related equipment.
9
(1) While starting an RHR pump in shutdown cooling, the operators did not follow the procedure as written. The operators involved indicated that if the procedure was used as written, there was a potential for an excessive cooldown rate when the RHR pump was started. After the team.
brought this procedural issue to the operator's attention, the operating shift drafted a recommendation to change the procedure; however, the licensee concluded that the procedure as written was acceptable.
i (2) While inerting the Unit 1 drywell, the containment atmosphere monitoring ,
system (CAMS) hydrogen / oxygen analyzers were being used to monitor the oxygen concentration. The team observed that the common failure alarms were lit on both analyzer channels. The operators did not know why the alarms were lit, and the alarm response procedure had not been performed. Subsequent performance of the procedure cleared one of the.
alarms.
(3) During surveillance testing of the standby liquid control system, the' team observed that an operator did not complete the prerequisites of the procedure before beginning the body of the procedure. The operator did not know what actions were Scessary to complete one of the prerequisites and did not know the location of a valve required to be verified by another of the prerequisites.
The first two examples were observed during the initial 2-week onsite portion of the evaluation. In response to these observations, Operations management revised QCAP 1100-12 (Procedure Use and Adherence Expectations) to supply more explicit guidance on procedural adherence,- including alarm response ,
procedures. Additionally, training sessions were conducted on procedural .:
adherence. During the second onsite evaluation period, the team observed that control room operators were, in general, more rigorous about adhering to procedures. A marked increase in the number of procedure change requests submitted by control room operators was also observed. However, another example of a failure to adhere to procedures involving safety-related equipment was also observed.
2.1.3.2 Weaknesses in Implementation of Technical Specifications Some specific TS requirements were not operationally verified or properly controlled. These were observed by the team.
(1) During observation of a plant cooldown, the reactor. vessel temperature limits of TS 3.6 were not~ operationally verified. The procedural requirements for the operators to periodically check the cooldown rate and verify compliance with other limits in the TS were insufficient.
The licensee considered that chart recorders were adequate.to comply ,
with the TS and was not performing operational verification of the requirements. Two 1992 Quad Cities LERs (254/92-11 and 265/92-10).
specifically addressed problems involving reactor pressure vessel and' recirculation loop temperature limits. In response to these observations, the licensee established procedural controls to ensure the '
TS requirements were being verified.
(2) During observation of a reactor water level instrument surveillance test, there were no procedural requirements (in control room (CR) logs or in the testing procedure) to log or monitor the inoperability period 10 9
w of the instruments to ensure compliance with TS during surveillance testing. Later that same day, the licensee identified a TS violation:
action statements were not implemented when a TS instrument was removed from service for surveillance testing. The licensee subsequently established procedural controls in this area.
In addition, NRC issued a Notice of Significant Enforcement Action'(EA 93-162) on August 6, 1993, that addressed the fact that no written procedures were established to track opposite-unit / shared-unit safe-shutdown components when equipment was taken out of service. Because of procedure deficiencies.-the licensee was unaware that a potentially significant 10 CFR Part 50 Appendix R safe-shutdown component limiting condition for operation had been exceeded.
In addition, there were two other occurrences in 1993 where the licensee failed to properly adhere to 10 CFR Part 50 Appendix R requirements.
2.1.3.3 Other Procedural Weaknesses Some plant operating procedures did not reflect the manner in which the plant was actually operated. A few examples of poor procedural controls were noted.
Operations personnel at Quad Cities were not aggressive in their attempts to revise or enhance procedures.
(1) The procedures for the RHR system indicated that the minimum flow valves, RHR MOV 18's, remained in service during shutdown cooling operation. As .such, these valve could allow inadvertent draining of the vessel to the torus Control room operators indicated that the valves were disabled by the shift engineer (SE) tagging out the valves after .
RHR had been placed in the shutdown cooling mode. In previous outages,
- this was done as a nonintent change to the procedure, and the licensee had not evaluated the need for minimum flow protection prior to the
- tagging of the valves. After questioning by the team, the licensee performed an evaluation and concluded that the minimum. flow valves could be disabled.
(2) The procedure for de-inerting the drywell did'not address that feed and
, bleed may be used to maintain the drywell-to-torus differential pressure when the pumpback compressors were out of service. Also, the procedure for inerting refers to the oxygen analyzer as the normal means of monitoring oxygen concentration. In practice, the. CAMS was used because of repetitive problems with the oxygen analyzer.
(3) A procedure did not exist for draining the HPCI system turbine exhaust drain pots. In August 1993, the licensee found that a system engineer, while draining the drain pots, erroneously directed an equipment operator to close a valve that isolated and disabled the drain-pot level switches for both Units 1 and'2. Also, plant operators did not perform any verification that the system was correctly aligned after draining the pots.
(4) The licensee identified that Quad Cities Appendix R Procedures, as written, would not have supported the achievement of a safe plant shutdown in the required amount of time (LER 93-016).
11
J 2.1.4 Decreased Sensitivity to Control Roorn Annunciators The team found that Operations management had not established appropriate standards or expectations regarding control room annunciators. The team observed operators sometimes failing to question or pursue annunciator indications. The designation of some annunciators as " expected" alarms c illustrated acceptance and lack of resolution of underlying equipment problems. As a result, operators were less sensitive to control room annunciators.
(1) The team observed that a number of annunciators were lit for extended-periods of time due to equipment problems that were not expeditiously resolved. No one considered disabling the. alarms. As an example, the "RB Low DP" annunciators were continuously lit. The alarms received inputs from the reactor water cleanup (RWCU) heat exchanger room and the RB. The RB ventilation system rarely gets the RWCU heat exchanger room differential pressure to a high enough value to clear the instrumentation setpoint. After the team discovered that the alarms were not providing useful information to the operators, the licensee implemented a temporary alteration which removed the RWCU heat exchanger room input from the alarm. Other examples of unresolved equipment problems resulting in continuously lit annunciators included: Unit 1
" Charcoal Adsorber Vessel Temperature High," " Crib HSE A Floor Drn Sump i High Level," and Unit 1 "0 2Analyzer Low Flow."
(2) The team observed that some annunciator windows were' colored green.
Apparently this indicated that these alarms would be expected at power. ,
As a result, less attention was given to those alarms. However, expectations of actions to be taken when these alarms were lit had not been promulgated. The fuel pool cooling low discharge pressure alarms (green) were lit on Unit 2 but not on Unit 1. Subsequent investigation found that-the alarms in Unit I were not performing as expected and that-the operators did not clearly understand how the system was supposed to function. Also, the alarm (green).for the reactor vessel head seal-leakage in Unit 2 was lit. The team concluded that if the system was operating as designed, the alarm should not be expected.
(3) The team observed that annunciators in the CR which had reached the reset point were not cleared for long periods of time. The operators did not acknowledge the cleared status because that would result in the-alarm sounding again if it reached its alarm setpoint. In the case of repetitive alarms, the horn could be silenced from the sequence-of-events recorder panel . The operators had not been told how many alarms could be disabled or for how long.
2.1.5 Limited Oversight of Control Roorn Activities During Busy Periods The assignments of tasks in the CR at Quad Cities permitted only limited oversight of control room activities during some busy periods. Although CR staffing levels exceeded the TS minimum, the team observed that it was sometimes difficult for the shift control room engineer (SCRE) to appropriately oversee CR activities. The SCRE was the only senior licensed 12
I operator normally in the CR during power operations since the offices for the 1 shift engineer (SE) and the shift foreman were located outside the locked CR doors. The SCRE would also function as the shift technical advisor in a casual ty. During several minor incidents, the team noted that instead of overseeing the incident, the SCRE was initially occupied by contacting the SE.
It was not uncommon for both units to be changing power because Quad Cities was operated as a load following plant. Interviews with corporate personnel indicated that the SCRE could choose not to load follow if he judged that the plant conditions could not support the power change. The team observed, however, that site personnel almost always attempted to accommodate the load dispatcher's request for load following, even if one unit was in the process of shutting down. In some instances, both units were changing power, several potentially significant equipment problems were occurring, and surveillance testing was in progress. Under such conditions, the SCRE's ability to provide appropriate oversight was limited.
During observation of simulator training, the team noted that only about 25 percent of the scenarios have the SEs outside the control room at the beginning of the scenario. This does not accurately simulate the normal CR staffing, the SCRE to SE turnover, the initial absence of a shift technical advisor, snd the absence of an " extra" individual (the SE) at the beginning of a plant incident. Such training tends to promote reliance by the SCRE on the SE to provide initial oversight of the incident. This tendency was confirmed by actual plant observations that the SCRE's first actions were to contact the SE rather than provide ovefsight.
2.1.6 Strong Overall Operator Training During observation of simulator training, the team noted high levels of confidence, operational performance, plant knowledge, and teamwork.
Interviews with the operators indicated that they found training was at the proper level of difficulty, that it was important, and that they took training very seriously. The operators have specifically acknowleJged that the Training Department's efforts have prevented or mitigated actual events in the plant. The team concluded that the use of a site-specific simulator has contributed to significant improvements to overall operator performance.
Operator performance errors made during simulator evaluations and walk-through examinations were detected'and addressed by the facility's evaluators.
Critiques of operators and' crews after the simulator drills were effective' in denoting strengths and weaknesses. The Training Department evaluators effectively identified those individuals who- required remediation.
Performance standards used in conducting evaluations were communicated to the ,
evaluators directly by training management. These standards were validated by '
the Operations Department through feedback from operator supervisors, Operations Department managerial overview reports, and station management overview. The team observed the Station Manager's overview of a crew evaluation on the simulator. The Station Manager gave direct feedback to both !
the crew and the simulator evaluators on his expectations of performance for the crews.
13
The team reviewed the process for identifying operator performance deficiencies and how they were incorporated into the licensed operator training program. The facility identified operator performance deficiencies through simulator evaluations and direct discussion with operating crew members and their supervisors. The Training Department had assigned one specific instructor to each of the six operating crews to facilitate feedback to the Training Department for incorporation in future training. The facility provided training for the crews for performance deficiencies through a mixture of training methods, including free simulator time when possible during training sessions; classroom discussions, including the Assistant Station Operations Manager's training session; and such other methods as required reading or night orders.
In reviewing the effectiveness of the licensee's process for revising its licensed operator training program to maintain it up to date, the team assessed the use of feedback from the Operations and Maintenance Departments. ,
The Training Department was responsible for obtaining training feedback through the facility's training request system. Nearly anyone could submit a training request at computer terminals within the plant. This allowed training to adjust to changes in plant design or procedures and the occurrence i of plant or industry events. Each of the training requests reviewed by the team could be traced to a legitimate training need, and each was assigned a Training Department contact for resolution and followup.
2.1.7 Positive Observations The team concluded that the performance of the licensed CR operators was good even though Operations Department management had not been successful in getting plant equipment fixed. Communications between operators was particularly strong. The team noted a high level of performance in overall crew communications. The use of the "three-peat" system of communications (order-repeat back-verification) was practiced extensively in the control room, in the simulator, on the phone, and during informal communication outside of the CR. During training activities, good communication was strongly emphasized. Shift turnover and activity briefings were usually thorough. The operators were confident, exhibited a high level of knowledge and positive attitudes, and worked well as a team.
2.2 Maintenance and Testing In its evaluation of maintenance and testing activities the team focused on the residual heat removal (RHR) system. The evaluation included lateral interfaces and support to maintenance, the maintenance process and its implementation, preventive maintenance, predictive maintenance, root cause analysis, motor-operated valves (MOVs), industry operating experience, risk insights, work backlogs, planning and scheduling, work prioritization, inservice testing (IST), and other maintenance program initiatives.
The team observed that the maintenance process was cumbersome to control and exhibited weaknesses in its implementation. Overall, maintenance management was not fully effective in sustaining available and reliable plant equipment.
The team noted many instances in the IST program where actions were not taken to identify or fix the root causes of equipment problems.nor were evaluations 14
performed of potentially safety significant degradation. Within the licensee's MOV program, root causes were not identified which led to numerous repetitive MOV problems. Management did not fully recognize the significance of equipment problems, failed to implement recommendations provided by both external and internal sources, and failed to ensure sustained corrective action programs. The team concluded that support to Maintenance from other organizations was insufficient to assure an effective maintenance program.
The team concluded that plant appearance and housekeeping were good due in part to initiatives such as the residual heat removal service water (RHRSW) pump room restorations, pump mechanical seal installations, and valve packing improvements. The site had few packing, valve, or pump leaks. Radiological control practices were also good.
I 2.2.1 Failure to Fix the Root Cause of Known Testing Deficiencies A number of safety-related pump and motor vibration readings routinely exceeded the American Society of Mechanical Engineers (ASME)'Section XI alert threshold without timely corrective actions being taken, and some pump capacities were inconsistent with design requirements and varied significantly betwean identical pumps. As stated in the licensee's Vulnerability Assessment Team (VAT) report, in October, 1992 there were 74 pieces of rotating equipment that were at or above their vibration alarm levels. Some safety-related check valves were neither tested in accordance with Section XI requirements nor were there requests submitted to the NRC for relief from those requirements. The site had experienced an unusually high failure rate of RHR system relief valves without taking appropriate corrective actions to address the root cause of test failures. Tne overall relief valve program did not include adequate trending and root cause of lift test failures. Several weaknesses were noted in the control of Section XI requirements for pumps, check valves, and safety-related MOVs. Specific examples of these weaknesses are noted below:
(1) Vibration readings for the Unit 1 and 2 high-pressure coolant injection (HPCI) pumps were in the alert range for an extended period of time. To determine the cause of the Unit 2 pump vibration, a laser-based alignment inspection of the pump was performed which showed that the pump was out of alignment. The licensee viewed correcting the condition i as too costly to lower the vibration readings into the acceptable range.
The licensee limited the realignment to the low-pressure portion of the Unit 2 HPCI pump. Following the alignment in August 1992, vibration-readings were reduced but were still in the alert range.
Instead of correcting the cause of the vibration, the licensing organization submitted a relief request to the NRC in May 1993 to raise the ASME Code Section XI alert threshold for the Unit 2 HPCI pump so it L would be in the acceptable range. The relief request stated that "The specific limits assigned to the HPCI pumps are based on extensive experience with these pumps and the inherent high vibration levels associated with pumps of this design. The HPCI pump impellers have been modified to reduce vibration levels (approximately 50 percent) yet absolute levels remain high. The turbine and pump rotating components have been re-balanced and extensive re-alignment work has been performed with little overall improvement in vibration levels." The relief request did not mention that the pump was found to be out of alignment 15
or that minor alignment of the booster pump and gear box was performed in August 1992, but not to the extent prescribed by the laser alignment examination. A followup paper submitted to the NRC to clarify information surrounding the relief request referenced irreversible corrective actions, inaccurate alignment results, and skepticism, none-of which could be supported.
Similarly, several RHRSW purrss experienced vibrations in the alert range and erratic flow test results for several years without corrective action being taken. Vibration trending data indicated that RHRSW pump 1A had IST readings in the alert range for both vibration and flow; pump 1B was in the alert range for low flow; vibrations for pump IC were consistently in the alert range; vibrations on pump ID were erratic and generally exceeded the alert threshold until recently; and vibrations on pump 2C consistently exceeded alert range values. The licensee'is currently addressing these issues.
Additional vibration readings taken on safety-related pumps and motors for predictive purposes (not within the scope of the IST program) indicated that these safety-related pumps and motors had also routinely-exceeded the administrative alert values for years without adequate corrective action being taken. Examples include the (1) standby liquid control (SBLC) pumps lA, IB, 2A, and 2B; (2) core spray (CS) pumps IA, 18, and 2A; (3) RHRSW pumps IA and 2C; and (4) RHR 2C pump motor.
Adverse trends were also noted with the DGCW pumps and HPCI pumps.
(2) Discrepancies discovered between the CS pump head data and vendor pump ;
head curve data were never evaluated. The IST reference value for the 1A CS pump was 450 gpm lower than the IB pump and significantly below .
the manufacturer's pump curve. When this condition was questioned by )
the team, the licensee evaluated the pump performance characteristic for the 1A pump and concluded that there were consistent unexplainable a deviations from the manufacturer's pump curve with one significant deviation of 800 gpm in August 1991. The discrepancies were scheduled to be evaluated and corrected during the December 1993 outage.
The Unit 1 HPCI pump capacity was significantly below the vendor pump curve and approximately 700 gpm less than the Unit 2 HPCI pump. . This condition existed .for years without being evaluated and corrected. The Unit 1 HPCI governor controls had a chronic history of oscillation problems (rpm fluctuations as high as +/- 500 rpm). The turbine was-designed to be operated at 4000 rpm, the operating procedure listed a value of 3900 rpm, while the actual test rpm varied between 3700 and 3800 rpm as recorded in IST quarterly operability insts. Despite the test discrepancies, the operability tests had been s gned off as acceptable in several instances with no evaluation performed.
IST flow test results for. DGCW pumps were erratic for several . years,.
producing both alert low and al_ert high test data. These conditions were being addressed.during the evaluation with the installation of new pump casings. After the changeout of the pumps and Belzona coatings on the interior of the volute, the pumps actually exceeded design flow.
Engineering's review of the high-flow condition failed to consider the potential deleterious effects of high flow on the DGCW heat exchangers.
16
(3) Some safety-related check valves were not tested in accordance with Section XI requirements. For example, the IST program required the CS 9 A/B and RHR 68 A/B check valves to be cycled open and closed,. and for the position indication function to be checked. These activities were not being performed in all cases, and relief requests from Section XI testing requirements had not been submitted to the NRC. For example, the position indication function of the CS 9A and'9B valves was tested only sporadically and was eventually disabled, and the back-leakage test was performed only once. RHR 68 A/B testable check valves were never tested for seat leaks in accordance with Section XI.
Also, in response to Generic Letter 87-06, " Periodic Verification of Leak Tight Integrity of Pressure Isolation Valves," the licensee committed to continuous monitoring of CS and RHR pump discharge pressut e annunciators as an indication of excessive check valve leakage. The use of control room annunciators to determine back-leakage was not only inaccurate (some alarms were in a constant alarm condition when the CS injection valve was open), but did not satisfy the ASME Code requirements for leakage testing. General Electric also determined that this approach should not be used because of inaccuracies in the approach.
(4) The results of check valve inspections, testing, and failures were not :
trended or tracked. The licensee's responses to address a 1986 industry '
initiative involving check valves were slow to develop. The initiative was slowly developed as check valve og rability questions surfaced and the scope of the program broadened. The licensee did not issue its Check Valve Preventive Maintenance Program to address the 1986 initiative until 1993. Repetitive events during the period 1990 through 1993 involving reactor building floor drain check valve failures resulted in additional reviews, and the licensee was still determining the root cause of multiple' failures in 1993 of floor drain check valves located in Unit 2 CS, RHR, and RCIC rooms. -
(5) During refueling outages Q1R10 and Q1R11 for Unit I and Q2R10 and Q2R11 for Unit 2, more than 75 percent of the RHR relief valves failed lift "
setpoint testing, but no appropriate corrective actions were taken to address the root cause of test failures. The failures were attributed to poor design, corrosion product (rust) in_the system, and corroded internal components. Corrective Action Record (CAR) 04-92-036 developed
, after the outages stated that the Quad Cities staff failed to identify l and address the problem of RHR relief valves failing lift setpoint
- tests. The CAR also stated that "There are plans to test several of these relief valves during the next outage (Q2R12)." This plan to test the relief valves was never implemented. No RHR relief valves were tested during either Q1R12 or Q2R12. The CAR was closed in August 1992.
The team noted that no trending was done of relief valve failures to indicate test results, dates, lift pressures, valve type and manufacturer, or root cause.
(6) A review of work requests on the RHR system dating back to 1987 indicated a tendency to increase torque switch settings in an attempt to reduced the valve leakage and pass local leak rate testing (LLRT). More l
recently, this was also done to pass differential pressure (dp) testing. -t Solid wedge design gate valves are difficult to seal if the valve disk 17 l
and valve seat surfaces are not properly mated. Increased torque switch settings were still evident during the evaluation in that stem thrusts were typically set near the design maximum. However, recent efforts have included maintenance of some valve internals resulting in improved valve seat conditions and lower torque switch settings.
(7) Weaknesses were noted in the programmatic control of Section XI stroke time testing of safety-related MOVs. The licensee's Quality Assurance Program (QAP) listing of power-operated valve surveillance values included a matrix table of valve number, stroke direction, maximum allowable stroke time, and completed surveillance tests. A review of the report showed that several safety-related MGVs had exceeded.the non-TS maximum allowable stroke times without being declared inoperable or being evaluated for operability. The origin of the non-TS maximum allowable stroke times was apparently the vendor, who established the times based upon operator size, gearing, and valve characteristics. The licensee had not evaluated the vendor-supplied maximum allowable or actual stroke times to determine if a limiting system design requirement would take precedence.
(8) IST scope and test acceptance criteria for some valves was inconsistent.
Valves that were designed to provide a containment isolation function 1 were not always tested to verify acceptability, and some acceptance '
criteria were changed without adequate evaluations. For example, the maximum allowable IST stroke time for the RHR injection throttle valve (RHR 28) was administratively changed from 25 seconds to 90 seconds which was inconsistent with the RHR injection valve (RHR 29, the next valve downstream in the injection path). This containment isolation valve had a maximum stroke time of 25 seconds. Containment isolation gate valves RHR 7 and 28 were not seat leak tested. Similarly, containment isolation testable check valves RHR 68 and CS 9 were not seat leak tested in the past but will be in the future.
Fire protection valve 1-8941-705 was not tested or repaired in :
accordance with ASME Section XI requirements. This valve, including other similar valves, was added in response to an NRC inspection. The '
last time this valve passed IST stroke time testing was almost two years ,
ago. Maintenance has been unable to fix the valve which was still inoperable while the team was on site.
(9) Testing documentation weaknesses were noted: (1) trending reports did not indicate whether a recorded stroke time was acceptable or unacceptable, (2) the licensee had to search maintenance history records to determine if maintenance activities' contributed to stroke time deviations, (3) new stroke time reference values were not clearly ,
identified in reports, and (4) software was not used to track and trend ;
valve stroke performance. (AdditionM M0V ha dware and programmatic weaknesses are discussed later in Section 2.2.3.)
2.2.2 Observed. Weaknesses in Maintenance implementation l
The team observed several maintenance activity weakne::ses such as limited pre- )
job briefings, poor communications, and inaccurate drawings *. Maintenance l management did not always effectively utilize or seek engineering support.
18 )
l Failures to incorporate engineering requirements into maintenance plant procedures led to repetitive work and unnecessary personnel radiation exposure.
(1) Following the repair of the B feedwater check valve in Unit 2, the licensee failed to retorque the valve bonnet nuts while at operating pressure to compensate for plastic deformation of the pressure seal ring, which was a vendor recommendation. A futile attempt to retorque the bonnet nut at power to eliminate the leakage resulted in unnecessary radiation exposure. The team found that the failure to incorporate these engineering requirements into instructions in the maintenance work package resulted in a unit shutdown on September 2, 1993, to repair excessive leakage past the valve bonnet. Another failure to incorporate appropriate engineering re:;uirements into the work procedure occuried in May 1993 when maintenance craft were working on CS check valves 9A-and 98 to eliminate leakage past the indicator shaft. A review of work .
history indicated that a cheater bar was used to tighten the indicator mounting plate and housing retainer screws and that vendor-recommended torque values for the screws (18 ft-lb and 150 ft-lb, respectively) were most likely exceeded. Craft personnel failed to follow procedures, and again, vendor technical- requirements were not correctly implemented.
(2) The team observed maintenance activities associated with the replacement of an equipment drain sump pump. Limited pre-job briefings and inaccurate drawings made Faintenance activities more difficult in August 1993 when maintenance crews made an "at power" entry into the Unit I drywell to replace the drywell equipment drain A :: ump pump and inadvertently replaced the B sump pump. The four drawings used for verification of the correct components did not correlate with the actual pump configuration. The pre-job briefing placed little emphasis on the -
work performed or the information in the maintenance work package.
During this activity, communications between the' control room and the maintenance craft in the containment were confused and contributed to the error. Because of the additional work that had to be performed, this activity tripled the expected radiation exposure.
(3) On August 26, 1993, an incorrect weld procedure was used to make a seal weld on the Unit 2 B regenerative heat exchanger outlet isolation valve, a non-safety-related valve. An investigation performed by the licensee also identified that an incorrect weld filler material was used (i.e., a carbon steel to carbon steel procedure was used to weld a carbon steel yoke to the stainless steel bonnet). As a result of using the incorrect procedure and filler material, the weld leaked several times during-post-maintenance testing and, in one case, the weld cracked. In addition, the weld record in the package indicated 6 entries were made ,
by maintenance personnel to perform the welding on the valve. It was also discovered that a similar mistake had been made on the Unit I regenerative heat exchanger outlet isolation valve. .However, the licensee determined by evaluation that this weld was sufficient for its-intended service.
A significant example of maintenance implementation weaknesses had also been recently identified by the NRC. The NRC issued a Notification of Violation and Proposed Imposition of Civil Penalty (EA 93-127) to the licensee on July 30, 1993, involving the Unit 2 DGCW pump not being capable of performing 19 m
its intended function. This occurred as a result of a pump overhaul in which the EDG lubricating oil piping was incorrectly assembled, resulting in no :
lubrication to the pump bearings. This degraded condition existed from ;
January 1992 until it was discovered in March 1993.
2.2.3 Failure to Fix the Root Cause of Motor-Operated Valve Deficiencies Progress had been made in correcting the historically high number of valve deficiencies at Quad Cities, but management's failure to fix the root cause hindered significant improvement. Corporate and site groups deferred most of the responsibility for implementation of the MOV program to an individual in 4 the Maintenance Department who had Limitorque experience and a staff of contractors. Many of the MOV problems were related to deficiencies in engineering, plant design, system operation, and training. The responsible individual was unable to obtain the management support needed to make the necessary improvements, such as getting a dedicated staff and sufficient outage time to make the needed valve repairs, performing root cause determinations, improving equipment history documentation, and tracking repetitive failures to fully resolve the equipment problems.
Maintenance management did not correct the root cause of M0V problems even though recommendations had been made to correct these types of deficiencies in internal audits or evaluations. For example, the licensee's On-Site Quality Verification Section performed an MOV root cause analysis audit and included-recommendations to the plant staff in the January 1993 Monthly Quality Verification Report. Among the recommendations were the following: provide guidance to prevent thermal binding of valves; provide guidance on motor operator starting duty cycles to prevent damage; evaluate the use of cheater bars on MOVs; consider including valves with a history of chronic problems in the MOV testing program; consider documenting all MOV troubleshooting activities so the MOV coordinator can trend, prioritize, and fix valves; consider writing work requests for all valves found thermally bound; and provide cross-training for system engineers, operators, and maintenance personnel on M0V operations. Few of the recommendations were fully implemented, and in most cases, newly drafted procedures addressing the recommendations had not been approved. A 1988 Maintenance History Evaluation had many of the same comments regarding MOV program deficiencies.
The work history review of the RHR system indicated a relatively high level of valve operator work with little valve body work. In one case, following corrective maintenance on valve internals for the full-flow test return valve 2 RHR 34B, the required closing thrust was found to be significantly reduced to 34,000 lb. The closing thrust for three other identical valves was 50,000 lb. This indicated that the practice of limiting valve repairs to the valve operator, and not including valve body internals, might be a contributing cause of the higher-than-expected deficiencies.
In order to determine actual conditions and corrective maintenance activities, the team as well as the licensee had to review hard copies of vault records and personal logbooks in detail. The lack of detailed work history interfered with planning work activities and affected the ability of the-site to track and identify repetitive failures.
20
l l
l From 1985 until June 1993, the work histories of 62 RHR MOVs disclosed that ;
more than 200 valve repairs had been accomplished: 56 associated with the '
Limitorque operators or motors, 26 repairs associated with breakers or '
overloads,19 replacements of worn stem nuts,16 associated with loose or sheared bolting and cracked yokes,13 yoke replacements, and 5 stem ,
replacements. During the same time period, 9 repairs were performed on valves '
bodies with 5 of the repairs coinciding with stem replacements. This work was only performed after the valve component had failed its function. The number ,
of repairs indicated that the root cause(s) of the valve failures were not '
being corrected, and there were generic problems causing the deficiencies.
The team identified several examples (containment isolation valves) of licensee corrective actions being untimely or lacking:
(1) On February 24, 1992, a nuclear work request (NWR) identified valve guide problems on the full flow test return valve (2 RHR 348). An NWR, written on January 1,1993, identified a bent stem in the same valve.
Valve disk repairs and replacement of the bent stem were not completed until May 4, 1993. -
(2) On November 14, 1988, an NWR identified that the operator on injection throttle valve (2 RHR 28A) had excessive grease leakage. The NWR was cancelled on March 3,1990, with indication that there was no problem.
Another NWR, written on September 1, 1992, noted that oil had leaked from the gear case of valve 2 RHR 28A and further noted that the gear case was likely empty because the leak had stopped. When the. work was finally completed on May 4, 1993, the valve operator had extensive internal damage.
(3) On November 15, 1986, an NWR identified that injection throttle valve (1 RHR 28A) would not close while under pressure during shutdown cooling. This NWR was cancelled and was eventually combined with an NWR to perform a static valve operation test evaluation system (VOTES) test, completed on December 3, 1990. The licensee indicated to the team that the root cause had never been identified, and consequently corrective actions had never been implemented to address the deficiency. ;
(4) An NWR written on November 11, 1992, documented that RHR pump suction valve (1 RHR 78) was leaking back to the torus' because the water level .
in the reactor was dropping at about 200 gallons per hour. - Another NWR, ,
written on September 28, 1987, stated that valve 1- RHR 7D _would not isolate the RHR pump from the torus because-it leaked through. Both of these NWRs were found to be open during the evaluation. This leakage is only of concern while the valves are shut, which would occur during shutdown cooling operations. Operators worked around this deficiency by.
monitoring the torus level more frequently.
The team noted specific e.xamples (containment isolation valves) where the licensee did not identify the root cause of equipment failures:
4 (1) In March 1992, during the overhaul of the Limitorque operator on injection throttle valve 2 RHR 28B, the licensee found that the grease was separated, the oil sample was 5 percent copper by volume, and _the grease was heavily embedded with brass, resulting in excessive wear to i
21
i the worm gear. The licensee corrected the deficiencies but did not address why the deficiencies occurred or what actions could be taken to prevent recurrence.
(2) In April 1993, the licensee found that all 12 (1-1/2-inch-diameter by 6-inch-long) yoke-to-actuator bolts on injection throttle valve 2 RHR 28B were loose. The licensee assumed that the cause was due to the bolts having been stretched. Rather than addressing the cause, the bolts were replaced with stronger bolts and tightened " snug to fit." During interviews with the team, the M0V coordinator indicated that he was not aware of this potentially significant problem.
(3) In June 1990, an NWR was written for repair of a motor failure on shutdown cooling suction isolation valve 1 RHR 50. The NWR stated that the root cause of the motor failure was " heat failure due to excessive-current being drawn by the motor while trying to pull the valve off of the closed seat." The NWR also stated that the licensee did not determine why the disk was stuck in the seat or why the.0perations staff reset the tripped breaker and overloads several times.while attempting to operate the valve.
(4) On February 25, 1990, an NWR documented that the motor operator for full-flow test return valve 2 RHR 34A was replaced due.to a. cracked casing. An NWR, completed on February 24, 1992, documented that the motor operator for the same valve was rebuilt because the grease was in a fluid state and the grease on the worm gear was embedded with brass.
The licensee had not evaluated these failures to determine if they were related.
(5) On February 21, 1991, an NWR documented two cracked welds at the yoke to bonnet on torus cooling return valve 1 RHR 368. An NWR, completed on June 18, 1992, documented four broken bolts on the operator yoke of the same valve. Both failures could be related to excessive thrust or cyclic fatigue. The licensee had not recognized either of the examples as repetitive failures.
~
(6) On May 7,1993, an NWR documented stem replacement for torus cooling .
return valve 2 RHR 36A. On June 2, 1988, another NWR documented stem replacement for the same valve. It appeared that the stem was replaced twice in the last 5 years without an evaluation for repetitive failure.
2.2.4 Ineffective Maintenance Work Process Resulted in the Failure to Get Equipment Fixed The maintenance process was cumbersome to control and implement. The work process was burdened with such a large number of NWRs such that only high priority corrective maintenance that directly affected plant operation _could be assured of being worked. Additionally, work history documentation was confusing, incomplete, and' difficult to track. Corrective actions generally-addressed the symptom or condition but not the root'.cause.
There were a number of barriers for the maintenance technicians to overcome.
Work packages were cumbersome and difficult to implement. The standardized work packages were almost equivalent to modification packages in size and 22
detail for all jobs. Difficulties were identified in the process of making field changes to work packages, and it usually took at least two shifts to make even minor documentation changes. In addition, a large number of signatures (up to 15) were required before the NWR was ready to be workcs This detail frustrated many craft personnel. This process also contribereJ to the large backlog of NWRs maintained by the work analyst and the lengthy time to complete a work activity (average of one year).
Maintenance management seemed more focused on industry performance indicator goals rather than focusing on the level of effort required to reduce the backlog. The team noted that the published maintenance backlog indicator was approximately 1700 corrective maintenance NWRs which site and corporate management used to evaluate maintenance effectiveness. The team, however, found a total of approximately 5000 open NWRs of which 2950 were safety related, the oldest dating back to 1981. There was a major backlog of approximately 3300 NWRs awaiting planning by the maintenance analysts. In t addition, many NWRs were combined, making the corrective maintenance NWR indicator appear smaller than it really was. The licensee was unable to document the number of NWRs that were combined, but estimated that at least 25 percent of all NWRs were combined. The licensee defined an abnormal amount of what is normally considered corrective maintenance (CM) as preventive maintenance (PM). During the last year, more than 300 NWRs categorized as PM involved substantial CM actions. This PM indicator provided the appearance that more equipment was being maintained, its CM indicator appeared less than what actually existed, and the ratio of preventive to corrective maintenance improved the perception of the Maintenance Department's effectiveness. The Quad Cities July 1993 Monthly Performance Update stated that the total non-outage work request goal was 580 NWRs and that the actual was 773 NWRs.
Additionally, the report stated that only 921 NWRs would be worked during the future Q1R13 refueling outage. The team concluded that the maintenance backlog was not being accurately reported or effectively managed.
The licensee could not demonstrate the effective use of work history documented in various data bases and had to resort to a detailed time-consuming search of vault records to determine what work had actually been
, performed. Vault records also reflected poor documentation of actual work performed and could not always substantiate that each work item of each NWR was actually completed.
Maintenance management was not involved in the root cause determination process. The team noted that maintenance procedures directed maintenance technicians to document the root cause of failures following the completion of the maintenance activity. The root cause would then be screened to evaluate whether-a root cause determination would be requested to be performed by engineering. However, because of a lack of maintenance management involvement ~
during the root cause and screening process, engineering support was not requested to fully evaluate the cause of a number of equipment problems. In fact, the team determined that, per the screening criteria, engineering was not involved unless the equipment failure caused significant operat'ional problems. Failure of maintenance management to. recognize the need for engineering involvement in the root cause process resulted in a number of .
inadequate root cause evaluations and repeat equipment failures.
23
4 +
2.2.5 Support to Maintenance Was Not Sufficient to Maintain an Effective Program Support to Maintenance from other organizations was insufficient to maintain an effective maintenance program. In many instances, the maintenance department did not actively seek engineering support and felt that it could resolve most issues. The team concluded that (1) system engineers were not significantly involved with maintenance activities on their assigned systems, (2) support engineering involvement in the maintenance process was not proactive, (3) design deficiencies and incorrect control drawings contributed to poor maintenance, (4) operators were reluctant to identify degraded components to the Maintenance Department, and (5) there was a lack of sufficient time to complete maintenance on degraded components scheduled for repair during refueling outages.
Site manager interviews emphasized the importance of meeting outage schedules.
Consequently the outage did not allow sufficient time to complete needed maintenance on degraded components. This condition was worsened by the high number of unplanned NWRs, the number of planned but unworked CM NWRs, and the number of NWRs that were routinely deferred or cancelled during outages. >
Outage management was reluctant to schedule certain maintenance activities because of the probability of exceeding the outage windows for those activities. Corrective maintenance identified during an outage was routinely deferred until the next outage as a result of management's desire to meet the '
restart schedule. The team noted that although approximately 1400 NWRs completed during the last (Q1R12) outage, 225 NWRs were either cancelled or postponed.
Although the system engineers were responsible for all work performed on their assigned system, they were not significantly involved with maintenance activities on their systems. System engineers were not proactive in determining whether root causes were identified for repetitive failures occurring with their systems and were only involved upon request. Nor.did system engineers request other engineering support to resolve longstanding
, problems with their systems. Interviews indicated that they were not aware of the open work requests or the work history of their assigned systems.
Maintenance technicians also indicated that system engineer assistance was not desired because of their lack of experience and understanding of the immediate problem and what was needed to fix it.
Plant Support. engineering involvement in the maintenance process was- not -
proactive. Engineers were seldom involved in root cause analysis of the ,
- longstanding vibration problems or MOV hardware failures. Design engineers were seldom involved as they were.only required when requested by the system ~
engineers. However, notwithstanding the many design problems that existed at Quad Cities, system engineers seldom. requested support.
l Operations Department personnel expressed frustration with the' efforts needed 1 to get equipment fixed and were reluctant to write NWRs on components that, in '
their view,- would not be fixed in a timely manner. Operators expressed a lack of trust in the ability of Mechanical Maintenance to fix equipment in an efficient manner citing the numerous times operators were required to reestablish plant conditions during repetitive attempts to fix equipment.
Often times what operators attributed to Mechanical Maintenance 24 4
ineffectiveness was due in part to insufficient support to Maintenance and an ineffective work process. The team's review of longstanding operator work arounds indicated that approximately 50 percent of the NWRs written to resolve operator concerns were not initiated until June 1993. In addition, of the NWRs written to resolve operator work arounds, a large percentage was not scheduled at the time of the diagnostic evaluation.
2.2.6 Preventive Maintenance Program Not Effectively implemented The PM process had several barriers hindering the implementation of recommendations provided by the PM coordinator. The PM program was the responsibility of a single individual within the Maintenance Department. PMs required approvals by the system engineer and a cost benefit committee which could reduce the frequency, modify the extent, or cancel the PM if it was determined that it would be too costly or the immediate benefit of the PM could not be realized. The licensee failed to implement several PMs, including a number of PMs that were recommended by the reliability-centered maintenance (RCM) program evaluation. The funding for implementation of the RCM was later cancelled because site management found it too costly. Had this effort been fully implemented, the HPCI rupture disk event and associated personnel injuries may have been prevented since the PM program would have included the HPCI Magnetrol level switches. The NRC issued a Notice of Violation and Proposed Imposition of Civil Penalty (EA 93-210) to the licensee on September 21, 1993, regarding the licensee's failure to promptly correct and identify PM program deficiencies.
In addition, recommendations resulting from Maintenance History Evaluation of 28 systems completed in 1988 were never implemented. This evaluation provided the licensee with detailed analytical insights of equipment reliability, unavailability, and maintainability characteristics. Due to the lack of management involvement and commitment, sustained performance enhancements were sometimes limited. For example, recent improvements were noted with the predictive maintenance techniques using laser alignment and infrared thermography techniques. However, the success of these programs and the extent of their implementation was highly dependent upon the initiative of one individual.
A Site Quality Verification Audit identified that the torus high level switches had not been tested as recommended by the RCM evaluation and concluded that Quad Cities staff did not adequately review the RCM report.
2.2.7 Positive Observations l Plant outward appearance and housekeeping were good due in part to initiatives ;
such as the RHRSW pump room restorations, mechanical pump seal installations, l and packing improvements. The site had few visible piping, valve, or pump _
leaks. The' appearance of the EDGs was good and modifications to reduce fuel oil leaks had been completed. The licensee had made several improvements within the last two years. Upgrading the 480-volt bus breakers with solid-state trip devices was a positive initiative. Improvements in maintenance l training were also recently implemented. Discussions with ma;ntenance i technicians indicated that training was being conducted. Training facilities '
had a number of mock-ups and state of the art calibration equipment.
25
Radiological housekeeping practices and controls were good. Few areas were significantly contaminated, and most radiologically controlled areas were of low radiation levels. The licensee's total man-rem exposure was at an acceptable level. However, the licensee's exposure levels may increase substantially when CM activities are performed on the degraded equipment.
(There were several instances noted in this report where poor maintenance management controls and documentation contributed to unnecessary exposures because of repetitive work).
2.3 Engineering and Technical Support The team evaluated, in-depth, the residual heat removal (RHR) and RHRSW systems, and other significant equipment issues. The team also evaluated the effectiveness of the engineering and technical support function by reviewing engineering evaluations, plant modifications, operability assessments, root cause determinations, and plant and corporate support.
Engineering had failed to effectively address plant equipment degradation problems associated with excessive vibration that had persisted over the plant's lifetime and had failed to fully evaluate other equipment design issues. Site Engineering did not always support the plant, and Corporate Engineering was not sufficiently involved in site problems. The team had some concerns about plant modifications regarding timeliness of implementation and quality. Some recent initiatives demonstrated good engineering capabilities.
2.3.1 RHR Equipment Problems Resulted in System Degradation The RHR system was degraded in several areas largely due to valve and vibration problems. In addition, design deficiencies had recently been' identified by the licensee and NRC inspections. A large number of problems in the RHR system had not been fixed as exemplified by 408 pending RHR system work requests. The licensee was not fully aware of the impact of system degradation and unreliability because a cumulative assessment had not been performed, the operability assessment process was weak, and design-basis documentation was lacking in some cases. The Vulnerability Assessment Team (VAT) findings had not explicitly been considered in the development of the draft Individual Plant Examination (IPE). The team also found evidence indicating similar conditions existed in other systems. As described in the <
previous two sections, the Quad Cities staff was tolerant of degraded equipment and delayed fixing equipment problems.
A_large number of problems were associated with motor-operated valves'(MOVs) I in the RHR system, most notably vibration. At least two M0Vs failed because of loose torque switch adjustment screws. One of these failures occurred-while the team was on site. Several MOVs failed due to broken or loose bolts.
Few root cause assessments regarding vibration-related failures had been '
performed.
Quad Cities had a high percentage of gate valves of solid-wedge design in the RHR system that industry experience had shown to be vulnerable to thermal binding. In October 1989, a licensee review of thermal binding operating experience in the industry concluded that since thermal binding failures had no prior history at Quad Cities, no action was warranted. A shutdown cooling 26
- - . _ _ _ _ - _. ____-____ _ __ _ _ - ______2 ___-_2__
suction isolation valve failed at Quad Cities in 1990 due to thermal binding.
Following MOV failures in May 1993 at LaSalle, thermal binding and pressure locking screening criteria to identify susceptible MOVs were written for all Commonwealth Edison Company (Ceco) sites but had not been implemented at Quad Cities.
Another problem that had the potential to affect MOVs in other systems was the use of certain magnesium alloys in the rotor bars of actuator motors that were used in high-temperature environments. One alloy type (referred to as "Dow-M") was susceptible to intergranular stress-corrosion cracking and oxidation / corrosion and had an increased susceptibility to corrosion in high-temperature and high-humidity environments. A failure in 1988 of a reactor recirculation loop isolation valve at Quad Cities was of the Dow-M alloy, and failure was attributed to catastrophic stress-corrosion cracking of the rotor.
~
As of September 21, 1993, the licensee had identified 16 MOVs (10 RHR MOVs) that were in use with Dow-M alloy rotors. The licensee had changed the purchasing' specification to prevent reordering of the Dow-M alloy bar rotors.
However, the licensee had no plans to inspect or replace any of these 16 Limitorque motors. The Maintenance staff had earlier proposed a 10 CFR Part 21 report to Corporate Engineering, but it was evaluated as not meeting the threshold for reportability.
Valve misapplication apparently caused or contributed to some MOV problems.
For example, the RHRSW heat exchanger bypass MOVs were originally air-operated valves that were converted to MOVs. However, when motor operators were mated to these valves, the valve stems were undersized for this type of duty, contributing to several bent stem failures. Also, globe valves, such as the RHRSW heat exchanger outlet valves (RHR MOV 5), and the torus cooling return motor-operated valves (RHR M0V 36A and 36B), were used extensively for throttling but were not fitted with anti-cavitation trim. Some of these .
valves had failed as a result of cavitation-induced vibration.
As discussed under Maintenance and Testing in this report (Section 2.2.1), the 2C RHR pump motor, the ID RHRSW pump, and the suction piping for 2D RHRSW pump all experienced excessive vibration. Also, a number of RHR pump suction
, relief valves and RHR heat exchanger tube-side relief valves had routinely failed during inservice testing (IST). The root causes were not appropriately identified and corrected. Trending was not performed even though it was recommended in a 1992 corrective action report (CAR) and the high failure rate of these valves (75 percent over the last two outages) did not trigger attention to these valves.
2.3.2 Engineering Failure to Effectively Address Plant Vibration Problems Quad Cities had equipment degradation problems, in addition to those in the :
RHR system,. associated with excessive vibration that had persisted for years. '
They included failures of Electromatic relief valves (ERVs); mechanical ;
snubbers; core spray (CS) system pumps, valves and piping; and the Unit I diesel generator cooling water (DGCW) line, as well as excessive vibration in-
. the high pressure coolant injection (HPCI) steam discharge line to the torus.
27
An excessive number of failures of ERVs, part of the automatic depressurization system (ADS), had occurred. Vibration of the main steam lines (MSLs) had been the apparent cause of five scrams and numerous deviation reports. During the last 3 years, three ERV failures had occurred at each
' unit. An investigation of a 1977 failure led to the conclusion that a 140-Hertz low-amplitude vibration in the MSLs was due to MSL routing and hanger arrangement.
Other failures due to vibration included mechanical snubbers. During the last 3 years, six Unit 2 snubbers had failed. Since 1980, 12 snubbers had failed, 7 directly on the MSLs or on ERV discharge lines. A 1993 license event report (LER) attributed failure of two MSL snubbers during testing to dried grease and high vibration, respectively. A 1992 LER attributed failure of five snubbers during testing to dried grease. No root cause had been found for the grease hardening. The vendor was performing grease tests, and the licensee planned vibration measurements on the MSLs in the drywell early in 1994.
The licensee found that the core spray (CS) system full-flow test valves (CS.MOV 4A and 48) were experiencing vibration as a result of cavitation in the downstream orifices. Vibration had created cyclic impact loading on the valve actuator motor end bell bolts and caused these bolts to vibrate off the actuator motors. In addition, these full-flow test valves were experiencing torque trips when going closed against the pressure of a running CS pump. The licensee performed vibration testing on the CS piping during the diagnostic evaluation (DE) and identified at least one case where the piping displacement was 0.153 inch instead of an allowable displacement of 0.044 inch. The licensee reported that planned detailed analysis would be necessary to determine the impact of these displacements. Vibration tests on another portion of the piping measured 10 g of acceleration. Displacement data were not obtained at this latter location because the licensee believed that the greatest displacement (and highest piping stress) was at the locations discussed above. A modification was planned to remove the flow-reducing orifice and install anti-cavitation trim in the CS test return valve.
The licensee's VAT report identified high vibration on the upper motor bearings in CS pump 1A as a concern. The VAT also identified a problem with excessive vibration of the Unit 1 emergency diesel generator (EDG) cooling-water line downstream of the Unit 1 EDG heat exchangers. Although these '
issues had been identified in November 1992, no actions had been taken at _the time of this evaluation.
During observation of a HPCI surveillance test, the team observed that the torus experienced significant displacement which'the licensee had not measured.' The displacement was attributed to unstable steam condensation of-a main vent downcomer in the HPCI steam discharge to the torus, defined as
" condensation oscillation" and/or " chugging." A modification was planned to replace the straight-end exhaust pipe with a sparger to improve the steam condensation performance of the system.
2.3.3 Engineering Faiiure to Fully Evaluate Equipment Design issues Engineering failed to fully evaluate and resolve equipment design issues in a-timely manner. Safety-related electrical switchgear were susceptible to failure from breaks or leaks in overhead non-seismic-fluid piping. Other 28
equipment issues included a safety-related l'oad that was not considered in the EDG loading calculations and inoperable standby liquid control (SBLC) system heat tracing that had not been fixed or properly evaluated.
2.3.3.1 Potential Loss of Safety-Related Switchgear Due to Seismic Concerns Safety-related (Class IE) electrical switchgear was located in the turbine building, and non-seismic lines carrying water were routed over the switchgear. The following Class 1E buses were susceptible to failure from breaks or leaks in overhead lines: 29, 13-1, 24-1, 14-1, 13, and 14. A 1989 evaluation of seismic Category II/I issues concluded that the design of non-seismic piping was such that it would not break during a design-basis seismic event. In fact, in 1991, water from an overhead line caused Quad Cities Unit I to shut down. As a result, a tarpaulin drip shield was placed over bus 14-1 and was later removed. All of the buses were open at the top for ventilation. This problem had been re-identified in the VAT report, but no additional action had been taken. The licensee planned to address the issue in the Seismic Qualification Utility Group effort scheduled for completion in 1996.
The RHR pump rooms were susceptible to flooding from the stairwells and unsealed ceiling piping penetrations. A source of flood water was non-seismic piping at and above the 595-foot elevation. This problem was identified in the VAT report, but no effective action had been taken because the 1989 evaluation cited above had also shown that these lines would not fail. The licensee provided draft documentation to the team indicating that this issue was screened as part of the IPE and was judged to be a sufficiently low probability event such that it did not need further evaluation.
2.3.3.2 Emergency Diesel Generator Loading Concern The intent of the original design was to manually load the control room air conditioner compressor on the swing EDG sometime into the postulated loss-of-coolant accident (LOCA) and before the control room temperature exceeded its design basis. The control room heatup time, the limiting time that the loading would need to be completed, had never been calculated and 15 minutes had been suggested by Engineering as conservative. In addition, no calculation had been completed to show this additional load could be put on the swing EDG prior to shedding some other load. Neither Operations nor Engineering seemed aware that this problem had not been solved until it was brought up by the team. The licensee performed calculations while the team was on site which indicated that loading of the compressor onto the swing EDG, prior to shedding another load, could not be assured without a more rigorous calculation. The licensee was reevaluating the issue when the team left the site.
2.3.3.3 Inoperable Heat Trace Line on Standby Liquid Control System j The heat trace line from the Unit 1 SBLC tank to one of the pumps had been inoperable since December 1990. The licensee had shown by calculation that the SBLC system would remain operable with the degraded heat tracing circuit j given certain temperatures and tank concentration limits. The team's review l of these calculations indicated they might not envelope actual off-normal 1 conditions such as ambient temperatures of 60 'F combined with tank :
concentrations above 15.6 weight percent. While the team was on site, the I 29 !
licensee initiated a site engineering service request (SESR) to generate-guidelines, and procedural enhancements to assure off-normal system operation.
2.3.4 Sita Engineering Did Not Always Support The Plant The site engineering staff at Quad Cities, Site Engineering and Construction, was reorganized into two groups in June of this year, Plant Support and Design. The former was responsible for day-to-day plant engineering support.
As part of this reorganization, approximately 5 corporate personnel were transferred to the site. Plant Support Engineering had not developed a proactive role on site. Engineering management was not fostering this type of role. Operability assessments and root cause determinations were often weak or nonexistent. Operating experience reviews (0ERs) were not comprehensive and failed to address problems and recommendations. In addition, Plant-Support was not effective in correcting significant longstanding or emergent plant problems. When involved, Plant Support was often narrowly focused and omitted broader implications. As a result, Maintenance and Operations management had limited expectations regarding engineering support. The NRC had previously identified that weak engineering support resulted in equipment operability issues not being promptly addressed at Ceco licensees (reference Commission paper SECY-92-228, dated June 25,1992).
Althoughthesystemenginebrsidentifiedanimprovedresponsiveness,the reorganization had not yet become fully effective. The primary mechanism which connected the system engineers and Plant Support engineering organizations was the SESR process. SESRs were not prioritized or trended, and the status of overdue SESRs was not closely monitored. Systems engineers were intended to be the system focal point and thus prepared most SESRs. (The NRC has recently identified significant weaknesses in the system engineering program, and the licensee was working to make program improvements.) However, because of the system engineers' extensive responsibilities and relative inexperience, they frequently did not identify either repetitive problems or individually significant problems which required additional engineering.
support in order to effect adequate and timely corrective actions. In addition, Plant Support engineering staff and management did not actively seek to address significant issues or repetitive problems across systems that may have required a more intensive problem solving approach than that which resteo with a single system engineer. Examples included problems with site-wide vibration, HPCI pump alignment, ERVs, feedwater check valves, and snubbers.
The IST program was an additional significant example of an issue which lacked effective engineering involvement.
The team also noted examples of communications between organizations which did-not define the issue well and did not ensure an appropriate response. These problems were exacerbated by the number of interfacing organizations at Quad Cities. In the entrance meeting for the diagnostic evaluation, the licensee said the large number of contracting . firms previously used by site engineering had been consolidated into a single prime architect / engineer contractor and two to three specialty contractors. However, the team found that up to:seven engineering contractors were still onsite. The site engineering group'had a heavy reliance on contractor support, much of which was offsite.
30
2.3.4.1 Weak Operability Assessments and Root Cause Determinations The team reviewed various engineering operability assessments and evaluation efforts with respect to the quality and adequacy of these documents. The team concluded that most of these engineering operability assessments were lacking in scope and engineering detail, root cause evaluations had not been performed, and proposed corrective actions and appropriate findings were either not included or in error. In addition, the team also reviewed an example of degraded equipment where no engineering operability assessment or root cause determination was performed.
(1) Engineering Operability Assessment #0121779 documented the effects of.
system vibration on the RHR 36A/B MOVs. This assessment was performed by the licensee at the prompting of the team because, in 1992, four sheared yoke-to-actuator bolts and several yoke cracks were found on the Unit 1 RHR 36B M0V. An examination had shown the bolts failed due to fatigue; however, no root cause had been performed to determine why the bolts had suffered fatigue or why the yoke cracked. The yoke was fixed by welding, the bolts were replaced with stronger bolts, and the valve was placed back in service. A report from the licensee's contractor concluded that the allowable fatigue stress for the valve yoke was being exceeded by 46 percent and was "high enough to eventually contribute to fatigue." The report went on to state that if the bolts were not properly torqued or if they became loose, then significantly higher
, stresses would result in the bolts. Engineering's operability assessment, based on this report, failed to mention the conclusions given above. Although a new torquing procedure was used with the stronger bolts, the team was concerned that bolt loosening due to vibration might be an unevaluated root cause that could recur. Two NRC information notices (ins) issued in 1983 and 1985 discussed vibration-related MOV failures at Quad Cities, with a failure of this same MOV discussed in the IN.
(2) The team reviewed Engineering Operability Assessment #184260, "1/2 Diesel Generator Cooling Water Pump Operability Assessment," dated April 16, 1992, which evaluated a design deficiency where the swing DGCW pump fan cooler was only powered from one source. This deficiency during a design-basis accident could render the swing EDG CWP inoperable due to loss of the associated fan cooler. The licensee made a determination of operability based on the operable status of the swing EDG CWP and other equipment. However, the licensee failed to fully analyze the effects of the proposed modification, which corrected one design error and introduced another. Together, these two design errors had effectively rendered the swing EDG inoperable to perform its intended design function for Quad Cities Unit 2 since initial plant-startup. (This issue was responsible for an escalated enforcement action by the NRC staff, EA 93-162 issued on June 2, 1993.)
(3) Several concerns regarding the licensee's operability determination process arose from the-team's review of a problem identification form (PIF) questioning the operability of RHR spring cans. First, the ,
' licensee's inservice inspection-(ISI) group performed field verifications of spring can settings consistent with a new piping analysis in November 1992 and found four RHR spring cans, which were out i of tolerance and had not been reset per the new analysis. No
]
31
operability evaluaticn was initiated to evaluate the spring cans' condition at that time nor was an effort made to promptly reset them in accordance with the new analysis. Second, a licensee review of the work packages raised the operability question on August 20, 1993, but a PIF raising an operability concern was not generated until August 24, 1993.
Lastly, the operability assessment was not performed in accordance with approved procedures, and the appropriate condition was not evaluated for -
operability.
(4) The NRC identified sheared bolts on the full-flow test return MOVs (RHR 34A and 34B MOVs) in June 1992. Bolting problems had been found on 16 RHR system nuclear work requests (NWRs) since 1985. NWRs documented that failed bolts were replaced with stronger bolts. For example, on December 11, 1992, during valve operation test evaluation system (VOTES) testing on the RHR injection throttle valve (1 RHR 288), twelve 1-1/2-inch-diameter actuator mounting bolts were stretched. The cause was found to be improper ungraded bolts, and stronger bolts were installed.
On April 26,1993, all 12 actuator mounting bolts were found to be loose on valve 2 RHR 28B. The bolts were replaced with stronger bolts.
However, no formal operability assessment or root cause investigation was performed.
(5) The licensee had not performed an engineering operability assessment for a scenario in which both units at Quad Cities were vulnerable to loss of both low-pressure coolant injection (LPCI) trains and one of two CS trains during a degraded grid condition. A degraded grid, combined with a single failure of the degraded grid detection circuitry for one of the 4 KV buses, could result in degraded voltage to the associated 4 KV bus and the unit's 480 volt swing bus. The swing bus provided power to both trains of LPCI MOVs. In addition, the CS pumps were powered separately from one of the 4 KV buses, with one pump on the degraded 4 KV bus.
This scenario could result in only one CS train available from the LPCI and CS systems. Although the licensee was evaluating installation of a modification that would alleviate this single failure vulnerability, it contended that this accident scenario was outside the plant's design basis. (This issue has been referred to the Office of Nuclear Reactor Regulation.)
2.3.4.2 Weak Operating Experience Reviews and Feedback Industry and_ site OERs performed by the licensee were not comprehensive or timely and failed to address problems and recommendations.
(1) General Electric (GE) Services Information Letter (SIL) No. 531 documented recurring industry problems with Magnetrol level switches used to detect high condensate water levels in HPCI and reactor core isolation cooling (RCIC) steam line drain pots and recommended certain actions for GE boiling-water reactor (BWR) 4/5/6 owners. As discussed in the maintenance section of this. report,-inattention to this SIL may have contributed to the turbine exhaust rupture disk burst at Quad Cities on June 9, 1993.
(2) SIL No. 371, "RCIC Turbine Exhaust Pressure Trip Setpoint," recommended that BWR owners consider raising the turbine exhaust pressure trip setpoint to increase RCIC availability during events such as small-break 32
LOCAs which cause significant containment pressurization. Although this SIL was dated February 1982, it was still "under investigation." .The VAT report of November 1992 also identified this item and recommended that the system engineer initiate setpoint change documentation to increase the setpoint.
(3) The team selected a sample of four NRC Information Notices (91-61, 92-23, 92-41, and 92-70) related to Limitorque operators for MOVs to determine if Quad Cities had a process to screen operational experience.
In each sample, the Regulatory Assurance records on information notice screening, assigning action, and closing gave no evidence of actions taken or the reason for closure. The OER program had recently been transferred from the corporate Safety Review Department to the stations with no additional resources provided to the stations.
2.3.5 Corporate Engineering Was Not Sufficiently involved in Plant issues Many corporate engineering' managers were not sufficiently aware of Quad Cities' problems. Corporate had not provided the necessary leadership tc effect aggressive attention to VAT issues. The licensee's design basis documentation program was hot comprehensive, and the draft IPE was not plant specific in some areas.
2.3.5.1 Lack of Ownership and Accountability for Plant Engineering Problems Corporate engineering was involved with generic programs, but did not focus enough on specific site issues related to the implementation of these programs. Programs that required more of the corporate interface with site engineering; e.g., Electrical Distribution _ System Functional Inspection (EDSFI) followup, seemed to be more successful than those that required more corporate interface with plant personnel; e.g., M0V testing. One complaint of Quad Cities personnel was that in Corporate engineering, everything was done on a corporate basis (i.e.i what was good for one site was good for all).
However, Quad Cities had not assumed enough active participation in the implementation of generic programs to ensure that site-related specific issues were managed and resolved.
Corporate engineering controlled the funds for all of the generic programs.
This did not provide an incentive for Corporate engineering to support the plants nor an incentive for the plants to seek corporate assistance. In addition, the corporate engineers did not fill out time cards, so management could not accurately track where time and-money were expended. This contributed to a lack of clear ownership and accountability for the programs.
For example, the MOV test program in response to NRC Generic letter (GL) 89-10, " Safety-Related Motor-0perated Valve Testing- and Surveillance,". had corporate sponsorship but neither corporate nor Site Engineering was assuming a leadership role at Quad Cities. As a result, the site MOV program suffered from a lack of leadership as indicated in Corporate Nuclear Engineering and Technical Support meeting minutes'of July 26, 1993: "The most important !
problem at Quad Cities is the lack of upper management understanding of the entire NRC GL 89-10 program requirements and commitments" and " Quad needs to l have a Driver of the MOV program from a management level high enough to remove '
obstacles and assist in taking 89-10 to completion." Only 6 MOVs had been
{
33
4 reported as tested under this program, and two of these valves had failed their initial tests. In a letter to CECO, dated September 9, 1993, NRC discussed a number of concerns which in E J that Ceco had not satisfied commitments made to the NRC to complete
- G9-10, had not incorporated significant M0V findings into the proper methodology for ensuring adequate M0V sizing, and continued to use an outdated valve factor of 0.3, whereas the industry generally use. 0.7 in its methodology for sizing and setting many gate valves.
2.3.5.2 Lack of Aggressive Actions on Issues Identified in the Vulnerability Assessment Team Report As part of CECO's Mature Plant Improvement Initiative, the Vice President of Nuclear Oversight and Regulatory Services formed the VAT to identify vulnerabilities at Quad Cities and assess their safety significance. The VAT, which was comprised of 6 individuals, 3 of whom were from the site, issued a report in November 1992, identifying 53 vulnerabilities and over 80 observations. Although this effort was a significant safety initiative by Ceco corporate management, it was not brought to the team's attention during the entrance meeting or initial onsite activities. The team only learned of the VAT effort and its report at the end of its first week onsite during routine interviewing of staff personnel.
The team concluded that the VAT report was a comprehensive and useful documentation of safety-significant problems at Quad Cities. However, most senior managers had not read the report. Its usefulness also had been diminished in several other ways. The team concluded that the VAT's ranking system did not accurately reflect the issue's significance and status. No individual issues, such as' the potential of operation above rated power, were identified as significant. In addition, the cumulative effect of the issues was not assessed. The view depicted by the rankings, in combination with the understanding that not many new items were identified and that all were being addressed, contributed to the failure of many site managers to read the VAT report.
Management had not communicated its expectations regarding the' priority VAT implementation should receive and how the implementation of these items was to be managed. For example, the VAT report identified the Dresden-based Systematic Evaluation Program (SEP) review done f' Quad Cities.as a generic ,
item. The report indicated that an action plan would be developed by August 31, 1993, to address the Quad Cities SEP topics. The action plan had not been developed as of September 24, 1993.
Some issues were not given a higher priority because the' licensee maintained they were outside of the plant's design basis or not in Technical Specifica-tions. The report repeatedly used rationalization such as ."this meets Our original design basis and so don't need to fix" and "the failure has not occurred yet therefore no action is necessary." In addition, no criterion'was utilized in determining whether an item was a vulnerability or an observation.
A PIF had not been utilized for any of the VAT items. Utilizing the-PIF process was the only way to obtain a formal root cause evaluation on site, and, in the case of observations, ensure their closure, since observations are l not entered in the nuclear tracking system.
34
i
, =
l l
A VAT update report was issued in August 1993. Once again, the team was not .
provided this report by corporate or site management until the final week of '
the evaluation. This report updated the status of the items in the original VAT report and identified 6 additional vulnerabilities and almost 50 new l observations. Almost half of the vulnerabilities-in the VAT report had no corrective action indicated, and the update report still indicated no action on each of these items.
2.3.5.3 Design-Basis Documentation Program Not Comprehensive Corporate engineering had a design-basis documentation (DBD) effort under way to reconstitute the plant's design basis. Engineering judgment had been used to a large extent rather than rigorous calculations in areas in which original calculations were not available. The team concluded that the DBD effort was an improvement, but it was basically a document-gathering effort, and important information abe'It the system design was not included. The team requested a list of' systems that had completed DBDs for the purpose of aiding' in the decision to choose a system for in-depth review. The licensee's initial list to the team indicated that tht DBD review for the RHR system was complete. However, when the team announced that the RHR system was selected for in-depth review, the licensee subsequently stated the BDB was not completed. The team found the licensee's efforts in performing operability determinations, engineering evaluations, and root cause determinations were sometimes ineffective partially due to a lack of complete design-basis documentation. The licensee allowed the contractors to retain ownership of the design-basis calculations.
2.3.5.4 The Draft Individual Plant Examination Was Not Plant Specific In Some Cases The licensee's draft IPE assumed that the original design margins and equipment important to safety were being maintained so that generic industry reliability data could be used in many areas. However, since-the operability determination process was flawed, some assumptions and data used in the draft IPE might not reflect actual plant conditions and risk to plant safety.
GL 88-20, " Individual Plant Examination For Severe Accident Vulnerabilities,"
requested that licensees participating in the IPE program, examine and understand the design, operation, maintenance and surveillance aspects of plant operation and ensure that the IPE and/or PRA reflected the current plant design and operation. As a result of a cursory review of the IPE, the team was concerned the licensee might not be meeting the-intent of GL 88-20. For example, M0V failure rates which resulted from dynamic testing of MOVs under the GL 89-10 program appeared to be more representative of current M0V status than the MOV failure rate value assumed in the draft IPE. In addition, the draft IPE did not consider appropriate plant-specific failure data, significant plant degradation affecting numerous safety systems (such as plant-wide vibration problems with pumps and valves), a high number of snubber and relief valve failures, and certain plant. design features which were considered weaknesses, such as past single failure problems with the swing EDG and the LPCI swing bus design.
The licensee's VAT report, which contained many of the above described weaknesses, also was not explicitly considered in the IPE. The licensee indicated that the draft IPE was only intended to detect vulnerabilities in the original plant design and not to account for " cyclic" aspects of the plant 35
design. However, the team noted that the relatively high number of recent safety system failures and number of longstanding equipment degradation issues at Quad Cities indicated that equipment degradation had occurred over a significant period of time and not as a result of " cyclic" plant performance.
2.3.6 Modification implementation Was Often Untimely and incomplete Several safety-significant modifications were not implemented in a timely manner. There was a large backlog of 512 outage and non-outage modifications that had been approved by the Modifications Review Committee. This backlog was being reported to site and corporate management as 225 largely because the same modification on multiple equipment was counted as one. modification in some instances. The backlog was also being shown as trending down; however, this trend was uninterpretable because it was unknown how much of the downward trend was due to minor design changes that were being removed from the backlog and modifications that were being cancelled. The team identified several concerns regarding the quality of modification packages and 50.59 reviews.
2.3.6.1 Untimely Implementation of Safety Significant Modifications Approximately 3 percent of the open modifications were approved more than 7 years ago; 15 percent were older than three years, and 67 percent were two or less years old. Of concern to the team was approximately 25 modifications with safety consequences that were more than a year old since approval. For example, EDG engine modifications to install fuel cutoff valves, flexible fuel oil hoses, and flanged pump connections to reduce fuel leakage (a high-priority VAT item) had been approved since early 1992, but implementation was not planned until late 1994. Modification MR4-2-91-010 was supposed to complete Modification M4-2-82-049 (more than 10 years old) that seismically mounted the 250-V de battery charger #2, rerouted conduit, and replaced circuit breakers and power cables. An open 1990 modification required the changeout of fuses and breakers on existing 125 V de batteries.
Unit 2 had experienced repeated spurious Group I isolations following a scram; the last occurred on June 13, 1993. Although at least three occurred during the 1990-1991 time frame, an SESR to evaluate special tests for possible fixes was not prepared until January 3, 1992, a modification request was not prepared until September 15, 1992, and the modification was not scheduled to start until an outage in early 1994.
Repeated failures during local leak rate testing (LLRT) of the HPCI turbine exhaust valve were discussed earlier in Section 2.2. A modification was prepared to enhance the exhaust configuration, including the sparger installation. The Unit I modification was deferred from QlR12 and was scheduled for Q1R14. The Unit 2 sparger modification was not installed during Q2Rll, was deleted from Q2R12, and was scheduled for Q2R13.
2.3.6.2 Failure to Perform 10 CFR 50.59 Reviews .
The team found instances where the licensee had changed or altered plant operation as described in the Updated Final Safety Analysis Report-(UFSAR) and had not performed a 50.59 review. For example, one of the two pumpback air 36
compressors (described in the UFSAR) had not been operational for the past 9 years. The team found that when the remaining compressor failed, the normally-closed drywell-to-torus vent valves were routinely opened to provide venting of the torus (see Section 2.1.1 for further discussion). The licensee -
had not performed a 50.59 evaluation with regard to continuous use of only I compressor or the routine use of opening the normally-closed torus vent valves. Another example wa.s the change to the MOV stroke times for the RHR 28 M0Vs (see Section 2.2.1). The licensee had administratively changed the stroke time requirements from 25 seconds to 90 seconds for all of the RHR 28 MOVs without performing a 50.59 evaluation. In addition, numerous RHR yoke-to-bonnet bolt modifications and the use of Belzona in safety-related pumps and valves (see Sections 2.3.3 and 2.3.7, respectively) were other examples where the licensee failed to perform a 50.59 evaluation.
In addition, Quad Cities had an ongoing history of incomplete 50.59 reviews and modification packaaes. The lack of design-basis documentation also hindered the licensee's 50.59 reviews. The team found deficiencies in the 50.59 evaluation for a minor modification to replace the 60 ft-lbf motor of the 1A RHR torus cooling and the 1A torus test return valve actuators with an 80 ft-lbf motor. The 50.59 evaluation did not consider the increased thrust capability of the actuator' as a potential adverse affect on certain valve components.
2.3.7 Ceramic Fill and Coating Compounds Not Controlled Several work requests and modifications included application of a ceramic fill (Belzona Ceramic-R) and a ceramic coating (Belzona Ceramic-S) to the interior of safety-related pumps and valves. Although the licensee stated. it strictly controlled the use and application of other fill and coating compounds such as Fermanite, Belzona applications received little Site Engineering evaluation, and the work process was not well controlled or monitored.
In reviewing licensee evaluations regarding the application of Belzona and in discussions with the vendor, the team found (1) the application process t includes several critical vendor application and service requirements; (2) formal training or vendor supervision is highly recommended; (3) Belzona contains leachable chlorides, fluorides, and sulfur compounds which neither the vendor nor licensee evaluated; (4) Belzona should not be used in dry applications exceeding approximately 450*F - 500*F and wet applications exceeding approximately 180*F - 200*F; (5) Belzona loses its structural properties when exposed to radiation and acids; and (6) other vendors manufacture similar products which, like Belzona, are widely used.
An SESR of January 3,1992, requested approval of Belzona Ceramic-R and Ceramic-S for repair of DGCW casings. Usage was approved by site engineering-with the provision that coatings should be installed-according to the vendor's recommended procedures. The work package for #2 DGCW pump completed January 30, 1992, did not contain any vendor cautions, such as maintaining.
minimum casing wall thickness and that buildup should not exceed that necessary to achieve original casing contour. No mapping of the erosion, casing thickness measurements, or the amount of Belzona applied were available in the work request.
'. 37 i
Belzona was also used to repair steam cuts along the seating surfaces of the seat ring and on the main body of the HPCI testable check valve (2-2301-7) on October 21, 1985. Because of the potential for the valve to experience temperatures in excess of approximately 300 'F in routine conditions and excess of 450 *F in emergency conditions, the licensee obtained a memorandum from the vendor stating that the application may work if the Belzona was reinforced by being held between two flange faces. However, neither the ,
evaluation nor the vendor's memorandum addressed a non-reinforced application '
such as the repairs made to the valve body. In addition, engineering cautions relating to mapping, thickness application, and minimum wall thickness were not included in the evaluation or work package. The licensee finally properly repaired the valve in March 1992 after it was identified (during a station valve inspection program) that the valve had been modified. The valve vendor recommended that a special weld process be performed to build up and machine the total pressure seal area because the as-found clearances of the pressure seal area were beyond the valve vendor's specifications. However, although the work has been performed, the licensee was unable to provide the team an evaluation or work package which approved or documented the valve >
modification. '
The team concluded that there was a lack of management involvement on the part of Maintenance and Engineering in the performance of temporary repairs and modifications of safety-related equipment. Several examples of insufficient evaluations, uncontrolled work processes, inadequate documentation of work !
performed and inadequate tracking of temporary repairs were identified. These weaknesses resulted in the potential for performing inappropriate temporary repairs or modifications rather than a special process as required.
2.3.8 Positive Observations P
The team noted several issues that the licensee was recently addressing effectively including RHR heat exchanger testing and EDSFI followup. The team concluded that effective response to these issues indicated that Quad Cities and Ceco did have the necessary technical resources available to address plant problems . However, the team also concluded that these resources were applied only after the issue had received significant attention from the NRC and/or industry groups.
2.4 Management and Organization The team evaluated how management performed in the areas of organizing, planning, direction and control, and problem resolution. The performance of -
corporate management was evaluated in those areas in which station performance-was affected. The team also evaluated the timeliness and effectiveness of the corrective action processes, the performance of the quality oversight organization, and the status at Quad Cities of those actions initiated by the Dresden and Zion diagnostic evaluations (des).
The team conducted 110 formal interviews, observed numerous meetings, reviewed !
relevant documents,-and used the hardware and process issues identified in the l Operations, Maintenance, and Engineering sections of this report (Sections 1 2.1, 2.2, and 2.3, respectively) as a basis for evaluating management- :
effectiveness. I 38 l I
The team found that corrective. action processes were neither timely nor effective and that the site quality verification (SQV) group was also ineffective. The SQV group had previously identified a number of issues but had not been fully successful in resolving the problems it had identified.
The team found that the number of previous improvement programs initiated by corporate management were not successfully improving station performance.
Site management did not always demonstrate its ability to organize, plan, execute, evaluate, and resolve issues.
Management had not taken advantage of the Quad Cities strength, which was the capabilities and dedication of its staff. The team found that the recent significant management realignment was needed. >
2.4.1 Untimely and ineffective Corrective Action Processes-Ineffective site management of corrective action processes and site management's failure to correct known deficiencies in these processes led to a number of equipment problems, including issues identified by the Vulnerability Assessment Team (VAT). The failure to trend and analyze for repetitive equipment or performance problems, shallow (or nonexistent) root cause analyses, failure to perform operability and safety impact evaluations, and lack of aggressive problem resolution resulted in short-term rather than long-term solutions to station problems. Many of the process and management problems had been identified previously through internal, industry, and NRC assessments, including the May 1992 Event Assessment Team (EAT) Report and the 1993 corrective action self-assessment. Management had taken few actions to resolve concerns identified in these assessments. Site management acknowledged previous failures in assuring that appropriate corrective actions were taken to identified problems.
The licensee had implemented a new corrective action process, the corporate Integrated Reporting Program (IRP), 2 days before the team's first onsite period. Before the IRP was established, multiple department-level programs existed at Quad Cities. This fragmented approach to problem identification ,
resulted in redundancies and inefficiencies within site and corporate programs. Identified problems could not be tracked or trended effectively as manual analysis was needed to identify trends. Individual departments independently developed lists of equipment issues for resolution, without direction or coordination from site management. Previously there were 22 '
individual corrective action programs, and 7 remain. The number-of programs '
used by the various departments to track known equipment problems clearly illustrates this fact (see Operations and Training, Section 2.1.2)
The IRP was developed initially by Commonwealth Edison Company (Ceco) at the Braidwood Nuclear Power Station in 1987. In 1989, Ceco initiated a task force to develop a corporate-wide IRP. Industry organizations had reviewed this program and, with permission from CECO, recommended this program to other licensees. In April 1991, the corporate nuclear oversight' group recommended to senior corporate management that this program be implemented at all of the ,
CECO facilities. However, senior corporate management decided not to implement the program at that time.
At the time of the DE, management accelerated the implementation of the IRP.
The team found that most site personnel had not received training associated 39 l
with either the IRP or the new Problem Identification Forms (PIFs), which resulted in a number of implementation errors and a reluctance to report problems. The team also noted that existing failure data from departmental corrective action processes were not being transferred to the IRP. Therefore, trending was not accomplished until sufficient data existed in the new program.
Failure to implement an effective corrective action program in a timely manner challenged the adequacy of previous root cause evaluations. The team found many examples of inadequate root cause analyses related to equipment failures, particularly in maintenance. The licensee had previously identified inadequate root cause analyses as an area of concern. However, a formal root cause methodology had only recently been established. Additionally. the Process Experts Group (PEG) did not fully meet corporate staffing recommendations.
The new IRP established an Event Screening Committee to review all PIFs, assign severity levels (1 through 4, with 1 being the most severe), and assign responsibility for resolving problems. The PEG was required to perform root cause analyses on Severity Level 3 PIFs and assist with root cause determinations on Severity Level 1 and 2 PIFs. Root cause determination was not required for Severity Level 4 PIFs. These lower level problems were expected to be escalated to higher levels, thus requiring a root cause analysis, when adverse or repetitive trends were identified. However, as previously noted, no trending was performed. Therefore, repetitive equipment problems were not identified nor resolved. The team identified several examples of repetitive problems that occurred during the evaluation that were not escalated to site management.
In response to the team's observation that issues were not being escalated, the licensee completed a review of only Level 2 root cause analyses from 1989 to the present. This did not include the number of deficient Level 3 root ,
cause analyses that existed, nor did it provide sufficient evidence that the root causes of many identified problems had been analyzed. Thus, this review did not provide a realistic assessment of the problems.
P In many cases, failure to establish and maintain accountability for problem resolution contributed to continuing problems with equipment and personnel.
For. example, in March 1993, the licensee. found that the number of discrepancy records (DRs), a departmental corrective action device previously used by the Quality Control group, associated with. procedure violations had increased 181 percent in 1992. This was reported to senior site management and a DR was written to document the substantial increase. During this evaluation, the e team' found that almost no corrective actions were implemented and actually observed several members of the operations staff fail to follow procedures.
In addition, the team found that the same report indicated the. following: ,
(1) problems involving work practices had increased 100 percent, (2) personnel errors had increased 144 percent, and (3) equipment failures had increa. sed 190 percent. Despite these large increases, no corrective actions had been-planned.or initiated.
i 40
2.4.2 Ineffective Quality Oversight The Nuclear Oversight groups reported to the Vice President and Chief Nuclear Officer (VPCN0) through the Vice President of Nuclear Operations (VPN0). The SQV Group reports to the VPN0 through the corporate quality verification function. The team reviewed a number of reports applicable to Quad Cities performed by corporate and site Quality Verification Groups such as, comparative audits, shutdown risk assessments, density charts, daily statistics on safety system availability. These reports were usually of good quality. Weekly conference calls were conducted between the corporate and site groups. '
Independent oversight was also provided to the Board of Directors by the .
Nuclear Oversight Committee (NOC). The NOC was a small group of consultants that periodically reviewed activities at each site. The team was informed that no minutes of meetings or correspondence were available. The team also found that a Nuclear Safety Review Board (NSRB), reporting to the VPN0, was being formed to provide independent peer oversight. The Quad Cities NSRB planned to hold its first meeting in early October 1993. Regardless of these ,
initiatives, the nuclear oversight function had not been effective in improving Quad Cities performance.
l The corporate and site quality verification groups were ineffective in raising problems and other concerns to the appropriate management levels to ensure adequate resolution. Usually, the group had not assessed or understood the collective significance of their findings. The team noted that a number of the issues identified by the team had been previously identified by SQV.or other self-assessment initiatives, such as the VAT and the EAT but had not been fully resolved. SQV had repeatedly expressed specific concerns about ineffective corrective actions, lack of attention to detail, failure to follow procedures, and failure of site management to communicate expectations and directions during normal audits at Quad Cities. SQV concerns were documented in reports that were routed to site management and SQV corporate management.
Little, if any, improvement in these areas was noted until recently. Monthly summaries were routed to all senior corporate managers; however, these rarely-contained sufficient detail to alert senior management to' ongoing concerns.
Corporate management had weakened SQV by staffing reductions and redirection of efforts. SQV's corporate management had' eliminated a reporting mechanism to elevate those concerns that remained unresolved for more than 60 days.
Since a reorganization 3 years ago, corporate quality verification stopped the practice of evaluating SQV. implementation effectiveness. !
In October 1992, CECO management mandated a reorganization of the quality oversight organizations as part of an overall corporate restructuring. Two- ;
thirds of the personnel assigned to the SQV Superintendent were eliminated. '
These reductions were not matched in other departments, nor were they in line with realistic needs. In addition, the Onsite Nuclear Safety Department, which performed the independent safety engineering group (ISEG) function (not~
required by Quad Cities Technical Specifications), was combined with the SQV I group. Further, the SQV Superintendent was required to perform audits in "
order to meet Technical Specification requirements. This last change reduced the perceived authority and position of the SQV Superintendent.
4
)
l 4 41 i
l
, - _ = . _ _ _ - _ _ _ _ _ _ .
At the same time, corpceate management directed SQV to become more involved in site activities, further diluting SQV resources and confusing the function of the organization. Auditors were assigned to participate in such site activities as event investigation teams and event review / screening committees and were expected to audit these activities concurrently. The team observed two event screening committee meetings and noted concerns involving dual responsibilities as well as during interviews and reviews of event investigation team reports. The team also found that some audit findings replicated-committee / team findings in which the auditor had participated.
These changes, coupled with SQV corporate management's failure to facilitate elevation of problems and other concerns, resulted in an ineffective quality oversight organization. The team verified that all audits required by the Technical Specifications were performed in a timely manner; however, the quality and depth of these reviews was limited and problems were not always resolved appropriately. For example, in mid-1992, the report on reliability-centered maintenance (RCM) identified several issues related to failure to test various instruments. In November 1992, the VAT report noted that the RCM program had been abandoned and that high-pressure coolant injection-(HPCI) drain-pot switches were " vital features." In April 1993, SQV identified that closed cooling water storage tank (CCST) and torus level switches had never been tested and issued a corrective action record (CAR) to document and track this issue. SQV failed to note that the RCM program had been abandoned. In May 1993, the response from site management to the CAR stated that this w:s an isolated event and that the review process for the RCH was adequate. On June 1, 1993, SQV asked the licensee to reconsider its response. Only after the Unit 1 HPCI rupture disk failed and personnel were injured did SQV elevate this issue to a Level 1 finding. At the time of the DE, SQV and corporate SQV were still unaware that the RCM program had been abandoned. A second example involved the identification in the VAT report that the chain hoists over the HPCI turbines and pumps were not secured and represented a potential threat to safety-related equipment during a seismic event. In February 1993, SQV reported that this condition remained uncorrected and recommended corrective action. On September 23, 1993, licensee personnel reported that the hoist for Unit 2 remained unsecured, and the hoist for Unit I had been improperly secured to a relief valve.
At the end of the second onsite DE period, corporate management had completed an effectiveness review of SQV and arrived at similar conclusions. During discussions with the team, corporate management indicated that it planned to reinstate the 60-day reporting mechanism, increase SQV staffing, separate the Independent Safety Engineering Group and quality assurance audit functions, and review the advisability of SQV auditors being assigned dual responsibilities. Corporate management also stated that plans had been made to place higher level SQV management onsite to facilitate elevation and resolution of identified problems.
2.4.3 Corporate Management Failed to Fully Ensure that Lessons Learned from Dresden and Zion Were implemented Just as corporate management failed to transform lessons learned during the Dresden DE to tne Zion plant, the team found that approximately 25 percent of the lessons learned from the Dresden and Zion des were still not implemented 42
at Quad Cities. The licensee had completed a number of actions however, most of the more significant actions had been postponed with management endorsement.
1 Neither CECO or Quad Cities management emphasized the importance of completing Dresden and Zion DE actions. For example, the Dresden DE concluded that there j was no viable plant wide root cause determination and analysis program and the Zion DE concluded that corrective actions were not timely or effective. Quad ~
Cities failed to implement the IRP until August 21, 1993.
Both the Dresden and Zion des found that repetitive equipment problems had not been addressed and there had been no objective evidence that actions resulting from trending of equipment failures had improved station maintenance i, activities. Quad Cities initiated the Recurrent Equipment Problem (REP)
Program in response. As previously noted (Section 2.1.1), this program was i not fully functional and the associated management committee had been inactive i until recently. The licensee plans to list the repetitive equipment problems ;
by the end of 1993.
The Zion DE also found that the licensee had not implemented a System Engineering Program. In addition, closure of several actions was tied to establishing this program. Quad Cities continued to postpone actions to fully implement this program until the end of 1994.
The Zion DE noted that responsibilities had not been clearly assigned at the station, station management oversight and control were weak, and numerous operator workaround problems existed. As discussed in this report, these same factors contributed to the declining performance at Quad Cities.
2.4.4 Corporate improvement Programs Did Not Correct Declining Performance Major corporate organizational restructuring took place in 1987 and 1988'that affected Quad Cities and the other nuclear stations. These changes were largely reorganizations of existing functions and did not alter the fundamental management process. In early 1991, CECO recognized it was experiencing problems and initiated efforts directed at understanding and improving its performance. Quad Cities initiated the 1990-1991 Performance Enhancement Program (PEP), followed by the 1992 Management Plan. In July 1992, the Board of Directors announced integrated actions for fundamental improvement of the Nuclear Operating Division (N00) to include organizational l
realignment. The organizational realignment was completed in February 1993.
I A subsequent reorganization took place in June 1993 to consolidate functions, and another imminent reorganization will affect the nuclear oversight function. The plan announced by the Board of Directors was thorough. The i positions of Vice President and Chief Nuclear Officer (VPCN0) and a Site Vice President at each station reporting to the VPCN0 were created. The Site Vice Presidents were delegated the responsibility for safe and efficient operation of their plants. The VPCN0 had created a Nuclear Operating Committee that met twice per month and served as a forum to maintain consistency and to allow timely sharing of experiences between the six Site Vice Prcsidents and the Vice President of Nuclear Operations.
43
1 l The team concluded that the organizational realignment completed in February L 1993 was a necessary action. However, in analyzing past and present I management, the team identified several areas of concern. The team found that I organizational instability over the last six years had caused a loss of l consistency of purpose and hindered team building at Quad Cities. The reorganizations had also contributed to a loss of focus that diluted progress on improvement initiatives in the 1990-1991 PEP and subsequent' Management Plans. Further, the Reliability Centered Maintenance Program was developed and subsequently suspended without translating pertinent requirements to current programs. Other programs were delayed because of priorities of the new management; for example, the Procedures Upgrade Program and the Design Basis Reconstitution Program.
In the past, the failure of corporate management to hold plant management accountable for hardware improvements and initiatives, and corporate management's endorsement of the expectation that improvement would take place )
slowly, played a role in the failure to cause improved performance. For example, approximately 50 actions from the 1990-1991 PEP remained to be completed, and of the 122 actions completed, some were being readdressed. A typical response to a question regarding accountability was for the person being intervieead to say he had only been in the position a few months, that .
his predecessor was responsible, or that the problem was now the responsibility of another department. The reorganization, therefore, was being used as an excuse for failure'to improve.
Although corporate management had defined plant goals and objectives in broad I terms, and created the detailed plans and pugrams to meet these goals and j objectives, site management retained decision making authority and I responsibility for implementation. Corporate management did not insist on a response or ensure timely implementation of solutions to problems affecting-safety. Therefore, it was assumed. improvements would be implemented. For a example, the IRP and Systems Engineering Program were not implemented in a i timely manner and corporate management did not recognize the delayed action.
The new plan was initiated to improve overall performance. The new management plan attempted to improve accountability by the utilization of- management personnel rotated from other corporate plants not associated with past performance problems. However, the team found that corporate management had not validated its assumption that key site management personnel had the leadership skills to assume their new position. A major change in management philosophy was to hire experienced managers from the industry. An experienced industry manager was recently hired to assume the position of Plant Manager in October 1993.
The present performance monitoring system (windows and supporting analyses) had not always brought potentially significant issues to management's attention until an industry organization or the NRC had raised the issue. The system did not detect that hardware, process, and management problems were uncorrected. In the view of the team, only after the diagnostic evaluation was announced, did many of the problems at the plant surface, or become revitalized. The performance monitoring system also used data which was submitted by the site and was sometimes found to provide information that did not represent the actual workload; e.g., the maintenance and modification backlogs.
44
Because corporate failed to recognize the scope and extent of problems, past actions indicated a lack of commitment to provide the necessary funds and. .
personnel for improvement. In addition, the resources allocated for the 1993 management plan were initially based on those found in the 1992 management ,
plan, rather than on an analysis of the work. The corporate group also had planned that the 1994 and 1995 funding would be less than funding originally .
allocated for 1993. However, throughout the DE, senior corporate officials !
assured the team that the resources needed to improve performance at Quad Cities would be available.
The team noted the present staff was approximately 1.5 times what it was in 1987 and the expenses had approximately doubled. Most of the increases were in the 1988-1990 time frame. Senior corporate management admitted it may have lost efficiency and focus on the work when resources were increased while in a reactive mode, and without commensurate plans to improve planning, budgeting, or work monitoring processes to the extent necessary.
2.4.5 Site Management Was Not Effective The team found that Quad Cities management did not always demonstrate its ability to organize, plan, execute, evaluate, and resolve issues.
Responsibility and accountability were not always understood, demonstrated, or established for performing activities important to safety. In addition, there was a lack of ,llow-through and a failure to achieve plant improvement.
Further, commu! ations internally and externally at Quad Cities were often ineffective.
2.4.5.1 Absence of Responsibilit, and Accountability The team found that responsibility and accountability were not always understood, demonstrated, or established. The site manager's task was complicated by three revisions to the 1993 Management Plan subsequent to the announcement of the DE. Even though the DE and efforts by the licensee had identified numerous hardware issues, the focus of the plan was to implement the strategic objectives of corporate plans. Further, one of the fundamental objectives of the February 1993 reorganization was to clearly establish responsibility and accountability. The September 17, 1993, Management Plan postponed clearly defining roles and responsibilities, interfaces, and expectations of each Quad Cities department until February 1994. This 1-year delay clearly contributed to delayed actions for initiating plant improvements at Quad Cities.
Site management did not always assure that hardware, process, and management trends were identified, evaluated, and corrected in a timely manner. In addition, department managers did not always act on numerous assessments, audits, and other available information that affected plant hardware and ,
operation. Some site managers did not identify root causes or take timely corrective actions. Further, the lack of leadership fostered a safety culture .
that was prone 'a repair equipment when it failed, but rarely fixed the~ root cause. Department managers acknowledged problems and frequently failed to take corrective action. Management exhibited little sense of urgency to solve equipment problems and generally accepted a low level of performance.
45
, . ., - ~ - -- - - .- .. .
. ( ,
1
,v w
l As d'escribed in'the previous section, a cornerstone in the present ,
organization was the creation of a Site Vice President to be' held fully ,
accountable for the business and technical activities of the site. The team found that his authority, as defined in the corporate plan, was not always . .:
clear. For example, the Site Vice President was responsible for engineering, '
but corporate management kept approval authority of engineering' scopes, s schedules, and budgets that affected Quad Cities. In addition, the Site Vice President was responsible for making commitments to the NRC. However, licensing responsibilities were managed and controlled by the corporate office without appropriate oversight. R Throughout the evaluation, many department managers rarely exhibited awareness of plant problems'or assessments affecting safety performance. Site (
management had delegated many of its responsibilities to subordinates, and .
abrogated much of its management oversight function. The-team observed that 1 management'was absent from most problem-solving sessions and other critical 1 activities. For example, the leadership' of the Station Modifications Review ;
Committee, the Recurrent Equipment Program, and the-Events Screening Committee '
was delegated to lower levels with little management oversight and/or feedback. The team observed that there was no management involvement in the i root cause investigation of the August 22, 1993, HPCI event involving the; . >
failure to follow procedures, although appropriate involvement was obtained after the team made this observation. As a result of being uninvolved or ;
uninformed of these activities, some mangers could not discuss a single plant issue of concern to the team during the evaluation.
7 The team noted that some managers had recognized performance problems and' had.
demonstrated performance towards improvement. Unlike some other department- ;
managers, both the work control and support services managers were aware of. '
identified problems, had' taken steps to improve, and obtained results.
2.4.5.2 Inability to Implement Initiatives for Improvement In areas affecting the safe operation of the plant, there was a lack.of '
follow-through and failure to achieve results to improve plant performance. ;
The team observed that there was no assessment or integrated plan to resolve the large number of equipment problems -in- the plant. The lack of such a plan contributed to the declining performance in safety' system availability and ,
power production goals for the last-three consecutive years.. During the-onsite evaluation period, a senior manager had been assigned to develop this plan, y The licensee had not recognized the need to quickly fix processes' that. prov'ide feedback for action and results. Many measures of performance were neither relevant nor completely accurate, and did not always reflect; the need for.
1 prompt management action. For example, Quad Cities had been tracking the - :*
ratio of positive to negative NRC comments.but managers interviewed had-no understanding of what this indicator meant. The maintenance and modification 1 backlog performance indicators did not always. represent the actual. work that i needed to be performed. ~ As a result, neither site' nor corporate management , , t were aware of the status of the work. . Further, the licensee was: measuring the~ !
success of leadership initiatives and .the system engineer.ing program to the j
-results .of. a employee survey rather to than measurable changes in plant- r performance. 1 46
[
, , _ _ . - . _ . __ i
Site managers continued to defer resolution of problems to a committee, group, or program. These programs were usually detailed, elaborate, and comprehensive, and took substantial time to develop. For example, a committee ;
had been formed by management to address recurring equipment problems. As management delegated its leadership of the committee, the committee did not meet frequently, had no minutes of its meetings, had no goals for improving equipment problems, and was not operating as defined in an Administrative Procedure. (SQV had issued a CAR in July 1993 on the latter.) The committee issued a list of the 10 most significant recurring problems to another committee, the SMRC. As another example, one objective in the September 17, 1993 Management Plan was to establish a clear mission for the system engineers program. To meet this objective, the plan required that newly created Site Planning Group, establish a Site Working Group responsible for defining the system management function.
The team found that there was a weak project / issue management system, and most of the managers were not aware of the issues management procedures. The use of an issue / project manager to scope, schedule, budget, and assure completion of projects / issues had not been practiced. As a result, many of the programs lost their identity and the desired results were not achieved. '
Weak management information systems (MISs) also contributed to the failure to obtain results. Management had created numerous computer programs and systems at Quad Cities. Additionally, many other systems and programs were developed by individual organizational elements to list and track problems unique to that group. Only recently had an effort been made to . integrate and coordinate these systems, eliminating redundancies and. inefficiencies and ensuring completeness of data collection. As a result, the team was able to ident.ify many weaknesses. For example, the nuclear tracking system (NTS), the primary tracking system for Quad Cities, contained more than 235. items.that had been delayed or deferred two times or more. Additionally, the team noted that the NTS would not notify management when action items and commitments were delinquent until responsibility for that item had been assigned to an individual. Further, the team identified several examples where inadequacies in data entry prevented accurate NTS data sorting.
2.4.5.3 Ineffective Communications Communications from senior site and corporate managers to other staff at Quad Cities were obstructed by the department managers. The. Chief Nuclear Officer (CN0) communicated management goals, objectives, and strategies through the Site VP and to all who came to the onsite meetings with management.
Interviews revealed that some of the staff had expanded the VPCN0's messages into personal expectations. However, the team found that some department ,
managers did not have a clear understanding of senior management's expectations and could not describe the actions that needed to be implemented in their areas of responsibility. For example, many department managers stated they " walk the talk" and spent time in the plant as expected by senior management; however, many of these managers could not explain to the team what they did on these occasions. One manager stated he needed to give more attention to long term planning to implement the Management Plan and less attention to emerging plant problems.
The communication of management expectations was not always evident. For ,
example, the team observed a number of procedure problems that occurred while 47
the team was on site. Although procedure problems at Quad Cities have been recurring for some time, they have not yet been fully resolved. Therefore, to evaluate management expectations in this area, the team conducted an evaluation of the Operations Manager's effectiveness in communicating his expectations to his subordinates. The evaluation involved one of several procedure non-compliance events that were observed by the team. The Operations Manager was asked to define his performance expectations for each supervisor involved. The supervisors were then interviewed to confirm that the expectations of the Operations Manager had been communicated. The evaluation indicated that the communication of the Operr.tions Manager's expectations to his subordinates was weak. At least six miscommunications occurred which resulted in delayed and incomplete corrective actions.
The failure to appropriately cominunicate expectations sometimes sent employees conflicting messages. The corporatir: olanned to financially reward all employees if the organization reduce :
operating and maintenance costs by 1 to 3 percent. Although the team ac- -. edged it was prudent to reduce these costs, these incentives were not tieu to any performance improvement criteria.
This reinforced the attitude to accept workarounds and degraded conditions. '
Senior corporate management agreed such an attitude could conflict with the corporateimprovementinit)atives.
In interviews during which!some plant problems were discussed, no department manager was willing to accept ownership of problems. The team found that Operations pointed to Maintenance, Maintenance to Operations, and Engineering groups to each other. It was obvious that these groups did not communicate in a satisfactory manner, and did not perform as a team.
The team found many examples where external communications should be improved.
The team noted that the site was hesitant to communicate with corporate resources, particularly in Engineering Technology and Technical Services. The team found examples where corporate management attempted to communicate '
successful fixes from other facilities without success (such as the LaSalle repair to fix the leaking testable check valves). In another case the team found that the Quad Cities staff was frustrated by the low-priority given an 1
old request for an exemption that had the potential to save 38,000 manhours per year and many manrem. This request had been given low priority review by corporate management over a 20-month period. The team found that Quad Cities staff had communicated the_importance of the request to site management; however, the site staff had not advised corporate management of the need to redirect its priorities with the NRC.
2.4.6 A Strong Staff The Quad Cities staff was capable and dedicated. The staff recognized that improvements were necessary. The staff was waiting for its leadership to direct the changes. The team obtained much of its information and insights from the staff, particularly when plant problems were involved. The Quad Cities management had not capitalized on these strengths to implement change.
e.
I 48 i
2.4.7 Positive Observations The corporate plan for the improvement of performance is very comprehensive and complete. The plan contains detailed strategic, business and executive objectives and presents a very systematic approach to sustained improvement.
The executive portion of the plan gives each plant the opportunity to identify, evaluate and plan all the actions necessary to improve its performance. As stated previously, senior corporate officials assured the team the resources would be made available to support the plan.
The management plan was developed as a living document which focused on defining, deploying, and demonstrating responsibility and accountability;.
providing effective managerial and technical support on site; aligning and improving the business, work, and people processes; educating and improving ,
the leadership; and removing barriers to performance as they are discovered.
The plan identified actions for implementation, and contained completion dates for both corporate and site activities. Full implementation by the stations was expected to take three to five years.
The corporate commitment to the management plan was forthright and commendable. The VPCN0 was open and willing to effect change. The VPCN0 acknowledged that aggressive corporate aversight was needed if the plan was to evolve in a timely manner and be succcssful, t
t t
- r 49
3.0 ROOT CAUSES 3.1 Ineffective Corporate Leadership, Oversight, involvement, and Follow-Through Corporate management had not been able to effect significant improvement in ;
the operational safety performance at Quad Cities. Corporate management had defined goals and objectives in broad terms, but neither monitored nor "
measured site management's progress toward them. Corporate management did not insist on timely implementation of solutions to problems affecting safety, and thus did not identify or correct declining performance at Quad Cities.
Lessons learned from the Dresden and Zion diagnostic evaluations such as repetitive equipment problems, weak system engineering programs, motor- .
operated valve problems, and ineffective root cause determinations were not effectively addressed at Quad Cities.
Organizational instability over the last 6 years had caused a loss of ,
consistency of purpose for program initiatives. Each reorganization resulted in previously initiated programs being superseded before full implementation.
Since 1990, substantial improvement initiatives were to materialize at Quad Cities, but did not. Corporate focused on overall generic programs rather '
than site-specific issues related to those programs. Corporate performance ,
initiatives were not appropriately considered during the budget process. t Corporate incentives encouraged expenditures to be minimized without '
concurrent consideration of plant performance improvements.
~
The existing performance monitoring system was not effective in bringing significant performance problems to corporate management attention. Quality oversight organization effectiveness had been reduced because' of staff reduction and poor reporting mechanisms. Corporate management often waited until performance problems'were raised by outside organizations before addressing them, and still' site management was not held accountable for performance improvement initiatives.
3.2 Site Management's Failure to Resolve identified Safety Problems Site Management often was not able to recognize the safety importance of technical problems. A clear indication of design limits being exceeded was needed before a rigorous and formal operability evaluation would be performed.
Degraded, deteriorating, or indeterminate conditions were 'not always evaluated i resulting in some safety significant issues not being given a priority for.
resolution. In many cases, because of the absence of clear design limits 'or regulatory requirements, the licensee did not correct the degraded conditions. ;
Site Management accepted a large number of equipment problems and lacked focus and a sense of urgency to resolve them. The safety significance of hardware and work process control deficiencies was not recognized. Critical processes needed to understand equipment condition and achieve sustained improvement i were not receiving adequate attention. Inaccurate reporting of work backlogs i obscured recognition and extent of plant degradation. ,
50
Once problems were identified, there was a failure to achieve results. The ,
failure to accept full responsibility and accountability at the appropriate management levels, the absence of management in problem solving sessions, and the ineffectiveness of management processes that evaluate performance resulted in an inability to obtain results. The Performance Enhancement Plan, the Critical Self-Assessment, the Events Assessment Team, the '/ulnerability Assessment Team, and the Management Plan were not fully effective because management did not implement the actions necessary to improve performance.
3.3 Low Standards of Performance Corporate management had endorsed " slow measured improvement" for many years.
However, many managers had been at Quad Cities for a long period and did not have a good standard by which to judge performance. Low standards for performance contributed to a lack of ownership and accountability and hindered measurable improvement. As a result, site management had little sense of urgency regarding plant performance improvements.
The affects of low performance standards were evident across the site. The operations staff exhibited a willingness to work around degraded equipment problems. Root causes to problems often were not fixed, wasting resources due to repetit*ve equipment repairs. Engineering was not sufficiently involved in providing solutions to the numerous technical problems that existed, and Site i Management was usually the last to implement corporate improvement initiatives even after they had proven successful at other CECO plants. !
3.4 Site Management's Failure to Exercise Effective Leadership As a result of weak leadership, site management had not fully utilized the considerable capabilities of the staff. Managers did not fully acknowledge their own responsibilities and establish clear.-expectations or make clear ;
delegations of responsibility and authority to each level in the organization.
Absent delegations, responsibility often remained by default at the lower- '
levels where expectations were not understood, performance was not well monitored, and there was little accountability. Site managers were detached from their staff, reserved in interactions with each other, and very cautious in their dealings with the NRC which resulted in communication difficulties.
Most site managers lacked understanding of senior management's expectations, contributing to their inability to establish high performance expectations.
51-
4.0 EXIT MEETING On November 8, 1993, the Directors AE00, NRR, the Administrator for-Region III, the Team Manager of the Quad Cities Diagnostic Evaluation Team, and other NRC staff members met with the President and Chief Executive Officer, CECO; the Vice President and Chief Nuclear Officer, CECO; the Site Vice President, Quad Cities; and senior managers and staff from the Corporate offices and Quad Cities Station to review the results of the evaluation. This exit meeting was open for public observation. Representatives from the non-operating owners of Quad Cities, Iowa-Illinois Gas and Electric Company, were also present. Briefing notes, summarizing the team findings and conclusions, are attached as Appendix A.
52
- f;u<;;;, - ..
- s > >
~
i t'
i
?
't .
DIAGNOSTIC EVALUATION ]
p' EXIT SLIDES ;.
ON !
QUAD CITIES NUCLEARLPOWER STATION NOVEMBER -8',1993 ;
i i
U.S. Nuclear Regulatory Commission Office for Analysis and Evaluation of' Operational' Data Division of Operational Assessment-Diagnostic Evaluation and Incident:. Investigation Branch 2
3
~
.i i
'?
SELECTION OF QUAD CITIES BASED ON e EXPECTED-RATE OF IMPROVEMENT NOT MAINTAINED e
. REPETITIVE EQUIPMENT PROBLEMS l e UNRECOGNIZED HIGH RATE.OF SAFETY SYSTEM FAILURES e' RECENT. MAJOR ORGANIZATIONAL AND PERSONNEL 1
' CHANGES a e- LORGANIZATIONAL PERFORMANCE PROBLEMS NOT-WELL .
UNDERSTOOD ,
-f l-1
.z__
DIAGNOSTIC EVALUATION TEAM 1
GOALS AND OBJECTIVES e PROVIDED'INFORMATION TO SUPPLEMENT OTHER -
ASSESSMENT DATA AVAILABLE TO NRC SENIOR MANAGEMENT e DETERMINE LCAUSES FOR THE SIGNIFICANT NUMBER OF-
-SAFETY SYSTEM FAILURES.
o EVALUATE'THE EFFECTIVENESS OF ENGINEERING e EVALUATE THE IMPACT OF CORPORATE MANAGEMENT ON OPERATIONAL SAFETY.
a ~_ m _ __m___ ____m-__mm__am____ .
_ _____.__mm . _ ,..~,. . ... ...%_. 4 ,. . . _,~ . . , . . , , , . .% ....m. p., [ , , ,, .y_ ',,
, ,, i..y. ,, ,,
- METHODOLOGY .
e- 15-MEMBER TEAM, FOUR FUNCTIONAL AREAS e 6-WEEK EVALUATION: 3 WEEKS ON-SITE,1 WEEK IN CORPORATE OFFICES, 2 WEEKS IN-OFFICE e 'OVER 110.lNTERVIEWS CONDUCTED i .
l e 5 DAYS OF ROUND-THE-CLOCK CONTROL ROOM -
OBSERVATION-3 e INDEPTH:REVIEWLOF: RESIDUAL HEAT REMOVAL SYSTEM i
m OPERATIONS AND TRAINING WEAKNESSES .
e EQUIPMENT DEGRADATION - ACCEPTANCE AND LIMITED AWARENESS e OPERABILITY OF DEGRADED EQUIPMENT FREQUENTLY NOT '
EVALUATED
-e PROCEDURAL DEFICIENCIES INCLUDING ADHERENCE.AND TECHNICAL-SPECIFICATION CONTROL WEAKNESSES e DEGRADEDLSENSITIVITY TO CONTROL ROOM ANNUNCIATORS f
-e LIMITED OVERSIGHT 10F CONTROL ROOM ACTIVITIES DURING BUSY PERIODS t
.g i
+ ,a . , .. ..~.-r - ~~e v,.- . e,.r..,,_...,m.y..__ - ,,... _ mm ,, ._ , . .___. _ a_~ _ . _m_ _ _ _ _ _ _ __.___ ___ _.__ . _ _ _ __ _ __._._ - , ,,_ _ _ _ _ _ __m. __,._
q OPERATIONS AND TRAINING i
STRENGTH
-e OPERATOR TRAINING WAS STRONG i
POSITIVE OBSERVATION e STRONG OPERATOR PERFORMANCE a l .
L e -- OPERATORS SHOWED GOOD TEAMWORK AND
! COMMUNICATED WELL
-i
- u. . . . - - -
MAINTENANCE AND TESTING f WEAKNESSES .
e FAILURE TO FIX THE ROOT CAUSE OF KNOWN TESTING ,
DEFICIENCIES l
e MAINTENANCE lMPLEMENTATION WEAKNESSES ,
l e FAILURE TO FIX ROOT CAUSES OF MOTOR-OPERATED VALVE l DEFICIENCIES .
1 e INEFFECTIVE MAINTENANCE -WORK PROCESSES li e SUPPORT TO MAINTENANCE lNOT SUFFICIENT TO MAINTAIN AN EFFECTIVE l PROGRAM i
-e PREVENTIVE-MAINTENANCE- PROGRAM NOT EFFECTIVELY '
IMPLEMENTED '
t i G
(
i MAINTENANCE AND TESTING POSITIVE OBSERVATIONS
-* . GOOD HOUSEKEEPING AND OUTWARD PLANT APPEARANCE l e- GOOD RADIOLOGICAL CONTROL PRACTICES AND
- MAINTENANCE TRAINING FACILITIES
)
F re e w '..e.--,5 w w ,_y-, ..+3. +-c. 4 s p. , - , *- . . ' , -n',.m = ...< . :--on. rhe"t . -,[
7_-_ - .-- -. -
n
- ENGINEERING AND TECHNICAL SUPPORT .
LWEAKNESSES e RHR EQUIPMENT PROBLEMS' RESULTED IN SYSTEM "
DEGRADATION e ENGINEERING FAILED TO EFFECTIVELY ADDRESS PLANT -
- VIBRATION PROBLEMS o ENGINEERING FAILED TO FULLY EVALUATE DEGRADED l
EQUIPMENT
~
e SITE ENGINEERING DID NOT ALWAYS SUPPORT THE PLANT e CORPORATE ENGINEERING WAS NOT SUFFICIENTLY. ]1 lNVOLVED lN PLANT ISSUES e MODIFICATION IMPLEMENTATION WAS OFTEN UNTIMELY j
- AND INCOMPLETE
]
e CERAMIC: FILL AND COATINGiCOMPOUNDStWERE.NOT
. CONTROLLED
- --- _ - - - . - . -. - . . - .. ~ ..- -
~~
4 1
L ENGINEERING A. ND TECHNICAL SUPPORT 1
- POSITIVE OBSERVATION
- ENGINEERING WAS ABLE TO IDENTIFY PROBLEMS (VAT REPORT)
+
t
~
MANAGEMENT AND ORGANIZATION
-j WEAKNESSES e -MANAGEMENT WAS NOT ABLE TO UTILIZE VAT REPORT c
l
- UNTIMELY AND INEFFECTIVE CORRECTIVE ACTION 3 PROCESSES .
- INEFFECTIVE QUALITY OVERSIGHT DRESDEN AND ZION LESSONS LEARNED NOT YET FULLY IMPLEMENTED 1 l
- CORPORATE IMPROVEMENT PROGRAMS DID NOT CORRECT :
DECLINING PERFORMANCE +
e- SITE MANAGEMENT lNEFFECTIVE 4
_ . . _c__;._... .._...:;..._ . . . _ . , , . . . . , , . . . _ ,-- . . . , , . . . - . . . . . . .
MANAGEMENT AND ORGANIZATION
.3 STRENGTH 3 e
~
CAPABILITIES AND DEDICATION OF THE STAFF .
I POSITIVE OBSERVATION e 'VICE PRESIDENT AND CHIEF NUCLEAR OPERATING OFFICER IS COMMITTED TO IMPROVING. PERFORMANCE t
I e
_...<-2 __ .. w .- ,- --
a n. v- -vw+- _ . - ,- , ,,e-n v ..v+m. __ su +_ _ -e__ _.- y,
1 CHRONLOGY OF INITIATIVES 1990-1991 CORPORATE ASSESSMENTS SEP'1990 DRESDEN AND ZION LESSONS LEARNED LATE 1990 PERFORMANCE ENHANCEMENT PROGRAM
. 1992 MANAGEMENT PLANS FOR IMPROVEMENT j MAY 1992- EVENTS ASSESSMENT TEAM REPORT .;
L JUN 1992 .SECY-92-228 PERFORMANCE OF CECO PLANTS j JUL 1992. BOARD ANNOUNCES MAJOR CHANGES OCT 1992 NUCLEAR OVERSIGHT REORGANIZED NOV 1992 VULNERABILITY ASSESSMENT TEAM REPORT FEB 1993 SITE VICE-PRESIDENT POSITIONS. STAFFED 1993 MANAGEMENT PLANS FOR IMPROVEMENT JUL 1993 DIAGNOSTIC EVALUATION ANNOUNCED ,
JUL 1993 BDT SELF-ASSESSMENT AUG '1993 lNTEGRATED REPORTING PROGRAM '
lMPLEMENTATION i 1993' PENDING QUALITY VERIFICATION REORGANIZATION.
1993- PENDING MAINTENANCE OUTAGES -
l
_. . . __. ._2 - . . _ ~ . _ _ _ __ _ . . u. - . _ , _ .. . _ . ~ _ . _ .. u
~
e f
ROOT CAUSES e "
INEFFECTIVE CORPORATE' LEADERSHIP, OVERSIGHT, INVOLVEMENT, AND FOLLOW. THROUGH-
]
o -SITE MANAGEMENT DID NOT ASSURE RESOLUTION OF IDENTIFIED. SAFETY PROBLEMS 1 Le- LOWLSTANDARDS OF PERFORMANCE o QUAD CITIES MANAGEMENT FAILED'TO EXERCISE EFFECTIVE 4 LEADERSHIP. ;
L i
i
. .._. .. -. ~ . - _ _ . - . ..__.:_.--.. ._...-.;-_._..-.;;.
,_ _ _ . ~ . _ _ , . . . - . _ . . , _ . - _ . . -.
_ _ _ _