IR 05000206/1989034
| ML13329A142 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 12/28/1989 |
| From: | Johnson P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML13329A141 | List: |
| References | |
| 50-206-89-34-MM, 50-361-89-34, 50-362-89-34, NUDOCS 9001180159 | |
| Download: ML13329A142 (46) | |
Text
Q U.S. NUCLEAR REGULATORY COMMISSION
REGION V
Report No /89-34, 50-361/89-34, 50-362/89-34 Docket No, 50-361, 50-362 License No DPR-13, NPF-10, NPF-15 Licensee:
Southern California Edison Company Irvine Operations Center 23 Parker Street Irvine, California 92718 Facility Name:
San Onofre Units 1, 2 and 3 Meeting at:
Walnut Creek, California Meeting conducted: November 29, 1989 Prepared by:
C. W. Caldwell, Senior Resident Inspector Approved By:
P. H/i f[ohnson, Chief Date Signed React r Projects Section 3 Summar A Management Meeting was held.on November 29, 1989 to discuss tssues related to the failure of an ASCO solenoid valve on August 23, 1989 anderecent operator performance problems. In addition, the licensee provided a status on the enhancements to the Emergency Preparedness and Root Cause Assessment Programs, engineering program improvements, the Design Basis Documentation (DBD) review, and the activities of Nuclear Oversight Organization PEJR 900110159 9122
DETAILS 1. Meeting Participants Nuclear Regulatory Commission J. Martin, Regional Administrator B. Faulkenberry, Deputy Regional Administrator R. Zimmerman, Director, Division of Reactor Safety and Projects D. Kirsch, Chief, Reactor Safety Branch G. Yuhas, Chief, Emergency Preparedness and Radiological Protection Branch P. Johnson, Chief, Reactor Projects Section 3 C. Caldwell, Senior Resident Inspector Southern California Edison Company H. Ray, Vice President, Nuclear Engineering, Safety, and Licensing (NES&L)
R. Bridenbecker, Vice President and Site Manager H. Morgan, Station Manager D. Shull Jr., Nuclear Oversight Manager, NES&L D. Nunn, Manager of Nuclear Engineering and Construction D. Brevig, Supervisor, Onsite Nuclear Licensing San Diego Gas and Electric Company R. Lacy, Manager, Nuclear Department 2. Management Meeting On November 29, 1989, a Management Meeting was held in the Region V
Office in Walnut Creek, California with the individuals identified in Paragraph 1. The purpose of this meeting was to discuss issues related to the failure of an ASCO solenoid valve on August 23, 1989 and recent operator performance problems. In addition, the licensee provided a status on enhancements to the Emergency Preparedness and Root Cause Assessment Programs, engineering program improvements, the Design Basis Documentation (DBD) review, and activities of Nuclear Oversight Organization The meeting convened at 10:00 Introduction Mr. Martin opened by stating that the purpose of the meeting was to review recent concerns and to assess the progress of program initiatives since the last Management Meetin Slides used during the licensee's presentations are enclosed.as an attachment to this repor Mr. Ray began by identifying actions being taken in response to the June 27, 1989 Management Meeting. In particular, he indicated that
Southern California Edison (SCE) has placed specific emphasis in the following areas:
-
Enhanced training of all personnel
-
Reduction in the backlog of work items
-
Consolidation and enhancement of the Root Cause Assessment Program
-
Evaluation for necessary qualified staff resources in all areas
-
Maintenance of positive progress toward goals in the Nuclear Engineering, Safety, and Licensing Organization Failure of the ASCO solenoid for CV-304 Mr. Caldwell introduced this topic by noting that on August 23, 1989, an Automatic Switch Company (ASCO) solenoid valve operator for CV-304, the normal charging valve in Unit 1, failed, rendering CV-304 inoperabl This issue was discussed in special Inspection Report No. 50-206/89-3 Valve CV-304 is required to be closed for hot leg recirculation (HLR)
when it is necessary to prevent boron precipitation in the core in the event of a loss of coolant accident (LOCA).
Mr. Caldwell briefly summarized the history surrounding the failure of this solenoid, the apparentiroot cause of the failure, and the number of weaknesses found in the corrective action program that were associated with this issu Mr. Caldwell further noted that when CV-304 failed, the licensee entered a 72-hour administrative action to fix the valve or initiate a plant shutdown. This was done since SCE did not believe that the Technical Specifications (TS) were applicable to the HLR flowpath, since it had been added to the plant in 198 However, the NRC considered that when CV-304 failed, TS 3.3.1 was applicable. In the absence of a specific action statement concerning this valve, TS 3.0.3 would have required that a plant shutdown be initiated within one hour of the valve failur The licensee agreed with the NRC's characterization of the problems surrounding the failure of CV-304 and the interpretation of the TS requirements for this valve. SCE representatives also stated that the company is committed to conservative implementation of TS requirements and would consult with the NRC as necessary to ensure conservative implementation in the futur Mr. Martin stated that this issue was being discussed to ensure that future situations do not arise in which TS interpretations compound issues when activities (such as engineering reviews) find problems in the plan Operator Performance Problems Mr. Johnson summarized recent performance problems observed in the Operations area. In particular, a number of events have taken place that were attributable to training (insufficient emphasis on routine plant operations) or to a need for additional formality in the performance of routine duties. The most recent of these-events included (1) a partial draindown of approximately 700 gallons of water from the reactor coolant system due to operation of the wrong valve and (2) failure to recognize
that a test conducted in Unit 2 rendered one of the Unit 3 off-site AC power sources-(via a cross-tie breaker) inoperabl Mr. Morgan responded that SCE had been closely monitoring performance problems that have occurred and were not merely reacting to NRC concerns in this area when they initiated their review. The licensee believed that these recent problems were due to training weaknesses; insufficient formality; and performance, design, and human factors problems, with the two most significant causes being training difficulties and lack of formality in the performance of normal activities. Mr. Morgan indicated that they were attempting to reduce the number of performance errors through various means, including the Formality, Attention to Detail, Communications, and Teamwork (FACT) program, professional operator development, and simulator improvements. The licensee also indicated that they were evaluating other potential factors for performance problems such as the use of extensive overtime during outages and operator morale. Long term actions would then be implemented based upon the results of these evaluation Mr. Martin stated that there was a perception on the part of the NRC that Operations performance has not recently been as strong as in previous years. He encouraged SCE to make program adjustments as necessary to prevent continued perceptions of a decreasing trend in the Operations are Program Enhancements SCE provided a status on enhancements to the Emergency Preparedness and Root Cause Assessment Programs. In addition, engineering program improvements, the DBD review, and the activities of Nuclear Oversight Organizations were also summarized. Slides used by the licensee during these presentations are provided as an enclosure to this repor Closing Remarks Mr. Martin stated that he considered this to have been a useful session, and indicated that efforts to implement program improvements appeared to be going as well as could be expected. He noted that licensee management appeared to share NRC perceptions about recent trends in Operations performance. He emphasized that the NRC did not want to overemphasize recent operator performance errors; however, it is desirable that any perceived adverse performance trend be reverse The meeting adjourned at 2:00" OPERATIONAL EVENTS CAUSE CATEGORIES EVENTS TRAINING
- CREACUS DOOR [A]* FORMALITY/
- PLCEA DOWN POWER [AD]
ATTENTION TO DETAIL
- ASI TRIP [A] PERFORMANCE
- D-G AUTO START [D,AB,C] DESIGN/HUMAN FACTORS
- OFF-SITE ELECTRICAL SOURCES [A,C]
- ADV MANIPULATION [C,BI
- VALVE MANIPULATION [CBJ
- First Category is Primary
CORRECTIVE ACTIONS
PERFORMANCE ISSUES DESIGN/HUMAN FACTORS
TRAINING/FORMALITY/ATTENTION TO DETAIL
INITIATIVES UNDER DEVELOPMENT BACKGROUND
PROFESSIONAL OPERATOR DEVELOPMENT AND EVALUATION PROGRAM
SIMULATOR IMPROVEMENTS
PROFESSIONAL OPERATOR DEVELOPMENT & EVALUATION PROGRAM (PODEP)
AREA ACTION COMMUNICATIONS DEVELOP STANDARDS TAILBOARDING DEVELOP CHECKLIST PLANT MONITORING RADIX TRENDS PROCEDURE USE JOB PERFORMANCE MEASURES (JPM)
PLANT MANIPULATION PROBLEM SOLVING SIMULATOR ALARM RESPONSE
SIMULATOR IMPROVEMENTS
LESSON PLAN UPGRADE V
LONGER PROBLEM SESSIONS V
CONCENTRATION
/
IDENTIFIED PROBLEMS (NORMAL PLANT EVOLUTIONS)
DEDICATE ONE WEEK TO NORMAL OPS V
PLANT MANEUVERING V
RESPONSE TO THE UNEXPECTED V
STRESS PODEP FEATURES
PROVIDE PODEP TRAINING TO THE INSTRUCTORS
UPGRADE SIMULATOR MODELING
EVALUATION
ATTENDANT ISSUES OVERTIME V
1989/OUTAGE
/
RELATIONSHIP TO INCIDENTS ATTRITION/MANNING V
ANNUAL V
CAUSES MORALE V
PERSONAL OBSERVATION V
OPS/TRNG OD EFFORT V
QA CONCERN
Management Meeting - November 29, 1989 SCE Has Reviewed the NRC and SCE Memoranda Concerning the June 27 Meeting
- Actions Are Being Taken in Accordance With That Discussion Agenda for Today Includes Items of Concern to SCE As Well As the NRC
- Of Particular Concern Are Efforts to Maintain Operational Excellence U -Overall SCE Assessment, Based on Last 3 Months There Is a Need to Increase Training In All Areas, Both "How-To" and
"Why", With Emphasis on Excellence In Normal Operations
-
Thereis a Need to Reduce the Backlog of Work in Both Technical and Related Program Areas There is a Need to Consolidate and Enhance Root Cause Determination Processes Action Is Required to Maintain and Increase Qualified Staff Resources Significant, Positive Progress Has Been Made Toward Our Goals, Including Much Greater Involvement In--and Awareness Of--Ongoing Plant Activities by NES&L
Application of Technical Specification Requirements U Conservatively Implement Requirements At All Times
Inform Region V and Consult With NRR When Clarification Is Required to Ensure Conservative Implementation
Where Functions or Components Have Been Added and/or Taken Credit For Subsequent to Issuance of the Relevant Technical Specification Requirements, Include These Functions or Components Within the Requirements
.
Propose Changes Where Necessary to Clarify Applicability
Emergency Preparedness Program
SCE Has Included Emergency Classification Among the Responsibilities of the Emergency Coordinator
This Worked Well So Long As the Emergency Coordinator Remained In the Technical Support Center (TSC)
When SCE Decided to Transfer the Emergency Coordinator Function to the Emergency Operations Facility (EOF) Following Its Activation, Emergency Classification Was Slow
- Region V Comment in Connection With 1988 Exercise
In 1989 Exercise, SCE Attempted to Duplicate TSC Emergency Classification Timing at the EOF
- Results Successful With Respect to Timing
- Other EOF Functions Adversely Impacted
SCE Will Modify Its Program to Retain Emergency Classification in the TSC Following Emergency Coordinator Transfer to the EOF
Result Will Improve the Interface Between the EOF and TSC
Root Cause Assessment Program I
Need for SCE Reassessment Discussed by NRC at the June 27 Management Meeting
SCE Has Been Strongly Oriented Toward the Principle that the Most Effective Corrective Action Results When Root Cause is Determined by the Organization Responsible for Implementing the Action
-
This Remains an Important Factor in Our Development of a Revised Program
-
Nevertheless, We Must Pull Together Scattered Activities Into a Single Program Which Must Be Developed
Root Cause Assessment Program
The Program Will Be Administered by Oversight Engineering, Within the Nuclear Oversight Division of NES&L
-
Dr. Chlu Will Be the Manager of Oversight Engineering Effective 1/1/90
- The Program Will Be Fully Functional by 6/1/90
-
Root Cause Program Staff to be Engineers Drawn from Quality Organization - Additional Staff Will Be Hired If Needed
-
In Addition to Root Cause Program Group, Oversight Engineering Will Include Quality Engineering, Independent Safety Engineering Group and Nuclear Safety Group
Root Cause Assessment Program
The Program Will Provide for:
-
Systematic Oversight of All Root Cause Determinations by the Program Staff - This Will Include Tracking to Provide Visibility to Timeliness and Backlog
-
Independent Root Cause Determinations by Oversight Engineering for Designated Events or Conditions
-
Establishment of Ad Hoc Evaluation Teams When Directed by the Manager of Oversight Engineering
-
Systematic Evaluation of Data to Identify Conditions by the Manager of Oversight Engineering
-
Definition and Training In the Use of the Appropriate Methodology to be Applied to Root Cause Determinations In Various Circumstances
UPDATE ON ENGINEERING IMPROVEMENTS AND SONGS DESIGN BASES DOCUMENTATION (DBD)
November 29, 1989 Southern California Edison
Review BASIC SSFI CONCLUSIONS -.JUNE 1988 I
SCE Lacks Full Understanding of Basic Design of Systems Reviewed I
SCE Lacks Ready Access to Accurate Design Information I
Many identified Deficiencies Result from Inadequate Access to Basic Design Information I
Technical Work Is Not Always Complete and Technically Correct I
SCE Relies Heavily on Contractors
Assessment I
Improvement in Quality of Design I
increased in-House Design I
System Ownership I Continuing Improvements
- Interface
- Teclyieal Expertise
- Training
- Backlog 2e
Nuclear Engineering and Construction TASK FORCE RECOMMENDATIONS I
Consolidate Nuclear Functions
- Increase Level and Quality of In-House Design I
Develop a Design Bases Documentation Program
NUCLEAR ENGINEERING &ONSTRUCTION DIVISION NUNN, D. MANAGER OF NUCL ENGINEERING
& CONSTRUCTION PILMER, D MYASAT TECHNICAL SE REAR CONSULTANTA BALOG, C. K O' CONNOR, K MYERS, P. MERLO, M. SHORT, M. P*
HONLEY, P. SITE NUCLEAR NUCLEAR NUCLEAR NUCLEAR ENGFR DESIGN BASES TECHNICAL I
ENJGIN.JEERING OTRUC~TI FUEL DESIGN DOCUIMENTATON SUPPOT BRO)UGH, ( open THOMSEN, WHARTON, CARUSLE, B OGOSIAN, G FUEL SCHONED ENGINEERING COSRUTON]
ANALYSIS MECANCA NULER SEIO I
~ENGINEERS MIHALIK, HUEY, BEERS. KANEKO FREY, TECHNCAL t
FUEL ENIING KRAMER, TECHNICAL PLANNING PAOCUREMENT ELCRCALNGNEING KAEC SUPPORT ELECTICL
[
AI DBD ENGINEERS HADLEY, MRAN, RODGERS, E MILLER, R.PR I
t
--
PR(1JECT PROCUREMENT MATERIAL ADMINISTRATION CIVIL / PLANT ENGE ENGINEERING DICKINSON. I RICE, TECHNICAL SERVICES WRL
Nuclear Engineering and Construction INCREASE LEVEL AND QUALITY OF IN-HOUSE DESIGN
- Resources
- Level of In-House Design'
- Quality -- Engineering Excellence Program
Nuclear Engineering & Construction 1989 Resource Plan PERSONNEL 2W 195
..............................
190 180 4C)...
Proposed Actual 170 161 160
150
.f 140 130 126 120 I
I NO DE JA MARCH JUNE SEP DE In-House Design 100%
100
.i Planned
Actual
Forecast
50
30 30 -2 %
%0
"
10 U1CO U2C5 U3C5 U1C11 1988 1989 - 1990
Engineering Excellence UEngineering Quality Monitoring Program-Unit 1 Cycle 10:
Complete 29 Action Items
- Unit 2 Cycle 5:
Ongoing 12/89 Forecast
- Independent Review: Unit 2 Cycle 5
- Cygna and Sargent & Lundy
- Four (4) DCPs
- Forecast 12/89
Engineering Excellence
- Design Engineering Training
- Consolidated as part of NES&L Department Training Group
- Increased resources - 1990
- INPO Accreditation Criteria
- Training Needs Analyses - Complete
- Training curriculum - Approved
- Training Program Description - 12/89
- Priority / Schedule - 1/90
- Process Training (50.59, DCPs, systems)
Engineering Excellence
- St'anding Committees Established
- Standards
- Design Review
- Communications Improvements
- Roles & Responsibilities
- Procedures and Work Process Enhancements
- Analytical Tools Acquired
- BECAPandPTI
- ME101
Nuclear Engineering Design Organization PROGRESS SINCE CONSOLIDATION
- System Design Engineering Concept Implemented
- Engineering Excellence Program Initiated to Change Culture
- Engineering Quality Monitoring Program Established
- Engineering Resources Increased Design Change Process Improved and Training Program Under Way
Design Bases Documentation Program Overview
Design Bases Documentation RECOMMENDATIONS FROM 1988 ASSESSMENT
,'Evaluate and Refine Configuration Management Program Develop Design Bases Documentation Program
..Maximize Use of SCE Engineering Resources
"Verify Existence, Correctness and Consistency of Design Documents J Verify Incorporation of Design Basis Information Into Operations, Surveillance, Maintenance and Training Programs Verify Final Safety Analysis
Design Bases Documentation PILOT PROGRAM GOALS I
Prepare DBD Program Plan by January 1, 1989 I
Prepare Three to Four Pilot DBDs by End of 1989 I
Field Verify Pilot DBDs I
SSFI Type Review of One SO1 and One S023 Pilot DBD System I
Review Results and Revise DBD Program as Necessary I
Prioritize Remaining S01 and S023 Systems for DBD Development
Design Bases Documentation PROGRAM STATUS & SCOPE Status I Pilot DBDs Under Way
- S023 Instrument Air
-
SO1 SLSS (Sequencer)
- S023 Component Cooling Water
-
EQ Topical I NSSS / Industry Participation
- CE / W / BPC Document Retrieval
- NUMARC / INPO / Region V Utilities Efforts Scope I All Systems with Safety Functions I Major Topical Areas
Design Bases Documentation LESSONS LEARNED DURING PILOT PROGRAM g Schedule Changes
- Accident Analysis Topical
- Electrical Systems
- Component Cooling Water IVerification Program
-
Operability derived from design bases
-
Post-installation testing program derived from design bases
-
Vendor supplied skids (i.e., black boxes)
-
FSAR commitments
NUCLEAR OVERSIGHT I. Status of Items to Address NRC Observations and SCE Commitments made from last meeting II. State of the Organization
NRC - OVERSIGHT MANAGEMENT PRESENCE IN THE PLANT UNACCEPTABLE SCE ACTION
- AREA MONITORING PROGRAM
- ACTIVITY MONITORING AND SURVEILLANCE PROGRAM
- NOD OUTAGE PLANNING PROGRAM
AREA MONITORING PROGRAM
- All Oversight Participates except QC Inspectors
- Site Engineers Monitor Assigned Areas Weekly
- Offsite Engineers Monitor Assigned Areas Bi-Monthly
- Area Monitoring Program Status Reported to Management Monthly
- Look at Activities in Areas, Evaluate Material Condition and Housekeeping
- NOD Staff Participation for Qctober was 85% (309 Area Monitoring Reports submitted)
ACTIVITY MONITORING AND SURVEILLANCE PROGRAM
- Formal Procedure Implemented
- Typical Assignments
- Shadow PEO from Pre-Shift Brief to end of Shift Turnover
- Monitor Reactor Stud Final Tensioning
- Average of 35 Activities Monitored each month for last 6 months and a total of 88 findings resulted
NOD OUTAGE PLANNING PROGRAM
- Formal Procedure Implemented
- All Six Oversight Groups participate
- Vulnerability Analysis used to Select Areas of Interest
- Examples of Selected Oversight Areas - Unit 2 Outag Refueling Activities were covered 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />s/day
- Balance of Plant Work was covered 2 shifts/day
- Steam Generator Inspections and Repairs
- Surveillance of FME Control Program
- Monitored Shutdown Cooling and Mid-Loop Entry
- Surveillance of HP Controls and Programs
NUCLEAR OVERSIGHT SEPTEMBER OCTOBER DIVISION WEEK WEEK WEEK WEEK WEEK WEEK WEEK WEEK PROTECTED AREA ENTRIES 1*
3
1 2*
4 NOD MNGMT/SUPVSN
10
6
6
5, ISEG
2
3
2
2 NSG
3
1
2
0 SUPPLIER QA
3
7
4
4 SITE QA
28
30
18
22 QUALITY PROGRAMS
2
4
1
1 TOTAL
48
51
33
34
- HOLIDAY
NRC - OVERSIGHT NEEDS TO BE MORE AGGRESSIVE, INTRUSIVE AND ACTIVE SCE ACTION
- De-Certify 30% of the Auditors/Lead Auditors Developed' Get Well Program for Each
- Gave NSG & ISEG Corrective Action Capability
- All Violations in the Field Immediately Corrected or Formal Stop Work Issued
- All Violations reported to Supervision
- Independent Evaluation of Causal Factors for Specific Events
- 0 NRC - OVERSIGHT NEEDS TO BE MORE AGGRESSIVE, INTRUSIVE AND ACTIVE SCE ACTION continued
- Moved NCR Approval from Site GA to ISEG
- Hired Operators - Implemented Operator Surveillance Program
- All NOD Trained on INPO Observation Techniques and Performance Based Auditing
STATE OF NOD CHALLENGING THE CORPORATION LEAVING FOOTPRINTS ADDING VALUE EXAMPLES
- Crimper use Out of Control
- Background Investigation Vendors Uncontrolled
- Unit I Trip - Feedwater Valve Control Logic
- 50.59 Evaluation ADV Modifications
STATE OF NOD (continued)
- Maintenance Staff Passing Tools under Contamination Zone Barriers
- Test Engineer asking Security to Help Perform Leak Rate Test of Emergency Hatch
- Operator requests HP Technician to take Reactor Vessel Level Readings in a Zone III (Fuel Flea Zonej
- Pipe Support in Turbine Building found deficient
- Struts. Disconnected
-
Loose Fasteners
MANAGERS ASSESSMENT
- Trending of deficiencies is not where I want it to b * Audit training program is not where I wish it wa * Although 40 corrective action documents were issued in the area of Design this effort not as cohesive as I would like it to b I
- Evaluating performance and effectiveness rather than merely reporting complianc * Are asking the next question - pulling the strin * Organization has embraced the requirement to be part of the team and add value to Nuclear operation e*
MANAGERS ASSESSMENT (continued)
- Exceeded my expectations of June
- Must keep the pressure on
- Cannot allow ' We're good enough' to settle in_