ML20198J837
| ML20198J837 | |
| Person / Time | |
|---|---|
| Site: | Crystal River  |
| Issue date: | 01/10/1998 |
| From: | Rencheck M FLORIDA POWER CORP. |
| To: | |
| Shared Package | |
| ML20198J675 | List:
|
| References | |
| NUDOCS 9801140223 | |
| Download: ML20198J837 (12) | |
Text
-- .. . . - - _ - - _ _ _ _ _ _ _ _ _ _ _ _ - -
T
}
Power SFlorida C0RPORATaON r
~
CRYSTAL RIVER U NIT 3 l ENGINEERlhG :
1 l
l . .
4f & ^<-$ i M W. Rench'eck Director, Nuclear Enginesing and Projects l
gligggg gyb2 PDR 1
P .
~
l Key issue How does FPC have reasonable assurance that Engineering personnel are properly focused on safety, have a sufficient questioning attitude, l and demonstrate sufficient leadership, knowledge, and skills to assure 1 that engineering contro!s and equipment are maintained within the prescribed design and licensing bases and that the plant operates in a reliable manneri Executive Recruitment of experienced Engineering managers brought a new Surnmary perspective to the previous issues of engineering performance. These managers have aggressively addressed prior performance problems.
Ihrough systematic evaluations, assessments, and anal sis, we have identified the underlying root causes of the engineering deficiencies at CR-3. These evaluations clearly demonstrated that a key focur, of the corrective actions must be on improving Engineering personnel knowledge, skills, safety ethic, and sense of ownership. Other assessment activities demonstrated that there was a need to enhance practices and processes, as well as to reduce the backlog and improve equipment reliability. The corrective actions implemented in the Engineering area support safe plant restart and continued operation.
Initiatives Training and The level of Engineering performance for design and license basis issues Quafi/ication were directly tied to the knowicdge and skill possessed by personnel, in this regard, FPC initiated comprehensive remedial training of Engineering personnel. The training focus areas included:
- Safety Assessment /Unreviewed Safety Question Determination
- Design and Licensing Basis e Configuration Management furthermore, Engineering had issued and trained our staff on Administrative Instruction Al1700 which defines the structure, functional responsibilities and management expectations for the Nuclear Engineering and Projects Department, it is a roadmap or guide to address the conduct of Nuclear Engineering and Projects.
Specialized Saining in the form of Failure Prevention International's (FPI) Human Error Reduction Training for Supervisory Personnel was initiated to enhance the knowledge of our supervisors to guard against situations that lead to human error, in addition, over 150 Engineering personnel received FPI's Apparent Cause or Root Cause analysis Training in support of our new Corrective Action Program initiated in 1996. To reinforce human performance prior to restart of CR-3, Eng.ocering supervisors and managers received Event free Operation and Probabilistic Safety Assessment Training from FPC personnel to refresh their knowledge base.
2
l l
l Meastirement of effectiveness was obtained by an independent l Engineering Readiness for Restart Evaluation.
Engineering Management benchmarked our training procedure TDP-308 against INPO's ACAD 91-017 guideline to ensure that Engineering Support Personnel (ESP) are trained and qualified for their assignment.
TPC recognized that the effectiveness of this training and the maintenance of basic knowledge and skills for Engineering personnel could be better assured through the use of a review board. Specifu::,,
a Training Review Cc,mmittee was established. The committee is manned by experianced senict level managers who meet to discuss training activities and needs for station personnel (i.e., our 1998 ESP Training curriculum).
Management expectations regarding training have been raised. The acceptance level for engineer competency (80% score to pass, previously 70%) has been established well above that for minimal performance. Engineers who fall to meet this standard or exhibit marginal competence are placed in remedial programs, as appropriate.
Enhancing Based on an initial FPC screening, key engineering practices and Programs processes were selected for a ccmprehensive assessment. The screening and Processes not only included engineering-specific programs / practices, but other key interfacing processes (e.g., Corrective Action Program). The assessment primarily evaluated the practices / processes with regard to their: (1) interface; Q) consistency of implementation; and (3) adequacy when compa,ed to purpose. The assessment team effort used:
- Personnel involved in the implementation of the practices / processes
- Input from the departments that depend on the practices / processes
- Outside process experts The first step was to develop self assessment measurement criteria.
These criteria were derived from various industry "best practices" sources (e.g., INPO, EPRI and " top performing" utilities). With these criteria, the second step of the assessment was to bueline each practice / process to determine how it functions, what works well, what is not working satisfactorily, and why. This was accomplished by various means, including self assessments, procedure reviews, process flowcharting, and team focus groups. Once the function and effectiveness was understood, each practice / process was benchmarked against industry "best practices".
3
Based on the assessment results, several programs and processes rea" ired follow-up reviews. T hese programs included:
e In Service Testing and in Service inspection
- Pipe Supports and Analysis e Appendix *R"
- Configuration Document Integration Project e l&C/ Electrical Calculation Upgrades
- Instrument Setpoint Calculation Verification
. Electrical loads for Diesel Generator e Configuration Management e Control Room Habitability
. Once Through Steam Generator Inspection
- P;>nt Equipment Equivalency Replacement Evaluation
. Modification Approval Record e Deficiency Reports e System Readiness Review Program issues identified for corrective action during these follow-up reviews were included in Precursor Cards (PCs) within our Corrective Action Program. " Restart required
- PCs are being completed and Engineering Departments are affirming that they are ready for restart and will support Operations througn the operating cycle.
The design change process known as Modification Approval Records (MARS) was specifically targeted as an area for improvement, given a number of selfidentified concerns, in response to these findings, FPC took a number of remedial actions. Chief among these actions were the following:
- Training for Engineers on 10CFR50.59 and Design Basis e Reinforced the need to perform a pre and post design field walkdown to verify and validate plant configuration
. Established interdepartmental (Operations, Maintenance, Licensing, Training, and Engineering) reviews of the conceptual and final MARS by implementing a formal Design Review Board 4
1 hen we ensured the process improvements were effective by conducting:
- Several third-party assessments of safety significant modifications
- Design Review Boards to evaluate the effectiveness of root cause investigation results to resolve plant deficiencies requiring modifications and the conceptual and final MAR design Illgh Quality and Engineering management recognized that proactive and responsive Responsive engineering services were firmly grounded in the perceptive attitudes fngineering and motivations of its work force. To gain a better understanding of Products and these attributes, f PC used input from a number of assessments and self-Services assessments to determine the view of the quality of Engineering services and products.
A key process, the System Readiness Review 'SRR) Program was used to identify restart required activities using interdepartmental teams (Operations, Design Engineering, System Engineering and Licensing).
T he SRR process is discussed separately.
Results from this enhancement in engineering produced:
- 180 MARS to correct plant condi' ans
- 8 major modifications (MARS) to enhance safety margin and reduce operator action:
a) High Pressure injection (IIPI) Recirculation to the Reactor Building Sump b) Control Room Emergency Ventilation System Upgrades c) Low Pressure injection Pump Mission Time d) Reactor Building Spray Pump 1B Net Positive Suction Head Margin e) Emergency feedwater System Upgrades f) Emergency Diesel Generator Loading g) Failure Modes and Effects ior loss of DC Power h) Generic Letter 96-06 Post Accident Containment Heatup on Coolers, Piping, and Penetrations Other process enhancements produced the following results:
- Comprehensive Once Through Steam Generator inspections
- Upgrades to in-service inspection and Test Programs 5
e Control Room Habitability Verification and Validation e Appendix *R' Upgrades e Quality of 10CFR50.59 Evaluations e Upgrades to Design Basis Tools Training in the form of 10CFR50.59, Design Basis, Configuration Management, Root Cause Analysis, and the Corrective Action Process was effective in enhancing the skill base of Engineering and in increasing the quality and responsiveness of Engineering products ar services.
Independent assessments performed a FPC Quality Assurance and third party assessors have confirmed engineering effectiveness and quality of engineering services; i .e., Configuration Management, Safety Evaluations, REA Dispositions, Precursor Card Dispositions, and Vendw Manual updates, are improving.
Design and Understanding the CR-3 design and licensing is the foundation for Licensing Engineering success. The three major tasks completed in this area Basis Assurance included: (1) Completion of the CDIP project FSAR Chapter 14 VerificationNalidation, which provided assurance that CR 3 can operate within its design and licensing basis; (2) Completion of the Systems Readiness Review Program, which performed vertical SSF1-type assessments of CR 3 systems; and' (3) Implementation of training recommendations identified in our Management Corrective Action Program.
The CDIP and System Readiness Review projects verified that Chapter 14 Safety Analysis assumptions and input parameters were used consistently within our desig- and licensing bases documentation. The Safety Evaluation Reports associated with Technical Specification amendments for CR 3 were reviewed to assure their requirements were correctly incorporated. In addition, enhancements to the (cIlowing design / license bases tools gNe CR-3 reasonable assurance that it can operate within the Design and Licensing Bases:
e .- Integrated Technical Specification e FSAR and related Chapter 14 Safety Analysis
. Enhanced Design Basis Documents e Analysis Basis Documents and Calculations -
- Special Emergency Programs' and Surveillance Procedure and Emergency Operating Procedure
. Equipment and Modification 6 l
The CDIP process and results are described separately.
lhe Nuclear Steam Supply System (NSSS) Vendor, framatome ,
Technologies, Inc. (fil), provided dedicated site services to support CR 3 restart. Fil provided their Vice President of Engineering and a large percentage of Engineering staff to provide input and resolution of design basis deficiencies. While on site, these resources reported to, and took direction from, the FPC Director of Engineering. FTl supported resolving design and licensing basis issues such as Boron Precipitation, Small Break loss of Coolant Accident analysis, and completion of CDIP.
This level of participation has provided additional assurance that the NSSS design basis is integrated into f PC solutions and documentation.
Another key aspect of Design and Licensing Basis assurance is to have an effective 10Cf R50.59 Safety Evaluation Program. FPC provided 10Cf R50.59 training for the engineers and created a Safety Analysis Group (SAG) within the Design Engineering organitation. SAG provides independent review of 10CFR50.59 safety evaluations ,
performed by site personnel for MARS, selective procedures, and other evaluations. The effectiveness of this group has been confirmed by independent assessments and Plant Review Committee results.
Reduction of The Engineering backlog is comprised of items from numerous sources Engineering (l.c., Precursor Cards, Request for Engineering Assistance, NOTES, Backlogs Modification Approval Records, Document Change Notices, MCAP 11 Items, and Licensing Commitments), in the past, this diversity contributed to the less than adequate management oversight and employee accountability in completion of backlog items. 10 address this situation, the various tracking mechanisms were screened for outstanaing en gineering open items and against restart criteria. This resulted in creating two components of the Engineering backlog:
" restart" and " post restart" items.
A comprehens!ve set of Performance Indicators was created to monitor progress toward resolving engineering actions. Managers are assigned ownership for specific engineering activities and are required to provide progress and direction at Engineering staff meetings. For example, Request for Engineering Assistance, Vendor Manual Updates, Drawing Change Notices, etc. are statused, for the restart items, over 16,000 work activities were completed for CR 3 Readiness Review Process, including the Engineering backlog reduction effort. items identified as restart were incorporated into the Systern Readiness Review Program and _ a responsible supervisor or f- manager was assigned to monitor success. Some activities, by necessity, will be performed during power ascension; however, to date, most activities have been closed, as indicated in the System Readiness Review ,
Program discussion.
7
i improved Root As noted above, comprehensive root causes and effective corrective Cause Analysis actions were identified as a weakness. Several principal actions were and taken to address these weaknesses: (1) establishing and implementing a f//ective process for the identification and resolution of technical issues; (2)
Corrective providing training; and (3) establishing a requirement to perform in.
Actions - depth root cause analysis for significant events.
As part of the first action, Engineering initiated a daily Engineering management goup raceting to identify and review emerging Engineering issues, and to work more effectively to address customer 9 needs. Upon identification of such issues, the group assigns responsibility for corrective action and continues to monitor progress to ensure prompt and effective resolution. The e fectiveness of the group is demonstral.ed by the increased number of identified issues that are
'ffectively resc hed.
For the second action, engineers have been trained in Root Cause Analysis (RCA) techniques. The Corrective Actbn Procedure (CP-111) was revised to require a " formal" RCA for Level "A" and "B" Precursor Cards. Also, a deficiency report has been implemented to specifically address issues that are nonconformances to the design and license basis.
For the third action, root cause analyses are performed for significant events. These analyses are reviewed interna!l f by line management to ensure quality. An independent review board, the Corrective Action Review Board, then assesses the overall effectiveness of the root cause evaluation and grades performance.
Results and Performance Indicators Organizational The engineering organization has been restructured to provide more improvements and effective and proactive support to CR-3 Operations and Maintenance.
Training /
.Quali/ications The Engineering leadership has been changed to ensure effective
( management oversight of CR-3 Engineering. Approximately 85% of the Engineering managers are new to their positions since CR-3 shutdown.
About 40% of these people are experienced bdustry personnel from other utilities.
. Engineering Support Personnel have had their training records updated as required per INPO ACAD 91-017 guidelines. The training provided for Design Basis (DB), Licensing (LB), and Configuration Management (CM) has yielded the following improvements. Prior to training, only 11 of 212 engineering personnel passed (>80% test score) a pre-training test. Upon completion of DB/LB/CM training, 200 people passed
(>80% test score) with an average score of 90%. The remainder of 8
t . . . - . . . - . . .
'peSonnel received remedial training.
Total Avg. Test % Passing Attendance Score % (> 80% Test Score)
Pre-Training 212- 59 5 Post-Training 212 90 94 Training initiatives have improved our design basis, licensing basis and configuration management knowledge and skills. Similarly, FPC conducted further assessments using the Failure Prevention International (FPl) Quality Index Review. Specifically, FPC measured the Engineering _
Judgment (EJI) and Problem Solving and Knowledge (PSKI). An Engineering Quality Index is the average of the EJI and PSKI. An Index of 50 to 70 has been correlated by FPI to an Engineering organization at a SALP 2 rating. The following reflects the assessment results:
Date of Survey EJI PSKI EQI Sept.1997 73 46 59 Overall, FPC Engineering EQI is within the 50 to 70 point SALP 2 correlation. Strong engineering judgment scores were offset by lower problem solving and knowledge skills. This assessment provides input to further improve engineering performance. For example, the Quality Index Review Results are being used to further improve Engineering knowledge and skills by providing specific input into the 1998 Engineering Training Curriculum, improved Work The work processes have been enhanced at CR-3.
Processes The CR-3 MAR Design Review Board (DRB) results are tracked. These results are indicative of engineering's enhanced ability to satisfy customer needs (Operations and Maintenance) and resolve plant equipment deficiencies. The DRB approval rate is monitored for adverse trenus. Within the past five months, the DRB approval rate is approximately 94%.
DRB Results Number [
of MARS '
Reviewed F
==
E1 i
, E J
^
N l Fx ,
f
\l h
~
l ,
[ l ll l 9
The Plant Review Committee (PRC) results are also tracked.- _ These -
results are indicative of the higher safety quality of design changes prepared by Engineering. Specifically, the PRC rejection rate for 10CFR50.59 evaluations, as depicted below, are meeting management expectations (at 95% approval rate).
PRC Reviewed SA/USQDs Number of . .
=
SA/USQDs m + cT.en ten.
.t...
ass s stor Oct enn tat Backlog The restart Engineering backlog has been significantly reduced. The c Reduction restart required activities will be resolved prior to restart.
For example, Request for Engineering Assistance (REA) had been a significant backlog for Engineering five months ago. The items were reviewed by the SRRP and prioritized te support plant activities. As depicted below, Engineering has reduced the REA workload from approximately 800 REAs to under 200. As part of our Mode Ascension Assessment the quality of REA responses have been evaluated by QA and third party assessors to ensure quality dispositions. The results demonstrated improved engineering performance O
Weekly Request for Engineering REA Report l
Number -
of REAs ,"' '
- g we,.,,
EMB R($? ART
" "%rq .ut et Ctkt e tap .
N
- AM4WERED
~ ~ ~ ' ~
\ m.2 n
. . A. tr.:l 11.L.a.3;,._..in11'._L
- . . . . . l.1.. .~3.1. .L.:Ilti X a Months Other examples are the Vendor Manual Updates and Drawing Change Notices. -Typically, utilities experience backlog growth in these areas due to modifications during outages. Although CR-3 has completed 180 MARS, the Vendor Manual Updates and Drawing Change Notices have been managed to support olant operation. As shown below, only 12 Vendor Manual Updates aLJ ~50 Drawing Change Notices are 1
10
]
pending. All Control Room Drawings are updated within an average of two days from MAR Return to Service.
Vendor Manual Updates Number of -
Vendor Manual * -
Updates e e
m - - - -
! san == M -
m DCN Status Numbti ,
of DCNs e _ __
._._[eum l m _ _ __ _
0- -e- - - - + - ' i- m ChwWr* CM MWm wwwwk W3dWik DowWik On3dvek Quality The quality of Engineering products and services to supoort station Engineering operations has improved. The Engineering organizations are integrating Products and Root root cause analysis as part of the normal Engineerin5 review process.
Cause Analysis For example, CR-3 Corrective Action Review Board evaluates all Precursor Cards requiring Root Cause Analysis. Engineerias performance has met or exceeded management expectations for the last two months. The graph that follows illustrates the conclusion that Nuclear Engineering has sustained the high quality of its root cause analyses, l
1 11
l Quality of Root Causes 24 ,
Root Cau?e - 7; Quality Index ~
, g 16 1
~
34 r
j - __ Mr$SSTAR ARDS k-12 -
10 DS 8 WMN 6
4-2d O . , , . , , . . .
1 2 3 4 5 6 7 8 9 10 LAST 10 EVALUATIONS goo 16OR cKATER Independent As discussed in the text of this report, various assessments have been Assessment completed to confirm that engineering results will support CR-3 restart.
Results These assessments have confirmed the quality of specific MARS, Maintenance Rule implementation, and other engineering work activities. Specifically, to confirm Configuration Management practices and overall Engineering readiness to support CR 3 operation, broad and intrusive assessments were performed by independent teams consisting of industry peers, senior consultants, and INPO personnel.
. The Configuration Management Assessment confirmed that reasonable assurance exists for effective configuration management support of CR-3 rrerstion.
. The Engineering Readiness Evaluation confirmed that CR-3 Engineering was expected to be ready for restart upon completion of all restart related activities.
Conclusion FPC has conducted aggressive engineering enhancements to improve performance. Personnel and process changes have been made, and performance has improved. The standards have been raised across the organization as demonstrated in training, problem identification, and corrective action. Prior backlogs have been reduced, which will result in greater equipment ieliability and more prcmpt response time to emergent issues. As a result of these actions, Engineering department leadership and personnel are focused on safety, have a questioning attitude and have demonstrated the ability to effectively resolve engineering issues.
Further, a continuous Leadership Plan will be used by Engineering to sustain improved performance. This plan focuses on continued improvement using performance indicators and assessments to ensure effective results, i.
12
_ _ _ _ - _ - _