ML20244B116
Jump to navigation
Jump to search
| ML20244B116 | |
| Person / Time | |
|---|---|
| Site: | Rancho Seco |
| Issue date: | 03/24/1989 |
| From: | Caldwell C, Correia R, Hooker C, Kirsch D, Ramsey C, Toth A, Wagner W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V), Office of Nuclear Reactor Regulation |
| To: | |
| Shared Package | |
| ML20244B108 | List: |
| References | |
| 50-312-89-01, 50-312-89-1, GL-81-12, GL-83-28, GL-88-15, IEB-88-009, IEB-88-9, IEIN-88-045, IEIN-88-45, NUDOCS 8904190014 | |
| Download: ML20244B116 (79) | |
See also: IR 05000312/1989001
Text
{{#Wiki_filter:,,.
y \.)
. V[ ' -1-
,; ' .> ,
'
'
U. S.' NUCLEAR REGULATORY COMMISSION
REGION'V
iReport No. 50-312/89-01
1
Docket No. .50-312-
l
'
License No. DPR-54
!
'
Licensee: . Sacramento Municipal Utility District
_
P. O. Box 15830
Sacramento, California 95813
Facility Name: Rancho Seco Unit l'
Inspection at: Herald, California (RanchoSecoSite)
Inspection Conducted: January 23 to February'17,1989
1
Inspectors:
A. D. oth, Team Leader, RV
3 2 2 J'f
Date Signed
.A Y m
C. B. Ramsey, Reactof Inspector, RV
3f.no /J'9'
Date Signed ;
h$YM/AO
W. gWagner, Rpfctor Inspector, RV.
1/Zz/89
Ddte Mgned
. ..
- $ YY
C. W. Caldwell, Reactor Inspector, RV
,3/ b/R'1
Ddte Sign'ed 1
- - \
'
a h h
C. A. Hooker, R rat'on Specialist, RV
3f22fcP9
Date Signed
.
~
t . ,, f h, f g 3l9yky
,
,
i'
:R.P.Correi9,PefformanceEvaluationBranch,NRRDateSigned
.
'
Consultants: ,
l
,i '
f. C'. .
Hsu,1 Reactor'0perations Analysis Branch, AE00 J
,
1", - ; zD. Baxter; INEL/EG&G .i
>4
'
:G. Overbeck,,ERCI
3 .t 3
~
~g
4
Prepared'By: < 9
Date Signed
'
A. D. Toth, Team Leader .
sDivision of Reactor Safety and Projects, Region V
'
Approved By: '
th00L
D.M. Kirbch', Chief
_3
Dafte Sign'ed
'
Reactor Safety Branch, DRS&P, Region V
..
"
C904190014 890327
PDR ADOCK 05000312
G PNU
I
_
- 7 .. .-
_
,
-
.- . g ,- -
3-
_
-..
g- , _ _
,
- -
[* k t}
~
I [' ;;g
@7 , . . , i
.-n+
.
. \;w.:>. i my; , ., -v.
- y; f aae.2 nsg , :, . ,
., w ,t o
_
, g .5 t/ g: , .
,g.
,
, , ,
;.s.2
,
.,
, l?
-
. ,
^[ j
,
'
'
,
a a s t ,_ , , a3
'
i
d ,3 1 , t t. p
Y <
4 ' f -
, ; ,
'
i
7 r '
7
4 i -
,
.
, -
,
,
. d. . v . <.
.
s .
% > -{g s.
Jr a
' '
~
_
,
(n.
4
..,
;
,
Table"of Contents
-
a .
tu
m; } ~.
-c .; ;Page
Table of~ Contents 1...-.................................. .
.............. 2= ,
.r
. Summary ..............'............. .................................. 3-
DETAILS ................................................................ -
8
*
,
'1. Pers'ons' Contacted .......................................... 8-
2. ' Inspection Objectives ...................................... 9
..
3. Inspection' Approach......................................... 9
.
.. . ..
4.: # Significant Issues Identified During the Inspection . . . . . . . 11-
. . .
~ 5. General. Inspection Findings - Maintenance Program Overview . 11
'
APPENDIX A'-'SIGNIFICANT ISSUES IDENTIFIED DURING THE INSPECTION ....
-
14
' APPENDIX 8 - REPORT: #1 0F DESIGN. CONSULTANTS . . . ., . . . . . . . . . . . . . . . . . . . . . 40 -
' APPENDIX'C " REPORT,#2 0F DESIGN. CONSULTANT:.........................
~
44
APPENDIX D - SPECIFIC IDENTIFICATION OF DOCUMENTS REFERENCED-
IN THE INSPECTION REPORT ............................. 51-
. .
: APPENDIX.E - GENERAL INSPECTION FINDINGS,
: MAINTENANCE INSPECTION TREE .......................... 52-
; APPENDIX 'F' - MAINTENANCE TRE E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 '
t
I
e
f
4
9
e
j
.J
t
j'.
. -
-
- _ _ _
-,
i
9 ,~f a ! .'
L, * "
_.- ;w y ,
[v gf . .
.3.,
, . . - 'a L, % 4
*
p J! L -.
'l
L ; Summary: ,
'
r J
,t , ~ .
_ .
j gf jInspection"during the period January 23 - February 17,1989
.
)
.c ! ' ((Report No. 50-312/89-01). i
.
, Special announced team inspection of the Rancho Seco
j 2', #, P Maintenance Areas Inspected: ,
program and implementation of:related activities. The inspection 1
W - 4 team utilized ~NRC. inspection procedure TI-2515/97 and related procedures ..f
, '
1 - ! creference'd therein. ,~O
,,4 y+y -
. ;y 3 p
.- V 5?'ISafety Issue Manag'ement System Items: ,5N -
- Q
f.(
,
,
.a
,
,
i W Nonei
;. x r
#
;*
a .
39' C
fJ r '
-j
x
[(i
?
< ~ <=
f( q.; L iInspection Objectives: ~
6 (' '
Qv /
'
s
,_
.,
l N J ,., y '; , The-objectivecof.this inspection was to determine the effectiveness of the '
q '( l ,j ,
f f( ?' + Ranch'o Seco total integrated maintenance process, to assure that all- , y
- F /. ' ~ components,isystems, and; structures of Rancho Seco are adequately maintained ' '
Mi yj
so thatethey; are+available-to perform their intended functions. * <
'.- m
>
'
fa
Resul ts' : ' e
{
/ General Conclusions and' Specific Findings:
' The te'am consensus was that the maintenance program at the Rancho Secol Unit 1
was; adequately addressed and implementation was in place, but could be
strengthened. There were-areas where the' licensee has been'proactive in
_
.
establishing 1 improvements in the. maintenance program.
!The maintenance-process for the Rancho Seco Unit I has been improved since
sthe December 1985 shutdown'and shows continued and recent dedication of
; resources to further~ improve the program and its implementation. Substantial
progress has been made in the reduction of backlogs, and in addressing prior
troublesome. issues,1such as permanent repairs of temporarily repaired leaks
'in piping ~ systems. The' maintenance program currently appears adequate, but
:could be further: strengthened. . Specific areas of maintenance program
weakness; include: communications, engineering support, support interfaces,
post- maintenance testing, procedure. adequacy, utilization of probabilistic
risk assessment, and electrical circuit breaker maintenance.
p =
Some areas of weakness reflected inattention to detail in either establishing
or implementing the program:
.
1. Technical oversights, omissions and inattention to m - meering detail.
^ EXAMPLES'
4
a.- Proper periodic testing and corrective action had not been
<'
' '
establish'ed for the AFW governor and turbine overspeed trip device, ,
: . including the governor oil-dump solenoid feature. l
# ,
.b.- The; addition of a nitrogen purge to turbine governor increased
r
,
-
vulnerabilitycof the Auxiliary Feedwater System turbine.
g ,4, , t , c , p '"
u
. n( , p
__._'. -
'
;
.
.f,
_ . _ _ _ ___ _ _ _ _
, __
,
.m
-
, .
,
w. y
,
t,j - '> ,4-.
,.:
An inadequate evalation of the' condensate . storage tank drain valve
l /<
-
*.c. positioning during operations. ..
,
, , , . d .' ' Lack of:calhulati.'ons' to support the temporary, bypass piping H -
connectedtofthe'AFW" system.
_ t .
.
>
: ;-
e. ' Lack of calculati.oris- toiverify'the load capability of the main ,
'
steamsa'fety,valvegag-mechanism +fabricatpdon-site.
<
*
>
. . v ..
. , o ., .
- - f. Lack of an evaluation of~ generic consider tions of debris found in 'l
the instrunient air'sy' stem. ,
, . ,
>, -
.
'
'
. , , s
'
g.' Omissions 'in the, determi' nation of the qu'araritine . boundary for.
equipmentpotentiallyaffectedbythe"AFW[oyerspeedevent.
c ,y
-
,
7
.h. Inadequate' con' sideration'toev'aludting';circuilbreaker:andfuse ,
j
j, . problems, with respect to weaknesses iricoordination data.-
2. Flawed work? instructions.' ,
, /
EXAMPLES' ,
4 ; 5, y
- a. JPlans for investigation
- lof" equipment' failures involved in the AFW.
h
, y. Overspeed event were in omplete and did.not meet commitments.
b. Work instructions / procedures for investigation'of AFW equipment
< involved;in the overspeed event (turbine governor and mechanical
overspeed trip device) contained omissions and were cumbersome.
' c. Turbine control test procedures / instructions contained a high risk
-
approach to the test,'and omitted prerequisites and precautions.
d. Installation instructions lacked appropriate testing of a nitrogen
system check valve.
O e. - There were delays in correcting inadequate / flawed ; work procedures.
: f. C mpleted " Work Requests" contained instructions which were in some
ceres ambiguous, and included minor. data omissions; such items were
not been' identified by final quality assurance reviewers.
.
3. Weak control of' documents.
. > EXAMPLES
.a. Vendor manuals did not reflect as-built conditions of the turbine l
governor. !
l
3'
..
,
[ \ ., -r
I
5 _ _ _ _ . . _ . _ . _ _ _ _ _ _ _
' ^
.
. .
>-
'
.. 5-
1
. . ...
Inattention to' industry experience and initiatives.
*
~4.
"
EXAMPLES
a. Maintenance test procedures did not incorporate NRC published
lessons learned regarding turbine control / protection failure modes.
,
b. Debris in the instrument air system was not recognized for its
potential generic implications, and risks were apparently not
recognized in the use of sealing tape on threaded joints.
,
3 5. Weaknesses in communications and engineering support.
, ,
'j EXAMPLES 1
,
. .y
J a. There were no requirements for engineers to review the technical ' *
y ,,-
q, s content of post-maintenance procedures, or otherwise become ' *
*
,
'
. . involved in the post-maintenance test activities. Responsibilities
of the.several engineering groups were ill-defined, and there
'
, ,' seemed to be no single group responsible for engineering
:!; y coordination of specific systems activities. "
r, ,
,
, ,
.
'
;
M b. There was inadequate communication between the enginee' ring and2 ,
,
procurement groups, regarding the AFW governor vendor questions.
.
<f 6. Delays in completion of maintenance craft training. ,
!
EXAMPLES
a. Completion of on-the-job training, following completion of
instructional training, was not timely. ...
,
b. Documentation of completed-certifications /traini.ng was deferred due
to workloads.
7. Prioritization of backlogs.
EXAMPLES
L- a. More timely completion of identified procedures and vendor manual l deficiencies could have helped avoid the AFW turbine overspeed
event. j
L
8. Failure Monitoring - Nuclear Plant Reliability Data System (NPRDS) j
EXAMPLES
a. The method used to determine NPRDS reportable failures appeared to
contain potential oversights and omissions. !
9. Each of the concerns addressed above also reflect examples of missed
opportunities by the quality assurance organization to iderr ify and
"
l inform management of existing problems. l l l l 1 L _ _ _ _ _ _ _ _ _ _ _ . . _ . l
-
.
* *
L ,,-
, -6-
.
Significant Safety Matters:
The vulnerability of the Auxiliary Feedwater System (AFW) to overpressure, as
a result of failure of the pump turbine governor and its mechanical overspeed
trip, was a self-revealing event on January 31, 1989, when both devices
failed during a test after maintenance. The event resulted in both trains of
AFW system piping being pressurized beyond the design stress values, thereby
making the system inoperable due to questionable piping integrity. A failed
mechanical overspeed protective device may have existed since the last time
' the device was tested on July 24, 1987. With existence of this passive
failure, bnth trains of the AFW system were vulnerable to failure upon AFW
activation, in event of active or passive failure of the turbine governor
during that activation. Such an event occurred January 31, 1989.
,
Summary of Violations Identified:
The following items, not related to the occurrence of the January 31, 1989
AFW turbine overspeed event, appeared to be violations of NRC requirements:
1. Failure to comply with design control procedures requiring performance,
verification, and documentation of calculations for design' changes, i.e.
basis for design of temporary piping connections to the AFW system, and
for design of a hold down (gagging) device for a pressure relief valve
on the main steam system. (Appendix A, paragraph 1.d and paragraph 1.e)
2. Failure to implement (PDQ) procedures for identification and evaluation
of conditions adverse to quality, to assure that cause is determined and
corrective action taken to prevent repetition; i.e. plastic debris .n
air system check valves in accumulator supply lines for valve' actuators.
(Appendix A, paragraph 1.f)-
1
3. Failure to provide acceptance criteria in. detailed work instruction for
investigation of the AFW turbine overspeed event of January 31, 1989.
(Appendix A, paragraph 2.b.(2))
Open Items Summary:
During this inspection, 10 new items were opened; no previously identified
items were closed.
Management Meetings
The Team Leader met with several licensee managers on February 16, 1989
to display and discuss the diagram representation of the results of the
maintenance program evaluation. Issues identified during the inspection were
conveyed in detail to the licensee's inspection coordinator, who discussed
these with plant and corporate management prior to the formal exit meeting on
Februa ry 17, 1989. During the exit meeting, the issues were summarized.
This was followed by a presentation of the maintenance diagram, to illustrate
how the issues related to various elements of the maintenance program.
1
1
- _ _ _ - _ _ _ - _ - - - _ _ _ _ -
7_, _ . . , - - _ _ .
- 4 - :r. ,
4. :. . . ' .-
'
f. ,
-7-
, .'
J
>
' Unresolved Items
An unresolved item is a matter about which more information is required to
. ascertain whether it is an acceptable item, a deviation, or a violation.
. Unresolved items identified during this inspection are discussed.in Appendix
'A .
paragraphs 1.c, 2.c, and 2.d. ..
Open Items
'
. An Open Item is a matte'r that requires further review and evaluation' by the
inspector, including an. item pending specific action by the licensee and a
j previously identified violation, deviation, unresolved item, and program
related weakness. Open Items a.'e used to document, track, and ensure
'
adequate follow-up by the inspectors. Open Items identified during this
inspection'are discussed in Appendix A, paragraphs 1.a (two), 1.h, and 6.
<
;
,, ,
3
4 t Y
! '
1_ > ^
~
k 'g 1 , '
Y
'$_, g
g- 5Y 4
i
}., ,
*
t -
.
j ..
4
) ' s..
,
'
. +
,
-
}- U t ;
,
, ,
,
*
*
.[,' _
s .
.
)
Y t
j' L I: e - - - - - _ - - - - - - _ - - -
p. w + A , -t u
-
-
7
*
,p , ii
' ' % ' j% p y ' i *T .
[ g ?'-'
,'
," ( ;;
.
*
'J. ' -8" ['
:.. '
# #
P
*
'
.
,
3
;
'
e , l 7
DETAILS
j. ,
-
L
7 ,
h 4
_.
4 ,
. ""
1. PERSONS CONTACTED -
.
3
' Sacramento Municipal' Utility District-
..,
*J.'F. Fir 11t, CEO Nuclear
*D. R. Keuter,-AGM, Nuclear Power Plant Manager
"
1 *J. R. Shetler, AGM, Nuclear Support Services
*B. : G. Croley, AGM, Nuclear Technical Services
*J. Vinquist, AGM, Quality & Industrial Safety
.
*W. F. Peabody, Nuclear Engineering Manager
,
*P.' J.-Bender, Deputy Maintenance Manager
~*S. Redeker, Operations Manager
*T.- E. .Redican, Material. Control Manager
*Ri L. McAnnaly, Training. Manager
*J. Del _ezenski, Supervisor, Regulatory Coordination
*A.;Bonino, Licensing Engineer
*G..Legner, NRC Coordinator
y; <
D. Brock, Maintenance Manager
P. E.-Turner,' Plant Performance Manager
I" M. J. Bua, Radiation Protection Manager
M. E. Herre11, Nuclear Training Manager
-K. Helman~ Nuclear Human Resources Manager
,
S. Crunk, Nuclear: Licensing Manager
L. Fossom, Assistant Nuclear Plant Manager
C. Linkhart,' Instrumentation & Control' Superintendent
B. Chapin, Planning Superintendent l
, In addition to the personnel listed above, during the course of the
inspection the inspectors also contacted other licensee employees,
including: operations staff, health physics and maintenance technicians,
engineers,. quality assurance' staff, and various supervisors.
NRC Staff and Consultants
*D. Kirsch, Region V, Reactor. Safety Branch Chief
p *S. Richards, Region V,, Engineering Section Chief
.
*A. D'Angelo, Senior. Resident Inspector
G. Kalman, NRR Project Manager (Rancho Seco)
'*A. Toth, Region.V Team Leader
*C. Ramsey, Region V Team:En'gineering~ Inspector
*W. Wagner, Region V Team Engineering Inspector ~
*C. Caldwell, Region V Team Projects Inspector
*C Hooker',1' Region V-Team Radiation Specialist
*R. Correia, NRR/ Human Factors Branch Team Engineer
C. Hsu, AE00/ Reactor; Operations Analys_is Branch Analyst
*D. Baxter, INEL/EG&G' Team Consultant
G. Overbeck', ERCI . Consultant ;
* Identifies individuals who attended the exit management meeting on !
February 17, 1989. )
I
a
' -
- _ _ _ _ _ - _ _
_ _ _ - _ _ _ _ _ _ ____ _ _ -
...
;; . d, s. '
. , ..- -9-
:.
I "
-
'
2. ' INSPECTION OBJECTIVES
i " .;
'
-
.The. principal objective of this team inspection was to determine the
+
effectiveness of the~ total integrated maintenance process at Rancho Seco /
>
? iUnit 1, to assure that all components, systems, and structures are -
:* ,,
'
adequately maintained sotthat they are available to perform their ;
,
/F' lintended functions,,and that the maintenance processes at the facility '
,1 ',
provide for.the prompt repair of plant components, systems, and '
'
H
structures as appropriate to their prescribed functions. _
,
'Several contributing objectives were addressed to focus the inspection:
) a. Determine wh' ether all components, systems, and structures of the
facility'are adequately maintained so that they are available to e
c perform.their intended functions. Does the maintenance process
. provide prompt repairs as appropriate.
b. Characterize the' effectiveness of licensee's maintenance program to
avoid challenges to safety systems from transients initiated or
,
.made more severe by equipment failures due to maintenance
weaknesses.
c. Assess the effectiveness of quality verification organizations in
contributing to the identification, solution, and prevention of
safety.significant technical problems and deficiencies in plant
systems and operations. This includes the-thresholds for management
involvement in ensuring that identified deficiencies are responded
to promptly and completely.
d. Assess the utilization and performance of System Engineers in
avoiding or mitigating plant events.
e. Specifically consider components and systems which have been
recognized and well documented within the industry as significant
problem areac; which were identified as predominant risk items by
Probabilistic Risk Assessment methods; and/or which have
experienced unusual or frequent failures at Rancho Seco. Unit 1.
This included:
(1)- The auxiliary feedwater system turbine overspeed and system
overpressure event of January 31, 1989;
(2) The main feedwater system components which have experience
instability problems;
(3) The main steam pressure relief valves which have experienced
setpoint problems; and
(4) Electrical systems associated with the above systems.
l
.. g ,
_ _ . . _ . _ _ _ _ _ _ _ - - _ - - _ _ _ _ _ _ . _ -
y G ,, ; .
']l .s
' n' % '
,
', ' ~
-~10 -
<
1
3. ! INSPECTION APPROACH
The team attempted to find, vulnerabilities in hardware,. personnel, or
program centrols which could lead to plant transients if uncorrected.
Emphasis was placed on equipment failures having root causes or elements
arising from maintenance oversights or omissions,
a. Inspection' Methodology
A diagram (MAINTENANCE TREE) and associated inspection guidance of
'NRC inspection procedure (TI-2515/97) was used to focus efforts of
individual team members, and to at.sure that all potential
contributors to selected plant problems were explored.
During a two-week on site review of procedures, records and work: i
in progress, and interviews of licensee staff, the resulting
inspection data was reviewed'in team meetings at the site.
Inspection findings were considered relative to each block of the
MAINTENANCE TREE,"to identify maintenance control program aspects
which may have" contributed to discrepancies and.which warranted
-
further team review. Near the'end of the. inspection, findings
. applicable to;each block'were discussed by the team,;and a rating
of 'progra'm . and implementatiori ' effectiveness was arrived at by
. consensus'. ,,The individual block ratings were then averaged and
adjusted ~ qualitatively to a rating'for each key block.
b. . Sample Selection' l
v.
Generic PRA studies', such as:NUREG-1050, identify the frontline
systems for typical PWR reactors, and such systems were considered
for inspection' candidates. Equipment failure data of the NPRDS was
also considered, along with consideration of topics of past team
inspections. Additionally, recommendations of the NRC Senior
Resident Inspector were considered. Dur% , a one week on site
preparation / inspection effort, the team selacted as an initial
. working / familiarization sample, the AFW punt turbine governor.
This component appeared to be an example of hardware with a
troublesome history and repetitive repairs and modifications.
During the week after the initial team site work, a significant
- . event occurred at the site; this involved overspeed of the AFW
L system turbine due to failure of the turbine governor, resulting in
excessive pressure in redundant piping trains. The team elevated
this item to the principal focus of the subsequent two-week
l detailed inspection effort, addressing both the cause of the event,
and the licensee troubleshooting efforts to investigate the event.
In addition to the above, the team also examined a variety of I
maintenance activities relating to the following:
*
Main Feedwater System active components
l
*
Main Steam System Pressure Relief Valves J
i'
*
Electrical distribution system components i
*
Temporary / permanent repairs to piping / component leaks. )
!
l
l
.
..m__m__u-._ _ _ _ - _ _ _
, y- ._
^
'
'
a', *',
s, f, ,
._ 11 -
-
,
-
o. ,
.c. Additional Event A'ssessment' Assistance
The' inspection team was augmented with two technical' specialists
during the week of February 6,1989:
E -(1) 'An NRC staff member, author of AE0D Case Study A'E0D/C602 l
" Operational Experience Involving Turbine Overspeed Trips",
dated August 1986, assisted the team in review of-the
technical details of the turbine governor and overspeed
mechanisms, generi.c implications of the turbine overspeed
.
. event, and expected status of Rancho Seco relative to
previously issued industry and NRC reports of similar
operating experiences.
(2) A design consultant, who had participated in the 1986 NRC
follow-up activities for a December 1985 Rancho Seco event,
-' " Augmented System Review and Test Program (ASRTP)", assisted
3 .
i
;.
'
the: team in the technical review of the design bases of the
~
, AFW system and turbine / pump, the relationship of the current
overspeed event to concerns expressed by NRC during the ASRTP
reviews, and an assessment of the adequacy of the boundarns
established by the licensee for quarantine of equipment
affected by the turbine overspeed and resultant pump discharge
pressure.- Two reports by this consultant are attached as
Appendix B and Appendix C to this inspection report. '
The design consultant also assisted in a review cf prior ;
design changes and modifications that appeared to have an ;
impact on the operability and/or repetitive failures
experienced by the AFW turbine governor. Results of this
,
assistance review have been incorporated into the body of this
,, inspection report, and also are discussed in Appendix C.
4
~
' '
d. Team Interfaces With Other NRC Inspection Activities
..
' ~
, ,
!, The-inspection team assisted the NRC Resident Inspector office and s'; ;
g
#^
*
ths Region.V office in assessing the 'iicensee event investigation o ' :)
plans and activities. Team members reviewed the individual action f*
y +
'
plans and'provided recommendations regarding concurrence with the '
, ,
t' '
'
s- ' action plans. The team observed investigation activities of the -
; , AFW turbine governor and mechanical overspeed mechanism as part of
e ,,
the work-in progress scope of the Maintenance Team Inspection;
s
'
sign'ificant findings were identified to the Resident Inspector
'
office for subsequent action following completion and assessment of ^ '
,
." the. licensee investigation efforts. Details of.these findings are i
in'cluded.in the body of this. inspection report.
#
4. SIGNIFICANT ISSUES IDENTIFIED DURING THE INSPECTION '
The inspection team identified several regulatory compliance issues and
other observations which appeared to be weaknesses in the maintenance
program and/or its implementation. These items were grouped into
i general categories as discussed in Appendix A, and were considered b during the rating process documented in Appendix E of this report.
I
__
---__ _ - .
. .
!
- - 12 - !
l 9 l'
i
!. 5. GENERAL INSPECTION' FINDINGS - MAINTENANCE PROGRAM OVERVIEW i
I
L Appendix E of this report' presents the' team assessment of 'the licensee
maintenance program, using the guidance of NRC Inspection Procedure l
TI-2515/975 Where applicable, the assessments presented in numbered j
subparagraphs of Appendix E refer _.to~ issues;which have been discussed in 1
detail in Appendix A.
. \
Appendix F contains<a marked diagram summarizing the team findings, as {
discussed with the licensee management representatives at the conclusion i
of the inspection.
~
Near the end of the on site inspection, the team j
developed a consensus. judgement of a rating of each assessment element,
relative to perceivedtlicensee performance in fulfilling maintenance '
program objectives of'that element. A summary presentation of the team
ratings of all maintenance program elements is provided on the diagram. j
The ratings are presented using color coding and scales, as described in l
the legend-to the Inspection Tree. The ratings have been broken down
into three, parts, as follows:
Element Adequacy: A measure of how well the licensee maintenance
program has described and documented the requirements of the element.
Green: The element was determined to be fully included
in the licensee maintenance program,
vellow: Thc element was determined to be adequately addressed
in the licensee maintenance program.
Red: The element was determined to be missing or inadequately
addressed in the licensee maintenance program.
Element Implementation: A measure of how well the licensee maintenance i
process has implemented the requirements of the element.
Green: The element was determined to be functioning and
functioning adequately.
'
Yellow: The element was determined to be in place, but
could be strengthened.
Red: The element was determined to be missing or inadequate.
Composite Element Rating: Element Adequacy and Element Implementation
ratings of individual level 4 blocks of the tree were combined to
reflect a composite rating for the level 2 and 3 blocks (Blocks I, II,
III, and 1.0 through 8.0) and for the overall rating of the maintenance
program:
Good: More than minimal efforts have been made in this area, and
this area has desirable qualities with only a few minor areas
requiring improvement.
1
,
f
._--_-.._._-.x _-._ . - _ - . . - - _ _ . . . _ . _ _ _ - ._ - - _
, - _,-.
,
., ..
a
- 13 -
.
'"k
Satisfactory:
~
Applicable requirements of this' element have been
c.
'
developed, documented and effectively implemented.
' Areas requiring improvement are approximately offset by I
better performance in other areas. i
Poor: Inadequate or no effort has been made in this specific area.
!
The' handling of color coding of Section.I of the Inspection Tree was an
exception to the above procedure. Section I deals with historic data ,
'
and plant appearances, for which a two part (program and implementation)'
l- breakdown was not appropriate. For Section I blocks, a single color
code was assigned, representing adequacy of the physical plant and its
- operational history; i.e. functioning well (Green); adequate (Yellow);
.
! inadequate (Red).
'
In addition totthe. color code characterization of the individual and
~~
soverall block:ratin~gs, the team also gauged the degree of conformance
,
.with the evaluation criteria, (a composite of both the program '
definition and implementation aspects of each block); this was
~
g . . ." . 4J represented by a mark;in the sliding scale below each level 4 block of
the tree.' These scaled ratings were compiled in histogram form to
'
n ; arrive lat similar relative ratings for the level 2 and 3 blocks, and the
, -
,
;1eveli1 overall maintenance program rating. -
The paragraphsLof Appendix E are numbered to correspond to individual
,
, blocks.of the Inspection Tree, and summarize the individual ratings
r- discussed above, and their bases.
4
,
1- h. L
:
l
!
- - _ . _ _ _ _ . _ _ _ _ - _ _ _ _ - - _ _ _ _ _ _
. _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ _ _ _ _ _ _
.
,
*e '
14
. .
APPENDIX A
SIGNIFICANT ISSUES IDENTIFIED DURING THE INSPECTION
The inspection team identified several regulatory compliance issues and other
observations which appeared to F- weaknesses in the maintenance program
and/or its implementation. These items were grouped into general categories
i as discussed below, and were considered during the rating process documented
in-Appendix E of this' report.
I
1. Technical Oversights, Omissions, and Inattention to Detail
a. AFW Surveillance / Preventive-Maintenance Programs
The program for periodic surveillance and preventive maintenance of
AFW turbine components was inadequate. Had proper periodic testing
been developed, and performed following turbine governor
maintenance work, the January 31,.1989 AFW turbine overspeed event
could possibly have been avoided.
Rancho Seco had'previously experienced problems with the AFW
turbine governor and the mechanical overspeed trip mechanism (OST),
_
and had identified the need for testing of these components several
years ago. However, the devices were not incorporated in a
periodic testing program prior to the governor replacements.
During a July 1987 startup test of the AFW mechanical overspeed
trip (OST) mechanism (special test procedure STP-1025), the OST
failed to operate properly the first time. The device actuated,
but due to mis-adjustment of the trip bar linkage, the trip and
throttle valve.(HV-30801) did not close. The trip bar linkage was
adjusted and the 0ST functioned properly during two subsequent
attempts. Subsequently, the system engineer identified that this
device should be incorporated in,to .a routine," periodic testing
program, and an appropriate procedure' prepared for such testing.
The system engineer' identified the need for a procedure to perform
replacement, venting, and adjustment of the AFW pump turbine
governor on April' 28, 1988. Testing of the.0ST was identified as a
priority 2 item on.the'LRSL (i.e. a procedure should be prepared
prior to startup from the next refueling). The procedure had not
yet been prepared, nor routine testing commenced, before the
overspeed event occurred on January 31, 1989. In view of the
repeated replacements of the turbine. governor, and the experience
with passive failure of the trip mechanism, it appears that more
timely implementation of the routine OST periodic testing was
warranted. Open Item (50-312/89-01-01).
Also, a-governor control-oil dump solenoid was included as an
integral part of the AFW Terry turbine governor, to prevent
overspeed conditions should the turbine be shut down for several
minutes and then be restarted. Industry experience reports
identified a number of overspeed conditions that occurred due to
the long amount of time (up to 30 minutes) necessary for the oil
pressure to decrease in the speed demand part of the governor.
_ _ _ _ - _ _ _ _ _ _ _ -
s ., ,
s.. ,
-
y 7
-
p ,
,g' p' Q -l >
"
]q
,; .i d c
#
..15 - IU
y
3 . ~
.
4 L># {>
'
f ; - ,
*
tt; :n _
, <
'
' O .
p # t
- j 1 .i
'
At. Rancho Seco the licensee had tested operation of the governor
yy: 1V dump solenoid only once during.the operational lifetime of the s
~
g' (<
'
? plant.- That occurred during an August 1987 performance of
'
i
e. "
. procedure STP.1070. The-licensee functionally tested the dump ,
^
solenoid by running the turbine, tripping it for several minutes,
~
v i
an,d restarting the turbine. Circuit continuity checks were not.
performed nor was individual (component' level) testing performed.
<
Discussions with the' licensee indicated that they had added testing
. "
of the dump solenoid and OST to the LRSL for action. For the
governor dump solenoid, the periodic' surveillance'were scheduled
to.begin in June 1989, in conjunction with motor operated valve
k
analysistand testing.
Test records showed that a problem had been encountered with the-
dump solenoid during a July 1987 turbine startup test (STP-1025).
, ~ The solenoid became stuck and a blown scienoid diode was found.
'These problems limited turbine speed to 1500 rpm. The. licensee
replaced the solenoid and contacted the vendor who recommended that
'the solenoid diode be jumpered during the performance.of the test.
' The jumper was installed and the test was completed on July 24,
" 1987;,a new governor solenoid was installed on December 17, 1987.
'The test records did not demonstrate that'the licensee had fully
evaluated the effect of this solenoid diode jumper on subsequent
governor' operations (i.e. would the t'urbine operating speed have
been limited to'.1500 rpm, and was there a need for some sort of
compens'atory action). Open Item (50-312/89-01-02).
b. Inadequate Design of Nitrogen Purge to AFW Turbine Governor
As' discussed in Appendix C of this report (Report #2 of Design
Consulta'nt), Item 3, addition of a nitrogen purge to the AFW
governor (intended to eliminate moisture buildup in the governor
oil reservoir) introduced a mechanism for water intrusion via
draining water.from the Condensate Storage Tank, under some
operational circumstances. This has subsequently been corrected.by
addition of a check valve in the nitrogen line, but the valve was
not functionally tested (see item 2.d of this Appendix).
I, c. Inadequate Evaluation of Condensate Storage Tank Drain Valve
u
As discussed in Appendix C of this report (Report #2 of Design
'< .
Consultant), Item 3, a complete evaluation was unavailable to
![ T demonstrate the acceptability of operation with an open valve
g
'
'(MCM-146) on the Condensate Storage Tank (CST). This item is
_
- unresolved pending NRC further review of the licensee decisions ,
'
tLhat the CST was not inoperable, and issuance of an Licensee Event
3 i,
Report"(LER) was not required. Unresolved item. (50-312/89-01-03) +'
o ,4
; !I t -
.
,
? i 1
/5 4
.s 3 -
O
g
*
+ -
,.
k
~
' <
,
.
.
+ 4, i
> . _ _ _ . ______-__3
- --
- = - '
,m . ,o , .
,, ; .: s . -
.mn s: *
in9 ,_.16 -
. ,
,
. . ,
'
4' d. Lack ofJCalculations for AFW Bypass Temporary Modification-
f.:
O .4i
,
*
:SMUD engineering safety review-(10 CFR 50.59) of temporary 'pihing, j
,
:
connected'to the:AFW system vent / drain lines, did not consider the
"
z impact of the, connection on the seismic qualification of the AFW
Lpiping system.
"
~ Temporary Modification (TM) 89-11 was developed to allow the
sq recirculation system-for the steam generators (SG)'to operate with
= inoperableLportions-of the AFW piping system'(normally used for-
s'ch
u recirculation). TM'89-11 required, installation of two 1-inch
diameter, non-safety gra'de, carbon steel pipelines', by passing the
: inoperable portions of the safety-related AFW system. The
: temporary pipes, v'alvssiand their connections to the AFW system
appeared to be installed.with no abnormal or' unusual items. All
supporting engi.neering documentation appeared _ adequate and in
conformance with.. applicable: Rancho Seco procedures, with the'
. exception of-thei10 CFR 50.59 esaluation.
,L.
,
X ' Jg 4 .
, ', ,
;The?SMUD engineeringistaff hadd 'etermined that an unreviewed safetyc
,que tion 4did not exist. ;The basis /for this determination was that:
. ., , ..
;l sr sThe AFW ' valves: utilized 'ibTM-89-11 would be closed to protect'
' "U # ' SGrintegrity when or if ' plan't conditions existed which would
f ', require:
t
'
AFW. system-operation! .
.
'
.
O "O -
'The probability of~an accident.previously evaluated in the
L ' Safety Analysis, Report would .not be increased.
,
< +
~
/ . . ; , .
The erigineers' considered that~the -1l inch non-safety-related lines
'weFe only'a temporary modification, t'o be used during cold
shutdown,.but isolated (by closing the AFW valves at the connection
~
,
. points) during' normal' plant operations. They considered that a 1
stress analysis.of the AFW system was not required.
The SMUD engineers apparentlyfoverlooked the fact that the pipes
~
would still be physically connected to the AFW system, even during'-
normal plant operations. The ' potential therefore existed for
- 3 .overstressing the AFW piping system if a seismic event occurred.
Without an adequate pipe stress analysis for the AFW system and the
support system of the -temporary piping, the AFW reduction in margin
- .of safety (if any) was unknown.
10 CFR 50.59(a)(2)(iii) provides that a proposed change, test or
experiment shall be deemed to involve an unreviewed safety question j
if the margin ofLsafety as defined in the basis for any technical !
specification is reduced. ]
The USAR, Appendix 58, Section 58.2C, lists the' Auxiliary Feedwater
components and associated piping as a Class I system. Class I
systems are required to have the ability to withstand postulated
-earthquakes. 10CFR 50.59(b)(i) requires that the licensee maintain
records of changes in the facility and of changes in procedures
made pursuant to this section, to the extent that these changes
, ,
,
- * , . -
,
_
_
.
.' ,
H - 17 -
s.
,
constitute changes in the facility as described in the safety
analysis report. These records must include a written safety
evaluation which provides the bases for the determination that the
change, test, or experiment does not involve an unreviewed safety
question. The TM-89-11 was in violation of the requirements of
10CFR 50.59(b)(i) in that a record did not exist which provided the
complete bases that the TM-89-11 modification did not involve an '
unreviewed safety question. Violation (50-312/89-01-04; also see
paragraph 1.e.)
Prior to the exit meeting, the licensee agreed that the seismic
qualification of the AFW system had not been considered with the
TM-89-11 pipes installed and that a revised 10CFR 50.59 evaluation
J would be generated. The licensee's operations staff determined
that the integrity of the AFW system can and would be maintained by
physically disconnecting the temporary pipes prior to declaring the
system operable.
)
e. Lack of Calculations for Main Steam Safety Valve Gag-Mechanism i
The design engineer failed to perform / document calculations to
substantiate the load capacity of the device used to gag PSV-20544.
Main steam safety valve PSV-20544 lifted three times during an
August, 1988 reactor trip test. PSV-20544 was designed and set to :
lift at a system pressure of 1050 psi. However, during the reactor
trip' test, the valve lifted at 1018 psi and reseated at 930 psi.
This occurred three times despite main steam system pressure
* '
reduc; ions'when six other safety valves lifted. The licensee
issued Potential Deviation from Quality (PDQ) report 88-1459 to
address these abnormal lifts of PSV-20544. s-
p The SMUD plant I&C maintenance group tested the valve after the
event, and found it to have a lift set point of 1030 psi.
Maintenance personnel then adjusted the set point to 1045 psi,
which is within the 1050 +0/-15 psi acceptance criteria.
For revision 2 to PDQ 88-1459, the licensee engineering staff
determined.that a short term disposition of the problem was to gag
the valve (install a fixture which would prevent valve opening),
since PSV-20544 had a history of out-of-tolerance lift set points.
)
The long term disposition would be to remove, refurbish, reset and
L retest the valve to specifications and return it to service. l l
The documented technical justification for gagging PSV-20544
j included compliance considerations with the plant technical l
specifications, original NSSS system analyses, and seismic
qualifications of the valve. The justification appeared to
adequately address each issue. However, the licensee could not 3
l find any supporting calculations which justified the adequacy of i l the on-site shop design of the gag mechanism. !
l
l Rancho Seco procedure RSAP-0303, Section 5.1.1, " Responsibilities," l
states that Nuclear Engineering has overall design responsibilities l
!
l
I _ __ ________ D
, _
, ,
---- - - _ - - .
_ _ ,
'
g y " '.g. ..
,t
.
t - t . ,
- 18 -
. ,
r
includingdevelopmentof'theconchptual.designandthedetailed-
o design. Section-4.8 of RSAP-0303 states that the detailed design ! includes design' definition documents ~and design substantiation ;
documents'which present the design and verification that the design '
is correct. Design' substantiation documents include: design basis
reports, design verification report, safety analysis report,
calculations and interdiscipline document review notices. The toam
determined that'the mechanism used to gag PSV-20544 did not have a
calculation that' substantiated it's design. This lack of a design
calculation is considered a violation of the plant modification
requirements established in Rancho Seco procedure RSAP-0303.
' Violation (50-312/89-01-04; also see paragraph 1.d.) 1
i
f. Evaluation of-Nonconforming Conditions (Air System Debris)
Established nonconforming condition reporting programs were not
utilized to obtain engineering evaluation of modifications cork
practices which resulted in debris in the instrument air system.
SMUD performed Special Test Procedure STP-774 to determine if the
Instrument ' Air System (IAS) back-up accumulators and associated j
piping and valves would function as designed following piping i
modifications. Part of,STP-774 tested the IAS check valves in l
supply lines to the accumulators. These check valves prevent the
accumulators from losing their contents upon a loss of normal ~
instrument air supply. During the test, five check valves failed l
to _ seat properly. Two of these valves functioned properly after
system operators cycled shut-off valves adjacent to the check
valves. The remaining three valves required corrective maintenance i
to restore them to proper operating conditions. Of these three
valves, two valves were found to have white plastic particles on
their seats which prevented the valves from closing completely.
Associated work requests for these three valves indicated that the
valves were cleaned and operated properly during subsequent retest.
The licensee system engineer stated that the debris probably was
introduced during piping modifications work at the accumulators.
'A nonconformance report (NCR) was not issued to obtain an
' engineering evaluation of the root cause of the problem.
Conditions control procedure (QAP-17), which was in place at the
time STP-774 was performed, defined a nonconformance as: "a
deficiency in characteristic. . . which renders the quality of an
item (materials, parts, components, or system) unacceptable or i
'
indeterminate. .-Examples of nonconformance include: physical
defects, test 1 failures. . . . Nonconformances are hardware
related."
'
The SMUD engineering staff stated that corrective maintenance on j
the specific check valves had remedied the problem and subsequent
IAS blow downs should have cleaned the system of any other debris.
The engineers . stated that an NCR was not required since the
deficient conditions had been corrected, i.e. the valves functioned
as required after. corrective maintenance and retest. They noted
!
,
mm .__ _ _ . _ _..____l__ _ _ _ . . . __ __
j
, *
_
, ,
- -- --- .- - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
, ' e f e, $ h ,, i
,.
"^gs :
%. ,
[M4 q_
<
a.
*
_ 19 ,
9 1," ,
.
1'; f, <
'l .
, 1
- <
'ds . ' '
that a judgement was made thatht'e source of the white particles
-
was'probably from the revised piping construction work (i.e.
l'
^
.
ineffective cleanliness control or control of pipe. joint' sealant),
.g - A
' '
and would by geometry be limited to the specific valves where
y9, 1 problems,were noted, and probably not impact the entire IAS; ;
*
.
< :
, ,
1
-During. plant tours, NRC, team members noted that numerous gas system.
' "
<
,y
'
.pipe joints (threaded connectors) were also' sealed with a white )
,.J ! sealing tape. Such tape characteristically can. shred internal to g * '7
'
y
*
l; .the piping system, resulting in debris such as' described above. (V
'; '
Such tape was observed on joints in radiological sampling systems,r ;
i 3and containment isolation valve gas' supply lines, upstream of flow:, -
"
,
'
, control solenoid valves or pressure regulators (e.g. HV-1503, .~ ;
4' 4 -
,
L ,HV-15037, HV-037; HV-15044 A,B,C; SFV-53613). )
' ,
., n
ML
'
~
c , : The nuclear industry'has reported numerous problems with air '
[ .
-
f 3
2
systems thatuinclude, for example, malfunctioning check valves,
: accumulators not properly sized and. water and debt is-in the system.- 3
Such problems have rendered air systems and' numerous safety-related
air-operated components inoperable. Responsible SMUD plant staff
should have been sensitive to this issue, for which licensees have
received repeated NRC correspondence.
. .
. . .
-Failure:to.. issue an NCR was based upon an evaluation of specific
, . , <
malfunctions of certain valves, but did not address the more
'
. fundamental'issueLof unacceptable' work-practices, which'resulted in
-
debris in air supply piping systems. Issuance of such an NCR could
~have triggered a' review of other uses of sealing tape and {
' identified'the generic situation described above.
Apparent. violation (50-312/89-01-05). '
The current SMUD procedure governing the reporting of nonconforming
*
i ~ items, (RSAP-1308) stated that a PDQ is necessary when ..
;,"
" Test / Surveillance failures cannot be resolved by corrective
, maintenance and retest." However, RSAP-1308 also states a basic i
: criteria for initiating a PDQ is a " degradation, damage, failure,
malfunction or loss of plant equipment which is unexpected or not
the result of normal wear". These criteria appeared contradictory !
in the manner in which they may be interpreted to allow recurrence 4
of the air system. debris issue discussed above. The SMUD Quality
Assurence (QA) personnel agreed that RSAP-1308 required
3 clarification to ensure that any abnormal or unexpected conditions
found during corrective maintenance, as part of a test, or as part
of a surveillance was reportable as a nonconforming item. The QA
staff stated that RSAP-1308 will be revised to include this
t clarification.to preclude any repetition of a situation similar to
the IAS check valve failures during STP-774.
'y .
~- "
'The significance of this issue is that potentially generic problems j
'or conditions adverse to quality may go unreported. Appropriate
corrective actions'and adequate root cause evaluations would not be
effected,'to assure that plant equipment, components, and systems
4 perform reliably and safely. j
.
4
T
i s
lm.
. w
-
y
,
g
. 3
. , - - - -
'
y '
,
- 20.- 1
(9- , I J
g. Omiss' ions in Determination of'AFW Equipment Quarantine Boundary-
As discussed in Appendix C of this' report-(Report #2 of Design
J
Consultant), Item 1, the Nuclear Engineering Department Action Plan
included oversights in the determination of the equipment
potentially affected by the January 31, 1989 AFW turbine overspeed
event. Although of minor significance, the omissions showed some
inattention to detail on a subject.of.high visibility and
sensitivity.
,
h. Circuit Breaker and Fuse Coordination
.There' appeared to be inadequate consideration given to maintaining
circuit: breaker / fuse coordination schemes during the design-review
'
process, and on occasions where circuit' breakers were replaced
under Work Requests.(WRs), without failure:cause/ design reviews.
Limitations on the ability of engineering' to properly consider
coordination,.and the need for engineering sensitivity to breaker
malfunctions which occur in the plant, relate to.the following:
(1) Only a partial circuit breaker / fuse coordination study had
been performed on the Class 1E electrical' distribution system;
(2) There were no; Time-Overcurrent Characteristic Curves available
for .a large portiori of the as-built Class 1E ano non-Class 1E
electrical distribution system (although Time-Overcurrent
Characteristic Curves for,the largest family.of load center
circuit breakers..(70. Amps).were available and maintained).
(3)c Portions of the electrical distribution system do not have
^ ground' fault protection.
One't'ypical, Worki Request '(WR 159982)"andassociateddeficiency
.. report (PDQ 88-0869).showed that a'non-class-1E circuit breaker did
not,have proper amperage rating nor conform to as-built drawings.
Associated equipment was deemed operable, although the records did
not indicate that relay coordination implications were evaluated.
The licensee' demonstrated that'no una'cceptable loss of equipment
would ~ have occur'r ed,during fault conditions. However, the licensee-
acknowledged that each circuit breaker replacement should be
evaluated on=a case by case basis because a coordination study has
_not been performed on the balance of the' plant electrical
distribution system.
i
Numero'u's examplesiof the above type deficiency have been cited in
the licensee's quality assurance audits and maintenance work
packages. The licensee was planning the following future
corrective actions, which appear to be responsive to NRC Generic
letter No. 81-12, Generic Letter 88-15. Information Notice 88-45 l
and IE Bulletin 88-09. These related to AC and DC Class 1E and
non-Class 1E plant electrical distribution systems considerations
that need be given to verifying the adequacy of the voltage
.available to Class 1E equipment.
e m , , - - - - - - - - - - - - - - -
p
e .,..a.. ' '
S
2.[ '
#[
)
.
, %[) S #p ,
. - 21 -
i
- i y/
'~
'
g ., P
,
_
) i
e _
'
- -
Develop electrical distribution system design criteria and .]
*
.F guide revisions so that concise and accurate design criteria i
are provided as basis-for future system design modifications. j
. . 1
-
' Revise electrical load design date sheets so that the 'i
information. contained in E1011 series drawings represent
'
accurate information about electrical equipment and design ,
parameters, including. relay settings on a single drawing. !
-
Develop circuit breaker Time-Overcurrent Characteristic Curves ,
for Llass.1E 4.16KV and 480 V switchgear,-including 480 V j
MCCs, Non-Class 1E 6.9 KV and 480 V switchgear; investigate ~
instances of' spurious tripping. The curves will depict each
[- breaker's various protective devices and respective E
Time-0vercurrent relationships.
t
-
Modify 480 V switchgear.and MCC ground fault protection
systems to improve their ability to detect and isolate ground
faults and reduce the sensitivity of existing MCC supply
breakers ground fault sensors to achieve a selective
coordination with a larger percentage of a given MCCs breaker
population.- Install ground fault protection systems on
individual MCC feeder breakers rated 35 ampere or larger to
provide selective breaker coordination for the entire breaker
population. ,
i
However, there was no current program requirement to notify
engineering staff when problems are experienced with circuit
breakers in the plant, and it appeared that simple like-for-like
replacements were being made under routine work requests. In view
of.the incomplete status of corrective actions for the breaker / fuse
coordination discrepancies, it appeared that engineering
notification and evaluation would be appropriate to assure that
, , failures were not indicative of points of vulnerability
, .g -attributable to poor breaker coordination. Thus, absence of
'
required engineering involvement in routine repairs of protective
devices appeared to be missed opportunity for early identification
, of problem areas. Open Item (50-312/89-01-06).
2. Procedure Weakn' esses and Flawed Work Instructions
j a. -Equipment Investigation Task Plans
; Investigation plans. developed for investigation of equipment '
;
C ,t ,
. failures were incomplete and reflected an inattention to detail and J
< lg g g communications oversights between licensee organizational elements.
'
*
*
4 An NRC Confirmatory Action Letter (CAL) was issued February 1,1989
'
l 'to confirm planned SMUD actions as a result of the January 31, 1989
" -auxiliary feedwater (AFW) overspeed event. In particular, the
, licensee committed to develop a plan for investigating the event
I anc. presenting it to the NRC prior to commencing troubleshooting of
K the AFW pump turbine governor, overspeed device, and AFW system
valve alignment. The CAL also specified that the licensee would
- _ - _ _ _
_
_ _ -
- - - - - - - - - - - - -
l
- t.
N %
- 22, , ,
4 j
1
conduct a thorough investigation and obtain a full understanding of
the, overspeed . vent.and define the pre-restart and post-restart
corrective actions.
Th'e SMUD Auxiliary Feedwater Investigation Action Plan, dated
-
February 3, 1989,-and the. Auxiliary Feedwater Incident Failure
Investigation Work P.lan; dated February 7,1989, were presented to
-the NRC inspection team'for concurrence. These plans did not
contain,the information-defined in the NRC reviewed master plan; 1
f.e. detailed information such as engineering evaluations, basis !
for establishment'of the quarantine boundary,' controls for shipping
.
the governor, an inspection plan for: the. governor at the vendor's
facility, andscoordinationof failure . history reviews, data
: gathering, and evaluation plans. .The submitted plans provided only
items such as background information on the event, definition of :
the quarantine ' boundary, and outlines of what SMUD intended to do
in work requests. .
The'NRC team later'found that-the submitted documents were
superfluous'in view of more complete work control documents' that l'
had been generated under the established Rancho Seco work control
program, i.e. Potential Deviation from Quality (PDQ) documents.
The NRC team also.found that, although identified during SMUD
preplanning meetings, the submitted task plans did not incorporate ;
review of vendor plans and possible effects of the water deluge '
incident that occurred soon after the overspeed event (this
incident could have potentially damaged or altered the overspeed
trip-linkage).
The-submitted plans clearly reflected a lack of communication I
between management, engineering, maintenance, and task planning
organizations,,and apeared to be a result of hurried efforts to
present " Task Plans" '.n the context of terminology used in the NRC
CAL (rather than submittal of the established SMUD work control
documents). The licensee subsequently withdrew the task plans and
submitted the PDQs and work requests (WRs), which contained the
specifirs to' implement the master plan, (PDQ 89-0152 and
WR-01f.41420-0 for the turbine overspeed trip (OST) mechanism, and
PDQ 89-0148 and WR 01541360-0 for the governor).
b. Work Instruction and Procedures For AFW Equipment Investigation j
Work Requests (WR), developed by SMUD to investigate the cause of
equipment malfunctions during the AFW overspeed event, contained
poorly written instructions; e.g. WR 01541360-0 (investigate / remove
the turbine. governor) and WR 1541420-0 (investigate the OST
mechanism).
Although SMUD procedure MAP-006 specified how to write a WR, the
requirements were not as stringent as those for other types of work
instructions (e.g. SMUD did 'not. consider WRs to be procedures, and
thus not subject to the stringent INP0 guidelines for Procedures,
as incorporated in the Rancho Seco Procedure Writers Guide). As a
result, the WRs did not follow the INP0 guidelines, e.g. these WRs
l I
- _ _ _ .
p-
- .
-.
.
+.
1
.i.
- 23 -
.
1
1
used' inconsistent terminology (within the body of the WR and
between the WR and the vendor manual), had inconsistent placement
of notes, and required several actions to be performed under a
single procedure, step. 1
?
(1) Work Requests WR 01541360-0 and WR 1541420-0 did not include
as-built drawings of the AFW governor Overspeed Trip (OST)
' device, or trip / throttle valve HV-30801. The SMUD planners
prepared and included sketches of the OST linkage and attached
an uncontrolled drawing (SK-M-011) to'the WR (so that the
craft would know where to take measurements specified in the
WR). The resulting WRs were hard.to follow and led to
'
~ confusion on the part of the craft that had to use them. The
, procedures had to be rewritten'several. times (01541360 - once
, and 01541420 - three times) due to work flow deficiencies and
inconsistencies found during field implementation.
7
~
-
.(2) ' Revised WR 01541360-1 was used by the craft to remove the
4
governor and investigate the governor linkage;.it did not
.
.
'
contain acceptance criteria for measurements of the governor 3-
valve linkage clearances. Licensee staff stated that this was: -
'
lf ,
,
'
, intentional, and that measured values would be evaluated by -
, ,
,
the engineering staff. However, the WR also did not contain .
'
toleran'ces for measurements, to assure the accuracy required ' .;
4
J for the engineering evaluation. Without dimensional . \ ?
'
-
tolerances, valuable information could have been lost when
,
.
,
;r'ecording data during this investigat' ion phase. After NRC . >
. ' team questioning, separate instruction sheets were provided to
)
'
, 'the crafts. (ANSI N45.2.8-1975 defines acceptance criteria,
'
'.in part, as limits such as dimensional tolerances.) Step , ,
'
* *
6.2.6.3.(2) of MAP-006 required that a work instruction .
#~
utilizing technical manuals, drawings, and Planner technical
expertise shall include acceptance criteria. The lack of
acceptance criteria in WR 01541360-1 is an apparent violation '
(50-312/89-01-07).
SMUD work order packages generally did not appear to provide
crafts personnel with tolerances for measurements, unless they
happen to be in a specific maintenance procedure that is
referenced for use. The underlying basis for this is that the
crafts personnel "know" that unless otherwise spec 1 N ad that ,
all measurements are to be taken to the standard + 1/16 inch.
'
This value is identified in the " Machinist's Handbook";
however, not all of the crafts personnel were required to be
familiar with this handbook, nor was a 1/16 inch tolerance
necessarily adequate for assessing trip linkage malfunctions.
, (3) Revision 2 of WR 1541420 was not adequate to investigate all
potential problems with the OST linkage. It specified that
valve HV-30801 be opened manually, prior to manually tripping
it using the OST. When that step in the procedure was
performed, the valve tripped as required; however, valve
HV-30801 should also have been opened using the motor
operator, to properly simulate the conditions at time of
- __
m - , ,
, ,
n .= y + -+
,
, ,
( -*'
,
'
% 7 , , ,
'l
- 24'-
. 1
' I
f- . .
.. o
ir. - (,
sfailure;(valua61b 'information in the licensee's investigation
would have been lost,Jsince the plan was-to disconnect the
linkage for subsequent investigation of the status of the
~
Terry turbine).
~
In response to the NRC concern, the licensee
issued WR 01541420-3, and opened HV-30801 using the motor
operator, thereby discovering that the manual OST device
~
failed to trip the valve.
,
(4) During observation of the work required by WR. 01541360-1, the
craft reached step 17 which required several actions.to be
performed. In particular, the WR required the craft to:
Manually open the governor valve, and Measure and record
measurements #1 & 6 as shown on WR Attachment "A". . After this
measurement is complete, close the valve. If required, ,
manually move the governor " shutdown rod" to release the
governor's servo rod." The craft completed the first two i
parts of the step, but the inspector noted that they forgot to- j
close'the valve as required in the step. The step.was signed '
as completed, and the craft went on to step 18. The QC
inspector;apparently did not notice the missed. action; (the QC
inspector was required only to verify and sign that the
measurements were taken accurately).
-(5) Step 18 of WR 01541360-1 required the craft to: " Disconnect-
the north "Heim Joint".of the governor linkage... Lift the
governor's servo rod end measure the additional. stroke iength
available from the gcvernor." This would have been impossible
to perform since the governor valve would have.still been open
from the previous missed step and valuable data to this
investigation could have been lost. The NRC inspector had to
identify to the personnel performing the WR that they had
missed an action in step 17. The inspector considered that
use of multiple actions within a step in the WR contributed to
the craft missing the action.
c. Inadequate Post-Maintenance Testing of AFW Turbine
Work instructions / procedures were inadequate to properly control
PMT (Post-Maintenance-Testing) after governor maintenance:
,
Work Request WR 01452120-0 prescribed the testing to be
accomplished after replacement of the AFW turbine governor; several
discrepancies were noted: ]
(1) No incorporation of vendor procedures for a non-emergency
-start of the Terry turbine; (nor did the operations l
' surveillance procedure A-51 contain the vendor manual
directions for 'a non-emergency start of the Terry turbine).
A non-emergency start of the turbine could have allowed
gradual increase in steam flow and turbine speed, until the
turbine approached set' speed and.the governor started to exert
control. This would have allowed time for operators to take
shutdown actions in the event the governor failed to control
" properly.
1
_ _ . _ _ _
g a
- - - - - - - - -
.; , , ,
7----
.
*--
q: g q , , , :
w v. n,,;-
p, 'y,,
.~
- gs : "
Ty ,e
]*4 g ::- ,3, ta p,'e
'
s J
7 , ,
'- ('
F
, ,,
' t s
'. ., . ,
,
s(2).. .No . valve lineup specifications to be_used for the Terry
*
d i *
i'I,f
~3 i
turbine test, other than a simple statement that operations
'
(j
,
' s,,
gf i ' < > ' i - should conduct la valve lineup. .F
y, ,
4
/ l
)- -
.
'
~ Engineering' consideration of' valve lineup and flowpaths. , .1
jh -p ; '
,A may;have' permitted a lower resistance flow path to avoid-
.
1
,
4 -- ,
s
overpressure ofLAFW piping in event-of turbine
-
Y g :V f: J.:. -
-
/
'
.overspeed. .
~
[ ,
'
'i 'k3) Notrequkr~ements'formonitoringturbinespeedorcontingency '
;
.ft- .#
.
1
plans,*despite the fact'that'the vendor manual identified a ,~
P9 number'of precautions and warnings;ie.g. j
e-
3
0 A -
4" WARNING, Before making any adjustment to an operatingL
*
Jgovernor.on a prime mover, check that the. overspeed
, shutdown' system'is operating properly", and
'
-
" WARNING, To protect against possible personal injury, ;
,
loss of life, and/or property damage when starting the '
L engine, turbine, or other type of prime mover, BE L
PREPARED TO MAKE'AN EMERGENCY SHUTDOWN... should the l
mechanical-hydraulic governor (s)... fail." j
'
r (4) No acceptance: criteria provided for testing the new governor. j
j
~
I, (5). No requirements for testing sub-components in the governor, L
.
such as the oil; dump solenoid. !
~
(6) No evidence.that the WR was reviewed by the system engineer, !
maintenance engineer, or design engineer for technical
content.
! ,
!
Program controls governing Post Maintenance Testing.(PMT) were
defined in SMUD procedure MAP-006 and its Attachment 15, but were
, minimal. l The MAP-006 defined maintenance verification testing as )
testing conducted to verify. satisfactory completion of the work l
,g request as specified in the body of the work plan using the
guidelines of Attachment 15. Revision 5 of MAP-006 specified that:
~
j
"If maintenance verification testing is required, the discipline
planner shall specify the testing in the body of the work plan. !
Guidelines are outlined in Attachment 15." ,
.
.
l
Attachment 15 addressed the AFW turbine; it included only the i
following requirement.for PMT following maintenance on the turbine l
governor system: ;"... the turbine shall be demonstrated operable
; "
by performance of a turbine manual start test. Ensure that the ;
turbine operates at rated speed and pump flow ... Ensure that the {
- - . governor control system is capable of controlling the turbine !
throughout its' full speed range and the governor limiter stops are i
'
operable." A precautionary note for AFW governor testing was
limited tota discussion of the need to ensure that the oil level
*
. , m/ ; wasproper;priortoattemptingastart,toensurethattheturbine 1
'l -
.,,
; starts on demand, the turbine accelerates smoothly to the governor j
,
- 1
.
i
. 1
. .
+ . x
.g . . .
._
_ - - _ _ _ _ _ _ _ ..
p ... 3-
J '
- 26 - l
s.
-l
L
j
setting, the governor takes control and maintains turbine speed, ll
and the oil and bearing parameters were correct. !
Work Request WR 0154212-0 was written in such a manner that it l4
assumed that the new governor would operate properly; i.e. it
relied solely on the OST to protect the AFW pump, turbine, and
motor in the event of a governor failure. This was despite
considerable industry experience concerning problems with governor
failures. Also, it presumed that the OST device would operate
properly. .This was also despite.the fact that there have been OST
.
failures both in the industry and at Rancho Seco (as noted below). ,
SMUD personnel' apparent 1y'did'not consider such data, or did not !
consider it to be significant, because no follow up action was
taken to add warnings / precautions (concerning AFW governor testing)
<
to Attachment 15 of MAP-006.
L .
:The lack of p'ro' visions for testing.the' Terry turbine in a
"
non-emergency start configuration, precautions or warnings in the
WR as identified _in the-vendor manual, valve line-up requirements, i
acceptance; criteria for testing the governor, testing requirements
'
for sub-components such asithe oil dump? solenoid, Land review by
applicable. organizations for technical content are' considered an a
unresolved item (50-312/89-01-08).' , i
.The inspector' discussed concerns over,.the adequacy of WRs with
responsible District personnel.' During these discussions, the
licensee indicated'that flaws with WRs had been recognized and
-
evaluations were in progress to determine corrective actions. Some
of the actions being considered for implementation prior to startup !
were the following:
(1) Maintenance verification and operational tests would be linked
together. j
(2) Each WR would have a maintenance verification test or a i
justification why not (this was already identified in i
procedures during this inspection).
(3) Each clearance would have a PMT. The threshold was already
established for some maintenance verification tests (i.e.,
leak checks) with others possibly to come.
(4) The PMT would be a WR with the same number as the clearance
number and they would be computer controlled.
(5) Each WR given approval to work would have a clearance number
and that number would be entered into the computer by
operations.
(6) PMT type WRs would be prepared by maintenance planners (due to
quality considerations). This would include maintenance
verification tests, operations tests, and variations. The
system engineers, quality engineers and operations engineers
would then review and approve the PMT WR.
!
_ _ _ _ _ _ _ . _ _ _ _ _ - _ _ _
-- -
.
, ^ -
_
.
- 27 -
t t.
I
i
(7) WR would be. written under guidance of the procedures writers i
' guide.
'
i
In addition, as a f result of the concerns identified above, the j
licensee indicated that MAP-006 would be revised'to specify the
'
~j
.particular' vendor manual sections and drawings to be used in the q
WR. >The' licensee was also reviewing the need to. implement the f
' Procedure Writers Guide through MAP-006, which would include '
training:for personnel who write WRs.
d. Inadequate Post-Modification Testing of a Check Valve
As discussed in Appendix C of this report (Report #2 of Design ,
'
. Consultant), Item II, installation instructions did not provide for
functional tests of a newly installed check valve in the nitrogen
supply to the Condensate Storage Tank. Proper functioning of this
valve is necessary to prevent water intrusion into the governor
control. oil,'and assure reliability of.the AFW turbine governor
(the governor had been replaced several times due to moisture
. intrusion problems). Proper testing.of this valve, prior to plant
startup, is an unresolved item..(50-312/89-01-09)
e. Delays in Correcting Inadequate Procedures
. Management Observation Forms indicate the crafts personnel are
* ' unhappy with procedures.that cannot be worked as written and yet
the revisions are not getting incorporated in a timely fashion and
they ar'e continually trying to work with procedures and: work.
, , instructions that are not compatible with the work they're trying
to perform (e.g. Procedure SP-627A). Craft personnel had
e- Lidentified. specific procedures that were used for surveillance ;
a,nd/or preventative maintenance that were in need of revision to .
, i
,e make them acceptable; several months later when the work was issued'
3
to be' performed again, the same unrevised procedures were available. '
.to use with the work order. A " Management Observation Form" showed '
, data in consequence of this problem: On August 14, 1988, PM c
l
.
. procedure EM-175 Revision 6 was being used to complete required
. maintenance on a spare CRD breaker; 6 months previous to this the
'
' ? '
f_
.same procedure and revision had been used for the same task and at
"
,
,
'
*
that time a marked up copy of the procedure had been submitted for ,
#~
revision (the one used at that time was not clear and very hard to
'
' , Jc ,
-
{ '
work through). Another " Management Observation Form" identified
that some procedure changes (PICNs) were still not being
.
incorporated into the procedures in a timely manner.
f. Incomplete and Flawed Work Requests
Work requests contained instructions which were incomplete / flawed.
..
Some such work instructions (used as procedures) were not
technically correct,'nor consistent in format, nor tested. The
Terry Turbine governor replacement is an example; it was even
i
identified by the craft personnel in April 1987 (during that
1 i. ___-___
_ _ . - - - _ _ . - _ _ . .__ .
. . . .
m._-
. 3 S a.. ,
% gg m c
; , .
,' .
,
,
,
,j
,y
e ._~[
L +f;mp
; ; s
-
28 - .
1
,
,
1+ c ,
j
*
> ,-
- i? i
,, . _2 q
. q "
2 .
j
y' .
replacement) that;a'new procedure.needed to be generated before'the
, ;
governor was replacediagain. The - replacement, on January 30, 1989- J
: was the second' replacement since April 1988. 1
'. . . x '. .. i . 'j
Work! instructions"did'not' alwajs reference all of the applicable q
,docume'nts.that should be referenced- '
i
'
'
1
p i
.; .q , The Terry ' Turbine) governor. work ; order did not go into
,
-V
.
g. < sufficient'det' ail'as;.to1 exactly what> portion (s) of the vendor
-
technical manual should:be used for the replacement and-
subsequent post maintenance testing.
_.
, .i . , , ; ,
. *The Engineering Change'NoticeJ(ECN), which specified
Linsita11ation of a nitrogen supply line to the governor casing,
did not identify that an'.0perations Procedure provided .for an
'
'
"t _
interfacing water line valve'(condensate storage tank drain
v'alve) to be normally aligned open.
^
Work packages had been' released to the field'with' indications that.
the planner may not'have performed an effective jobLwalkdown before
he planned the work. This was most obvious.with work packages that
had been worked previously, with' the newest work package " cloned".
.frnm a previous package:
. The Work' Request for. inspection of the Terry Turbine overspeed ~ \
, trip mechanism demonstrated.that the planner was unaware of-a-
plug in.the side of-the turbine journal bearing cap, that
'
allowed direct. observation of the'overspeed trip unit fly
, weights and trip plunger.
. On at least one occasion, the planner provided'. detailed work.
,
instructions for the disassembly of a component when there was
an approved SMUD procedure.(M-22) in existence. This was
contrary to MAP-0006, section'6.2.6, which states in part that -
the order of preference for work order instruction preparation
is to utilize all'of an approved procedure or a portion of an
approved procedure, and the last resort is to provide' work
. instructions in the work order.
" Post maintenance testing requirements being incorporated into work
orders did not necessarily ensure equipment or system operational
readiness, e.g. the occurrence of the overspeed of the AFW Terry
turbine / pump when the turbine trip valve failed to close with an
overspeed trip signal applied to it. (The team did not assess what
'
post maintenance testing was performed after the last time it was
M0 VATS tested.)
Repeat maintenanceLactivities did not appear to receive the same
degree of planning, review and approval scrutinies as did original
- packages. The turbine governor had been changed out 3 times and
'
during all of the changeouts it was never recognized that possibly
the system could fail. . Additionally, " Management Observation
Forms" included several interviews of crafts personnel who were
concerned that repeat jobs were,being performed by " cloned work"
,
*
9%
-'
., !4
-
'
"g .
>: _ _ _ _ _ _ _ _______ _
crpm -
,
,- .; ,
~ '
,
fij
-
f M
vwy
,
- , '-
si
v>
.
*-
gf i4 f t: N )y,j< #;, m >
n
> . 29 -
. . ,
,
1,'.
,
,
.- ;
,
_( e
.
p_ , , , l E , -- r ?
L, >
,
.
g
[t '
'_
AI o .Nw ,
, i
kg '4 /[~
. ,
pack' a ges; t.h'e y feitlthat all' work 'should be treated.'as.an_ original ,
task and'should receive l1ndividual preparation,
~
t y
V
.
"'
1 ..s . . - . . .. .
m
iSafety reviews of work order packages did not always consider th'e' t '
ii :
'
,'
; ultimate possibilities for personnelcinjuries,from. malfunctioning _
equipment.l This was' highlighted by the' turbine overspeed' event in 9
-
which there were craft personnel standing right next to,the- ~1 .]
-
. 1. equipment;'they.-were'apparently not. prepared for the possibility ofe ,
!
< *. '
" the overspeed .nor conscious of-the potential dant;ers presented when
1 ' it. did occur.. They'. remained in the area with:this equipment in an
D . unsafe conditionifor at least one minute. ,
j
cWork~ order packages did not' cont'ain special tool identifications;.
'
'
it' appeared lto be,left up to the crafts personnel (during'their _' '
: walk-down or tail-board meetings) to identify what tools.and other
equipment will be. required. A review of " Management Observation
H. . Forms". revealed that on several jobs the craft. personnel were not
,. aware ofJ special! toolsthatould w be required for:the work; for one
"
job they were not aware in advance that'they would,have to'obtain ap
.
-
key to unlock'a cab.inet containing the needed special tools.
4 ' Work packages were'. reviewed and accepted for completeness and .
accuracy and sent'to the records storage for permanent storage withl'
various minor' ambiguous, missing or incomplete data., These packages
,
w ,
>
had been' reviewed by quality' assurance. staff,;who apparently, .j
overlooked or accepted the observations discussed'balow. q
l(1);.WR'01370820 was written to vent :and adjust the governor'after -
'
'
'it was changed out and filled with new oil on WR 0137082C:
;
Pg 4(1) There was no= signature for'the prerequisite that i
required verification'nthat a prior work request was complete'd
and the. governor was filled.with oil to'the proper level; 1
Pg 4(7) Step 10 of the ~ work instructions. required the
-
craftsman to record the rpm'at which an action occurs; but the
comment section of this step stated that the system engineer
had this information. For complete equipment history records
and to ensure that the step was completed accurately,gthis
.
value should have been recorded-
- !
s ,*
-
Pg 8(2) Step 7 of the QC instructions required verification of ,
the oil added to the governor via the green tag attached to .
'
, j ,
ef f, n , - the oil container; this step had been left' blank'(however, the y yI(.)
"'
l"
)
craftsman's description of work indicated that the'prope'r oil
was-added to the governor); '
3 ,
-;3[ ( *
.
Pg 8(3) Step 9 contained two sub-steps'(a and b) with a single .
_l f);9N}g :
,* g ,
9 QC sign-off. Applicability of the verification'to both
sub-steps was ambiguous and not a good practice. , ,
.l
j
# p syv ; 1
j
W ig . .
,
b b
Jw A # \
g
,
~g .
.
", e
.
. .
l
_
, , , - - -
' ,
i %, '4.
,
.. .- 30 - ~
A , g
p
W
'
*
f.7y Pg 10 The workers' write-up indicated that the governor was
<
filled, vented and adjusted as required; there was no- < 4
1 signature to identify who did the work- - 4
[Q, ,
i
L
b / J General - There was no warehouse material' withdrawals slip in
,!J; - this package for the oil'added to the governor, r
'
, < . ,
,
, LP g o (2) WR 01382480 waswrittentofunctionallychecktheinstrument$ e
* *
. installed to! supply nitrogen to the Terry Turbine. governor-as: .
L N 1,u part of ECN-2990:
'
,
;
'
, \
a w
- 'I; Pg 7 There was no method of testing identified; the requ' ired _l
,
'
data was ambiguous; '
'
>
c
Pg 8 There was no QC required according to this page'of the
work request, under which a craftsman calibrated a Class 1
component. ,QC involvement in this calibration of Class 1
components appeared warranted; 4
'
Pg 10-The craftsman's write-up. indicated that the output of !l
flow indicator FI-30801 was set at .5 scfh, and an instrument
'
data" sheet was generated;for the component. However, a full:
range function test was apparently not performed for the
'
component. -The licensee stated that accurate values of flow
indication were not necessary, and only an indication of'
positive flow (through the full' flow range) was needed for' !
this nitrogen supply line. This appeared to be a compromise- :
F to'a rigorous calibration approach to instre4ents in general.
(3) WR# 0137082C was written to change out the terry turbine
7 governor and modify the oil cooler: i
Pg 11 step 12 was marked "N/A",'without a brief' explanation
(as recommended by work request procedure MAP-0006). The i
licensee noted that QC concurred / signed the "N/A", based upon I
the work step also being included in the vendor manual, which j
~
was invoked and verified in step 14 of the instruction. A
note to this effect would have been appropriate at step 12,
and consistent with the intent of MAP-006;
!
'
Pg 12. step 15 contained two sub-steps (a and b) with a single
QC sign-off. Applicability of the verification to both !
sub-steps was ambiguous and not a good practice;
General - There were no QA green tags, nor warehouse
withdrawal slips, attached to the completed work request.
(4) WR 01594540 was written to reset the low discharge pressure
switch PSL-31762 for the "B" Main Feedwater Pump as an interim
disposition of PDQ 88-1292:
" Pg 2 There was no issue number assigned to the withdrawal of
the snubber from the warehouse,
f , ,
j _
. ;
/
f
4y 2.
h.
$ {
' I k _ _ .___.._____ __ _ _
]
, , , _
_ _
,
-_. _ _ - _ _ - _
[, ) he
~
m < -
' ' '
7
y ,wr > -
1
<
:
S,
g..
Mk
,N. ~',
'
'. ~
J
L- 31~-
'
l
;
, , , .
.;
'
r
,
'
'..n<
. A '- 1
#
r I
s
..
,
Pg;3 of the PDQ required that an evaluation be ' performed 4or
^
#
' theLneedito clo'se both MFW block valves (on the loss.'of both.
MFW pumps); there was1no documentation with the WR.to indicate
i._* ,
.
'that this evaluation was performed or'was in process.
-
, q
# ^ a ..
,
o .. ,
(5);:WR 01594530 was written to reset;the. low. discharge pressure:
i c
4 -
'
<
1
7 witch PSL-31761 for the "A" Main,Feedwatef Pump as an intarim4
j disposition'of PDQ-88-1292: , [g Q
,,2 , , , i ny t . - ....
,
6 Pg 2 There'was no issue numbe,r assigned to'the'w'ithdrawal *
of?
,
>sj W , the snubber.from the Warehous'e.* ,, 4 b , >
g.
'
'
,F"-
-
"
f yt ,
<
.
.
Pg 6 The write-up ab'out the'tieflon tape on rthe' new snubber did
~
'9
'
1
, not. have the s'amet follow up comment .that w'as'made' for the
.. previous workerequ'est for the other feed pums iitjappeared
' '
, < that--this. work' request should alsolhave noted,that the
<
" '
'
equipme'nt i
installed had (prohibi,ted)t teflon ' tape iri it, and.
v o.
was'also'subjsct,itoevaluationlunderPDQ-88-1347t
, ,
, , js ~
3 "1
,5 , .. e i i
Pg' 3 of the PDQ' required that,an evaluation:for the need to
close~both MFW block' valves;on'the loss ,of'both MFW pumps be
'
performed; there was no documeritati,on with the WR to indicate
.] cthat this evaluation had been. performed or was in process.
, ,
*
Pg 1'of. Attachment B did notiindicate that the available
W7 . quantity;ofthesnubbers~ha's'[be.en'reducedduetothiswork
,
f . request'(or-thejwork request for the~other feed pump); it .
y , ' appeared that the work requests should have been coordinated
,- to reflect' current materials status.
.*
(6)'.WR 01601280 was written to repair FV-91007 on the service air -
-system.
'
Pg 6 step 6.. required that'the deficiency tag be removed and-
attached to-the work package; the. tag was-missing. There was
.,
_a rote on the Cover Sheet which stated that the Deficiency Tag-
was removed and thrown away; the individual that made this
,
entry.only' initialed it; the initials did not appear to be
'those of the worker who.inic.ialled parts of the WR; i.e.'it
was not obvious why this was done and who made this entry.
Cover sheet - The craftsman who signed as having completed the
work did not print his name, as well as signing it, on the
cover sheet; this did not meet procedure MAP-0002 step 5.3.17,
which specified that the craftsman will do both.
(7) WR 01379660 was written to support STP-1070, which was the
checkout of the Auxiliary Feedwater Pumps: ,
, Pg 4 step 3 of the work instructions required that the drawing
number'and revision number .Or tne tubing installation be
K , documented; there was no documentation of these items. 0
th
4
. .
(
.f
- _ _ - _ _
s .;
i -o <
:v - 32 -
.
t
Completed work packages reveal a lack of consistent indications
that the crafts personnel were attempting to identify why a
particular piece of equipment had failed; (MAP-0019 specified that
the craft personnel are-to make a root cause determination of the'
'possible failure mechanism and include it in the work description
write-up.).
3. Weak Control of Vendor Documents '
Some vendor manuals contained insufficient current data, and were not
maintained in such a manner to/ assure availability.of proper information
to. personnel. planning and conducting work.
Such deficiencies with the control of vendor manuals had apparently been
previously identified. ' ' As';a result, SMUD instituted a corrective action
plan, thejVendor Equipment Technical'Information Program (VETIP). ;This.
program was, initiated in mid-1988. However, full implementation of the
program was not expected to be completed for Class 1 and Technical
~
Specification related equipment until mid-1990.
a. 'Personn'el perfdrming'the! investigation of the AFW turbine overspeed;
eyent had available only uncontrolled drawings and/or drawings'that
~
-were not reflective of as-built conditions, and a technical manual
~
in which it was very' difficult to locate information.
-
(1) : Some design ' drawings in the vendor manual were'not entered j
. into the plant drawing system and the drawings:in the
~
'
'
. technical manual were not identified "for information only"
(however, the licensee indicated that craft persor.nel were
traired that vendor drawings-were to be used for information
only). For. example,' drawings 8-1074-X-37168 and 67270E were
not located in the document control system (DCS); drawings
719960 and NP 796 were in the DCS, but were not marked as "for l
information only" in the vendor (Woodward)' manual.
'
(2) The. drawings of skid mounted vendor equipment in the technical
manual and the DCS aid not reflect the as-built conditions of
the Terry turbine governor, OST mechanism, nor the linkage for
the trip and throttle valve, HV 30801, as noted previously.
As a result, the personnel using WR 01541360-1 and WR
1521420-3 (the revisions to the WRs that were actually used by
the craft to remove the governor and investigate the OST
device) did not have drawings that could be used for the-
investigation. As < result, the planners had to make sketches
of the OST linkage so that craft knew where to take proper
measurements. In addition, WR 0145212D-0 that was used to
install.the governor prior to the overspeed event also did not
have drawings that reflected as-built conditions.
(3) The Terry turbine technical manual did not have each page
- . uniquely identified by a specific page number. As a result,
it was hard to locate pages or to communicate with personnel
as to specific items in the manual, and difficult to reference
acceptance criteria for quality control inspection points. i
L
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
,
.
a. . . o- ~
.- -
33 -
{
b. Vendor technical manuals.were not being revised'in a timely manner
to ensure the planners and the crafts personnel were using the most
current.information when planning or performing the requested
maintenance.
(1) Woodward Governor Technical Manual (M5.06-57) for the Terry ,
-Turbine governor had 2 outstanding revisions that had been l
sent to engineering for approval', but were not yet approved
and ready for inclusion in the manual (one had been-there
'
since July 1988 and the other since December 1988),
c. Vendor technical manuars .ere not controlled as rigorously as
required by the SMUD r.rog.'am:
(1) Location unknown, of controlled copies of technical manuals.
Y
(2) Manuals checked out in excess of the 14 days allowed by
procedures, (no formal or informal letters issued requesting
return of the manuals or for updated withdrawal requests).
(3) A controlled copy of a Vendor Manual had been identified as !
lost; a replacement copy was made and identified as copy 1A;
this' copy was,then checked out to two individuals at the same-
time; the original number 1 was identified as having been
*
. ,
checked out to a third individual during the same time period. ~
., ,
~^ ' '
4.' 4 Inattention to; Industry Experience and Initiatives
.(4 . t
+ . 8
4
,;t' ,
'
[a.';TestingofAFWTurbineGovernor
w w ,
,'
As discussed. in item 2.c of this Appendix, -industry experience was
[' , insufficiently considered in development of testing instructions
!for the.AFW' pump-turbine governor, i.e. Work Request WR 0154212-0- t
s ,1 did not consider the considerable industry experience regarding AFW -
, 3
'
turbine speed control failures and failures of mechanical overspeed
devices, r
b. Debris in-Instrument Air System
As discussed in item 1.f of this Appendix, industry expedience was
insufficiently considered in recognition and action on debris found
in the instrument air system, after piping modifications for valve
control back-up gas supply accumulators.
5. Weaknesses in Communications and Engineering Support
a. Support Interfaces and Communications
Engineering support and communications appeared inadequate relative
to the preparation of maintenance work requests and verification
testing after maintenance of equipment.
l
!
l
!
g !
_ ____-__-_ ~
W (f ~
' '
~ ~ - ~ ~ - ~
'
;
-3 o
-
<
'
f 34 - '
. , s 4
Is
'
:
It appeared that recent program changes and reassignments of
responsibilities had created a gap between' support engineering and
maintenance, concerning such maintenance related activities as WR
preparation, procurement and PMT; i.e. no single. point of
accountability _among the engineering groups, any discussions that
took place between these groups was on an. informal basis and not
documented, and no clear lines of communication established between
these groups.
Engineering general support for maintenance appeared adequate, with
the following exceptions:
..
(1) SMUD procedures did not require a technical content review of
PMT instructions by system engineers, maintenance engineers,
or design engineers.
(2) There was no requirement for planners to contact system,
maintenance, or design engineers (for review of WRs) during
the planning phase. (H.vever, it appeared that craft or
planners informally contacted the engineering groups, but only
if the craft or-the planners felt the need to do so.)
The system engineers previously were the focal point for all
information concerning the system. As such, they were aware of
maintenance and other activities that could affect their system.
However, as of mid-1988 their responsibilities were redefined;
since then, they only track and trend system performance.
Consequently, the system engineer does not get involved with WR
preparation or procurement.
Procedure MAP-001 denoted the responsibilities of the maintenance
engineering group, i.e. engineering support of the department
groups'as requested. This included reviewing and revising
maintenance department procedures needed to support maintenance and
modification activities. Notably absent were requirements for the
maintenance engineers to also review work requests which involve an
unusual equipment failure, temporary modification, special
processes (e.g. . temporary pipe leak seals), replacement parts
equivalency, and. interpretations of design information.
Of 15 maintenance engineers, five were electrical, four were I&C,
and six were mechanical engineers. Some'of the maintenance
engineers were*Osed as specialists (e.g., pump and valves, motor
operated valve testing, motors, ' snubbers, and AFW (due to large
amount of problems with'that system)). Most-maintenance .
,
engineering personnel on-site were relatively new and were l
contractors. They appeared to be experienced contractors, but in
general had limited hardware experience. They did not appear to
have a detailed knowledge o,f Rancho Seco specific problems and j
systems. '
, l
The maintenance engineers.had started 24 h'our a day coverage just a
few weeks before the NRC team inspection, to provide better support
~
, i
_ _ _ _ _ _ _ _ -
'
K .
., ,.
.
- 35 -
J' .
l'
l
' j i
1
to other groups'. The maintenance engineers considered their
'
. workload to be fairly high during outages.
i ,
The NRC team discussed these concerns with responsible licensee
[/ '
,
y
"
i
personnel, who indicated that a variety of engineering personnel
constituted a large portion of the overall budget and include
,.
t, procurement engineers, quality engineers, nuclear design engineers', ,
*
'
>< '
maintenance. engineers, and system engineers. The team noted that
:' each group of engineers had assigned responsibilities, but the
interrelationships and communication channels between these groups,
~
f '
,
'
>
had not been established. The NRC team considered that this could'
5 . lead'to confusion by personnel as exemplified by the overspeed *
-
,. event. Maintenance program elements did not tie system engineers
,
.o '
to WRs or procurement, but the licensee felt that this was in fact
'
g " accomplished on an informal basis. The licensee was aware that the
system' engineers were not involved with work and the maintenance
_. engineers were busy preparing procedures, etc. The licensee
acknowledged that some items may'be overlooked, and identified that i
'
they were reevaluating the need for system engineer involvement l
with maintenance activities. In addition, the licensee indicated
that~they were evaluating the need to increase communication
between the engineering groups and more clearly define their roles.
b. Inadequate Communications for AFW Governor Procurement
Communications between procurement and engineering staff lacked the
formality and documentation to assure that vendor originated
critical changes to equipment were identified and evaluated for
potential impact on equipment operation.
The governor on the AFW turbine had been replaced several times
'
during the past year because of water intrusion problems. During-
the most recent replacement, the vendor made undocumented changes
to the device, whi.ch made the direction of rotation a critical
parameter for preper operation. Undocumented telephone inquiries
by the vendor could have alerted the SMUD staff to the change and
avoided p1 ant. damage.
!
Previously, the governor rotational direction was not critical,
because of the arrangement of it's internals. Vendor (Woodward)
representatives stated that at the time the governor 9as being !
modified, they had contacted Rancho Seco purchasing personnel to
ask what direction their AFW pump turbine rotated. (Responsible
SMUD purchasing personnel stated that they in turn had referred the
Woodward question to the plant engineering staff. However, the ,
plant engineering staff reported that they did not recall receiving -
the Woodward question.). Woodward personnel then contacted the
-
turbire vendor, Dresser, as to the' direction of rotation of the
turbine. Dresser's records indicated that the Rancho Seco K-308 '1
turbine rotated in the clockwise direction. Accordingly, Woodward
modified the governor for clockwise rotation.
'
. The SMUD investigation of the January 31, 1989 AFW turbine
overspeed event revealed that the pump-turbine configuration was
'
.
#
';. e .]
*
% a
b = < 'e
,
& ._ 4
. - _ _ _ _ . _ _ _ _ _
_ _ _ ._
. 1'
ac
u.
7
.
,
- 36 - l
1
'
I
such that the governor' rotated counterclockwise, and not clockwise
as thought by the turbine vendor (Dresser). By the close of this '
' inspection, the licensee had removed the governor and was preparing .I
shipping and source inspection plans to witness the disassembly and
inspection of-the governor at Woodward faciliths. Part of the
investigation included determining whether or'not.the additional
modification made by Woodward contributed to the overspeed event of
the AFW pump-turbine.
Purchase order (PO) No. RQ-88-26-26893 had requested the vendor
.
(Woodward) to modify the governor:
-
weather proofing,
-
. change a buffer spring', and add a nitrogen purge connection,
-
set the governor to maintain turbine speed 3600 +/-10 RPM,
-
maintain a mechanical speed setting to 1500 +10/-0 RPM,
-
relocate the oil sito glass, and hydrotest the oil cooler.
The P0 included the Part Substitution clause required by Rancho
"
Seco procedure RSAP-0401: ... the supplier shall not substitute
other items for the items requested without specific SMUD approval
prior to shipment ... if the supplier identified a change,
nonconformance, or seeks waivers from other requirements of this
purchase order,,the supplier shall describe such conditions on an
attached supplier disposition request (SDR). This information
shall be transmitted, in writing, to Rancho Seco Plant
Procurement." The procurement package contained an SDR on which
~
Woodward notified SMUD.that addit.ional changes to the governor
would be- required as a' result of the requested SMUD modifications.
SMUD procurement, materials control and the nuclear engineering
staff performed the required evaluation of the' Woodward changes in
compliance with RSAP-0401. However, Woodward apparently did not
submit any SDR.for 'the modification which resulted in a specific
direction of governor rotation.
SMUDreceivinginspectionrec$rdsforthegovernorindicatedthat
all of.the required specification had been met by Woodward. TN
only governor-rotation-direction reference provided by Woodward was !
one line on a data sheet " Governor Specification For PQ-PL", which
stated: " Case Rotation-Plugged for CW."
6. Training and Qualification of Maintenance Personnel
The licensee had apparently provided very little oversight and emphasis
in the area of maintenance on-the-job (0JT) training and qualification,
especially in the Mechanical Maintenance Department.
During this inspection, there were no instances where the team observed
work being performed by unqualified workers. However, January 19, 1989
mechanical mainten.'nce On-Job-Training (0JT) index records (maintained
-and updated by the Training Coordinator), indicated that a number of
craftsmen had completed the required classroom training in 1987 for
several tasks but have not been formally taok qualified. One of the
maintenance supervisors stated that a number of the craftsmen had
n
___
.
w @
*
r - 37 -
..
i
L
completed their OJT for several of the tasks listed; however the paper
work had not been completed to document their qualifications due to
.n scheduled work and time constraints. Of nine valves listed on the OJTi
index, one crew of seven workers had only one person dstignated as '
: ' -/,
, qualified (on only one valve), although records showed..that instructive
a ,>
.~ .
training was completed'in 1987 on six valves. Another worker.on the
/
,
' 3
same' crew had completed instructional training on eight v'alvestin 1987-
'
,
i'
ij! $? but had not been OJT qualified on any valves. Simulator observations '5,, ,
#
*
were also noted for all six work crews for valves, pumps, snubbers, air
'
c . y
e
,
; compressors and other OJT tasks. ~ '
7 .
,
Jl' r. L *
BasedondeficienciesidentifiedintheOJTprogramduringan'INP0f, -
'( : evaluation conducted October 3-10, 1988, the licensee committed to1 ; s -
ij perform a comprehensive review of their training program and ; implement , 'e-
%e/ '
appropriate corrective actions to improve the DJT program.' Th'e !i~
' .-' *
,
? licensee's- review was scheduled to be completed in March 1989.
p
,' [j[ .' .
Open Item (50-312/89-01-10). . e
-
!
'
.
i
i; y ? 1
3, '
7. Backlog Controls
In general, the licensee appeared to have made significant progress in
. reducing the backlog of items on the Long Range Scope List-(LRSL).* s ,
'
'
However, several items ~on the LRSL, if completed in a more timely
manner, could have helped avoid the AFW overspeed event.
a. LRSL Item 4058 identified that vendor manuals were not controlled
in accordance with procedure AP 46 (which referred to NRC generic
letter 83-28). This was identified as a priority 2 item, scheduled'
for completion prior to startup from the next refueling outage.
b. LRSL Item 2292 indicated that training of personnel in vendor
manual control had not been implemented and'that a clear
responsibility for control of vendor manuals had not been accepted.
c. LRSL Item 182 identified that terry turbine overspeed trip testing
was not performed (the last time'that the OST device had been
tested was July 1987). This item was generated in response to a
number of QC tracking system items that identified the same
deficiency. In April 1988,. the system engineer identified that a
procedure was necessary for AFW pump governor replacement and
venting, but the procedure had not been prepared and implemented.
L
8. Failure Monitoring - Nuclear Plant Reliability Data System (NPRDS)
The method used by the licensee to determine NPRDS reportable failures l
l
appeared to contain potential oversights and omissions.
Assigned Rancho Seco NPRDS operators evaluated maintenance work request
data for input into NPRDS. The operators stated that they review lists
of corrective maintenance (CM) work requests which contained one-line
descriptions of work performed. Based on the lists, the NPRDS operators
determined which CM items involved reportable failures of plant
components and equipment. The selected CM work requests were then
j obtained and reviewed for the information necessary to report to NPRDS. k i
'
(,'e ,, !
,
;
, . - __.
.
w. .
~ *
. 4 9~ j
.
,
- 38 - -
p .. -
'
:7 '
-
.
f A
p. 1The team reviewed ' numerous Ranchof Seco maintenance. work requests for, the
-Maio Steam (MS) and Instrument' Air-Systems (IAS). The team's review
p.
"
'
revealed numerous- sporadic actuations of the MS safety. valves. _In
addition,:five.IAS checkLvalves, installed:to prevent a' bleed-down'of:
the back-up; air' supply ' accumulators on~ numerous safety .related and
.
;
"
4
-non-safety-related'airroperated valves,tfailed to seat during a special
surveillance-test'of the IAS. The team asked;the staff.of the
' Maintenance Engineering' organization,'who tre responsible for the-
, ' management of the Rancho Seco,NPRDS, to scan NPRDS data for all reported.
failures of MS safety valves'and,IAS components at Rancho Seco. .After
?
several' attempts, NPRDS operators found only'a fed reports of problems.
-with the MS'sarety valves and no reports of-any IA5 ' component'fLilures.
,
,
The . team' then reviewed maintenance work requests / written to investigate
why (1) a MS safety valve lifted _ bel _ow it's expected set point and (2)
why several-IAS' check valves failed to seat properly. This review was-
conducted to provide insights as-to why these failures were not reported
to NPRDS. Details found within each of the work requests revealed that
(1) the MS safety valve's lift point pressure setting had drifted below
'it's required setpoint'and (2) the IAS check valves were found'to'have
. white plastic particles on their seats that~ prevented proper closing of
;
the valves. After reviewing these work requests, the licensee's NPRDS
operators determined that these failures shauld have been reported to
NPRDS. They determined that the as-found data-on the work request was-- l
not reported to them on the CM work request list, which would have !
flagged the reportable failure.
_
The team determined that this method of determining NPRDS reportable
'
-failures'is a weakness in the Rancho Seco maintenance program. Valuable
equipment: failure data could be overlooked and trending. data, which;
might be used to enhance the licensee's preventive maintenance program,
could'be missed. In addition, NPRDS data used by other licensees with
similar equipment would not be available to allow them to evaluate.it,
and determine if they too could have similar" failures unless appropriate
l . actions'are'taken. k
1
!
l
!
,
,
i
.,
i
( '.
_ __ _ _.______________i
_ _ _ _ _ _ - _
.
.
. - u.
'
- 39 -
4
APPENDIX B
REPORT #1 0F DESIGN CONSULTANT
The following report was prepared by a design consultant who assisted the
Maintenance Team in'the evaluation of engineering aspects, and the historic
perspective, of the AFW overspeed event which occurred on January 31, 1989.
The consultant addressed specific questions posed by the team leader, as
follows:
REPORT BY: Gary Overbeck, Design Consultant; ERC International
1. The USAR Section 10.2.2.2 states that "An operator is stationed at
manually-operated valve FWS-055 in the recirculation line during pump
surveillance testing until the AFWS upgraded design is installed."
Should an operator have been stationed at FWS-055, such that he could
have relieved the overpressure condition? 1
The original design did not have an automatic isolation feature on AFW
' system initiation; therefore, if the valve (full flow test valve
. .FWS-055) was open, significant AFW flow could be diverted from the S/G
to the condenser. As a consequence, the Licensee committed to station '
an operator,at FWS-055 to close the valve if AFW was initiated during
,
,
the surveillance test. 1
The AFWS upgraded modification has been installed and tested; therefore,
the need for an operator to be stationed at FWS-055 was not required. '
)
'
In addition, this event occurred during post-maintenance testing, not
surveillance testing.
i
2. On the day of the event, the AFW trains were cross-connected at the
discharge of the pumps. What is the design rationale which led to a
potential loss of both AFWS trains? l
Rancho Seco has two Once-Through-Steam-Generators (OTSG), one of which
must be available to cool the core. Likewise, there are only two
auxiliary feedwater pumps. One of these pumps is powered by a motor,
while'the other has two drivers - a motor and a turbine. The turbine is
the lined-up driver and is the device initiated on EFIC initiation.
Assuming a main steam line break inside containment in one generator and
a single failure, such as a diesel does not start or an AFW pump does
not start, then the remaining AFW pump must provide the good 0TSG.
Given the two AFW pump arrangement and twa OTSGs, the trains must be
cross connected to assure water can be feo to the good 0TSG. This
design rationale was reviewed thoroughly during the Augmented System
Review and Test Program. Valves HV 318T.6 and HV 31827 can be operated
from the control room to isolate the affected header, if required.
. l l.
_ - _ - _
_ - _ -
;_ _ __
,
,. _
,
l
km
f ,
s-
;, -
;, .
,
,
-;40 -'
'
[y;
~
'
(4 7~.
-
A ):
'O
.c m ; 3
' ,
4
}p -
,, ,
* r 3c
~
#
Since;bAthtrain'sarecrossconn~ected,'the;overpressureaffectedboth~
, : trains. -Why is there no overpressure, protection of the'AFW system
'
' downstream of the AFW pumps?.
'
u : There is no regulatory requirement' or ' code requirement for a full flow
" '
. relief valve.in a system. application like the AFW system. .The-governor
m"
. -
and-the mechanical overspeed device both'had to failLto cause the event
observed. Both; devices are ~ considered safety related. .so from a single-
failure point of view, either device should be sufficient to' prevent-
-
overpres s ur.e. On 1/31/89,:the governor was in an unknown condition
-(might; work by might not) while the<overspeed trip should have worked.
:From a system design point of view,^the design was not deficient per--
~
requirements. ~Instead, the importance of having only one source of
, , overpressure protection was'not recognized by operating or maintenance.
personnel. performing the. test.
Given that the two.trainsimust be' lined-up cross-connected,'then any-
overpressure protection -in the' form of a relief valve would be a single -
"
point of vulnerability. Single failure of'one relief valve to stick
'
open or failiopen could divert flow from both OTSGs.
4. -. During;the ASRTP, the NRC team had concerns with respect to turbine
overspeed. Briefly describe :those concerns, how they were resolved, and
s
how they relate'to this event?
'
'There were four ASRTP: concerns related to turbine ovorspeed. These are
111sted below:
a. 'The turbine overspeed_setpoint including instrument error exceeded
'Lthe design' limit for revolutions / minute for the AFW motor drive.
1 b. The mechanical turbine overspeed device had'not been tested and was
'not scheduled to b? tested before restart.
c. yAnalysis did not exist to prove that AFW system components were not
3 ' overpressure at t's turbine overspeed setpoint with instrument
error,
y d. -The operators'.were permitted to start, stop and restart the AFW
turbine'without guidarice-on how long.between stops before the l
governor control oil had bled down sufficiently to permit a normal'
.
!
start.
HTo resolve concern a, the turbine overspeed setpoint was reduced such
that the design limit for the AFW motor revolutions per minute are not
exceeded,jConcern'b was~ resolved by testing the overspeed device before
.- . restart. ,For~ concern c; analysis'byf the Licensee subsequent to the
restart' led to r,eplacement,of some carbon steel piping. To resolve
. concern d, the restart? capability'of the turbine pump was tested. The
maintenance inspection team has' determined that a solenoid-operated dump
_ valve (actuates ;on steam supply valve close limit. Therefore, if this
b, Edes%n: attribute works properly, then there should be no problem with l" respect to bleed down of governor' control oil.
Based upon the foregoing ^and my understanding of the AFW overpressure.
event astof; February 10, 1989, the ASRTP concerns do not relate to the
current event.
s
b f
'
'\_,
s)
- p '
_.
7-
--
,
* =0 . ;
. .. :-,
3 - 41 -
{' n
, .
q,2
,
-
. , ,
"
,. c . *
'
^l '
5. Provide a short summary of 'the other activities in which you provided
( team assistance.
V. ? 4
In addition- to' the ' three in _ction observations and an evaluation of
4
the relationship of past ASRTP concerns with the AFW overpressure event,
J '
1 a system review was conducted to identify the list of affected
*
components. A review of the following documents was performed:
-
< ,,
,- 1. PDQ #89-0127, .Rev. O, Equipment ID: K-308/P-318, 1/31/89 i
/ 2. PDQ #89-0143,'Rev. O, Inspection of Pressure Retaining Components, l
I
k.,l 2/3/89.
3. PDQ #89-0144, Rev. O, Equipment ID: P-318, 2/3/89-
"i 4. PDQ #89-0145, Rev. O, AFW Turbine, 2/3/89
5. PDQ #89-0146, Rev. O, AFW Pump Motor Driver, 2/3/89
6. PDQ #89-0147, Rev. O, Instrumentation, 2/2/89
7. PDQ #89-0148, Rev. O AFW Turbine Governor, 2/3/89
8. PDQ #89-0149, Rev. O, Pacific Valves HV-20851 and HV-20582, 2/3/89
.9. PDQ #89-0150, Rev. 0, Anchor Darling Valves HV-31826 and HV-31827,
2/3/89
10. PDQ #89-0152, Rev. 0,' Mechanical Overspeed Trip Device, 2/4/89
11. Rancho Seco, Auxiliary Feedwater System Procedure A.51, Rev. 37,
System Lineup Prior to Event
12. Rancho Soco Auxiliary Feedwater Investigation Action Plan, Report
No. 1, Rev. O, 2/3/89
13. Rancho Seco Nuclear Engineering ~ Department Action Plan, Plan Number
89-M-101, Rev. O, 2/4/89
14. Rancho Seco Calculation Z-FWS-M2492, Auxiliary Feedwater System
Overpressurization, Rev. O, 2/3/89
15. Rancho Seco P&ID M-533, Sheet 3, High Pressure Feedwater Heater
System, Rev. 18, 1/23/89
' 16. Rancho Seco P&ID M-530, Sheet 3, H.P. & Auxiliary Turbines System,
Rev. 31, 1/22/89
17. Rancho Seco P&ID M-532, Sheet 1, Steam Generator System, Rev. 23,
1/18/89
18. Rancho Seco P&ID M-545, Nuclear Service Cooling Water System, Rev.
22, 11/1/88
Initially, my review concentrated on determining the extent of the
overpressurization and the peak pressure reached. Reference 14 above is
the design calculation that established the peak pressure of 3850 psi.
Although, one could argue with the methodology used to model the
min-flow bypass flow resistance, the calculation is conservative and the
actual pressure during the event probably did not exceed 3850 psi. My j
inspection observation concerning the quarantine documents my review to
determine the extent of the overpressurization. Using the P&ID's and
. isometrics attached to PDQ #89-0143, I identified the components
subjected to the overpressure condition (piping, valves, orifices, and
instrumentation). Then compared my list of components to those being
developed by the Licensee. Agreement between the list I had generated
and that of the Licensee was very good. During this period, I had an
<
opportunity to witness very intense and competent activities by the
a' ' Licensee's engineering organization in evaluating the extent and
"i magnitude of the problem caused by the overpressurization. These
intense and" positive engineering activities are representative of a good i
d engineerii.g organization reacting as expected.
,o ,3 )
x
fl1* . .
. l?
1
__iL.J._i.._
-, -' ,
.
-
.
-- - --- - _
- -- , - - -
c .r . ; ,
<
y- , .
<
, ,
,
r ; ..y w, x ,
(
e + ~ -r
.- , -F 42 -
*
,
g s
o
-
,
,s . 4
h.
&- , -
s-
r,a ;
.
.
~
B: 7- - In: summaryF I: b'elieve the engineering support in ' responding' to this .
-
<.
'.,;,
' event to-be' good even;tho' ugh the' indirect cause of;the governor.- .
> replacement may be(linked to engineering (i.'e. governor replaced,Jin.'
part,tbecause'of' water in'the control oil partially attributed to a poor:-
-design'and-test'of-the modification supplying' nitrogen to the-AFW-
'
[(, . turbine: governor).
. . . .;; ,
u-
s
e
, ,( .l.
p. ,
..
I
!
!
> Si- j,
, < <
[ .,
- ' (
. s
' ' '
{ ig
:p y ~ , ,
,
' ~
n. A! , ,<
s s
e,
-
^ ,
,
,__
<t _
.. g
\
. r -
"
.i,
#
i. ,
,
,
.m ,
, I,
,. > .u, q
.. . )-
.
e. f. ,
j -
r ,
' '
p.
f
9
l' e i $
'_
<
.
'
. , ... a,< i
>
r
i i
.
-
..
1
1
l
,
'
,
o
:j
,
'
!
;q
l
!
v- 4
11 1
sh_
c
$
n>
1
;
. _ - - _ _ - _ - _ __-_-____O
_- - _
'
l
9 ',s'
s i
- 43 -
a
i
APPENDIX C
!. REPORT #2 0F DESIGN CONSULTANT
The following report was prepared by a design consultant who assisted the
Maintenance Team in the evaluation of engineering aspects of plant
modifications, which appeared to be related to the AFW overspeed event which
occurred on January 31, 1989. The consultant also reviewed the engineering
basis for the equipment quarantine established in conjunction with the
licensee event investigation efforts. The consultant report of findings
follows below:
I
REPORT BY: Gary Overbeck, Design Consultant; ERC International
1. Equipment Quarantine Boundary Discrepancies
Description of Discrepant / Suspect Conditions:
Weakness in the quarantine boundary established to ensure preservation
of the as found conditions of piping and components exposed to AFW
system overpressure.
Nuclear Engineering Department Action Plan, Plan Number 89-M-101, Rev.
" 0, dated February 4, 1989, identifies the valve lineup in that portion
, , of the AFW system subjected to the overpressure condition. Figure 2,
identified as Overpressurization Boundary, indicates that the supply
isolation valve to the nuclear cooling water surge tanks, FWS-079, was
,, shut and', therefore, the piping downs. eam was not overpressurized
, et .duringlthe event. While performing an independent review of components
< ri affected by the overpressurization, the team observed that FWS-079 was a *
'
, ,
,normally'open, valve per P&ID M-532, Steam Generator System, Sheet 1 - -
'
' dated 1-18-89. This inconsistency was brought to the attention of the ,
.
'
Licensee on or about 1500 on February 6, 1989, and the team was informed ,
,s.
that theivalve was shut prior to the event per valve lineup procedures.
The system status file for the Auxiliary Feedwater System A.51,
completed on>1-28-89, indicated that FWS-079 was lined up open;
therefore, when the event occurred, the piping and components downstream' ,
'
were also pressurized. i
Regulatory Requirement and/or Commitment:
'
;
4
,
i
None "f
F Extent o'f Observed Problem:
This observation is not indicative of multiple problems; however, this ,
weakness is apparent in other documents: !
a. PDQ #89-0147, Rev. O, 2/2/89.
This PDQ identifies instruments to be evaluated and refers to an
Attachment A which is the same as Figure 2 described above.
l- Although the overpressurization boundary was not extended far 1
_ _ _ _ - _ _ _ - _ _ -
7m-
\ :,
'
~i,(?
l. . n.
,
y
~
'
* '
. 44 .
c. 4
Y
q
6
4 \ c
, g
'
enough,noadditionaj~instrumentationislo'atedinthenew- c
,
boundary. y _.7
.
, ,
i;*; t;
i
s b. ActioniPlan, Report:1*,' Revision.0;'2/3/89.
, ,
, ,
Appendix,l',"Quaran. tine,1 did not identify. valves affected by the'
overpressurization on P&ID M-545', Nuclear Service Ccoling Water
~
"
' System,1Rev. 22, 11/1/88.' For example, valves FWS-112 and FWS-114
were not: listed. The quarantine was established from the, auxiliary
~ feedwater gvstem to,FO-48477,;just downstream of FWS-079. On
4 -
, February)/,?1989, the team confirmed that these valves were added 1
,
'
to Nuclear Engin'eering's list of affected valves.
~
p: ' Safety'S1gnificance of Observed Problem:
L - -
,
This' observation has minor safety significance. The failure to
adequately define the overpressurization boundary could result in the
failure to inspect / test all affected safety related components. If left
undetected, some AFW flow could be diverted away from the steam
generators through a failed 1 inch line or to the NSW surge tanks
through a failed valve. However, the pressure reached was not
significant enough to exceed the design capability of the 1 inch line
y and the flow through a failed valve is expected to be small. Therefore
the amount of diverted flow would be small and the AFW pumps have
'
sufficient head margin to overcome the additional flow demand. In
addition, it is highly-likely.that the Licensee's . subsequent reviews
would have also detected this oversight. Therefore, the likelihood of
r not recognizing that FWS-079 was open during the' overpressure is low.
Licensee Corrective Actions Planned:
On the evening of February 6, 1989, the Licensee extended the quarantine
boundary to include FWS-079 and piping downstream to normally' closed
isolation valves FWS-112 and 114.
Specific References:
l.
1. Nuclear Engineering Action Plan,' Plan Number 89-M-101, Rev. O, !
2/4/8?.
l 2. System Status File for the Auxiliary Feedwater System A.51, j, 1/28/89. j '
3. Action Plan, Report 1, Rev. O, 2/3/89.
:
J
i
i
!
l
l >
-
l .
k
;-
+,
'
L A _._ _ _ . _ . _
x c. 3 i
( *
,; ,
~- 45 -
j
l: ) f ~ 1
2. Ina'dequate Post-Modification' Testing of a Check Valve
;
l
Description of Discrepant / Suspect Conditions:
p '
, , A temporary modification installed a check valve which was not
. functionally tested prior to returning the system to service.
A check valve was installed in line 35801-3/8"-HE1 to prevent back flow
of water.from the condensate storage tank into the nitrogen system. The
check valve was" installed by WR#01415330-0 on 1/24/89 and authorized by
Temporary Modification 89-06.on 1/23/89. No post-installation testing
was specified.by the temporary modification. The work request required
a verification that no leakage existed at system pressure; however,
functional testing was not performed. The testing performed did not
confirm that the check valve was installed in the correct direction or-
.that the disk and seat interface was leak tight.
'
'
The team is concerned that if the check valve is not leak tight, then
. water can.be introduced into the AFW turbine governor'under certain
circumstances; e.g. during CST nitrogen purging (MCM-146 open) an
undetected loss of nitrogen supply to the header may occur (i.e. an .
operator shuts nitrogen header supply isolation valves, etc.). )
i
a- Regulatory Requirement and/or Commitment
10CFR50, Appendix B, Criterion XI, Test Control, requires preoperational l
tests. The purpose of these tests is to demonstrate that systems and i
components will perform satisfactorily in service.
Extent of Obse.ved Problem:
This observation is the'only one noted by the inspector during the
limited time onsite; therefore, it is not indicative of multiple
problems with post-modification testing. However, RSAP-1606, Temporary
Modification Control, Rev. O, dated 11/08/88 does not address
. post-modification testing. The lack of post-modification testing in
this procedure appears to be a weakness.
Safety Significance of Observed Problem:
Although the check valve is classified as QA-2, it serves a function to
prevent water introduction to the nitrogen system and in turn the
nitrogen purge to the AFW turbine governor. The check valve needs to
seat in the reverse-flow direction if the nitrogen purge to the CST is j
being used (MCM'146 open) and2 N pressure is lost. '
Licensee Corrective Actions Planned:
The team was informed that a design change package is in progress.
~F This valve should be functionally tested prior to restart.
Specific References:
1. RSAP-1606, Temporary Modification Control, Rev. O, dated 11/08/88.
2. Work Request #D1415330-0, 1/24/89.
3. Temporary Modification, Tag No. 89-06, 1/23/89.
<
_---_______________J
_ - _ . - _ _
_,
-
.
s s
..
- 46 - i
e
3. Inadequate Attention to Detail for Engineering Work
i
Description of Discrepant / Suspect Conditions: 'I
I
Insufficient attention to details resulted in the installation of an I
inadequate modification, failure to quickly detect incorrect valve position,
'
and water intrusion into the AFW turbine governor.
Background:
i
While performing Special Test Procedure 1080 on 4/6/88, the Licensee
determined the AFW turbine governor control oil contained water and debris.
PDQ-88-0087 (Ref. 1) was issued on the same day to replace the governor while
Nuclear Engineering determined the need for preventive measures. Engineering
Change Notice R2990, Rev. 0 (Ref. 2) was developed to add a slight, constant
purge of nitrogen to the governor assembly to reduce the likelihood of water
intrusion. ECN R2990 was reviewed, approved, and issued for work on 4/22/88.
The ECN was revised on 5/5/88 (Ref. 3) to add an isolation valve so that the
nitrogen supply to the condensate storage tank (CST) nitrogen blanket could-
be isolated without isolating the nitrogen purge to the AFW turbine governor.
-The modification was installed and tested on.5/7/88 (Ref. 4). Figure 1 shows
the arrangement after modification.
On'10/28/88, one of the CST rPicf valves was determined to be leaking. To I
prevent excess nitrogen usage Operations isolated'the nitrogen purge to the
CST by closing MCM-590, which also isolated the nitrogen supply to the AFW
turbine governor and to CST. nitrogen blanket (Refer to Figure 1). Because of
elevation differences between the CST and the AFW turbine governor, water
back flowed to the governor l housing. PDQ #88-1808.(Ref. 5) was prepared on
the same day to flush and replace the governor oil. The"long term fix was to
install a check valve to prevent back flow from the CST on loss of nitrogen
pressure (Ref. 6 andL7).
On 11/1/88, while at' 92% reactor power, the Licensee found that the plant
operating procedure for the condensate system (Ref,, 8) differed from the P&ID
~
for manual position of MCM-146, CST drain' isolation valve (Ref. 9). This
valve's safety functiori is,to' be closed to preserve the seismic Class 1
boundary from non-seismic;'however, the plant was operated with the valve l
open to permit nitrogen bubbling /sparging to the CST. PDQ #88-1820 was
prepared on 11/1/88 (Ref. 10) and the NRC notified by red phone.
I-
Discussion:
Weaknesses that contrQuted to this condition:
a. ECN R2990, Revision 1, did not recognize that the proposed design was
weak because the failure modes and effects analysis was not implemented
thoroughly in accordance with the requirements of NEAP-4113 (Ref. 11).
Although a Failure Mode and Effects Analysis (FMEA) section was
prepared, it only addressed the consequences of overpressurization of
the AFW turbine governor. The consequences of a loss of nitrogen supply
or mistaken valve operation was not addressed. This weak FMEA was not
_ _ _ _ _ _ - - _ _ -
wy,r. - -
w -
.
3rx4: +>
f m.. y r 4
i * .
. a ;, y .
,
',
;g' f
*
[[{ L V
l} ' f r
> <
- 47.- i
t '
.y* ( %
, . , <
-
t
,
,
p
/'J
;* 1
l }., i lg + .4 ,
~
!
c .
d'etected_by ,the design verification performed on either Revision'0 or s i.s '!
1
,
# , Revision 1.
,
b. The team was informed that-the design engineer relied upon the valve
indication depicted on the P&ID, which showed MCM-146 to be closed.
However, the system operating procedure in effect at the time (Ref.12)
is the. correct source'for valve ~ position and should have been used.
Had the: system operating procedure been used, there is reasonable
likelihood that the incorrect operating position of valve MCM 146 would
~
have been recognized six months earlier. The team noted that the ;
-Operating Procedure was not identified in the ECN as a referenced
document nor as an affected document.
c. The Operating staff did not recognize the system interaction caused by
the addition of the nitrogen purge to the AFW turbine governor. They
failed'to recognize that isolation of the nitrogen supply provided a
flow path for water from the CST to the AFW turbine governor.
Regulatory Requirement and/or Commitment:
ANSI Standard ANS-3.2/N18.7-1976, paragraph 5.2.7, " Maintenance and
Modifications," specifies that " Maintenance or modifications which may affect
. functioning or safety-related structures, systems, or components shall be
performed in'a manner to ensure quality at least equivalent to that specified
in original design bases'and requirements.
ANSI Standard N45.2.11' requires that the preparer and verifier of a design
change confirm that the proposed design change does not alter or adversely
(
affect the function of safety-related systems, components, or structures.
l,
Extent of Observed Problem:
Only one, relatively simple, modification was reviewed in detail; however,
the problems observed suggest a general weakness may exist in the assessment
of t_he details in preparing a modification, verifying its adequacy, and
t
i
,commun i cat i on of the des i gn intent to the operating staff. This general
<
.weakness may be limited to the smaller or less significant changes; i.e. for
those minor changes that_are beneficial but not complicated, a perception may
develop that close scrutiny or detailed review is not required. Overall
conclusions ,about the strengths or weaknesses of the modification process
"
cannot be made based upon this limited sample size. ,
'
Safetir Significance of Observed Problem: +
s
<
f
',, .; This entire obse'rvation is Licensee identified. However, the means existed
s e for_the Licensee to correct an inadequate valve position much sooner. The '
"
,
=
'
valve (MCM 146) is a' seismic boundary, so being open jeopardized the ability- ,
,
5
.of_the AFW to-perform its safety function following an earthquake. *-
.
.
.
_
-
, ' ;, , (
(Just prior to the inspector's departure from the site, licensing informed ,
him that.an LER was not' written. The notification and deportability
~
4 3
worksheet'(Ref. 13) attached to PDQ #88-1820 states, in part, that reasonable
operator actionican be expected to maintain safety systems in service to
s
:
a
_ _ _ _ _ _ _
P- ,..
3 .o,
.O ' - 48 -
:..
fulfill their function. It infers that there are three alarms on the
condensate storage tank (i.e. , Lo, Lo-Lo, and Lo-Lo-Lo) and the operator
would have sufficient time to recognize the problem and correct it. First,
no analysis was provided with the worksheet to indicate how fast the
condensate storage tank level would decrease from a non-seismic pipe break
downstream of open valve MCM 146. In addition, the inspector remembers the
low alarm sensor to be at a high point in the tank to alert the operator that
condensate makeup has initiated. . The low-low alarm sensor is also located
high in the tank to alert the operator that the water level does not meet the
Technical Specification limit for CST water volume for continued operation.
Finally, the low-low-low alarm point is low in the tank to alert the operator
that 40 minutes remain (based upon AFW pump capacity) before suction pressure
will be lost to the pump.' The time is the~ time needed to realign the suction j
from the CST to the Folsum South Canal. Following a seismic event,
initiation of=the'AFW system ~in' response to'a turbine trip and loss of off
site power is anticipated. Therefore,.the operator would not find the alarms
at to and Lo-Lo to'be necessarily unexpected for that condition. If the loss
of level is due to AFW pump operation and loss of water through an unisolated
leak through a failed non-seismic line and an undetected open seismic I/II
boundary isolation valve.(MCM-146),'then 40 minutes may not be available
after the low-low-low, alarm is sounded to shift to a reliable source of AFW
supply. As a minimum it appears that the Licensee's analysis of this event
is weak and based upon the assumption that the operator will have more than
enough time to' recognize and correct the problem. The team or regional NRC
personnel should e'xamine'the reason why an LER was not prepared and a more
detailed analysis performed. (This is an Open Item, discussed elsewhere in
the team inspection report).
Licensee Corrective Actions Planned:
1. The operating procedure has been revised to have MCM-146 locked closed l
(Ref. 14) i
2. Temporary Modification 89-006 has installed a check valve to prevent
reverse flow.
3. Permanent Modification DCP-R89-0013 is in process to revise the valve
arrangement at the bottom of the CST. Preliminary review indicates a
correct arrangement is proposed. )
Specific References:
1. PDQ No. 88-0087, Terry Turbine Governor, 4/6/88
2. ECN R2990, Addition of a Nitrogen Purge Supply to the AFW Turbine
(K-308) Governor, Rev. O, 4/22/88
3. ECN R2990. Addition of a Nitrogen Purge Supply to the AFW Turbine
(K-308) Governor, Rev. 1, 5/6/88
l
4. Work Request #01382460-0, K-308, 5/6/88
)- l. 5. PDQ No. 88-1808, K-308 Terry Turbine, Rev. O, 10/28/88
_2___. _
_ _ _ _
- , >4
.
..
,,
49 - 1
f
i
'
6. Temporary Modification Tag No. 89-06, 1/23/89
7. Work Request # 01415330-0, Equipment ID 35801-3/8-HE1, 1/24/89
8. Rancho Seco Procedure A. 47, Condensate and Feedwater System, Rev. 26, 1
5/20/88
..
9. Rancho Seco P&ID M-536, Condenser System, Rev. 34, 4/14/88
10. PDQ No. 88-1820, MCM-146, Rev. O, 11/1/88
..
11. Nuclear Engineering Procedure Manual, NEP-4113, Design Basis Report,
Rev. O.
12. Rancho Seco Procedure A. 47, Condensate and Feedwater System, Rev. 25,
1/25/88
'
13 Rancho'Seco Operations and Licensing Department, Notification and
.
. Deportability Worksheet, 11/8/88
14. Rancho Seco Procedure A. 47, Condensate and Feedwater System, Rev. 27,. ,
1/1/89 l
,
t *
,
, t
<
! ,
.
'
,.
Y
t
!
!
,
i
r-
..
h%-
_ -
4. . iI ,
.
- - _ _- . _
~> e
'! .
{k . -- ,
.
, .,
.
1\;: -49a -
g
+
.
,.
'
4
l -'. .
.
,
,
:.GU E :. -
.
!
.N2 SYSTEM-
SUPPLY
Are!uPCV
e 35801'
x A
4
HCH-897< /n'ie,\
'
<
CST i
MCH-146
;(HCH-602
*
. r1
- 35801 l[HCH-589
!. uD
r.
x
HCM-603 I
-
'PI
35801
x
FVS-827'
l
I.
AIV / " !
6
ll FVS-828
TURBINE
GOVERNOR * $
u
1
1- K-308
FVS-829 PCV-30801
o.
i
,
e
b
' ' ' " ' - t- _____________'_,.__.__..____... _ _ _ _ . _
-_
-.
4
i & ,
--' - '
- - 50~- ,
4
1- APPENDIX D
I
SPECIFIC IDENTIFICATION OF DOCUMENTS REFERENCED IN INSPECTION REPORT
'
1. USAR Updated Safety Analysis Report (Rancho Seco)
2. IEEE 338-1977, Criteria for the Periodic Testing of Nuclear Power
Generating Station Safety Systems
3. IEEE 352-1975, Guide for General Principles of Reliability Analysis of
Nuclear Power Generating Station Protection Systems
i
4. IEEE 650-1979, Standard for Qualification of Class 1E Static Battery
Chargers and Inverters for Nuclear Power Generating Stations
PROCEDURES AND INSTRUCTIONS:
1
1. Auxiliary Feedwater Investigation Action Plan, Report Number 1, I
Revision 0, dated February 3, 1989
2. Auxiliary Feedwater. Incident Failure Investigation Work Plan,
Revision 0, dated ' February 7,1989
3. RSAP 0309,. Vendor Equipment Technical Information Program (VETIP)
Revision 0.
4. RSAP-1308, Potential Deviation from Quality (PDQ),
Rev. 2, August 23, 1988
t-
5. RSAP-0303, Plant Modifications
6. RSAP-0401, Preparation and Processing of Purchase Requisitions,
Rev. 1, March 8, 1988
7. MAP-0009, Preventive Maintenance Program
8 .' M-159, Vibration Monitoring Manual (Maintenance Procedure)
9. QAP-17, Nonconforming Material Control, Rev. 7, dated November 25, 1987
10. WP36808/D-0199B, Inservice Testing Program Plan For Pumps and Valves
Rev. 4
11. Repetitive Task Number 12162 (PM), " Sample Collection For Lube Oil
Analysis," for AFW Pump and Motor P-318:-
12. Repetitive Task Number 12163 (PM),'" Drain and Flush Lubricating System,
Lubricate Governor Linkage,"'for AFW Pump turbine
13. Repetitive Task Number 07099 (PM), " Collect Vibration and Temperature
Data on Pump and Motor," for AFW pump and motor'
14. Repetitive Task Number 05864'(PM), " Hand Rotate Pump / Motor During Lay Up
' Exceeding Ninety Days," for AFW pump and turbine
11 5 . Repetitive Task Number 05370 (PM), " Oil Analysis; Turbine Sample
Collection for Lube Oil Analysis," for AFW pump, motor, and turbine
16. Repetitive Task Number 06191'(PM), " Change Governor / Governor Gear Box
011; Observe Power Piston and Valve Linkage for Smooth Operation,"
17. Repetitive Task Nu' mber 14760 (PM), " Monthly AFW Pump Operational
_
Verification" -
'
18. STP-774, Special Test Procedure
l
L.- .
_ _ - . . - -- -
r ym : ,'%. ,, , .
.
,
y w 3- .4
[ 1{ w wf.fb{y - 51 -
i ,
_p ',
5 'U;
- / APPENDIX'E-
r =
,-,M ['
-gn. ,
.
c , .
1
j , . . ..
t i
,
.
' ~s
,. GENERAL INSPECTION FINDINGS r <
& #y Np; ,p,, s - ; 3; x e 3 ,s j, , < g N tfvi er '
-
P
Mm 't (! t : I . . ' ', ' a MAINTENANCE INSPECTION TREE ' .a , ,. $ ) , -- I g[ '
g4 3
'* f-),, 4, , 11.0 ND'I RECT SEASlIN S
'
,, , ,
.
, . . , , ,
~
f Q ,$;r, w
p
= .;
2,11The)teamJconsensuswasthatthiselementwasfunctioningadequately.
<
'
.i s s s .
r i,
N N*/ s l Q Direct' measures of. plant [,
Levaluat'ed'from data performance associatedindicators
relating to performance with' maintenance were
compiled by the f
[h" 4
.
NRC" performance indicator program, data reported by.the licensee in
,
-
,
7 monthly 4 operating data reports to the NRC, NRC SALP reports and licensee' t_
, d internalf reports'on' performance goals and indicators. 1
#
From the above sources, the!following principal direct measures of
", performance were evaluated to assess'overall plant performance related
to m'aintenance.
:1.1 Historical Data
'
N rating was not' assigned to this block.
.
There isLno. distinction-for program adequacy versus implementation
-adequacy,made in rating this area.
The reviewfof historic. data is intended to determine what impact-
maintenance has'had on plant availability., operability, reliability, and
radiation exposures. The team did not perform a review of Rancho Seco
- historical .dat'a'since the plant restarted from an extensive shutdown
les.; than one year ago. The team consensus was'that insufficient
historic data was available-for the period since startup to permit
: assessment of all the elements of-this block.
-
However, during the inspection of radiological controls,
the following STRENGTH-was noted in this area:
a. The licensee maintained occupational exposure well below the
established SMUD goal for maintenance activities during the last
year. In 1988,-the licensee expended about 65 person-rem for all
utility and contract personnel involved in maintenance activities,. j
with an established goal of 108 person-rem. The industry median i
was about 163 person-rem for maintenance activities.
E 1 '22. P1 ant Walkdown Inspections l
The rating assigned by the team to this block was Satisfactory.
-The team consensus was that this element was functioning adequately.
,
-
' There is no distinction for program adequacy versus implementation l
y.'
,
: . adequacy made in rating this area. l
H The STRENGTHS noted in this area included:
*
..
Of
* '
<
,
4 , ,f, ' ,
,
. , ,
9
-r. -
,
v _' g j '( j h '\ r ,
A
5_ f. L
'% ,,
r *
%[s
'
; / * * /.. 8 1 . >
r, , a. _-_._____:_t _
- _ _ _ _ - _ - _
. . . ,
-
:n ; a
-
52 -
.
,a. Material storage an'd housekeeping were effectively controlled by
administrative procedures and verbal instructions by management.
b. . Equipment physical condition (such as electrical-switchgear) !
appeared to be. excellent. Equipment requiring maintenance'or repair
was appropriately tagged and monitored on a status board in the
Control Room.-Interior plant electrical controls appeared to be
free of moisture / foreign material and circuit breake'rs were in
'
'their proper position'and tagged appropriately. No obvious damage.
c. Plant walkdowns by 1k. m ee management personnel were being
conducted at regular intervals and. documented.
d. A security printout of licensee management personnel access to
plant vital areas confirmed that spot licensee inspections were
conducted frequently by.all level.s of management.
'The WEAKNESSES identified in this area included:
a. Although it was the team consensus that the physical condition of
equipment was appropriately tagged and monitored on a status board
in the Contro1 Room, the team felt that many of the conditions
#
identified on the tags were minor discrepant conditions that could
have.been more promptly corrected by a more aggressive maintenance
program. '
b. Teflon type tape was found.to be'used extensively on safety related
systems throughout the plant. This. appeared to be contrary to
_
_
st'ated policy by the licensee and,could~ result in-particles from
this materialfenter.ing, solenoid' valves / pipe fittings, and creating
the potential for. malfunction of certain systems, i.e. air and
sampling systems'(See ISSUES).
.
2.O MANAGEMENT-COMMITMENTAND' INVOLVEMENT -
The overall rating,0f this area was:' Satisfactory
Applicabli requirements of this element have been developed, documented
and effectively implemented. Areas requiring improvement were
approximately offset by better performance in other. areas. j
- STRENGTHS
- No" areas were rated as Good:
I l WEAKNESSES: No areas were rated as Poor. l
2.1 Application Of Industry Initiatives
ll The overall rating of this area was: Satisfactory [ Program elements appeared to be adequately addressed. I
Implementation was in place, but could be strengthened.
I [I The STRENGTHS noted in this area included: l .. l a. The assignment of responsibilities and tracking of commitments for
each NRC and INPO issue.
L L _ _ __.
s x .
'
4 , j +,.;> f s
,1 ,
f j p% % l pf d
y
%
{fi j. 4 . 53 --
, ; ,
b. 'bataoneachinitiativewas.readilyaccessibleandretrievable.
< c .' ' A self-assessment-program was established and working.
The WEAKNESSES identifled'in.this area included:
i= H .
a. Mr.ny weaknesses. identified by the self-assessment program were.not
P .yet corrected.
- J
,
b. The plant did not implement engineering recommendations regarding
AFW turbine _. testing. -(See ISSUES).
c. Some engineering evaluations of industry AFW turbine problems were:
-shallow and did not recognize potentially similar problems at-
Rancho'Seco.
. d .- Evaluation and implementation of industry /NRC concerns over plant
~
! , air systems and check: valves appeared to be slow despite noted I '.'
problems with instrument air system check valves. (See' ISSUES).
4
e. 'The-licensee' evaluations of EPRI, NUMARC, ASME, and IEEE
i m' .information appeared to'ba dependent on individual efforts and not-
directed by' licensee programs; i.e. licensee efforts appeared to
=only be confined to NRC-and INP0 initiatives.
2.2 Management Vigor"And' Example
r. "
. The overall rating of this' area was: Satisfactory.
Program elements appeared to be adequately addressed.
Implementation ~was-in place, but.could be strengthened.
The STRENGTHS noted in this area included:
a. 'A demonstrated strong interest in~ maintenance by managers.
b. A computerized Work Request program - readily accessible and
- s .available.
c.' , Feedback to the maintenance organization regarding the importance
'of maintenance,
d. Management plant walkdowns conducted with reported evaluations of
*
*
, . work performed and deficiencies found.
- e. 'CEO. initiated plant performance indicators trended and posted .,
' '
, throughout the plant.
'
,3 .The WEAKNESSES' identified in this area included: n
y
J:
L 'a , 4 NPRDS program implementation only recently made full-time. (See
,
' '
r
'
e
<
. ISSUES).
'1 t <
,
b; Use.of NPRDS appears to be limited to Maintenance Engineering and
'
, some systems engineers. (See ISSUES).
F
s .
3.Oi MANAGEMENT ORGANIZATION AND ADMINISTRATION
.
n . .
'
. g The overall . rating of this. area was: Satisfactory
Applicable requirements of this element have been developed, documented
b #,
.
, and effectively implemented. Areas requiring improvement are
'approximately offset by better performance in other areas.
.
STRENGTHS were noted in rating of the following areas as Good:
;
n_L ____ '
'l
..
_ _ _ _ - . _
. .
,
'k ng
J. . -
- 54 -
.
3.1 Identification Of Program CoverageL or F Maintenance l
' 3. 2 Establishment Of Policies, Goals,10bjectives
For Maintenance
3.3 Allocation Of . Resources * .
3.5 Conduct Performance Measurement-
3.6, Document, Control System For Maintenance
3.7 Maintena'nce Decision Process
4
WEAKNESSES: No areas were rated:as Poor.
3.1: Identification Of Program Coverage For Maintenance
The overall rating-of.this area was: Good .
;
Program elements appeared to be adequately addressed. j
Implementation was functioning well. !
The STRENGTHS noted in this area included:
a. Long range maintenance plan issued.
b. Maintenance plan implemented, up-dated, and contains organizations
with responsibilities identified.
c. Long range maintenance activities ideritified, scheduled and
tracked.
No significant weaknesses were noted in.this area. !
3.2- Establishment Of Policies, Goals, Objectives For Maintenance
The.overall rating of this area was: Satisfactory
Program elements appeared to be well documented.
Implementation was not sufficiently evaluated to determine a conclusion.
STRENGTHS and WEAKNESSES were not evaluated by the team, although the
. team consensus was that policies, goals and objectives appeared to be
well defined, in many cases posted in prominent locations throughout the
plant site.
3.3 Allocation Of Resources
The overall rating of this area was: Good
Program elements appeared to be well documented.
Implementation was functioning well.
!
The STRENGTHS noted in this area included: I
a. All Priority 1 items on the Long Range Scope List (LRSL) had been
completed. The licensee indicated that there were 388 priority 2
items on the list. Of those, 69 items had been completed since
December 1988. .
b. The number of backlog maintenance items had been reduced l
l substantia'11y since October 1988. I c Management had added a large number of contract staff to help
reduce the backlog of maintenance items,
d. There appeared to be few delays of maintenance items attributed to
understaffing of maintenance.
l
'
c ._ _____________;
q
m [y)ip' f _ ,
i
g , , - 55:-
6 , ,. - * '
, yy a
,
f li':q l
W' e. The licensee had established a 12.houe :hift rotation for
-
): *
~maintwaance personnel to ensure that. sufficient maintenance
y ? personnel were available to perform work 24 hours a day
( , (accomplished by six crews rotating on a 12 week schedule of 12
L . . .hourJshifts).
f "#; . f. , .A duty maintenance engineer and a duty system engineer had been
p,' .provided for backshift coverage.
y .e
,
;g. The maintenance manager was knowledgeable about areas needing
; improvement and was. focusing resources in those areas.
,
, -
k , , ' .. D * *,- i h. 0vertime controls were established to prevent excessive use of .
4i ' * ' overtime.
[" '
f i; A maintenance shift supervisor was added to support operations
1
<
g,4 during backshift hours
M . . The WEAKNESSES identified in this area included: 4
J' ;ilp z
'
_
p ,
a. There was'a heavy reliance.on contract support which may lead to .c'
,
fI , h} ? ',., j , .C. discontinuities in the future. .'-O *
??, * '
,
b. ' More prompt completion of some Priority 2 LRSL items' could prevent
f: ;
'
, impacts on. plant operations, e.g. AFW overspeed event. (See ISSUES)" .,
Fe[p; .f . ~
u
i .' 3.4 i Definition Of Maintenance Requirements
"
,
. ,
,
*
N
;- ,, ,
u
h cg ,g' ;. i' .
7 / c y ,,The*overall rating of'this area was: Satisfactory
L. .1 e - Program _ elements appeared to be adequately addressed, '
s
)* a ' Q " Implementation was in place, but could be strengthened. [
(. ,
, on
0' The. STRENGTHS noted in this area included:
g
<* ~ a. Maintenance. requirements for' preventive and corrective maintenance
were clearly. established.
h , , b. Preventive maintenance requirements included periodic, predictive b and planned maintenance.
,
c. Corrective maintenance requirements were based on equipment
conditions such as time' life limits (expected failure)', known
b, failures, and required. actions to restore, repair or replace. b d. In the areas'of Electrical and Mechanical maintenance, the h preventive maintenance program requirements' contained in Procedures P MAP-0009, M-159'and WP36808/D-0199B identified governing code [:.^ references, e.g. ASME Section XI, ANSI N18.7 and INPO Good Pract:ce
guidance.
e. Technical Specification requirements appeared adhered to in.the
maintenance process.
The WEAKNESSES identified in this area included:
[ a. In the electrical maintenance program requirements, no reference b was made to vendor instructions or the ifcensee's commitment to the
.
testing ~and maintenance criteria of applicable IEEE Standards, i.e.
IEEE 338-1977, IEEE 352-1975, and IEEE 650-1979. (See ISSUES)
.
.b. Some corrective, maintenance procedures did not include instructions
1 for testing of critical performance parameters of some electrical
'
and mechanical components in accordance with vendor instructions
,
and governing codes, i.a. running current of electric motors and
l
Mx s.. .
.,
*
(? 2,
'
,
3.. % y s
+
- sg ,
p.
/
solen'oid valves, and turbine overspeed mechanical trip device.
, c. Preventive or corrective maintenance test procedures did not
include appropriate equipment testing precautionary measures, i.e.
turbine AFW pump periodic and post-mainttaance testing. (See
ISSUES)
d. The maintenance history of equipment, contained in the plant
computerized information system (NUCLEIS), was incomplete, i.e.
history of previous AFW turbine governor and mechanical overspeed
trip failures. (See ISSUES)
e. The maintenance program did not appropriately incorporate industry
and plant experience, i.e. NPRDS trend data and PRA data not used
effectively, such as previous AFW pump-turbine failure experiences.
(See ISSUES)
3.5 Conduct Performance Measurement
The overall rating of this area was: Satisf actory
Program elements appeared to be well documented.
Implementation was in place, but could be strengthened.
The STRENGTHS noted in this area included:
a. A maintenance activity plan was established for daily review of
maintenance piocedures.
b. Maintenance work sampling was performed regularly.
c. Walkdown inspections of maintenance work performed was conducted,
d. A feedback system was established for maintenance improvement.
e. Maintenance performance indicators were established and
implemented.
f. Maintenance performance was included in QA audits.
The WEAKNESbES identified in this area include:
a. The feedback system' established for maintenance improvement had
experienced poor implementation (See ISSUES, i.e. nuclear
engineering and operations communications regarding CST nitrogen '
.
sparge line design and interfacing nitrogen supply to the turbine
AFW pump governor).'
b. There was no apparent periodic surveillance plan for periodic
review of' maintenance procedures, procurement, training,
maintenance equipment and tools as they relate to performance
measurements (See ISSUE;).
c. Root cause analyses of maintenance failures were not routinely
-performed. Such. analyses were apparently performed only at the
discretion of the Maintenance Superintendent by the Incident l
Analysis Group lor Maintenance Engineering (See ISSUES i.e. numerous !
failuresz of turbine AFW governor).
8
h___. _ _ . . _ .
, - _ _ - _ __ -_ -___ _ -
,
'
?,
'
t
4
A yt;,
<" 'tf ,
a
57
,~,. ,
r '
l{: a t m ,; $ ,
,
.
Q t^
'. 3. 6L Document Control System For Maintenance
a f ;~ .c ,-
The'overall rating of this area was: Satisfactory
. Program elements appeared to be well documentsd. (
, Implementation was in place, but could be strengthened.
, 4
.
The STRENGTHS noted in this area included:
., a. The computerized NUCLEIS Work Request Control System was a strong
asset to the program.
. b. The overall maintenance program was well documented.
c. Accountability fer work requests in the field.
d. Retrievability of completed work request documentation, including
hard copies before final storage, and microfilm copies after final
storage.
;
The WEAKNESSES-identified in this area included:
a. Work Request-work instructions were less than adequate.
b. System Engineers were not utilized for work instruction inputs.
c. Approved procedures were not always utilized with work
instructions.
d. Reviews of completed work request packages were not effective in
identifying all inconsistencies.
e. Work requests indicated the planners were not always walking down
work locations / activities prior to planning.
f. No program to identify or track rework items,
g. Vendor Technical Manuals were not revised in a timely manner.
h. Vendor Technical Manuals were not controlled as rigorously as the
program required.
i. Post maintenance testing requirements were inadequate and routinely
did not specify acceptance criteria.
- 3.7 Maintenance Decision Process
The overall rating of this area was: Good
Program elements were not evaluated to determine a conclusion.
Implementation was functioning well.
The STRENGTHS noted in this area included:
a. Plant management involved in and aware of decisions regarding
maintenance activities for maintainabi'.ity, upgrading, replacement
and plant aging.
The WEAKNESSES identified in this area included:
a. Plant management decisions to defer permanent repairs to degrading
components and equipment with a history of repeated maintenance and
repairs, i.e. Main Steam Safety Valves, Main Feedwater Control
Valves, temporary leak sealing of 80 valves throughout the plant.
Better use of the NPRDS data (both input and output) should have '
allowed plant staff to recognize these adverse trends and in turn
raise these issues with management. (See ISSVES).
.
$
. . . _ _ _ . _ _ _ _ _ . _ _ _ _ . _ _ _ _ _ _ Q
. - ._
. . _ _ _ _ _ _ _ _ _ _ - _ - _ - _ _ _ _ _ - _ - _
.
s- -a -
' - '
- 58 -
.
,4. 0 TECHNICAL SUPPORT
The overall rating of this area was: Satisfactory
Applicable requirements of this element have been developed, documented
and effectively implemented. Areas requiring improvement are
approximately offset by better performance in other areas.
STRENGTHS were noted in rating of the following areas as Good:
.
4.4 Role Of Quality Control In The Maintenance Process
4.5 Integration Of Radiological Controls Into Maintenance Process
4.6 Safety Review Of Maintenance Activities ~
WEAKNESSES were noted in rating of the following areas as Poor:
4.1 Internal /CorporakeCommunicationsChannels
4.2 Engineering' Support '
4.1 Internal / Corporate Communications Channels
The overall rating of this area was: Poor
Program elements appeared to be adequately addressed.
Implementation was. inadequate.
The STRENGTHS noted in this area include:
a. Daily maintenance status and planning meetings were conducted.
b. A computerized plant information system (NUCLEIS) was available to
all departments and contained the status of all maintenance
activities.
The WEAKNESSES identified in this area included:
a. Inadequate identification and resolution of technical issues
associated with maintenance, i.e. design deficiencies associated
with the turbine AFW pump. (See ISSUES)
b. Inadequate equipment failure determination, i.e. turbine AFW pump
governor failures, CST nitrogen sparge line design interface,
temporary modifications and lack of use of NPRDS information in
failure determinations. (See ISSUES)
c. Inadequate planning and scheduling, i.e. the planning function
drives the scheduling function instead of vice-versa. (See .T SUES)
d. Failure Effect Mode Analyses were not considered in the
implementation of temporary modifications, i.e. CST nitrogt t.parge
line/ turbine AFW pump governor nitrogen supply line temporary
modification design interface. (See ISSUES)
e. Design and system engineers were not involved in the review of
Maintenance Requests, i.e. turbine AFW pump overspeed event. (See
ISSUES)
i
I
.- _ -_ _-____ -
.
_ _ _ _ _ _ _ ._
,
*
..o r.
4
* ' t-
p
._ ,
,
- 59 -~ ,
,
<
'
..
_~4;2 Engineering Support
E(
*
,
The. overall' rating of this area was: Poor.
Program elements appeared to be. adequately addressed.
Implementation was1 inadequate.-
'
'
a
-
=
R 'The ' STRENGTHS noted in this area included:
"
:- '
' a .' The' Preventative Maintenance planning program was.well documented,.
'
staffed,~ and implemented.
i b. Equipment Qualification program requirements were well documented
and were computer identified / controlled for work requests (i.e.
n ..
NUCLEIS system).
' c. Use and control of qualified personnel were well documented.
>
,
The WEAKNESSES identified in this area included:
a. Inadequate performance of failure determination analysis, e.g. see
Appendix C, Item 3, regarding nitrogen purge line. modifications,
b. Excessive.use of contracted engineering personnel.
g - c. . Insufficient usage'of human factors considerations in work
'~ '
instruction preparations, illustrated by " Maintenance Observation
Forms'! . indications of in process work problems.
' d. ' Absence:of integrating System Engineers into the work process,
e.- Failure to provide engineering guidance in the establishment of
. post maintenance testing requirements.
O f. Inattention to detail. .
"
f .
,
4.3 Role Of PRA in'the Maintenance Process ,
b u The overa[1 rating of'this area was: Satisfactory " o;. '
.
+
'
s Frog' ram' elements appeared to be missing or inadequate. ,
V
Implementation was in place, but could be strengthened. , -
The STRENGTHS noted in this area included:
[ f x. l_ p : a. . Safety significance of. maintenance issues appeared to be considered
* '
g, .through planning, scheduling, and prioritization of work. 5
The WEAKNESSES. identified in this area included: -
*
a. No:PRA had been prepared for Rancho Seco.
b. Specific PRA based techniques were not applied by system engineers
.
or other plant staff.
- 4. 4 Rcle Of Quality Control In The Maintenance Process
The overall rating of this area was: Satisfactory
- Program elements appeared to be well documented.
"' Implementation was in place, but could be strengthened. r
. .__ __ ________ _ _ m
_ _ _ _ _ _ .
_ _ _ , _ _
t
. ~ s i
~
.A 't
'
l . '
- 60 -'
t . .e ,
,.
l
-I
The STRENGTHS noted in'this area included: l
l' r. e i l ~ a. , Quality Assurance has' performed 142 surveillance of maintenance 1
.~ activities in 1988, with,55 findings and/or observations reported. f
b. ,The licensee.had successfully; implemented a program to ensure that j
quality deficiencies are promptly brought to management attention. 1
The WEAKNESSES identified in.'this area included:
a. Initiation of a Potential . Deviation from Quality (PDQ) was not
required for test failures that can be resolved by corrective
maintenance ~and retest (see ISSUES).
b. Quality.. review of maintenance Work Requests (WRs) were not always
reviewed for completeness and adequacy in that work completion and
QC' verification signatures were not obtained as' required by the
SMUD program.
c. Full engineering evaluation of unusual, unexpected, and apparent
abnormal conditions were bypassed by using a Work Request rather
than a PDQ to address the problem.
d. Quality Assu ance vendor audits did not contain provisions to
ensure that oesign or manufacturing changes made to spare or
replacement parts are identified and reported to the licensee.
4.5 Integration Of-Radiological Controls Into Maintenance Process
The ~ overall rating of this area was: Good
Program elements appeared to be well documented.
Implementation was functioning well.
The STRENGTHS noted in this area included:
a. Chief Executive Officer Nuclear Directive, (CE0-ND-88-06), clearly
established a SMUD corporate commitment to keep radiation exposures
ALARA, consistent with the requirements of 10 CFR 20.1(c).
Procedure RSAP-1101 clearly defined plant personnel
responsibilities for implementation of the policy, and the Rancho
_
Seco ALARA Implementation Manual invoked implementing procedures
for pre-work ALARA reviews, ALARA job planning guidelines, job
exposure tracking, and post ALARA reviews; these provided
guidelines for ALARA work practices to' minimize, control, and
evaluate workers' exposures.
b. An individual in each key plant department was designated to act as
an ALARA coordinator; they were provided additional training to l
assist them in performing this function. These individuals were
assigned responsibility for incorporating ALARA considerations in
their respective department procedures and the work process.
c. The licensee's procedur'es included Radiation Protection (RP) and
ALARA planning in the initial phases of processing work requests.
The RP Department utilized and had direct input into' new work
requests as they were being developed on a 24-hour basis via a site
computer based system. This system allowed for assignment of
existing Radiation Work Permits (RWPs) or generation of new RWPs,
input for RP. hold points during the job evaluations, and special
comments related to RP controls.
_ _ _ _ - _ _ _ _ _ _ . _ _ _ _ _ _ _ - - _ - _ - _ _ _ _ _ _ _ ._ _ - _ . . _ _ _ _ __ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ - -
,7- = F Y k~
>
.
-
61 -
d. The RP Department maintained a computerized photo retrieval system
of.various plant areas and components that aided in planning
special aspects of proposed work.
e. The licensee had established a Hot Particle (HP) Program; this
program required pre-job briefings with the workers, pre-job site
inspections by RP supervisory personnel, use of additional
r .p rotective clothing, and special controls for entering and exiting
HP Zones;,it also established frequencies for surveying workers.
f. The licensee had reduced the number of contaminated plant areas,
'and established tracking and trend' analysis to ensure radiation
s
reduction' efforts were maintained and evaluated.
, / g. Annual exposure estimates and ALARA goals were established for each , E site department, based on planned and unplanned tasks. ,,
h. Worker awareness of ALARA principles and practices was. evident;
i+ performance indicators were posted at several key locations on
'7 +
site.
, i. The licensee had been effective in implementing corrective
actions for previously identified weaknesses in this area.
'The WEAKNESSES id'entified in this area included:
?
'a. The licensee appeared to maintained a limited number of mockups for
radiological training; however, additional mockups were being
'
considered for purchase,
b. Based on interviews with cognizant licensee representatives, it
appeared that department ALARA coordinators needed to be given more
time in order to be fully effective in this function.
4.6 Safety Review Of Maintenance Activities
The overall rating of this area was: Satisfactory
Program elements appeared to be well documented.
Implementation was not sufficiently evaluated to determine a conclusion.
Although implementation was not evaluated by the team, the following
program STRENGTHS wre' observed in this area:
a. The GET program included training of all personnel in the
principles of plant safety including, safety equipment
requirements, confined space hazards, asbestos controls, chemical
hazards, electrical hazards, ladder and scaffolding hazards,
compressed gas cylinder hazards, first aid services, fire
protection, entries into carbon dioxide fire protected zones and
equipment tagging.
b. Administrative procedure RSAP-0105 adequately defined the policy
and responsibilities for the licensee's Occupational Safety and
Health Program.
c. The Work Request Planning procedure (MAP-0006) referenced and
incorporated plant safety requirements in the work process.
d. The licensee had recently received an INP0 " Good Practice" for
their heat stress control program.
e. The licensee maintained a Registered Nurse onsite 24 hours / day to
provide medical emergency assistance for injured personnel.
- _ _ _ _ _ _ _ _ -
_ _ - - __ _ _ . _ _ _ _ . _ _ - _ _ _ - _ - _ _ _ - _ _
_
s. 4 j
'
l,
- 62 -
! l l l
The following WEAKNESS was noted in implementation of this area:
l a. Industrial safety reviews of work packages appeared not to be j
consistently incorporated into the work package. j
4. 7 Integration of Regulatory Documents Into the Maintenance Process
l
The overall rating of this area was: Not determined.
Program elements were not evaluated to determine a conclusion.
Implementation was not sufficiently evaluated to determine a conclusion. ]
5. 0 WORK CONTROL
The overall rating of this area was: Satis factory
Applicable requirements of this element have been developed, documented
and effectively implemented. Areas requiring improvement are
approximately offset by better performance in other areas.
STRENGTHS were noted in rating of the following areas as Good:
5.2 Establishment of Work Order Control I
5.6 Maintenance Work Scheduling
WEAKNESSES were noted in rating of the following areas as Poor:
5.8 Maintenance Procedures
S. 9 Conduct Of Post-Maintenance Testing
5.1 Review Of Maintenance In Progress
The overall rating of this area was: Satisfactory
Program elements appeared to be adequately addressed.
Implementation was in place, but could be strengthened.
The STRENGTHS noted in this area included:
a. Maintenance personnel, as a whole, appe red to understand work
control concepts.
b. The appropriate management and supervision authorizations were
received and health physics reviews of WRs were performed.
Security and fire protection reviews were not performed on all WRs,
but for the WRs reviewed this appeared adequate.
c. The licensee used up-to-date WRs for the performance of work,
revised WRs as necessary, and identified these revisions.
d. WRs identified work to be accomplished, QC hold points, and
references, and defined work package completion. '
e. QC appeared to review each WR.
f. The exact status of a WR could be determined at any point from
preparation to final closecut.
g. Proper operational personnel were identified and clearances were
issued as necessary. l
h. Operational Limiting Conditions for Operation (LCOs) were
identified in WRs.
i. Housekeeping and cleanliness appeared to be well maintained.
___ - _
. , . . . - . _ _
- - - - - _ - - - -
% 7 .s <
,
*~
,,
rp -
;
f;( U s [pi
.,
,
- G3 -
'
t, ' 1' , ,
,
,
j. Proper restoration of barriers and covers was accomplished.
t
> The WEAKNESSES identified in this area included:
a. During the performance of maintenance controlled by a work request,
instructions were not always adhered to. (See ISSUES).
b. The latest Technical Manuals were not used during replacement of
the AFW pump governcr. (See ISSUES).
c. Adequate functional testing was not performed before returning
equipment back to service. (See ISSUES).
-d. Safety precautions to protect equipment were not adequately defined
prior to commencement of work. (See ISSUES).
e. WRs were not considered to be procedures. As a result they did not
adhere to a stringent format as defined in the Rancho Seco
Procedure Writer's Guide, and did not always include 'INPO Good
Practice Guidelines.for effective control of work. (See ISSUES).
f. Drawings used during. investigation of the AFW overspeed event did
not fully reflect as-built conditions. (See ISSUES).
g. Some personnel were not present for the AFW investigation pre-work
'"tailboard" meeting; as a result, they were not fully aware of
their job responsibilities or special requirements specified in the
Work Request. (See ISSUES).
h. Equipment, tools, a'nd M&TE were not identified in the WR; however,
these were identified by personnel performing work at the pre-work
tailboard meeting.
i. Correct parts and materials-were not identified in the WR. These
were identified by personnel performing work at the pre-work
'tailboard meeting.
5.2 Establishment Of Work Order Control ;
i
The overall rating of this area was: Satisfactory' l
Program elements appeared to be well documented.
Implementation was in place, but could be strengthened, i
'
The STRENGTHS noted in this area included: ,
l
a. The work order control program was well documented. j'
$ b. The. lines of authority were clearly established.
y c. All required crat cs necessary for the work were identified.
d. Sp'ecial processes and required clearances were identified. '
-
, ..
The WEAKNESSES identified in this area included:
*
a. Lack of special qualifications and training requirements. !
r
b. Failure to' include any special tools or test instrumentation
requirements in the work package.
c. Inadequate post maintenance testing requirements identified and +
.
documented in the work packages. i
,
i
&
d
_ _ _ _ , - - _ _ _ - _ _ _ _ - - . -
- - -- -
---
p g ;, -
' '
.f i
! ,y ' ,
-
- 64 - !
!
5.3 Maintenance Of Equipment Records and History ,
+ .
l
The overall rating of_ this area was: Satisfactory
Program' elements appeared to be adequately addressed.
Implementation was in place, but could be strengthened.
The STRENGTHS noted in this. area included:
a. Maintenance records were being kept' current and were closely i
correlated with the Master Equipment List. s
The WEAKNESSES identified [in this area' included:
a. A maintenance hi' story program did not exist.
b. Historical records were;available, but not. easily obtainable.
Equipment history files were not all consolidated by the NUCLEIS
system.' For a plann_er;to' utilize equipment history, he had to pull
eup.all of the past work orders for that' item and review all of.the
'" work, performed" write-ups to familiarize himself with what was
done. This. is where he identified if there are any special
considerations that have to be dealt with for the newest work
package.' The NRC team! perceived'that only the last work file may
*
be receiving this detail.of review.
5.4 Conduct Of Job Planning
The overal16 rating of this area was: Satisfactory
Program elements appeared to be adequately addressed.
Implementation was in place, but could be strengthened.
The STRENGTHS noted in this area included:
a. The licensee maintained occupational exposure well below their
established goal for maintenance activities during 1988.
b. The' program provided for adequate control of job planning.
c. The program and procedures provided for early coordination with
technical support groups.
-The WEAKNESSES identified for this element included:
a. Scheduling of work appeared to be a function of the planning
organization instead of the schedulers,
b. Work packages did not include tool /part availability.
c. Work instructions did not identify when special qualifications may
be required for the performance of the work.
d. Scheduling of personnel to perform the work was the responsibility
of the lead craft cjroup and was not effectively coordinated for all
jobs,
e. Job planning was not adequately performed, as evidenced by the pen
and ink changes and revisions necessary to complete tasks.
f. Technical support groups were not routinely being requested to
support the work planning.
- - _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ - ._.
,
_ _ _ - - - - ____ -.__ _ _ _
,
ly -
a .: y
l
:. e
a -
,
.)
/,, '* { s -
65 -
, , .-
cp_- _.
5.5.' Performance Of Work Prioritizations
'
The overall rating of this area was: Satisfactory
Program elements appeared to'be adequately addressed.
Implementation was in place, but could be strengthened.
-The STRENGTHS noted in this area included:
~
a. Work prioritization considered the safety significance of the work.
. b. Work.prioritization coordinated balance of plant maintenance with.
its'effect on safety,
c. The plant had implemented a q'uarterly surveillance system tracking
program.
.
'
d. Priority' categories were based upon the INPO guidelines in the
procedure controlling work prioritization
e. ;Prioritization methods were clearly defined in procedure MAP-006.
The WEAKNESSES identified in this area included:
a. : Work prioritization was' not based upon (PRA) criteria.
5. 6 Maintenance Work Scheduling
,
'
The~ overall rating of'this area wasi Satisfactory
Program elements! appeared to be well documented,
~
s "/ '.. '
Implementation.was in place, but could be strengthened. ,
a The STRENGTHS noted'in' this area included:
1
a. 'The maintenance organization had established a goal to have WRs
j prepared seven days prior to commencement of work. '
<
b. A. requirement had been established for planners to walkdown the job
site during the WR planning' stage.
-
.,
' c. . Planners took a great deal of photographs when planning a job.
'
d.- WRs we're tracked through the planning process to completion,
j e. Preventive, corrective, and predictive maintenance as well as
surveillance activities were scheduled and controlled.
f. Environmental / seismic qualified equipment components were
identified on the WR.
g. Maintenance activities were scheduled to assure availability of
adequate supervision.
h. A work experience feedback form was attached to each WR to aid the
planner in preparing similar WRs in the future.
i. WR5 were sent to appropriate personnel to be reviewed for ALARA
'
considerations.
j. A daily plan of activities included maintenance activities.
k. The Maintenance Manager was observed personally involved in
verifying scheduling of work area preparations for a difficult work
effort in a high radiation area.
The WEAKNESSES identified in this area included:
a. Post maintenance testing was prescribed on a work request and was
not tracked separately.
_ _ _ _ _ _ - _ - _ _ _ .
.
= .- ..
7. - ,;.-. . .
.
-
,
.<
+
g, c4 , i
'
ll^
.x ,
, ,
,
,s "./3
,
66 -
,
.
'
, 3
5. '7 Establishment Of Backlog' Controls
c -The overall rating of this area was: Satisfactory.
Program elements appeared to be adequately: addressed. j
>4
Implementation was'in place, but'could be strengthened. ~~
;
The STRENGTHS noted in this area included:
'
j
.V .
.;
.The backlog of' maintenance items had been reduced from 1300 in
"
a.- 1
October 1988, to 383. in February 1989; this was very close N the 1
licensee goal of 300 items.
b. Operations staff prioritized the backlog items, "j
c. All Priority ~1 items on the Long Range. Scope List (LRSL) have been i
,
completed. :The licensee indicated that there were 388 priority 2,
ittms on the: list . Of those, 69 items had been completed sincei . l
December 1988. {
d. Procedures require that LRSL items may only be downgraded by.the '
. Assistant General Manager (not the responsible manager). :
e. Backlog reduction appeared to consider safety significance. l
f. Additional ' personnel were added to track backlog items. 3
.g. Preventive maintenance deferred by more than 125% of the allowable 'l
amount was identified to the maintenance manager. j
h .~ Deferred maintenance was tracked (but not specifically controlled). I
1. Maintenance scheduling contributed to maintaining occupational ;
. radiation exposure ALARA goals. i
1
.The WEAKNESSES identified in this area included: I
a. Some' maintenance items were allowed to be deferred for up to two
years.
. b. ~The maintenance engineering staff appeared to be stretched thin
during outages (See IS' SUES). .
c. The ratio of safety related maintenance to non-safety maintenance ;
was not trended. 1
d .' The ratio of emergency to non-emergency maintenance was not ,
trended. l
'
5.8 Maintenance Procedures', l
< >
*v 1
The overall rating,.of this area was: Poor .
)
Program elements appeared to be missing or inadequate. ,
Implementation wt.s inadequate, , j
Th'e STRENGTHS noted in this area included: i
~
a. There was h documented review ard approval cycle for maintenance )
procedures. . . ;
b. The program identified indiviaual responsibilities for procedu*e !
development. i
l
i
4
-_-__ -_a_-__ ______ O
-- ~
,
! 'a' %. t
' '
'
- 67 -
1 .
i y;
* *
h b ,
.-
.
g
. ..
.
"'
;, ja .The WEAKNESSES identified in this area-included: , J"
,.
>
. ..
D'
fi >
<
, tia. Maintenance procedures were not in a consistent format, apparentlyf g, ' +R
4
;
'
.due,to age or due to organizations having differing interpretations 1tc
'
of the' format guidelines. fE
' '
.- 7
' '
<
b .- Some work procedures cannot be worked as written and yet revisions - C #
'
/- I j
'
were not. incorporated in a timely fashion; crafts were trying to-
/
: 4
' work with procedures and work instructions that were not compatible
with'the work. (Management Observation Forms documented several
,
.
r ,
.such instances and employas complaints). '
, .
t' -c. ~ Work * instructions were written to the detail and level of more 6
formal ~ procedures, but were not required to comply with similar
format and content criteria, nor go through a similar' formal review ,*,
f and approval cycle; (e.g. WR 154142 instructs how to disassemble
,
and inspect the Terry Turbine and contains dettiled step by step
instructions),
d. Work instructions used as procedures were not technically correct,
consistent or tested; (e.g. The Terry Turbine governor change out;
the inadequacies also were identified in April 1987 by the craft
personnel, but were uncorrected for *.he repeat work in April 1988
and January 1989).
e. Procedures and work instructions were not developed to address the
possibility of an action not occurring as designed.
f. Procedures did not identify tools, instruments, spare parts or
limitations for accuracies, ranges, sensitivities, etc.
5.9 Conduct Of Post-Maintenance Testing
The overall-rating of this area was: Poor
Program' elements appeared to be missing or inadequate.
Implementation was inadequate.
l
The STRENGTHS noted in this area included:
a. The maintenance work order process recognized that post maintenance
testing should be accomplished.
WEAKNESSES were identified in the following areas:
a. Post maintenance testing performed on the Terry Turbine governor
did not include any of the warnings or cautions contained in the
vendor technical manual as to what may occur during the filling and
venting of a newly installed governor, nor did any of the previous
work packages written for the same job. '
b. Post maintenance testing instructions did not include adequate
acceptance criteria. The post maintenance test that was completed
prior to the attempted filling and venting of the governor did not
,
specify any acceptance criteria as to what constituted an
acceptable test of the equipment after the lube oil changeout and
the filter replacement.
c. Post maintenance testing did not include any criteria as to what
'
I constitutes operational readiness of the equipment or system.
After the installation of the check valve in the nitrogen sparge ,
'
+ line for the condensate storage tank, there was no post maintenance ,
.
e
'
)
.
v4 r .. . gn.. >-
e
* '*
-
-,s 'm ~
f ;g.wgm; gJ,7
, ,
' '
_
=o
9 . ,, ,
,
'y\
* g f, ' * YU y(Q Q
, . .
gf% M /b(, - 68 - 1
f ; r.. i i'
s tue. ,
, a.
.s
'ls ;j q; 'p , y 7
, ._ _ y v 5
_ , +_ .t> ,
y r E
,
'*+ *
W 7 ;y " A -
m,
,
,
3 +
+
. fl V 1
! testing : identified to insure the. check valve would perform its: .,
,
. , ,
,
'
-intended function. ; i <Tb ,i'?
'
d. . Post. maintenance testing _did not' insure equipment or sy' stem
operational readiness. After the Terry, Turbine Trip throttle. valve ,' ;-
iL was;M0 VATS tested in'1988, the post maintenance test did not insure 4 r f
operational readiness of, the equipment, because the trip failed: thel i
i -first' time.1,t subsequently was called upon'to work.
5.10 Review Of' Completed Work Control Documents
The'overall rating of this area was: Satisfactory
Program elements appeared to be adequately addressed.'
Implementation was~in' place, but could be strengthened.l
Th'e STREN'GTHS noted in this area included:
f =
a. The program for the review and control of completed work control i
documents was well documented.
b. The work control ' packages had a built in feedback mechanism that
could be useful' if used consistently,
c. Completed work packages, which the licensee noted to have errors or
' ,
inconsistencies within them, were corrected before being
transferred to records storage.
'
, 'The WEAKNESSES identified'in this area included:
a .' . Work control documents were found in records s torage with minor
ambiguities and' omissions.
b. Feedback forms were.not always actively pursued to ensure the
follow-up was' completed in a timely manner.
c. Work ~ packages were not reviewed for compir.,eness in a timely
. manner, i.e. less than 30 days from wrk completion.
d. Procedures, identified as inadequate for the work, were again
.
. identified 'to be- used to perform the task several months later,
without revision or upgrade.
6.0 PLANT MAINTENANCE ORGANIZATION
., The overall rating of this area was: -Satisfactory
Applicable requirements of this element have been developed,Ldocumented
and effectively implemented. Areas requiring improvement'are -
*], ,
3
-
approximately offset by better performance in other areas. '
,
.
79 ,
, o
STRENGTHS were noted in rating of the following areas as Good: ' O'
,
ff- / 4
-
# h
- yj s
,
'
6.2 Contracted Maintenance i
'g
l k a e
4 ,
; /, WEAKNESSES: No areas were rated as Poor.
,x _
*
%)l -
w J
}-. _
L i .
m
'
- _ . _ _ __ _
"l-
. 2 1
'
-
- 69 -
P
f
'6.1 Establish Control Of Plant Maintenance Activities
,
'
The overall rating of this area was: Satisfactory
+- Program alements appeared to be adequately addressed.
Implementation was in place, but could be strengthened.
The STRENGTHS and WEAKNESSES noted in this area are' delineated in
paragraphs 6.1.1 through 6.1.3, below:
6.1.1 Mechanical Maintenance
The overall rating of this_ area was: Satisfactory
Program elements appeared to_be adequately addressed.
Implementation was in place, but could be strengthened.
The STRENGTHS noted in'this area included:
a. Vendor manuals and drawing control program requirements were
generally established.
b. Work Request Programs were computerized, allowing easy access to
maintenance data by use of terminals throughout the site.
c. Maintenance procedures were written.using INP0 and Rancho Seco
Procedures Writer's Guides. i
d. Continued development of the Rancho Seco maintenance self
assessment, with compilation'of prior and newly identified needs of
maintenance with regards to facilities, equipment environmental
factors, radiation protection, lifting aquipment, human factors
considerations, labeling, sp;re parts storage, tools, technical
= data, safety, equipment protection, communications (public address
systems etc.) preventive maintenance program enhancement, control
;of overtime, and trainir.g.
e. Established flexible work instructions to allow for in process
deletions of activities not applicable to a particular portion of-
' maintenance.
Established control of calibration of ools and equipment.
~
f.
*
Jg. Established work performance accountability in the work request
,
process.
.
-y
The WEAKNESSES identified in this area included: .4
[ a. Plant systems and components integrity not always maintained
- because of rework and temporary repairs (e.g. temporary leak seals '
j ,_ of 80 valves).
Inadequate implementation of the vendor manual and drawing control
'
. .b.
programs (Woodward Governor manual and drawings).
c. Training on mock-ups and component specific classroom training
,
could.be improved. -i
d. Control of materials / components used in the maintenance process was l
l~ > not always adequate (e.g. AFW turbine governor out of storage in l
excess of 14 days). l
l 1 1 ________.____m
w =
;;5 - .,
, c 9 ,
s ^l,l
~ ~c -
:. ' 70 l. , s
.; '
,
, ;. ,
*
~ s
T.f-- ,
. ,
:; , s+c ' ., ,,
,
m e, ,
?
_
,
,
, ,
.
3
6.1.2 Ele'trical
c Maintenance * ." ' 1 ,." *
s .a .t
,
~a. ^
The overs 11 rating of thisLar'ea was: ' Poor; ) .
,
Program elements' appeared lto be; adequately addressed.
*
Implementation was inadequate. s
,
^
The' STRENGTHS ' noted" iri this ~ area" include: ,"
'
. . ^
1
a. ProceduresMAP-Ob09fand'M-159includ'ed;identificationof. electrical-
<
'
maintenance needs.
1
b. . Controls were establish'ed*
to monitor electrical maintenance
activities.
.c. Measures were established for resolving discrepancies detected
during, maintenance activities.
:d. Maintenance procedures.used were contro13ed by assuring that the
proper and latest revision of the proceduNs were used.in the;
maintenance' activities.
e. Guidelines were' established to allow portions of a. procedure.to be
marked."not applicable" when an activity may not be required for a
particular. phase of maintenance.
'The WEAKNESSES identified in this area included:
a. Appropriate measures were not established to ensure that plant
electrical distribution system integrity was maintained, i.e.
circuit breaker / fuse coordination schemes,. functional testing of-
running current to solenoid valve to turbine AFW pump governor.
(See ISSUES)
; b. . Appropriate measures were not established'to' control rework and
i- temporary repairs of electrical maintenance, i.e. use of
unqualified parts.'(See ISSUES).
c. Appropriate measures were.not established to control and implement
vendor technical manuals, i.e. Time-Overcurrent Characteristic.
Curves for as-built Class 1E and balance of plant circuit breakers. j
were not available.:(See ISSUES) i
d. . Appropriate measures were not established for configuration control
to assure maintenance. activities always return a system to its
normal operating configuration, i.e. temporary modifications and
post-maintenance testing procedures did not include appropriate l
test and acceptance criteria. (See ISSUES) '
6.1.3 Instrumentation and Controls Maintenance
The overall rating of this area was: Not determined. :
'
Program elements were not evaluated to determine a conclusion.
Implementation was not sufficiently evaluated to determine a conclusion.
6.2 Contracted Maintenance
The overall rating.of this area was: Good
Program elements appeared to be well documented.
Implementation was, functioning well.
,
9
__.2 ._ _. 2 __ .. m ._ _ . _ _ _ _ _ . . _ _ _ _ _ _ _ _ _ _ _
-
T s .y
3 '
;
'
; .c - 71 -
.The STRENGTHS noted in this area included:
a. Personnel selection, qualification, training and personnel control
were well addressed in licensee procedures for contract maintenance
personnel. Training was established according to the work to be
accomplished, with specific required procedure reading for all
workers.
' b. The individual responsible for selection of contract maintenance
personnel had been on site for a long period, and could make
'
,
selections of workers from the union hall by name and specific
,
qualifications. ,
f There appeared to be no union constraints that would hinder work
'
-c.
/
"c
/ activities, in controls or discipline of contract workers.
-
g
'
, 3 s
,
.
,
;
No signif.,1 cant', weaknesses were noted in this area.
. .. . ,
-
'
4 '
6.3 Establishment Of Deficiency Identification and Control System
,, '# ,.
-
,_
' ,
6
;The o~verall rating of this area was: . Satisfactory '
,
Program elements appeared to be adequately addressed.
'
+
.
,
l
.
. Implementation was in place, but could be strengthened.
'
* '
y - ,
4 x
# ' ~
!' "
The STP.ENGTHS noted in this area included: j
'
-
a. . The'PDQ process was an easy and effective means of reporting ,
'
deficient conditions.
*
b. The PDQ process was followed t,y all plant and contractor personnel.
J
The' WEAKNESS identified in this area included: l
!
a. Root cause analyses were not performed, even when equipment
' underwent repeated maintenance for the same failure.
b. Inadequate engineering of plant modifications to correct identified ;
deficiencies. '
l
6.4 Performance Of Maintenance Trending . j
L
The overall rating of this area was: .Not determined.
Program elements were not evaluated to determine a conclusion.
Implementation was not sufficiently evaluated to determine a conclusion. I
However, the following WEAKNESS was noted in this area:
a. There was no established mechanism to identify or track " rework"
items. The SMUD staff was aware of this and had committed to
establish a rework program; a commitment date had not yet been
established for its implementation. ,
?
_ _ - _ _ - _
?.t
7,-
~; . ; ,i
"
* * 1
,
( 1'
. . .
l
,Y a
"
. r
l,
6.5 Establishment Of Support Interfaces
!
The overall rating of'this area was: Satisfactory
Program elements. appeared to be missing or inadequate.
. Implementation was in place, but could be strengthened.
The STRENGTHS noted in this area included:
l a. 'On-site computer systems were used extensively by various-
' departments for tracking maintenance items.
The WEAKNESSES identified in this area included:
a. System engineers were not involved with WR planning, procurement,
post maintenance testing, etc. (See ISSUES).
b. System engineers and maintenance engineers did not have a clearly'
defined role in interfacing with :.aintenance craft (See ISSUES).
c. There appeared to be no single individual, such as tne system
engineer or maintenance engineer, knowledgeable of all activities
on a system (See ISSUES). j
d. System engineer and maintenance engineer responsibilities were
described 'in a number of different procedures, but not defined in
any one document for clarity and integration.
e. Maintenance personnel did not appear to document discussions with
various organizations on a routine basis.
f. Procurement personnel did not appear to have contacted any
engineering personnel to resolve vendor questions concerning
rotation-direction of the AFW turbine governor. (See ISSUES).
g. Support personnel for the AFW overspeed investigation were not'
contacted to attend the pre-job tailboard meeting; as.a result,
difficulties were encountered while performing the investigation
(See ISSUES).
7.0 MAINTENANCE FACILITIES, EQUIPMENT AND MATERIALS CONTROL
The overall rating of-this area was: Satisfactory a
:. Applicable re'quirements'of:this element have been developed, documented ;
and effectively implemented. Areas re4uiring improvement are
approximately offset by better. performance in other areas.
STRENGTHS: were noted in' rating of the following areas as Good: ,
'
f
7.2 E, establishment Of' Materials Controls
WEAKNESSES: No areas were rated as Poor.
t
&
P
k
- _ _ - - _ - - _ _ _ _ _ _ _ _ _ _ -
,_
.- . .
- - - - . - - - - -- - - - -
- ' r. '.
,. 43 '
w ,
,
. , . . p
" ' '
'
' '"
., - 73 - .-
2-
'
* ,
'
u st '
I
q +
,
'.~
, - , .. .
.
, .. . . .
c',
, <
"
-
. 7.1. Provision Of Maintenance Facilities and Equipment 9-
J c
.
.
,
.
.
.
F
'
.The overall rating of this area was: Satisfactory -
' P'cogram elements' appeared to be adequately addressed.
-
'^
'
1 implementation was in place, but could be strengthened.
'
:The STRENGTHS n tid in this area included: J
? a. The I&C and Electrical work shops were optimized in regard to plant
location, nearness to plant procedures and technical libraries and
being close to their respective supervisory and management staff.
b. The clean tool room was centrally located to the plant and appeared
to be well organized. .!
.c. Fabrication of a new environmentally controlled weld test shop had
'
been completed.
d. Although limited in space, the contaminated tool storage, hot,
machine' shop and tool decontamination facilities were centrally
located in the Auxiliary Building.
The WEAKNESSES identified in this area included:
a. The mechanical maintenance work / fabrication shop was not optimized
in regard to location to'the plant, being close to the respective
supervisory and management staff, and plant procedures and
technical libraries. Although the mechanical workers in this area
maintained their own tool boxes, they-were not provided with
individual work benches. The grinding and abrasive cut-off saw
area was not provided with an exhaust ventilation system. The shop
area was not insulated and lacked adequate heating and air
conditioning for worker comfort.
b. The mechanical test and machine shop was in a separate building
from tha mechanical work shop area, making it inconvenient for work
on equipment that involved both areas.
c. Storage space appeared to be limited and not easily accessed for
contaminated equipment used for outage activities. The licensee
utilized "C" vans for most of such storage.
d. Most of the plant support groups operated out of temporary
buildings (trailers).
e. The licensee appeared to maintain a limited number of mockups for
worker radiological training. The licensee plans on expanding this
capability and was in the process of purchasing a mockup for
reactor coolant pump seal work planned for their 1989 refueling >
outage.
, t
J
l [
$
is ,
'
I
.i.!
. _ _ _ _ _ _ _ _ _
_
! 4.- r.
'
,, - 74 -
7.2 Establishment Of Materials Controls
The'overall rating of this area was: Good
Program elements appeared to be well documented.
Irolementation was functioning well.
The STRENGTHS noted in this are.a' included:
a. The procurement of spare 'and rep 1'acement commercial grade items for
intended safety-related applications had been strengthened by
requiring suppliers to identify and notify'the licenaee of any
design or manufacturing changes 1made to the original item. 1
b. The dedication of commercial grade items were' thoroughly evaluated,
by the procurement ( engineeringna'd quality departments, to verify
that all applicable technical and quality requirements were met for
the intended safety-related application,
c. Esta'alishment of an aggressive material control program to ensure
that materials'and equipment were procured from qualified sources,
properly received'and inspected, with good controls for issuance.
The WEAKNESSES identified in this area included:
a. No provisions to ensure that suppliers of commercial grade items
understand the purpose and need for proper. implementation of the
procurement requirement for reporting design or manufacturing
changes made to spare or replacement parts.
7.3 Establishment Of Maintenance Tool and Equipment Control-
The overall rating of this area was: Not determined.
Program elements were not evaluated to determine a conclusion.
Implementation was not sufficiently evaluated to determine a conclusion.
The STRENGTHS noted in this area included:
a. The tool room was centrally located to the Turbine and Auxiliary ,
buildings. l
The WEAKNESSES identified in this area included:
i'
a. The tool room was not routinely manned during backshift hours nor
on weekends. Shift craft workers had to contact the Maintenance
Shift Supervisor for tools needed from the tool room, which caused
delays in maintenance work when this individual was not readily l
available.
7.4 Provide Control and Calibration Of Meter and Test Equipment
The overall rating of this area was: Not determined. ;
Program elements were not evaluated to determine a conclusion. ~
Implemeritation was not sufficiently evaluated to determine a conclusion.
-
. . - - . _ _ - - - - _ _
--
o- ..,.
'
y - 75 -
8.O PERSONNEL' CONTROL
The overall rating of this area was: Good. ,
More than minimal efforts have been made.in this area, and this area has l
desirable qualities with only a few minor items requiring improvement. 'i
STRENGTHS: were noted in rating of the following areas as Good:
8.1 Establishment of Staffing Control
8.2 Provide Personnel Training
8.3 Establishment of Test and Qualification Process
WEAKNESSES: No areas were rated as Poor.
8.1 Establishment Of Staffing Control
'The overall rating of this area was: Satisfactory
Program elements appeared to be well documented.
Implementation was not sufficiently evaluated to determine a conclusion.
,
" The STRENGTHS noted in this aree acluded:
a. In 1988 the licensee initiated a prograra to provide plant coverage
24 hours per day, 7 days per week for core and key support groups
'
.that included the Maintenance Department (core).' This was *
'
f "' , accomplished by either 6 or 5 rotating crews, on 12 or 10 week '
.~ rotating" shifts of 12 hour shifts (7:00 A.M to 7:30 P.M on days and k, -
..
<
b o
,
7:00 P.M. to 7:30 A.M. on nights). Procedure RSAP- 0226 detailed .
'
-
#
e- the work schedules and worker compensation. The rotating shifts (-
LA !J 7 , y allowed for 2 weeks per quarter for class room training. > i. ,
The WEAKNESSES identified in this area included: ;
f6
'
,.
, , ,
e
<3
:,: a. Base'd on discussions with several maintenance supervisory and craft. >
; ~~ personnel, it appeared that there was a general. consensus that the .
(f
o ,
f shift' work was~ not cost effective, there were less available j" , i H" personnel for daytime work activities and the workers efficiency '
J
L decreased after 8 hours of work; especially, on the night shift. a
' 8. 2 Provide Personnel Training
l
The overall rating of this area was: Good
'
Program elements appeared to be well documented.
l
Implementation was in place, t,ut could be strengthened.
'
i
L L The STRENGTHS noted in this area included: L
a. Existence of a well documented INP0 accredited training program for
maintenance and technical staff training.
b. INP0 accredited programs were supported by a Job Task Analysis. l
c. The licensee had recently upgraded their General Employee Training
(GET) program to implement the guidance provided in INPO 87-004 for
GET training.
1
, _ _ _ _ _ _
...
-(;~
j, ,'e' tw L
"
- 76 - '
l+ . .
. . ;
4 -- 1
,
'
i e
T
. '.
, ;. . ', '
.
,
\
d .- Procedures existed;that' detailed the maintenance training and job l
'taskLqualification programs. l
e. . A ned training .and simulator facility was being constructed which
_
i
, was expected;to be completed in November 1989.
l
,
'.
f. 'The licensee'had purchased a diesel engine that will be set up for
training and qualification purpose's. .
j
g. Weekly. schedules of scheduled trainit:g courses were published in i
the-licensee's " Watts Happenin,g" site newsletter. {
The WEAKNESSES identified ~in this area included:
-
a. It appeared that the flicensee had provided very little oversight
and emphasis on implementation of the Maintenance Department
on-the-job training (0JT) and task qualification programs (See
ISSUES).
b. Maintenance Department Training Coordinator responsibilities were
assigned.to non-technical persons (secretaries) who appeared to
have very little time to devote to this important function'and
appeared not to be fully knowledgeable of their responsibilities or
the system.
c. OJT requirements have'not been established for work on diesel
generators.
d. Mechanical Maintenance personnel replacing and testing the AFW
turbine governor had not received any specialized in-house or
vendor training on this equipment.
e. Although the Rancho Seco GET program had been upgraded, the new GET
Handbook being provided to students exhibited an inadequate
technical review prior to publishing. The stated objectives for
each lesson could not be satisfied with the material provided in
the handbook.
8. 3 Establishment Of Test and Qualification Process
.The overall rating of this area was: Good
Program elements appeared to be well documented.
Implementation was in place, but could be strengthened.
The STRENGTHS noted in this area included:
a. The Craft training programs were documented and supported by an
INP0 accreditation and job task analysis.
b. Personnel training and qualification records were maintained in a
computerized data base system that provided for easy traceability.
The WEAKNESSES identified in this area included:
a. It appeared that the' licensee had placed very little oversight and ;
emphasis on implementation of their OJT program (See ISSUES)
i
8.4 Assessment Of the Current Personnel Control Status
The overall rating of this area was: Not determined.
Program elements were not~ evaluated to determine a conclusion.
Implementation was not .sufficiently evaluated to determine a conclusion.
,
- - _ __ _ _ _ . . _ _ _ _ . _ _ _ . _ _ _ _ _ _ _
_ __ - - _ _ _ _
_ _ _ _
'
s ,
e - 77 -
1
AP,PENDIX E
MAINTENANCE TREE l
The attached diagram is a similitude of the MAINTENANCE TREE (diagram) used ,
at the exit meeting on February 17, 1989. The version used at the meeting
was , colored in each block, to indicate:
, Green - Good '
l
'
Yellow- Satisfactory i
*
Red- Poor *
Blue- Not inspected
>.
,
Coloring of the blocks corresponded to the ratings indicated in each
, subparagraph of Appendix D of this report.
,
*
A rating mark was annotated below each block of the diagram, to indicate
the team consensus rating of activities represented by each block of the ,
diagram. 'The overall rating of a block, as summarized in each paragraph '
of Section 5 of this report, reflected heavy emphasis on the ,.
implementation of each activity, with less emphasis on program #
definition adequacy; e.g. a " Yellow" code for " Implementation" may
result in a " Satisfactory" overall rating for the block in Appendix A of
this report, even though a strong " Green" code was indicated for the
" Program Definition".
!
i
a
L ,
t
- _ _ . - - - _ . . _ - _
_ _ _ _ _ _ . _ _ . _ . _ _ _ . _ - _ _ - - - - - _ - - - - - ... 4& mN '
c -. .
-s s s
.
I, i,
,
1, . 1"e .
I '
\ , s. . \s!jj!!!![kis !I li! !!!a !!
j.4;
~
ik
m
.](~ sillJ i ga a' i il
1
:l-_ . -
p$
i,
l I
,
! l
<
\' t !, i-
. . ,
~ *
E ' 21
- \
'
.
s
_
"
l u -
r ; o- m ; liv ,uav , ,i!
sin ,i,Is.
:
,
, g ,g :.
- s
ll
ll \d
,
' '
Il L \,
g g _ k4 \ y \i.
git)if
_
j
_s
y H
u
;, , a a.ic .A: A
M
, ,
55 s N L N; Ni Ni
,y
b 'l l !I[: l![, ! m! !s .I
e til'Itii'
di ' u
13 II
! U E %- y~ fi g e C i\ _
I
A ::h
_
.
j
' ~
ll 6 II ME : I![ : I!E , II: !!!(
fe
Jti~C ,T-
F
'
,
5 %_ \] . l\ .
4 j g
Q'
+ CJIT . -
4e s.ls 4,-
1
3
!
'
.d.~s-
1
. ,.9 <-
.
I
o .
.
j
,
l
; i
.I \ \. -
'I y
,' 1
3 !, I,!!Y j!ls- l
-lll4
i ;
~ill!sl
.
=i '
2 ,
.
:
-
-
r-
r.i...!.$mi
- . j _
,3
, _
-
g,'
. r
a18'li
_
g
E ;- s=
{l,h. '
_ ,
-
- _
. . _ _ _ _ . _ _ _ _ _ . - .
.-
.
n n
i. 4
.- I 6,
Q
j? o h ' PRESENTt
TREE INITIATORS MAINTENANCE i
i = =><= , ,no.E.
= mnem
& RFICS & B1 DEST (DECE v4K1 em% A8 $41DfL Steaul% anUIDS)
4 perdoul BrPLCBCh itemG5
8 GBOvalla W PLmr1 ACEWE3
m
i
I I I i
w',a .v
,Q 1u___s---_-
3 '
_
ig i
u
' }.R .'
..
,
.
-
,.
.. i
igi gg uomme
s.
Sm=:{:: .
e
$e3h4].
~
3 A'
w ti ti
At
ai
IK i ;
J
:
a i
-
88tenus e' - i
81E5 . : fi f
.x. j
. <
St I
_ . 'd8'W yg
. ,, ,
m f , ,
iz i -
.massen cf
== anse.. . ,
2-
'
.
h
13
..
.:
~
, ~ , . . .- /
ea
'[l
sets esfigung.
gg -
j fit. -
K 3 t ! ..
s i ix
L g j
p
.,
,
. .
le 40
,
, , , ,
,
.,
_ ledtWIWAS
m:Ima ' "I
_
_ W1tmat d'
.a
ESAAfMT'
j 'A,1
1 I
most us car. a van a cowenom sov 42 sam,e2ssot<2ssoloissos.stwa 425ac7 e e2ws
WA
i_____________________ . _ _ _
-_ - _ _ _ - _
r
t
l TREE lCTION TREE .
.
=
aa= l uwe-i i l l
{ 1
06 4 PUFOshmeG D uAk2
1
{
)
a
ene
.
:,m=1 u==
. a ,,-
~
l E
et #
4
p.______.___+,m a==== -
' s i
E
i. I o I ,, I ..
- ,ust :
=
g> e - ,.s; y
e y =.=.a.r ..my
i.mm
ynnu . . , w_ , s.,
K i M 3 i V I M
7, - o n a, 4 insum - enoer maim a . -
u -
ge W Pusrr raauins -
!
~
f.
jW 8 NM ' ... )[ l.
i r){~
i
um '" K i
i
IX i
77_ iX i
m.e==ma -> W
-
pai . a., . a n- A na "
i.
i x -
+- .
_'
"
K '
7 .o
e [
~
_.
-
a y
em
o s' .= ue f .
~.m
3 # Y' ,,
s
ix -
i.
ww.g ' '
- _ _ /
gs ,
-a ,a .mi . e
if e a m.-
27_ e ,i K i i
u
k i. ,,
i
- mem M _ .x %
NOJ: '
M0"J'"
' " i i
X' .
.
..
h. K_ wd
-
'
3_".,l /. - .
4- i , ,
LEGEND
-
a.a,um w / o
a ~
23fAltJr Kid naumt EkN L'BDD4
[M -
a'=*=- ino._,xw.
xi 5 """ " ., pgc vuysgi
' M p.4-n/uor name s arwmi sata to huan
.
neau **oa 2 === n=r nac
.m- eno, c un oo
949_7 -
" W a u- ggry, igg,g;g.gn,ma
-
g== M sawuoi no.o . .wmi au re nu.==
,* "'-. rw = m. aua oun anoiin e u w
i i a r .r.,,,ua o .u
..JgNNd$s
4% "I zw.- en aus a ==mo
1AIKk
-./ues ou.sp?JeG
o e 3wwruiaure
MAXhag
u g - - - - - - -
n====
fr.arX_
apraas
igp,tiTURFa y "a"va*s'""*""""'***'"
'X' CARD ' ' ' .m.
Rased haasp
.
425768-C
Also Avadable Ori
Aperture Card O A1
- - _ - _ -_------
-
QO D
-
Ui4Dl --
.
}}