IR 05000361/1993027
| ML20058K711 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 11/24/1993 |
| From: | Miller L, Narbut P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML20058K699 | List: |
| References | |
| 50-361-93-27, 50-362-93-27, NUDOCS 9312160014 | |
| Download: ML20058K711 (50) | |
Text
{{#Wiki_filter:- __- .-. - . -. - - i - - . . . - . - - _ ~ -{ . - . ' ~ ~ - U.S. NUCL' EAR REGULATORY. COMMISSION - - ~
. ~ REGION Y - - . - - . , , . . - Report Nos.
50-361/93-27, 50-362/93-27 ' _ , ' Docket Nos.
50-361, 50-362 ', License Nos.
NPF-10, NPF-15 Licensee: Southern California Edison Company
Irvine Operations Center > 23 Parker Street - - Irvine, California 92718 . . . . . . _ . Facility Name: San Onofre Units 2 and 3 _ _ ' Inspection At: - San 0nofre, San Clemente, California.
~ _ , Inspection Conducted: September 20 through October 22, 1993 ~ I
Inspectors: P. Narbut Regional Team _ Leader D. Acker - Assistant Team Leader ' C. Clark Reactor Inspector - N. Mamish Reactor Inspector , J. Mel Ti Reactor Inspector C. Phillips Resident Inspector (LaSalle) Contractor: D. Prevatte _ - . ~ U
Submitted by: - P. P. Narbut, Team Leader Date, . N/ 2-Y/($ ~ Approved by: Date L. F. Miller Jr., Chieff I , Reactor Safety BranV., _ M Summary: ' Inspection on September 20 - October 22. 1993 (Report Nos. 50-361. 362/92-27) Areas inspected: An announced team inspection of San Onofre Nuclear Generating Station (SONGS) testing areas. The areas examined were post-modification, post-maintenance, technical specification surveillance, inservice test (IST), and predictive maintenance testing.
Also', the testing
1
9312160014 931124 ' gDR ADOCK 05000361 i PDR ' - - . -.. -.- -. - -.. . ----
. - . , - - _ - . - .. ~ ' - - . _ ~ ~. . . . . _ , ' associated with-Generic Letter 89 '13, "Se[vice Water System Probfems Affecting - . ~ ' Safety-Related. Equipment," was examined. NRC Inspection Manual Chapters .61725, 62703, 37700, 40500,-73756, and Temporary Ipstruction 2515/118 were _ used for guidance.
. Results of Inspection and General Conclusions
In general, the testing areas examined were performed in a manner which adequately demonstrated system and component operability.
The identified strengths were , Strengths were noted in the testing programs.
i that the engineering organization was strongly involved.in. establishing . testing requirements, the pre-job briefs held before tests.were thorough and detailed, quality assurance engineers' surveillances of test activities were detailed and the interactions with ~the audited. organizations were ob~ served to.
be in depth.
- After discussions with the licensee regarding the identif.ied violations, the ' ~
inspectors developed a concern regarding potential problems in the . implementation of the licensee's programs for procedure strealining and for The licensee was urged to exercise caution
verification of. work completion.
- and discretion in their efforts to streamline procedures so that important l procedure steps with cautions and verification signat'ures were-not eliminated.
t The inspection noted that the maintenance department generally had well written and detailed maintenance procedures, but maintenance personnel did not always follow the instructions specified in these procedures or change the
! A concern was noted in the cases involving the ! procedure when required.
The failure to follow procedures for making verification signatures.
) personnel performing verification signatures were not always the personnel who - The concern was heightened by j had performed or observed the specified action.
the fact that the maintenance people involved were supe.rvisory.
, Sianificant Safety Matters . ~
_ None Summary of Apparent Violations and Deviations Violations were identified for failure to follow a procedure requirement to . personally verify any work signed for, failure to follow procedure (five examples), inappropriate acceptance criteria in procedures (two examples), procedures that were not appropriate to the circumstances, failure to do a 10 CFR 50.59 safety evaluation, and failure to implement a piping code requirement.
No deviations were identified.
Three open items were identified for followup: An unresolved item concerning the isolation valves for the reactor coolant pump seal cooler which were not in the inservice test program.
(section 4.3.3) -
) . . -. _ .. - . <
' $>M: .- -.. -,
+ w.JA4 .
- - - ,
_ - l - . . . - ^ - _ . . , An ope,n item concerning the salt water cooling _ discharge valves-to the - _ ~ - (section 4.3.5) - - -beach w'hich were not class IE powered.
An open i~ tem regarding a design change which added an eight inch globe The change subsequently t valve to the component cooling water system.
! _ required a field change to add a bypass line to provide sufficient flow.
' (section 4.3.6) i One open item was closed: . 1 Followup item 50-361/93-13-02 regarding inservice test pump flow rate l
reference flow values.
(section 3.4.3.1)
- Safety issues Manaaement System (SIMS) Items: None , I . . _ - . ~
. . i ' . w - ! .
! i l ! -
) , -
- _
.
i . .
- l
, $ . i
1 ! ! t . . . -.. . - -. _ - .-. -.. - -... .. --,.- -
... . - - - -_ _ -. , - _ i - ' _
. . s - l _ -; _ U.S. NUCLEAR REGl[LATORY COM!ilSSION ~ ' - -
REGION V
- - l _ Report Nos.
50-361/93-27, 50-362/93-27 i Docket Nos.
50-361, 50-362 l License Nos.
NPF-10, NPF-15 Licensee: Southern California Edison Company Irvine Operations Center 23 Parker Streei , -- Irvine, California 92718 ~ Facility Name: San Onofre Units 2 and 3 ~ , Inspection At: San Onofre, San Clemente, California.
, Inspection Conducted: September 20 through October 22, 1993 i Inspectors: P. Narbut Regional Team _ Leader D. Acker Assistant Team Leader C. Clark Reactor Inspector , ! N. Mamish Reactor Inspector J. Melfi Reactor Inspector
C. Phillips Resident Inspector (LaSalle)
P Contractor: D. Prevatte l - U!b!
Submitted by: I
P. P. Narbut, Team Leader Date , l A Approved by: - N/f_$ L. F. Miller Jr., Chfef/ I Date Reactor Safety Bran y Summarv: Jnspection on September 20 - October 22. 1993 (Report Nos. 50-361. 362/92-27) Areas Inspected: An announced team inspection of San Onofre Nuclear Generating Station (SONGS) testing areas.
The areas examined were post-modification, post-maintenance, technical specification surveillance, inservice test (IST), and predictive maintenance testing. Also~, the testing '
1 _ _ _ _, _ -. . ._-
-
- _, . - -
' ~ associated with-Generic Letter 89-13, " Service ater System Problems Affecting , Safety-Related Equipment," was examined.
14RC Inspection Manual Chapters 61726, 62703, 37700, 40500, 73756, and Temporary Instruction 2515/118 were used for guidance.
- Results of Inspection and General Conclusions , ! In general, the testing areas examined were performed in a manner which
adequately demonstrated system and component operability.
. Strengths were noted in the testing programs. The identified strengths were , that the engineering organization was strongly involved in. establishing testing requirements, the pre-job briefs held before tests were thorough and detailed, quality assurance engineers' surveillances of test activities were detailed and the interactions with the audited organizations were observed to ~ be in depth.
After discussions with the licensee regarding the identif.ied violations, the inspectors developed a concern regarding potential problems in the , implementation of the licensee's programs for procedure streamlining and for ' verification of work completion. The licensee was urged to exercise caution and discretion in their efforts to streamline procedures so that important procedure steps with cautions and verification signatures were not eliminated.
, ! The inspection noted that the maintenance department generally had well written and detailed maintenance procedures, but maintenance personnel did not i
always follow the instructions specified in these procedures or change the procedure when required. A concern was noted in the cases involving the ' failure to follow procedures for making verification signatures.
The personnel performing verification signatures were not always the personnel who had performed or observed the specified action. The concern was heightened by i ' the fact that the maintenance people involved were supervisory.
J Sionificant Safety Matters lione Summary of Apparent Violations and Deviations Violations were identified for failure to follow a procedure requirement to personally verify any work sioned for, failure to follow procedure (five examples), inappropriate acceptance criteria in procedures (two examples), procedures that were not appropriate to the circumstances, failure to do a 10 - CFR 50.59 safety evaluation, and failure to implement a piping code l requirement.
fio deviations were identified.
, Three open items were identified for followup: An unresolved item concerning the isclation valves for the reactor ' coolant pump seal cooler which were not in the inservice test program.
(section 4.3.3) .
. . ... - -, -- -. ..... . .. - - rw - i .
-
' h _ .- An open item concerning the salt water cooling. discharge valves to the ~ , '- beach which were not class IE powered.
(section 4.3.5) - i - i ~ I An open item regarding a design change which added an eight inch globe valve to the component cooling water system. The change subsequently
required a field change to add a bypass line to provide sufficient flow.
j
(section 4.3.6) , One open item was closed: , ! Followup item 50-361/93-13-02 regarding inservice test pump flow rate } l reference flow values.
(section 3.4.3.1) ! n . Safety Issues Manaaement System (SIMS) Items: None . I - .
~ - . ,
i l ! - !
! , i - , i ! !
l ! ! i ( I ! i
- !
i , ,- em - ,- - .., ,,. -. - . --
__ __ _ . . _.. _ _ _ _ _ _._ _ _ _ ~ . _ w - .- . _ _
- ~ Table of Contents - .\\ - .
1 - 1.0 Executive Summary......................... _ 2.0 Introduction...................',........
< ' t
l 2.~ 1 Inspection Scope....................... '
2.2 Sampl e Sel ecti on....................... 2.3 Persons Contacted-......................
, . t
' 3.0 Test Programs........................... - 3.1 Post-modification Testing..................
' 3.i.1TestProgramDescription
. ...............
3.1.2 Sample and Criteria.................. '
3.1.3 Findings ........................ . 3.1.3.1 - Test Equipment
.............. 3.1.3.2 Agastat Relay Testing........... . 8 3.1.3.3 Certification of Completion of i Construction Work Orders (CW0s)......
~ 3.1.3.4 Modification of Construction Work Orders
. , 3.1. 4 Concl usi on s...... ;...............
i 3.2 Post-maintenance Testing..................
-
3.2.1 Test Program Description
! ............... 3.2.2 Sample and Criteria..................
' ,
3.2.3 Findings ....................... 3.2.3.1 Failure to Follow Procedure to Verify Pump to , ,
Motor Clearance..................
! 3.2.3.2 Inappropriate Acceptance Criteria for Motor
.
Rotation Check , .................
3.2.3.3 Failure to Follow Procedure to Personally-
Observe Work .................. 3.2.3.4 Failure to Follow Procedure for Packing Gland ~ Leakoff.....................
I 3.2. 4 Concl usi ons......................
l . !
' 3' Surveillance Testing ....................
16 l 3.3.1 Sample and Criteria ................. 3.3.2 Findings.......................
l 3.3.2.1 Review of Completed Surveillance Test Data
i .. 3.3.2.2 Review of Technical Specification Test.
-;
[ Scheduling ................... 3.3.2.3 Review of LERs and NCRs
........... 3.3.2.4 Observations of Surveillance Tests
t .....
< 3.3.3 Conclusions .....................
i
- i
i f -
i
. .. - - -..-.- .~ -- -- - -.. . . _.. ~ > - _ , . ' - . , -
_ ' - 19 ~ ! Inserv ce Testing (IST)...........,....... ! i - 3.4 . -
[ ' 3.4.1 Test Program Description ...............
3.4.2 Sample and Criteria........~.. ....... 3.4.3 Findings ................~.......
l
l 3.4.3.1 Previous Open Items.............. '
3.4.4 Concl usi ons......................
1 3.5 Predictive Maintenance Testing ...............
22-l 3.5.1 Sample and Criteria .................
l 3.5.2 Findings....................... 3.5.2.1 Vibration Monitoring.............
. 3.5.2.2 Failure To Follow Procedure for an Oil ! -Sample..................... . 23~ !
3.5.3 Conclusions .....................
24 ! 3.6 Generic Letter (GL) 89_13 Testing.............. ,
3.6.1 Test Program Description
................
3. 6. 2 S ampl e and Cri t eri a..................
1 3.6.3 Findings ..................-.....
l 3.6.3.1 Piping Inspections.............. 3.6.3.2 Heat Exchanger Tests.............
3.6.3.3. Failure to Follow Procedure for Test .: Completion Verification Signatures
l , ....... ' 3.6.3.4 Failure to Perform a 10 CFR 50.59 Evaluation ! Upon Isolating the CCW Heat Exchanger Vent
r V al v e......................
l 3. 6. 4 Concl u si ons......................
l 4.0 Design Basis Verification in Testing ................ Objective.......................'... 28' 4.1
4.2 Sample and Criteria..................... , .
4.3 Findings ..........................
4.3.1 Incorrect Acceptance Criteria References -..... .. 4.3.2 Inadequate CCW Check Valve Testing Acceptance !
Criteria .................... .. 4.3.3 Inservice Testing of Reactor Coolant Pump Seal' Water l 31-l Heat Exchanger Isolation Valves........... -. 4.3.4 Auxiliary. Feedwater Gravity Lube Oil System
( Discrepancy...................... 4.3.5 Saltwater Cooling-System Beach Return Valve Power i ,
l Supplies ............_...........
4.3.6 Improper Sizing of Containment Cooler Throttling l'
Valves-........................ 4.3.7 Surveillance Testing of Air Operated Valves......
l
4.4 Conclusions......................... ' '
'
, ) i > , . - -.,, .._ , ._ ., -. e- ,,,.5 - ,
.-._. _ - .. . . _. -. - . .. .. - . - . - - _ ~ , _ i - -
_ _ , - . i - _ 5.0 - Quality Assurance Involvement and Effectiveness.......... 35_ ~ ! _ -
!
l 5.1 Sample and Criteria ..................... , , ~
5.2 Details ........................... 5.2.1 Quality Assurance Audits...............
! 5.2.2 Effectiveness of Corrective Actions
......... 5.2.3 QA Auditors' Performan,ce...............
, > i '39 5.3 Conclusions .........................
! 6.0 Other Observations .......................... i . 6.1 Relief Valve not-Installed per Code Requirements
_ ...... i 6.2 Relief Valve Test Procedure Enhancement. 40- ' ...........
~ i 6.3 Adequacy of Salt Water Cooling (SWC) Low Flow Alarm - i
} Setpoint .......................... > ).0 General Conclusions
........................ .
8.0 Exit Meeting..........,.................
l
l l ! !
_ i - i ' i i i l l
$ ! ^I .
i i s m.
' . . . . . .
_- -- - - -.- . .- . .-. _ . - - , _ '\\ . ' , ' . DETAILS ~ 1.0 Executive Summary , A team inspection was conducted at the San Onofre fluclear Generating Station ~ (S0!iGS). The inspection was conducted on-site for three separate weeks..The l team consisted of a team leader, four inspectors from Region V, one inspector j ~ from Region III, and one contractor.
, The purpose of the inspection was to assess the licensee's testing programs to i verify that the programs provided adequate test verification that plant system performance parameters were being met or maintained.
i f The inspection was generally performance based and examined six testing areas: post-modification testing, post-maintenance testing, technical specification
testing, inservice testing (IST),- Generic Letter 89-13 related testing, and predictive maintenance testing.
Important systems-and components were i selected for examination using generic and plant specific probabilistic risk l assessment (PRA) data.
The various tests associated with those components were then examined.
l The testing areas examined appeared to be performed in a manner which_provided generally adequate assurance of system operability.
Strengths were noted in ' the testing programs. The identified strengths were that the engineering
organization was strongly involved in establishing testing requirements, the i pre-job briefs held before tests were thorough and detailed, quality assurance
engineers' surveillances of test activities were detailed and interactions i with the audited organizations were observed to be in depth.
Six violations were identified.
The violations-involved a failure to follow a j procedure requirement to personally verify any work signed for, failure to l follow procedure (five examples), inappropriate acceptance criteria in. i procedures (two examples), procedures that were not appropriate to the
circumstances, failure to do a 10 CFR 50.59 safety evaluation, and failure to l implement a piping code requirement.
.f Of particular note were violations that dealt with personnel signing for work j that they did not personally perform, and a failure to follow a procedure for
verifying a motor-pump gap clearance, that led to damage of a salt water i cooling pump. The violations were discussed extensively with the licensee's
middle management to clarify the manager's positions toward procedure detail i and verification of work completion. The managers were urged to exercise caution and discretion in their efforts to streamline procedures so that _j important procedure steps with cautions and verification signatures were not l ~ eliminated.
_; ! The inspection noted that the maintenance department generally had well written and detailed maintenance procedures, but maintenance personnel did not _
! always follow the instructions specified in these procedures or change the procedure when required. A concern was noted in the cases involving the 'i failure to follow procedures for making verification signatures. The 'l l personnel performing verification signatures were not always the personnel who had performed or observed the specified action.
The concern was heightened by i the fact that the maintenance people involved were supervisory.
! l t f -- ...
- . _- _ _ _ - -
- _
-
2.0 Introduction - _ _ Testing demonstrates that plant systems or components meet required performance parameters.
Proper testing is one of the principles of the assurance of quality, along with proper design, construction, operation, and ~ maintenance. The Test Team Inspection is a comprehensive performance-based evaluation of the adequacy of the licensee's testing programs.
2.1 Inspection Scope The areas examined were post-maintenance testing, post-modification testing, technical specification surveillance _ testing, ASME inservice testing, predictive maintenance testing, and testing associated with Generic Letter 89-13.
Important systems and components were selected for examination using . generic and plant specific probabilistic risk assessment ~ (PRA) data.
The tests associated with those components were then-examined.
- The inspection evaluated the licensee's testing programs to. determine if they.
met regulatory requirements and were effectively used to confirm system conditions. The inspection assessed whether the licensee programs tested the proper items, at the proper frequency, for the proper attributes, utilizing adequate methods and measurements. The inspection assessed whether the tests demonstrated that the system design basis was met or maintained.
In addition, the inspection verified that the purpose of testing, as defined'by the technical specifications or the licensee procedure, was fulfilled by the details of the test.
The inspection included observation of field testing or observation of as-left hardware in several cases. The inspection also reviewed records in most The inspection included samples from the mechanical, electrical, and cases.
the instrumentation and control areas.
In areas where problems were indicated, ti.e inspection attempted to identify the reasons for the problems and examine the involvement of the engineering and quality assurance departments.
2.2 Sample Selection The inspectors selected their inspection samples to ensure that important risk significant components were examined. The inspectors utilized the - probabilistic risk information described in the licensee's Individual Plant Examination (IPE), " Individual Plant Examination Report for San Onofre Nuclear Generating Station Units 2 and 3 in Response to Generic Letter 88-20, April 1993." The IPE identified and ranked the contribution to core damage frequency of each significant safety function. The inspectors used Table 3.3-19 which listed dominant. cut sets from the level I PRA model, Table 3.3-20 which listed events by Fussel-Vesely importance, and Table 3.3-22 which listed events by risk increase values. The inspectors also used a list of the top one hundred components for risk significance. The list had been prepared by the licensee's IPE staff for general site us r - - - ! t . - - . . [ . -
- _ _
_ - 3.
l 2.3 Persons Contacted ~ , _ _ , ' Southern California Edison Company
- R. Krieger, Vice President, Nuclear Generating Station l
- R. Rosenblum, Vice President, Nuclear Engineering and Technical
- Support !
- J. Reilly, Manager, Nuclear Engineering & Construction
!
- B. Katz, Manager, Nuclear Oversight R. Waldo, Manager, Operations l
+*L. Cash, Manager, Maintenance i
- D. Breig, Manager, Station Technical i
- M. Wharton, Manager, Nuclear Design Engineering
,
- J. Fee, Assistant Manager, Health Physics
- D. Herbst, Manager, Site Quality Assurance
. ~
- V. Fisher, Plant Superintendent, Units 2/3 ~
_
+*G. Gibson, Supervisor, Onsite Nuclear Licensing .
- D. Irvine, Supervisor, Technical Support W. Strom, Supervisor, Independent Safety Engineering Group
, +*R. Giroux, Engineer, Onsite Nuclear Licensing , i W. Frick, Supervisor, Assessment Engineering
- J. Hirsch, Supervisor, Power Generation
+*R. Joyce, Manager, Maintenance Units 2/3
+*W. Marsh, Manager, Nuclear Regulatory Affairs ' D. Schone, Project Engineer / Supervisor, Design Basis Documents
- R. Browne, Shift Supervisor
+*R. Douglas, Senior Licensing Engineer, Onsite Nuclear Licensing
- J. Curran, Manager, Design Basis Documents
- - '
- J. Thomas, Senior Engineer, Safety Engineering j
j
- T. Elkins, Supervisor, Nuclear Construction
- P. Schoefield, Performance Monitoring Supervisor
- ! !
- T. Yackle, Nuclear Engineering Design Organization
- J. Brower, Supervisor, Plant Coordination, Unit 2
' !
- K. O' Conner, Manager, Construction R. Borden, Supervisor, Maintenance Quality Assurance
! ' G. Grinning, Supervisor, Maintenance Procedures Writers R. Lee, Supervisor, Nuclear Safety Group { ' ! The inspectors interviewed other licensee employees including shift foremen, i > reactor and auxiliary operators, maintenance personnel, plant technicians ~and
engineers, and quality assurance personnel.
, United States Nuclear Reculatory Commission l
- L. Miller, Chief, Reactor Safety Branch, Region V l
- J. Russell, Resident Inspector
.;
- C, Regan, NRC Intern i
Attended the exit meeting on October 22, 1993. l
+ Attended the telephone exit meeting on November 15, 1993.
, , . - . .. ,-. - -.
- . .- -. . . =. - ,, - - ! - - _ _ - -
- I . . _ ..
! i 3.0 Test Programs - ' < - i - ~ ' 3.1 Post-modification Testing ~ When licensees make design' changes and modifications to systems and l ! components, an important ast.ct of the modification process is to verify that the modified component or system will function as intended prior to returning
the component or system to service.
Post-modification testing is important to ! verify that components or systems will perform satisfactorily in service for
all conditions encompassed by the design requirements.
l
3.1.1 Test Program Description - . The' inspector reviewed the licensee's prx:ss for specifying and reviewing . modification tests. The following paragraphs discuss aspects of the _ l ~ licensee's post-modification test process: - -
Procedure S0123-XXIX-2.10, " Design Change Process," required that Nuclear Engineering Design Organization (NEDO) personnel prepare and include testing
criteria for a design change in the design change package. Procedure S0123- ! - l XXIX-2.7, " Test Wori.ing Group," required NEDO, Nuclear Construction Engineering (NCE), and Station Technical personnel to review and approve l safety related modification test procedures.
Procedure S0i23-XXVI-2.5, i " Preparation, Review and Approval of Preoperational Test Procedures," required [ quality assurance (QA) review and approval of safety related modification test >
procedures, and in-proces.s inspection of test activities.
c Normally, NCE personnel directed performance of modification test procedures.
! Procedure 50123-XXVI-2.6, " Review, Evaluation and Approval of Test Results," , ! required that NEDO, NCE and Station Technical personnel review the results of modification tests. When all the design specified testing (component level t and pre-operational testing) was completed, then the modification package was
turned over to operational personnel for operability testing.
[ l Procedure 50123-XX-5, " Work Authorizations," required, for complex safety , related work, that modification operability testing be specified by a retest l
committee, consisting of Operations,- Station Technical and NCE personnel.
l
Changes, except to correct minor editorial errors, required the concurrence of l the organizations which specified the test.
~ '
The inspector concluded that the licensee's process for specifying and I accomplishing modification tests had strong engineering involvement and was a ~ licensee strength, i 3.1.2 Sample and Criteria I The inspector selected recent modifications for review based on the safety ! significance of the modified components and systems.
The inspector reviewed j approximately 70 pre-operational tests, construction work orders with i I associated component tests, and operational tests from the following design l change packages (DCPs): l
h
' __.
. _ - -
- - - - ~. . . - _ i - - h - i, _ - - - i, . _ -
-
, DCP 6742.02 CCW [ Component Cooling Water] Throttling Valves
I ~ , This DCP installed new globe valves to throttle CCW flow to the _ containment emergency coolers to minimize system voiding during postulated system transients.
' DCP 6818.00 Diesel Generator Air Start System This DCP increased the capacity of the air receivers for the emergency diesel generator air system to meet committed cold engine start - requirements. The licensee installed new receivers and hard piped them - , to the existing receivers.
- ~ DCP 6826.00 Main Steam Isolation Signal (MSIS) - Cycling Relay and rSF Test . Switch Modifications . - , This DCP modified MSIS wiring to improve testing of MSIS relay functions , while reducing the possibility of a spurious activation causing a reactor trip. On February 23, 1990, a reactor trip occurred during _
performance of MSIS relay testing, due to an existing failed component.
Test relays were added to preclude the type of component failure which , occurred on February 23, 1990, from causing another plant trip. The
wiring change did not change the MSIS safety functions.
In addition, a
push to operate test switch was replaced with a two position key operated switch to relieve an operator of having to continually depress the test switch throughout the test.
DCP 6858.00 Low Pressure Safety Injection (LPSI) Pump Seal Upgrades l The DCP modified the LPSI pump and motor and associated CCW piping to , improve accessibility for-replacement of the LPSI pump seal..The-existing system precluded replacement of the pump seal within the 72
hour time requirenent for operability of the LPSI pump. The licensee
had noted' that during certain shutdown cooling operations the LPSI pump i experienced flows significantly below the pump design, leading to
reduced seal life. The DCP allowed seal replacement without motor , removal.
DCP 6863.00 Hardpipe Cross-connect Shutdown Cooling System to Containment ~ Spray Pumps Suction - . The DCP modified the shutdown cooling and containment spray systems to ! ' provide a backup means of cooling the Spent Fuel Pool (SFP).
The modification allowed removal of both trains of the dedicated SFP cooling The system or CCW non-critical loop from service for maintenance.
modification provided a capability to perform shutdown cooling using l containment spray pumps, and to allow removal of both LPSI pumps from service, without the need to off-load the core from the reactor vessel j ! . ~ ! !, l ... _. - -. ... . . -.
'I - -
- - - _ _ - - . _ _ _
DCP 6869.00 Auxiliary Feedwater (AFW) Pump Turbine Overspeed Trip - _.
The DCP was installed to preclude recurring AFW turbine overspeed trips, and water damage and hinge pin wear to system check valves, by minimizing water in the system. The DCP changed the steam tap off point for the AFW piping to be at the top of the relief header, in lieu of the bottom, and made other modifications to enhance the quality of the steam supplied to the AFW -turbine.
The inspector reviewed post-modification test records for these DCPs to verify: . Test acceptance criteria were coniistent with applicable design i requirements.
~ All affected safety functions of new or modified systems and components were tested.
~ Testing was technically sound.
Testing was adequate to the circumstances.
' Testing was properly performed and documented.
Test results were reviewed in a timely manner.
t 3.1.3 Findings ~ The inspector had six findings and observations within the licensee's modification program. Two of the findings, incorrect test equipment and .i inadequate relay setting instructions, involved technical issues.
The remaining findings / observations involved the proper documenting of testing.
I In general, the inspector determined that the licensee's modification tests were technically adequate, provided clear acceptance criteria, were adequately documented, and were reviewed in a timely manner.
3.1.3.1 Test Equipment / Dew Point Indicator . The inspector determined that an insufficiently accurate test instrument was used in a test procedure.
Procedure 503-XXVI-9.6813.0.1, Revision 0, " Emergency Diesels Starting Air System Compressors Performance Test," included a test of the dew point of the starting air compressor output. - Section 8.2.15 of this procedure required that the diesel compressor deliver air with a dew point of equal to or less than 22 degrees Fahrenheit (F) at normal operating pressure of 185 to 200 pounds per square inch gage (PSIG). The dew point recorded for this step was 22 degrees F.
Section 8.2.15 also required that the meter and-test , '
equipment (M&TE) used for this step be recorded in Section 6.0.
L _ . . . _ .. . _... - _ _ .,
_ _ ._ - _ _. -. _.. _ _ _ _ _ ~ . - _ . - . . _ . - _ -
Section 6.0 of Procedure 503-XXVI-9.6818.0.1 required that the accuracy ~ of the dew point indicator be plus or minus one degree F.
_ The inspector determined that the equipment identification number recorded in Section 6.0 of Pr~ocedure S03-XXVI-9.6818.0.1 was for a Shaw dew point indicator with an accuracy of plus or minus four degrees Celsius, in lieu of the required plus or minus one degree F.
The dew point tests for the various compressors were performed in Unit 3 between November 1991 and February 1992.
The inspector noted that with~the more inaccurate test equipment, the-recorded dew point of.22 degrees F could have been outside the test ~ procedure requirements. The inspector discussed the possibility of the' recorded value being outside the procedure requirements with the , _ licensee.
, The licensee reviewed the bases for the 22 degree F dew point requirement and determined that the limiting condition for acceptable performance was a dew point of 26 degrees F.
The licensee obtained the 26 degree F dew point performance standard from NUREG 800, Revision 2, " Standard Review Plan," Section 9.5.6.II.4.j.
This Section noted that - the dew point should be controlled to at least 10 degrees F less than the lowest expected ambient temperature..The licensee noted that their lowest expected ambient temperature was 36 degrees F.
The licensee noted that the inaccuracy of the dew point indicator used could have allowed the actual dew point temperature to have exceeded the _ 26 degree F limit. The licensee checked the calibration records for the ' dew point indicator used and determined that it read the exact required reading during the latest calibration check. The licensee concluded that their limit of a dew point of 26 degrees F had not been exceeded, ~ and that no retesting was required.
The inspector reviewed the licensee's action and agreed that the data supported that the actual dew point was less than 26 degrees F.
However, the inspector noted that failure to use test equipment as specified in Section 6.0 of Procedure S03-XXVI-9.6818.0.1 was an example of a violation involving failure to follow a procedure (50-361,362/93-27-01).
, inconsistent Specification of Test Eouipment Accuracy During reviews of other preoperational test procedures and construction work orders (CW0s), the inspector observed inconsistent and potentially confusing specifications for accuracy of test equipment.
Some procedures listed the required measuring and test equipment (M&TE) accuracy, others required only the actual accuracy be recorded.
In many cases the actual accuracy recorded was "per manufacturer."
Occasionally, several different meters of the same general type, such as voltmeters, would be listed, without specifying which meter was used for which part of the test.
It was not clear to the inspector how the reviewers of the completed procedures could ensure that the proper test
- . , _ _ ! , o ' _ .- - - -
_ _ - .- 8
equipment had been used.
' r ~ , . The licensee The inspector discussed M&TE accuracy with the licensee.
reviewed the information provided by the inspector and concluded that
the procedures for specifying M&TE accuracy could be modified to - , The licensee representative stated
preclude potential M&TE misusage.
that they would modify their process instructions for construction work i orders and preoperational tests by May 1,1994, to clarify M&TE accuracy
requirements. The licensee also noted that they were already in the process of eliminating "per manufacturer" as an acceptable description
of test equipment accuracy. The inspector concluded the licensee's , ' actions appeared to be appropriate for_this matter.
- 3.1.3.2 Agastat Relay Testing The inspector observed modification component testing being performed in Unit 3 in accordance with CWO 93081834000. This CWO required Agastat pneumatic time delay relays be set in accordance with Procedure S0123- , XXVI-6.14, Revision 0, " Circuit and Calibration Tests," Section 8.1.2.
Section 8.1.2 set the delay time based on a one time test.
The inspector noted that Agastat technical literature for their pneumatic time delay relays required the delay be set by, " Record (ing) f as many time delays as required to establish a stable average."
t The inspector asked the technicians performing the work how they set The technicians stated that they normally Agastat time delay relays.
used a different procedure, Procedure 50123-II-11.152, to set-the - ! Agastat relays and that this procedure required the time delay be set based on the average of three separate delay tests in keeping with the ' Agastat technical literature. The technicians set the relays and . verified the setting using the average of three delay tests.
The , inspector considered that the performing of two additional time delay tests was not precluded by the instructions of Procedure 50123-XXVI- ' 6.14.
The inspector considered that use of three delay tests was-adequate to meet the vendor criterion of establishing a stable average.
Procedure 50123-11-11.152 was a site procedure which was applicable to l normal maintenance activities, while Procedure 50123-XXVI-6.14 was a NCE-procedure, applicable to modification testing.
, The inspector concluded that failure of Procedure S0123-XXVI-6.14 to ! require that a stable average time delay be established was a procedure weakness. The licensee issued Temporary Change Notice 0-1 to Procedure 50123-XXVI-6.14 on October 21, 1993 to include guidance to perform three i
tests for Agastat relays. The inspector considered the licensee's i actions appropriate for this item.
, > > ! + . -. - - - -.
. . , . - - - . - . ~ . . - - . . _
, 3.1.3.3 Certification of Completion of Construction Work Orders ~ i (CW0s) ' .The inspector noted an apparent problem with the sequence of signoffs in - CW0s.
. The inspector reviewed a number of CW0s which had a standard Section V which read: t " Final Instructions: 1.
Review the CWO for completion of required signatures, dates, craft #s and data, . - , 2.
Return the CWO work package to the cognizant-FE [ field . engineer] for final review.
, ~ Final instructions performed by - Date - " - The inspector observed that, although all CWO verification signatures had been completed, a number of the CW0s had Section V signatures dated l" prior to other signatures in the CW0s certifying completion of various non-craft work steps, such as engineering and QA verification signatures " for completion of required welding.
The inspector questioned whether the signatures in Section V were in compliance with the CWO instructions.
. l The licensee reviewed the subject CW0s and concluded that Section V was [ intended to read, "1.
Review the CWO for completion of required craft i signatures...." The licensee immediately changed their standard CWO i format for Section V to verify completion of only craft signatures. The ! licensee stated that the CW0s questioned by the inspector had all been properly completed and verified by NCE personnel. Because all the work i i was completed and reviewed, and no verification signaturas were missing, the licensee concluded that the Section V signature statement had not caused any problems in the completion of work and therefore had no ! safety significance. The inspector agreed with the licensee's . ! evaluation and actions.
) 3.1.3.4 Modification of Construction Work Orders ' The inspector discovered two potential problems with the methods used by - , the licensee to change CW0s.
Conflictina Instructions for Work Orders l Procedure 50123-XXIX-2.14, Revision 1, " Construction Work Orders,"
Section 5.13 described conditions where a planner or foreman could make l changes to CW0s and required that, " Pen and ink changes are made by one lining the item to be changed and then initialling and dating the
) change."
The inspector determined that many CW0s had an administrative section which potentially conflicted with Procedure 50123-XXVI-2.14.
During . , _.. _ - -. _,. ._ .. . _.. . . J
. . ..-. . - .. . 7 - -- . _ _ _ __ - . . - - - . _ . e - 10 _ ~ review of CWO component tests, the inspector determined that many of the CW0s had a standard Section I which read: i l " Detailed CWO steps are a guide for technicians and a description of test methods for station technical.
Record-in the work done - section any changes to steps not performed as written.
Step ! changes must be within the scope of the description on this CW0, t but may test any circuit on this CWO in its entirety."
The inspector considered that the words of Section I conflicted with the [ administrative requirements of Procedure 50123-XXIX-2.14 for pen and ink changes. The inspector reviewed Section I with the licensee.
The
f-licensee stated that Section I was only intended to apply to component ' test methods within the authorized work boundary and not to production, -
precautions, and prerequisite steps.
- _ l The licensee changed the wording of Section I to state that only test ' i ~ methods could be changed and documented in the work done section of'a CWO; all other changes required a pen and ink change per Procedure ! , The licensee representative stated that test S0123-XXIX-2.14.
technicians personnel were trained in testing methods and needed to be l l i allowed some flexibility in testing components within the procedure ' ' guidelines and the work permit boundary. The licensee emphasized that craft supervision and engineering personnel reviewed the completed CW0s to ensure their completion and the adequacy of any alternate test , methods used.
t The inspector reviewed the licensee's revised Section I and concluded-that it complied with Procedure S0123-XXIX-2.14.
l Low Pressure Safety Iniection Pumo Testina ' The inspector determined that not all data required by a CWO had been taken. The licensee had evaluated and accepted the missing data but had ! not changed the CWO to delete the requirement to obtain the data.
Construction Work Order 93050248000, Section 3.0 required running low pressure safety injection (LPSI) pump P015 for 100 hours and collecting data. During the process of running the pump for 100 hours only 70 l I hours of data was taken.
n The engineer in charge of the test contacted the cognizant NEDO and l Station Technical engineers and obtained their telephone concurrence
that 70 hours of data was acceptable. The engineer recorded the results !
of his telephone conversations in the work continuation section of the CWO.
The inspector considered that since the procedure required 100 hours of I data, a pen and ink change should have been made to the CWO as required >
by Procedure 50123-XXIX-2.14.
! The inspector discussed this issue with the licensee. The licensee stated that they considered that no pen and ink change was required i l l f - _.
. _. _. - _ . - -
. - - _. -. . - -- . . _ - ! _ ' - .
.
- ~ - . ~ J _
. . because approval of the 70 hours of data was normal dat'a review.' Since j ' the data was determined to be acceptable and no further work was t required, the licensee concluded that no pen r.nd. ink change would have been required. However, the licensee stated that they would make an administrative change to Procedure 50123-XXIX-2.14 to clarify the . ! documenting of discontinued tests by February 1,1994.
The inspector determined that the 100 hour test data was a licensee j requirement and that all required inservice testing of LPSI pump POIS
had been accomplished. The inspector concluded that the licensee l actions were adequate.
. j 3.1.4 Conclusions ! The inspector concluded that the licensee's post-modification testing met the . review criteria of SEction 3.1.2, except for the violation for failure to follow procedure, for test equipment accuracy, discussed above.
Five , additional observations were noted with the licensee's documenting of
completed test results. The licensee committed to resolve these five items.
The inspector considered that the licensee's modification program had strong i engineering involvement and concluded that this involvement was' a licensee j
strength.
i 3.2 Post-maintenance Testing j
i 3.2.1 Test Program Description The inspector examined the licensee's post-maintenance testing program. The - i licensee's Maintenance Department administrative controls for post-maintenance i testing activities were described in Maintenance Procedure 50123-I-1.7, j < l Temporary Change riotice (TCf4) 4-12, * Maintenance Order Preparation, Use And Schedul e. " The requirements for maintenance verification testing were l outlined in Maintenance Procedu're 50123-I-1.25, TCf41-1, " Maintenance
Verification Testing." The licensee's guidance for the post-maintenance i ' testing program was established and described in licensee General Procedure l 5023-XV-1, 2, TCf4 2-9, " Post-maintenance Retest Program." As stated in i procedure 5023-XV-1, the objective of post-maintenance testing was to demonstrate that the malfunction leading to the equipment repair, replacement or adjustment was corrected by the maintenance activity, and that the affected equipment was capable of performing its design function.
3.2.2 Sample and Criteria
. . l The inspector reviewed a list of the top 100 risk-significant components, i generated from the April,1993 individual plant evaluation (IPE) report for
SOfiGS Units 2 and 3.
Additionally the inspector reviewed a September 10, l 1993, list of maintenance orders (MOs) issued in 1991, 1992, and a portion of i
1993 for work on the top 100 risk-significant components.
This information was used to select twelve components for review.
Thirty maintenance orders associated with the twelve components were selected for
review. The maintenance orders were reviewed to verify that: , ! ... . . . _ i
__.
_ -- - - - - - .. _ ~
- - - ! j -
- - - .
-12 _ ! . Any testing specified in the documents was correct p'er applicable.
l
! requirements of the licensee technical specifications, procedures, vendor instructions, American Society of Mechanical Engineers (ASME) Boiler and Pressure Vessel Code (Code), and good engineering judgement.
i The acceptance criteria of the specified tests were technically adequate
per the requirements of the technical specifications, vendor instructions, ASME Code, and good engineering judgement.
Specified tests were, in fact, conducted.
~ Recorded test data for the specified tests met the test-acceptance _
criteria.- 3.2.3 Findings _ - j _ The majority of the maintenance orders (MOs) reviewed had the correct post-maintenance testing specified and performed.
In the majority of M0s reviewed, [ the acceptance criteria identified was technically adequate and the recorded l test data met the acceptance criteria.
Exceptions are detailed below: i L 3.2.3.1 Failure ~ to Follow Procedure to Verify Pump to Motor Clearance j
i s The inspector identified a failure to follow procedure. The procedure
required a check for adequate clearance between an uncoupled motor and t pump prior to energizing the motor.
l . Details ~ A screen' wash header flange leak sprayed saltwater on the motor for
safety related Saltwater Cooling Pump (SWCP) S21413MP113, resulting in j motor failure.
j Maintenance order (MO) 93091669000 was issued to replace the motor. The ! M0 invoked maintenance procedure 5023-I-5.4, TCN 4-3, " Pumps - Saltwater
Cooling Pump Disassembly, Inspection and Assembly " Step 7 of the M0
stated, "Begin motor installation IAW Section 6.7 of procedure S023-I-5.4, Steps 6.7.1 through 6.7.12 and 6.7.16 through 6.7.18."
Step G.7.18 , stated, " Verify coupling adjustment nut has satisfactory clearance to , motor coupling to allow rotation without contact," and required a l " performed by" signature verifying the clearance was satisfactory. The l next step, Step 6.7.19, stated," Bump the motor and check for proper l rotation." A motor bump is the brief application of electrical power -to -
check for proper direction of motor rotation.
l
, The electrician and his supervisor, stated that the uncoupled new motor ' for SWCP MP113 was bumped to verify the direction of motor rotation, without first physically verifying or obtaining a step 6.7.18 signature . that there was adequate clearance to allow rotation of the motor ! coupling without contact.
As a result of inadequate clearance, the motor and pump shaft couplings acted like a screw jack and were jacked
l
, . , . _ . . . -
_ _ _
- together. Significant loads were applied to each component.
The failure to verify adequate clearance prior to bumping the motor was another example of the violation (50-361/93-27-01) for f ailure to follow procedure discussed in section 3.1.3.1.
The licensee subsequently replaced the motor, and performed an evaluation of the loads applied to pump MP113.
The evaluation determined that pump MPll3 was operable. The maintenance department had not completed their root cause evaluation of this event during this inspection.
. , 3.2.3.2 Inappropriate Acceptance Criteria for Motor Rotation Check The inspector identified a failure to specify the appropriate acceptance criteria in a maintenance procedure. The acceptance criteria specified~ in the procedure incorrectly identified the direction of motor rotation for a motor bump test.
Details Motors were removed and replaced on Saltwater Cooling Pumps 521413MPll2 and S31413MP307 in accordance with instructions issued, respectively, in maintenance orders 93051741001 and 93060267000.
Step 5 of MO 93051741001 and step 9 of MD 93060267000 stated in part that, "... Ensure motor is turning in the proper direction.... Complete steps 6.7.19 and 6.7.20 of procedure 5023-I-5.4."
A note prior to step 6.7.19 of procedure 5023-I-5.4, TCl4 4-3, stated " Proper motor rotation is clockwise looking from the top down."
Step 6.7.19 stated " Bump the motor and check for proper rotation," and required a " PERFORMED BY" signature that the acceptance criteria (rotation clockwise) was satisfactory or unsatisfactory.
The completed work packages for MO 93051741001 (for MPll2) and MO 93060267000 (for MP307) included copies of Step 6.7.19 in procedure S023-1-5.4, TCN 4-3, signed off as satisfactory on July 8, 1993, and July 10, 1993, respectively.
The inspector noted that the pump vendor manual specified that the direction of motor rotation was different than that specified in the licensee's procedure.
The Byron Jackson Pump Division (Borg Warner) instruction manual No. S7210 IF-7735 (SCE lio. 4108, S023-405-3A-49-0), was identified by the licensee as applicable for SWCPs MP112 and MP307.
Paragraph 1.9 of this manual stated in part that for the driver (motor), " Rotation, viewed from top, is counter-clockwise." Step 10 of paragraph 3.9 of this manual stated in part that for the driver, "... Viewed from the top, shaft rotates correctly counter-clockwise...." The inspector confirmed that the proper direction of rotation was counter-clockwise by examining SWCPs MPll2 and MP307, and noted both pumps had label plates attached with arrows indicating the direction of pump shaft rotation was counter-clockwise viewed from the top of the shaf __ _ ~ _ _ - _ . w
1 - ~ i Further, the inspector observed the direction of actual hotor rotation ~ for SWCPs 521413MP112 and S31413MP307 and verified that both motors were rotating in a counter-clockwise direction (as viewed from the top of the motor).
- The incorrect acceptance criteria (motor rotates clockwise versus counter-clockwise) identified in step 6.7.19 of the procedure, was an example of inappropriate acceptance criteria (Violation 50-361/93-27-02).
The licensee stated they were preparing a TCN to procedure S023-1-5.4, to provide appropriate acceptance criteria for the. direction of_ motor rotation.
.
3.2.3.3 Failure to Follow Procedure to Personally Observe Work .
The inspector identified a failure to follow procedure. The procedure required the verification signature for a motor rotation check be made by the person who performed the work. However, in two instances, the signatures were entered by machinist supervisors who had not personally performed the rotation. Both machinist supervisors stated that their signatures were based on verbal information from an electrical supervisor.
In both cases, the motors did not rotate in the direction specified by the procedure.
Details Motors were removed and replaced on Saltwater Cooling Pumps 521413MPll2 and S31413MP307 in accordance with instructions issued, respectively, in maintenance orders 93051741001 and 93060267000.
Step 5 of M0 93051741001 and step 9 of MD 93060267000 stated in part that, "... Ensure motor is turning in the proper direction.... Complete steps 6.7.19 and 6.7.20 of procedure 5023-I-5.4."
Step 6.7.19 stated " Bump the motor and check for proper rotation," and required a " PERFORMED BY" signature that the acceptance criteria (rotation clockwise) was satisfactory or unsatisfactory.
Administrative procedure 50123-VI-0.9, Rev. 3, " Author's Guide for the Preparation of Orders, Procedures and Instructions," Attachment 7, requires that the signature of an individual who performs a verification for a step designated as " PERFORMED BY" be "The signature / initials of one who personally conducted an operation identified by the document."
The completed work packages for MD 93051741001 and MO 93060267000 included copies of Step 6.7.19 in procedure 5023-I-5.4, TCN 4-3, which were signed off as satisfactory by machinist supervisory personnel on July 8,1993, and July 10, 1993, respectively, for SWCPs MPil2 and MP307. In both cases, however, as described earlier in this section, the motors did not rotate in the direction specified by the procedure.
On November 8, 1993, in telephone discussions with the inspector, the _ _. _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _
. - _ - ~ . .- _ , . _- ~ - . . . '
- - machinist supervisory personnel who made the step 6.7.19 signatures stated that they had not actually observed the actual rotation of the ' subject motors.
They stated that their signatures were made based on verbal information received directly from electrical supervisory . personnel.
- -!
The failure to personally observe the work for which mechanic supervisors made verification signatures is considered a violation for ! failure to follow procedures.
(violation 50-361/93-27-03) 3.2.3.4 Failure to Follow Procedure for Packing Gland Leakoff The inspector ~ identified a failure to follow procedure. The diesel fuel i
transfer pump packing adjustment procedure provided an acceptance criteria for a packing gland _ leakoff rate, required after packing ~ ! adjustment. Work records showed that the packing leakoff rates for two , pumps, did not-satisfy this acceptance criteria.
- _ Details Maintenance Orders 93032292001 and 93032293001 were issued, , respectively, 'to replace the pump shaft packing in Diesel Fuel Transfer Pumps (DFTPs) S22421MP094 and S22421MP095. Step 5 of these M0s stated, '" Repack pump I.A.W. Section 6.2 of 5023-I-5.31."
Step 6.2.16 of the procedure stated, " Perform Section 6.1 of this procedure." Procedure Section 6.1, " Packing Adjustment" provided the following instructions ! for pump packing adjustment: Step 6'.l.3, stated, "If the leakoff rate does not meet the
sp'ecification listed in Attachment 2, repeat Steps 6.1.1 through
' 6.1.2 until the proper leakoff rate is obtained." Attachment 2 of procedure 5023-I-5.31, Rev. 3, specified a packing gland leakoff l rate of 40-60 drops per minute for DFTPs P093-P096, and stated in ' pa.rt that, "...by no means shall the leakoff rate be less than what is specified...." Step 6.1.4, stated, " Record the final leakoff rate." A space was
provided to record a final leakoff rate in " drops per minute."
Step 6.1.5, stated, "Section 6.1, Packing Adjustment, completed
satisfactorily." A verification signature space was provided.
The completed work package for maintenance order 93032292001 (pump S2241MP094) had a final approval signature dated July 26, 1993. The work package included a copy of Step 6.1.4 of procedure S023-I-5.31 with a recorded final leakoff rate of "less than 30 drops per minute," which did not meet the acceptance criteria. Step 6.1.5 was improperly signed ' off as satisfactory on July 2,1993.
The completed work package for maintenance order 93032293001 (pump S2241MP095) had a final approval signature dated July 26, 1993. The . h " ,..
~ _ _ - _ - , ~
. -.
- _- - . i work package included a copy of Step 6.1.4 of procedure 5023-I-5.31~ with a recorded _ final leakoff rate of'"3 drops per minute." Step 6.1.5 was improperly signed off as satisfactory on July 2, 1993.
The licensee stated they would perform an evaluation of the above identified "as left" packing gland' leakoff rates.
t The final packing gland leakoff rates (30 and 3 drops per minute) ' documented in the closure packages for the maintenance orders identified above, did not meet the acceptable packing gland leakoff rate (40-60 drops per minute) specified in Attachment 2 of procedure 5023-I-5.31, Rev. 3, for the specified pumps. These are two additional examples of-the violation (50-361,362/93-27-01) for failure to follow procedure - described in paragraph 3.1.3.1.
- 3.2.4 Conclusions
! The inspector examined thirty maintenance orders and concluded that testing was technically performed adequately in 29 M0s, the exception being the motor
rotation check test of Saltwater Cooling Pump MPil3.
However, there were procedure and documentation problems in 5 of the 30 M0s. Specifically, the , inspector found that maintenance personnel had failed to follow procedures for - work and tests on five safety significant components (SWCP MPil2, MP113, .' MP307, DFTP MP094 and DFTP MP095).
t
From this sample it appeared to the inspector that although the maintenance department generally had well written and detailed maintenance procedures, maintenance personnel di.d not always follow the instructions specified in these procedures or_ change the procedure when required. The inspector was specially concerned in the cases involving the failure to follow procedur< for making verification signatures. The personnel performing verification signatures were not always the personnel who had performed or observed the specified action. The inspector was concerned, since the maintenance people , involved were supervisory.
L 3.3 Surveillance Testing In the area of surveillance testing the inspector reviewed the results from ' previous surveillances to verify that the technical specification requirements ' for the systems and components selected were met.
In addition, the inspector
used interviews, observations, and data review to verify that the surveillances were: , Properly scheduled and conducted within the required frequency
Reviewed in a timely manner
Adequate for the circumstances
Properly performed
, 3.3.1 Sample and Criteria l The inspector reviewed 60 surveillance test records associated with the i ' < k e-e , - ~ ~.
, . ~ _ - _ _ _ _ - . ~ - . _ _ _ -
- _ _ following systems and components: ~ _ 125 VDC Batteries
Auxiliary feedwater -
Diesel Generators
Component Cooling Water
Emergency Feedwater Actuation Signals
In general, the surveillances selected were the last three conducted for each system, for each unit. The inspector interviewed schedulers and reviewed technical specification surveillance schedules for the Electrical Maintenance - and the Operations Departments to verify that there was an ongoing program to ensure that the surveillances were performed on time. The inspector also - reviewed the Licensee Event Reports (LERs) for Units 2 and 3 for the previous year and 6 Nonconformance Reports (NCRs) for the auxiliary feedwater system to determine if equipment failures should have resulted in increased frequency of testing. The inspector observed the performance of four surveillances to assess the use and adequacy of procedures._ 3.3.2 Findings . 3.3.2.1 Review of Completed Surveillance Test Data The inspector verified through the review of completed surveillances that the testing performed met the technical specification requirements, was completed within the required technical specification periodicity, and was reviewed with proper disposition of test results. The inspector verified that the surveillances reviewed were performed correctly with the exception of the 'following observations of minor administrative . ! errors of minimal safety significance from the surveillance records reviewed: l S0123-I-2.3, "125 VDC Battery Inspection," TCN 1-5, performed on
May 12, 1993, step 6.10.2 was marked N/A. The step required the inspection of the battery post " plastic nuts" for leakage. The plastic nuts were the battery post seals for the older battery
' style. New batteries had been irstalled, however, which had battery post seals made of rubber. The change in post material had been noted by the electrician in the notes at the beginning of the procedure but the step was marked N/A. Therefore, it was not I possible to determine if the battery post seals had been inspected as was still required. The licensee's procedure for dealing with j the situation was 50123-VI-1.0.3, " Methods of Handling Invalid i Steps / Sections," Revision 1, TCN 1-1, step 6.1.2 which stated, i "Unless specifically directed in an approved procedure or instruction the terms not applicable, N. A. or N/A shall not be , used to indicate invalid steps or sections." The electrician { should not have marked the step N/A.
The inspector noted, - however, that the procedure was corrected in a subsequent change, and correctly called out the post material. The N/A'd step had minimal safety significance. The inspection was required because
__ _ _ . . 'I - - _ _ , - - - . . - . > - - .-
-
- _ . - . leaking post seals could lead to long term battery degradation. A ' failure to perform the inspection one time, assuming it was not inspected, would not have affected battery operability.
S0123-I-4.53, " Battery Equaiize Charge," Revision 0, TCN 0-10, ~
i step 6.3.10 had a typographical error in the acceptance criteria for battery bank terminal voltage.
It read, "All other 125 VDC < Battery Banks: 131.5 (131.9 to 132) VDC."
131.9 should have been ! 131.0. This procedure was used on June 15, 1993, to perform an ! equalizing charge on battery 2B009 and also on July 14, 1993, on battery 2B007. A TCN, to correct the typographical error,.was not ~ generated.
50123-I-1.7, " Maintenance. Order Preparation, Use and - ! i Scheduling," Revision 4, TCN 4-12, step 6.15.1.2.3.2.3.2 states, "If a maintenance procedure is incorrectly written, the i ' foreman / planner shall ensure a Temporary Change Notice (TCN), per S0123-VI-1.0.1 is approved to correct the' procedure prior to ! continuing the work." At the time of the inspection TCN 0-10 was the current version of the procedure and still contained the i ' error. The observation of.the typographical error has little
safety significance since, it was obvious that "131.9" was a wrong
number.
j These observations were identified to the licensee for any actions they
deemed appropriate.
, 3.3.2.2 Review of Technical Specification Test Scheduling
The inspector interviewed schedulers from the electrical maintenance and the operations department's to verify testing was being properly < scheduled on an ongoing basis. Bcth departments had computer software that aute.natically revised the required completion date and the 25
percent allowed extension date when a test completion date was entered.
Both divisions had assigned individuals with trained backup personnel to ensure the data base was updated and reviewed daily.
i ' 3.3.2.3 Review of LERs and NCRs
. ! The inspector reviewed the Licensee Event Reports (LERs) for Units 2 and
3 for the previous year and.six Nonconformance Reports (NCRs) for the ,' auxiliary feedwater system to determine if equipment failures should have resulted in increased frequency of testing. The results of this review did not identify any cases where an increased frequency of
' testing was warranted and not implemented.
3.3.2.4 Observations of Surveillance Tests
! The inspector observed the performance of S023-V-3.4.1, " Auxiliary Feedwater Inservice Pump Test," on September 22, 1993; S023-V-3.4.4, "High Pressure Safety Injection Inservice Pump Test," on September 23, 19P; S023-3-3.5, "CEA (Control Rod Assembly) Monthly Operability Test," on October 6,1993; and 5023-V-3.4.8, " Salt Water Cooling Inservice Pump t r
___ _ . _. _ _.
_ -.. __ __ . - [ . _ i
- - . l - __ _ _ > _ _ ,- , - - ! _ ' -
- ~ Test," on October 7, 1993.
Pre-job briefings conducted by the cognizant engineers were detailed in nature, included individuals involved in the ' test, and specifically stated what actions were expected at different points in the procedure if something went wrong. The inspector considered that the briefings were conducted well.
i 3.3.3 Conclusions ' . The inspector concluded that technical specification surveillance testing was l well managed and administrative controls of the testing were effective. The i
tests sampled met the technical specification requirements and had been scheduled and performed within the periods required by the technical -! specifications. The surveillances sampled had test data dispositioned as l required by licensee procedures. The conduct of the surveillances observed j - was excellent and the pre-job briefings were consi.dered to be a significant ! , strength.
- }
i f 3.4 Inservice Testing (IST) -
- r 3.4.1 Test Program Description Section 4.0.5 of the SONGS Units No.' 2 and No. 3 technical specifications (TS) ~ requires that IST be performed on ASME Code Class 1, 2, and 3 pumps and valves , ' in accordance with Section XI of the ASME Code and applicable addenda. The Code testing requirements are the basis for inservice tests conducted during t the initial 120 months of commercial operation IST interval, which began on August 18, 1983. The licensee's first ten year IST program was required to be i in accordance with Section XI of the ASME Code,1977 Edition through the ! Summer 1979 Addenda.
l ! ' The licensee's first ten year IST program plan interval ended on August 17,
1993. The licensee had not issued their second ten year IST program plan at ! the time of the inspection. The licensee stated that it was their intent to
extend the first ten year inspection interval until three months after the
i completion of the Unit 3 Cycle VII refueling outage, approximately April 1, 1994, as allowed by-Section XI of the ASME Code. The licensee stated that
they were developing a single second ten year IST program plan that would apply to both Units 2 and 3.
The SONGS IST program for Units 2 and 3 ASME Code Class 1, 2 and 3 pums; was } described in Engineering Procedure 5023-V-3.4, TCl No.5-3, " Inservice Testing i of Pumps Program." The SONGS IST program for Units 2 and 3 ASME Code Class 1, ! 2 and 3 valves was described in Engineering Procedure 5023-V-3.5, TCN No. 7-l 29, " Inservice Testing of Valves Program."
l 3.4.2 Sample and Criteria
The licensee's IST program was reviewed in depth during a recent previous
inspection and the inspection results were documented in Inspection Report
Nos. 50-361/93-13 and 50-362/93-13. Because of this inspection the ! inspector's sampling was reduced for this inspection.
, -
, $ ? -., _. _ __
. . - _ _.
._. . _ _. - - _. _ _ _ . ,. - '- > _
- - - l _ ~
~ ~ _
.
- l - ? To ensure current IST was being performed in accordance with the ASME Code
' requirements,- the inspector observed and evaluated portions of six tests performed during the inspection. Test performance and applicable portions of , test procedures for the following selected system.iests were reviewed:
i 22, 1993, test of Auxiliary feedwater Pump S21305MP140 per ! A September '
procedure 5023-V-3.4.1, TCN 6-10. " Auxiliary Feedwater Inservice Pump Test."
r 23, 1993, test of High Pressure Safety Injection Pump A September i
S21204MP018 per procedure S023-V-3.4,4, TCN 6-11, "High Pressure Safety ! Injection Inservice Pump Test."
23, 1993, test of High Pressure Safety Injection System i A September ~
! Check Valves per Attachment 3 of procedure S023-V-3.5.4,-TCN 3-25, ~ l " Inservice Testing of Check Valves (Quarterly Freliuency)."
A October 6, 1993, test of-Charging Pump S3120BMP192 per procedure S023- .
V-3.4.11, TCN 7-2, " Reactor Charging Inservice Pump Test."
A October 7,1993, test of Saltwater Cooling Pump S31413MP114 per S023- !
V-3.4.8, TCH 7-7, " Saltwater Cooling Inservice Pump Test."
A October 7,1993, test of Saltwater Cooling System check valves, per
Attachment 8 of procedure 5023-V-3.5.4, TCN 3-25, " Inservice Testing of j Check Valves (Quarterly Frequency)."
l , l 3.4.3 Findings
, i During the inspector's review of the observed IST, associated procedures, and l records, the following was identified: ! Personnel performing IST observed during this inspection were generally . !
knowledgeable and qualified to perform their assigned functions.
Prior to each test observed, the operations department held a detailed
pretest meeting (tailboard) to familiarize operations personnel with the The meetings the inspector observed were conducted in a ': testing method.
professional manner and appeared to be very beneficial to all personnel > involved in the testing.
i IST work activities observed were performed in accordance with the ll
licensee's approved program and procedures.
! -
i > ! t i l ., e r -., - ,. ,-
_ ,_ __ -
--_ - , _ - - _ _ - ~ _ - . -. _ - ~
_ - 3.4.3.1 Previous Open Items (ClosedL Followup Item 50-361/93-13-02. IST puma reference values ~ The followup item involved the inspector's observation that the licensee did not always perform pump performance tests using the same initial The inspector had reviewed data for the saltwater pump flow rate.
cooling pump and concluded, from the data sets selected, that the-licensee also changed the ASME section XI reference value each time the The reference values, described by ASME Section XI.
test was performed.
IWP-3110, are selected pump performance parameters which are compared from test to test to detect pump degradation.
. . - This inspection The inspector reviewed the SWC data sets cited in Inipection Report 50-361/93-13 and all the other quarterly IST test data sets for the eight Unit 2 and 3 saltwater cooling pumps for a one year period from September 1992 to September 1993. The inspector observed that the licensee did not routinely establish new pump flow. rate reference values but.rather only established new values for situations that warranted new values such as after a pump motor replacement.
The inspector observed,-however, that the tests were not always.
' For performed with the system flow rate set at the reference value.
example, the reference value flow for Unit 3 saltwater cooling pump P114 in November 1992 was 15,000 gpm but the test was run with flow at 16,250 This starting condition does not violate the requirements of the gpm.
ASME Code provided that the other pump parameters are within their acceptance values specified by the Code. However, not setting the initial flow rate to precisely the reference value does detract from the value of the other data, such as pump vibration,. from a trending standpoint. The inspector discussed the fact that flow rates were not ' being set to optimize IST data trending with operations and engineering personnel. Operations personnel indicated that they could set the system flow rate precisely under normal circumstances. Engineering management committed to improve the trendability of pump performance test data by establishing the system flow rates precisely to the reference values whenever possible. A similar opportunity, for enhancing the trendability of air operated valve IST performance data, was identified in section 4.3.7 of this report.
The inspector considered the followup item closed.
3.4.4 Conclusions The inspector observed the performance of six inservice tests and concluded that the observed testing was performed in accordance with requirements and that the tests adequately assessed the operational readiness of the pumps and valves.
A problem with an inadequate procedure for testing the Component Cooling Water
- ! _ - - - _ _ ' - - _ _ - - , .. - -
- - ' pump discharge check valves was. identified in a different section of this ~ report (section 4.3.2).
Additionally, components were ident~ified which were not in the IST program. These included' components which should have been in , the IST program, as identified by QA audits as discussed in section 5.2.1 of this report. Also, the inspection identified that the CCW isolation vahes j for the reactor coolant pump seal water heat exchanger were not~in the IST - ' Further NRC evaluation is required to determine if these valves program.
should be in the program (section 4.3.3).
3.5 Predictive Maintenance Testing - r 3.5.1 Sample and Criteria ~ The inspector reviewed three programs that were associated with predictive maintenance: reliability centered maintenance (RCM), vibration monitoring, and oil sample analysis.
Reliability Centered Maintenance -
. The RCM group designated 29 systems for analysis. Of those,19 had completed analysis.
Seventeen of the 19 had items that were designated for further evaluation.
The inspector reviewed a sample of' items that were designated for further licensee evaluation. The inspector sampled from the Auxiliary Feed ! i Water (AFW) and Component Cooling Water (CCW) systems.
Examples of items i designated for further review by the licensee were the addition of a conditioning monitoring program for large AC motors and the evaluation of additional CCW components for IST program applicability.
, Vibration Monitorina ~ Cognizant engi-zrs were observed collecting data during the following tests: 5023-V-3.4.1, ' Auxiliary Feedwater Inservice Pump Test," on September 22, i 1993; S023-V-3.4.4, "High Pressure Safety Injection Inservice Pump Test," on September 23,1993; S023-3-3.5, and 5023-V-3.4.8, " Salt Water Cooling Inservice Pump Test," on October.7, 1993. The inspector interviewed the test , personnel regarding their knowledge of vibration analysis equipment and l ~ techniques.
Oil Analysis Procram The inspector interviewed the chemistry department personnel responsible for the administration of the oil sampling and analysis program.
In addition, the inspector reviewed oil analysis data reports from the Units 2 and 3 diesel generators and turbine driven auxiliary feedwater pumps.
3.5.2 Findings The reliability centered maintenance program had been implemented for the systems analyzed.
. .
_.. ._ __ . _ . - . ._ . _. _ _ ' ~ ~
_ . _ _ - - . ' s.
- , , ~
-
3.5.2.1 Vibration Monitoring
The inspector discussed the vibration monitoring program with the licensee representatives. The licensee stated that' vibration test data, - obtained during surveillance testing, was recorded and evaluated for trends of potential system degradation. During test observations, the inspector observed that cognizant engineers demonstrated that they were . I well trained in the use of the vibration monitoring data collection
hardware. They also demonstrated that they were knowledgeable in the l use of the vibration monitoring software to detect anomalies in the - recorded data.
, ! 3.5.2.2 Failure To Follow Procedure for an Oil Sample - l . Througn discussions with the licensee, the inspector determined that oil samples were drawn from procedurally specified points by mechanical-j i maintenance. The chemistry department then sent the samples out to a contracted lab for analysis. The a'nalysis results were then logged into a computer data base and forwarded on to the cognizant engineer with ,
recommendations as required.
The inspector reviewed lube oil sample analy' sis data reports for the Unit 2 and 3 turbine driven auxiliary feed water pump bearings and the-l diesel generators for the last five years. The licensee was requested, , but was unable to provide data to show that the Unit 2 turbine driven-l auxiliary feedwater pump outboard turbine bearing samples had been i analyzed. The responsible licensee personnel stated they were unaware ! of the missing data until pointed out by the inspector and stated that
they were unable to explain why the data was missing.
, 50123-III-6.5,"0i1 Sampling and Analysis Program," Revision 0, Attachment 3 required that a-sample be drawn and analyzed annually from,
among others, the Unit 2 AFW pump turbine outboard bearing housing.
In addition, step 6.9.1 of the same procedure states, " Chemistry
! Engineering shall obtain the trending and analyses data, enter it into the database, and provide reports for review by the Cognizant Engineer."
l Step 7.2 states, " Chemistry Engineering shall maintain a file copy of
all oil sample (s) test results for 10 years." The licensee had a repetitive Maintenance Order, 10200657000, which required that a sample be drawn from the outboard bearing housfrig.
j The oil analysis for particle contamination was technically important
because the vendor manual stated that particles greater than 40 microns in size could cause wear and damage to the thrust bearings and particles-greater that 100 microns in size could cause wear and damage to the journal bearings.
Oil changes on the turbine were scheduled based on the oil sample analysis. The inspector determined from records that the oil in the Unit , 2 turbine driven auxiliary feedwater pump turbine was last changed in 1990. The 01.1 change was apparently due to sample results from other - turbine bearings. The licensee stated that the outboard bearing in
l .- -
_... ._u._ ..
m ._.,m
m_.
1.- .e.._._m.. . - w_.. ...u.. . m.
,. ..s . . _ _ _ - - -. _ -
-. f ~
- . , ' question shared an oil sump with other bearings which were sampled.
The apparent failure to sample, analyze, and retain a record of analysis f for the AFW turbine outboard bearing oil was considered an example of
the violation (50-361,362/93-27-01) for failure-to follow procedures i' discussed in section 3.1.3.1 of this report.
! 3.5.3 Conclusions i l The inspector conclLJed that the RCM program appeared generally effective.
- However, based on the apparent procedure violation involving missing oil analysis data for the Unit 2 auxiliary feedwater pump turbine outboard bearing
and the. fact that responsible personnel were unaware that-the data was , missing, the inspector concluded that the controls involved in the
- administration of the oil sample and analysis program required improvement.
j 3.6 Generic Letter (GL) 89-13 Testing - i
_ _ 3.6.1 Test Program Description ,
GL 89-13, " Service Water System Problems Affecting Safety-Related Equipment,"
requested that licensees implement measures to assurb that safety-related ' service water piping and heat exchangers do not adversely degrade due to l l fouling.
The licensee responded to GL 89-13 with letters dated January 26, 1990, and
B
December 18, 1991, describing their program and commitments.
, To assure that the service water system at San Onofre, the Salt' Water Cooling
a ' (SWC) system, would not degrade during operati~on, the licensee implemented l several measures. These measures were:
- Monitoring the Salt Water Cooling / Component Cooling Water (SWC/CCW)
heat exchanger flows and differential pressures, to assess if the heat l exchanger was becoming plugged.. ] l
- Performing periodic heat treatments of the SWC/CCW heat exchanger.
These heat treatments effectively kill marine growth in the heat
, exchanger.
y '
- Inspecting all the SWC piping interiors for piping coating degradation in accordance with procedure 5023-V-2.8, " Salt Water Cooling Piping i
Internal Inspection."
- Performing periodic heat exchanger performance tests (with available shutdown cooling heat loads) and extrapolating the results to verify that _the heat exchangers could perform their heat removal function under
, design basis conditions. The applicable licensee test procedure was l 5023-V-3.25, " Component Cooling Water. Heat Exchanger Testing."
, ! I l ! i ! " _
_. . _ _ _. _ . -.. _.
. _.
- _ _ _ _ _ __._ _ . , _ _ _ _ - . .- _ - _ ~ ~ - __ ' - 25 - 3.6.2 Sample and Criteria , The inspector reviewed the licensee's programs for the inspection of SWC - piping and heat exchanger performance testing. The inspector' reviewed historical documentation associated with the six SWC piping inspections and eleven heat exchanger performance tests.
The licensee's internal piping inspection criteria were to note and repair degradation in the rubber liner and heat exchanger, note locations of biological growth, and inspect all the piping within three refueling cycles (6 years).
. _ The licensee's acceptance criterion for the SWC/CCW heat exchanger tests was the design basis maximum outlet temperature for CCW. The licensee's heat exchanger tests measured the heat removal capacity of their SWC/CCW heat ~ exchangers using the available heat loads during the cooldown for refueling outages. The licensee extrapolated these measured values to. assess if the heat exchangers would be adequate under design basis conditions.
Using these - extrapolated values, and accounting for instrument error, the. licensee verified that the CCW outlet temperature would be below the design basis maximum of 105 F at design basis conditions.
, ~ 3.6.3 Findings 3.6.3.1 Piping Inspections The inspector reviewed documentation for six piping inspections done by the licensee. The inspector found that the licensee performed system piping inspections and identified any torn liner 5, degraded piping or areas of marine growth.
The licensee had not observed any excessive ', piping degradation and had repaired the identified items. Additionally, the licensee ultrasonically tested portions of the SWC pipe to assure ! that the pipe walls were not thinning.
i
3.6.3.2 Heat Exchanger Tests . ! The inspector reviewed eleven SWC/CCW heat exchanger tests.
Eight tests i passed and three tests failed to meet the test acceptance criteria. The failures appeared to be due to conservative test result extrapolation to l design basis values. Visual inspections of the heat exchangers did not reveal significant fouling.
For each failed test, the licensee wrote a lionconformance Report (f4CR). The licensee issued 14CR 93040051 to document a test failure in Unit 3, CCW heat exchanger E001A in cycle 6, (January,1992). The test results showed an extrapolated CCW heat exchanger outlet t'emperature of 105.54 degrees F versus the maximum allowed of 105. The licensee evaluated the heat exchanger as operable based on conservatisms in the extrapolation.
The other two I4CRs (93100046 and 93100047) were written for two failed tests that occurred on October 12, 1993.
In their preliminary review of the test failures, the licensee concluded that a recent procedure revision was part of the reason the tests failed.
The licensee stated - - - . -
-- - . . . - . - - . . t - -. . , _ .
that the revision had added unrealistically conservative assumptions for
data extrapolations. The licensee was continuing to investigate for
other possible causes for the teat failure. The licensee was still evaluating these NCRs at the conclusion of the inspection.
-
3.6.3.3. Failure to Follow Procedure for Test Completion Verification Signatures
During the inspector's review of completed test results, the inspector , found that.both Unit 3 Cycle 6 (January 26,1992) test results did not I have all the required steps signed off. The steps for the removal of
temporary instrumentation and final ' acceptance of the test results by i th_e supervising engineer were unsigned for both heat exchangers tests.
~ The licensee obtained the original documents from their records' , , repository and confirmed that the steps had not been signed. The.. ! documents had been received by the records department on July 8, 1992.
q The inspector verified that the licensee had removed the t.emporary-o -! instrumentation. The responsible supervising engineer stated that he had reviewed the test results but had failed to make the verification signatures due to-an oversight.
. , ~ 10 CFR 50, Appendix B, Criterion V requires that activities affecting l quality be accomplished in accordance with procedures.
. Procedure 5023-V-3.25, TCH 0-2, required verification signatures for > certain completion steps. The following steps did not have completed verification signatures:
- Step 6.2.9 stated (for heat exchanger E001A)i " Verify removal of
. temporary instrumentation and reinstallation of. permanent plant gauges."
l t i
- Step 6.3.9 stated (for heat exchanger E002B): " Verify removal of I
temporary instrumentation and reinstallation of previously removed plant
instrumentation."
~
- Step 6.6 stated: " Provide the completed procedure and trends of data to the Supervising Engineer for review and approval."
l The failure to provide verification signatures for completed work steps in the test procedure for the CCW heat exchangers is another example of the violation (50-361/93-27-01) for failure to follow procedures discussed in section 3.1.3.1.
3.6.3.4 Failure to Perform a 10 CFR 50.59 Evaluation Upon Isolating the CCW Heat Exchanger Vent Valve During the licensee's investigation for possible causes of the Unit 3, cycle 7 CCW heat exchanger test failurcs, the licensee found air entrained in. the outlet waterbox of the SWC side of one heat exchanger (E002B). Due to difficulties with a leaking fitting, the licensee did ,
.
. . ._ _ - - -- .~ . , ! ' _ - - _ _ ~ - - , , ' . ' , ~ .. ! l
not determine if there was air on the inlet side of the-E002B heat i exchanger.
If air were present through the heat exchanger at the level , observed at the outlet waterbox, about 5 per cent of the heat exchanger tubes would not have had saltwater flow through them.
_
- The licensee did not find air in the other CCW heat exchanger (E001A) water box outlet. The licensee did find a small air pocket on the inlet j side but the pocket was well above the level of the heat exchanger
tubes.
The licensee stated that the air in the heat exchanger.was due to ! ! operations having closed the isolation valve to the heat exchanger's t automatic vent valve, Valve W544, on January 27, 1993, without having i provided for periodic manual venting. The licensee stated the automatic ' l vent valve had been isolated due to excessive leakage.
The isolation ~ ! had been authorized by the shift supervisor and upper operations
management in accordance with procedure S0123-0-23, " Abnormal System Alignments." The licensee did not perform a 10 CFR 50.59 safety- . ! evaluation when the automatic vent valve was isolated.
10 CFR 50.59 states that the holder of a license may make changes in the
facility as described in the safety analysis report, unless the proposed change involves an unreviewed safety question.
It states further that t the licensee shall maintain records of changes in the facility, to the - l ! extent that these changes constitute changes in % e facility as described in the safety analysis report, and that these records must , i include a written safety evaluation which provides the bases for the determination that the change does not involve an unreviewed safety question.
. , ' The San Onofre Final Safety Analysis Report (FSAR), Revision 9, Figure
' 9.2-1, "P&I Diagram Component Cooling Water System Heat Exchangers
System 1203, Drawing 40127CS03 Revision 14," shows Valve W544, an-I automatic vent valve, for the salt water cooling portion of the j Component Cooling Water (CCW) heat exchanger, E002. The inspector observed that the isolation of the automatic vent valve W544 to the SWC heat exchanger and the subsequent potential air pockets could have reduced the heat removal capacity of the SWC/CCW heat exchanger.
The inspector concluded that the isolation of the automatic vent valve !' - M544 was a change to the facility that could have degraded system performance.
The failure to perform an 10 CFR 50.59 safety evaluation , for a change made to the facility described in the FSAR is an apparent
- !
violation (50-362/93-27-04). i 3.6.4 Conclusions
i The inspector concluded that the licensee's implementation of their
commitments made in response to Generic Letter 89-13 were being properly > implemented. The commitments included inspection for salt water cooling system piping degradation and heat exchanger performance tests.
" , - , _ _, _ _ _ ._ _ I
- . . - - - . _ - _ _ _ ! ~
- _ _ l The inspector noted that three of eleven tests sampled had failed to meet the - acceptance criteria but that tMs appeared to be due to conservative criteria j ~ l for extrapolating test results to design basis conditions.
r The inspector identified.two examples of failure :o follow procedures for ! completion of verification signatures. Additionally, the inspector identified l a violation for failure to perform a 10 CFR 50.59 safety. evaluation when the i automatic vent valve for a CCW heat exchanger was isolated without compensatory measures.
' t 4.0 Design Basis Yerification in Testing 4.1 Objective - ,
- ! The inspector's objective was to verify that the design bases were clear and - complete, and that they were correctly reflected in the test procedures and , ! results.
- ! 4.2 Sample and Criteria Three safety related systems were sampled: the salt water cooling (SWC) i ! system, the component cooling water (CCW) system, and the auxiliary feedwater - .f system (AFW). Additionally, a common supporting system, the instrument air system, was reviewed at its interfaces with these three systems.
! > The inspector determined the design and licensing bases for these systems by I reviewing applicable design and licensing documents, by physical inspection of l the equipment, and by interviews with personnel from the plant technical,. i operations, and the Nuclear Engineering Design Organization (NEDO) l organizations. The inspector also reviewed design and licensing documents
including 25 drawings, two design basis documents,11 calculations,.four j modification packages, four design memorandums, and 10 licensing documents, ! including the applicable FSAR, SER, and Technical Specification sections.
j v The inspector reviewed 16 testing procedures for.these systems to determine if l the design and licensing bases had been correctly reflected. The procedures j included six surveillante test procedures, six inservice test procedures, i three preoperational test procedures, and one engineering test procedure.
l Additionally, the inspector reviewed the results of several completed tests to
verify that they met requirements and were consistent.
l 4.3 Findings In all cases reviewed, except those described in the following sections,-the .j licensee had :orrectly reflected the design bases in the test procedures.
In
the case of the salt water cooling system, the inspector performed extensive { review of the bases for the safety analyses performance limit curves used as i the acceptance criteria in the system IST test procedure, and no discrepancies j were detected.
. l The following seven findings and observations resulted from the design basis ! verification portion of the inspection: la ! ! !
-.. _ . -. . - = _ - _ _ - _ - _ -.. . __.. . . - , ~ ~ - . ~ - , _ _
- I
- , - 29' r - 4.3.1 Incorrect Acceptance Criteria References ' The inspector noted the procedure for performance testing the CCW pumps had.
- incorrect references for the test acceptance criteria.
Specifically, procedure S023-V-3.4.2, Temporary Change Notice (TCN) 7-8, " Component Cooling Water Inservice Pump Test," did not reference the intended pump performance . t curves.
In addition to verifying that the CCW pumps met the IST requirements, this
test procedure also verified that they met the safety analysis minimum ' performance requirements. The erroneous references to acceptance criteria for ~ the safety analysis limits 'were given in _ Attachment 1, Step 6.14, Note 1, which directed the user to Attachments 2 and 6 of Engineering Procedure S023- ' V-3.4, TCN 5-3, " Inservice Testing of Pumps Program" for this information.
However, neither-these attachments nor any other section of the referenced ~ procedure contained the intended information.
Attachment 6 further referred the user to a memorandum from V. Barone to S. Gosselin dated June 9, 1992, but this reference did not contcin the intended CCW pump acceptance criteria.
_ i The responsible licensee representatives stated that drawing 41066, Sheets 1 i through 6, Rev 0, dated March 9, 1993, " Component Cooling Water Pump Tag No.
_ l 2P024 (2P025, 2P026, 3P024, 3P025, 3P026) IST Curves," provided the correct , safety analysis acceptance criteria. These drawings contained the pump vendor-l curves as well as the minimum safety analyses performance curves for each of
the pumps. The licensee :tated that the responsible test engineers knew that
these were the correct acceptance criteria curves even though they were not referred to in the applicable test procedure.
I ' The failure to provide the correct acceptance criteria in the CCW pump performance test procedure is considered another example of the violation (50- -, 361,362/93-27-02) for failure to provide appropriate acceptance criteria in procedures described in section 3.2.3.2.
4.3.2 Inadequate CCW Check Valve Testing Acceptance Criteria _ The inspector noted that the test acceptance criteria for the CCW pump discharge check valves were not appropriate. The acceptance criteria allowed for a very large amount of reverse flow through the closed check valves.
The CCW system was equipped with discharge check valves for each of the CCW pumps to prevent back flow through idle pumps. Three separate test procedures addressed in-service testing of the reverse flow closing function of these i valves: e ,
- Quarterly test procedure, 5023-3-3.30, TCH 7-26, "In-Service Valve Testing, Quarterly," Attachment 19, " Component Cooling Water Check Val ves. " This procedure was used by the plant engineering staff until
. July 1993 when responsibility for this test was transferred to operations.
- Quarterly test procedure, 5023-V-3.5.4, TCN 3-25, " Inservice Testing of Check Valves." Attachment 1, " Component Cooling Water System Check-
, - m ,
_ _ _ _. _ _ ___ . . _ _ __ ._ _ . - -- - _ _.
~ . - _ _ , - - _ . - ~ l '
l . ' Valves." This procedure was used by the plant operations staff after they had taken over the test responsibility.
- P,efueling outage test procedure, 5023-3-3.31.2, TCH 0-14, " Inservice t
Testing of Check Valves (Cold Shutdown Frequency)," Attachment 10 " Component Cooling Water Check Valves."
< . ' All three procedures assessed the back flow prevention function of the pump discharge check valve by looking for back flow across an idle pump. The , discharge and suction pressures of the idle pump were read-and compared, and the presence of differential pressure was considered indicative of back flow due to the check valve failing to close.
~ The inspector considered this methodology to be inadequate to detect
acceptable levels of back flow through the check valves. This was ~ demonstrated by an informal calculation performed by the inspector addressing . i the back flow which would correspond to the possible instrument and i readability errors which could exist with the installed instruments . _ (approximately 2 psi). This flow was found to be approximately 4,500 gpm, which was considered unacceptable by the inspector, in a system designed to provide 10,000 to 15,000 gpm through the CCW heat exchangers using only cne " pump rated at 14,000 gpm design flow.
~ ~ The acceptance criteria in the three procedures were different from each other, but all were considered inappropriate.
Procedure ~S023-3-3.30, contained an acceptance criteria that the idle pump discharge and suction pressure gage readings be "approximately equal", which was considered , insufficiently precise.
Procedures S023-V-3.4.5 and S023-3-3.31.2 contained ! acceptance criteria of 5 psi and 10 psi, respectively, for the differential l i pressures across the idle p.mps. As described above, differential pressures of just 2 psi would provide unacceptable back flow.
The inspector also reviewed actual results of these tests and found several
instances of~ inconsistent data. -In these cases, the idle pump suction pressures were greater than the discharge pressures (opposite direction from the expected differential pressure), some by as much as 2 psi for tests where
other parameters indicated no significant back flow. This provided further indication of the inappropriateness of the acceptance criteria.
. The licensee noted that, although not required by procedure, the closed
function of the pump discharge check valves had been inherently demonstrated i by the pump inservice test procedure results in that the active pump's
performance requirements could not be met if the idle pump discharge check valves were leaking excessively. Therefore, the operability of the check valves was not in question.
However, the preparation, review, and approval of l three separate safety-related procedures containing improper acceptance ' criteria called into question the attention to detail being exercised in the ! ! process for these procedures.
, The failure to provide an adequate procedure for testing the CCW pump discharge check valves was considered a violation for failure to provide r adequate procedures. (Violation 50-351,362/93-27-05) , ! ._
t ~ - . _ _ _ - - , .
_
. 4.3.3 Inservice Testing of Reactor Coolant Pump Seal Water Heat Exchanger l Isolation Valves ' The inspector noted that the isolation valves separating the high' pressure
i portion of the CCW system from the low pressure portion of the system were not in the IST valve test program for testing of the isolation function.
The reactor coolant pump (RCP) seal water coolers were cooled by the non- , critical loop of the CCW system.
For the case of leakage from the high
pressure side of the coolers (reactor pressure) to the low pressure side (CCW pressure), the CCW inlet and outlet piping for the coolers was designed for . reactor. pressure and was provided with high pressure isolation valves.
The high pressure isolation valves were a check valve on the inlet side and a~ motor operated valve on the outlet side for each reactor coolant pump's heat - ' exchanger. These valves performed the safety function ~ of isolation of the high pressure (reactor coolant system design pressure) portion of the CCW system from the low pressure portion (150 psig design pressure). The system design did not provide any other high pressure isolation valves or relief . - devices between the heat exchangers and the critical loop.
. ASME Code 1977, Section XI, Subsection -IWV-3411 requires that valves which perform safety functions be exercised at least once every 3 months, and i Subsection IWV-3522 requires that check valves be exercised to the position required to fulfill their function. The inspector noted and the licensee i acknowledged that the CCW seal water cooling isolation valves were not included in the inservice testing program for valves for the closed or isolation function. The licensee further stated that there was a rationale I for their decision but the inspector was unable to follow this up due to time constraints.
- The licensee's rationale for not including the CCW reactor coolant pump seal
water heat exchanger isolation valves in the IST program is considered an unresolved item. (Unresolved item 50-361,362/93-27-06) .
, 4.3.4 Auxiliary Feedwater Gravity Lube Oil System Discrepancy ' The inspector noted that there were some nozzles on an auxiliary feedwater system gravity lube oil collecting tank that could exceed code allowable stresses in certain situations. The nozzles were not s&fety related.
The auxiliary feedwater (AFW) system is equipped with gravity feed backup lube oil systems for the AFW pumps. These lube oil systems were provided to assure l that pump bearing temperatures did not exceed allowable values in the event of , a steam line break in the AFW pump room.
' , In the event of a steam line break, cool lube oil flows from storage tanks on the building roof to the pump bearings and from there to collection tanks.
l t The lube oil systems are totally closed with no vents or relief valves.
e I
. . . . .- , i _ _ i - - , . _ -, i -
,
Pursuant to discussion with the inspector, the licensee calculated that for the steam line break event, if no system leakage were considei i, the systems > could be pressurized to 6 psig due to the heat up effects. This would over
! pressurize the ASME Code, Section VIII, Division 1 collection tanks which had ' a design pressure of only 3 psig. The licensee calculated that for this case, stresses at the tank nozzles could be approximately 14,000 psi; the Code allowable stress was 10,000 psi.
I Although the allowable stresses in the material could be exceeded for the , steam line break case, the yield s. tress for the material was found to be I 30,000 psi. Therefore, failure of the collection tanks would not be expected.
Even if the tanks were postulated to fail, the worst case consequences would l be spillage of the lube oil on the floor. This would increase the amount of j combustibles in the room.~ However, the level of hazard associated with lube . l oil is relatively low, particularly in the steam filled room environment which , . would be present in this event. Therefore, the safety significance of this ] l finding is considered minimal.
This item was considered an observation and was identified to the licensee for their consideration and action as they deemed appropriate.
! 4.3.5 Saltwater Cooling System Beach Return Valve Power Supplies ^ ' The inspector noted that the electrical power supplies for the saltwater cooling system emergency discharge line beach discharge valves were not Class , IE. Therefore, the valves could not be counted on to be electrically operable
in a design basis event.
Details - i The salt water cooling system is the ultimate heat sink at San Onofre. An i important portion of the saltwater cooling system is not Seismic Category 1.
Specifically, the normal discharge path through the circulating water ! discharge piping to~ the ocean, which is a common path for both trains is not Seismic Category 1.
However, FSAR Section 9.2.1.1.A, describes "An emergency discharge line [which is Seismic Category 1], common to Units 2 and 3, is provided in the event of blockage of the normal discharge line " l , The inspector noted that the isolation valves in the emergency discharge path are required to be opened to make the beach discharge path operable and were not Class IE powered. Therefore, the valves may not be operable from the , control room under accident conditions.
In that event, the valves would have ! to be opened manually using their handwheels. Typically, local manual actions are assumed to require 30 minutes to accomplish. The inspector questioned , whether a delay of 30 minutes was technically appropriate for assumed l combinations of conditions. The licensee stated that some conclusions about the loss of saltwater cooling for 30 minutes could be inferred from an > ' informal analysis they had performed, for the total loss of CCW for one hour concurrent with a loss of coolant accident. The licensee stated that the study showed that the containment design pressure would not be exceeded, and that this, together with other similar existing analyses, indicated that the i , plant could experience loss of saltwater cooling for 30 minutes without , , i e-n
. - , - , - - - _ _ ' . , ' ' -
> exceeding any design safety limits.
[ The inspector reviewed available relevant licensing documer.ts including the l FSAR, Question and & Answer 010.32, Section 9.2.1 of the SER dated February l 1981, and updated FSAR Section 9.2.1.1 dated February 1993. None of these clearly indicated that the lack of Class 1E power for the emergency discharge valves had been considered in the licensing bases. However, in the response to FSAR Question 010.32, the licensee did perform an analysis which showed that the normal non-Seismic Category 1 discharge path would not be fully blocked and a flow path sufficient for salt water cooling flow would remain.
Therefore, the alternative discharge path would not be required.
~ To resolve the matter, the licensee committed to provide a letter to NRR ~ ! stating the complete design bases for the saltwater cooling discharge flow
paths and to provide copies of existing analyses which would indicate the , ability of the plant to experience loss of saltwater cooling without exceeding any design safety limits.
.
This is considered an open item. (Followup Item 50-361,362/93-27-07) . , 4.3.6 Improper Sizing of Containment Cooler Throttling Valves . The inspector noted a design change to the CCW system that required an ! extensive field change to allow the system to function with sufficient flow.
The design change involved the addition of an apparently undersized valve which subsequently required the addition of a bypass line to achieve system . flow.
~ Details An NRC Safety System Functional Inspection (SSFI) performed in 1988 identified that the containment coolers could be susceptible to water hammer for certain loss-of-power events.
Consequently, the licensee instituted two CCW system , modifications.
One of these, DCP 2(3)-6742.02SM, Rev 0, dated July 1, 1991, involved moving the throttling location for the containment coolers from high in the containment to relatively low in the return piping in order to provide i greater assurance that this piping would remain filled for the loss-of-power i event, and therefore reduce the potential for water hammer. The modificathn ! left the existing throttling valves in the system, positioned them full open j
so that they would have minimal throttling effect, and installed new throttling valves just downstream of the return line containment isolation valves. The original throttle valves were 10 inch gate valves; the new valves
selected were 8 inch globe valves.
The inspector's discussions with technical personnel indicated that they had expressed concerns with design in the review stage with regard to the ability of the new valves to pass the required flow of 2,000 gpm. Post-modification
testing revealed that the valves could not pass the required flow. As a corrective action, the licensee issued a revision to the DCP, ASC-1, dated i December 19, 1991, to install 3 inch bypass lines around the new valves to increase the flow area.
j i I .
. - . _ - t _ - , - - . ~ - , , ! , . - - ,, . s* I The licensee appeared to deal with the error responsibly and performed a full
root cause analysis _ documented in Root Cause Evaluation Report 91-032, dated l i December 20, 1991.
Further review of the CCW valve modification, and its implications regarding ! design engineering, is considered an open item. (Followup item 50-361,362/93-27-08) ,
4.3.7 Surveillance Testing of Air Operated Valves ' , The inspector examined surveillance testing of air operated valves and noted two items which could be improved. The first item was that testing was not done at the minimum allowed system pressure. The second item was that the ' required testing was not always done at the same ' starting air pressure which - e resulted in data that was not as useful for trending as it could have been.
i
Details , . I I The saltwater cooling system and the component cooling water system contained numerous air operated valves which were designed to fail to their safe
positions upon loss of non-safety-related instrument air.
Examples included i 2(3)HV-6202, saltwater pump P307A discharge valve, and 2(3)HV-6219, component , cooling water non-critical loop return header isolation valve. These valves were not typical spring actuated valves which returned to their fail-safe , position by spring force upon loss of air.
Instead, the valves were pilot valve controlled air-to-open, air-to-close valves.
Backup air supply i i accumulators provided the motive force for moving the valves to their fail-safe position.
Procedures controlled the pressure in these accumulators and j required the pressure be maintained at greater.than 70 psig.
-
, Surveillance Operating Instruction S023-3-3.30, TCN 7-26, "In-Service Valve - Testing, Quarterly," was the procedure for testing the fail-safe feature of , these valves.
The test timed and trended the valve movements to the fail-safe ' i positions, and verified that the accumulator pressures were maintained above 70 psig.
, j To test the fail-safe feature, the procedure required the normal instrument air supply to be isolated and the pressure to be bled off to simulate a loss of normal instrument air. With this loss of air, the valves were verified to i move to their safe position.
The inspector noted that the procedure did not specify that the accumulator pressure be at the minimum value of 70 psi at the start of the fail-safe test.
Additionally, the licensee stated that they had no documentation that these valves had ever been tested at their minimum control pressure. Also, the licensee stated that they did not have a design basis documented for the minimum system pressure they had selected. The licensee's periodic testing of the fail-safe valves at the as found accumulator pressure met all regulatory requirements and provided good assurance that the valves would function if called upon. The need for a design basis document for the selected minimum system operating pressure is not a regulatory requirement.
-
_- ~ ~ - . - - _, . .
'l _ -{ ~
~ [' ~ _ The lack of a test at minimum allowed system pressure and the lack of a. design ! i basis document were identified to the licensee as observations for their l action as they deemed appropriate.
! Trendino 5023-3-3.30 also required that after the fail-safe tests, the normal l Procedure air supply be restored, and valve stroking to the safe position be timed and l trended. For these tests, the air pressure was not controlled to a fixed i value from one test to the next. The inspector examined the backup air accumulators for several salt water cooling valves and found their pressures . Since the time of the valve stroke appeared
to range from 74 psig to 83 psig.
i to be a function of supply air pressure, the inspector noted that the trending results required by the procedure could be enhanced by using the same pressure
each time the tests were done.
This was identified to the licensee as an
observation for their action as they deemed appropriate. A similar < opportunity, for enhancing the trendability of IST pump performance data, was - ' identified in section 3.4.3.1 of this report.
4.4 Conclusions Generally, the inspector found that the design bases were clear 'and complete, ! ' and that they were correctly reflected in the test procedures and results.
In all areas of testing reviewed, it appeared that tSe design bases were being j properly verified.
The inspector also concluded that both engineering organizations, NEDO and Station Technical, were generally competent, effective and well integrated with each other. The only exception noted was discussed in section 4.3.6 regarding the improper sizing of the CCW throttling valves. However, this problem was raised and adequately analyzed by the licensee's root cause analysis program.
In the area of inservice testing, however, it appeared that the attention to detail in the preparation, review, and approval of procedures could be _ improved as evidenced by the incorrect references for CCW pump performance acceptance values discussed in section 4.3.1 and the inappropriate acceptance. criteria for CCW pump discharge check valves discussed in section 4.3.2.
5.0 Quality ' Assurance Involvement and Effectiveness The inspector examinad the licensee's Quality Assurance (QA) involvement with test activities by interviewing cognizant personnel, reviewing pertinent documents and procedures, conducting facility tours, and accompanying QA auditors on surveillances.
5.1 Sample and Criteria To assess QA's involvement and effectiveness in testing, the insp6ctor examined the licensee's compliance with the license technical specifications, effectiveness of corrective actions, and QA auditors' performance. The
_ __ . _ .-_ _ .. __ , - i ~ - - - . wr ! - , - s ~
- - inspector's evaluation included a review of the following test related- ~ documents: j Licensee procedures and surveillance guidelines-
i i i QA's 1993 master audit schedule
. Eight audits !
Fifteen surveillances , ,
Three Corrective Action Requests (CARS) -
Three Problem Review Reports (PRRs)
,. . Additionally, the inspector held meetings and discussions with various members i ! of the licensee staff to discuss -the above documents, verified the completion - of corrective actions, and accompanied QA auditors on two surveillances
involving a pump performance test of the Unit 3 Component Cooling Water (CCW) system and post-modification testing of the condenser air ejector wide range , gas monitor.
. i j 5.2 Details 5.2.1 Quality Assurance Audits j Technical specification 6.5.3.5.a. requires the licensee to perform audits, at least once per 12 months, that verify unit operations comply with provisions of the technical specifications.
Additionally, technical specification l 6.5.3.5.d. requires the licensee to conduct audits of activities required'by j . the QA program to meet the criteria of 10 CFR 50, Appendix B at least once i every 24 months. The inspector examined a sample of eight audits to verify ! the licensee's compliance with technical specifications 6.5.3.5.a. and l i t 6.5.3.5.d.
The following are summaries and conclusions of the QA audits
reviewed by the inspector: , l Audit SCES-519-92, " Technical Spacification and Topical Quality Assurance Manual Requirements for the Integrated and Local Leak Rate Test Program," was ]
performed to verify that requirements for integrated and local leak rate
testing had been effectively implemented. The audit identified an j administrative error concerning the lack of records indicating that an interlock test was conducted for the Unit 3 personnel and emergency hatches during a Local Leak Rate Test (LLRT). The procedural weakness was perceived . as inattention to detail and was corrected by revising the applicable - procedures to ensure that interlock testing was specifically documented when conducted. The audit, however, concluded that station technical engineering and operations were effeuively perfo-ming the required periodic tests and surveillances.
Audit SCES-521-92, " Cable and Elu .1 Apparatus Monitoring Program," was conducted to verify that the Cabl 1 Electrical Apparatus Monitoring Program l -. . a
_.
- . _ _ _ _ _ . . _.. _. _ ._ _ _ _.. _. ___ . - - _ - _.
, , . - , ' . , - l -
- - ! (CEAMP) utilizing the Electronic Characterization and Diagnostics (ECAD) . ! System 1000 conformed to regulatory and program requirements.
The audit-
identified deficiencies ~ related to program implementation regarding long-term j . i cable monitoring utilizing ECAD.
Additionally, procedural inadequacies were
noted regarding the ECAD cable monitoring program.
I
Audit SCES-539-92, " Operational Surveillance Program " verified that a program
was established to ensure that all applicable surveillance requirements were-l ! appropriately conducted. Additionally, the audit verified that applicable tests, checks, and calibrations were being performed as required by station f technical specifications. No deficiencies were noted relating to the j implementing procedures, surveillance frequencies, or test results.
Audit SCES-544-92, " Inservice Testing Program," was performed as a result of i ' an NRC inspection that identified weaknesses in the Inservice Testing (IST) ~ ' The audit had confirmed that weaknesses existed in the IST program . program.
as follows: , The program did not identify all pumps and valves required to be
included and tested. This was evidenced by the continual identification of components that were not included in the IST program. scope.
' A programmatic inconsistency was noted between the IST valve program !
procedure and the implementing testing procedure for valves in the Units ! 2 and 3 Emergency Diesel Air Start Systems. The program procedure ! listed certain valves be tested at specified frequencies.
However, the , operations procedures that test the diesel generators did not identify [ - or test these valves to meet the IST program requirements.
, , The IST program did not fully implement all of the recommendations of i
. Position 11 of Generic Letter (GL) 89-04, " Guidance on Developing ' Acceptable Iriservice Testing Programs," dated April 3,1993.
Specifically, Position 11 of the Generic letter identified a potential '; generic deficiency concerning the fact that licensees do not always
identified that certain components were not in the San Onofre IST . ! include the necessary equipment in their IST programs. The audit ' program and should have been. The components identified were the , control room chilled water system pumps and valves, and the Unit I boric j acid transfer pumps.
Audit SCES-551-92, "TQAM Chapter 4-G, Test Control," was performed to verify + that the maintenance, operations, station technical, and nuclear engineering ! and design organizations complied with the requirements of the Topical Quality i Assurance Manual -(TQAM) Chapter 4-G, " Test Controls." The audit determined i that nonconformance reports were not always written when test results were i outside the required acceptance limits. The cause of the deficiency had been i due to inconsistent procedural requirements and inconsistent maintenance order j guidance.
Audit SCES-307-93, " Containment Systems," verified the that technical specification requirements had been effectively implemented. The audit, l i ! ! ! - -.,. _ _- . , !
. - . _ _ w - - __
_ - -
however, identified six administrative deficiencies and one item for potential improvement. The administrative errors were related to procedural errors, lack of training records, and the inability to retrieve maintenance orders.
Audit SCES-308-93, " Emergency Core Cooling System, Plant Systems, and Electrical Power Systems," concluded that testing requirements related to the emergency core cooling system, plant systems, and electrical power systems had been fully implemented.
Minor deficiencies were identified during the course of the audit concerning inconsistent diesel generator logs, incomplete records, and procedural inconsistencies.
Audit SCES-313-93, "Bi-Annual Corrective Action Audit With Special Emphasis on Test Activities," reviewed nonconformance reports, problem review reports, licensee event reports, and Regulatory Commitment -Tracking System (RCTS) corrective action _ documentation in the area of testing to ensure adequate corrective action implementation. The audit noted a discrepancy concerning RCTS pending files not being maintained current.
The inspector noted that the findings outlined in audits, surveillances, and other QA related documents appeared to be thorough, probing, and were appropriately categorized. With the exception of Audit SCES-544-92 which - identified weaknesses in the IST program, the inspector did not note indications of specific trends or weaknesses relating to test programs (i.e., discrepancies identifie. by QA were diverse and from different organizations).
The inspector concluded that the audits of testing performed by QA met the requirements of technical specifications 6.5.3.5.a and 6.5.3.5.d.
No - violations or. deviations were identified.
5.2.2 Effectiveness of Corrective Actions The inspector followed up on corrective actions outlined in audit SCES-521-92, " Cable and Electrical Apparatus Monitoring Program," and surveillances SOS-298-92, " Post-maintenance Operability Test of Fire Pump Diesel SA-2301-MP-220," SOS-120-92, " Design Change Package 3-6783 AFW Check Valve Installation Turnover Review," and 505-279-92, " Unit 3 High Pressure Safety Injection Pump Testing."
The inspector noted that corrective actions were appropriate, thorough, and completed in a timely manner. The lack of repetitive problems or deficiencies indicated that corrective actions had been effective. No violations or deviations were identified.
5.2.3 QA Auditors' Performance The inspector accompanied QA auditors on two QA surveillances involving Inservice Testing (IST) of the Component Cooling Water (CCW) system and post-modification testing of the condenser air ejector wide range gas monitor.
Additionally, the ins;ector examined the qualifications of three QA auditors and a lead auditor.
_
. - . - - -. ~ - - _ - _ - , ! _ i ~
- . t The QA auditor performing the component cooling water IST surveillance and the ' f
engineers being audited demonstrated an adequate depth of knowledge regarding the system being tested. The inspector specifically noted that the exchange , of information between the QA auditor and the engineers to be quite detailed.
! The QA auditor identified a problem concerning the lack of hearing protection for the engineers.
l As in the component cooling water IST surveillance, the. inspector noted during i the gas monitor testing that the interaction between the QA auditor and the j nuclear construction engineer was thorough. The testing was halted by the l test engineer when a discrepancy was discovered involving calculations of the isokinetic flow ratios.
The inspector examined the qualifications and certifications records of three QA auditors and one lead auditor. Certification records indicated that QA l
- staff qualifications met or exceeded the requirements of ANSI N45.2.23, ! " Qualifications of Quality Assurance Program Audit Personnel for Nuclear Power l
Plants," and QA Procedure N2.19, " Qualification and Certification of Auditing .l Personnel. " [ 5.3 Conclusions .
The licensee's audits and corrective actions appeared to be probing and ' -l While seveillances varied considerably in scope and detail, q comprehensive.
- identified discrepancies such as procedural and record inconsistencies had
been corrected in a timely manner. The inspector concluded that QA s , j involvement in testing, in the areas reviewed, appeared to be effective in identifying test related deficiencies and in ensuring appropriate corrective j actions were taken in a timely manner.
! .
6.0 Other Observations ! i 6.1 Relief Valve not Installed per Code Requirements i , During a system walk down on September 21, 1993, the inspector noted that Unit
2 Pressure Safety Valve (PSV) 8450 did not have a drain hole on the discharge
line of this safety valve. Valve 2PSV-8450 was the relief valve on the , nitrogen supply line to tank 2T212, a recently installed nitrogen accumulator.
, The nitrogen from tank 2T212 supplied the seismic category I motive force to
operate the main steam atmospheric dump valve (ADV), 2HV-8419. The discharge i
i line from 2PSV-8450 connected horizontally to the relie' valve and had a 90 degree elbow pointing upwards. The piping was in the main steam isolation i , valve enclosure, exposed to the atmosphere and rain intrusion.
l
l Design Change Package (DCP) 6763.00BJ installed valve 2PSV-8450 in early'1992.
DCP 6763.00BJ, section 9, " Quality and Seismic Category," required 2PSV-8450 to be installed in accordance with ANSI /ASME B31.1.
Further, the licensee's Q-list notes that "the ADV actuator pneumatics and associated backup nitrogen system is ANSI B31.1."
ASME B31.1 (1973), section 122.6.2(D) stated that: " Discharge lines from pressure relieving safety devices within the scope of this Code shall be - - - - -. - -.. _
. . . .- - . ~... -. - . - _ - - - - _ o - _ -.
. . ' ~
! i designed to facilitate drainage." and section 122.6.2(F) stated that: ! " Drainage shall tse provided to remove water collected above the safety valve ! ! seat."
! The failure to provide for discharge line drainage in accordance with code ! requirements is considered a violation of 10 CFR 50, Appendix B, Criterion , III, " Design Control," (50-361/93-27-09).
l i After the inspector notified the licensee of his observation, the licensee walked down other relief valves and found an additional seven relief valves I
that did not provide for discharge line drainage..The licensee promptly issued field change notices (FClis) F4433J, F4434J, F4435J, F4436J, F8740M, l F8741M, F8742M, and F8743M, to the affected drawings to provide drain holes.
- The licensee completed the work specified by the FCfis during the inspection l period..The inspector examined three of the eight relief valves and verified ' - that drain holes were installed. The inspector considered the licensee's i actions appropriate.
I 6.2 Relief Valve Test Procedure Enhancement j ! The inspector reviewed the licensee's procedure for setpoint testing of relief l valves, 5023-I-8.88, " Valves - Cold Bench Testing and Calibration of SR and ! NSR Safety Relief Valves." The licensee uses this procedure to verify the ! lift setpoint of relief valves.
ASME code requirements for testing safety-i related relief valves are in Performance Test Code (PTC) 25.3-1976, " Safety l l and Relief Valves."
i The inspector noted that one quantitative criterion contained in the ASME code i
was not incorporated in the licensee's procedure.
Specifically, the code noted that, "the rate of pressure increase for safety relief valves shall be
less than 2.0 pounds per square inch (psi) per second or whatever lesser rate
of pressure increase is necessary to accurately read the set pressure." The ! licensee's procedure was less specific and stated, " slowly raise pressure j until the valve setpoint is reached."
I Subsequent to discussion with the inspector, the licensee issued Temporary Change Notice (TCN) 2-I to procedure S023-I-8.88 to specifically incorporate l the code guidance.
j s The licensee informed the inspector that they considered the rate of pressure
increase was generally at or below the 2.0 psi /second during their relief , valve testing. The inspector considered the licensee's procedure enhancement
! to be sufficient action for this item.
l 6.3 Adequacy of Salt Water Cooling (SWC) Low Flow Alarm Setpoint < . The inspector examined the adequacy of the licensee's SWC control room
' annunciator low flow alarm setpoint, which was 12,000 gallons per minute (gpm).
The licensee stated that they did not have a documented design basis for this setpoint but the minimum pump flow recommended by the vendor was 14,000 gpm.
, ,. - - - - , - -
. . . . - - - -. - . -
. - , _l _ - ! ' t i .
The cognizant system engineers stated that the vendor requirements for flow > were based on long-term (40 years) bearing life, and the pump could operate . for extend periods of time below 14,000 gpm.
, The inspector observed that the system operability curves for the SWC flow and -
temperature showed that about 14,000 gpm was needed if the maximum design i basis inlet temperature was assumed. Further, the operability curves showed l that the flow rate can be much less than 14,000 gpm for the usual seawater
temperatures.
The licensee stated that operators had administrative controls to maintain SWC ] ~ !- flow above 14,000 gpm, and within the operability curve. Operations pe'rsonnel stated that they believed that the alarm setpoint was. set low, at 12,000 gpm, > to avoid nuisance alarms but that the set point was sufficient to provide annunciation in the event of a line break. Since the flow was maintained above 34,000 gpm, the safety significance of the inspector's question regarding the alarm setpoint was low. This ubservation was given to the j licensee for their action as they deemed appropriate.
7.0 General Conclusions - In general, the testing areas examined were performed in a manner which ' adequately demonstrated system and component operability.
The identified strengths were Strengths were noted in the testing programs.
that the engineering organization was strongly involved in establishing l testing requirements, the pre-job briefs held before tests were thorough and detailed, quality assurance engineers' surveillances of test activities were detailed and interactions with the audited organizations were observed to be in depth.
i The inspection identified six violations. The violations involved a failure
to follow a procedure requirement to personally verify any work signed for, l failure to follow procedure (five examples), inappropriate acceptance criteria in procedures (two examples), procedures that were not appropriate to the circumstances, failure to do a 10 CFR 50.59 safety evaluation, and failure to ] i implement piping code requirements.
' . , Of particular note, were violations that dealt with personnel signing for work ! ! that they did not personally perform, and a failure to follow procedure for verifying a gap clearance, that led to damage of a Salt Water Cooling pump.
i The violations were discussed with the licensee's middle management to clarify j the manager's positions toward procedure detail and verification of work
l The managers were urged to exercise caution and discretion in completion.
their efforts to streamline procedures so that important procedure steps with cautions and verification signatures were not eliminated.
3 , - . . - - . . . , }}