IR 05000528/1990038
| ML17305B203 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 11/06/1990 |
| From: | Huey F NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML17305B202 | List: |
| References | |
| 50-528-90-38, 50-529-90-38, 50-530-90-38, NUDOCS 9011270185 | |
| Download: ML17305B203 (38) | |
Text
U.S.
NUCLEAR REGULATORY COMMISSION REGION V
Report Nos.
50-528/90-38, 50-529/90-38, 50-530/90-38 License Nos..
NPF-41, NPF-51, and NPF-74 Licensee:
-... Arizona Nuclear Power Project P.O.
Box 52034 Phoenix, Arizona 85072-2034
, Facility Name:
Palo Verde Nuclear Generating Station Units 1, 2, and
Inspection at:
Palo Verde Site; Wintersburg, Arizona Inspection conducted:
September 10 through October 23, 1990 Inspectors:
M. Miller, Reactor Inspector P. Galon, Reactor Inspector D. Proulx, Reactor Inspector Approved By:
~
~
~
Forrest
.
uey, 1 e Engineering Section
~Summar:
D a
e cygne Inspection during the period September 10 through October 23, 1990 (Report Nos. 50-528/90-38, 50-529/90-38, 50-530/90-38)
Areas Inspected:
A special unannounced inspection by regional based inspectors of the licensee's design, engineering, and associated quality verification activities.
The licensee's quality and root cause of failure programs were inspected to determine the quality of licensee engineering work.
Inspection procedures 30703, 37700, 37702, and 2515/105 were used for guidance in this inspection.
Results:
General Conclusions and Specific Findings:
The licensee's program to identify and resolve nonconforming situ'ations and conditions adverse to quality appears to have improved.
However, areas of weakness in these programs have been identified.
The licensee and inspectors have identified several instances regarding the use of improper work control and engineering documentation.
t Several inaccurate references were identified in engineering and quality procedures.
901127018 901106 PDI>
ADOCK 05000528
PI3C
II
The licensee's root cause evaluation program and implementation appears to be adequate.
Si nificant Safet Matters:
None Summar of Violations and Deviations:
Unit I began initial operation without resolving a material nonconformance of a safety related breaker.
Because the licensee's corrective action appears acceptable, this violation is non-cited.
0 en Items Suomar
.Four open items were closed, three remain ope DETAILS Persons Contacted Arizona Nuclear Power Project (ANPP)
- R. Adney, Unit 3 Plant Manager
- J. Auston, Fire Department Chief
- J. Baxter, Compliance Engineer
- D. Best, Primary Training Instructor
- H. Bieling, Emergency Planning/Fire Protection Manager
.
- M. Czarnylas, Fire Department Deputy Chief
- E. Dotson, Site Nuclear Engineering/Construction Director
- Z. Elawar, Nuclear Safety Department Engineer
- R. Fongamie, Component/Specialty Engineering Supervisor
- R. Fountain, guality Audits and Monitoring Coordinator
- R. Fullmer, guality Audits and Monitoring Hanager
- S. Guthrie, guality Assurance/Control Deputy Director
- K. Johnson, Site Technical Support
- S. Kantor, Part Services Coordinator
- G. Overbeck, Site Technical Support Director
- R. Rouse, Compliance Supervisor
- G. Sowers, Site Technical Services
- P. Wiley, Unit 2 Operations Manager'l Paso Electric
- S. Gross, Site Representative NRC
- D. Coe, Senior Resident Inspector P. Galon, Reactor Inspector
- M. Hiller, Reactor Inspector D. Proulx, Reactor Inspector
+J. Sloan, Resident Inspector The inspectors also held discussions with other licensee and contractor personnel during the course of the inspection.
- Attended the exit meeting on September 25, 1990.
Descri tion of ualit Pro rams Ins ected 37701 The licensee's problem identification and resolution programs have..been revised and have been implem'ented in the tevised form for about
'a year.
Although this inspection concentrated on the root cause of failure evaluations, the inspector also reviewed the. areas of the programs which were noted as weaknesses in earlier NRC inspection reports.
These areas were:
Willingness of the plant staff to identify non-conformances and conditions adverse to quality, Timeliness of resolution of identified problems, Timeliness of root cause of failure analysis, Adequacy of root cause analysis and troubleshooting, Preservation of the as-found condition of failed material components by the maintenance staff, Adequacy of corrective action, fc
'dentification of nonconformances during ASME Section XI Testing and In-Service Inspection (ISI),
Adequacy of plant procedures associated with the areas identified above.
The licensee appeared to have made specific improvements in these areas.
The findings in the above areas are documented in this report.
3.
Licensee Problem Identification, Corrective Action, and Root Cause ssessment ro rams 377
The licensee uses several programs to implement the quality process associated with problem identification, corrective action, and root cause assessment.
The inspector found that, for many of the issues identified, several of the licensee's problem resolution documents were involved.
Therefore, the following listing is provided to summarize the licensee's programs.
Nonconformin Materials, Parts, Or Com onents
CFR 50, A
endix 8, rater>on
To implement these requirements, the licensee uses Material Nonconformance Reports (MNCRs).
Identification and Correction of Conditions Adverse to ualit (10 CFR'0, en sx 8, rsterson YI To implement these requirements, the licensee uses:
Work 'Requests (WRs), for problem identification only, Material.,Nonconformance Reports (gNCfts),
guality Deficiency Reports (gDRs),
Problem Resolution Sheets (PRSs),
and, V
Corrective Action Requests (CARs), for significant programmatic failure Although the licensee uses other documents to perform evaluations and identify and correct problems, the only documents the licensee credits for identification and correction according to the requirements of Appendix B of 10 CFR 50, Criterion XV and XVI, are those mentioned above.
Determination of the Cause of Si nificant Conditions Adverse'to ua st
ents icatson o
oot ause o
as ure endix 8,
riterson V
,~t To implement these requirements, the licensee uses:
V Engineering Evaluation Reports (EERs) to-document root cause of failure analysis associated with material deficiencies, Human Performance Eval'uation Summaries (HPESs) to determine root cause of failure associated with human performance based failures, and Incident Investigation Reports (IIRs) to determine root cause of failure for complicated and interrelated failures with multiple root causes.
The inspectors reviewed about 120 of the problem identification and root cause reports listed above to determine the licensee's ability to identify and correct conditions adverse to quality.
The licensee appeared to have identified several conditions adverse to quality.
The most frequently identified conditions adverse to quality are in the area of documentation; the lack of documentation of engineering decisions and improper documentation of the work control process.
Several examples of these conditions are discussed in this report.
Also, with minor exceptions documented later in this report, all the conditions adverse to quality which were independently identified by the inspector had already been identified and documented by the licensee.
Therefore, the licensee appeared to be implementing the problem identification area of the quality process.
Problem Identification and Corrective Action (37702 The inspector reviewed the applicable procedures and sampled reports. -;z which identified problems and corrective action.
These reports were PRSs, gDRs, WRs, and MNCRs (MNCRs are also discussed later in this inspection report).
Willin ness of Licensee Em lo ees to Identif Problems:
The.inspector discussed formal identification of pro ems wit about 40 members of the licensee maintenance, operations, and engineering staff.
All.of the individuals appeared knowledgeable and familiar enough with the problem identification process to initiate problem identification documents'",lthough in five cases, it appeared that individuals may have been more inclined to initiate work request rather than MNCRs or gDRs if it appeared that only a minor problem had occurred.
This was considered a minor concern because the inspector observed that work requests were specifically reviewed by planners to determine if an HNCR or other problem identification
document should be initiated.
Because earlier NRC inspection reports had identified that several licensee employees felt restrained in identifying problems, this is considered an improvement.
Dis osition of Problem Identification Documents:.
The documents for prob em reporting appeared to have been initiated, validated and:
dispositioned in a timely fashion.
In many instances this occurred within 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />, and in most cases within 1 day.
In the cases which took longer than 1 day to disposition, the operability and reportabil)ty:~g,,-
assessments appeared to have been performed within an hour or two, and the delay in disposition of the document. appeared to have been caused, by waiting for additional data which more completely characterized the issue.
these delays did not appear to affect the final. disposition of the report in a non-conservative manner.
g$
QDR Pro ram:
The licensee implemented the QDR program in October, 1989.
e inspector found that the QDRs are categorized by root cause and evaluated for trends.
QDRs must be resolved in 30 days or the issue is immediately escalated at least one level of management.
The inspector noted several examples in which that process was apparently effective in obtaining timely resolution.
Listin of Incorrect Re uirement for a DR:
QDR 90-291 dated July 17,
9
,
iscusse proce ura e scsencies associated with an investigation IIR 2-1-90-2 into failures of the steam bypass control system.
The requirement recorded in the QDR stated
"The investigation director shall assure the initiation of a QDR for those issues which have an impact on nuclear safety."
Although this statement is true, the issue associated with this QDR was inadequate procedures.
Therefore, the requirement line should address the requirements associated with the adequacy of operations procedures rather than the requirements of the investigative director.
The licensee agreed that this requirement was not the appropriate requirement to be listed in this QDR, and stated that the QDR system was still relatively new, and the review and validation process had been reducing these types of errors over time.
Since the QDR appeared to have been resolved satisfactorily, the inspector did not identify a significant safety concern.
Lack of Procedural Re uirements to Review Work Re uests to Identif Con itsons dverse to ua it :
evision 14, CN 4 of procedure 30 C-9ZZ01, or ontro
,
i not appear to require review to determine if a condition requiring documentation by an MNCR or QDR.
However., the inspector had observed that work requests were reviewed for reportability, operability, and conditions which meet MNCR and QDR thresholds.
The inspector had observed'everal cases in which the appropriate problem "
identification and corrective action documents have been generated from work requests.
The inspector did not observe cases in which a work.
request appeared to identify a condition 'adverse to"quality, without"the issuance of an MNCR or QDR.
After discussions with the inspector, the licensee issued PCN 1 to Revision 2 of procedure 30DP-9WP01, Work Initiation, which required a prompt formal review of work requests to determine of conditions adverse to quality exist, and to initiate MNCR or QDR documents as appropriate.
No violations of NRC requirements were identifie.
Nonconformin Condi tions 37702 The licensee program to identify non-conforming conditions appeared to have appropriately identified, validated and dispositioned these conditions.
The dispositions appeared to have been'timely (within I or 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to perform initial evaluations of operability and reportability),
with the exception noted later in this report.
"i ASME Section XI Testin
Previous NRC inspection reports had identified t at urging SME Section XI In-Service Testing (IST), the licensee may not requi,re that an MNCR be issued.
Similar concerns were documented for In-Service Inspection (ISI).
The inspector reviewed the requirements of the applicable procedures, and found that, for both IST and ISI (inspection of pipe welds) procedures, a step existed which required that MNCRs be initiated in accordance with applicable MNCR procedures.
The MNCR procedure, however, required the initiation of an MNCR unless subsequent steps in the work order or. surveillance test procedure corrected the non-conforming condition.
Therefore, for some failures of IST surveillances, no MNCR need be initiated.
However, whether or not.an MNCR was initiated, the licensee stated that the components and systems must be promptly declared inoperable, an immediate operability and reportability review initiated, and an engineering evaluation initiated.
These prompt reviews were required by the surveillance testing procedure which governs operations during these tests.
During independent inspection activities, the inspector reviewed two separate cases in which these required actions appeared to have occurred promptly.
Therefore, the licensee actions appeared to meet the requirements of Appendix B to
CFR 50, Criterion XV and XVI for identification and disposition of nonconforming items, in that appropriate rework appeared to be specified by procedure and appropriately implemented.
Warehouse Discre ancies:
The inspector verified that revision I of proce ure
P-MC29, arehouse Discrepancy Notice (WDN) step 2.8.2.,
required that an MNCR be initiated for those nonconforming materials which are conditionally released for service; Effect of EER Backlo on MNCRs:
Previous NRC inspection reports had i entifie a signsf)cant ac og of EERs, and had documented concerns that issues significant to safety may not be resolved on a timely,basis as a result of this backlog.
The licensee stated that upon initiation'f the MNCR program, all outstanding EERs were reviewed to determine if nonconformances or conditions significant to safety were identified in any of these EERs.
During the course of this inspection, the inspector
.,identified one issue which was apparently delayed as a result of the EER backlog.
The issue was documented in PRS 524, dated January 26, 1990, IIR 3-1-90-1; and EER 87-SS-18.
The chemistry sampling procedure for gas stripping used volume amounts which were not consistent with EER
'7-SS-18.
The licensee stated that the initial indications showed that the discrepancies resulted in conservative calculations, and no Technical Specification limits were exceeded.
However, the upper limit of the dissolved hydrogen operating range (greater than 50cc(STP)/Kg(H20)
could have been exceeded.
The corrective action for this condition was to bring the dissolved hydrogen down within 7 days, and if not, perform a
technical evaluation.
This appeared to be a satisfactory evaluation of the issue.
Exam les of Resolutions of Nonconformin Conditions:
A.
B.
MNCR 90-SP-19 documented corrosion of carbon steel, keys in the safety related spray pond valves.
The licensee determined that carbon steel was inappropriate for the ser vice, and had stainless steel. keys made for the valves in all 3 units.
The licensee evaluation determined that the vendor delivered unacceptable safety related items.
The licensee issued LER 1-90-5 according to $0 CFR 21.
This appeared to be appropriate action.
The inspector reviewed EER 89-SI-308 dated October 27, 1989,'.Which described failure of a safety injection system motor operated valve.
The troubleshooting and analysis of the failure appear to have been adequate.
However, no MNCR was initiated although this appeared to meet the criteria of a material nonconformance.
The licensee stated that the MNCR system started in October of 1989.
This EER which identified a nonconformance appeared to have been dispositioned in accordance with the existing EER procedures rather than canceled and reinstated as an MNCR.
Since the EER appeared to have been appropriately dispositioned, and the MNCR system was in administrative transition, no violations of NRC requirements were identified.
C.
The inspector reviewed QDR 90-333, dated August 24, 1990.
The initiator reported that the Quality Control (QC) had not promptly notified operations of a nonconforming condition.
This QDR appeared to document examples of effective problem identification and awareness by licensee staff of the required nonconformance and quality processes for the following reasons:
(I)
Operations was independently aware of the nonconforming condition (damaged braid on a nuclear indication cable),
(2)
Operations was aware of the requirements for QC to promptly notify operations of MNCRs, (3)
Operations was aware of and promptly used the appropriate problem identification document (QDR) to report that the notification had not taken place, (4)
The root cause of the deficiency was identified (that QC had placed the MNCR in the wrong action basket),
and (5)
The corrective action included a review of all previous MNCRs to determine that proper notification of operations had'ccurred for all past MNCRs, as well as briefings of QC personnel and correction of the human factors concern by establishing an action basket for items which require additional prompt action.
No violations of NRC requirements were identifie p
Root Cause of Failure 37701 The licensee determined the root cause of failure and cause of significant conditions adverse to quality using the following three programs described below.
The quality of root cause assessments appeared adequate for each of these programs.
Engineering Evaluation Reports (EERs) to document root cause of failure analysis associated with material deficiencies, Human Performance Evaluation, Summaries (HPESs) to determine root cause of failure associated with human performance based issues, and Incident Investigation Reports (IIRs) to determine root cause'f failure for complicated and interrelated failures with multiple "root causes.
Lack of Follow U of Condition Adverse to gualit
The inspector identifie a case in w ic t e in ings o
events analysis identified conditions which may be considered adverse to quality, but which the licensee did not identify as such and pursue corrective action.
HPES 90-33 identified, among other findings, that an MNCR was not promptly initiated when incorrect oil was added to the charging pump.
The licensee initiated a (DR to follow the resolution of this deficiency after discussions with the inspector.
This concern is mitigated by the fact that the licensee identified six other conditions adverse to quality associated with this event, and that this type of finding appeared to be an isolated example.
This particular event is discussed later in this report as an example of use of improper work control documentation identified by the licensee.
Closure of Investi ation Before Com rehensive Review of Root Cause
~an in s: If con itvons a verse to qua 1ty ave een s eats le uring an investigation, procedures required the licensee to issue appropriate problem identification and root cause of failure documents.
Before an investigation can be closed, the licensee must be satisfied that the root cause of the failures are sufficiently understood to preclude further events.
The inspector was concerned that the understanding of individual root causes of the event and associated corrective actions are not reouired to be reviewed to determine if the interaction of root causes are appropriately addressed by corrective action.
The licensee stated that, as specific root causes and corrective actions are determined, they are reviewed by an investigator to ensure the corrective action is appropriate.
The licensee stated the reviewer is sufficiently feamiliar with the particular event to identify cases of inadequate or inappropriate corrective action for interrelated root causes.
Investi ative Methods:
The inspector reviewed Revision I of licensee procedure 790P-OIP02,
"Investigative Methods.", which discusses human performance evaluations and various root cause investigation techniques.
This appeared to be adequate to implement standard industry techniques for root cause analysi Trainin
The inspector reviewed the training records of the individuals w o a
performed investigations to determine if they'ad received appropriate training to perform the investigations.
The training records showed that each of the individuals performing human performance and engineering root cause of failure evaluations had received appropriate root cause and/or human performance evaluation training.
Destruction of As-Found Condition:
In NRC inspection report 89-28, the R
maintenance team ocumente the licensee's intent to provide training to maintenance craftsmen in preserving the as-found condition of defective components.
This intent was based on the licensee findings.
that several root cause evaluations could not.be accomplished as a result of craftsmen destroying the as-found condition pf a defective, component.
At the time of this inspection, the licensee had, plans to provide this training to maintenance supervisors.
The inspector reviewed EER 90-CH-70, which documented a recent case in which maintenance craft appeared to have destroyed the as-found condition of a defective component, thus hampering a root cause assessment.
After discussions with the inspector, the licensee established a formal commitment to complete training to the maintenance craft on preservation of the as-found condition of apparent failures by March 31, 1991.
In addition, the licensee established a formal commitment of December 31, 1991 to provide an abbreviated root cause training to all maintenance supervisors.
Exam les of Root Cause Assessments:
A.
IIR 3-2-89-35 dated December 22, 1989; associated with PRS 437 dated October 16, 1989, and EER 89-DG-103 dated October 16, 1989; investigated the inadvertent actuation and subsequent failure of the emergency diesel generator as a result of steam cleaning the diesel.'he IIR found that housekeeping had requested to steam clean parts of the generator.
After permission had been obtained from the operations staff and the electrical equipment associated with the diesel had been wrapped in plastic, the steam cleaning commenced.
The investigation determined that soon after, the water intrusion into an ESFAS relay caused a short circuit to actuate the diesel, and water intrusion into an air start solenoid caused the diesel to fail.
The licensee corrective action informed operations staff in all three units of the need to declare equipment inoperable for cleaning and maintenance, and did not add specific guidelines for steam cleaning to plant procedures, because this event was determined to be an isolated occurrence.
However, the Unit 2 housekeeping guidelines were revised to add the requirement to not steam clean 'any operable or energized safety related equipment.
B.
IIR 3-2-90-31 dated July 19, 1990, investigated the improper operation of the azimuthal power tilt computed at about 20 per"cent power.
The root cause was the transposition of detector sensitivity values while updating the parameters in. the plant computer.
The investigation evaluated the human factors concerns and provided event and causal factors analysis.
The investigation appeared to have appropriately identified the root causes associated with the even No violations of NRC requirements were identified.
7.
Use of Ina ro riate Documentation (307702 The licensee, along with NRC inspectors, has identified several examples of events and instances which appeared to have been associated with inappropriate documentation.
These problems appeared to be the result of inappropriate documentation of work control, and inadequate or nonexistent documentation of engineering evaluations.
The following,were identified:
4
A.
Ina ro riate Documentation of Mork Control
~
'
+i 44 The following lists occurrences in which improper documentation.of work was identified.
Addition of Incorrect Lubricant to a Char in Pum
On July 5, 1990; urging routine maintenance, 5/8 of a ga on of incorrect lubricant was added to the crankcase of Unit 1 Charging Pump CHA-P01, and the pump was operated with the incorrect lubricant for about 20 minutes.
The pump was declared inoperable and the appropriate action statement entered.
The conditions adverse to quality were identified by PRS No. 893 dated July 5, 1990, and evaluated by Engineering Evaluation 90-CH-88.
Human Performance Evaluation (HPES)
90-33 found that several inappropriate actions had occurred during the actions to recover from the addition of incorrect lubricant.
The associated (DR No.90-307 identified 5 deficiencies associated with the event.
These documented that:
(I)
a maintenance work package was prepared without a written work request; (2)
corrective maintenance was performed using a modified preventative maintenance task instead of a corrective work order; (3)
the reason for the work was not documented on the work order; (4)
the work proceeded without an approved interim disposition, of the engineering evaluation; and (5)
the pump was declared operable prior to obtaining the required
,final disposition on the associated Engineering Evaluation
,, Report (EER) No.,90-CH-88.
These inappropriate actions were considered adverse to quality and were documented on guality Deficiency Report No 90-307.
An additional deficiency identified by the inspector was that an INCR was not promptly initiated, as discussed earlier in this report.
RCS Leak as a Result of T on Tubin Ru ture:
During the Unit 2 outage of anuary, 1990, a
essgn c ange package to install the reactor vessel level indication was installed.
The equipment was tested using a temporary procedure rather than a work order.
As.a
result, a plastic hose remained attached to the pressurizer and was exposed to reactor coolant system pressure after the work was finished.
Operations was not aware of this configuration, since the system was never properly restored.
In preparation for a mode change, operations initiated a fill and vent procedure.
The tube ruptured at, about 100 psia, resulting in a reactor coolant'eak....
B.
Inade uate Documentation of Root Cause of Failure and En ineerin va uatsons In the following,'.instances, the licensee or, the inspector,.identified-cases in which root cause of failure and=engineering evaluations
'ere performed, but ~were-not documented, or did not appeat to have been documented properly.
(1)
On March 25, 1990, during mid-loop operations, the licensee determined that, in order to both minimize leakage and support other work in the plant, RCP flanges should be removed at a reactor coolant level height which was greater than that specified in the procedure (center of the hot leg, which is 101'4"). Engineering and plant management performed engineering evaluations of removal of the flange at a higher coolant level.
The final level determination was 103'8", at which the flanges were removed.
However, the licensee did not document the engineering evaluation before the flanges were removed.
QDR 90-180 reported the problem that the flanges were removed at a higher reactor coolant level (103'8") than required by the procedure, which specified the middle of the hot leg (101'4").
The licensee determined, in QDR 90-180, that the specific statement
"center of the hot leg" did not meet the intent of the work, and that the procedure. should have been changed when an engineering decision was made which apparently conflicted with the procedure.
They also determined that an EER was not initiated to document the engineering decision.
The inspector noted that the QDR appeared to have appropriately escalated the issue to higher managem'ent because ineffective responses were obtained, and eventually obtained a discussion of the engineering
,analysis.
However, the QDR did not document the detai:ls 'and engineering basis of the analysis.
(2)
QDR 90-251 documents the licensee's failure to follow a vendor recommendation during plant work.
The licensee had concluded that the vendor recommendation was not required.
However, the engineering decision was not documented.
(3)
HPES 90-33, which documented addition of an incorrect lubricant to the charging pump, stated on page '10 that a QA/QC supervisor had recommended incorrect technical actions to correct the pump condition based on incorrect assumptions of the event.
The inspec'tor was concerned that a
QA/QC supervisor appeared to have given a technical evaluation without recommending initiation of a formal engineering evaluation.
The licensee stated a formal engineering evaluation had been initiated, but
had not been documented clearly with respect to the QA/QC supervisors involvement.
The inspector considers this an example of vague documentation.
(4)
EER 89-SS-24, associated with HNCR 90-SP-002, reported the-failure of a RCS sample valve for the pressurizer steam space.
This EER also noted that water obtained downstream of the valve was a reddish color and did not appear to:be RCS liquid. - The
'inspector was concerned that another piping system~ay have
'been mistakenly connected to the RCS, and that this concern was
,,-,not:addressed in the EER.
During discussions,with~the
-
inspector, the licensee stated that. the failure "of 'the valve
,,was due to improper maintenance, that the EER.and INCR were..
initiated as a result of post-maintenance testing, and,that the red color was caused by the dye penetrant used in the maintenance procedure, and that no system other than the RCS was connected to the sample line.
The inspector noted that this pertinent information was not included in the EER and, although the resolution appeared acceptable, the documentation did not address the information which the engineer discussed with the inspector.
(5)
(6)
PRS 713 dated Nay 25, 1990, identified that the Boric Acid Nake Up Pumps hold down bolts did not meet design requirements.
MNCR 90-CH-ll. -12, and -13 appeared to have been promptly and properly initiated.
The determination of the operability of the boron injection flow path was made based on analysis and testing of the existing bolts, and was documented in calculation 13-NC-CH-318.
The PRS attachment which documented that the condition was not reportable does not reference the calculation by number, nor do the associated HNCRs and EERs.
Although the technical conclusions are substantiated, this was one of several examples of the lack of reference to associated technical documents.
I EER 89-DG-22, which evaluated failures of Agastat relays installed in the alarm circuits of the emergency diesel enerators, omitted the number identification of the document PCO) which lists an acceptable replacement.
This EER,,is
~
incomplete.
(7).
The inspector reviewed EER 90-ZJ-20 which evaluated the seismic
.qualification of a scaffold in the Unit 2 Control'Building.
< The calculation appeared to conservatively consider the. design basis earthquake and applicable seismic response modes.
The inspector'oted that the calculation incorrectly referenced the vendor specification rather than the ap'plicable Regulatory,
'uide (RG) 1.61 for damping values -for bolted
. structures However, since the conservative values of RG 1.61 were used, this finding has low safety significance.
No violations or NRC requirements were identifie s.
9.
Inade uate En ineerin Work 37701 EER 90-AF-022 dated July 31, 1990, associated with EER 90-AF-025 dated September 4,
1990, and MNCR 90-AF-002 dated August 23,'1990; documented that the governor had difficulty controlling the speed of the Unit 2 turbine driven auxiliary feedwater (AFW) pump.
The licensee determined that condensation caused the governor to lose control, and that, after troubleshooting, determined that the source of the condensation was the liquid colletting in the steam traps and liquid condensing on cold~piping in the steam he'ader to the turbine.
The licensee, therefore, required the drain:lines -for each unit to be blown down every six.hours, and later required the temperature of the steam supply header to be monitored with a contact pyrometer every six hours using procedure 410P-1ZZ15,
"NUOPS 5 OPSSUP Personnel Work Stations" (and the corresponding procedures for Units 2 and 3).
This procedure required that the turbine driven feed pump be declared inoperable if the temperature reading was less than 190'F.
Discussions with licensee operations and engineering personnel found that the temperature of the inlet header changed by a few degrees of temperature as a result of draining the condensation.
There did not appear to be a formal calculation of tolerance in the operability criterion allowed for this change in temperature in the procedure, and the order and time interval between operations (draining or temperature monitoring) was not specified in the procedure.
The inspector reviewed the logs of the temperature readings.
There readings recorded below 195.8'F (for an operable pump) in the three units.
Therefore, there does not appear to be an operability concern.
However, the lack of inclusion of the uncertainty in the temperature limit for operability is considered inadequate engineering practice.
After discussions with the inspector, the licensee promptly established that, for Unit 2, a 1.4'F change in temperature occurred as a result of draining condensation.
Therefore, the licensee changed
'the criteria to 193'F.
The licensee stated that, although a formal calculation of the tolerance of the earlier operability criteria of of 190'F was not made, testing had verified operability of the pump at a header temperature as low as 188'F.
The licensee also stated that the lack of formal calculation of tolerance would be used as a "lessons learned" briefing for system engineers.
Followup of the licensee's actions in the area of AFW pump operability and associated quality of engineering work will be done by NRC resident inspectors as part of routine inspection activities.
Pro rammatic Inconsistencies and Inade uacies 37702 The inspector identified the following programmatic concerns, which appear to be inconsistencies in quality programs.
Re uirement to Review EERs and MNCRs for Additional Conditions Adverse to ua it :
e inspector note t at some s and s appeare to iscuss conditions which may meet the requirements of conditions adverse to quality.
The inspector noted 'that neither the MNCR procedure nor the EER procedure appeared to address the need to initiate a (DR if the condition which initially appeared to require an MNCR or EER turned out to meet (DR requirements.
The following are examples of MNCRs and EERs which may have met these requirement e
EER 90-CH-50 dated April 25, 1990, documented a continuing discrepancy between charging and letdown flow.
The root cause was determined to be that flow orifice 3JCHNFE0202 was installed backwards (with the "inlet" arrow facing downstream).
As a result, MNCR 90-CH-14 was initiated to document the condition.
This appeared to be caused by a personnel error.
The inspector noted two additional problem identification reports which noted orifices installed backwards.
The root cause of the incorrect
= orifice installation was discussed in only one of these reports.
P EER 90-ZC-14 dated February 8, 1990, noted a broken stud on the containment polar crane rail.
The polar crane is a.level 2 over level.1 material failure concern, and, therefore, the.possible failure may be significant to safety.
MNCR 90-ZC-0003 performed an engineering "
analysis to show that the failure of a bolt in the polar crane rail during a seismic event would not result in damage to the plant.
EER 90-ZC-66 provided a root cause analysis and interim and final disposition of the condition.
The information available in these documents concluded, based on physical evidence, that the stud was installed improperly as a result of improper fit up of the stud hole and filler plate.
The EER documented failure of two other crane rail studs.
These two broken studs may be considered a condition adverse to quality requiring a
(DR to determine and correct the cause of the additional broken studs.
The EER did not note that in the event of a design basis earthquake, the failure of the crane rails would not result in the crane damaging the reactor components since the polar crane would stay pinned in the upper levels as a result of the crane's configuration, regardless of bolt failures in rails.
The licensee did not include this relevant information, which resolves the 2 over I concern, in the EER.
A ro riate Level of Investi ation:
IIR 3-2-90-017, which discussed a
RCS ea as a resu t of a p astic tubing rupture, was assigned as a level 3 investigation, Ahich meant a moderate level of effort is required, but the investigation is assumed to be straightforward.
After review of the IIR and discussions with the licensee, it appeared that the investigation met the criteria for a level 2, requiring more resources, because the situation leading up'o the event was relatively complicated.
The licensee stated that considerable discussion had taken place before the investigation started on whether the level was 2 or 3, and 3 had been assigned.
After discovery that the background facts were much more:=:.
complicated than expected, and more resources may be needed, the licensee retained the lower classification of 3. It appeared that the resulting report was not particularly clear and concise, and that a level
investigation using more resources may have been appropriate.
Control and Stora e of Work Re uests 'Rs in Accordance with the'
uirements or ua
)t ocuments:
Rs s
not appear to e treated as qua ity documents for the fo owing reason.,
Problem identification;...
documents (MNCRs, CARs, gDRs, PRSs, and'WRs)-were assigned a number immediately after an initiator submitted the document to the appropriate group.
At that point, except for work requests, procedures require that the document become a quality record regardless of its determination of validity or disposition.
However, WRs were not stored or maintained as
, quality documents, but were maintained on a computer database (SINS).
The licensee stated that storage of work requests in a computer database rather than in permanent quality record storage was not a concern for the following reasons:
2.
The content of a work request was repeated in the associated work order.
The inspector notes, however, that if the problem was already identified separately or was determined not to be necessary, the work request was not repeated on a work'rder.
The work requests stored in the database were evaluated for trends on a quarterly basis.
'i
~
'.
Work requests which identified work which~was determined.to be not necessary were considered invalid.
In accordance with plant
'rocedures, invalid work requests were returned to the initiator with a notification that the request was considered invalid.
The licensee stated that, in this case, the initiator could issue another problem identification document (QDR or MNCR) if he desired.
4.
The WRs are reviewed hy planners to determine if a condition adverse to quality or a nonconformance exists.
This appeared to meet the intent of 10 CFR 50 Appendix B, Criterions
and 16, that conditions adverse to quality and nonconformances be identified and corrected.
The inspectors noted several errors in procedures.
They are listed below.
A.
Step 16.4.3 of Revision 5 to the Operations QA Manual stated that documents adverse to quality included, but were not limited to those listed in Article 16.3.3.
However, 16.3.3 did not appear to exist.
Instead, Article 16.3.2 listed abbreviations for documents which may identify conditions adverse to quality.
This list appeared to include many documents which no longer existed, and did not include HNCRs, QDRs, and PRSs.
Therefore, the Operations QA manual did* not appear to apply to the quality systems in use, and it described documents which were no longer used.
This appeared to be of low safety significance since the quality program appeared to be satisfactorily implemented in the lower tier procedures, and the licensee planned to implement,a revised quality manual.
B.
Procedure 60AC-OQQ02, "Corrective Action", Step 3.3.1.3;.stated, that if a CAR was found to be invalid, the"Manager, QASN or designee shall state the justification for invalidation in Section 5 and shall close the CAR and distribute in accordance with step 3.6.5.
The inspector noted that step 3.6.5 did not appear to exist.
The
licensee stated that, to this date, no CAR had been determined invalid.
C.
D.
In the Revision 2 of licensee procedure 60AC-OQQ01, the validation process was in step 3.3. 1, and the requirement to number the MNCR was in a later step, 3.3.3.
It is considered inappropriate to number a problem identification document only after validation.
However, in revision 1 to the lower tier implementationvprocedure 63DP-OQQ05,
"Handling of Nonconforming Documents and Conditional Releases",
the procedure-appeared to properly require the MNCR to be numbered end, therefore, entered into the plant quality,.records, before the validation process takes place.
The inspector verified that in practice, the licensee numbered MNCRs before. validation,,
-.
Procedure 70-DP-'EE01,'Root Cause of Failure Analysis" steps 1.1.1, 3.7 and 5.1. 1 referred to procedure 70AC-OEE02, "Engineering Evaluation Request" This procedure should be listed as 73AC-OEEOl E.
Procedure 73AC-OEEOl, "Engineering Evaluation Request",
steps 3.5.2. 1, 3.9.3.b'7, and 5.1.5 referred to 73DP-OEEOl,
"Root Cause of Failure Analysis within Engineering Evaluations Department."
The correct procedure reference should be 70DP-OEE01.
F.
Revision 2 of procedure 60AC-OQQ01, "Control of Nonconforming Items", step 3.1.4.1, stated Root Cause of Failure Analysis shall be performed and documented in accordance with 73AC-OEEOl, "Engineering Evaluation Request" (Reference 5. 1.5).
5. 1.5.
was incorrect, and should have been listed as 5. 1.2.
G.
73AC-OEEOl, step 5.2.6 referred to 79AC-ONS08, "Post Trip Reporting".
The licensee stated that this procedure should be listed as 79PR-OIP01, 79AC-OIP01 and -.02.
H.
Procedure 79PR-OIP01, Incident Investigation Reports",
step 2.15.1 referred to procedure 92GB-ONS01,
"IST Surveillance and Special Investigations".
The licensee stated that the procedure should list 92AC-ONS01.
Step 4.4 of this procedure lists procedure 79AC-9NS09,
"Technical Specification Interpretation."
This procedure should be listed as 79AC-OAP11,
"Review and Approval of Technical Specification Interpretations."
Step 5.1.12 of this procedure lists 79AC-OIP03, "Corrective Action and Report Closeout Incident
Investigation."
The licensee stated that this procedure should be listed as 79DP-OIP03-.
Initially, Document Control (DDC) personnel stated that the appropriate reference for procedure 92GB-ONS001 could not be identified, and'"that other plant personnel had requested this particular procedure without-results.
This appeared to be an example of an ongoing problem with"
-..'rocedure references, and lack of timely corrective actions.
Because engineers were able to identify the correct reference for this and the other incorrect references after a search, this was not considered a
major concern.
However, the significant number of incorrect references and the lack of correction was noteworth No violations of NRC requirements were identified.
Review of Desi n Chan es The inspector reviewed design change PCP 85-02-SI-067-00, which replaced the hard seats on 4,containment sump isolation safety injection system butterfly valves with soft seats'.
The inspector reviewed the design package to evaluate its ability to be installed and its compatibility with existing plant systems.
The design change was initiated because the existing nickel coating was peeling and the valve was subject to binding.
The design change appeared to address appropriate design and function..
requirements, including operating temperature, seating torque changes, and vendor maintenance documentation.
p v
The inspector reviewed the design change PCR No 87-13-SI-045, which replaced two-rotor limit switches on motor operated valves with four-rotor limit switches.
This replacement of limit switches has been discussed as a desirable change in several NRC and industry notices.
The design change appeared to address the appropriate concerns for limit switch setpoints and require review by the valve design group to ensure appropriate documentation of setpoints.
In addition, the licensee was standardizing the safety injection system motor operated valve limit switch settings.
The inspector considers this to be in accordance with NRC initiatives in this area to improve maintenance and human factors performance.
No violations of NRC requirements were identified.
Review of Emer enc Li htin Records 64704 The inspector reviewed work orders which documented the maintenance and testing of emergency lighting units.
The inspector noted that, according to Mork Order No. 00315914 dated November 22, 1988, emergency lighting unit EXIDE No. l-l appeared to have been tagged out on an improperly set clearance associated with Clearance 88-01192, set on November 21, 1988, about 3:30 am.
As a result, the lighting unit power supply breaker was open for about 57 hours6.597222e-4 days <br />0.0158 hours <br />9.424603e-5 weeks <br />2.16885e-5 months <br /> without the unit being declared inoperable.
The topic of emergency lighting has been the subject of extensive NRC and licensee discussions in the recent past and this finding will be".combined with future discussions and examinations.
Potential Safet -Related Pum Loss TI 2515/105 The inspector reviewed the licensee's activities which address design concerns associated with safety related pump miniflow.,configuration'."'he licensee response to NRC Bulletin No. (IEB) 88-04 discussed the adequacy of installed miniflow capacity and the possibility of a pump being dead-headed by improper miniflow line sizing.
"
- '4"~"-
A.
Existence of a Common Header The inspector confirmed that a
common header exists for the licensees safety related pumps.
This is also documented in the licensees responses to IEB 88-04.
The inspector did not review the
flow resistances calculation details.
This will be followed in a later engineering inspection.
B.
Pum s Vendor Assesments I
The licensee reviewed the designs of the safety related pumps with respect to the design concerns noted above and requested vendor evaluation of short and long term pump performance under.miniflow pump recirculation.
The area of concern identified by the vendor (Ingersoll-Rand) of the high and low pressure safety injection Pumps;-the containment spray pumps, and -the condensate:transfer pumps, was the need for quarterly surveillance measurements of vibration, pump head, and leakage.
The licensee plans to implement these requirements.
Future engineering inspections will review the surveillance results and other test data.
C.
Plant Modification The licensee and vendor have determined that no plant modifications are required.
D.
Pum Test Results and Maintenance Histor ASME Section XI Test results and pump maintenance will be reviewed in association with ASME Section XI inspections.
No violations of NRC requirements were identified.
14.
Radiation Protection Area Walkdown During a routine walkdown of radiation areas, the inspector observed a
tool and protective clothing glove outside the charging pump area.
A health physics supervisor was'informed.
These items are not considered safety significant and no violations of NRC requirements were identified.
15.
Follow U of 0 en Items 92701 A.
Closed Enforcement Item 528/89-02-03, Interface with the Phoeni'x Fire e artment Inspection Report 50-528/89-02 identified that the Phoenix" Fire Department had no preplanned emergency plans for the site since no planning with the licensee had been performed.
The inspector'erified that the licensee had performed three drills with the Pho'enix Fire Department in>,the last year.
Also, the inspector discussed the emergency response with"the Phoenix fire fighters stationed in Fire Station No. 40, which would be expected to respond to the licensee.
Most of the fire fighters at the station had received training and were familiar with the site location.
The inspector verified that the maps to the site were in the cab of the fire truck and that, in response to an alarm at Palo Verde, the
i
n tg
computer dispatch system would dispatch, only those fire stations with Phoenix fire department trained members.
Initial test alarms resulted in dispatching the Tolleson and Glendale stations which were not part of the Phoenix Fire Department, and 'which had not been trained.
After discussions with the licensee, this appeared to have been corrected.
The inspector verified that the licensee would be able to supply a sufficient number of radios to the Phoenix Fire Department upon arrival at the plant.
A+
0 en Follow U Item 528/89-08-01 Emer enc 0 eratin Pr ocedures An NRC team inspection of the emergency operating procedures found that the procedures presented significant problems to the operators, particularly in the areas of consistency and format.
NRC inspection report 90-02 documented that on February 17, 1990, licensee implementation of corrective action for these problems had been delayed from July 1990 to July 1991, and that the licensee was in the process of identifying these delays to the NRC.
At the time of this report, the contractor assistance to prepare the writers guide for the procedures was late.
The licensee stated that technical and human factors comments for the procedures have been incorporated.
However, the procedures will be delayed further as a
result of the delay to the writers guide.
Increased attention is warranted to minimize the delays in implementing the procedures.
0 en Unresolved Item 528/89-16-02, Leak Rate Testin Methodolo An NRC inspector identified that the methodology for leak rate testing may not be conservative.
The FSAR appeared to implicitly allow only the inflow method.
However, most testing by the licensee used the outflow method.
The licensee evaluated the concern and identified changes which may be implemented in the leak rate testing method.
The inspector reviewed the licensee's resolution, and considered that the licensee had identified potential resolution of several issues associated with this concern.
However, since the licensee has not implemented the resolution of these issues, the inspector was unable to verify satisfactory resoIution of the item.
This item remains open.
0 en Follow U Item 528/89-28-03, Vendor Notifications The licensee did not appear to have taken adequate measures to ensure that notifications of equipment deficiencies from vendors were properly obtained, reviewed and dispositioned.
The 'licensee formally notified primary contractors of the need to provide any notifications of equipment problems.
.The inspector was concerned that the vendors notified did not appear to be complete and o'mitted vendors such as Rotork, Raychem, Limitorque, Dresser Valves, and other products.
The license'e stated that, although separate correspondence had taken place with Limitorque, there was no need to work with suppliers other than the primary contractors.
The inspector noted that several of the vendor products which are
e
supplied by secondary contractors have experienced pr obl ems, including the vendor's product which initiated this open item.
The licensee appeared to be taking step's to to ensure that secondary vendors of plant equipment will provide vendor notices of equipment concerns, with the vendor consolidation project in response to Generic Letter 90-03.
Since this project is subject to licensee commitments which appear timely, this item is, closed.
(0 en Follow u Item 528/89-17-Ll Diesel Generator Fuel Oil Free
,s< $8
~
The licensee analyzed the likelihood that both trains of diesel generators would be rendered inoperable-as a result of flammable liquid flowing through a gap in a fire barrier between the ~trains, of emergency diesel generators.
This analysis determined that the probability was low.
However, the use of probabi listic risk analysis is not c'onsidered adequate to resolve issues involving separation of trains.
The licensee plans to request an exemption from the requirements for separation of trains.
Because the licensee has not yet resolved this issue, this issue
'remains open.
Closed Licensee Re ort 528/89-50-L1, Inade uate Desi n of the utomatic um a ve DY ctuator.
During the March 3, 1989 event at Palo Verde, the ADVs did not operate properly.
The licersee and Control Components International (CCI) reviewed the design of the Automatic Dump 'Valve (ADV) actuator and determined that the design of the actuator was inadequate.
As a
result of testing and analysis, the cause of the ADY failure was determined to be excessive leakage past a piston ring in the valve actuator.
The licensee has determined that the requirements of 10 CFR 21 were met and reported the design inadequacy accordingly.
The licensee corrective action for this issue and other issues associated with the event are being followed by the resident inspectors in the post-restart items.
Therefore, this item is closed.
Closed Unresolved Item 528/89-53-01, Lack of Dis osition of Non-Conformin tern NRC inspection report 89-53 identified that nonconformance report (NCR) 0-037-85-1 was initiated on February 12, 1985 and.closed December 6, 1986.
Unit 1 achieved initial criticality on May 25, 1985.
R'eview of documentation showed that the plant.apparently operated without appropriate disposition of the *NCR.
Section 4.6 of
'he applicable procedure, 6N417.21.00,
"Control of Nonconforming Items," required that the nonconforming items be appropriately dispositioned 'before declaririg'he system operable.
"The only disposition available for this NCR appeared to be procedure ECE-ZZ-A017, revision 1, dated July 24, 1986.
The licensee performed a review to determine if another disposition of the NCR occurred before criticality; however, no such disposition was found.
This issue appears to be a violation of Criterion XV of Appendix B to 10 CFR 50 (50-528/90-38-01).
The existing program to identify nonconforming
e items (MACR program)
appears to have corrected this weakness in the quality system.
The final resolution of this issue has been verified by the inspector.
This violation meets the criterion of a non-cited violation as defined in 10 CFR 2, Appendix C.
This item is closed.
18.
Exit Meetin 30703 The inspectors met with licensee representatives denoted in paragraph
on September 25, 1990.
The scope and findings of the inspection were discussed as described in this report.
Licensee representatives
- ,-..--
acknowledged the inspector's finding t