ML17305B204

Forwards Response to NOV & Payment of Civil Penalty in Amount of $125,000 Re Failure to Implement Updated FSAR

ML17305B204
Person / Time
Site:	Palo Verde
Issue date:	11/15/1990
From:	Conway W ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR
To:	NRC OFFICE OF ENFORCEMENT (OE)
References
161-03592-WFC-M, 161-3592-WFC-M, NUDOCS 9011280122
Download: ML17305B204 (80)

Text

. ACCELERATED DEMONRATION SYSTEM DISTRIBUTION REGULATORY INFORMATION DISTRXBUTXON SYSTEM (RIDS)

ACCESSION NBR:9011280122 DOC.DATE: 90/11/15 NOTARIZED: YES DOCKET FACXL:STN-50-528 Palo Verde Nuclear Station, Unit 1, Arizona Publi 05000528 STN-50-529 Palo Verde Nuclear Station, Unit 2, Arizona Publi 05000529 STN-50-530 Palo Verde Nuclear Station, Unit 3, Arizona Publi 05000530 AUTH. NAME AUTHOR AFFILIATXON CONEY,N.F. Arizona Public Service Co. (formerly Arizona Nuclear Power R RECIP.NAME RECIPIENT AFFXLXATION Ofc of Enforcement (Post 870413) I

SUBJECT:

Forwards response to NOV & payment of civil penalty in D amount of $ 125,000 re failure to implement updated FSAR.

DISTRIBUTION CODE: IE14D COPIES RECEIVED:LTR ENCL SIZE:

TITLE: Enforcement Action Non-2.790-Licensee Response /

NOTES:STANDARDIZED PLANT 05000528 A Standardized plant. 05000529 Standardized plant. 05000530 RECIPIENT COPIES RECIPIENT COPIES D ID CODE/NAME LTTR ENCL ID CODE/NAME LTTR ENCL PD5 LA 1 1 PD5 PD 1 1 PETERSON,S. 1 1 TRAMMELL,C. 1 1 INTERNAL: AEOD/DOA 1 1. AEOD/DSP/TPAB 1 1 DEDRO 1 1 NRR/DOEA/OEAB11 1 1 NRR/PMAS/ILRB12 1 1 NUDOCS-ABSTRACT 1 1 0

REG GN

~~

ZR RSS/EPRPB 02

.1 1

1 1

1 1

OE FXLE RGN5 FILE 03 01 1 1 1 1

EXTERNAL: NRC PDR NSIC 1 1 NOTES: 1 1 D

A D

D NOTE TO ALL "RIDS" RECIPIENTS:

PLEASE HELP US TO REDUCE WASTE! CONTACT THE DOCUMENT CONTROL DESK ROOM Pl-37 (EXT. 20079) TO ELIMINATEYOUR NAME FROM DISTRIBUTION LISTS FOR DOCUMENTS YOU DON'T NEED!

TOTAL NUMBER OF COPIES REQUIRED: LTTR 18 ENCL 18

Arizona Public Service Company P.O, BOX 53999 ~ PHOENIX. ARIZONA 85072.3999 WILLIAMF. CONWAY EXECUTIVEVICE PRESIDENT 161-03592-t6'C/MEP/KLMC NUCLEAR November 15, 1990 Docket Nos. STN 50-528/529/530 Director, Office of Enforcement U. S. Nuclear Regulatory Commission ATTN: Document Control Desk Mail Station Pl-37 Washington, D. C. 20555

References:

1) Letter from J. B. Martin, NRC, to W. F. Conway, APS, dated October 16, 1990;

Subject:

Notice of Violation and Proposed Imposition of Civil Penalty - $ 125,000 (NRC Inspection Report Nos. 50-528/90-02,90-025, and 90-35).

2) Letter to J. B. Martin, NRC, from W. F. Conway, APS; dated July 20, 1990;

Subject:

Evaluation of the Fire Protect:ion System Equipment (161-03349).

3) Letter to J. B. Martin, NRC, from W. F. Conway, APS, dated August 1, 1990;

Subject:

Response to NRC Inspection Report Nos. 50-528/90-25, 50-529/90-25 and 50-530/90-25 (161-03373).

Subject:

Palo Verde Nuclear Generating Station (PVNGS)

Units 1, 2 and 3 Reply to Not:ice of Violation and Proposed Imposition of Civil Penalty - NRC Inspection Report Nos. 50-528/90-02, 90-25, 90-35 File: 90-070-026; 90-019-026 As required by Reference 1, Arizona Public Service Company (APS) submits its Reply to the Notice of Violation and Proposed Imposition of Civil Penalty.

A restatement of the violations and APS's responses are provided in Appendices A and B and Attachments 1 and 2, respectively.

Enclosed is a check payable to the Treasurer of the United States in the amount of $ 125,000, the amount: of the proposed civil penalty.

90 i i2+0i 22 90i i i5 PDR AOOCK 0 000528 0 PDC

l I

I

Director, Office of Enforcement Nuclear Regulatory Commission Page 2 APS agrees that it did not implement all of its Updated Final Safety Analysis Report (UFSAR) commitments for Quality Assurance (QA) applicable to the Fire Protection Program and that some emergency lighting units, as described in the Notice of Violation, Violation B, were not available during certain periods of time. In the initial development of the Fire Protection Program, applicable QA requirements were not fully addressed. Both violations resulted primarily from this oversight.

APS has taken extensive corrective actions, and others are being taken.

These actions are described in the attachments to this letter. Since the March 1989 event, APS has completed a number of corrective actions to improve emergency lighting and other aspects of fire protection. These actions include revision of the preventive maintenance program procedural controls governing the waiving of preventive maintenance tasks, modifications to add additional emergency lighting and to improve lighting system availability, increasing the preventive maintenance frequency on selected lighting units, and upgrading the corrective maintenance. priority for emergency lighting.

These actions demonstrate APS's recognition and acceptance of its responsibility to assure that emergency lighting meets the applicable NRC requirements and APS's commitments.

1 Violation A addresses APS's failure to fully implement its QA commitments applicable to the Fire Protection Program. This was caused by an ambiguity in the PVNGS Updated Final Safety Analysis Report (UFSAR) Table 3.2-1 and in the procedures that implement the QA requirements. This resulted in inconsistent application of the'riteria, both in the identification of structures, systems and components subject to the QA criteria, and in the implementation of the criteria to such structures, systems and components.

The effect of not fully applying the QA Program, as discussed in Reference 2, was that some QA requirements were not applied, although these omissions were generally compensated for by other programs and procedures.

Violation B addresses. emergency lighting system failures. Lighting availability, as discussed in Reference 3, is assured through compliance with NRC requirements for design, procurement, installation, testing and correction of identified deficiencies.,'These measures cannot prevent the occurrence of isolated equipment failures, but should assure that such failures are identified and promptly corrected. Reference 3 provided the NRC with an analysis of emergency lighting availability from June 1989 to June 1990. Seven of the eleven lighting failure examples cited in the violation

i I

Director, Office of Enforcement Nuclear Regulatory Commission Page 3 occurred prior to June l989, the remaining four were included in the availability analysis. The results of the analysis indicate that emergency lighting had a generally high availability and further analysis indicated that the availability of emergency lighting had improved over the past three years.

APS expects that the best lighting units will experience out-of-service periods, some of which may be prolonged by extenuating circumstances.

Compensatory measures, as recognized by the NRC guidance, have been established by APS to cover such periods. In APS's view, such out-of-service periods would not constitute a violation of NRC requirements unless APS fails to take required action when the lights become inoperable. However, since the failures were corrected in an untimely manner, APS does not contest the violation.

Reference l discusses the involvement of system engineers in maintenance activities. APS agrees that the emergency lighting system engineer was involved in corrective and preventive maintenance activities only when requested. The PVNGS System Engineer Program evidences a clear expectation that system engineers will be involved in maintenance't the time of these inspections the system engineer for emergency lighting was routinely involved in the following activities:

Performance of periodic walkdowns of the emergency lighting system.

Quarterly reviews of Failure Data Trending Reports, which summarized the Corrective Maintenance performed in the last quarter for components which had experienced maintenance activity that exceeded

.established thresholds.

APS does not dispute that routine reviews of emergency lighting test results by the system engineer were not performed; nor that such reviews could have been effective in identifying problems for corrective action. Although, APS does not expect system engineers to review every work order after closure, system engineers are expected to maintain cognizance of the condition of their respective systems. System engineers are generally very conscientious and have a good working interface with their maintenance counterparts. The details of this violation will be reviewed with system engineers as a lesson learned, to further improve this interface.

Director, Office of Enforcement Nuclear Regulatory Commission Page 4 If you have any questions or comments concerning the Reply to the Notice of Violation, this letter, or any other related matters, I would appreciate the opportunity to respond to them.

Sincerely, WFC/MEP/KLMC Attachments Enclosed Check: ¹50003050 CC: J. B. Martin (All W/ Attachments)

D. H. Coe A. CD Gehr A. H. Gutterman

161-03592-WFC/PKP/KLMC November 15, 1990 STATE OF ARIZONA )

) vs COUNTY OF MARICOPA )

I, William F. Conway, represent that I am Executive Vice President - Nuclear, that the foregoing document has been signed by me on behalf of Arizona Public Service Company with full authority to do so, that I have read such document and know its contents, 'and that to the best of my knowledge and belief, the statements made therein are true.

William F. onway Sworn to before me this ) S day of 1990.

Not y Public My Commission Expires:

My CovmissiptI Expires April 6 1%i

t APPENDIX A NOTICE OF VIOLATION AND PROPOSED IMPOSITION OF CIVIL PENALTY During inspections conducted from January 8 through March 23, and May 16 through August 31, 1990, two violations of NRC requirements were identified.

In accordance with the "General Statement Policy and Procedure for NRC Enforcement Action, " 10 CFR Part 2, Appendix C (1990), the Nuclear Regulatory Commission proposes to impose a civil penalty pursuant to Section 234 of the Atomic Energy Act of 1954, as amended (Act), 42 U. S. C. 2282, and 10 CFR 2.205. The particular violations and civil penalty are listed below.

A. Failure to Im lement Fire Protection ualit Assurance Pro ram License no NPF-41, Condition 2.C(7) for Palo Verde Unit 1, License No.

~

NPF-51, Condition 2.C(6) for Palo Verde Unit 2, and License No. NPF-74, Condition 2.F for Palo Verde Unit 3, provide in part:

APS shall implement and maintain in effect all provisions of the approved Fire Protection Program as described in the Final Safety Analysis Report (FSAR) for the facility, as supplemented and amended and as approved in the NRC Safety Evaluation Report (SER) through Supplement 8 for Units 1 and 2 and through Supplement ll for Unit 3.

FSAR Table 9B.3-1(C) requires the development and implementation of a Quality Assurance Program to satisfy the guidance of Branch Technical Position (BTP) 9.5-1 for design, procurement, installation, testing, and administrative controls for the Fire Protection Program for safety-related areas. The Table sets forth ten criteria, including those requiring adequate corrective actions and test controls.

BTP 9.5-1 defines the term "Fire Protection Program" as "the integrated effort involving all components, procedures, and personnel utilized in carrying out all activities of fire protection. It includes system and facility design, fire prevention, fire detection, annunciation, confinement, suppression, administrative controls, fire brigade organization, inspection and maintenance, training, quality assurance, and testing." (Emphasis added).

FSAR Section 17.2 '.2 provides; "the Operations QA program, as described in the Operations Quality Assurance Criteria Manual, shall be applied to Fire Protection Program activities associated with those fire protection systems and equipment used or installed in areas housing safety related equipment, and other areas where an unsuppressed fire could potentially damage safety-related structures, systems or components."

I

)l I

The original Palo Verde Safety Evaluation Report (SER) dated November 1981, Section 9.5.1.7, approved the implementation by the licensee of a Fire Protection Program consistent with the provisions of the NRC staff's guidance in "Nuclear Plant Fire Protection Functional Responsibilities, Administrative Controls and Quality Assurance," dated August 19, 1977

("FRAQUA"), including those provisions concerning quality assurance. to the FRAQUA guidance recommends a Quality Assurance Program that applies, among other things, to emergency lighting.

Contrary to the above, at the time of the inspections and since initial startup of each unit, emergency lighting system components required as part of the Fire Protection Program were classified as Non-Quality Related (NQR), and thus were not subject to the Quality Assurance provision of administrative procedures and did not meet the FRAQUA provision concerning tested and deficiencies were not properly corrected.

In addition, the licensee's letter of July 20, 1990 identifies additional items of the Fire Protection Program improperly classified as NQR to which the appropriate provision of the Operations Quality Assurance Manual or the FRAQUA had not been applied.

I ATTACHMENT 1 REPLY TO NOTICE OF VIOLATION A I. DMISSION OR DENIAL OF'VIOLATION A APS admits the violation.

II. REASON FOR VIOLATION A The NRC, during review of the application for Operating Licenses for PVNGS Units 1, 2, and 3, asked APS in Question 260.10 to clarify an apparent inconsistency between FSAR Tables 17.2-1 and 3.2-1. Table 17.2-1 indicated that the Fire Protection Quality Assurance Program is met as part of the Quali,ty Assurance Program under 10 CFR Part 50, Appendix B. Table 3.2-1 indicated that the Quality Assurance Program did not apply to the fire protection systems. In response to this question, APS amended Table 3.2-1, Section 17.2.2.2 and Table 17.2-1 and provided the following answer to NRC Question 260.10:

"Amended table 17.2-1 indicates that the Quality Assurance Program during the operations phase complies with the Quality Assurance Program guidelines of Appendix A to (BTP)

APCSB 9.5-1. Application of the Quality Assurance Program to Fire Protection Program activities is described in amended paragraph 17.2.2.2. Amended table 3.2-1 indicates that the pertinent requirements are applied 'o Fire Protection Program activities during the operations phase."

Amended Table 3.2-1 listed certain fire protection system components, but did not include emergency lighting and certain other fire protection system components (other than supports and hangers)..

In-the design and construction phase, PVNGS used a cl'assification system of "Q", "R" and "S" to define the application of Quality Assurance Program requirements. Items classified as "Q" were considered safety related and required full application of the 10CFR50, Appendix B, QA program. Class used to identify that some quality assurance controls were applicable for "R"'as equipment important to power generation. Class "S" identified the remaining items which were designed, procured'and installed in accordance with industry practice. The Emergency Lighting Syst'm was classified as quality class "S" during the design and construction phase of Palo Verde.

During the transition from construction to operations, the APS Operations Quality Assurance Program was implemented and the quality classification system was modified to allow proper application of the QA program to operations phase activities'tems previously classified as "Q" became "SR" and were, subject to the full 10CFR50, Appendix B, QA program.

1

4 I

Items previously classified "R" or "S" were reclassified Important-to-Safety

("ITS") only if a specific commitment was identified in Table 3.2-1 requiring application of the QA Program. When developing and implementing this methodology for reclassification it was not recognized, due to an ambiguity in UFSAR Table 3.2-1, that other commitments (e.g. Sections 9.5 and 17.2) should also be used when classifying plant components. As a result, the emergency lighting system and certain other fire protection system components (except supports and hangers) were classified as non-quality related (NQR).

As operational experience increased, APS recognized the need for a more detailed classification program for systems, structures and components including documentation of the basis for the classification. In 1988, a new classification procedure was adopted and a project was initiated which provided for a detailed evaluation of major components in the PVNGS equipment database. The component contribution to safety was defined and the basis for classifying the item as safety related ("Q"), quality augmented (QAG) or non-quality related (NQR) was established. The new classification process determined, in the spring of 1989, that the emergency lighting required to meet 10CFR50, Appendix R, should be classified QAG ~

Corrective Action Report (CAR) 90-10, initiated on March 19, 1990, identified deficiencies in the timeliness of completing procedure changes and other implementing document changes associated with classifying the emergency lighting system QAG. The CAR also specifically cited failure to include maintenance and procurement activities associated with emergency lights under the QA Program. Following completion of the corrective actions prescribed by this CAR and associated corrective action documents, the QA Program will be applied to the Emergency Lighting System as required by APS commitments.

The Fire Protection QA Program deficiencies were evaluated as part of an Incident Investigation, performed in July and August of 1990, which identified the following root causes:

The Classification process for structures, systems and components did not consider the requirements of the Operations Quality Assurance Criteria Manual (OQACM), Appendix 1, and they may not have been classified correctly per APS'ommitments in all cases.

The Fire Protection Test Program did not encompass the entire scope of testing required by the QA Program.

The fire protection impairment pr'ocedure had not been applied to the full scope of equipment defined in the UFSAR section 9.5.

QA had not consistently initiated corrective action documents as a result of audit findings identifing portions of the Fire Protection System, and associated activities that were being treated as non-quality related.

The administrative program which controls procedures did not require the identification of those procedures used for the conduct of quality related activities.

III. CORRECTIVE STEPS THAT HAVE BEEN TAKEN AND THE RESULTS ACHIEVED APS prepared a Justification for Continued Operation (JCO) to address the application of the QA Program to fire protection system equipment, including Appendix R emergency lighting. During this process, the following actions were completed:

The OQACM was evaluated for compliance with the Branch Technical Position, 9.5-1, Appendix A position on QA criteria.

The PVNGS administrative controls were evaluated to assure that programmatic controls were in effect to implement the applicable QA criteria.

The fire protection equipment classification was evaluated for completeness and accuracy.

The Fire Protection Program was evaluated for completeness and adequacy.

Interim compensatory measures were implemented to address identified equipment or Quality Assurance Program deficiencies.

An Incident Investigation was conducted to identify root causes and initiate corrective actions.

As described in Reference 2, many of the QA requirements which should have been applied to Fire Protection have, as a practical matter, been compensated for by other procedures and programs. Nevertheless, certain Fire Protection Program deficiencies remain. None of these remaining deficiencies compromise the ability to achieve and maintain safe shutdown in the event of a fire.

As a result of the investigation during preparation of the JCO and the subsequent performance of the Incident Investigation, activities were initiated to fully address the identified Fire Protection Program deficiencies and to address their more generic implications. Specifically:

1) Corrective actions were initiated to address the deficiencies identified in the Incident Investigation Report (IIR). Briefly, these actions included a review of the Fire Protection Program for proper classification, revision of the fire protection test and impairment programs to ensure the'roper scope, and revision of the administrative program for control of procedures to assure proper classification of procedures.

2) A review of the Fire Protection QA Program was initiated to identify any required changes to the program.

3) A Fire Protection review, as delineated below, was initiated to resolve action items/issues which resulted from the JCO effort, and

4) A separate quality augmented requirements review, also delineated below, was -.initiated to conduct a comprehensive review of the implementation of the PVNGS QA Pr'ogram for systems, equipment and activities in areas other than fire protection where requirements exist to apply the Quality Assurance QAG Program.

FIRE PROTECTION REVIEW APS undertook development of a comprehensive listing of open items and commitments soon after completion of the response to Inspection Report Number 90-25. The listing of action items was derived, in part, from the following:

1) Recent regulatory correspondence (since January 1990) related to emergency lighting or fire protection,

2) Several internal documents identifying open questions or concerns related to emergency lighting or fire protection, and

3) The ABB/Impell Corporation Emergency Lighting Study Report, which was transmitted to the NRC by letter dated August 3, 1990.

This review identified over 200 action items. A number of these items are related to the evaluation of the Quality Assurance Program applicable to fire protection system equipment. The effort to address these action items was estimated to require approximately 20,000 man-hours. APS management has established a dedicated,,multi-discipline team with a full-time proj ect manager to develop a detailed plan and schedule to address all open action items. The evaluation of the application of fire protection quality assurance requirements is complete (see APS's letter to the NRC, dated November 13, 1990; 161-03585).

Currently, the multi-discipline fire protection review team has completed approximately 20% of the open actions. The priority assigned to the actions is based on their importance to quality assurance applicability decisions, equipment concerns and concerns related to application of design criteria.

Reviews to date have not identified any operability or safety concerns not previously identified in Reference 2. The fact that no additional safety or operability concerns have been identified, provides confidence that operation under the JCOs, documented in Reference 2, continues to be valid and acceptable.

Should further deficiencies be identified, they will be resolved through the normal PVNGS Corrective Action Program. Also, the need for compensatory measures will be evaluated, and applied as needed.

U, UALITY AUGME TED P OGRAM REVIEW To address the generic implications of the violation, APS initiated a Quality Augmented Program Review, which has completed the following:

Determined scope and depth of QA requirements applicable to PVNGS by reviewing NRC regulations, UFSAR, Operating License Conditions and the OQACM to identify functional areas/equipment groups which are subject to a Quality Augmented Program.

Developed a list of systems and equipment from the identified functional areas/equipment groups which are subject to the requirements of a QAG program (except for expendables/consumables, security and Reg Guide 1.97 which are still in process).

Reviewed procedure 81AC-OCC06 "Quality Classifications for Structures, Systems, Components, Spare Parts and Activities" for completeness and clarity of instruction. A draft revision of the existing procedure has been prepared. In addition, another procedure was drafted to address the classification of quality related activities.

The work currently encompasses in-depth reviews of the systems/equipment and activities to determine compliance with the applicable QA criteria.

The first systems to be reviewed are the Radwaste System, RP Calibration Standards, and Emergency Plan Equipment. Should additional deficiencies be identified, they will be resolved through the normal PVNGS Corrective Action Program. Also the need for a JCO and compensatory measures will be evaluated, and addressed as needed.

IV, CORRECTIVE STEPS WHICH WILL BE TAKEN TO AVOID FURTHER VIOLATIONS The Quality Augmented Review will identify two types of actions requiring follow-up:

a) Actions that are related to the general applicability of QA criteria to specific systems/equipment and/or activities, and b) Actions necessary to address technical questions concerning specific systems/equipment or activities.

I I

It I

1

The actions that are related to the general application of the QA Criteria will be evaluated collectively. It is anticipated that these changes will be completely evaluated and proposed changes to the QA Program for QAG systems, equipment and activities at PVNGS will be finalized in February 1991. A plan to incorporate any necessary changes resulting from this evaluation into the QA Program for QAG systems, equipment and activities will be developed in April 1991.

V. DATE WHEN FULL COMPLIANCE WILL BE ACHIEVED Compensatory measures were established for the Fire Protection Program QA deficiencies as a result of the JCO effort. These will remain in effect until necessary corrective actions are completed.

The Fire Protection and Quality Augmented Program reviews are expected to complete their activities in February 1991. The resulting revisions to procedures, drawings, equipment classifications, database, etc. necessary to satisfy the QA Program requirements for QAG systems, equipment and activities will be completed by the end of 1991.

li t

APPENDIX B NOTICE OF VIOLATION AND PROPOSED IMPOSITION OF CIVIL PENALTY, B. EMERGENCY LIGHTING SYSTEM FAILURES .

License No. NPF-41, Condition 2.C(7) for Palo Verde Unit 1, License No.

NPF-51, Condition 2.C(6) for Palo Verde Unit 2 and License No. NPF-74, Condition 2.F for Palo Verde Unit 3, provide in part:

APS shall implement and maintain in effect all provisions of the approved fire protection program as described in the Final Safety Analysis Report (FSAR) for the facility, as supplemented and amended, and as approved in the SER through Supplement 8 for Uni.ts 1 and 2 and through Supplement 11 for Unit 3.

FSAR Section 9.5.1.1.1 (R), Safety Design Basis Eighteen, states in part:

"Emergency lighting systems shall be provided in accordance with the guidance provided in NRC Branch Technical Position (BTP) APCSB 9.5-1.. ~

Batteries for emergency lighting shall be rated for a minimum of 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />..."

NRC BTP APCSB 9.5-1, Appendix A, recommends suitable fixed emergency lighting with 8-hour minimum battery power supplies for safe shutdown equipment and in access and egress routes thereto.

FSAR Table 9B.3-1, D.5 (a), requires emergency lighting units with at least 8-hour battery power supplies in all areas needed for operation of safe shutdown equipment and in access and egress routes thereto.

Contrary to the above, on numerous occasions while Units 1, 2, and 3 were operating in Mode 1 between initial startup and the time of the inspections, the licensee did not have available emergency lighting units required for operation of safe shutdown equipment and in access and egress areas thereto that would have been able to operate for at least 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />.

Specifically, from 1987 to 1990, a significant number of the lighting units failed to operate or would not have been able to operate for the full eight hours if required during an emergency as evidenced by examples (for each lighting unit listed below, its respective the'ollowing reactor was in Mode 1 for a significant part of the time that the lighting unit was unavailable):

Unit 1:

Control Room emergency lighting unit 1EQDNN01 was not available for approximately 12 weeks in that the specific gravity of its electrolyte was low, i.e., outside licensee acceptance criteria,

t following a discharge test on February 3, 1987. Following the discharge test, the unit did not receive an equalizing charge and was not demonstrated to be satisfactorily recharged until April 5, 1987.

2. Control Room emergency lighting unit 1EQDNN02 was not available for approximately 21 weeks, due to low specific gravity following a discharge test on February 5, 1987. The unit required several equalizing changes and was not demonstrated to be satisfactorily recharged until July 6, 1987.

3. Control Building emergency lighting unit 1EQBN001 was not available for approximately on January 28, 1988.

ll weeks, following failure of a discharge test During the test, the inverter smoked and the unit breakers remained open until the unit was'restored on April 13, 1988.

4. Control Room emergency lighting unit lEQDNN01 was not available for approximately 29 weeks, due to low specific gravity problems following a discharge test on January 29, 1988. The unit required several equalizing charges and was not demonstrated to be satisfactorily recharged until August 19, 1988.

5. Control Room emergency lighting unit 1EQDNN02 was not available for approximately 34 weeks, due to low specific gravity problems following a discharge test on January 29, 1988. Although the battery received several recharges during periodic preventive maintenance activities, low specific gravity problems continued until the battery bank was replaced on September 29, 1988. The unit did not satisfy test requirements until February 27, 1990.

Unit 2

6. Control Building emergency lighting unit 2EQBN002 was not available for approximately 61 weeks, following identification of a defective inverter transfer relay on March 8, 1987. The unit failed two consecutive discharge tests (September 9, 1987 and May 5, 1988) due to the defective relay. No discharge time was achieved during the first test and only four hours of discharge were achieved during the second test. The unit was not'epaired until May 12, 1988. The unit did not satisfy test requirements until July 3, 1989.

7. Control Room emergency lighting unit 2EQDNN02 was not available for approximately seven weeks, due to,low specific gravity following a discharge test on December 4, 1989. Battery bank recharging did not satisfy test requirements until January 27, 1990.

8. Control Room emergency light unit 2EQDNN01 was not available for approximately 15 weeks, following failure of a discharge test on January 3, 1990. The unit failed two consecutive discharge tests (January 3, 1990 and April 5, 1990), operating only about 7-1/2 hours during each of these tests. The unit was not restored to operable condition until April 20, 1990.,

Unit 3'uxiliary Building emergency lighting unit 3EQBN003 was not available for approximately 52 weeks, following failure of the lights to operate during a discharge test on April 17, 1987. This unit also failed two successive preventive maintenance tests on February 8, 1988 (six of eight cells were missing from the battery bank) and on March 4, 1988 (all eight cells were missing from the battery bank). The batteries were not replaced and did not satisfy test requirements until April 19, 1988.

10. Control Building emergency lighting unit 3EQBN002 was not available for approximately 31 weeks, Xollowing the discovery of low battery bank voltage and damaged battery cells on September 12, 1987. This unit also failed two successive preventive maintenance tests on February 5, 1988 and March 7, 1988, due to continuing low battery bank voltages. Furthermore, during the March 1988 maintenance activity, two of the eight cells in the battery bank were found to be missing. The batteries were not replaced and did not satisfy test requirements until April 18, 1988.

ll. Auxiliary Building emergency lighting unit 3EQBN003 was not available for approximately six weeks, following failure of a discharge test on May 3, 1990, during which the lights operated only 6-1/2 hours. The battery and inverter unit were not replaced and did not satisfy test requirements until June 18, 1990.

I ATTACHMENT 2 REPLY TO NOTICE OF VIOLATION B I. DMISSION OR DENIAL OF VIOLATION B The examples cited in the Notice of Violation represent violations of two specific NRC requirements. In examples 1 through 10, APS failed to take prompt corrective action to correct identified deficiencies. Examples 2, 4, 5, 7, and 8 illustrate a repetitive failure; in that the cause of the deficiency was not properly identified and corrected to preclude repetition.

In example 11, a Holophane MPS (3EQBN003) failed a discharge test.

Troubleshooting activities began immediately and continued for approximately three weeks, including replacement of five of the eight batteries and performance of three additional tests. At the end of three weeks a decision was made to correct the deficiency by replacing the inverter with a different model, implementing a design change that had been previously scheduled to resolve the design deficiency reported in June 19, 1990 correspondence.

Expediting delivery and installation required approximately four more weeks.

APS believes its actions in this example were appropriate.

II. REASON FOR VIOLATION B For the most part, the examples cited in the violation occurred several years ago, and APS must rely on a review of maintenance records to determine the reasons for these events. The results of the record review are detailed in Appendix l.

The cause of the failures to take prompt action to correct the identified deficiencies noted in the Notice of Violation can generally be attributed to the assignment of an inappropriately low work priority.

The reason for the failure to identify the cause of significant conditions adverse to quality and take corrective action to preclude repetition is that required root cause analysis of significant equipment failures was not applied to emergency lighting, prior to March 1990.

III. CORRECTIVE STEPS THAT HAVE BEEN TAKEN AND RESULTS ACHIEVED Failure to take prompt corrective action As stated in the response to Notice of Violation 50-530/90-08-02, PVNGS management has promulgated guidance for prioritization of corrective maintenance for the operability of emergency lights.

This guidance directs that such work should be treated as priority

I 2 (i.e., work should commence within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />). This guidance, issued in March 1990, will remain in effect until alternate measures are adopted to assure that appropriate priority is assigned to such corrective maintenance.

B. Failure to identify the cause of significant conditions adverse to quality and take corrective action to preclude repetition.

Emergency Lighting has been classified as QAG, and the Nonconformance Program has been applied to the emergency lighting system since March of 1990. A review was conducted of the quality classifications of all emergency lighting components to assure all components were classified as QAG. Required corrections to the classifications were completed in October '1990. The current procedure that governs the "Control of Nonconforming Items" requires that, "Nonroutine equipment failures identified by MNCRs shall require a root cause analysis to assure the cause of the condition is determined and appropriate corrective action is taken to prevent recurrence."

IV. CORRECTIVE STEPS THAT WILL BE TAKEN TO AVOID FURTHER VIOLATIONS A. The corrective action identified above is appropriate to assure that identified deficiencies in emergency lighting are promptly corrected.

B. Quality Assurance will perform a review to verify that the Nonconformance Program is being implemented properly for emergency lighting. The review will be done prior to April 1, 1991.

V. DATE WHEN FULL COMPLIANCE WILL BE ACHIEVED PVNGS will be in full compliance with NRC requirements upon completion of the activities associated with Violation A, as documented in Attachment 1.

APPENDIX 1 ROOT CAUSE ANALYSIS

E I

APPENDIX 1 ROOT CAUSE ANALYSIS INTRODUCTION Appendix 1 discusses the root cause analysis of the eleven (11) cited examples in Notice of Violation B. The root cause analysis for each of the examples was performed by a review of the available maintenance records.

Included within this appendix are Exide battery bank availability determination criteria. For purposes of evaluation, APS has defined Exide battery bank availability necessary to support an 8-hour discharge as follows:

If specific gravities are low in four (4) or less cells, the battery bank is considered available provided none of the cells indicate a short circuit condition.

Pilot cells are considered a representation of the entire battery bank (i.e., one pilot cell represents 15 cells or 25%

of the entire bank) in Weekly and Monthly PMs. Thus, if one pilot cell indicates a low specific gravity, the battery bank is considered unavailable until the specific gravities are either documented or evaluated to be within acceptance criteria again.

I I

f

Example 1. Equipment ID: 1-E-QDN-N01 Work Document Review:

4 The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WO¹ Result 2/13/87 203036 Eight Hour Discharge - The 2/3/87 data referenced in the 8/15/90 correspondence to the NRC is incorrect. The discharge test was begun on 2/12/87 and successfully completed on 2/13/87.

From a review of available records, the UPS batteries were not placed on an equalizing charge following this discharge test. It should be noted that these battery banks are connected to a continuous float charge.

2/19/87 205674 Weekly PM - Pilot cell specific gravities failed to meet acceptance criteria.

2/26/87 210385 Equalizing Charge - Failed to bring specific gravities within acceptance crit'eria.

3/03/87 211573 Equalizing Charge - Failed to bring specific gravities within acceptance criteria.

3/07/87 206722 Quarterly PM - Continued to find specific gravities not meeting the acceptance criteria.

3/12/87 209153 Weekly PM - the four pilot cells were found to meet the specific gravity acceptance criteria.

Emergency lighting unit 1-E-QDN-N01 availability was established with all four pilot cells meeting the specific gravity acceptance criteria during the performance for the Weekly PM (WO 209153) on March 12, 1987.

Therefore, the period of unavailability is from February 13, 1987, to March 12, 1987, or approximately four weeks.

l As demonstrated below, the specific gravities remained within the acceptance criteria.

Date WO¹ Result 3/19/87 209154 Weekly PM - Pilot cell specific gravities continue to increase slightly and continued to remain within acceptance criteria.

3/25/87 212271 Weekly PM - Pilot cell specific gravities continue to increase slightly and continued to remain within acceptance criteria.

Analysis of Root Cause:

Failure to take prompt corrective action, in that the equalize charge was not initiated immediately following a discharge to recharge the battery bank. Further, subsequent to the second failure of the equalizing charge, no further corrective action was initiated.

P Example 2. Equipment ID: 1-E-QDN-N02 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WOO Result 2/13/87 203036 Eight Hour Discharge - The 2/5/87 data referenced to the NRC is

'n the 8/15/90 correspondence incorrect. The discharge test was begun on 2/12/87 and successfully completed on 2/13/87.

From a review of available records, the UPS batteries were never placed on equalizing charge following the discharge test. It should be noted that these battery banks are connected to a continuous float charge.

2/19/87 205675 Weekly PM - Pilot cell specific gravities failed to meet acceptance criteria.

2/26/87 210386 Equalizing Charge - Failed to bring specific gravities within acceptance criteria.

3/03/87 211574 Equalizing Charge - Failed to bring specific gravities within acceptance criteria.

3/07/87 '209157 Quarterly PM - Continued to show unavailability when nine cells were found below acceptance criteria.

3/07/87 208455 Weekly PM - Found two pilot cells below acceptance criteria.

3/12/87 209155 Weekly PM - All pilot cells within acceptance criteria.

Emergency lighting unit 1-E-QDN-N02 availability was established with all four pilot cells meeting the specific gravity acceptance criteria during the performance for the Weekly PM (WO 209155) on March 12, 1987.

Therefore, the period of unavailability is from February 13, 1987 to March 12, 1987, or approximately four weeks.

'1 As documented below, the specific gravities remained within the acceptance criteria.

Date WO¹ Result 3/19/87 209156 Weekly PM - Pilot cell specific gravities continued to meet acceptance criteria.

3/25/87 , 272272 Weekly PM - Pilot cell specific gravities continued to meet acceptance criteria.

4/02/87 213325 Weekly PM - Pilot cell specific gravities continued to meet acceptance criteria.

4/08/87 215105 Weekly PM - Pilot cell specific gravities continued to meet acceptance criteria.

4/18/87 216307 Weekly PM - Found one pilot cell below acceptance criteria of 1.205. A review of Weekly PMs before and after WO 216307 indicates the specific gravities taken on May 18, 1987 may not have been accurate (see Table 1 below) since no testing or work was performed and the following nine weekly PM inspections indicated no specific gravity deficiencies.

Table 1 Cell 0 Spec. Grav. Spec. Grav. Spec. Grav.

4/8/87 4/18/87 4/23.87 1.209 1. 203 1.211 27 1. 211 1.205 l. 211 1.217 1.207 1.214 50 1. 219 1.209 1.216 An additional period of unavailability occurred in June 1987. This work document review is detailed belo~.

Date WO¹ Result 6/17/87 93482 Corrective Maintenance - This work order was initiated on 6/14/85, . but not worked until 6/17/87. Since no documentation show existing deficiencies in the batteries, as of 6/17/87, the battery replacement is considered normal maintenance. Corrective maintenance was successfully completed on 6/23/87.

l

'f t

I

Emergency lighting unit 1-E-QDN-N02 availability was established within the Corrective Maintenance Work Order itself and its completion on June 23, 1987.

In summarizing, APS considers 1-E-QDN-N02 to have been unavailable in two separate and independent timeframes:

1) from February 13, 1987 to March 12, 1987 and

2) from June 17, 1987 to June 23, 1987, or a total unavailability time of approximately five weeks.

Analysis of Root Cause:

Failure to take prompt corrective action, in that the equalizing charge was not initiated immediately to recharge the battery bank. Further, subsequent to the second failure of the equalize charge, no further corrective action was initiated.

(

f I

Example 3. Equipment ID: 1-E-QBN-001 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WO¹ Resu t 1/28/88 WR 134798 Faulty LED display and smoke emitting from the Modular Power Station (MPS) 'unit. To prevent further damage, both the input and output breakers were opened.

2/25/88 273845 Monthly PM - Verified deenergized state of 1-E-QBN-001.

2/29/88 274363 Corrective Maintenance - As a result of WR 134798, troubleshooting identified one defective battery and a defective transformer. A decision was made to replace four batteries instead of only the one defective battery. Four batteries were replaced and remaining four good batteries removed to an external charger. (See 4/13/88 for completion of corrective maintenance) 3/21/88 278052 Monthly PM - Confirmed only four batteries installed.

4/13/88 274363 Corrective Maintenance - Four batteries reinstalled.

4/22/88 ,

269330 Eight Hour Discharge - Performed successful eight hour discharge to complete retest requirement identified in corrective maintenance WO 274363.

Emergency lighting unit 1-E-QBN-001 availability was established by meeting the acceptance criteria of the eight hour discharge test (WO 269330) on April 22, 1988. Therefore', the period of unavailability is from January 28, 1988, to April 22, 1988, or approximately 12 weeks.

Analysis of Root Cause:

Failure to take prompt corrective action, in that corrective maintenance was not begun or completed in a timely manner.

I

'l I

f

Example 4. Equipment ID: 1-E-QDN-N01 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WO¹ Result 1/29/88 273899 Eight Hour Discharge - Successful eight hour discharge.

1/30/88 273485 Equalizing Charge - None of the pilot cells met the specific gravity acceptance criteria. Work Request 224848 was initiated to replace battery bank.

2/08/88 269030 Monthly PM - Two of four pilot cells failed to meet specific gravity acceptance criteria. This gives an indication that the specific gravities were increasing as a result of the continuous float charge.

4/05/88 278013 Monthly PM - All four pilot cells meet specific gravity acceptance criteria.

Emergency lighting unit 1-E-QDN-N01 availability was established with the four pilot cells meeting the specific gravity acceptance criteria during the performance of the Monthly PM (WO 278013) on April 5, 1988.

Therefore, the period of unavailability is from January 29, 1988 to April 5, 1988, or approximately 9.5 weeks.

As demonstrated below, the specific gravities were either documented or evaluated as meeting the minimum acceptance criteria for the following three PMs.

Date WO¹ Result 5/10/88 284301 Monthly PM - All four pilot cells meet specific gravity acceptance criteria.

5/11/88 273818 Annual PM - Sixteen specific gravities fail to meet acceptance criteria. (See explanation provided below) 6/02/88 286799 Monthly PM - All four pilot cells meet specific gravity acceptance criteria.

t APS evaluation of the data collected during the Annual PM performed on May ll, 1988 is that the data taken for pilot cells 27 and 50 were incorrect and the PM results were invalid. This conclusion is supported by the fact that monthly PM's performed immediately prior and soon after found specific gravities above the minimum acceptance criteria of 1.205. (See Table 2 below).

Table 2 Cellf Spec,Grav. Spec. Grav. Spec. Grav. Spec. Grav.

4/8/88 5/10/88 5/11/88 6/2/88 4 1. 215 1. 210 1. 210 1. 210 27 1.214 1.210 1.194 1. 210 41 1.224 1.224 l. 224 1.222 50 1. 220 1. 216 1.206 1. 218 Analysis of Root Cause:

Failure to take prompt corrective action, in that no action was taken to restore unit availability between February 8, 1988 and April 5, 1988, when the battery again met acceptance criteria.

2. Failure to identify the cause of significant conditions adverse to quality and take corrective action to preclude repetition, in that difficulty in recharging the battery bank was identified one year earlier (Example 1) with no subsequent action to evaluate the condition.

Example 5. Equipment ID: 1-E-QDN-N02 Work Document Review:

The following timeline begins with the date the lighting unit'as considered out of service and ends on the date the unit was considered available.

Date WOO ges~t 1/29/88 273899 Eight Hour Discharge - Successful discharge test performed.

1/30/88 273823 Equalizing Charge - Specific gravity acceptance criteria not met for 56 cells.

WR 224847 was initiated to replace the battery bank. The battery bank remained on float charge.

2/29/88 275597 Corrective Maintenance - Work was begun to replace battery bank. However, no physical work was performed until 9/12/88 due to unavailability of replacement batteries. The battery replacement was completed on 9/29/88.

3/08/88 273816 Quarterly PM - Although the quarterly PM indicated one cell below specific gravity acceptance criteria, the other 59 cells were acceptable, thus the battery bank was available. This is further supported by the fact that the following five monthly PMs and one quarterly PM were performed without any deficiencies.

Emergency lighting unit 1-E-QDN-N02 availability was established with all four pilot cells meeting the specific gravity acceptance criteria during the performance of the Quarterly PM inspection (273816) on March 8, 1988.

Therefore, the period of unavailability is from January 29, 1988 through March 8, 1988, or approximately six weeks.

In trying to cover the same timeframes as the NRC, a review of an identified deficiency was provided for WO 302678 to once again determine if an availability issue existed. The two following work orders indicate availability was maintained at this time period.

10

f f

I I

Date WOO Result 8/19/88 302678 Quarterly PM - Identified pilot cell 44 failed to meet the specific gravity acceptance criteria.

Since cell t4 is a pilot cell, it would be concluded on a Monthly PM that the battery bank was unavailable. However, since this was a Quarterly PM all cells were examined, and the other 59 cells showed no deficiency. Therefore, the unit was considered available.

8/29/88 29955 Monthly PM - all four pilot cells met specific gravity acceptance appears the 8/19/88 criteria.

reading Therefore, have it may been incorrect.

As a result of WO 275579, an additional period of unavailability occurred in September 1988. This work document review is detailed below.

Date WO¹ Result 2/29/88 275579 Corrective Maintenance - The batteries were removed on 9/12/88, which resulted in the UPS becoming unavailable. From 9/12/88 to 9/29/88, the batteries were removed, 20 new batteries prepared and installed, terminated, and charged.

Availability for this time period was established with completion of the Corrective Maintenance activities of WO 275579 on September 29, 1988.

Therefore, the period of unavailability is from September 12, 1988 to September 29, 1988, or approximately two weeks. This period of unavailability represents a relatively prompt response to corrective maintenance activities.

In summarizing, APS considers 1-E-QDN-N02 to be unavailable in two totally separate and independent timeframes. Unavailability periods for this emergency lighting unit are

1) from January 29, 1988 to March 8, 1988 and

2) from September 12, 1988 to September 29, 1988, or a total unavailability time of approximately eight weeks.

11

l~

Analysis of Root Cause:

Failure to take prompt 'corrective action, in that there was no action to restore unit availability between January 29, 1988 and March 8, 1988.

Failure to identify the cause of significant conditions adverse to quality and take corrective action to preclude repetition, in that difficulty in recharging the battery bank was identified one year earlier (Example ¹2) with no subsequent action to evaluate the condition.

The second period of unavailability does not appear to represent a program deficiency.

12

Example 6. Equipment ID: 2-E-QBN-002 Work Document Revie~:

The following timelines document two periods of unavailability for the lighting unit. In each timeline the timeline begins with the date unavailability began. The final date is the date the lighting unit was made available again.

WO¹ Result 3/01/87 WR 233485 Work request 233485 was initiated to investigate what appeared to be a high Modular Power Station (MPS) output and two hot batteries.

3/08/87 211542 Corrective Maintenance - Found a loose connection at the load indicator. Reestablishing the connection resolved the condition identified by WR 233485.

3/10/87 208641 Monthly PM - Retest required by WO 211542 completed with acceptable results.

For the first time period, emergency lighting unit 2-E-QBN-002 availability was established with completion of Monthly PM (WO 208641) on March 10, 1987. Therefore, the period of unavailability is from March 1, 1987 to March 10, 1987, or approximately 1.5 weeks. This period of unavailability represents a relatively prompt response to corrective maintenance activities and indicates no particular program deficiencies.

Date WO¹ Result 9/08/87 216000 Eight Hour Discharge - Failed test due to lights blinking 'on'nd 'off'. Work Request 195584 was initiated.

2/11/88 257054 Corrective Maintenance - Work order issued to troubleshoot, 'no deficiencies found, no test activities performed, and work order closed.

5/05/88 282626 Eight Hour Discharge - Test failed again due to lights blinking 'on'nd 'off'.

5/16/88 293202 Corrective Maintenance - Replaced Short Circuit Detector (SCD) board and performed functional retest. However, no eight hour discharge test was performed.

13

I' For the second time period, emergency lighting unit 2-E-QBN-002 availability was established with the replacement of the SCD board and functional testing. Therefore, the period of unavailability is from September 9, 1987 to May 16, 1988, or approximately 36 weeks.

In summarizing, APS considers 2-E-QBN-002 to be unavailable in two totally separate and independent, timeframes. Unavailability periods for this emergency lighting unit are

1) from March 1, 1987 to March 10, 1987 and

2) from September 9, 1987 to May 16, 1988, or a total unavailability time of approximately 37 weeks.

Analysis of Root Cause:

The first period of unavailability does not appear to represent a program deficiency.

The cause for the deficiency represented by the second 'period of unavailability is as follows:

Failure to take prompt corrective action, in that the first attempt at corrective maintenance was five months after the deficiency was identified. This attempt was ineffective. Subsequent corrective maintenance and the functional retest to assure the identified deficiency was corrected was not completed for an additional three months.

14

I Example 7. Equipment ID: 2-E-QDN-N02 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WO¹ Jesuits 12/04/89 387946 Eight Hour Discharge - Terminated after four hours due to low illumination. WR 366832 was issued to relamp lighting fixtures.

12/04/89 376937 Equalizing Charge - Did not meet specific gravity acceptance criteria. No additional equalizing charge was performed or requested.

12/12/89 391079 Monthly PM - Specific gravity did not meet acceptance criteria for two pilot cells. WR 366834 was written to address deficient pilot cells. This resulted in the issuance of WO 400019. WO 400019 was cancelled by referencing WO 401624.

1/03/90 398915 Eight Hour Discharge - Successfully completed discharge test without benefit of the additional equalizing charge of WO 401624.

1/04/90 401624 Equalizing Charge - Did not meet specific gravity acceptance criteria.

1/15/90 395623 Monthly PM - Did not meet specific gravity acceptance criteria. As a result, APS issued WO 404283 to conduct an equalizing charge.

1/15/90 404283 Equalizing Charge - Did not meet specific gravity acceptance criteria.

1/27/90'05743 Equalizing Charge - Met specific gravity acceptance criteria on all but two cells.

Therefore, lighting unit was considered available.

Emergency lighting unit 2-E-QDN-N02 availability was established by meeting the specific gravity acceptance criteria following performance of an equalizing charge (WO 405743) on January 27, 1990. Therefore, the period of unavailability is from December 4, 1989 to January 27, 1990, or approximately eight weeks.

15

Analysis of Root Cause:

Failure to take prompt corrective action, in that failed equalizing charges did not result in further corrective action on December 4, 1989 and January 4, 1990.

2. Failure to identify the cause of significant conditions adverse to quality and take corrective action to preclude repetition, in that no action was initiated to determine the cause for the failure of the equalizing charge on December 4, 1989.

16

Example 8. Equipment ID: 2-E-QDN-N01 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WO¹ gesulus 1/3/90 398915 Eight Hour Discharge - Failed due to undervoltage relay tripping the input breaker.

1/8/90 401622 Equalizing Charge - Failed to meet specific gravity acceptance criteri'a.

1/15/90 404284 'qualizing Charge - Failed to meet specific gravity acceptance criteria.

1/23/90 405740 ,Equalizing Charge - Met specific gravity acceptance criteria. However, lighting unit was not considered available since a successful discharge test had not been completed.

4/5/90 418337 Eight Hour Discharge - Terminated after 7.75 hours8.680556e-4 days <br />0.0208 hours <br />1.240079e-4 weeks <br />2.85375e-5 months <br /> due to error in PM instructions.

Note: In Reference 3, this discharge test was evaluated as a successful test due to another test (WO 437694, run on July 31, 1990) which simulated the original test conditions, and was successful.

4/5/90 418654 Equalizing Charge - Failed to meet specific gravity acceptance criteria.

4/20/90 418654 Equalizing Charge - Second equalizing charge performed under the same work order met specific gravity acceptance criteria.

Emergency lighting unit 2-E-QDN-N01 availability was established with the successful discharge test of April 5, 1990 and the specific gravities meeting acceptance criteria on April 20, 1990. Therefore, the period of unavailability is from January 3, 1990 to April 20, 1990, or approximately 14 weeks.

17

Analysis of Root Cause:

Failure to take prompt corrective action, in that action to correct the cause of the relay trip on January 3, 1990 was not initiated.

Corrective action was ultimately performed in August 1990 with the implementation of Site Modifications 1,2,3-SM-QD-008.

2. Failure to identify the cause of significant conditions adverse to quality and take corrective action to preclude repetition, in that no action was initiated to determine the cause for the failure of the equalizing charges on January 8, 1990 and January 15, 1990.

18

Example 9. Equipment ID: 3-E-QBN-003 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

ate WO¹ Results 4/17/87 218633 Eight Hour Discharge - Failed to keep emergency lights lit.

8/16/87 222485 Corrective Maintenance - Initiated to resolve deficiency noted above. Troubleshooting activities determined batteries were disconnected.

Another work order was referenced as replacing the batteries. (However, no reference was made by number and no work order has been found describing this activity.)

The resulting corrective maintenance activity done under WO 222485 or the unknown WO removed six of eight batteries from the unit. (The dates of these activities have not been determined.)

2/08/88 271622 Monthly PM - Indicates batteries missing.

3/04/88 277033 Monthly PM - Indicates batteries missing.

4/14/88 222485 Corrective Maintenance - Batteries replaced.

4/19/88 282475 Eight Hour Discharge - Performed successful eight hour discharge test.

Emergency lighting unit 3-E-QBN-003 availability was established after the performance of a successful eight hour discharge (WO 282475) on April 19, 1988). Therefore, the lighting unit was unavailable from April 17, 1987 to April 19, 1988, or approximately 52 weeks'nalysis of Root Cause:

Failure to take prompt corrective action, in that corrective maintenance was not completed for one year after the deficiency was identified.

19

g 4

,f I

1

Example 10. Equipment ID: 3-E-QBN-002 Work Document Review The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

Date WO¹ Resu t 9/12/87 247122 Corrective Maintenance - Work activities removed batteries 3A and 3B. With two batteries removed, the unit was unavailable. After the removal of 3A and 3B, deficiencies were found in the remaining batteries. Efforts made to charge the batteries using internal Holophane charger unit failed. Work activities were suspended on September 17, 1987.

4/14/88 247122 Corrective Maintenance - All batteries replaced and the unit placed on charge. Batteries continued to fail specific gravity criteria.

Charger board was determined to be defective and was replaced. Batteries were placed on eight (8) hour equalizing charge and met acceptance criteria on April 14, 1988.

Emergency lighting unit 3-E-QBN-002 availability was established after the equalizing charge acceptance criteria was met (WO 247122) on April 14, 1988. Therefore, the lighting unit was unavailable from September 12, 1987 to April 14, 1988, or approximately 31 weeks.

Analysis of Root Cause:

Failure to take prompt corrective action, in that corrective maintenance was suspended for a period of approximately seven months.

1 20

,'I C

yl

Example 11. Equipment ID: 3-E-QBN-003 Work Document Review:

The following timeline begins with the date the lighting unit was considered out of service and ends on the date the unit was considered available.

DATE WO¹ RESULT 5/3/90 410539 Eight Hour Discharge - Failed after 6.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />.

Batteries were charged and brought into the specific gravity acceptance criteria. WR 393047 was initiated to replace the batteries.

5/4/90 418534 Corrective Maintenance - Loads on the inverter were verified.

5/6/90 418534 Corrective Maintenance - With no batteries available in the warehouse, five batteries were obtained from Unit 1 and placed on an external charger. l 5/7/90 410539 Eight Hour Discharge - Failed second discharge test, using the original batteries.

5/8/90 418534 Corrective Maintenance - Five batteries obtained from Unit 1 were installed in the unit.

5/9/90 410539 Eight Hour Discharge - Failed third discharge test using the newly installed batteries from Unit 1.

Engineering directed Maintenance to charge the batteries for 168 hours0.00194 days <br />0.0467 hours <br />2.777778e-4 weeks <br />6.3924e-5 months <br />, per vendor recommendations, prior to performing another discharge test.

5/22/90 410539 Eight Hour Discharge - Failed fourth discharge test.

5/23/90 418534 Corrective Maintenance - Troubleshooting activities found two fuses blown and replaced them. Additional troubleshooting activities resulted in the accidental shorting of wires which appeared to have an adverse affect on the output signal. Vendor assistance was requested.

21

gl 6/7/90 418534 Corrective Maintenance - New batteries arrived on site and were being prepared for installation when questions arose as to their acceptability. It was ultimately determined the batteries could not be used due to activities utilized in their onsite preparation.

6/12/90 429606 Corrective Maintenance - Work began on the replacement of 3-E-QBN-003 to the upgraded model as per recently designed modification 3-SM-QD-007.

6/15/90 430333 Corrective Maintenance - Batteries were installed in the new unit and charged. An eight hour discharge test was performed as required by the modification and the batteries recharged. Upon completion of the recharge, 6/20/90, the unit became available.

As a result of the deficiency identified on May 3, 1990, the inverter was out of service for approximately three weeks while troubleshooting was ongoing. This was unsuccessful in correcting the problem. Approximately four more weeks passed while a design change was implemented to change the inverter to a larger capacity model, as described in the June 19, 1990 letter to the NRC.

Emergency lighting unit 3-E-QBN-003 availability was established after the installation of the upgraded Holophane MPS (WO 429606) and the successful discharge test on June 15, 1990. Therefore, the lighting unit was unavailable from May 3, 1990 to June 20, 1990, or approximately seven weeks.

Analysis of Root Cause:

APS considers the actions taken to resolve deficiencies found with 3-E-QBN-003 to be appropriate. While the time of unit unav'ailability may appear to be excessive, activities to resolve the condition were being performed throughout the time period per existing programs and vendor recommendations.

22

i0 I