ML20056H339

Responds to Violations & Deviations Noted in Insp Repts 50-528/93-26,50-529/93-26 & 50-530/93-26.Corrective Actions: Test Log for Each Surveillance Test Updated to Properly Document Identified Problems

ML20056H339
Person / Time
Site:	Palo Verde
Issue date:	08/30/1993
From:	Conway W ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR
To:	NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
References
102-02623-WFC-T, 102-2623-WFC-T, NUDOCS 9309090169
Download: ML20056H339 (22)

Text

_ _ _ _ _ - -

Arizona Public Service Company P O BOv53999

• PHOEN:x. ARIZONA E5072-3999 102-02623-WFC/TRB/PJC August 30,1993 wi g- 'f,+i,,c g g U. S. Nuclear Regulatory Commission ATTN: Document Control Desk Mail Station P1-37 Washington, D. C. 20555

Reference:

Letter dated July 30,1993, from C. A. VanDenburgh, Chief, Reactor Projects Branch, NRC, to W. F. Conway, Executive Vice President, Nuclear, APS

Subject:

Palo Verde Nuclear Generating Station (PVNGS)

Units 1,2, and 3 Docket Nos. STN 50-528/529/530 Reply to Notice of Vloiations 50-528/93-26-01, 50-530/93-26-05, and 50-530/93-26-07 and Notice of Deviation 50-528/93-26-02 File: 93-070-026 Arizona Public Service Company (APS) has reviewed NRC Inspection Report 50-528/529/530/93-26 and the Notice of Violations and Notice of Deviation dated July 30,1993. Enclosure 1 to this letter is a restatement of the Notice of Violations and the Notice of Deviation. APS' responses are provided in Enclosure 2.

Should you have any questions, please call Thomas R. Bradish at (602) 393-5421.

Sincerely,

[9/#

WFC/TRB/PJC/rv

Enclosures:

1. Restatement of Notice of Violations and Notice of Deviation

2. Reply to Notice of Violations and Notice of Deviation cc: B. H. Faulkenberry J. A. Sloan l

9309090169 930830 I PDR ADOCK OS00o528 h O PDR ;j (j/) l

._ . . - ~ _ _ . . . . _ _ _ _ . _ _

i l

t i

i

~

i i

i

'I i

t I

ENCLOSURE 1 i i

i RESTATEMENT OF NOTICE OF VIOLATIONS I AND NOTICE OF DEVIATION  ;

NRC INSPECTION CONDUCTED JUNE 1 THROUGH JULY 12,1993 i l

NRC INSPECTION NO. 50-528/529/530/93-26 l 1

W

RESTATEMENT OF NOTICE OF VIOLATIONS 50-528/529/530/93-26-01.-05.-07 During an NRC inspection conducted on June 1 through July 12, 1993, four [ SIC) '

violations of NRC requirements were identified. In accordance with the " Genera!  !

Statement of Policy and Procedure for NRC Enforcement Actions," 10 CFR Part 2, i Appendix C, the violations are listed below:

A. Unit 1 Technical Specification 6.8.1 requires, in part, that written procedures shall be established, implemented and maintained covering surveillance and test [

activities of safety-related equipment. t Procedure 73AC-9ZZO4, Revision 9.03, " Surveillance Testing," Step 3.9.3, requires that unsatisfactory steps or data be marked "unsat," circled, and initialed. In addition, the "as-found" condition, the corrective action taken, and any ,

maintenance work request / work order, TSCCR, CR/DR, or MNCR initiated, shall be documented in the surveillance test log. [

i Contrary to the above:

(

1. On June 23,1993, during the performance of surveillance test 36ST-9SB02 in Unit 1, workers noted that the pre-trip indication failed to illuminate as required by the procedure, but failed to mark this step as "unsat" and did not document the deficiency in the surveillance test log.  ;

2. On June 30,1993, during the performance of surveillance test 41ST-1SP02  ;

in Unit 1, workers noted that spray pond pump *B" discharge pressure was i not within 50-55 psid as required by procedure step 8.2.2, but failed to mark the step "unsat" and failed to adequately describe the corrective action in the surveillance test log. This failure was also not identified during supervisory review.

This is a Severity Level IV violation (Supplement I) applicable to Unit 1.

B. Unit 3 Technica! Specification 6.8.1 requires, in part, that written procedures shall be established, implemented and maintained covering applicable procedures recommended in Appendix A of Regulatory Guide 1.33, February 1978.

Regulatory Guide 1.33, Appendix A requires,in part, that administrative procedures should be developed governing the review and approval of plant procedures.

Page 1 of 3

Procedure 01 AC-0AP02, Revision 02.02, Review and Approval of Nuclear Administrative and Technical Procedures," Step 3.2.3.1, states that a 10 CFR 50.59 screening and evaluation shall be performed for any intent change procedure action. Section 4.1.13, Step 5, states that an intent change exists if the change alters the acceptance criteria applicable to Ouality Class activities.  !

l Contrary to the above, revisions were implemented to procedure 31ST-9DG02 on September 12, 1992, and to procedure 31ST-9DG01 on March 5,1993, that l changed the criteria for a satisfactory surveillance test on the Emargency Diesel Generators without a 10 CFR Part 50.59 screening specified Ly procedure ,

01 AC-0AP02.  !

This is a Severity Level IV violation (Supplement I) applicable to Units 1,2, and 3.

C. Units 1, 2, and 3 Technical Specification 6.8.1 requires, in part, that written procedures shall be established, implemented and maintained covering the ,

applicable procedures recommended in Appendix A of Regulatory Guide 1.33, i February,1978. i Regulatory Guide 1.33, Appendix A requires, in part, that specific procedures for I the control of modification work should be prepared.

Procedure 81DP-0DC03, Final Engineering," Design Evaluation checklist item 16, l requires verification that adequate pre-operational and subsequent periodic test i requirements have been appropriately specified. '

Contrary to the above, on September 5,1991, verification of checklist item 16 was  !

inadequate in that the pre-operational test requirements were not appropriately l specified. The specified pre-operational test only considered the operational mode '

of the steam bypass control system and did not consider its test modes. This failure resulted in the inadvertent opening of five steam bypass control valves during subsequent testing. ,

l This is a Severity Level IV violation (Supplement 1) applicable to Units 1,2, and 3.

Page 2 of 3 l

l

- - - i

t 6

RESTATEMENT OF NOTICE OF DEVIATION 50-528/93-26-02 During an NRC inspection conducted on June 1 through July 12,1993, one deviation of l your Updated Final Safety Analysis Report (UFSAR) was identified. In accordance with  !

the " General Statement of Policy and Procedure for NRC Enforcement Actions," 10 CFR ,

Part 2, Appendix C, the deviation is listed below: l A. The UFSAR for Palo Verde Units 1,2, and 3, Section 13.1.3.1, states that, 'The ,

recommendations of Regulatory Guide 1.8, Personnel Selection and Training are used as the basis for establishing minimum qualifications for nuclear power plant personnel. The minimum requirements for station personnel are keyed to ANSI /ANS 3.1 1978."

ANSl/ANS 3.1-1978, paragraph 4.6.1, states that the Engineer-in-Charge of technical support shall have a Bachelor's Degree in Engineering or the Physical Sciences and have three years of professionallevel experience in nuclear services, nuclear plant operation, or nuclear engineering, and the necessary overall background to determine when to call consultants and contractors for dealing with  :

complex problems beyond the scope of owner-organization expertise.

Contrary to the above, on July 1,1993, the licensee reorganized their engineering department such that the managers of technical support (who were in a position to recognize and determine when to call consultants and contractors for dealing with complex problems beyond the scope of owner-organization expertise) did not have a Bachelor's Degree in engineering or the physical sciences. The personnel ,

designated as Engineer-in-Charge were not in charge of technical support, nor organizationally positioned to be able to determine the need to call consultants for dealing with complex problems.

This deviation is applicable to Units 1,2, and 3.

I i

Page 3 of 3 i

j 6

h r

ENCLOSURE 2 REPLY TO NOTICE OF VIOLATIONS AND NOTICE OF DEVIATION NRC INSPECTION CONDUCTED JUNE 1 THROUGH JULY 12,1993 NRC INSPECTION REPORT NO. 50-528/529/530/93-26

REPLY TO VIOLATION A (50-528/93-26-01J Admission Or Denial Of The AIIeaed Violation APS admits the violation. The types of concerns regarding surveillance test (ST) documentation cited in this violation were also identified by two APS Ouality Assurance (OA) audits, Plant Operations and Surveillance Testing, conducted from April 12 to June 8,1993, hnd from June 21 to July 9,1993, respectively. There are outstanding corrective action documents applicable to these, and other, concerns from both of the OA audits.

Reason For The Violation The two examples identified in the violation are attributable to personnel error on the part of the individuals who performed and/or reviewed surveillance tests 36ST-9SB02 and 41ST-1SP02 on June 23,1993, and June 30,1993, respectively and did not adhere to the administrative controls for ST documentation. The APS investigation found that there is I

misunderstanding among some operations and maintenance personnel as to the applicability of 73AC-9ZZO4, "Survei!!ance Testing," requirements to all portions of surveillance procedures. Some personnel mistakenly construed the documentation requirements of section 3.9.3,73AC-9ZZ04, as applicable only to those steps or data that are specific to Technical Specification acceptance criteria. In fact, some surveillance test Page 1 of 16 l

i procedures (e.g.,41ST-1SP02,8.2.2) designate certain steps as not being part of the ST  ;

acceptance criteria, or the acceptance criteria in many steps are more restrictive than required by the Technical Specifications (e.g., 36ST-9SB02). Such inconsistencies reinforce personnel misunderstanding of the applicability of 73AC-9ZZO4. An in progress, detailed review of 73AC-9ZZO4 has identified that the procedure contains excessive and confusing requirements that are subject to varying interpretations. Most of the problems identified with the procedure appear to be attributable to changes made  ;

over the years in response to corrective action documents issued as the result of  !

incidents in the field. The bulk of these changes added to the confusion, as the procedure was not rewritten to address the users' needs. These procedural complexities and inconsistencies are contributing factors to the cause of this violation. ,

An additional concern with the June 23,1993 performance of 36ST-9SB02 was that no f

work request was generated to correct the identified problem. During performance of a f step to verify the indication with a simulated input, the pre-trip indication on the local indicator did not illuminate until work group personnel tapped the bulb cover. The APS investigation found that this problem with the pre-trip indication apparently had existed for an extended, but indeterminate, period of time. There was general knowledge of the problem among Unit 1 I&C personnel, some of whom had informally discussed it with operations and/or engineering personnel. However, review of a computer data base of archived work orders dating from 1985 to the present did not produce any evidence that a work order had been initiated to address this problem. I&C personnel are very familiar l

Page 2 of 16 1

I

with the equipment tested by 36ST-9SB02. As the illumination failure affected only the a

local indicator, and all other indications (including redundant control board and remote panel indications) for this pre-trip operated correctly, Unit 1 I&C personnel came to accept the pre-trip indication response as normal for the equipment. Because so many  ;

of the technicians were aware of the problem, some assumed that it was already being addressed by engineering. Therefore, the pre-trip indication problem was allowed to continue without being properly documented as required by procedure.

Additional Information During performance of 36ST-9SB02 by l&C maintenance personnel on July 20,1993, a similar documentation incident occurred. Step 8.4.2.5 was unsatisfactory with regard to an as-found pre-trip setpoint. The work group circled and initiated the unsatisfactory data and notified both the shift supervisor and their foreman, but they had not made an entry in the test log to describe the as-found condition at the time of discovery.

Two further incidents occurred on August 5,1993, during performance of 41ST-1SG01.

In these incidents, the Operations crew encountered anomalies with the hand switch on main steam isolation valve SGE-UV-171, as well as the failure of a control solenoid on i valve SGE-UV-170. These deficiencies had not been described in the test log; however, it should be noted that the ST was still in progress. With regard to the failed valve, the Control Room Supervisor (CRS) concurred with the NRC Inspectors that a test log entry Page 3 of 16 l

. t i

j was necessary and would have required the entry to be made prior to closing the ST

! package. in the case of the hand switch anomaly, the CRS did not initially expect the )

1 crew to make a log entry; however, he brought the matter to the attention of the Shift Supervisor who agreed with the NRC Inspectors and directed that the appropriate test  :

log entries be made. Further, the CRS fo!! owed up with the NRC Senior Resident f i

inspector to advise him of the Shift Supervisor's position. A Unit 1 night order, issued on  :

t July 1,1993, addressed the requirement to circle unsatisfactory ST steps, but it did not t

cover the requirements for test log entries.

In these later incidents with the exception of the valve failure, the affected steps were, I

again, not specific to Technical Specifications acceptance criteria; and the test performers did not consider the test log documentation requirements of 73AC-9ZZ04, section 3.9.3,

]

to be applicable. At the time of these incidents, corrective actions for the cited violations had not been completed.

i Corrective Steps That Have Been Taken And The Results Achieved The test log for each of the surveillance tests was updated to properly document the  ;

l

identified problems. A work request was initiated to investigate and resolve the problem with the pre-trip indication lamp in the Unit 1 plant protection system. On August 3,1993, following his review of the incidents and the QA audit results, the Unit 1 Maintenance Manager directed that a briefirig on the requirements of 73AC-9ZZ04 be conducted with j Page 4 of 16

l each Unit 1 maintenance surveillance test performer and acceptance reviewer prior to  !

those individuals beginning or resuming any surveillance, testing. The next day, l August 4,1993, information and lessons learned regarding this violation were provided f

to the maintenance managers in Units 2 and 3, so they could evaluate their surveillance activities for impact. In addition, the Unit 1 I&C Supervisor briefed the technicians on this i

violation and the resulting investigation including his specific expectations for ST documentation.

On August 16,1993, Unit 1 Operations issued a comprehensive night order to reinforce I the requirements of 73AC-9ZZO4 and unit management's expectation for complete ST documentation. The night order included a copy of this violation and a copy of {

Appendix E, "Ouick Reference to Documentation Requirements," from 73AC-9ZZO4.  !

Copies of the Unit 1 night order were provided to Units 2 and 3, Operations Standards,  !

and Operations Training. l Corrective Steps That Will Be Taken To Avoid Further Violations i

Unit 1 Operations management will review this violation and the lessons learned with their l onshift personnel. Management in Units 2 and 3 will also review this violation with  !

personnel in their Operations and Maintenance Departments who perform STs.

I j PVNGS management reviewed the concerns regarding implementation of the ST Page 5 of 16

administrative controls and determined that program improvements were needed. On August 20,1993, a cross-organizational ST Program Review Focus Group was formed.

i The participating organizations include Site Technical Support, Plant Support, Operations, l  !

l Site Maintenance, Unit Maintenance, Quality Assurance, and Nuclear Regulatory Affairs. '

The Focus Group's objective is to expeditiously review and revise the ST administrative controls to provide " user friendly" guidance with sufficient instructions to allow uniform ST l 1

performance with consistently adequate, but not redundant, documentation. Once the  ;

revision is complete, appropriate training on the changes will be provided to ST ,

pedormers. A follow-up assessment will be pedormed to determine the effectiveness of this corrective action.

Date When Full Compliance Will Be Achieved  !

Full compliance was achieved when the applicable surveillance test logs were updated ,

to include the appropriate documentation.

)

Page 6 of 16

9 i REPLY TO VIOLATION B (50-530/93-26-05) ,

Admission Or Denial Of The Alleaed Violation  !

APS admits the violation in that the intent of section 4.1.13, step 5,01 AC-0AP02,

• Review and Approval of Nuclear Administrative and Technical Procedures," is that all approved design output documents, which include vendor documents, receive a 10 CFR 50.59  ;

screening.

i Reason For The Violation i

The violation occurred because of an inadequate interface between the vendor document j control and the procedure review and approval programs. Procedure 01AC-0AP02, l l

" Review and Approval of Nuclear Administrative and Technical Procedures, step. 3.2.3, i requires that a "O CFR 50.59 screening / evaluation be performed for any '1ntent Change" procedure action. Section 4.1.13.5 of the same procedure permits an exception to " intent Change" if the change is due to implementation of approved changes to design output documents.

Within the configuration management program and administrative controls, vendor documents are defined as design output documents, but they are excluded from the i

Page 7 of 16

\

I scope of design " changes." However, they have always been under the engineering )

change control program and are reviewed by engineering for conformance to the current l design specification and for any change in design. Vendor documents that do not meet ,

I these criteria are either not accepted, or they are implemented by a design modification l i

that receives a 10 CFR 50.59 screening / evaluation. Based upon reviews of the PVNGS i

UFSAR, PVNGS Technical Specifications, the 10 CFR 50.59 screening questions, and NUMARC NSAC/125, Guidelines for 10 CFR 50.59 Safety Evaluations, PVNGS  !

Engineering concluded that the UFSAR and Technical Specifications do not address equipment maintenance and operation to the level of detail that would be affected by changes to vendor documents. Thus, the requirement of procedure 01 AC-0AP02 that affects vendor documents as defined design output documents is more restrictive than i

10 CFR 50.59 regulations.

The changes to the two diesel generator survei!!ance procedures were processed in

l accordance with the 01 AC-0AP02 exception for procedure intent changes based upon ,

approved changes to design output documents, and the vendor technical manual had been processed in accordance with configuration management program requirements.

1 Therefcre, a 10 CFR 50.59 screening was not required by either program and was not included at any point in the process.

Page 8 of 16 ,

l l

Correct:ve Steps That Have Been Taken And The Results Achieved A complete evaluation of 10 CFR 50.59 requirements and PVNGS procedures was conducted in accordance with the investigation program. Although the results, as discussed previously, indicate positive control of the vendor document program, APS has determined that the additional action discussed below would be prudent.

Corrective Steps That Will Be Taken To Avoid Further Violations in order to treat changes to vendor documents conservatively, procedure 87AC-0CC08,

" Control of Vendor Documents," will be revised to include a requirement for 10 CFR 50.59 screenings of vendor submittals that contain technical changes. The revision is expected to be complete by October 31,1993.

Date When Full Compliance Will Be Achieved Full compliance will be achieved when the revision to the vendor document program is implemented on October 31,1993.

Page 9 of 16

REPLY TO VIOLATION C (530/93-26-07) i Admission Or Denial Of The Alleaed Violatfor.1 APS admits the violation.

Reason For The Violation This violation is the result of personnel error. Both the vendor-designed Steam Bypass Control System (SBCS) Ouick Open Tracking modification and retest and PVNGS -

l engineering's concurring design review were inadequate in that they did not consider the impact of the SBCS master contro!!er switch in manual mode operation. The quick open tracking feature was designed to improve the transition from quick open to modulation '

control following a large load rejection by forcing the low limit on the SBCS master modulation controller to a predetermined high value when the valves are quick opened.

During the vendor design and the PVNGS review of this feature, it was not recognized that once a quick open had been initiated by the interrupted input signals, it resulted in a modulation demand signal being locked in as a low limit. Thus, even when the quick open signal cleared, the affected bypass valves remained fully open. This low limit took effect even though the master controller was in manual mode; an effect not realized when this feature was designed and reviewed.

Page 10 of 16 l

The design modification testing requirements are part of the current overall retest process which involves several procedures among the various engineering organizations. The complexities induced by the combined process with cross-organizational ownership and 1

multiple procedures were a contributing factor in this violation. In addition, a review of system Engineering Evaluation Requests (EERs) is not part of the formal modification process, and Site Technical Support is not always involved in the development of retest i

specifications for non-quality-related design changes.

I Corrective Stens That Have Been Taken And The Results Achieved i

APS conducted a thorough and detailed investigation of the Unit 3 SBCS Overpower Excursion which occurred in April 1993. That investigation identified the modification design and review errors associated with DCP XJ-SF 31 and proposed corrective actions.

As an interim measure to preclude a repetition of the April 3,1993, SBCS event, the '

SBCS vendor tech manual and the applicable PVNGS procedures have been updated -

to include going to the emergency off position when moving the test panel mode l selection switch.

i Corrective Steps That Will Be Taken To Avoid Further Violations The Design Validation Testing Program will be strengthened to ensure that all design Page 11 of 16

1 1

intedaces and modes of operation (including test modes) are checked and understood.

In addition, the design procedures will be revised to involve Site Technical Support Engineering in the development of test specifications for design changes regardless of quality classification. These changes are scheduled to be completed by December 1,1993. '

A recommendation that design change package preparation should include a review of existing system EERs will be disseminated to appropriate engineering personnel by Scptember 1,1993. -

The process for testing design modifications is being separated from the maintenance retest process. Test development references / instructions for design modifications will be removed from other procedures and concentrated in 81DP-0CC23. " Inspection / Test Requirements." The inspection / test requirements procedure will be revised and strengthened to provide guidance for all design validation test development. The revised procedure will be entitled " Design Validation Testing." Completion of this revision is expected by December 1,1993.

Date When Full Compliance Will Be Achieved Full compliance will be achieved wher 9e SBCS Failure Modes and Effects Analysis (FMEA) is updated to include other operational and test modes and the recent SBCS Page 12 of 16

design changes. The FMEA update will either confirm that no similar conditions exist that could adversely impact the system, or it will identify any additional potentially adverse j l

conditions and provide recommendations for corrective action. The FMEA update will be completed on March 1,1994.  ;

t l

i

(

)

t

?

l l

i l

t I

i i

Page 13 of 16

._ _ .___ ~

i I

REPLY TO DEVIATION (50-528/93-26-02) l I

Reason For The Deviation ,

l During the PVNGS reorganization in June 1993, the position of Vice President of Nuclear Engineering and Projects was replaced with the position of Assistant Vice President of Eng:neering and Projects. At that time, PVNGS management decided that, based upon  ;

current UFSAR commitments regarding the Engineer-in-Charge, the Assistant Vice President for Engineering and Projects did not meet the requirements of section 13.1.3.1 because he did not hold a baccalaureate degree in engineering. The decision was made to have the Engineer-in-Charge n! aced at the manager level in the design organization. l Both the manager of mechanical / civil and manager of I&C/ electrical engineering were designated as Engineer-in-Charge. The position descriptions for these managers were  ;

revised to reflect this change.

t The deviation is attributable to differing opinions regarding the term " complex problems" f as used in ANS 3.1-1978. PVNGS uses consultants and contractors for assistance with projects that do not necessarily constitute complex problems, nor are they beyond the expertise of its engineering organization. It was the opinion of PVNGS management that  ;

complex problems beyond the scope of its engineering organization's expertise would be problems related to design, and any decision to bring in outside expertise should go i

Page 14 of 16 l

l

through the PVNGS design organization. Therefore, PVNGS management considered the engineering organization to be in compliance with the UFSAR and PVNGS' commitment to Section 4.6.1 of ANS 3.1-1978.

Corrective Steos That Have Been Taken And The Results Achieved ,

The position descriptions for the designated design organization managers have been r

revised to remove the Engineer-in-Charge requirement. PVNGS has completed a 10 CFR 50.59 evaluation to change section 13.1.3.1 to revise the commitment to ANS 3.1-1978 to include the exception that ". . .those positions requiring a baccalaureate in engineering or a physical science can be satisfied by a baccalaureate and ten years of nuclear power experience or a baccalaureate and a current or previous NRC senior operator license. In addition, the Assistant Vice President of Engineering and Projects is ;

now designated as the Engineer-in-Charge. This change will eliminate any potential  !

interpretations as to the level of management at which the decision is made to bring in contractors or consultants to assist with complex problems.

Corrective Steos That Will Be Taken To Avoid Further Deviations -

No 'urther action is required.

1 Page 15 of 16 1

I

. . j 1r , l Date When Corrective Action Will Be Completed ,

2 Corrective actions are complete.

i t

4 I

l l

Page 16 of 16 i

!