IR 05000528/1988007
| ML17303A825 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 02/24/1988 |
| From: | Andrea Johnson, Richards S NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML17303A824 | List: |
| References | |
| 50-528-88-07, 50-528-88-7, 50-529-88-07, 50-529-88-7, 50-530-88-07, 50-530-88-7, EA-88-062, EA-88-62, NUDOCS 8803140236 | |
| Download: ML17303A825 (15) | |
Text
U.
S.
NUCLEAR REGULATORY COMMISSION
REGION V
EA-88-62 Report Nos.
50-528/88-07, 50-529/88-07, 50-530/88-07 Docket Nos.
50-528, 50-529, 50-530 License Nos.
NPF-41, NPF-51, NPF-74 Licensee:
Arizona Nuclear Power Project P.
0.
Box 52034 Phoenix, Arizona 85072-2034 Facility Name:
Palo Verde Nuclear Generating Station Units 1, 2 and
Inspection Conducted:
January 27, 1988 through Januray 29, 1988 chal r.am A. Johnson, Region V Enforcement Officer S.
Richards, Chief, Engineering Section Approved By:
S.
Richards, Chief, Engineering Section
~Summar:
2 ZS-ee Date Signed Z z0 -eB Date Signed Q-3, 1-8&
Date Signed Ins ection on Januar
1988 throu h Januar
1988 Re ort Nos. 50-528/
88-07 50-529/88-07 and 50-530/88-07
.
Areas Ins ected:
Special announced inspection to followup on LER 50-528 87-025, which described the circumstances surrounding modifications made to the steam driven auxiliary feedwater pumps in Units 1 and 2, which rendered the pumps inoperable.
Results:
Of the areas inspected, four violations of NRC requirements were identified.
8803i40236 880224 PDR ADOCY>> 05000528
DETAILS 1.
Persons Contacted
- E.
E.
Van Brunt, Jr.,
Executive Vice President
"L.
G.
Papworth, Director, guality Assurance
"J. Allen, Unit 1 Plant Manager
"0. J. Zeringue, Unit 3 Plant Manager
"T. Shriver, Manager, Compliance
"G.
W. Sowers, Manager, Engineering Evaluations
- L. Clyde, Shift Technical Advisor Supervisor
"F. Buckingham, Unit 2 Operations Manager
~S.
M. Moyers, Plant Standards and Control
"G. Waldrep, Unit 3 Lead Shift Technical Advisor
"P. J. Coffin, Compliance Liaison R. Lindquist, Shift Technical Advisor M.
A. Corzo, Planner/Coordinator D. Ensign, Shift Supervisor D. Wittas, Mechanical gE Supervisor D. Harrison, Mechanical gE R.
M. Hickson, Electrician A. Rivas, Electrician M. Lencyewski, equality Control Inspector Ins ection Back round The Auxiliary Feedwater (AFW) System at each of the three units at the Palo Verde facility includes three pumps; one safety-related steam turbine driven pump, one safety-related electric pump, and one non-safety electric pump.
Each pump is rated at 100K of the design required flow.
Main steam flow can be aligned to the turbine driven pump from either of
,the two steam generators.
Two normally closed valves, SGAUV0134 and SGAUV0138, isolate the 'A'rain and 'B'rain steam supplies-respectively, from the pump turbine.
Each of these steam supply isolation valves has an associated bypass valve.
When the turbine driven pump is called upon to operate, one bypass valve opens and the pump starts and accelerates to an initial idle speed of approximately 400 rpm.
The main isolation valve then opens following a time delay.
When the control circuitry senses that the main isolation valve has opened, the turbine then ramps up to rated speed under the control of the turbine governor.
The motor operator for valves SGAUV0134 and SGAUV0138 each contain four position indicating rotors.
Each rotor has four position indicating switches, which are used for electrical control and indication circuits associated with the valves.
Two of the rotors normally change state when the valve goes fully closed and two of the rotors normally change state when the valve is fully open, however the point during the valve stroke at which each rotor repositions is adjustable, such that any rotor can be set to change state at any intermediate valve stroke position.
For the two valves in question, rotor 81 senses open valve position for three position indication lights and provides a limit switch which changes
state to deenergize the valve motor during an open stroke.
Rotor ¹3 senses open valve position for indication to the Emergency Response Facility Data Acquisition Display System (ERFDADS) and for the pump control circuit which allows the pump to ramp up to rated speed.
Rotors
¹2 and ¹4 change state as the valves near the closed position and provide control and indication primarily for the valve closing cycle.
On Novembery 27, 1987, with unit 2 running at 100 percent power, a
routine operability surveillance test was conducted for the turbine driven AFW pump.
The test was conducted by aligning steam flow through
- valve SGAUV0134.
The pump started and ran at the idle speed, however the pump did not ramp up to the rated operating speed.
The subsequent investigation by the licensee determined that the position indicating rotors for both steam supply isolation valves at Unit 1, and valve SGVUA0138 at Unit 2, had been incorrectly adjusted, such that during the opening stroke of the valves, rotor ¹3 would not consistently change state to generate a valve open indication, which allows the pump to. ramp up to rated speed.
The licensee documented the circumstances of the event in Licensee Event Report (LER) 50-528/87-025.
As noted in the LER, Palo Verde Technical Specification Limiting Condition for Operation (LCO) 3.7. 1.2 requires at least three independent steam generator auxiliary feedwater pumps and associated flow paths to be operable in modes 1, 2, and 3.
The LCO Action Statement requires that with one AFW pump inoperable, the unit shall be placed in a hot standby condition within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and in a hot shutdown condition within the following 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />.
The licensee's LER documents that this technical specification requirement was violated at unit 1 for a period of approximately 12 days and at unit 2 for a period of approximately 43 days (50-528/88-07-01).
The licensee determined that both the safety-related and the non-safety electric driven AFW pumps were continuously available during the affected period.
Each of the electric driven pumps will deliver the required flow rate to-mitigate the consequences of analyzed accidents.
Licensee Followu The licensee conducted a Special Plant Event Evaluation to determine why the AFW valves had been incorrectly adjusted without being detected by the review and testing requirements of the licensee's programs.
The results of the evaluation were being documented in Special Plant Event Evaluation Report (SPEER)
number 87-02-019 at the time of the NRC inspection.
The SPEER had not yet been approved by licensee management.
The licensee's review determined that during routine testing of the valves in June, 1987, licensee personnel identified that the valves were backseating with what was considered to be excessive force.
Engineering Evaluation Request (EER) 87-SG-117 was initiated to review the problem.
Although the valves are at an interface between the main steam system and the AFW system, the valve designation indicates the valves are part of the main steam system, the EER was directed to the system engineer for the main steam system.
The engineer contacted the valve vendor and determined that the valves should not be allowed to backseat.
The engineer resolved the EER by directing that the o en limit switch be
adjusted such that valve stroke would stop prior to backseating.
The EER disposition did not address resetting the other three position switch rotors contained in each valve.
The engineer performed a calculation to show that system design steam flow through the valve could be reached with the valve 60 percent or more open.
The engineer also recognized that the valve coasted somewhat after the motor was deener gized.
He therefore directed that the open limit switch be set as far towards the open position as possible without the valve disk impacting the backseat, but at a minimum the switch would be set at 65 percent open or greater.
The implementation of the resolution to this EER resulted in the valve motor rotors being reset.
The concerns identified by the licensee in the SPEER at the time of the inspection are summarized as follows:
a)
The engineer dispositioning the EER did not recognize that the valves contained multiple rotors and therefore did not provide instructions for resetting rotor ¹3.
The AFW system engineer was not consulted on the EER nor was the electrical engineering group consulted.
b)
The work package for disposition of the EER at unit 3 contained, as an attachment, another EER which generically discussed the need to reset rotor ¹3 to ensure that the ERFDADS input was correct.
This apparently resulted in the valves at unit 3 being adjusted such that the valves functioned properly.
However, this second EER was not attached to the work package instructions for units 1 or 2, nor were any other instructions for setting rotor ¹3 provided.
There was no overall program to control setting the rotors in motor operated valves and the knowledge level of the craft, the planner/coordinators, and the engineers, concerning these rotors, varied considerably.
c)
Retests following the adjustments of the limit switches did not ensure the proper functioning of the system.
Although the valves were cycled using inservice testing procedures to verify proper position indication and stroke time, the retesting did not run the pump.
The planner/coordinator initially recommended retest requirements in the work package, which were then approved by the shift supervisors at both units.
None. of these personnel recognized a need to test the operation of the pump following the rotor adjustments.
d)
Once the problem was identified, the valves were not declared inoperable pursuant to the containment isolation valve technical specification action requirements.
e)
In the process of taking immediate corrective action for the problem, the unit 2 shift supervisor requested the electrical craft to reset rotor ¹3 on the affected valves.
The licensee's procedures require an EER to adjust motor operated valve limit switches, therefore the shift supervisor violated the licensee's administrative controls in directing this action.
Additionally, in following out" the shift supervisor's directions, the electricians did not reset the rotors in accordance with EER 87-SG-200, which had been specifically dispositioned to address the problem.
87-SG-200 required rotor ¹1 and rotor ¹3 to be set at the same position.
The electricians adjusted rotor ¹3 so that it changed state at approximately 4 percent valve travel beyond rotor ¹l.
Although this was contrary to EER 87-SG-200, the error apparently did not affect the system operation.
The Shift Technical Advisors (STA) were in a position to identify that the retests performed after the initial rotor adjustment were insufficient; that the actions specified in EER 87-SG-200 were required to be completed as corrective action prior to declaring the pump operable at unit 2; and that technical specification 3.6.3 was required to be entered upon the initial discovery of the problem.
The STAs failed on all three. points.
The inspectors considered the licensee's actions to review the event to be extensive.
The licensee identified numerous concerns and failures during their review, and were considering corrective actions to address these deficiencies.
Ins ector Review Having reviewed the efforts of the licensee to investigate the event, the inspectors focused their own review on attempting to gain an understanding of how an error made in dispositioning an EER could go undetected by the various reviews, checks, and testing required by the licensee's programs.
The following personnel appeared to have.been in a position to identify the error.
Three onsite engineers, which included the engineer who dispositioned the EER, his supervisor and the engineering manager who approved the disposition.
The Planner/Coordinator who assembled the work orders to implement the EER disposition.
The various personnel, including QC/gA, who reviewed the work order prior to issuing the package to the field.
Several electricians who performed the field work.
Several gC inspectors who checked the work.
An unknown number of contractor personnel from the MOVATS Corporation.
Several Operations Assistant Shift Supervisors, who concurred with the retests specified in the wor k order.
The inspectors interviewed a majority of the personnel listed above and reviewed their actions in conjunction with a review of the applicable procedures and drawings.
The inspectors made the following observations.
a)
The licensee apparently did not consider the adjustment of the motor operated valve limit switches to be a modification of a nuclear
safety-related component.
As discussed above, the adjustment was directed by the disposition of an EER.
Technical Specification paragraph 6.5.2.3 required that proposed modifications to safety-related components receive an independent check and be approved by the Plant Manager or the Manager Technical Support.
The EER procedure, 73AC-OZZ29, required neither of these.
The EER procedure apparently is not intended to be used for modifications, however, the licensee's administrative controls do not provide clear guidance concerning when an EER is appropriate and when the design change process should be used.
The EER procedure defined a "repair" disposition as, "... allowing a minor change from original design without impairing its intended design basis function,...."
The procedure also required the Engineering Manager's approval of the EER disposition if the disposition constitutes a "design change".
The EER procedure therefore implied that minor design changes can be made through the EER process, however, what constitutes a minor change was not well defined.
Personnel interviewed by the inspectors stated that the licensee had internally been discussing how to appropriately limit the use of the EER procedure for the past year or more.
An unwritten policy had apparently been established that defined a minor change as a change which did not require a document revision.
The licensee personnel
'interviewed stated that the percentage of valve travel at which the limit switch rotors change state was not documented on drawings, and therefore the EER process was appropriate to adjust the rotors.
The inspectors pointed out that the electrical elementary diagram for the valves contained a limit switch'contact development graph.
The personnel interviewed apparently did not realize that the development graph existed, however they maintained that the graph did not clearly indicate the specific adjustments of the rotors, and therefore adjustments to the rotors would not require a revision of the electrical elementary drawing.
The inspectors concluded that the adjustment of the valve limit switches was a modification of a safety-related system, in that the adjustment, effectively rendered the turbine driven AFW pump inoperable.
Therefore the use of the EER procedure.to direct the modification was inappropriate and in violation of technical specification 6.5.2.3, which required an independent review of proposed modifications to safety-related components and approval of the modification by the Plant Manager or the Manager Technical Services (50-529/88-07-02).
The inspectors also concluded that the appropriate usage of the EER process was not adequately defined by the licensee's procedures.
The review of the EER disposition by the engineer's supervisor and the engineering manager was inadequate.
The supervisor had been significantly involved with the engineer's work to disposition the EER and therefore did not provide an independent check of the work.
The engineering manager's review of the work prior to granting his approval was apparently only an administrative review.
Neither the supervisor nor the engineering manager identified that the work
should have been done as a site modification, which receives a more rigorous technical review; nor did they identify the need for a review by the AFW system engineer on a cross discipline check by an electrical engineer.
The work order packages assembled by the planner/coordinator (P/C)
for units 1 and 2 made reference in several locations to resetting the open limit switches, however, as identified by the licensee, the packages contained no instructions for setting rotor 03.
The P/C indicated that he was constrained by policy to insert the EER disposition instructions, as written by engineering, word for word, into the body of the work order.
Therefore the detailed work instructions only direct the resetting of the "open limit switch".
The P/C indicated to the inspectors that he was aware that multiple rotors were contained in the valves, however he had felt confident that the MOVATS contractor personnel would ensure that all the rotors were correctly set and he was therefore not concerned that the EER disposition instructions only addressed the single open limit switch.
The P/C was responsible for initially specifying the retest requirements.
The shift supervisors were responsible for concurring in the retesting.
As identified by the licensee, the retest requirements were inadequate, in that only the associated valves were tested by individual valve stroking, rather than having performed a system test which would have checked the pump operation.
The P/Cs had been provided written guidance on what retests are appropriate for work performed on various valves.
The P/Cs had not been trained on the detailed operation of plant systems.
In this case, the P/C stated that he did not consider an integrated system type test to be necessary because the work only involved the individual valves.
He did not recognize that the valve limit switches were part of the pump control circuitry.
He further indicated that the shift supervisors approve the retesting and he had felt confident that the shift supervisors would ensure that the retests were appropriate.
The shift supervisor interviewed indicated that the type of review of the retest requirements performed by operations personnel did not typically include a review of component drawings nor a review of the integrated operation of the system involved.
The shift supervisor concurrence in the retests appeared to be based primarily on the same written guidance as that available to the P/C, and on the systems training routinely provided to operations personnel.
The shift supervisor interviewed, further indicated that if any special testing was required, that engineering would provide that information.
A further review of the EER procedure revealed that retest requirements are not specifically addressed by the procedure.
In this case, the EER disposition provided by engineering only contained general guidance to stroke the valve to ensure that the backseat was not impacted after the limit switch rotor was adjusted.
The inspectors agreed with the licensee's conclusion that the testing performed after the work was inadequate.
CFR 50,
P
Appendix B, Criterion XI; the licensee's Quality Assurance Manual, Criterion 11; and the Palo Verde FSAR, section 17.2.ll, all require that the test program assure that required testing be identified and performed to demonstrate that systems will perform satisfactorily in service following modifications.
Therefore, the licensee's failure to adequately test the AFM system following the work on the valves is a violation of NRC requirements (50-528/88-07-03).
The electricians interviewed, who had actually performed the work on several valves, stated that the normal practice was to set the two open position rotors at the same point in valve travel.
In this case, the adjustments were made in conjunction with "MOVATS" testing.
The electricians apparently placed their confidence in the MOVATS contractors to ensure that rotor ¹3 was set correctly.
The
"As Left" MOVATS test data sheets indicate that rotor ¹3 was checked when the valves were stroked closed, however the data sheets indicate that rotor ¹3 operation was not checked in the opening direction, which is the critical direction in this case.
Reportedly, the testing equipment did not have sufficient test connections to test all four rotors at once, so only three of four were tested in each direction.
A QA/QC review of the work packages was performed prior to the packages being issued to the field.
Discussions with QA representatives indicated that this review was not technical in nature, but rather a paperwork review to ensure that the process being used was correct.
This review apparently did not question whether the EER process was appropriate to use in this case.
The review did not question the adequacy of the QC holdpoints for the work.
As discussed below, the inspectors concluded that the QC holdpoints were inadequately specified.
Step ll of the MOVATS test instructions for each valve work order contained a
QC holdpoint.
The holdpoint required a
QC inspector to
"verify settings of limit switches per the attached instructions as indicated by "MOVATS"."
The inspectors interviewed a
QC inspector who,had performed this task and determined that the instructions attached to the work order were insufficient to determine the correct setting of rotor ¹3.
Additionally, the QC inspector had not been trained on motor operated valve operation to the point that he would be able to determine for himself that the limit switch settings were correct.
The QC inspector indicated that the performance of the holdpoint basically amounted to verifying that the rotors changed state at the point that the MOVATS personnel said they would.
The inspectors concluded that the holdpoint did not contain appropriate quantitative or qualitative acceptance criteria, as required by'0 CFR 50, Appendix B, Criterion V, and is therefore a
violation of NRC requirements (50-528/88-07-04).
The inspectors interviewed a total of nine individuals who were actually involved in the event.
Of these nine individuals, apparently only one had looked at the valve electrical drawing
during the performance of his part of the work.
Although this individual may have correctly read the drawing when performing his work, while being interviewed by the inspectors, he misread the drawing and indicated that the contact actually associated with the pump governor control, was a spare contact.
5.
Conclusions The inspectors reached the following conclusions based on their review of the event:
A large number of licensee personnel were involved in the implementation, review, or checking of the work, yet no one identified the problem.
Many of the personnel interviewed indicated that their responsibility was a very narrow one.
They often placed their confidence in the actions of others to ensure that the work was correctly performed.
Several of the personnel involved appeared not to be adequately trained to perform the work required of them by the procedures involved.
The licensee's programs indicate that the checks and reviews of work are numerous and extensive.
In this case, after interviewing many of the personnel involved, it was clear that the checks actually conducted were minimal.
The performance of technical work in this event was below expectations.
The inspectors considered the safety significance of the event.
As noted in the licensee's LER, both electric driven AFW pumps remained available at both units throughout the event.
The importance of reliable operation of the AFW system at Palo Verde is greatly increased, however, due to the absence of power operated relief valves (PORV) on the reactor coolant system pressurizer.
The lack of PORVs greatly increases the significance of a loss of the secondary heat sink.
Although the inspectors considered the undetected loss of operability of the steam driven AFW pumps to be unacceptable, the inspectors concluded that the main safety significance of this event lies in the fact that the numerous checking, reviewing, and testing requirements of the licensee's programs failed to detect a
significant error which was well within the licensee's capability to identify.
After the inspection was concluded, a second event occurred at Unit 2
.
which affected the AFW system.
The unit was shutdown on February 20, 1988, for a scheduled refueling outage.
The operators attempted to use the non-safety electric AFW pump to feed water to the steam generators for decay heat removal, however they were unable to achieve flow from the pump.
The discharge valve on the pump was found to be closed, although all documentation apparently indicated that the valve was open.
The NRC review has not been completed as yet, however early indications are
that three separate individuals had the opportunity to identify that the valve was mispositioned.
This event apparently represents a second instance where inattentiveness on the part of the personnel involved resulted in an important safety component being rendered unavailable.
6.
Exit Interview The inspectors met with the personnel identified in paragraph 1 of the report on January 29, 1988.
At this meeting, the inspectors summarized the scope and findings of the inspection.