IR 05000530/2009010

IR 05000530-09-010; 03/30/2009 - 08/12/2009; Palo Verde Nuclear Generating Station, Unit 3; Focused Baseline Inspection Report; Identification and Resolution of Problems

ML092460164
Person / Time
Site:	Palo Verde
Issue date:	09/02/2009
From:	Hay M NRC/RGN-IV/DRP/RPB-D
To:	Edington R Arizona Public Service Co
References
EA-09-140 IR-09-010
Download: ML092460164 (16)

Text

UNITE D S TATES NUC LEAR RE GULATOR Y C OMMIS SI ON ber 2, 2009

SUBJECT:

PALO VERDE NUCLEAR GENERATING STATION, UNIT 3 - NRC FOCUSED BASELINE INSPECTION REPORT 05000530/2009010

Dear Mr. Edington:

On August 12, 2009, the U.S. Nuclear Regulatory Commission (NRC) completed a focused baseline inspection at your Palo Verde Nuclear Generating Station, Unit 3 facility. This inspection examined activities associated with the Unit 3 train A emergency diesel generator failure that occurred on February 4, 2009. During two of three consecutive attempts, emergency diesel generator 3A failed to start within the time required by technical specifications. The enclosed report documents the inspection findings, which were discussed on August 12, 2009, with Mr. Bement, Vice President, Nuclear Operations, and other members of your staff.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.

The NRC's initial evaluation of this issue using the criteria in NRC Management Directive 8.3, NRC Incident Investigation Program, determined that the estimated Incremental Conditional Core Damage Probability was in the overlap region between a focused baseline inspection and a special inspection. Because this issue was considered to include a non-complicated failure of one safety-related component, a decision was made on March 16, 2009 to perform a focused baseline inspection, and the inspection began on March 30, 2009.

This report documents one NRC-identified violation of very low safety significance (Green). This finding was determined to involve a violation of NRC requirements. However, because of its very low safety significance and because it is entered into your corrective action program, the NRC is treating this finding as a noncited violation, consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the violation or the significance of the noncited violation, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001, with copies to the Regional Administrator, U.S. Nuclear

Arizona Public Service Company -2-Regulatory Commission, Region IV, 612 E. Lamar Blvd, Suite 400, Arlington, Texas, 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Palo Verde Nuclear Generating Station facility. In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Palo Verde Nuclear Generating Station. The information you provide will be considered in accordance with Inspection Manual Chapter 0305.

In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter and its enclosures will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Michael C. Hay, Chief Projects Branch D Division of Reactor Projects Dockets: 50-530 Licenses: NPF-74

Enclosure:

NRC Inspection Report 05000530/2009010 w/Attachment: Supplemental Information

REGION IV==

Docket: 05000530 License: NPF-74 Report: 05000530/2009010 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station, Unit 3 Location: 5951 S. Wintersburg Road Tonopah, Arizona Dates: March 30 through August 12, 2009 Inspector: D. Proulx, Senior Project Engineer R. Treadway, Senior Resident Inspector M. Runyan, Senior Reactor Analyst Approved By: Michael C. Hay, Chief, Projects Branch D Division of Reactor Projects-1- Enclosure

SUMMARY OF FINDINGS

IR 05000530/2009010; 03/30/2009 - 08/12/2009; Palo Verde Nuclear Generating Station, Unit 3;

Focused Baseline Inspection Report; Identification and Resolution of Problems This report covered a 5-day period of onsite inspection, with in office review through August 12, 2009, by one senior project engineer, one senior resident inspector, and one senior reactor analyst. One Green finding was identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

Green.

The inspectors identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion V, Instructions, Procedures and Drawings, for the failure to adequately implement foreign material controls during maintenance.

Specifically, on May 21, 2008, maintenance and quality control personnel failed to ensure that no foreign material entered the fuel injection pump 7R during refurbishment. As a result, a degraded fuel injection pump 7R resulted in the failure of diesel generator 3A to successfully pass surveillance testing on February 4, 2009. This issue was entered into the licensees corrective action program as Palo Verde Action Request 3280474.

The finding is more than minor because it is associated with the equipment performance attribute of the mitigating systems cornerstone and affects the cornerstone objective of ensuring the reliability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, the finding was determined to require a Phase 2 analysis because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. Using the Palo Verde pre-solved sequences and an exposure time of 3 to 30 days with one emergency diesel generator unavailable, the Phase 2 estimation determined this finding was of low to moderate significance. With credit for battery operation for seven hours, the Phase 3 analysis determined that the total delta core damage frequency from all of the combined scenarios was 5 E-7; and thus, the finding was considered to be of very low safety significance (Green). This finding has a crosscutting aspect in the area of human performance associated with decision making because the licensee did not communicate bases for decisions to personnel with a need to know such that work is performed safely, in a timely manner H.1(c)

(Section 4OA2).

Licensee-Identified Violations

None

REPORT DETAILS

4OA2 Identification and Resolution of Problems

Selected Issue Follow-up Inspection

a. Inspection Scope

The inspectors performed an in-depth review of issues associated with the February 4, 2009, failure of emergency diesel generator 3A to achieve rated voltage and frequency within the Technical Specification Surveillance Requirement 3.8.1.2 maximum allowed time of 10 seconds. The inspectors reviewed Condition Report-Disposition Report 3282707, Palo Verde Action Request 3280474, plant computer traces, the vendor report, and other documentation and procedures as listed in the attachment. The inspectors also conducted interviews and walked down applicable portions of emergency diesel generator 3A.

These activities constitute completion of one in-depth problem identification and resolution sample as defined in Inspection Procedure 71152-05.

b. Findings

Failure to Follow Foreign Material Exclusion Requirements

Introduction.

The inspectors identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures and Drawings, for the failure to adequately implement foreign material controls during maintenance. Specifically, on May 21, 2008, maintenance and quality control personnel failed to ensure that no foreign material entered fuel injection pump 7R during refurbishment. As a result, fuel injection pump 7R seized in place, rendering emergency diesel generator 3A inoperable and unavailable for 31.5 days.

Description.

On February 4, 2009, during performance of surveillance test Procedure 40ST-9DG01, Diesel Generator A Test, emergency diesel generator 3A achieved rated voltage and frequency (3280 V, 59 Hz, respectively) in approximately 10.7 seconds, which exceeded the Technical Specification Surveillance Requirement 3.8.1.2 maximum allowed time of 10 seconds. This surveillance test started the emergency diesel generator using a loss-of-coolant accident test signal. The starting time for emergency diesel generator 3A during the previous run on January 5, 2009, was 7.5 seconds, and the average starting time for emergency diesel generator 3A during the previous 8 months was 7.7 seconds. The licensee started emergency diesel generator 3A and operated it unloaded for approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> at rated voltage and frequency before shutting down and securing the emergency diesel generator.

Approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> later, with no adjustments or other actions taken with the exception of checking for obvious problems, the licensee again started emergency diesel generator 3A. However, this time emergency diesel generator 3A was started using a manual maintenance start. During this second run, emergency diesel generator 3A achieved rated voltage and frequency in 8.5 seconds, which was within the Technical Specification Surveillance Request 3.8.1.2 acceptance criterion of 10 seconds. The licensee operated emergency diesel generator 3A unloaded for approximately 12 minutes, then shut-down, and secured the emergency diesel generator. The licensee noted no obvious problems with emergency diesel generator 3A following this second maintenance run.

Approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> following the second start of emergency diesel generator 3A, the licensee performed a third surveillance test of emergency diesel generator 3A. This time, the licensee started the emergency diesel generator using the same conditions as the first test of the day (i.e., by inserting a loss-of-coolant accident test signal utilizing Surveillance Test Procedure 40ST-9DG01). In addition, the licensee staged two maintenance mechanics to locally monitor the engines response to the test signal.

During this third test, emergency diesel generator 3A achieved rated voltage and frequency in 18.7 seconds, exceeding the Technical Specification Surveillance Request 3.8.1.2 acceptance criterion of 10 seconds. The licensee continued to operate the emergency diesel generator unloaded for the next 30 minutes before securing emergency diesel generator 3A.

One of the mechanics observed that the fuel control cylinder did not fully collapse as expected, but only moved to the midposition. In a standby condition, the fuel control cylinder is extended and held in place by 80 psig control air acting on a piston internal to the cylinder. During a normal engine start sequence, the air is ported from the cylinder allowing an internal spring to expand and cause the cylinder to collapse. The collapsing of the fuel control cylinder provides the necessary force (approximately 200 pounds-force) to move the metering rods to the full fuel position. Due to the misoperation of the fuel control cylinder, the mechanics inspected the fuel injection pumps and noted that the metering rod for fuel injection pump 7R was stuck in the minimum fuel position, and could not be moved when applying moderate force by hand.

The other 19 fuel injection pump metering rods on emergency diesel generator 3A were checked as well and were noted to move freely by hand. Operations personnel initiated Palo Verde Action Request 3280474 and Condition Report Disposition Report 3282707 to place this issue into the corrective action program and generated a work order to replace emergency diesel generator 3A fuel injection pump 7R.

Maintenance personnel then removed and quarantined fuel injection pump 7R from emergency diesel generator 3A. On February 6, 2009, maintenance personnel installed a refurbished fuel injection pump and the surveillance test for emergency diesel generator 3A was completed satisfactorily. During this test the emergency diesel generator achieved rated voltage and frequency in 7.8 seconds, well within the technical specification required time of 10 seconds.

The licensee sent the failed fuel injector pump to an independent lab to determine the cause of failure, and sent engineering personnel to observe the vendor's investigation.

The licensee received the draft laboratory report on March 21, 2009. The draft report stated that, on a new fuel injection pump, the metering rod will move easily (i.e., fall by gravity). The apparent as-found condition of the removed fuel injection pump was that the metering rod was seized and would not move with moderate force by hand because of internal binding of the injector pump. Several attempts were made during disassembly of the injector pump to move the metering rod manually and identify the source of the binding. Normally, the metering rod moves laterally causing axial rotation of the control sleeve through a rack and pinion gear. The control sleeve is located between the housing and the barrel and positions an internal plunger to port more or less fuel to the emergency diesel generator as it rotates axially. Personnel continued to disassemble the pump until all that remained was the external housing, metering rod, control sleeve, and barrel. When the metering rod was finally removed, it was noted that the control sleeve and barrel internal to the housing were still seized. The vendor then chose to cut the housing off with a band saw to remove the control sleeve and barrel.

After sectioning, the barrel and control sleeve were still bound together, and the control sleeve could not be rotated on the barrel with moderate manual manipulation. The barrel and control sleeve were finally separated by hand, pulling them apart in an axial direction rather than rotational. Visual examination revealed a foreign material (brown/amber residue) in several locations of the barrel, control sleeve, control sleeve gear teeth, the pump spring, and spring plates.

The lab performed a spectral analysis of the foreign material. The spectral analysis eliminated the possibility that the foreign material was residue from normally approved oils or lubricants used during the refurbishment process. The spectral analysis indicated that the residue was most likely a coconut oil ethanolamide-based soap or detergent, which is not part of any cleaning solution recommended by the vendor manual for use during refurbishment. Palo Verdes engineering personnel observed the foreign material removed from the fuel injection pump, and agreed that the volume of material could not likely be justified other than if it was introduced during the refurbishment process.

The inspectors reviewed the licensees root cause investigation and noted that the emergency diesel generator fuel injector pump 7R had been refurbished on May 21, 2008, at a vendor facility, along with 13 other fuel injector pumps for emergency diesel generator 3A. This vendor was not an approved 10 CFR Part 50, Appendix B, supplier so the refurbishment of the fuel injector pumps was performed as a commercial grade dedication, with Palo Verde providing quality control oversight of the refurbishment process. The licensee used Procedure 87DP-0MC06, Material Engineering Evaluation, to identify the critical attributes necessary for dedicating the refurbished fuel injector pumps for safety-related use in the emergency diesel generators. Material Engineering Evaluation MEE 04128, Source Surveillance of Haynes Corporation for FDX Series Fuel Injection Pumps Utilized in the Emergency Diesel Generators, documented this evaluation and incorporated the Haynes Corporation inspection checklist for the commercial grade dedication. Haynes Corporation mechanics performed the refurbishment with 100 percent Palo Verde contractor quality control coverage. During their review, the inspectors noted that step 47 of the inspection

checklist required the mechanic and the quality control inspector to verify that no foreign material was introduced into the fuel injection pump.

The licensees root cause analysis revealed that the most likely cause of fuel injection pump 7R failure was the hardening of this soap-like foreign material on the internals of the pump. The inspectors concluded that the mechanic and licensee quality control inspector failed to adequately implement the maintenance instructions to verify that no foreign material was introduced into the fuel injection pumps during refurbishment.

The licensee implemented corrective actions to ensure that the extent of condition was addressed. The licensee verified that each of the 20 fuel injection pump metering rods for each of the six emergency diesel generators (two emergency diesel generators for each of the three units) had freedom of movement, and added a procedure requirement to check the freedom of movement of the metering rods following each emergency diesel generator surveillance test.

During their review, the inspectors noted that emergency diesel generator 3A had completed nine successful surveillance tests after May 21, 2008, prior to failing on February 4, 2009. Understanding that the metering rod for fuel injection pump 7R seized, while the emergency diesel generator was in operation, the inspectors concluded the seizure of the pump was a latent failure, and the timing of the failure was most likely influenced by the number of cycles of the fuel injection pump metering rod. The inspectors observed that the licensees root cause analysis also concluded the most likely cause of the irregular start times observed on February 4, 2009, was due to the physical displacement of the foreign material caused by metering rod movement.

The inspectors noted that while the lab report indicated the as-found condition of the metering rod for emergency diesel generator 3A fuel injection pump 7R was seized, the licensees final root cause analysis in Condition Report Disposition Report 3282707 indicated that the metering rod was operating sluggishly. The licensee stated that the metering rod was not stuck and the governor would have been able to allow the emergency diesel generator to load because the metering rod moved during the three attempted starts on February 4, 2009. During their review, the inspectors observed that during the three attempted starts of the emergency diesel generator 3A in February, it is unknown from one movement of the fuel injection pump 7R metering rod to the next how much friction would have occurred and how long the injector pump would have been stuck. Additionally, the inspectors noted that the licensee had not determined the amount of frictional force the foreign material presented, and therefore could not demonstrate that after the last movement of the fuel injection pump 7R metering rod, the frictional force would have been less than that generated by the governor.

The inspectors interviewed the system engineer and the mechanics that were present for the failed emergency diesel generator 3A surveillances on February 4, 2009, who reiterated their initial statements that the metering rod was stuck in place following the third attempted start. During their interview, the inspectors also noted that the mechanic who observed the fuel control cylinder stated the cylinder began to collapse (it normally takes less than half a second for the cylinder to fully collapse) and abruptly stopped halfway. The fuel control cylinder is mechanically attached to the fuel injector pump

metering rods through a series of mechanical linkages, and any movement of the cylinder causes the same affect for movement to each of the metering rods. The inspectors concluded this was not indicative of sluggish behavior, but instead demonstrated that during the third start the fuel injection pump 7R fuel injector pump was seized. The licensees root cause analysis report also stated that the foreign material in the fuel injection pump 7R only caused sluggish performance in low fuel regions. The inspectors reviewed the information and noted this was not supported by the mechanics observations during the third emergency diesel generator start that the metering rod stuck almost half way open.

The inspectors noted that, by design, the governor does not initially provide any force to move the metering rods during the emergency diesel generator starting sequence because it is already positioned for full fuel delivery. The fuel control cylinder is fully extended, and upon a start signal, fully collapses to impart approximately 200 pounds-force on the 20 fuel injection pump metering rods to position them to the full fuel positions. Once the emergency diesel generator is at rated frequency and voltage, the governor then imparts approximately 200 pounds-force to position the metering rods as needed to ensure adequate fuel is delivered to the emergency diesel generator. The inspectors noted that during the third emergency diesel generator start, the fuel control cylinder was not able to provide enough force to overcome the frictional forces caused by the foreign material and stopped half way. Since the magnitude of the forces exerted by the fuel control cylinder and the governor are very similar, the governor also would have failed to adequately control the fuel supply to the emergency diesel generator 3A during an actual event. The licensee stated in their root cause analysis report that following the three starts in February, the emergency diesel generator 3A did not exhibit any abnormal operation while running in an idle condition. However, the inspectors noted that the emergency diesel generator 3A was never loaded during those runs, which would have provided objective evidence to demonstrate that the foreign material did not affect the emergency diesel generators ability to fulfill its safety function.

Once complete with their review, the inspectors concluded that the emergency diesel generator 3A would most likely not have been able to meet its safety function to adequately start and load on February 4, 2009.

Analysis.

The failure to ensure foreign material was not introduced into the fuel injector pump during refurbishment was the performance deficiency. The finding is more than minor because it is associated with the equipment performance attribute of the mitigating systems cornerstone and affects the cornerstone objective of ensuring the reliability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, the finding was determined to require a Phase 2 analysis because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. Using the Palo Verde presolved sequences and an exposure time of 3-30 days with one emergency diesel generator unavailable, the Phase 2 estimation determined this finding was of low to moderate significance. With credit for the Unit 3 battery operation for 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, the Phase 3 analysis determined that the total delta core damage frequency from all of the combined scenarios was 5 E-7; and thus, was considered to be of very low safety significance (Green). This finding has a

crosscutting aspect in the area of human performance associated with decision making because the licensee did not communicate bases for decisions to personnel with a need to know such that work is performed safely and in a timely manner H.1(c).

Enforcement.

Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, states, in part, that activities affecting quality shall be prescribed by documented instructions, procedures, or drawings of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings. Procedure 87DP-0MC06, Material Engineering Evaluation, Section 3.1, implemented this requirement and stated, in part, that material-related issues may be addressed via a material engineering evaluation, including evaluation of quality-related purchases by source verification.

Material Engineering Evaluation MEE 04128, Source Surveillance of Haynes Corporation for FDX Series Fuel Injection Pumps Utilized in the Emergency Diesel Generators, documented this evaluation and provided work instructions for the Palo Verde fuel injection pumps. Step 47 of Material Engineering Evaluation MEE 04138 required the user and the quality control inspector to verify that no foreign material was present.

Contrary to the above, Palo Verde contract quality control personnel failed to accomplish an activity affecting quality in accordance with procedures. Specifically, on May 21, 2008, personnel failed to implement step 47 of Material Engineering Evaluation MEE 04138 by allowing foreign material to enter fuel injection pump 7R of emergency diesel generator 3A, resulting in binding of the metering rod. The failure to adequately implement this procedure resulted in the Unit 3 emergency diesel generator failure on February 4, 2009. Because this finding is of very low safety significance and has been entered into the licensee's corrective action program as Palo Verde Action Request 3280474, this violation is being treated as a noncited violation, consistent with Section VI.A.1 of the NRC Enforcement Policy: NCV 05000530/2009010-01, Failure to Follow Foreign Material Exclusion Requirements.

4OA6 Meetings, Including Exit

On April 3, 2009, the inspectors presented the preliminary inspection results to Mr. R. Bement, Vice President, Nuclear Operations, and other members of the licensee's staff at the conclusion of the onsite portion of the inspection. The licensee acknowledged the findings presented.

On August 12, 2009, the inspectors conducted a telephonic exit and discussed the final inspection results to Mr. R. Bement, Vice President, Nuclear Operations, and other members of the licensee's staff at the conclusion of the inspection. The licensee acknowledged the findings presented.

The inspectors noted that while proprietary information was reviewed, none would be included in this report.

4OA7 Licensee-Identified Violations

None ATTACHMENTS:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

G. Andrews, Director, Performance Improvement

S. Bauer, Department Leader, Regulatory Affairs

R. Bement, Vice President, Nuclear Operations

F. Burdick, Regulatory Affairs

R. Buzard, Section Leader, Compliance

K. Chavet, Senior Consultant, Regulatory Affairs

L. Cortopossi, Plant Manager, Nuclear Operations

E. Dutton, Acting Director of Nuclear Assurance

R. Edington, Executive Vice President, CNO

D. Elkington, Consultant, Regulatory Affairs

W. Grover Hettel, Director, Operations

J. Hesser, Vice President, Engineering

G. Hettel, Director, Operations

M. Karbasian, Director, Design Engineering

F. Lake, Performance Improvement

M. McGhee, Acting Director, Operations

D. Mims, Vice President, Regulatory Affairs and Performance Improvement

J. Molden, Director, Engineering

H. Ridenour, Director, Maintenance

J. Rodriguez, Engineer, Regulatory Affairs

J. Scott, Department Leader, Nuclear Assurance

M. Shea, Director, Safety Culture

J. Summy, Director, Plant Engineering

T. Weber, Section Leader, Regulatory Affairs

Nuclear Regulatory Commission

M. Runyan, Senior Reactor Analyst

R. Treadway, Senior Resident Inspector

M. Catts, Resident Inspector

J. Melfi, Resident Inspector

J. Bashore, Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Y

Opened and Closed

05000530/2009010-01 NCV Failure to Follow Foreign Material Exclusion Requirements (Section 4OA2)

Closed

None Attachment