IR 05000528/2010008

IR 05000528-10-008, 05000529-10-008, 05000530-10-008, on 11/29/2010 - 12/17/2010, Palo Verde Nuclear Generating Station Units 1, 2, and 3 Biennial Baseline Inspection of the Identification and Resolution of Problems

ML110280467
Person / Time
Site:	Palo Verde
Issue date:	01/28/2011
From:	Hay M Division of Reactor Safety IV
To:	Edington R Arizona Public Service Co
References
EA-11-006 IR-10-008
Download: ML110280467 (37)

Text

January 28, 2011 EA-11-006

Randall K. Edington, Executive Vice President, Nuclear/CNO Mail Station 7602 Arizona Public Service Company P.O. Box 52034 Phoenix, AZ 85072-2034 SUBJECT: PALO VERDE NUCLEAR GENERATING STATION UNITS 1, 2 AND 3 - NRC PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT 05000528/2010008, 05000529/2010008, 05000530/2010008, AND NOTICE OF VIOLATION

Dear Mr. Edington:

On December 17, 2010, the U. S. Nuclear Regulatory Commission (NRC) completed a team inspection at Palo Verde Nuclear Generating Station, Units 1, 2, and 3. The enclosed report documents the inspection findings which were discussed on December 17, 2010, with Mr. R. Bement and other members of your staff.

The inspection examined activities conducted under your license as they relate to identification and resolution of problems, safety and compliance with the Commission's rules and regulations and with the conditions of your operating license. The team reviewed selected procedures and records, observed activities, and interviewed personnel. The team also interviewed a representative sample of personnel regarding the condition of your safety-conscious work environment.

Based on the results of this inspection, the NRC has identified an issue that was evaluated under the risk significance determination process as having very low safety significance (Green). The NRC has also determined that a violation is associated with this issue.

This violation was evaluated in accordance with the NRC Enforcement Policy. The current Enforcement Policy is included on the NRC's Web site at (http://www.nrc.gov/about-nrc/regulatory/enforcement/enforce-pol.html).] UNITED STATESNUCLEAR REGULATORY COMMISSIONREGION IV612 EAST LAMAR BLVD, SUITE 400ARLINGTON, TEXAS 76011-4125 Arizona Public Service Company - 2 -

The violation is cited in the enclosed Notice of Violation (Notice) and the circumstances surrounding it are described in detail in the subject inspection report. The violation involved failure to correct a significant condition adverse to quality associated with the Unit 2 emergency diesel generator fuel oil transfer pumps. The violation is being cited in the Notice because Palo Verde failed to restore compliance within a reasonable time after the previous violation was identified (NCV 05000529/2009-004-002), as specified in Section 2.3.2 of the Enforcement Policy. You are required to respond to this letter and should follow the instructions specified in the enclosed Notice when preparing your response. If you have additional information that you believe the NRC should consider, you may provide it in your response to the Notice. The NRC review of your response to the Notice will also determine whether further enforcement action is necessary to ensure compliance with regulatory requirements.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure(s), and your response, will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html. To the extent possible, your response should not include any personal privacy or proprietary information so that it can be made available to the Public without redaction.

Sincerely,

/RA/ Michael C Hay, Chief Technical Support Branch Division of Reactor Safety Dockets: 50-528 50-529 50-530 Licenses: NPF-41 NPF-51 NPF-74

Enclosures:

Notice of Violation and Inspection Report 05000528/2010008, 05000529/2010008, and 05000530/2010008 w/Attachments: 1. Supplemental Information Initial Information Request 2. Palo Verde Fuel Oil Transfer Pump Failure Risk Assessment

Arizona Public Service Company - 3 -

Mr. Steve Olea Arizona Corporation Commission 1200 W. Washington Street Phoenix, AZ 85007 Mr. Douglas K. Porter, Esq Southern California Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Chairman Maricopa County Board of Supervisors 301 W. Jefferson, 10th Floor Phoenix, AZ 85003

Mr. Aubrey V. Godwin, Director Arizona Radiation Regulatory Agency 4814 South 40 Street Phoenix, AZ 85040

Mr. Dwight C. Mims Vice President Regulatory Affairs and Plant Improvement Palo Verde Nuclear Generating Station Mail Station 7605 P.O. Box 52034 Phoenix, AZ 85072-2034 Mr. Ron Barnes, Director Regulatory Affairs Palo Verde Nuclear Generating Station MS 7638 P.O. Box 52034 Phoenix, AZ 85072-2034

Mr. Jeffrey T. Weikert Assistant General Counsel El Paso Electric Company Mail Location 167 123 W. Mills El Paso, TX 79901

Michael S. Green Senior Regulatory Counsel Pinnacle West Capital Corporation P.O. Box 52034, MS 8695 Phoenix, AZ 85072-2034

Arizona Public Service Company - 4 -

Mr. Eric Tharp Los Angeles Department of Water & Power Southern California Public Power Authority P.O. Box 51111, Room 1255-C Los Angeles, CA 90051-0100 Mr. James Ray Public Service Company of New Mexico 2401 Aztec NE, MS Z110 Albuquerque, NM 87107-4224 Mr. Geoffrey M. Cook Southern California Edison Company 5000 Pacific Coast Hwy. Bldg. D21 San Clemente, CA 92672

Mr. Robert Henry Salt River Project 6504 East Thomas Road Scottsdale, AZ 85251

Mr. Brian Almon Public Utility Commission William B. Travis Building P.O. Box 13326 Austin, TX 78701-3326

Environmental Program Manager City of Phoenix Office of Environmental Programs 200 West Washington Street Phoenix, AZ 85003 Mr. John C. Taylor Director, Nuclear Generation El Paso Electric Company 340 East Palm Lane, Suite 310 Phoenix, AZ 85004 Chief, Technological Hazards Branch FEMA Region IX 1111 Broadway, Suite 1200 Oakland, CA 94607-4052 Jake Lefman Southern California Edison Company 5000 Pacific Coast Hwy, Bldg. D21 San Clemente, CA 92672 Arizona Public Service Company - 5 -

Electronic distribution by RIV: Regional Administrator (Elmo.Collins@nrc.gov ) Deputy Regional Administrator (Art.Howell@nrc.gov ) DRP Director (Kriss.Kennedy@nrc.gov ) DRP Deputy Director (Troy.Pruett@nrc.gov ) DRS Director (Anton.Vegel@nrc.gov ) DRS Deputy Director (Vacant) Senior Resident Inspector (Tony.Brown@nrc.gov ) Resident Inspector (Joseph.Bashore@nrc.gov ) Resident Inspector (Mica.Baquera@nrc.gov ) Branch Chief, DRP/D (Ryan.Lantz@nrc.gov ) PV Administrative Assistant (Regina.McFadden@nrc.gov ) Acting Senior Project Engineer, DRP/D (Gerond.George@nrc.gov ) Project Engineer, DRP/D (Peter.Jayroe@nrc.gov ) Public Affairs Officer (Victor.Dricks@nrc.gov) Public Affairs Officer (Lara.Uselding@nrc.gov ) Project Manager (Randy.Hall@nrc.gov ) Branch Chief, DRS/TSB (Michael.Hay@nrc.gov ) RITS Coordinator (Marisa.Herrera@nrc.gov ) Regional Counsel (Karla.Fuller@nrc.gov ) Senior Enforcement Officer (Ray.Keller@nrc.gov ) Congressional Affairs Officer (Jenny.Weil@nrc.gov ) OEMail Resource OEDO RIV Coordinator (James.Trapp@nrc.gov ) DRS/TSB STA (Dale.Powers@nrc.gov )

R:_REACTORS\PV 2010008 - PAJ ML SUNSI Rev Compl. Yes No ADAMS Yes No Reviewer Initials PAJ Publicly Avail Yes No Sensitive Yes No Sens. Type Initials PAJ RIV: PE: DRP/D RI/DRP/D RIV:DRS/EB2 RIV:DRS/PSB2 RIV: DRS/TSB PAJayroe MTBaquera EDUribe TDBuchanan MVasquez

/RA/ /RA/ /RA/ /RA/ /RA/ 1/25/11 1/25/11 1/25/11 1/25/11 1/26/11 RIV C:DRP/D C:DRS/TSB RLKellar RELantz MCHay

/RA /RA /RA/ 1/27/11 1/27/11 1/28/11 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax

- 1 - Enclosure NOTICE OF VIOLATION

Arizona Public Service Company Docket Nos: 50-528,-529,-530 Palo Verde Nuclear Generating Station License Nos: NPF-41, -51, -74 EA-11-006 During an NRC inspection conducted on November 29, 2010 through December 17, 2010, a violation of NRC requirements was identified. In a

REGION IV Docket: 50-528, 50-529, 50-530 License: NPF-41, NPF-51, NPF-74 Report: 05000528/2010008; 05000529/2010008; 05000530/2010008 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station Units 1, 2, and 3 Location: 5951 S. Wintersburg Road Tonopah, Arizona Dates: November 29, 2010 through December 17, 2010

Inspectors:

P. Jayroe, Project Engineer (Team Leader) M. Vasquez, Senior Reactor Inspector M. Baquera, Resident Inspector E. Uribe, Reactor Inspector T. Buchanan, Reactor Inspector

Approved By: Michael C. Hay, Chief Technical Support Branch Division of Reactor Safety

- 4 - Enclosure

SUMMARY OF FINDINGS

IR 05000528; 05000529; 05000530/2010008; 11/29/2010 - 12/17/2010; Palo Verde Nuclear Generating Station Units 1, 2, and 3 Biennial Baseline Inspection of the Identification and Resolution of Problems.

The team inspection was performed by a senior reactor inspector, a project engineer, two reactor inspectors, and a resident inspector. One green finding of very low safety significance was identified during this inspection. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609, "Significance Determination Process". Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Identification and Resolution of Problems The team concluded that the corrective action program at Palo Verde Nuclear Generating Station was generally effective. The team concluded that site personnel identify problems at a low threshold and enter them into the corrective action program. The licensee utilizes a rigorous screening process to characterize issues and that the vast majority of issues are appropriately evaluated and adequate corrective actions are taken. The team did identify isolated cases where problem evaluation could have been more effective at addressing the underlying causes of issues as well as a number of examples where corrective actions were not timely or adequate to address identified problems. The team also determined that though the overall process for identifying and correcting issues was well established, certain incidents of procedural violations associated with corrective action program processes led to delays and less than adequate actions to correct material deficiencies. Though the team identified areas in which the licensee could improve its corrective action program, the overall process was determined to be effective in identifying and correcting conditions adverse to quality.

The licensee appropriately evaluated industry operating experience for relevance to the facility, entered applicable items in the corrective action program, and subsequently utilized operating experience in root and apparent cause evaluations. The team did determine that that the licensee could improve its utilization of operating experience to prevent the occurrence of similar events at Palo Verde. The team determined that the licensee performed effective quality assurance audits and self assessments.

The team performed seven safety culture focus group discussions involving approximately 70 licensee personnel in order to assess the safety conscious work environment of the site. The team felt that a strong safety conscious work environment existed in most of the work groups interviewed; however, one work group interviewed exhibited weaknesses in this area. Specifically, the team found that although there were many individuals who felt comfortable raising safety concerns without fear of retaliation, there were some individuals in the operations department who expressed the perception that they would or might be retaliated against for raising certain safety concerns using certain avenues available to them. In all instances, these individuals stated they would use one avenue or another to raise their concerns.

Cornerstone: Mitigating Systems

Green.

Inspectors identified a Green cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action", for the failure of the licensee to correct a significant condition adverse to quality associated with the emergency diesel generator fuel oil transfer pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections. As an interim corrective action, splices have been placed in the cabling to prevent water from reaching the motor terminations. Due to the licensee's failure to restore compliance to a previous violation (NCV 05000529/2009004-02) within a reasonable time, this violation is being cited as a Notice of Violation consistent with the NRC Enforcement Policy. This has been entered into the licensee's corrective action program as Condition Report Disposition Request 3529151.

The performance deficiency associated with this finding was the failure of the licensee to correct a significant condition adverse to quality and prevent recurrence. The finding is more than minor because it affected the equipment performance attribute of the Mitigating Systems Cornerstone and affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.

Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was determined to require a Phase 2 and Phase 3 analysis by a senior reactor analyst because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. The senior reactor analyst performed a bounding Phase 3 significance determination and found the finding to be of very low safety significance (Green). The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump. The finding had a cross-cutting aspect in the area of Problem Identification and Resolution associated with the corrective action program component because the licensee failed to thoroughly evaluate problems such that the resolutions address causes and extent of condition, as necessary. [P.1.(c)] (Section 4OA2)

B. Licensee-Identified Violations

None.

REPORT DETAILS

4OA2 Problem Identification and Resolution

The team based the following conclusions on the sample of corrective action documents that were initiated in the assessment period, which ranged from March 1, 2009, to the end of the on-site portion of the inspection on December 17, 2010.

a. Assessment of the Corrective Action Program Effectiveness

(1) Inspection Scope The team reviewed approximately 250 Condition Report Disposition Requests, Action Requests, and Corrective Action Items, including associated root cause, apparent cause, and direct cause evaluations, to determine if problems were being properly identified, characterized, and entered into the corrective action program for evaluation and resolution. The team reviewed a sample of operability determinations, self- assessments, and various other documents related to the corrective action program. The team evaluated the licensee's efforts in establishing the scope of problems by reviewing selected logs, work requests, self-assessments results, audits, system health reports, action plans, and results from surveillance tests and preventive maintenance tasks. The team reviewed work requests and attended the licensee's daily action request review committee and the management review committee meetings to assess the reporting threshold, prioritization efforts, and significance determination process, as well as observing the interfaces with the operability assessment and work control processes when applicable. The team's review included verifying the licensee considered the full extent of cause and extent of condition for problems, as well as how the licensee assessed generic implications and previous occurrences. The team assessed the timeliness and effectiveness of corrective actions, completed or planned, and looked for additional examples of similar problems. The team conducted interviews with plant personnel to identify other processes that may exist where problems may be identified and addressed outside the corrective action program.

The team also reviewed corrective action documents that addressed past NRC-identified violations to ensure that the corrective action addressed the issues as described in the inspection reports. The inspectors reviewed a sample of corrective actions closed to other corrective action documents to ensure that corrective actions were still appropriate and timely.

The team considered risk insights from both the NRC's and Palo Verde Nuclear Generating Station's risk assessments to focus the sample selection and plant tours on risk significant systems and components. The team selected the following risk significant systems or components for review: emergency diesel generators, fuel oil transfer pumps, maintenance rule A(1) systems, spray ponds, and fire protection features. The team also expanded their review to include evaluations involving the aging of electrical systems to determine whether problems were being effectively addressed. The team conducted walkdowns of systems to assess whether problems were identified and entered into the corrective action program.

(2) Assessments

(a) Assessment - Effectiveness of Problem Identification The team determined that the licensee was identifying problems at a very low threshold as demonstrated by approximately 25,000 Palo Verde Action Requests (PVARS)initiated per year. The team found the Palo Verde staff to be conscientious about documenting conditions adverse to quality in the corrective action program. The component design basis review initiative performed by the licensee was determined to be an effective method for identifying design issues and entering them into the corrective action program.

(b) Assessment - Effectiveness of Prioritization and Evaluation of Issues The team determined that the licensee's prioritization and evaluation of issues was effective however certain areas for improvement were identified. The team reviewed approximately 50 issues associated with operability reviews to assess the quality, timeliness, and prioritization of operability assessments. The team noted that the immediate and prompt operability assessments reviewed were completed in a timely manner. The team noted several examples of weaknesses in issue prioritization and evaluation as identified below:

• The team found that the failure to correct the significant condition adverse to quality associated with water intrusion into the Unit 2 fuel oil transfer pump motor termination boxes was in part due to an inadequate evaluation which did not produce actions that would prevent future water intrusion into the conduit and motor termination box. As a result of the inadequate evaluation performed after the 2009 water intrusion and pump failure event, corrective actions were not adequate to prevent further water intrusion and another pump failure occurred in 2010.

• While reviewing corrective actions from a past NRC NCV, the team found that Apparent Cause Evaluation (ACE) 3444581 for Unit 2 was closed to ACE 3357761 and ACE 3425538, both for Unit 3, on 04/09/2010. Although similar, this was not a duplicate condition. The team also found that ACE 3521811 for Unit 2 was closed to Unit 3's ACE 3425538 on 09/24/2010, which was also similar, but was not a duplicate condition. The licensee determined that procedural guidance for closure of corrective actions caused confusion, which led to the inadvertent closures of the described Apparent Cause Evaluations. Procedural guidance allows for closure of cause evaluations to other cause evaluations if the condition is a duplicate condition. The licensee had previously addressed the underlying technical issue as the result of a separate ACE, so the equipment impact was insignificant. The licensee entered this issue into the corrective action program as Palo Verde Action Request 3565255.

• It was noted that there was virtually no guidance in procedure 01DP-0AP12, "Palo Verde Action Request Processing," Appendix E, "Condition Classification Instruction" concerning the classification and prioritization of PVARS related to the emergency plan. The existing guidance contains a reference to screening conditions that affect licensing basis documents such as the emergency plan as significant, however inspectors identified one example (CRDR 3296869) where incorrectly made protective action recommendations during a training evolution were screened as adverse, despite the fact that they are a critical attribute in NRC evaluations of the licensee emergency plan response.

• Corrective action documents associated with NRC findings were not clearly marked as such, and NRC findings were not always subject to apparent cause evaluations. While Palo Verde procedures currently allow this practice it is noteworthy that the licensee has received two notices of violation during the assessment period associated with a failure to restore compliance from a previous NRC identified noncited violation.

• The team identified one example of an inappropriately cancelled prompt operability (POD) request where the Senior Reactor operator (SRO) and engineering decided to cancel a POD request even though the procedure required the underlying issue to be resolved in order to cancel a POD request.

The inspection team observed multiple screening processes associated with the licensee's corrective action program including the action request review committee, the condition review group, and the corrective action review board. During these processes the team observed rigorous technical discussions and challenges to conclusions, and felt that these were effective efforts to improve issue screening and prioritization

(c) Assessment - Effectiveness of Corrective Actions Although in the majority of cases corrective actions appeared to be effective, the inspection team identified a number of examples where corrective actions were ineffective or absent and determined that this area is the licensee's biggest challenge in maintaining an effective corrective action program. Examples of inadequate corrective actions included:

• A corrective action associated with the 2009 fuel oil transfer pump failure consisted of actions to perform meggering of the wiring insulation as well as a visual inspection of the motor termination box. This action was closed to work orders that did not always complete the visual inspections as required by the corrective actions. This was a contributing factor in the 2010 demand failure because it could have detected the water intrusion before it caused electrical problems during pump operations. This was a violation of corrective action program procedures which resulted in a failure to detect water in the motor termination box. It is possible that if these inspections had been performed as directed the 2010 fuel oil transfer pump failure might not have occurred.

• A review of operations standing orders revealed a heat trace on safety injection piping that had been broken since the spring of 2007. The purpose of the heat trace was to prevent the piping from the Reactor Makeup Water Tank (RWT) to the safety injection pumps from freezing. The broken heat trace had been entered into the licensee's corrective action program and a work order to repair it had been generated, however the corrective maintenance was not performed in a timely manner. The work was deferred from online to outage then back to online. The standing order required operators to check the temperature of the line when outside temperatures were below freezing. Though the overall impact to safety was minor due to operating experience that showed that the affected piping would stay warm when outside temperatures were freezing, the impact to operators was not considered, and the condition adverse to quality was left unrepaired for over 3 years.

• Inspectors noted further examples of untimely corrective actions associated with the spray pond chemistry addition system. These adverse conditions had little to no impact on nuclear safety however they did present a burden to operators as well as an industrial safety hazard. In one instance, a PVAR was generated in June, 2010 to clean the acid residue on the spray pond acid skid. A corrective action was generated to create a routine maintenance action to clean the skid, and this action was closed to a work order to perform the same. As of December, 2010, when the NRC inspection team arrived onsite the original acid residue had not yet been cleaned up.

Inspectors interviewed the management team responsible for implementing the licensee's procedure improvement process, and recognized the licensee's investment of time and resources as a positive initiative to improve in this area. The team was impressed with the overall scope of the project as well as its flexibility to allow immediate procedural changes and fixes as well as planned upgrades to the site's library of procedures. It was noted in at least one focus group discussion that procedures are improving but there is still much to accomplish in this area.

Overall, inspectors acknowledge the efforts to improve in various areas, but the team felt that there is a challenge in the area of corrective actions based on the number of items observed that fell short of addressing the underlying issue either due to timeliness, inaction, or inadequate action.

b. Assessment of the Use of Operating Experience

(1) Inspection Scope The team examined the licensee's program for reviewing industry operating experience, including reviewing the governing procedure and self-assessments.

A sample size of 22 operating experience condition report/disposition records that had been issued during the assessment period were reviewed to assess whether the licensee had appropriately evaluated the notification for relevance to the facility. The team also reviewed a number of Root and Apparent Cause Evaluations as well as various Licensee Event Reports that covered the assessment period to verify if the licensee had appropriately included industry-operating experience.

(2) Assessment Overall, the team determined that the licensee evaluated and utilized industry operating experience, but noted that the licensee could improve in using industry operating experience to prevent similar events onsite. This was based on reviewing a sample of 35 industry operating experience documents. The team concluded that the licensee was evaluating for industry operating experience by reviewing generic industry guidance. The team also concluded that the licensee identifies weaknesses in specific operating experience evaluations when performing root and apparent cause evaluations as well as documenting licensee event reports.

The team noted that root and apparent cause evaluations were required in order to evaluate whether internal or external operating experience was available associated with the event or failure being examined, and whether the evaluation and actions to address those items had been effective. Additionally, all root cause evaluations reviewed included an assessment as to whether the issue being evaluated had potential application to other similar components or plants. Several exceptions were noted where recent evaluations identified relevant operating experience, which had been ineffectively addressed.

• The licensee is currently evaluating a deficiency related to the emergency diesel generator fuel oil transfer pumps; however, the site has had repetitive occurrences of pump failures. Industry operating experience exists concerning water intrusion into submerged cables, which is part of the pathway for water intrusion which resulted in pump trips.

• Significant (Sig) CRDR 3417311 identifies an example of poor use of operating experience in that spray pond missile hazards continue to challenge Palo Verde's ultimate heat sink. The site received a non-cited violation due to violation of requirements, but the site did not implement effective corrective actions. This condition report discusses the weakness of using internal operating experience, which prevented the site from restoring compliance and resulted in issuance of a Notice of Violation.

• Sig CRDR 3303334 identifies another example of poor use of operating experience in that condensate storage tank temperature monitoring failed to be controlled. The site received a non-cited violation due to design control and continued to be deficient. This condition report discusses the weakness of using internal and external operating experience, which prevented the site from restoring compliance and resulted in issuance of a Notice of Violation.

• LER 1-2009-001 for unanalyzed RWT recirculation alignment root cause analysis was documented in CRDR 3287805. The evaluation determined that industry operating experience existed and presented an opportunity to identify and mitigate the consequence of the violation.

• NCV 2009006-05 identified that the licensee failed to adequately evaluate operating experience related to inverters.

The examples above show signs of weakness in converting operating experience into effective preventive measures. Palo Verde appears to be identifying the potential causes of events, however, the licensee has not always implemented the actions to address and trend these as conditions adverse to quality.

c. Assessment of Self-Assessments and Audits

(1) Inspection Scope The team reviewed a sample size of 32 licensee self-assessments and audits to assess whether the licensee was regularly identifying performance trends and effectively addressing them. The team reviewed audit reports to assess the effectiveness of assessments in specific areas. The team evaluated the use of self- and third-party assessments, the role of the quality assurance department, and the role of the performance improvement group related to licensee performance. The specific self-assessment documents reviewed are listed in the Attachment.

(2) Assessment The team concluded that the licensee had an effective self-assessment process. The team observed that Palo Verde Nuclear Generating Station's management was involved in developing the topics and objectives of self-assessments. The team observed that the assignment of the assessment team included members with the proper skills and experience to ensure an effective self-assessment was conducted, and the team members included individuals from outside organizations. Inspectors observed that the licensee was very effective in identifying issues. The team observed that certain licensee organizations performed several self-assessments within a short period of time, going above and beyond established requirements. For example, the operations department has performed 10 benchmarking trips and self-assessments in the past two years. The team also noted that a self-assessment had caught Part 21 screening issues and improved the licensee's warehouse discrepancy notice procedure to prevent future missed screenings of component discrepancies.

d. Assessment of Safety Conscious Work Environment

(1) Inspection Scope During the week of November 29, the inspection team reviewed the last safety culture assessment for the site and conducted seven safety culture focus groups involving approximately 70 individuals. The interviewees represented four functional organizations and ranged across permanent employees and contractors. No supervisors were allowed to attend the focus group discussions. The team conducted these interviews to assess whether conditions existed that would challenge the establishment of a safety conscious work environment at Palo Verde Nuclear Generating Station.

(2) Assessment The team found that three of the four work groups represented had a strong safety conscious work environment (SCWE) in that workers felt comfortable raising safety concerns without fear of retaliation. However, the team identified weaknesses in the safety conscious work environment in Operations. Specifically, the team found that although there were many individuals in Operations who felt comfortable raising safety concerns without fear of retaliation, there were also some individuals who expressed the perception (to varying degrees) that they would or might be retaliated against for raising certain safety concerns. This was not isolated to a single focus group; rather, the team observed elements of a SCWE weakness in each of the three focus groups conducted with licensed and non-licensed operators. Importantly, the team found that the individuals who felt they could be retaliated against stated they would still raise their issues using one of the avenues available to them (e.g., raising them to certain shift managers or to the NRC).

Operators in all three focus groups raised concerns about equipment reliability and change management. Concerns about equipment reliability and change management were also expressed in other focus groups, but to a lesser extent. The operators expressed that unless equipment problems involve a technical specification or plant operation, it can take a long time for the problem to get resolved. As a result of continually raising some equipment issues, several of the more experienced operators do not believe their opinions are valued.

Some operators stated that they would not raise safety issues up to certain supervisors or managers if they suspected the supervisor or manager disagreed with them. In addition, operators stated they might not use certain avenues available to them for fear of retaliation. For example, some individuals stated that they would not raise certain safety issues with certain supervisors and managers, and some stated they viewed it as another arm of management and not as an independent entity who would objectively review their concerns. The majority of the operators stated they were comfortable raising issues to the NRC however there were some that were concerned that they might be retaliated against if plant management found out. As a result, the team concluded that some operators were not comfortable using all avenues available to them for raising safety concerns. Nevertheless, all operators stated they would raise safety concerns using one avenue or another.

In conclusion, the team found that most of the work groups interviewed had a strong safety conscious work environment in that workers felt comfortable raising safety concerns without fear of retaliation. However, the team identified weaknesses in the safety conscious work environment in the operations functional group. In response to the team's finding, the licensee planned to further evaluate the safety conscious work environment in operations and develop corrective actions to address the issue. The licensee planned to expand its actions to other work groups if it identified weaknesses in those areas.

e. Specific Issues Identified During This Inspection

(1) Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps.

Introduction.

The inspectors identified a Green cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action", for the failure of the licensee to correct a significant condition adverse to quality associated with the emergency diesel generator fuel oil transfer pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections.

Description.

On September 15, 2010 operations personnel started a monthly emergency diesel generator surveillance per Procedure 40ST-9DG01, "Diesel Generator A Test." During the surveillance, operations personnel received an alarm in the control room indicating trouble with the fuel oil transfer pump. It was discovered that the supply breaker for the fuel oil transfer pump was open, interrupting power to the pump.

Each emergency diesel generator has its own fuel oil transfer system which consists of one fuel oil storage tank located in a vault, one diesel fuel oil transfer pump located inside the fuel oil storage tank, and one fuel oil day tank. The fuel oil transfer pump takes suction from the fuel oil storage tank and pumps fuel oil to the fuel oil day tank. The fuel oil day tank then supplies fuel to the emergency diesel generators via gravity drain. Operations personnel declared emergency diesel generator Train A inoperable and maintenance personnel began troubleshooting. During troubleshooting activities, water was found in the motor termination box which houses electrical connections for the fuel oil transfer pump. The presence of water was determined to be the cause of a phase-to-ground fault resulting in a pump trip. The Train B emergency diesel generator was started and completed a one hour loaded surveillance run to comply with technical specifications to ensure operability of the opposite train. An extent of condition review found the Train B emergency diesel generator fuel oil transfer pump motor termination box filled with corrosion products, indicative of water intrusion into the box. Inspectors were present to observe the discovery of corrosion products in the motor termination box. Subsequent megger testing found the resistance of the cable insulation to be reduced by two orders of magnitude, from 1 Gohm at installation 1 year prior to the present reading of 5 Mohm, and below inservice limits of 50 Mohm established by Specification 13-EN-306, "Installation Specification for Cable Splicing and Terminations." Water intrusion into the motor termination box was the same significant condition adverse to quality present and identified during a failure of the Train B emergency diesel generator fuel oil transfer pump in April 2009. This significant condition adverse to quality was not corrected prior to its recurrence.

On April 22, 2009, the Unit 2 emergency diesel generator Train B fuel oil transfer pump failed surveillance test Procedure 73ST-9DF01, "Diesel Fuel Oil Pump Inservice Test," due to a supply breaker opening and interrupting power to the pump. Operations personnel declared the Train B emergency diesel generator inoperable. The April 2009 event was considered a significant condition adverse to quality, and was classified as such using Procedure 01DP-AP12, "Palo Verde Action Request Processing" Appendix E. A root cause evaluation was performed under CRDR 3317532. The cause was determined to be water intrusion into the emergency diesel generator fuel oil transfer pump motor termination box which resulted in corrosion and failure of the pump during its surveillance. Subsequently, the root cause analysis identified inadequate maintenance practices as the cause of the water intrusion and did not identify the problem with standing water in the underground cable conduit, even though it had been initially identified in 2004. Maintenance practices were thought to be inadequate to identify and prevent water intrusion, as such, maintenance in response to previous water intrusion into the fuel oil vaults of Units 2 and 3 in December 2004 was thought to be the source of the water intrusion for the April 2009 event. A self-revealing noncited violation, NCV 05000529/2009004-02, was issued for failure to take corrective actions to correct a condition adverse to quality associated with water intrusion into the emergency diesel generator fuel oil transfer pump motor termination box, in response to inadequate corrective actions from the December 2004 water intrusion event.

Corrective actions to prevent recurrence of the significant condition adverse to quality, as determined in April 2009, were to perform visual inspections of the motor termination boxes and test insulation resistance of the motor power cable. The corrective actions credited with the completion of this task did not always perform a visual inspection as required. During the root cause for the most recent September 2010 event, the water intrusion path was identified. The water intrusion path consisted of standing water in the underground cable conduit leaking into degraded cable jacketing, creating a path to the motor termination box. The degraded cable jacketing was initially identified in 2004, and this water intrusion path was likely active during the April 2009 event. Corrective actions are to change the design of the cable and conduit and replace it with a cable qualified for water submergence. In the interim, splices have been placed in the cabling located in the junction box to prevent any water from migrating to the motor termination box.

Analysis.

The performance deficiency associated with this finding was the failure of the licensee to correct a significant condition adverse to quality and prevent recurrence. The finding is more than minor because it affects the equipment performance attribute of the Mitigating Systems Cornerstone and affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was determined to require a Phase 2 and Phase 3 analysis by a senior reactor analyst because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. A Region IV senior reactor analyst performed a Phase 2 significance determination using the pre-solved worksheet from the "Risk Informed Inspection Notebook for the Palo Verde Nuclear Generating Station," Revision 2.01a. The analyst assumed an exposure period of one year. The finding was potentially Yellow, which warranted further review. The senior reactor analyst subsequently performed a bounding Phase 3 significance determination, which determined that since the Delta-CDF was less than 1E-6 and the Delta- LERF was not a significant contributor to risk, this finding was of very low safety significance, Green. The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump. Since most of this same equipment remained available, the components helped to mitigate the significance of the finding.

The finding had a cross-cutting aspect in the area of Problem Identification and Resolution associated with the corrective action program component because the licensee failed to thoroughly evaluate problems such that the resolutions address causes and extent of condition, as necessary. [P.1.(c)]

Enforcement.

Title 10 CFR, Part 50, Appendix B, Criterion XVI, "Corrective Action," states, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. Contrary to the above, from April 2009 to September 2010, the licensee failed to assure that the cause of the significant condition adverse to quality was determined and that corrective action was taken to preclude repetition. Specifically, the licensee failed to correct a water intrusion path to the motor termination box for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections. As a corrective action, splices have been placed in the cabling to prevent water from reaching the motor terminations until a design change for the affected cabling can be implemented. A Phase 3 analysis determined the finding to be of very low safety significance and it has been entered into the licensee's corrective action program as CRDR 3529151. Due to the licensee's failure to restore compliance within a reasonable period of time after the violation was identified in NRC Inspection Report NCV 05000529/2009004-0. This violation is being cited as a Notice of Violation consistent with Section 2.3.2 of the NRC Enforcement Policy. VIO 05000529/2010008-01 "Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps"

4OA3 Follow-up of Events and Notices of Enforcement Discretion

a. Event Report Reviews

(1) Inspection Scope The inspectors reviewed the below listed licensee event report and related documents to assess:

(1) the accuracy of the licensee event report;

(2) the appropriateness of corrective actions;

(3) violations of requirements; and

(4) generic issues.

(2) Findings and Observations (Closed) Licensee Event Report 05000528/2010-003-00, Technical Specification Violation - Loss of Containment Building Equipment Hatch Closure Capability On May 8, 2010, during the Unit 1 refueling outage, the licensee discovered that the containment building equipment hatch was not capable of being fully closed while core alterations were in progress on May 2, 2010. Core alterations were in progress on two occasions totaling approximately four hours and forty-seven minutes. The cause was determined to be due to an inadequate post maintenance test following maintenance on both the east and west hatch hoist upper limit switches. The licensee adjusted these switches and the hatch was closed and secured. Inspectors reviewed this issue and documented a Green noncited violation of Technical Specification 5.4.1, "Procedures," in Section 1R12 of NRC Inspection Report 05000528;529;530/2010003. The licensee documented the inadequate post maintenance test in Palo Verde Action Request 3478220 and Condition Report Disposition Request 3431177. Inspectors reviewed the root cause evaluation and the licensee event report and determined that no additional violations of NRC requirements exist. This licensee event report is closed.

4OA5 Other Activities

(Closed) Notice of Violation (VIO) 05000528,529,530/2009005-01, "Failure to Establish Adequate Procedures to Control Potential Tornado Borne Missile Hazards Near the Essential Spray Ponds." Inspectors reviewed the Root Cause Evaluation and Corrective Actions associated with this Notice of Violation. Inspectors reviewed commitments made in the NOV response letter dated March 11, 2010 and actions taken to comply with commitments and found them to be adequate. This Notice of Violation is closed.

OA6 Meetings

Exit Meeting Summary

On December 17, 2010 the team presented the inspection results to Mr. R. Bement, Vice President of Nuclear Operations, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

B. Berles, Nuclear Engineering Department Leader, Component Performance Engineering

J. Bungard, RP Technical Services Section Leader, Radiological Engineering

R. Bramlett, Nuclear Security Programs Department Leader, Emergency Services Programs

C. Clark, Control Room Supervisor, Shift Technical Advisors

J. Copsey, Employee Concerns Director, Employee Concerns

K. Chavet, Regulatory Affairs Senior Consultant, Regulatory Affairs

J. Dotson, Field Services Technician, Emergency Preparedness

D. Elkington, Regulatory Affairs Senior Consultant, Regulatory Affairs

A. Hartwig, Engineering Section Leader, Instrumentation and Control Design

M. Heider, Nuclear Engineering Department Leader, Procurement and Obsolescence

M. Hypse, Technical Management Assistant, Design Electrical / I&C

T. Hook, Engineering Section Leader, PRA

C. Karlson, Engineering Section Leader, Design Electrical

M. Karbassian, Nuclear Engineering Director, CDBR Group

P. Koss, Chemistry Unit Section Leader, Chemistry Work Management

F. Lake, Performance Improvement Team Department Leader, CAP 2A

J. Livorsi, Employee Concerns Senior Consultant, Employee Concerns

L. Leavitt, Performance Improvement Section Leader, CAP 2A

W. Liu, Senior Engineer, Transient Analysis

D. Mims, VP Regulatory Affairs and Plant Improvement, Regulatory Affairs / Plant Improvement

D. Myatt, Nuclear Maintenance Section Leader, Employee Concerns

H. Mckaig, Nuclear Engineering Department Leader, Systems Engineering

M. McGhee, Operations Support Department Leader, Shift Technical Advisors

M. Muhs, Work Management Department Leader, Work Management Outage

P. McSparran, Nuclear Training Department Leader, Total Operations Training

R. Meeden, Engineering Section Leader, Design Mechanical NSSS

T. McCloud, Performance Improvement Section Leader, Operating Experience

E. O'Neill, TMA Management, Nuclear Assurance Administration

F. Oreshack, Regulatory Affairs Consultant, Regulatory Affairs

S. Pobst, Engineering Section Leader, Maintenance Rule

B. Routolo, Radiation Monitoring Section Leader, RP RMS/RP Initial Training

J. Rodriguez, Engineer II, Compliance

M. Renfroe, TMA Management, Plant manager Admin

E. Sterling, Nuclear Assurance Department leader, NAD Engineering and Support Admin.

J. Shannon, Engineering Section Leader, Design Civil

J. Sontchi, Nuclear Training Department leader, Tech/Mtce Training Admin

K. Schrecker, Engineering Section Leader, System Engineering - BOP

M. Shea, Director ImPACT, Safety Culture

R. Stroud, Licensing Section Leader, Licensing

S. Sawtschenko, Emergency Planning Program Department Leader, Emergency Preparedness

D. Wheeler, Performance Improvement Team Department Leader, CAP 1 Admin

M. Webb, Compliance Section Leader, Compliance

- 19 - Enclosure

T. Weber, Nuclear Regulatory Affairs Department Leader, Nuclear Regulatory Admin

G. Zuniga, Nuclear Maintenance Team Leader, Maintenance Admin A

NRC Personnel

N. Okeefe, RIV DRS

G. Replogle, RIV DRS

E. Ruesch, RIV DRS

- 20 - Attachment

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000529/2010008-01 VIO Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality

Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps

Closed

05000528-2010-003-00 LER 2010-003-00 Technical Specification Violation - Loss of Containment Building Equipment Hatch Closure Capability

05000528,529,530/2009005-01 VIO

Failure to Establish Adequate Procedures to Control Potential Tornado Borne Missile Hazards Near the Essential Spray Ponds

- 21 - Attachment