IR 05000528/2010008

IR 05000528-10-008, 05000529-10-008, 05000530-10-008, on 11/29/2010 - 12/17/2010, Palo Verde Nuclear Generating Station Units 1, 2, and 3 Biennial Baseline Inspection of the Identification and Resolution of Problems

ML110280467
Person / Time
Site:	Palo Verde
Issue date:	01/28/2011
From:	Hay M C Division of Reactor Safety IV
To:	Edington R K Arizona Public Service Co
References
EA-11-006 IR-10-008
Download: ML110280467 (37)

Text

January 28, 2011

EA-11-006 Randall K. Edington, Executive Vice President, Nuclear/CNO Mail Station 7602 Arizona Public Service Company P.O. Box 52034 Phoenix, AZ 85072-2034

SUBJECT: PALO VERDE NUCLEAR GENERATING STATION UNITS 1, 2 AND 3 - NRC PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT 05000528/2010008, 05000529/2010008, 05000530/2010008, AND NOTICE OF VIOLATION

Dear Mr. Edington:

On December 17, 2010, the U. S. Nuclear Regulatory Commission (NRC) completed a team inspection at Palo Verde Nuclear Generating Station, Units 1, 2, and 3. The enclosed report documents the inspection findings which were discussed on December 17, 2010, with Mr. R. Bement and other members of your staff. The inspection examined activities conducted under your license as they relate to identification and resolution of problems, safety and compliance with the Commission's rules and regulations and with the conditions of your operating license. The team reviewed selected procedures and records, observed activities, and interviewed personnel. The team also interviewed a representative sample of personnel regarding the condition of your safety-conscious work environment.

Based on the results of this inspection, the NRC has identified an issue that was evaluated under the risk significance determination process as having very low safety significance (Green). The NRC has also determined that a violation is associated with this issue. This violation was evaluated in accordance with the NRC Enforcement Policy. The current Enforcement Policy is included on the NRC's Web site at (http://www.nrc.gov/about-nrc/regulatory/enforcement/enforce-pol.html).] Arizona Public Service Company - 2 - The violation is cited in the enclosed Notice of Violation (Notice) and the circumstances surrounding it are described in detail in the subject inspection report. The violation involved failure to correct a significant condition adverse to quality associated with the Unit 2 emergency diesel generator fuel oil transfer pumps. The violation is being cited in the Notice because Palo Verde failed to restore compliance within a reasonable time after the previous violation was identified (NCV 05000529/2009-004-002), as specified in Section 2.3.2 of the Enforcement Policy. You are required to respond to this letter and should follow the instructions specified in the enclosed Notice when preparing your response. If you have additional information that you believe the NRC should consider, you may provide it in your response to the Notice. The NRC review of your response to the Notice will also determine whether further enforcement action is necessary to ensure compliance with regulatory requirements. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure(s), and your response, will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html. To the extent possible, your response should not include any personal privacy or proprietary information so that it can be made available to the Public without redaction.

Sincerely,/RA/ Michael C Hay, Chief Technical Support Branch Division of Reactor Safety Dockets: 50-528 50-529 50-530 Licenses: NPF-41 NPF-51 NPF-74

Enclosures:

Notice of Violation and Inspection Report 05000528/2010008, 05000529/2010008, and 05000530/2010008

w/Attachments:

1. Supplemental Information Initial Information Request 2. Palo Verde Fuel Oil Transfer Pump Failure Risk Assessment

Arizona Public Service Company - 3 - Mr. Steve Olea Arizona Corporation Commission 1200 W. Washington Street Phoenix, AZ 85007 Mr. Douglas K. Porter, Esq Southern California Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Chairman Maricopa County Board of Supervisors 301 W. Jefferson, 10th Floor Phoenix, AZ 85003 Mr. Aubrey V. Godwin, Director Arizona Radiation Regulatory Agency 4814 South 40 Street Phoenix, AZ 85040 Mr. Dwight C. Mims Vice President Regulatory Affairs and Plant Improvement Palo Verde Nuclear Generating Station Mail Station 7605 P.O. Box 52034 Phoenix, AZ 85072-2034 Mr. Ron Barnes, Director Regulatory Affairs Palo Verde Nuclear Generating Station MS 7638 P.O. Box 52034 Phoenix, AZ 85072-2034 Mr. Jeffrey T. Weikert Assistant General Counsel El Paso Electric Company Mail Location 167 123 W. Mills El Paso, TX 79901 Michael S. Green Senior Regulatory Counsel Pinnacle West Capital Corporation P.O. Box 52034, MS 8695 Phoenix, AZ 85072-2034

Arizona Public Service Company - 4 - Mr. Eric Tharp Los Angeles Department of Water & Power Southern California Public Power Authority P.O. Box 51111, Room 1255-C Los Angeles, CA 90051-0100 Mr. James Ray Public Service Company of New Mexico 2401 Aztec NE, MS Z110 Albuquerque, NM 87107-4224 Mr. Geoffrey M. Cook Southern California Edison Company 5000 Pacific Coast Hwy. Bldg. D21 San Clemente, CA 92672 Mr. Robert Henry Salt River Project 6504 East Thomas Road Scottsdale, AZ 85251 Mr. Brian Almon Public Utility Commission William B. Travis Building P.O. Box 13326 Austin, TX 78701-3326 Environmental Program Manager City of Phoenix Office of Environmental Programs 200 West Washington Street Phoenix, AZ 85003 Mr. John C. Taylor Director, Nuclear Generation El Paso Electric Company 340 East Palm Lane, Suite 310 Phoenix, AZ 85004 Chief, Technological Hazards Branch FEMA Region IX 1111 Broadway, Suite 1200 Oakland, CA 94607-4052 Jake Lefman Southern California Edison Company 5000 Pacific Coast Hwy, Bldg. D21 San Clemente, CA 92672 Arizona Public Service Company - 5 - Electronic distribution by RIV: Regional Administrator (Elmo.Collins@nrc.gov ) Deputy Regional Administrator (Art.Howell@nrc.gov ) DRP Director (Kriss.Kennedy@nrc.gov ) DRP Deputy Director (Troy.Pruett@nrc.gov ) DRS Director (Anton.Vegel@nrc.gov ) DRS Deputy Director (Vacant) Senior Resident Inspector (Tony.Brown@nrc.gov ) Resident Inspector (Joseph.Bashore@nrc.gov ) Resident Inspector (Mica.Baquera@nrc.gov ) Branch Chief, DRP/D (Ryan.Lantz@nrc.gov ) PV Administrative Assistant (Regina.McFadden@nrc.gov ) Acting Senior Project Engineer, DRP/D (Gerond.George@nrc.gov ) Project Engineer, DRP/D (Peter.Jayroe@nrc.gov ) Public Affairs Officer (Victor.Dricks@nrc.gov) Public Affairs Officer (Lara.Uselding@nrc.gov ) Project Manager (Randy.Hall@nrc.gov ) Branch Chief, DRS/TSB (Michael.Hay@nrc.gov ) RITS Coordinator (Marisa.Herrera@nrc.gov ) Regional Counsel (Karla.Fuller@nrc.gov ) Senior Enforcement Officer (Ray.Keller@nrc.gov ) Congressional Affairs Officer (Jenny.Weil@nrc.gov ) OEMail Resource OEDO RIV Coordinator (James.Trapp@nrc.gov ) DRS/TSB STA (Dale.Powers@nrc.gov )

R:_REACTORS\PV 2010008 - PAJ ML SUNSI Rev Compl. Yes No ADAMS Yes No Reviewer Initials PAJ Publicly Avail Yes No Sensitive Yes No Sens. Type Initials PAJ RIV: PE: DRP/D RI/DRP/D RIV:DRS/EB2 RIV:DRS/PSB2 RIV: DRS/TSB PAJayroe MTBaquera EDUribe TDBuchanan MVasquez /RA/ /RA/ /RA/ /RA/ /RA/ 1/25/11 1/25/11 1/25/11 1/25/11 1/26/11 RIV C:DRP/D C:DRS/TSB RLKellar RELantz MCHay /RA /RA /RA/ 1/27/11 1/27/11 1/28/11 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax Enclosure NOTICE OF VIOLATION Arizona Public Service Company Docket Nos: 50-528,-529,-530 Palo Verde Nuclear Generating Station License Nos: NPF-41, -51, -74 EA-11-006 During an NRC inspection conducted on November 29, 2010 through December 17, 2010, a violation of NRC requirements was identified. In accordance with NRC enforcement policy, the violation is listed below: 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action" requires, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. Contrary to the above, from April 2009 through September 2010, the licensee failed to correct a significant condition adverse to quality and implement adequate corrective actions to preclude repetition. Specifically, the licensee failed to correct a water intrusion path to the Unit 2 motor termination boxes for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections and a pump trip.

This Notice of Violation is associated with a Green Significance Determination Process finding. Pursuant to the provisions of 10 CFR 2.201, Arizona Public Service Company is hereby required to submit a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001 with a copy to the Regional Administrator, Region IV, and a copy to the NRC Resident Inspector at the facility that is the subject of this Notice, within 30 days of the date of the letter transmitting this Notice of Violation (Notice). This reply should be clearly marked as a "Reply to Notice of Violation EA-11-006," and should include: (1) the reason for the violation, or, if contested, the basis for disputing the violation or severity level; (2) the corrective steps that have been taken and the results achieved; (3) the corrective steps that will be taken to avoid further violations; and (4) the date when full compliance will be achieved. Your response may reference or include previous docketed correspondence, if the correspondence adequately addresses the required response. If an adequate reply is not received within the time specified in this Notice, an order or a Demand for Information may be issued as to why the license should not be modified, suspended, or revoked, or why such other action as may be proper should not be taken. Where good cause is shown, consideration will be given to extending the response time. If you contest this enforcement action, you should also provide a copy of your response, with the basis for your denial, to the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001.

Because your response will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the Enclosure NRC website at www.nrc.gov/reading-rm/pdr.html or www.nrc.gov/reading-rm/adams.html, to the extent possible, it should not include any personal privacy, proprietary, or safeguards information so that it can be made available to the public without redaction. If personal privacy or proprietary information is necessary to provide an acceptable response, then please provide a bracketed copy of your response that identifies the information that should be protected and a redacted copy of your response that deletes such information. If you request withholding of such material, you must specifically identify the portions of your response that you seek to have withheld and provide in detail the basis for your claim of withholding (e.g., explain why the disclosure of information will create an unwarranted invasion of personal privacy or provide the information required by 10 CFR 2.390(b) to support a request for withholding confidential commercial or financial information). If safeguards information is necessary to provide an acceptable response, please provide the level of protection described in 10 CFR 73.21. Dated this 28th day of January 2011. Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION IV Docket: 50-528, 50-529, 50-530 License: NPF-41, NPF-51, NPF-74 Report: 05000528/2010008; 05000529/2010008; 05000530/2010008 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station Units 1, 2, and 3 Location: 5951 S. Wintersburg Road Tonopah, Arizona Dates: November 29, 2010 through December 17, 2010 Inspectors: P. Jayroe, Project Engineer (Team Leader) M. Vasquez, Senior Reactor Inspector M. Baquera, Resident Inspector E. Uribe, Reactor Inspector T. Buchanan, Reactor Inspector Approved By: Michael C. Hay, Chief Technical Support Branch Division of Reactor Safety Enclosure

SUMMARY OF FINDINGS

IR 05000528; 05000529; 05000530/2010008; 11/29/2010 - 12/17/2010; Palo Verde Nuclear Generating Station Units 1, 2, and 3 Biennial Baseline Inspection of the Identification and Resolution of Problems.

The team inspection was performed by a senior reactor inspector, a project engineer, two reactor inspectors, and a resident inspector. One green finding of very low safety significance was identified during this inspection. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609, "Significance Determination Process". Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649, "Reactor Oversight Process," Revision 4, dated December 2006. Identification and Resolution of Problems The team concluded that the corrective action program at Palo Verde Nuclear Generating Station was generally effective. The team concluded that site personnel identify problems at a low threshold and enter them into the corrective action program. The licensee utilizes a rigorous screening process to characterize issues and that the vast majority of issues are appropriately evaluated and adequate corrective actions are taken. The team did identify isolated cases where problem evaluation could have been more effective at addressing the underlying causes of issues as well as a number of examples where corrective actions were not timely or adequate to address identified problems. The team also determined that though the overall process for identifying and correcting issues was well established, certain incidents of procedural violations associated with corrective action program processes led to delays and less than adequate actions to correct material deficiencies. Though the team identified areas in which the licensee could improve its corrective action program, the overall process was determined to be effective in identifying and correcting conditions adverse to quality. The licensee appropriately evaluated industry operating experience for relevance to the facility, entered applicable items in the corrective action program, and subsequently utilized operating experience in root and apparent cause evaluations. The team did determine that that the licensee could improve its utilization of operating experience to prevent the occurrence of similar events at Palo Verde. The team determined that the licensee performed effective quality assurance audits and self assessments.

The team performed seven safety culture focus group discussions involving approximately 70 licensee personnel in order to assess the safety conscious work environment of the site. The team felt that a strong safety conscious work environment existed in most of the work groups interviewed; however, one work group interviewed exhibited weaknesses in this area. Specifically, the team found that although there were many individuals who felt comfortable raising safety concerns without fear of retaliation, there were some individuals in the operations department who expressed the perception that they would or might be retaliated against for raising certain safety concerns using certain avenues available to them. In all instances, these individuals stated they would use one avenue or another to raise their concerns.

Cornerstone: Mitigating Systems

Green.

Inspectors identified a Green cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action", for the failure of the licensee to correct a significant condition adverse to quality associated with the emergency diesel generator fuel oil transfer pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections. As an interim corrective action, splices have been placed in the cabling to prevent water from reaching the motor terminations. Due to the licensee's failure to restore compliance to a previous violation (NCV 05000529/2009004-02) within a reasonable time, this violation is being cited as a Notice of Violation consistent with the NRC Enforcement Policy. This has been entered into the licensee's corrective action program as Condition Report Disposition Request 3529151. The performance deficiency associated with this finding was the failure of the licensee to correct a significant condition adverse to quality and prevent recurrence. The finding is more than minor because it affected the equipment performance attribute of the Mitigating Systems Cornerstone and affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.

Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was determined to require a Phase 2 and Phase 3 analysis by a senior reactor analyst because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. The senior reactor analyst performed a bounding Phase 3 significance determination and found the finding to be of very low safety significance (Green). The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump. The finding had a cross-cutting aspect in the area of Problem Identification and Resolution associated with the corrective action program component because the licensee failed to thoroughly evaluate problems such that the resolutions address causes and extent of condition, as necessary. [P.1.(c)] (Section 4OA2)

B. Licensee-Identified Violations

None.

REPORT DETAILS

4OA2 Problem Identification and Resolution

The team based the following conclusions on the sample of corrective action documents that were initiated in the assessment period, which ranged from March 1, 2009, to the end of the on-site portion of the inspection on December 17, 2010. a. Assessment of the Corrective Action Program Effectiveness (1) Inspection Scope The team reviewed approximately 250 Condition Report Disposition Requests, Action Requests, and Corrective Action Items, including associated root cause, apparent cause, and direct cause evaluations, to determine if problems were being properly identified, characterized, and entered into the corrective action program for evaluation and resolution. The team reviewed a sample of operability determinations, self- assessments, and various other documents related to the corrective action program. The team evaluated the licensee's efforts in establishing the scope of problems by reviewing selected logs, work requests, self-assessments results, audits, system health reports, action plans, and results from surveillance tests and preventive maintenance tasks. The team reviewed work requests and attended the licensee's daily action request review committee and the management review committee meetings to assess the reporting threshold, prioritization efforts, and significance determination process, as well as observing the interfaces with the operability assessment and work control processes when applicable. The team's review included verifying the licensee considered the full extent of cause and extent of condition for problems, as well as how the licensee assessed generic implications and previous occurrences. The team assessed the timeliness and effectiveness of corrective actions, completed or planned, and looked for additional examples of similar problems. The team conducted interviews with plant personnel to identify other processes that may exist where problems may be identified and addressed outside the corrective action program.

The team also reviewed corrective action documents that addressed past NRC-identified violations to ensure that the corrective action addressed the issues as described in the inspection reports. The inspectors reviewed a sample of corrective actions closed to other corrective action documents to ensure that corrective actions were still appropriate and timely. The team considered risk insights from both the NRC's and Palo Verde Nuclear Generating Station's risk assessments to focus the sample selection and plant tours on risk significant systems and components. The team selected the following risk significant systems or components for review: emergency diesel generators, fuel oil transfer pumps, maintenance rule A(1) systems, spray ponds, and fire protection features. The team also expanded their review to include evaluations involving the aging of electrical systems to determine whether problems were being effectively addressed. The team conducted walkdowns of systems to assess whether problems were identified and entered into the corrective action program.

(2) Assessments (a) Assessment - Effectiveness of Problem Identification The team determined that the licensee was identifying problems at a very low threshold as demonstrated by approximately 25,000 Palo Verde Action Requests (PVARS)initiated per year. The team found the Palo Verde staff to be conscientious about documenting conditions adverse to quality in the corrective action program. The component design basis review initiative performed by the licensee was determined to be an effective method for identifying design issues and entering them into the corrective action program.

(b) Assessment - Effectiveness of Prioritization and Evaluation of Issues The team determined that the licensee's prioritization and evaluation of issues was effective however certain areas for improvement were identified. The team reviewed approximately 50 issues associated with operability reviews to assess the quality, timeliness, and prioritization of operability assessments. The team noted that the immediate and prompt operability assessments reviewed were completed in a timely manner. The team noted several examples of weaknesses in issue prioritization and evaluation as identified below:

• The team found that the failure to correct the significant condition adverse to quality associated with water intrusion into the Unit 2 fuel oil transfer pump motor termination boxes was in part due to an inadequate evaluation which did not produce actions that would prevent future water intrusion into the conduit and motor termination box. As a result of the inadequate evaluation performed after the 2009 water intrusion and pump failure event, corrective actions were not adequate to prevent further water intrusion and another pump failure occurred in 2010.

• While reviewing corrective actions from a past NRC NCV, the team found that Apparent Cause Evaluation (ACE) 3444581 for Unit 2 was closed to ACE 3357761 and ACE 3425538, both for Unit 3, on 04/09/2010. Although similar, this was not a duplicate condition. The team also found that ACE 3521811 for Unit 2 was closed to Unit 3's ACE 3425538 on 09/24/2010, which was also similar, but was not a duplicate condition. The licensee determined that procedural guidance for closure of corrective actions caused confusion, which led to the inadvertent closures of the described Apparent Cause Evaluations. Procedural guidance allows for closure of cause evaluations to other cause evaluations if the condition is a duplicate condition. The licensee had previously addressed the underlying technical issue as the result of a separate ACE, so the equipment impact was insignificant. The licensee entered this issue into the corrective action program as Palo Verde Action Request 3565255.

• It was noted that there was virtually no guidance in procedure 01DP-0AP12, "Palo Verde Action Request Processing," Appendix E, "Condition Classification Instruction" concerning the classification and prioritization of PVARS related to the emergency plan. The existing guidance contains a reference to screening conditions that affect licensing basis documents such as the emergency plan as significant, however inspectors identified one example (CRDR 3296869) where incorrectly made protective action recommendations during a training evolution were screened as adverse, despite the fact that they are a critical attribute in NRC evaluations of the licensee emergency plan response.

• Corrective action documents associated with NRC findings were not clearly marked as such, and NRC findings were not always subject to apparent cause evaluations. While Palo Verde procedures currently allow this practice it is noteworthy that the licensee has received two notices of violation during the assessment period associated with a failure to restore compliance from a previous NRC identified noncited violation.

• The team identified one example of an inappropriately cancelled prompt operability (POD) request where the Senior Reactor operator (SRO) and engineering decided to cancel a POD request even though the procedure required the underlying issue to be resolved in order to cancel a POD request. The inspection team observed multiple screening processes associated with the licensee's corrective action program including the action request review committee, the condition review group, and the corrective action review board. During these processes the team observed rigorous technical discussions and challenges to conclusions, and felt that these were effective efforts to improve issue screening and prioritization (c) Assessment - Effectiveness of Corrective Actions Although in the majority of cases corrective actions appeared to be effective, the inspection team identified a number of examples where corrective actions were ineffective or absent and determined that this area is the licensee's biggest challenge in maintaining an effective corrective action program. Examples of inadequate corrective actions included:

• A corrective action associated with the 2009 fuel oil transfer pump failure consisted of actions to perform meggering of the wiring insulation as well as a visual inspection of the motor termination box. This action was closed to work orders that did not always complete the visual inspections as required by the corrective actions. This was a contributing factor in the 2010 demand failure because it could have detected the water intrusion before it caused electrical problems during pump operations. This was a violation of corrective action program procedures which resulted in a failure to detect water in the motor termination box. It is possible that if these inspections had been performed as directed the 2010 fuel oil transfer pump failure might not have occurred.

• A review of operations standing orders revealed a heat trace on safety injection piping that had been broken since the spring of 2007. The purpose of the heat trace was to prevent the piping from the Reactor Makeup Water Tank (RWT) to the safety injection pumps from freezing. The broken heat trace had been entered into the licensee's corrective action program and a work order to repair it had been generated, however the corrective maintenance was not performed in a timely manner. The work was deferred from online to outage then back to online. The standing order required operators to check the temperature of the line when outside temperatures were below freezing. Though the overall impact to safety was minor due to operating experience that showed that the affected piping would stay warm when outside temperatures were freezing, the impact to operators was not considered, and the condition adverse to quality was left unrepaired for over 3 years.

• Inspectors noted further examples of untimely corrective actions associated with the spray pond chemistry addition system. These adverse conditions had little to no impact on nuclear safety however they did present a burden to operators as well as an industrial safety hazard. In one instance, a PVAR was generated in June, 2010 to clean the acid residue on the spray pond acid skid. A corrective action was generated to create a routine maintenance action to clean the skid, and this action was closed to a work order to perform the same. As of December, 2010, when the NRC inspection team arrived onsite the original acid residue had not yet been cleaned up.

Inspectors interviewed the management team responsible for implementing the licensee's procedure improvement process, and recognized the licensee's investment of time and resources as a positive initiative to improve in this area. The team was impressed with the overall scope of the project as well as its flexibility to allow immediate procedural changes and fixes as well as planned upgrades to the site's library of procedures. It was noted in at least one focus group discussion that procedures are improving but there is still much to accomplish in this area. Overall, inspectors acknowledge the efforts to improve in various areas, but the team felt that there is a challenge in the area of corrective actions based on the number of items observed that fell short of addressing the underlying issue either due to timeliness, inaction, or inadequate action.

b. Assessment of the Use of Operating Experience (1) Inspection Scope The team examined the licensee's program for reviewing industry operating experience, including reviewing the governing procedure and self-assessments.

A sample size of 22 operating experience condition report/disposition records that had been issued during the assessment period were reviewed to assess whether the licensee had appropriately evaluated the notification for relevance to the facility. The team also reviewed a number of Root and Apparent Cause Evaluations as well as various Licensee Event Reports that covered the assessment period to verify if the licensee had appropriately included industry-operating experience.

(2) Assessment Overall, the team determined that the licensee evaluated and utilized industry operating experience, but noted that the licensee could improve in using industry operating experience to prevent similar events onsite. This was based on reviewing a sample of 35 industry operating experience documents. The team concluded that the licensee was evaluating for industry operating experience by reviewing generic industry guidance. The team also concluded that the licensee identifies weaknesses in specific operating experience evaluations when performing root and apparent cause evaluations as well as documenting licensee event reports.

The team noted that root and apparent cause evaluations were required in order to evaluate whether internal or external operating experience was available associated with the event or failure being examined, and whether the evaluation and actions to address those items had been effective. Additionally, all root cause evaluations reviewed included an assessment as to whether the issue being evaluated had potential application to other similar components or plants. Several exceptions were noted where recent evaluations identified relevant operating experience, which had been ineffectively addressed.

• The licensee is currently evaluating a deficiency related to the emergency diesel generator fuel oil transfer pumps; however, the site has had repetitive occurrences of pump failures. Industry operating experience exists concerning water intrusion into submerged cables, which is part of the pathway for water intrusion which resulted in pump trips.

• Significant (Sig) CRDR 3417311 identifies an example of poor use of operating experience in that spray pond missile hazards continue to challenge Palo Verde's ultimate heat sink. The site received a non-cited violation due to violation of requirements, but the site did not implement effective corrective actions. This condition report discusses the weakness of using internal operating experience, which prevented the site from restoring compliance and resulted in issuance of a Notice of Violation.

• Sig CRDR 3303334 identifies another example of poor use of operating experience in that condensate storage tank temperature monitoring failed to be controlled. The site received a non-cited violation due to design control and continued to be deficient. This condition report discusses the weakness of using internal and external operating experience, which prevented the site from restoring compliance and resulted in issuance of a Notice of Violation.

• LER 1-2009-001 for unanalyzed RWT recirculation alignment root cause analysis was documented in CRDR 3287805. The evaluation determined that industry operating experience existed and presented an opportunity to identify and mitigate the consequence of the violation.

• NCV 2009006-05 identified that the licensee failed to adequately evaluate operating experience related to inverters.

The examples above show signs of weakness in converting operating experience into effective preventive measures. Palo Verde appears to be identifying the potential causes of events, however, the licensee has not always implemented the actions to address and trend these as conditions adverse to quality.

c. Assessment of Self-Assessments and Audits (1) Inspection Scope The team reviewed a sample size of 32 licensee self-assessments and audits to assess whether the licensee was regularly identifying performance trends and effectively addressing them. The team reviewed audit reports to assess the effectiveness of assessments in specific areas. The team evaluated the use of self- and third-party assessments, the role of the quality assurance department, and the role of the performance improvement group related to licensee performance. The specific self-assessment documents reviewed are listed in the Attachment.

(2) Assessment The team concluded that the licensee had an effective self-assessment process. The team observed that Palo Verde Nuclear Generating Station's management was involved in developing the topics and objectives of self-assessments. The team observed that the assignment of the assessment team included members with the proper skills and experience to ensure an effective self-assessment was conducted, and the team members included individuals from outside organizations. Inspectors observed that the licensee was very effective in identifying issues. The team observed that certain licensee organizations performed several self-assessments within a short period of time, going above and beyond established requirements. For example, the operations department has performed 10 benchmarking trips and self-assessments in the past two years. The team also noted that a self-assessment had caught Part 21 screening issues and improved the licensee's warehouse discrepancy notice procedure to prevent future missed screenings of component discrepancies.

d. Assessment of Safety Conscious Work Environment (1) Inspection Scope During the week of November 29, the inspection team reviewed the last safety culture assessment for the site and conducted seven safety culture focus groups involving approximately 70 individuals. The interviewees represented four functional organizations and ranged across permanent employees and contractors. No supervisors were allowed to attend the focus group discussions. The team conducted these interviews to assess whether conditions existed that would challenge the establishment of a safety conscious work environment at Palo Verde Nuclear Generating Station.

(2) Assessment The team found that three of the four work groups represented had a strong safety conscious work environment (SCWE) in that workers felt comfortable raising safety concerns without fear of retaliation. However, the team identified weaknesses in the safety conscious work environment in Operations. Specifically, the team found that although there were many individuals in Operations who felt comfortable raising safety concerns without fear of retaliation, there were also some individuals who expressed the perception (to varying degrees) that they would or might be retaliated against for raising certain safety concerns. This was not isolated to a single focus group; rather, the team observed elements of a SCWE weakness in each of the three focus groups conducted with licensed and non-licensed operators. Importantly, the team found that the individuals who felt they could be retaliated against stated they would still raise their issues using one of the avenues available to them (e.g., raising them to certain shift managers or to the NRC).

Operators in all three focus groups raised concerns about equipment reliability and change management. Concerns about equipment reliability and change management were also expressed in other focus groups, but to a lesser extent. The operators expressed that unless equipment problems involve a technical specification or plant operation, it can take a long time for the problem to get resolved. As a result of continually raising some equipment issues, several of the more experienced operators do not believe their opinions are valued.

Some operators stated that they would not raise safety issues up to certain supervisors or managers if they suspected the supervisor or manager disagreed with them. In addition, operators stated they might not use certain avenues available to them for fear of retaliation. For example, some individuals stated that they would not raise certain safety issues with certain supervisors and managers, and some stated they viewed it as another arm of management and not as an independent entity who would objectively review their concerns. The majority of the operators stated they were comfortable raising issues to the NRC however there were some that were concerned that they might be retaliated against if plant management found out. As a result, the team concluded that some operators were not comfortable using all avenues available to them for raising safety concerns. Nevertheless, all operators stated they would raise safety concerns using one avenue or another.

In conclusion, the team found that most of the work groups interviewed had a strong safety conscious work environment in that workers felt comfortable raising safety concerns without fear of retaliation. However, the team identified weaknesses in the safety conscious work environment in the operations functional group. In response to the team's finding, the licensee planned to further evaluate the safety conscious work environment in operations and develop corrective actions to address the issue. The licensee planned to expand its actions to other work groups if it identified weaknesses in those areas.

e. Specific Issues Identified During This Inspection (1) Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps.

Introduction.

The inspectors identified a Green cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action", for the failure of the licensee to correct a significant condition adverse to quality associated with the emergency diesel generator fuel oil transfer pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections.

Description.

On September 15, 2010 operations personnel started a monthly emergency diesel generator surveillance per Procedure 40ST-9DG01, "Diesel Generator A Test." During the surveillance, operations personnel received an alarm in the control room indicating trouble with the fuel oil transfer pump. It was discovered that the supply breaker for the fuel oil transfer pump was open, interrupting power to the pump.

Each emergency diesel generator has its own fuel oil transfer system which consists of one fuel oil storage tank located in a vault, one diesel fuel oil transfer pump located inside the fuel oil storage tank, and one fuel oil day tank. The fuel oil transfer pump takes suction from the fuel oil storage tank and pumps fuel oil to the fuel oil day tank. The fuel oil day tank then supplies fuel to the emergency diesel generators via gravity drain. Operations personnel declared emergency diesel generator Train A inoperable and maintenance personnel began troubleshooting. During troubleshooting activities, water was found in the motor termination box which houses electrical connections for the fuel oil transfer pump. The presence of water was determined to be the cause of a phase-to-ground fault resulting in a pump trip. The Train B emergency diesel generator was started and completed a one hour loaded surveillance run to comply with technical specifications to ensure operability of the opposite train. An extent of condition review found the Train B emergency diesel generator fuel oil transfer pump motor termination box filled with corrosion products, indicative of water intrusion into the box. Inspectors were present to observe the discovery of corrosion products in the motor termination box. Subsequent megger testing found the resistance of the cable insulation to be reduced by two orders of magnitude, from 1 Gohm at installation 1 year prior to the present reading of 5 Mohm, and below inservice limits of 50 Mohm established by Specification 13-EN-306, "Installation Specification for Cable Splicing and Terminations." Water intrusion into the motor termination box was the same significant condition adverse to quality present and identified during a failure of the Train B emergency diesel generator fuel oil transfer pump in April 2009. This significant condition adverse to quality was not corrected prior to its recurrence.

On April 22, 2009, the Unit 2 emergency diesel generator Train B fuel oil transfer pump failed surveillance test Procedure 73ST-9DF01, "Diesel Fuel Oil Pump Inservice Test," due to a supply breaker opening and interrupting power to the pump. Operations personnel declared the Train B emergency diesel generator inoperable. The April 2009 event was considered a significant condition adverse to quality, and was classified as such using Procedure 01DP-AP12, "Palo Verde Action Request Processing" Appendix E. A root cause evaluation was performed under CRDR 3317532. The cause was determined to be water intrusion into the emergency diesel generator fuel oil transfer pump motor termination box which resulted in corrosion and failure of the pump during its surveillance. Subsequently, the root cause analysis identified inadequate maintenance practices as the cause of the water intrusion and did not identify the problem with standing water in the underground cable conduit, even though it had been initially identified in 2004. Maintenance practices were thought to be inadequate to identify and prevent water intrusion, as such, maintenance in response to previous water intrusion into the fuel oil vaults of Units 2 and 3 in December 2004 was thought to be the source of the water intrusion for the April 2009 event. A self-revealing noncited violation, NCV 05000529/2009004-02, was issued for failure to take corrective actions to correct a condition adverse to quality associated with water intrusion into the emergency diesel generator fuel oil transfer pump motor termination box, in response to inadequate corrective actions from the December 2004 water intrusion event. Corrective actions to prevent recurrence of the significant condition adverse to quality, as determined in April 2009, were to perform visual inspections of the motor termination boxes and test insulation resistance of the motor power cable. The corrective actions credited with the completion of this task did not always perform a visual inspection as required. During the root cause for the most recent September 2010 event, the water intrusion path was identified. The water intrusion path consisted of standing water in the underground cable conduit leaking into degraded cable jacketing, creating a path to the motor termination box. The degraded cable jacketing was initially identified in 2004, and this water intrusion path was likely active during the April 2009 event. Corrective actions are to change the design of the cable and conduit and replace it with a cable qualified for water submergence. In the interim, splices have been placed in the cabling located in the junction box to prevent any water from migrating to the motor termination box.

Analysis.

The performance deficiency associated with this finding was the failure of the licensee to correct a significant condition adverse to quality and prevent recurrence. The finding is more than minor because it affects the equipment performance attribute of the Mitigating Systems Cornerstone and affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was determined to require a Phase 2 and Phase 3 analysis by a senior reactor analyst because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. A Region IV senior reactor analyst performed a Phase 2 significance determination using the pre-solved worksheet from the "Risk Informed Inspection Notebook for the Palo Verde Nuclear Generating Station," Revision 2.01a. The analyst assumed an exposure period of one year. The finding was potentially Yellow, which warranted further review. The senior reactor analyst subsequently performed a bounding Phase 3 significance determination, which determined that since the Delta-CDF was less than 1E-6 and the Delta- LERF was not a significant contributor to risk, this finding was of very low safety significance, Green. The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump. Since most of this same equipment remained available, the components helped to mitigate the significance of the finding. The finding had a cross-cutting aspect in the area of Problem Identification and Resolution associated with the corrective action program component because the licensee failed to thoroughly evaluate problems such that the resolutions address causes and extent of condition, as necessary. [P.1.(c)]

Enforcement.

Title 10 CFR, Part 50, Appendix B, Criterion XVI, "Corrective Action," states, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. Contrary to the above, from April 2009 to September 2010, the licensee failed to assure that the cause of the significant condition adverse to quality was determined and that corrective action was taken to preclude repetition. Specifically, the licensee failed to correct a water intrusion path to the motor termination box for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections. As a corrective action, splices have been placed in the cabling to prevent water from reaching the motor terminations until a design change for the affected cabling can be implemented. A Phase 3 analysis determined the finding to be of very low safety significance and it has been entered into the licensee's corrective action program as CRDR 3529151. Due to the licensee's failure to restore compliance within a reasonable period of time after the violation was identified in NRC Inspection Report NCV 05000529/2009004-0. This violation is being cited as a Notice of Violation consistent with Section 2.3.2 of the NRC Enforcement Policy. VIO 05000529/2010008-01 "Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps"

4OA3 Follow-up of Events and Notices of Enforcement Discretion

a. Event Report Reviews (1) Inspection Scope The inspectors reviewed the below listed licensee event report and related documents to assess: (1) the accuracy of the licensee event report; (2) the appropriateness of corrective actions; (3) violations of requirements; and (4) generic issues.

(2) Findings and Observations (Closed) Licensee Event Report 05000528/2010-003-00, Technical Specification Violation - Loss of Containment Building Equipment Hatch Closure Capability On May 8, 2010, during the Unit 1 refueling outage, the licensee discovered that the containment building equipment hatch was not capable of being fully closed while core alterations were in progress on May 2, 2010. Core alterations were in progress on two occasions totaling approximately four hours and forty-seven minutes. The cause was determined to be due to an inadequate post maintenance test following maintenance on both the east and west hatch hoist upper limit switches. The licensee adjusted these switches and the hatch was closed and secured. Inspectors reviewed this issue and documented a Green noncited violation of Technical Specification 5.4.1, "Procedures," in Section 1R12 of NRC Inspection Report 05000528;529;530/2010003. The licensee documented the inadequate post maintenance test in Palo Verde Action Request 3478220 and Condition Report Disposition Request 3431177. Inspectors reviewed the root cause evaluation and the licensee event report and determined that no additional violations of NRC requirements exist. This licensee event report is closed.

4OA5 Other Activities

(Closed) Notice of Violation (VIO) 05000528,529,530/2009005-01, "Failure to Establish Adequate Procedures to Control Potential Tornado Borne Missile Hazards Near the Essential Spray Ponds." Inspectors reviewed the Root Cause Evaluation and Corrective Actions associated with this Notice of Violation. Inspectors reviewed commitments made in the NOV response letter dated March 11, 2010 and actions taken to comply with commitments and found them to be adequate. This Notice of Violation is closed.

OA6 Meetings

Exit Meeting Summary On December 17, 2010 the team presented the inspection results to Mr. R. Bement, Vice President of Nuclear Operations, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

B. Berles, Nuclear Engineering Department Leader, Component Performance Engineering

J. Bungard, RP Technical Services Section Leader, Radiological Engineering

R. Bramlett, Nuclear Security Programs Department Leader, Emergency Services Programs

C. Clark, Control Room Supervisor, Shift Technical Advisors

J. Copsey, Employee Concerns Director, Employee Concerns

K. Chavet, Regulatory Affairs Senior Consultant, Regulatory Affairs

J. Dotson, Field Services Technician, Emergency Preparedness

D. Elkington, Regulatory Affairs Senior Consultant, Regulatory Affairs

A. Hartwig, Engineering Section Leader, Instrumentation and Control Design

M. Heider, Nuclear Engineering Department Leader, Procurement and Obsolescence

M. Hypse, Technical Management Assistant, Design Electrical / I&C

T. Hook, Engineering Section Leader, PRA

C. Karlson, Engineering Section Leader, Design Electrical

M. Karbassian, Nuclear Engineering Director, CDBR Group

P. Koss, Chemistry Unit Section Leader, Chemistry Work Management

F. Lake, Performance Improvement Team Department Leader, CAP 2A

J. Livorsi, Employee Concerns Senior Consultant, Employee Concerns

L. Leavitt, Performance Improvement Section Leader, CAP 2A

W. Liu, Senior Engineer, Transient Analysis

D. Mims, VP Regulatory Affairs and Plant Improvement, Regulatory Affairs / Plant Improvement

D. Myatt, Nuclear Maintenance Section Leader, Employee Concerns

H. Mckaig, Nuclear Engineering Department Leader, Systems Engineering

M. McGhee, Operations Support Department Leader, Shift Technical Advisors

M. Muhs, Work Management Department Leader, Work Management Outage

P. McSparran, Nuclear Training Department Leader, Total Operations Training

R. Meeden, Engineering Section Leader, Design Mechanical NSSS

T. McCloud, Performance Improvement Section Leader, Operating Experience

E. O'Neill, TMA Management, Nuclear Assurance Administration

F. Oreshack, Regulatory Affairs Consultant, Regulatory Affairs

S. Pobst, Engineering Section Leader, Maintenance Rule

B. Routolo, Radiation Monitoring Section Leader, RP RMS/RP Initial Training

J. Rodriguez, Engineer II, Compliance

M. Renfroe, TMA Management, Plant manager Admin

E. Sterling, Nuclear Assurance Department leader, NAD Engineering and Support Admin.

J. Shannon, Engineering Section Leader, Design Civil

J. Sontchi, Nuclear Training Department leader, Tech/Mtce Training Admin

K. Schrecker, Engineering Section Leader, System Engineering - BOP

M. Shea, Director ImPACT, Safety Culture

R. Stroud, Licensing Section Leader, Licensing

S. Sawtschenko, Emergency Planning Program Department Leader, Emergency Preparedness

D. Wheeler, Performance Improvement Team Department Leader, CAP 1 Admin

M. Webb, Compliance Section Leader, Compliance

- 19 - Enclosure

T. Weber, Nuclear Regulatory Affairs Department Leader, Nuclear Regulatory Admin

G. Zuniga, Nuclear Maintenance Team Leader, Maintenance Admin A

NRC Personnel

N. Okeefe, RIV DRS

G. Replogle, RIV DRS

E. Ruesch, RIV DRS

- 20 - Attachment

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000529/2010008-01 VIO Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps

Closed

05000528-2010-003-00 LER 2010-003-00 Technical Specification Violation - Loss of Containment Building Equipment Hatch Closure Capability

05000528,529,530/2009005-01 VIO Failure to Establish Adequate Procedures to Control Potential Tornado Borne Missile Hazards Near the Essential Spray Ponds

- 21 - Attachment

Section 4OA2: Identification and Resolution of Problems

Procedures

NUMBER TITLE REVISION / DATE 01PR-0AP04 Corrective Action Program 5 90DP-0IP10 Condition Reporting 47 01DP-0AP12 Palo Verde Action Request Processing 14

40DP-9OP26 Operations PVAR Processing and Operability Determination / Functional Assessment 29 90DP-0IP12 Root Cause CRDR Evaluation 5 70DP-0MR01 Maintenance Rule 30 90DP-0IP14 Adverse CRDR Evaluation 4 01DP-0FI01 Management of Nuclear Projects 5 12DP-0MC08 Control of Purchasing Material and Equipment 21

74DP-9CY04 Systems Chemistry Specifications 64 81TD-0EE10 Design Change Process 24 74OP-9SP04 Essential Spray Pond Chemical Addition System Human-Machine Interface Operation 2 40OP-9SP-04 Spray Pond Chemical Addition System Train B 33 40OP-9SP-03 Spray Pond Chemical Addition System Train A 34 01DP-0XX-01 Control and Monitoring of Potential Tornado Borne Missiles 1 75DP-0RP02 Radioactive Contamination Control 12

EPIP-04 Emergency Operations Facility Actions 51 87DP-0CC08 Control of Vendor Documentation 18

01DP-0EM09 Employee Concerns Program 2 40DP-9OP15 Operator Challenges and Discrepancy Tracking 26 40OP-9PC07 Miscellaneous Fuel Pool Operations 57 40AL-9RK3A Panel B03A Alarm Responses 27 40AO-9ZZ06 Loss of Instrument Air 32

01DP-0AP01-01 Procedure Preparer's Administrative Guideline 4 01DP-0AP01-02 Procedure Reviewer's Administrative Guideline 2 01DP-0AP01 Procedure Process 43 01DP-0AP21 NATM Procedure Replacement Project 0 70DP-0RA01 Shutdown Risk Assessments 35

40OP-9ZZ23 Outage

GOP 59 90DP-OIP13 Apparent Cause CRDR Evaluation 5 65DP-0QQ01 Industry Operating Experience Review 25 40DP-9OP26 Operations PVAR Processing and Operability Determination/Functional Assessment 29 90DP-OIP15 Review CRDR Evaluation 15 01DP-0AP16 PVNGS Self Assessments and Benchmarking 7

60DP-0QQ02 Trend Analysis and Coding 22 40OP-9DG01 Emergency Diesel Generator A 66

- 22 - Attachment Procedures NUMBER TITLE REVISION / DATE 40ST-9DG01 Emergency Diesel Generator A Test 40 73ST-9DF01 Diesel Fuel Oil Transfer Pump Inservice Test 22 40OP-9DG02 Emergency Diesel Generator B 62

70DP-0RA05 Assessment and Management of Risk when Performing Maintenance in modes 1 and 2 17 73ST-9DF01 Diesel Fuel Oil Transfer Pump Inservice Test 22 93DP-0LC07 10

CFR 50.59 and 72.48 Screenings and Evaluations 21 93DP-0LC17 10

CFR 50.59 and 72.48 Guidance Manual 5

Condition Report

Action Items

CRAI 3112430

CRAI 3182155

CRAI 3182163

CRAI 3182175

CRAI 3182179

CRAI 3182185

CRAI 3182198

CRAI 3183978

CRAI 3183981

CRAI 3337954

CRAI 3488855

CRAI 3337957

CRAI 3476074

CRAI 3486807

CRAI 3149172

CRAI 3149165

CRAI 3149057

CRAI 3297439

CRAI 3297444

CRAI 3367740

CRAI 3307571

CRAI 3307579

CRAI 3309913

CRAI 3340171

CRAI 3477407

CRAI 2941932

CRAI 3407034

CRAI 3407039

CRAI 2988522

CRAI 3032686

CRAI 3075733

CRAI 3400753

CRAI 3270067

CRAI 3348269

CRAI 3140933

CRAI 3333356

CRAI 3337943

CRAI 3288652

CRAI 3333353

CRAI 3047302

CRAI 3437919

CRAI 3445238

CRAI 3392785

CRAI 3270067

CRAI 3295068

CRAI 3295070

CRAI 3295078

CRAI 3327046

CRAI 3330347

CRAI 3350208

CRAI 3420003

CRAI 3420005

CRAI 3334253

CRAI 3298139

CRAI 3298141

CRAI 3479994

CRAI 3487566

CRAI 3127427

CRAI 3293601

CRAI 3294085

CRAI 3112414

CRAI 3112425

CRAI 3212753

CRAI 3421079

CRAI 3297448

CRAI 3243265

CRAI 3381398

CRAI 3243265

CRAI 3381398

CRAI 3258233

CRAI 3301301

CRAI 3318967

CRAI 3300928

CRAI 3300889

CRAI 3300892

CRAI 3300912

CRAI 3560968

CRAI 3419280

CRAI 2919416

CRAI 2919417

CRAI 2771662

CRAI 2785586

CRAI 3030118

CRAI 3333020

CRAI 3380080

CRAI 3333028

CRAI 3333032

CRAI 3452548

CRAI 3342549

CRAI 2902572

CRAI 3563117

CRAI 3560693

CRAI 3563121

CRAI 3563122

CRAI 3560705

CRAI 3333033

CRAI 3342549

CRAI 3563060

CRAI 3560747

CRAI 3560727

Palo Verde Action Requests

PVAR 3171090

PVAR 3484106

PVAR 3103044

PVAR 3283865

PVAR 3327597

PVAR 3302093

PVAR 3567358

PVAR 3491342

PVAR 3492005

PVAR 3296740

PVAR 3303150

PVAR 3292077

PVAR 3185240

PVAR 3055680

PVAR 2968212

PVAR 3124491

PVAR 3302472

PVAR 3419941

PVAR 3419732

PVAR 3563820

PVAR 3511452

PVAR 3527166

PVAR 3529476

PVAR 3425898

PVAR 3529475

PVAR 3562840

PVAR 3209329

PVAR 3209031

PVAR 3055297

PVAR 3055294

PVAR 3274141

PVAR 3566562

PVAR 3009375

PVAR 3567849

PVAR 3567337

PVAR 3475479

PVAR 3009375

PVAR 3562636

PVAR 3562621

PVAR 3560617

- 23 - Attachment

Condition Report

Disposition Requests

CRDR 3171340

CRDR 3139278

CRDR 3139279

CRDR 3105139

CRDR 3308660

CRDR 3476071

CRDR 3417311

CRDR 3382436

CRDR 3125620

CRDR 3456438

CRDR 3479992

CRDR 3279750

CRDR 3282706

CRDR 3311298

CRDR 3282707

CRDR 3283802

CRDR 3340170

CRDR 3310925

CRDR 3293411

CRDR 3282708

CRDR 3484837

CRDR 3466318

CRDR 3451431

CRDR 3048866

CRDR 2897810

CRDR 3470763

CRDR 3394172

CRDR 3393609

CRDR 3297924

CRDR 3173844

CRDR 3314991

CRDR 3383919

CRDR 3358222

CRDR 3288651

CRDR 3211958

CRDR 3434085

CRDR 3111432

CRDR 3398042

CRDR 3296869

CRDR 3446100

CRDR 3297229

CRDR 3296421

CRDR 3292700

CRDR 3434624

CRDR 3280774

CRDR 3304057

CRDR 3219742

CRDR 3202957

CRDR 3560585

CRDR 3281353

CRDR 3289353

CRDR 3336555

CRDR 3411547

CRDR3411547

CRDR 3283802

CRDR 3185716

CRDR 3229998

CRDR 3282704

CRDR 3312554

CRDR 3383898

CRDR 3211958

CRDR 3246250

CRDR 3255331

CRDR 3279838

CRDR 3285918

CRDR 3296259

CRDR 3301459

CRDR 3304518

CRDR 3299343

CRDR 3301540

CRDR 3315776

CRDR 3323052

CRDR 3337682

CRDR 3287805

CRDR 3418378

CRDR 3303334

CRDR 3343933

CRDR 3434665

CRDR 3481401

CRDR 3485725

CRDR 3411074

CRDR 3411033

CRDR 3411161

CRDR 3343933

CRDR 3401304

CRDR 3411031

CRDR 3311158

CRDR 3411155

CRDR 3411152

CRDR 3487756

CRDR 3405785

CRDR 3481402

CRDR 2761657

CRDR 3012697

CRDR 31202747

CRDR 3529151

CRDR 2882166

CRDR 3517526

CRDR 3303334

CRDR 3347523

CRDR 3287805

CRDR 3086170

CRDR 3317532

CRDR 3418378

CRDR 2882166

CRDR 3390784

CRDR 3431177

CRDR 3425538

CRDR 3426364

CRDR 3562875

CRDR 3192389

CRDR 3059185

Work Orders

3236820

3291430

3291434

3339386

3097215

3334856

2944206

3057308

2967863

2934353

3304408

3388525

3527172

3534552

3534548

3558553

3529375

3527183

2785583

3316981

3539192

3529213

3529215

3529223

3529237

26319

3317320

2910651

3540782

3426319

Audits 2010-002 Chemistry Audit Report 2009-010 Operations Audit Report

2009-009 Corrective Action Audit Report 2010-001 Emergency Preparedness Audit Report 2009-003 Special Processes, Inservice Inspection & Testing Audit Report 2009-004 Material Control/Procurement Audit Report 2009-005 Fire Protection Audit Report 2009-007 Design Control Audit Report 2010-007 Technical Specifications & Administrative Controls Audit Report

2010-008 Radiation Protection Audit Report 2010-004 Maintenance And Refueling Outage Activities Audit Report

- 24 - Attachment 2009-006 Document Control Audit Report 2009-011 Verification Process Audit Report 2010-003

Maintenance Rule Audit Report 2010-006 Training and Qualification Audit Report Self Assessments

3278891

3287745

3332917

3424019

3188986

3456721

3456891

3531574

3511365

3350944

3525944

3511217

351109

3525944

3511217

3456930

3362726

3420895

Drawing Numbers

Title

Revision 02-M-DGP-001 Diesel Generator System Rev 52 02-E-DFB-001

Elementary Diagram Diesel Fuel Oil & Transfer System Rev 5 13-E-ZVU-0014 Underground Electrical Installation Rev 15 03-E-ZYU-010 Diesel Storage Tank Conduit Plan and Section Rev. 7 12-E-ZYU-009 Diesel Storage Tank Conduit Plan & Section Rev. 12

Miscellaneous Documents

NUMBER TITLE REVISION / DATE EWR

3339327 DG B Trip on Lube Oil Low Pressure Turbo During Cooldown 0 EJT12C00102 Electrical Maintenance Training Program Polar Crane JITT 9/23/2009 NGW16-L-0001-13 Radworker Dressout Practical August 19, 2008 NGW01-C-0001-36 Radiation Worker Training March 4, 2010 Audit Report 2010-01 Emergency Preparedness March 2, 2010 N/A Performance Improvement Department Qualification Requirements and Training Program Description 008 N/A Effectiveness Review SIIP Task 6.1.1.c

CRAI 3032686 September, 2009

LER 50-528/2010-003-00 Technical Specification - Loss of Containment Building Equipment Hatch Closure Capability July 7, 2010 S-09-0039 10

CFR 50.59 Screening for DFWO

3286021 0

SP-485 Industrial Safety Assessment In Support of

EWR 3491766 0 (CRAI 3127427) Engineering Evaluation of EDG Fluid Leakage and Potential Operability Concerns 2

EWR 3283182 EDG Slow Start 0

- 25 - Attachment

ODP-7 Technical Specifications 9 ECP02 Employee Concerns Program Guideline 10 N/A Unit 1 Control Room Deficiency Log Dec 15, 2010 MP1 Site Corrective Maintenance Metric Summary Nov 2010 MP2 Site Elective Maintenance Metric Summary Nov 2010 OP1 Site Operational Focus Indicator Summary Nov 2010

N/A Site top 10 technical Issues Summary Dec 2010 N/A Active Standing Orders Dec 2, 2010 N/A Units 1,2,3 Night Orders Nov 2010 N/A NATM Procedure Replacement Project Schedule 2011 N/A NATM Procedure Replacement Project Schedule 2010 N/A CRG Daily Agenda

Dec 1, 2010 N/A ARRC Daily Agenda Dec 2, 2010 N/A CARB Agenda Nov 30, 2010 13-VTD-G080-0137 General Electric CR120B 600 Volt Industrial Relay - Series June 11, 1992 P.O.

500514424 Element Assembly, Lube Oil Thermostatic Valve; Cooper-Bessember Part Number 2-05V-419-109 Nov 15, 2007 U3R15 R8 10-7 Unit 3 R15 PRA model Nov 10, 2010 13-EN-0301 Installation Specification for Electrical Cables in Conduit and Duct Banks Rev 5

13-EN-0306 Installation Specification for Cabling Splices and Terminations Rev 11 N/A Updated Final Safety Analysis Report, Section 9.5 Rev 11 13-EM-057 600V Control Cables

Site Health Reports January through December 2009

Site Health Reports January through December 2010

NRC Information Notice 2007-01

Regulatory Guide 1.137 Fuel-Oil Systems For Standby Diesel Generators

- 26 - Attachment Information Request October 12, 2010 Biennial Problem Identification and Resolution Inspection Palo Verde Nuclear Generating Station Inspection Report Number 05000528/529/530-2010008

This inspection will cover the period from March 1, 2009, through December 17, 2010.

All requested information should be limited to this period unless otherwise specified.

To the extent possible, the requested information should be provided electronically in Adobe PDF or Microsoft Office format.

Lists of documents should be provided in Microsoft Excel or a similar sortable format.

A supplemental information request will likely be sent during the week of November 9, 2010.

Please provide the following no later than November 2, 2010.

1. Document Lists Note: for these summary lists, please include the document/reference number, the document title or a description of the issue, initiation date, and current status. a. Summary list of all corrective action documents related to significant conditions adverse to quality that were opened, closed, or evaluated during the period b. Summary list of all corrective action documents related to conditions adverse to quality that were opened or closed during the period c. Summary lists of all corrective action documents which were upgraded or downgraded in priority/significance during the period d. Summary list of all corrective action documents that subsume or "roll up" one or more smaller issues for the period e. Summary lists of operator workarounds, engineering review requests and/or operability evaluations, temporary modifications, and control room and safety system deficiencies opened, closed, or evaluated during the period f. Summary list of plant safety issues raised or addressed by the Employee Concerns Program (or equivalent)

g. Summary list of all Apparent Cause Evaluations completed during the period h. Summary list of all Root Cause Evaluations planned or in progress but not complete at the end of the period

2. Full Documents, with Attachments a. Root Cause Evaluations completed during the period b. Quality assurance audits performed during the period

- 27 - Attachment c. All audits/surveillances performed during the period of the Corrective Action Program, of individual corrective actions, and of cause evaluations d. Corrective action activity reports, functional area self-assessments, and non-NRC third party assessments completed during the period (do not include INPO assessments) e. Corrective action documents generated during the period for the following: i. NCV's and Violations issued ii. LER's submitted f. Corrective action documents generated for the following (for those that were evaluated but determined not to be applicable, provide a summary list): i. NRC Information Notices, Bulletins, and Generic Letters issued or evaluated during the period ii. Part 21 reports issued or evaluated during the period iii. Vendor safety information letters (or equivalent) issued or evaluated during the period iv. Other external events and/or Operating Experience evaluated for applicability during the period g. Corrective action documents generated for the following: i. Emergency planning drills and tabletop exercises performed during the period ii. Maintenance preventable functional failures which occurred or were evaluated during the period iii. Adverse trends in equipment, processes, procedures, or programs which were evaluated during the period iv. Action items generated or addressed by plant safety review committees during the period

3. Logs and Reports a. Corrective action performance trending/tracking information generated during the period and broken down by functional organization b. Corrective action effectiveness review reports generated during the period

c. Current system health reports or similar information d. Radiation protection event logs during the period

- 28 - Attachment e. Security event logs and security incidents during the period (sensitive information can be provided by hard copy during first week on site) f. Employee Concern Program (or equivalent) logs (sensitive information can be provided by hard copy during first week on site) g. List of Training deficiencies, requests for training improvements, and simulator deficiencies for the period

4. Procedures a. Corrective action program procedures, to include initiation and evaluation procedures, operability determination procedures, apparent and root cause evaluation/determination procedures, and any other procedures which implement the corrective action program. b. Quality Assurance program procedures c. Employee Concerns Program (or equivalent) procedures d. Procedures which implement/maintain a Safety Conscious Work Environment

5. Other a. List of risk significant components and systems

b. Organization charts for plant staff and long-term/permanent contractors

Note: "Corrective action documents" refers to condition reports, notifications, action requests, cause evaluations, and/or other similar documents, as applicable.

This information should be uploaded on the Certrec IMS website no later than November 2, 2010.

In addition, all electronic documents should be loaded onto a CD or DVD and sent via overnight carrier to:

U.S. NRC Region

IV 612 E. Lamar Blvd. Suite 400 Arlington,

TX 76011-4125

Attn: Harry Freeman

Please note that the NRC is not currently able to accept electronic documents on thumb drives or other similar digital media.

- 29 - Attachment Palo Verde Fuel Oil Transfer Pump Failure Risk Assessment

The senior resident inspector performed the initial significance determination for the fuel oil transfer pump failure using the NRC Inspection Manual 0609, Attachment 0609.04, "Phase 1 -

Initial Screening and Characterization of Findings."

The finding screened to a Phase 2 significance determination because it involved a loss of one train of safety related equipment for greater than its technical specification allowed outage time.

A Region IV senior reactor analyst performed a Phase 2 significance determination using the pre-solved worksheet from the "Risk Informed Inspection Notebook for the Palo Verde Nuclear Generating Station," Revision 2.01a.

The analyst assumed an exposure period of one year.

The finding was potentially Yellow, which warranted further review.

The senior reactor analyst subsequently performed a bounding Phase 3 significance determination, which is provided below.

Internal Events:

The analyst used the Palo Verde SPAR Model, Saphire 8, Revision 8.15, dated August 21, 2010 and assumed a truncation limit of 1.0E-13.

Only Unit-1 was modeled, but it could be used for any one of the three units.

Unit 1 terminology is used.

The fuel oil transfer pump failed on September 15, 2010.

The last time that the pump operated satisfactorily was August 18, 2010.

Since the exact time of inoperability was not known, the analyst assumed a T/2 mission time of 14 days.

To refine the estimate further, the analyst noted that the fuel oil day tank could have provided at least 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> of fuel.

Therefore, with the fuel oil transfer pump failed, the emergency diesel generator could have operated for at least 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />.

NUREG/CR 6890, Volume 1, "Analysis of Loss of Offsite Power Events: 1986-2004" specified a loss of offsite power non-recovery probability of 2.15E-1.

That means, following a loss of offsite power initiating event, there is a

79.5% chance that offsite power will be recovered before the day tank expires.

The analyst adjusted the loss of offsite power frequency to be consistent with these assumptions.

The base loss of offsite power frequency for Palo Verde was 3.59 E-2/year.

The 3.0-hour non-recovery of offsite power was 2.15E-1.

Therefore, the frequency for a loss of offsite power event that cannot be recovered in 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> was 3.59E-2 * 2.15E-1 = 7.7E-3/year.

This was the new loss of offsite power initiating frequency.

Resetting event time t=0 to 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> following the event required that the recovery factors for offsite power be adjusted.

Because the new loss of offsite power frequency included the assumption that offsite power was not recovered in the first 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />, the analyst calculated new nonrecovery values for offsite power given that the event of interest started at 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />.

The analyst used the following standard statistical equation for the calculation:

P(BIA) = P(A*B)/P(A), where:

P(BIA) = probability that offsite power will not be recovered given that it was not recovered at t=3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />.

- 30 - Attachment P(A) = nonrecovery probability of offsite power at t=3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />.

Equal to 2.149E-1.

P(B) = SPAR nonrecovery of offsite power at time = x

Since B is a subset of A and the two probabilities are dependent, P(A*B) = P(B).

The equation then reduced to:

P(BIA) = P(B)/(PA)

For example, for the 2-hour SPAR sequences, the offsite power nonrecovery probability at 5.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />, given that offsite power was not recovered at 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> is:

P(BlA) = 1.265E-1/2.149E-1 = 5.866E-1

The following table represents adjustments the analyst made to the SPAR model to accommodate these changes:

Offsite Power Recovery Period

Normal Offsite Power Non-Recovery Value Adjusted Offsite Power Non- Recovery Value (i.e.

+3 hours) Notes: 1.0 Hour 5.303E-1 7.278E-1

2.0Hours

3.181E-1 5.886E-1 Example cited above 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> 2.149E-1 4.484E-1

4.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> 1.566E-1 3.806E-1

6.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> 9.637E-2 2.741E-1

8.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> 6.718E-2 2.173E-1

10.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> 5.070E-2 1.801E-1

15.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> 3.079E-2 1.233E-1

24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> 1.79E-2 8.33E-2

The analyst performed two calculations, a "base case," and a "current case."

The base case assumed that the train A emergency diesel generator did not fail because of the performance deficiency.

The current case assumed that diesel did fail (at 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />).

Nominal recoveries were allowed.

The analyst set the train A emergency diesel generator "failure to run" basic event to True, which meant that a common cause failure of the train B emergency diesel generator was possible.

The analyst then solved only the loss of offsite power sequences for the base case and current case.

For a one year exposure period, the base case conditional core damage probability was 9.2E-7

and the current case (with the failure) probability was 1.2E-5.

The incremental conditional core damage probability was 1.1E-5.

For a 14 day exposure period, the delta-CDF for internal event initiators was:

Delta-CDFinternal = 1.1E-5 * 14/365 = 4.2E-7/year

- 31 - Attachment

External Events:

External events of interest included seismic induced loss of offsite power events and fire events that could result in a loss of offsite power.

Seismic:

The Analyst used the NRC's "Risk Assessment of Operational Events Handbook," Revision 1.01 to estimate the delta-CDF contribution from seismic events.

Appendix 1, Table 1, "Frequencies of Seismically-Induced Loss of Offsite Power Events," stated that the seismic induced loss of offsite power frequency for Palo Verde was 5.37E-5.

Seismic initiated losses of offsite power are considered non-recoverable.

The analyst use the noted SPAR model and calculated the conditional core damage probability (CCDP) for a non-recoverable loss of offsite power initiating event with a failed emergency diesel generator.

The analyst allowed the nominal emergency diesel generator recoveries to occur.

The day tank provided 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> of run time before the emergency diesel would have failed.

Operators would have had approximately 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> to troubleshoot the condition before failure occurred.

The CCDP was 1.4E-3.

The delta-CDF for a 14 day exposure period was:

Delta-CDFseismic = (3.5E-5*1.4E-3)*14/365 = 1.9E-9

Fires:

The fires of interest involved those that could have caused a loss of offsite power to Unit 2.

The analyst used the "Palo Verde Nuclear Generating Station, Units 1, 2, and 3 Individual Plant Examination of External Events (IPEEE) for Severe Accident Vulnerabilities," dated June 30, 1995 to evaluate the risk increase from fire initiators.

The IPEEE identified three fire compartments, where a fire in one of those compartments could cause a loss of offsite power.

Those compartments included:

5A - Train A ESF switchgear room 5B - Train B ESF switchgear room

- Control room

The IPEEE concluded that a loss of offsite power was not a significant risk in the control room (because of cable routing and limited exposure to affected circuits), unless the fire was substantial and warranted a control room evacuation.

In these cases, operators would rely on train B components to affect a safe shutdown.

Since the performance deficiency affected train A, it did not result in a quantitative change to the core damage frequency for control room fires.

Fires in fire compartment 5A were assumed to render the train A components inoperable.

Therefore, a fire in this area did not result in a quantitative change to the core damage frequency.

Fires in fire compartment 5B had the potential to cause a loss of offsite power coupled with the loss of train B components.

However, the analyst noted that the affected offsite power feeds had isolation breakers, outside of the fire area that should open to isolate offsite power from the affected bus if a short did occur.

If offsite power was lost to train

A (because a breaker did not open), operators could easily isolate the fault and restore

- 32 - Attachment offsite power to the unit within the 3.0 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> dictated by the day tank capacity.

The analyst qualitatively screened this fire area from further consideration.

The analyst noted that the actual risk associated with fires was not zero but, due to limitations in the IPEEE process, no quantitative result could be practicably obtained.

Nonetheless, the analyst qualitatively determined that the increase to the core damage frequency was very small.

Internal and External Events:

The delta-CDF for the performance deficiency was:

Delta-CDF = Delta-CDFinternal + Delta-CDFseismic + Delta-CDFfires = 4.2E-7 + 1.9E-9 + 0 = 4.2E-7/year

The licensee performed a similar calculation, at the request of the analyst, and calculated the Delta-CDF to be approximately 1.2E-7.

Large Early Release Frequency (LERF):

Using

IMC 0609 Appendix H, the SRA determined that this was a Type A finding for a large dry containment.

For pressurized water reactor plants with large dry containments, only findings related to accident categories intersystem loss of coolant accidents or steam generator tube ruptures have the potential to impact LERF.

Since this finding is not related to these two types of accidents, the analysts concluded that LERF was not a significant contributor to the risk associated with this finding.

Conclusion:

Since the Delta-CDF was less than 1E-6 and the Delta- LERF was not a significant contributor to risk, this finding was of very low safety significance, Green.

The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump.

Since most of this same equipment remained available, the components helped to mitigate the significance of the finding.