IR 05000528/2010008: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
Line 1: Line 1:
{{Adams
{{Adams
| number = ML110280467
| number = ML110670169
| issue date = 01/28/2011
| issue date = 02/28/2011
| title = IR 05000528-10-008, 05000529-10-008, 05000530-10-008, on 11/29/2010 - 12/17/2010, Palo Verde Nuclear Generating Station Units 1, 2, and 3 Biennial Baseline Inspection of the Identification and Resolution of Problems
| title = Palo Verde Nuclear Generating Station Units 1, 2 and 3 - NRC Problem Identification and Resolution Inspection Report 05000528-10-008, 05000529-10-008, 05000530-10-008, and Notice of Violation, Dated January 28, 2011
| author name = Hay M C
| author name = Mims D C
| author affiliation = NRC/RGN-IV/DRS
| author affiliation = Arizona Public Service Co
| addressee name = Edington R K
| addressee name =  
| addressee affiliation = Arizona Public Service Co
| addressee affiliation = NRC/Document Control Desk, NRC/NRR
| docket = 05000528, 05000529, 05000530
| docket = 05000528, 05000529, 05000530
| license number = NPF-041, NPF-051, NPF-074
| license number =  
| contact person =  
| contact person =  
| case reference number = EA-11-006
| case reference number = 102-06326-DCM/DFH, EA-11-006
| document report number = IR-10-008
| document report number = IR-10-008
| document type = Letter, Inspection Report, Notice of Violation
| document type = Letter, Licensee Response to Notice of Violation
| page count = 37
| page count = 6
}}
}}


Line 19: Line 19:


=Text=
=Text=
{{#Wiki_filter:
{{#Wiki_filter:10 CFR 2.201 EA-11-006 A':AO A subsidiary of Pinnacle West Capital Corporation Dwight C. Mims Mail Station 7605 Palo Verde Nuclear Senior Vice President Tel. 623-393-5403 P.O. Box 52034 Generating Station Regulatory Affairs and Oversight Fax 623-393-6077 Phoenix, Arizona 85072-2034 102-06326-DCM/DFH February 28, 2011 ATTN: Document Control Desk U.S. Nuclear Regulatory Commission Washington, DC 20555-0001 Reference:
[[Issue date::January 28, 2011]]
Palo Verde Nuclear Generating Station Units 1, 2 and 3 NRC Problem Identification and Resolution Inspection Report 05000528/2010008, 05000529/2010008, 05000530/2010008, and Notice of Violation, dated January 28, 2011


EA-11-006  
==Dear Sirs:==
Subject: Palo Verde Nuclear Generating Station (PVNGS)Units 1, 2 and 3 Docket Nos. STN 50-528, 50-529, and 50-530 Reply to Notice of Violation EA-11-006 In the above referenced letter to Arizona Public Service (APS), the NRC identified that APS failed to correct a water intrusion path to the Unit 2 motor termination boxes for the emergency diesel generator fuel oil transfer pumps and implement adequate corrective actions to preclude repetition.


Randall K. Edington, Executive Vice President, Nuclear/CNO Mail Station 7602 Arizona Public Service Company P.O. Box 52034 Phoenix, AZ 85072-2034
Pursuant to the requirements of 10 CFR 2.201 and Notice of Violation (NOV) EA-1 1-006, attached to the above referenced letter, APS hereby submits its reply to the NOV.Enclosure 1 to this letter contains a restatement of the violation.


SUBJECT: PALO VERDE NUCLEAR GENERATING STATION UNITS 1, 2 AND 3 - NRC PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT 05000528/2010008, 05000529/2010008, 05000530/2010008, AND NOTICE OF VIOLATION
Enclosure 2 contains the APS reply to the NOV.No commitments are being made to the NRC by this letter.A member of the STDARS (Strategic Teaming and Resource Sharing) Alliance ZZ7rJII-.Callaway * Comanche Peak * Diablo Canyon -Palo Verde -San Onofre -South Texas -Wolf Creek i ATTN: Document Control Desk U.S. Nuclear Regulatory Commission Reply to Notice of Violation EA-1 1-006 Page 2 If you have any questions, please contact Marianne Webb, Regulatory Affairs, Compliance Section Leader, at (623) 393-5730.


==Dear Mr. Edington:==
Sincerely,DCM/MNW/DFH/gat
On December 17, 2010, the U. S. Nuclear Regulatory Commission (NRC) completed a team inspection at Palo Verde Nuclear Generating Station, Units 1, 2, and 3. The enclosed report documents the inspection findings which were discussed on December 17, 2010, with Mr. R. Bement and other members of your staff.
 
The inspection examined activities conducted under your license as they relate to identification and resolution of problems, safety and compliance with the Commission's rules and regulations and with the conditions of your operating license. The team reviewed selected procedures and records, observed activities, and interviewed personnel. The team also interviewed a representative sample of personnel regarding the condition of your safety-conscious work environment.
 
Based on the results of this inspection, the NRC has identified an issue that was evaluated under the risk significance determination process as having very low safety significance (Green). The NRC has also determined that a violation is associated with this issue.
 
This violation was evaluated in accordance with the NRC Enforcement Policy. The current Enforcement Policy is included on the NRC's Web site at (http://www.nrc.gov/about-nrc/regulatory/enforcement/enforce-pol.html).] Arizona Public Service Company - 2 -
The violation is cited in the enclosed Notice of Violation (Notice) and the circumstances surrounding it are described in detail in the subject inspection report. The violation involved failure to correct a significant condition adverse to quality associated with the Unit 2 emergency diesel generator fuel oil transfer pumps. The violation is being cited in the Notice because Palo Verde failed to restore compliance within a reasonable time after the previous violation was identified (NCV 05000529/2009-004-002), as specified in Section 2.3.2 of the Enforcement Policy. You are required to respond to this letter and should follow the instructions specified in the enclosed Notice when preparing your response. If you have additional information that you believe the NRC should consider, you may provide it in your response to the Notice. The NRC review of your response to the Notice will also determine whether further enforcement action is necessary to ensure compliance with regulatory requirements.
 
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure(s), and your response, will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html. To the extent possible, your response should not include any personal privacy or proprietary information so that it can be made available to the Public without redaction.
 
Sincerely,/RA/ Michael C Hay, Chief Technical Support Branch Division of Reactor Safety Dockets: 50-528 50-529 50-530 Licenses: NPF-41 NPF-51  NPF-74


===Enclosures:===
===Enclosures:===
Notice of Violation and Inspection Report 05000528/2010008, 05000529/2010008, and 05000530/2010008
1. Restatement of Notice of Violation EA-1 1-006 2. Reply to Notice of Violation EA-1 1-006 cc: E. E. Collins Jr.J. R. Hall L. K. Gibson M. A. Brown NRC Region IV Regional Administrator NRC NRR Senior Project Manager -(electronic
 
/ paper)NRC NRR Project Manager (electronic
===w/Attachments:===
/ paper)NRC Senior Resident Inspector for PVNGS Enclosure 1 Restatement of Notice of Violation EA-1 1-006 During an NRC inspection conducted on November 29, 2010 through December 17, 2010, a violation of NRC requirements was identified.
1. Supplemental Information Initial Information Request 2. Palo Verde Fuel Oil Transfer Pump Failure Risk Assessment
 
Arizona Public Service Company - 3 -
Mr. Steve Olea Arizona Corporation Commission 1200 W. Washington Street Phoenix, AZ 85007 Mr. Douglas K. Porter, Esq Southern California Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Chairman Maricopa County Board of Supervisors 301 W. Jefferson, 10th Floor Phoenix, AZ 85003
 
Mr. Aubrey V. Godwin, Director Arizona Radiation Regulatory Agency 4814 South 40 Street Phoenix, AZ 85040
 
Mr. Dwight C. Mims Vice President Regulatory Affairs and Plant Improvement Palo Verde Nuclear Generating Station Mail Station 7605 P.O. Box 52034 Phoenix, AZ 85072-2034 Mr. Ron Barnes, Director Regulatory Affairs Palo Verde Nuclear Generating Station MS 7638 P.O. Box 52034 Phoenix, AZ 85072-2034
 
Mr. Jeffrey T. Weikert Assistant General Counsel El Paso Electric Company Mail Location 167 123 W. Mills El Paso, TX 79901
 
Michael S. Green Senior Regulatory Counsel Pinnacle West Capital Corporation P.O. Box 52034, MS 8695 Phoenix, AZ 85072-2034
 
Arizona Public Service Company - 4 -
Mr. Eric Tharp Los Angeles Department of Water & Power Southern California Public Power Authority P.O. Box 51111, Room 1255-C Los Angeles, CA 90051-0100 Mr. James Ray Public Service Company of New Mexico 2401 Aztec NE, MS Z110 Albuquerque, NM 87107-4224 Mr. Geoffrey M. Cook Southern California Edison Company 5000 Pacific Coast Hwy. Bldg. D21 San Clemente, CA 92672
 
Mr. Robert Henry Salt River Project 6504 East Thomas Road Scottsdale, AZ 85251
 
Mr. Brian Almon Public Utility Commission William B. Travis Building P.O. Box 13326 Austin, TX 78701-3326
 
Environmental Program Manager City of Phoenix Office of Environmental Programs 200 West Washington Street Phoenix, AZ 85003 Mr. John C. Taylor Director, Nuclear Generation El Paso Electric Company 340 East Palm Lane, Suite 310 Phoenix, AZ 85004 Chief, Technological Hazards Branch FEMA Region IX 1111 Broadway, Suite 1200 Oakland, CA 94607-4052 Jake Lefman Southern California Edison Company 5000 Pacific Coast Hwy, Bldg. D21 San Clemente, CA 92672 Arizona Public Service Company - 5 -
Electronic distribution by RIV: Regional Administrator (Elmo.Collins@nrc.gov ) Deputy Regional Administrator (Art.Howell@nrc.gov ) DRP Director (Kriss.Kennedy@nrc.gov ) DRP Deputy Director (Troy.Pruett@nrc.gov ) DRS Director (Anton.Vegel@nrc.gov ) DRS Deputy Director (Vacant) Senior Resident Inspector (Tony.Brown@nrc.gov ) Resident Inspector (Joseph.Bashore@nrc.gov ) Resident Inspector (Mica.Baquera@nrc.gov ) Branch Chief, DRP/D (Ryan.Lantz@nrc.gov ) PV Administrative Assistant (Regina.McFadden@nrc.gov ) Acting Senior Project Engineer, DRP/D (Gerond.George@nrc.gov ) Project Engineer, DRP/D (Peter.Jayroe@nrc.gov ) Public Affairs Officer (Victor.Dricks@nrc.gov) Public Affairs Officer (Lara.Uselding@nrc.gov ) Project Manager (Randy.Hall@nrc.gov ) Branch Chief, DRS/TSB (Michael.Hay@nrc.gov ) RITS Coordinator (Marisa.Herrera@nrc.gov ) Regional Counsel (Karla.Fuller@nrc.gov ) Senior Enforcement Officer (Ray.Keller@nrc.gov ) Congressional Affairs Officer (Jenny.Weil@nrc.gov ) OEMail Resource OEDO RIV Coordinator (James.Trapp@nrc.gov ) DRS/TSB STA (Dale.Powers@nrc.gov )
 
R:_REACTORS\PV 2010008 - PAJ ML SUNSI Rev Compl. Yes No ADAMS Yes No Reviewer Initials PAJ Publicly Avail Yes No Sensitive Yes No Sens. Type Initials PAJ RIV: PE: DRP/D RI/DRP/D RIV:DRS/EB2 RIV:DRS/PSB2 RIV: DRS/TSB PAJayroe MTBaquera EDUribe TDBuchanan MVasquez
/RA/ /RA/ /RA/ /RA/ /RA/ 1/25/11 1/25/11 1/25/11 1/25/11 1/26/11 RIV C:DRP/D C:DRS/TSB RLKellar RELantz MCHay
/RA /RA /RA/ 1/27/11 1/27/11 1/28/11 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax Enclosure NOTICE OF VIOLATION
 
Arizona Public Service Company Docket Nos: 50-528,-529,-530 Palo Verde Nuclear Generating Station License Nos: NPF-41, -51, -74 EA-11-006 During an NRC inspection conducted on November 29, 2010 through December 17, 2010, a violation of NRC requirements was identified. In accordance with NRC enforcement policy, the violation is listed below:
10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action" requires, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition.
 
Contrary to the above, from April 2009 through September 2010, the licensee failed to correct a significant condition adverse to quality and implement adequate corrective actions to preclude repetition. Specifically, the licensee failed to correct a water intrusion path to the Unit 2 motor termination boxes for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections and a pump trip.
 
This Notice of Violation is associated with a Green Significance Determination Process finding.
 
Pursuant to the provisions of 10 CFR 2.201, Arizona Public Service Company is hereby required to submit a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001 with a copy to the Regional Administrator, Region IV, and a copy to the NRC Resident Inspector at the facility that is the subject of this Notice, within 30 days of the date of the letter transmitting this Notice of Violation (Notice). This reply should be clearly marked as a "Reply to Notice of Violation EA-11-006," and should include: (1) the reason for the violation, or, if contested, the basis for disputing the violation or severity level; (2) the corrective steps that have been taken and the results achieved; (3) the corrective steps that will be taken to avoid further violations; and (4) the date when full compliance will be achieved. Your response may reference or include previous docketed correspondence, if the correspondence adequately addresses the required response. If an adequate reply is not received within the time specified in this Notice, an order or a Demand for Information may be issued as to why the license should not be modified, suspended, or revoked, or why such other action as may be proper should not be taken. Where good cause is shown, consideration will be given to extending the response time.
 
If you contest this enforcement action, you should also provide a copy of your response, with the basis for your denial, to the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001.
 
Because your response will be made available electronically for public inspection in the NRC Public Document Room or from the NRC's document system (ADAMS), accessible from the Enclosure NRC website at www.nrc.gov/reading-rm/pdr.html or www.nrc.gov/reading-rm/adams.html, to the extent possible, it should not include any personal privacy, proprietary, or safeguards information so that it can be made available to the public without redaction. If personal privacy or proprietary information is necessary to provide an acceptable response, then please provide a bracketed copy of your response that identifies the information that should be protected and a redacted copy of your response that deletes such information. If you request withholding of such material, you must specifically identify the portions of your response that you seek to have withheld and provide in detail the basis for your claim of withholding (e.g., explain why the disclosure of information will create an unwarranted invasion of personal privacy or provide the information required by 10 CFR 2.390(b) to support a request for withholding confidential commercial or financial information). If safeguards information is necessary to provide an acceptable response, please provide the level of protection described in 10 CFR 73.21.
 
Dated this 28 th day of January 2011. Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION IV Docket: 50-528, 50-529, 50-530 License: NPF-41, NPF-51, NPF-74 Report: 05000528/2010008; 05000529/2010008; 05000530/2010008 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station Units 1, 2, and 3 Location: 5951 S. Wintersburg Road Tonopah, Arizona Dates: November 29, 2010 through December 17, 2010
 
Inspectors:
P. Jayroe, Project Engineer (Team Leader) M. Vasquez, Senior Reactor Inspector M. Baquera, Resident Inspector E. Uribe, Reactor Inspector T. Buchanan, Reactor Inspector
 
Approved By: Michael C. Hay, Chief Technical Support Branch Division of Reactor Safety Enclosure
 
=SUMMARY OF FINDINGS=
IR 05000528; 05000529; 05000530/2010008; 11/29/2010 - 12/17/2010; Palo Verde Nuclear Generating Station Units 1, 2, and 3 Biennial Baseline Inspection of the Identification and Resolution of Problems.
 
The team inspection was performed by a senior reactor inspector, a project engineer, two reactor inspectors, and a resident inspector. One green finding of very low safety significance was identified during this inspection. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609, "Significance Determination Process". Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649, "Reactor Oversight Process," Revision 4, dated December 2006.
 
Identification and Resolution of Problems The team concluded that the corrective action program at Palo Verde Nuclear Generating Station was generally effective. The team concluded that site personnel identify problems at a low threshold and enter them into the corrective action program. The licensee utilizes a rigorous screening process to characterize issues and that the vast majority of issues are appropriately evaluated and adequate corrective actions are taken. The team did identify isolated cases where problem evaluation could have been more effective at addressing the underlying causes of issues as well as a number of examples where corrective actions were not timely or adequate to address identified problems. The team also determined that though the overall process for identifying and correcting issues was well established, certain incidents of procedural violations associated with corrective action program processes led to delays and less than adequate actions to correct material deficiencies. Though the team identified areas in which the licensee could improve its corrective action program, the overall process was determined to be effective in identifying and correcting conditions adverse to quality.
 
The licensee appropriately evaluated industry operating experience for relevance to the facility, entered applicable items in the corrective action program, and subsequently utilized operating experience in root and apparent cause evaluations. The team did determine that that the licensee could improve its utilization of operating experience to prevent the occurrence of similar events at Palo Verde. The team determined that the licensee performed effective quality assurance audits and self assessments.
 
The team performed seven safety culture focus group discussions involving approximately 70 licensee personnel in order to assess the safety conscious work environment of the site. The team felt that a strong safety conscious work environment existed in most of the work groups interviewed; however, one work group interviewed exhibited weaknesses in this area. Specifically, the team found that although there were many individuals who felt comfortable raising safety concerns without fear of retaliation, there were some individuals in the operations department who expressed the perception that they would or might be retaliated against for raising certain safety concerns using certain avenues available to them. In all instances, these individuals stated they would use one avenue or another to raise their concerns.
 
===A. NRC-Identified and Self-Revealing Findings===
 
===Cornerstone: Mitigating Systems===
: '''Green.'''
Inspectors identified a Green cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action", for the failure of the licensee to correct a significant condition adverse to quality associated with the emergency diesel generator fuel oil transfer pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections. As an interim corrective action, splices have been placed in the cabling to prevent water from reaching the motor terminations. Due to the licensee's failure to restore compliance to a previous violation  (NCV 05000529/2009004-02) within a reasonable time, this violation is being cited as a Notice of Violation consistent with the NRC Enforcement Policy. This has been entered into the licensee's corrective action program as Condition Report Disposition Request 3529151.
 
The performance deficiency associated with this finding was the failure of the licensee to correct a significant condition adverse to quality and prevent recurrence. The finding is more than minor because it affected the equipment performance attribute of the Mitigating Systems Cornerstone and affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.
 
Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was determined to require a Phase 2 and Phase 3 analysis by a senior reactor analyst because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. The senior reactor analyst performed a bounding Phase 3 significance determination and found the finding to be of very low safety significance (Green). The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump. The finding had a cross-cutting aspect in the area of Problem Identification and Resolution associated with the corrective action program component because the licensee failed to thoroughly evaluate problems such that the resolutions address causes and extent of condition, as necessary. [P.1.(c)] (Section 4OA2)
 
===B. Licensee-Identified Violations===
 
None.
 
=REPORT DETAILS=
 
==OTHER ACTIVITIES (OA)==
{{a|4OA2}}
==4OA2 Problem Identification and Resolution==
{{IP sample|IP=IP 71152}}
The team based the following conclusions on the sample of corrective action documents that were initiated in the assessment period, which ranged from March 1, 2009, to the end of the on-site portion of the inspection on December 17, 2010.
 
a. Assessment of the Corrective Action Program Effectiveness (1) Inspection Scope  The team reviewed approximately 250 Condition Report Disposition Requests, Action Requests, and Corrective Action Items, including associated root cause, apparent cause, and direct cause evaluations, to determine if problems were being properly identified, characterized, and entered into the corrective action program for evaluation and resolution. The team reviewed a sample of operability determinations, self- assessments, and various other documents related to the corrective action program. The team evaluated the licensee's efforts in establishing the scope of problems by reviewing selected logs, work requests, self-assessments results, audits, system health reports, action plans, and results from surveillance tests and preventive maintenance tasks. The team reviewed work requests and attended the licensee's daily action request review committee and the management review committee meetings to assess the reporting threshold, prioritization efforts, and significance determination process, as well as observing the interfaces with the operability assessment and work control processes when applicable. The team's review included verifying the licensee considered the full extent of cause and extent of condition for problems, as well as how the licensee assessed generic implications and previous occurrences. The team assessed the timeliness and effectiveness of corrective actions, completed or planned, and looked for additional examples of similar problems. The team conducted interviews with plant personnel to identify other processes that may exist where problems may be identified and addressed outside the corrective action program.
 
The team also reviewed corrective action documents that addressed past NRC-identified violations to ensure that the corrective action addressed the issues as described in the inspection reports. The inspectors reviewed a sample of corrective actions closed to other corrective action documents to ensure that corrective actions were still appropriate and timely.
 
The team considered risk insights from both the NRC's and Palo Verde Nuclear Generating Station's risk assessments to focus the sample selection and plant tours on risk significant systems and components. The team selected the following risk significant systems or components for review: emergency diesel generators, fuel oil transfer pumps, maintenance rule A(1) systems, spray ponds, and fire protection features. The team also expanded their review to include evaluations involving the aging of electrical systems to determine whether problems were being effectively addressed. The team conducted walkdowns of systems to assess whether problems were identified and entered into the corrective action program.
 
  (2) Assessments (a) Assessment - Effectiveness of Problem Identification The team determined that the licensee was identifying problems at a very low threshold as demonstrated by approximately 25,000 Palo Verde Action Requests (PVARS)initiated per year. The team found the Palo Verde staff to be conscientious about documenting conditions adverse to quality in the corrective action program. The component design basis review initiative performed by the licensee was determined to be an effective method for identifying design issues and entering them into the corrective action program.
 
  (b) Assessment - Effectiveness of Prioritization and Evaluation of Issues The team determined that the licensee's prioritization and evaluation of issues was effective however certain areas for improvement were identified. The team reviewed approximately 50 issues associated with operability reviews to assess the quality, timeliness, and prioritization of operability assessments. The team noted that the immediate and prompt operability assessments reviewed were completed in a timely manner. The team noted several examples of weaknesses in issue prioritization and evaluation as identified below:
* The team found that the failure to correct the significant condition adverse to quality associated with water intrusion into the Unit 2 fuel oil transfer pump motor termination boxes was in part due to an inadequate evaluation which did not produce actions that would prevent future water intrusion into the conduit and motor termination box. As a result of the inadequate evaluation performed after the 2009 water intrusion and pump failure event, corrective actions were not adequate to prevent further water intrusion and another pump failure occurred in 2010.
* While reviewing corrective actions from a past NRC NCV, the team found that Apparent Cause Evaluation (ACE) 3444581 for Unit 2 was closed to ACE 3357761 and ACE 3425538, both for Unit 3, on 04/09/2010. Although similar, this was not a duplicate condition. The team also found that ACE 3521811 for Unit 2 was closed to Unit 3's ACE 3425538 on 09/24/2010, which was also similar, but was not a duplicate condition. The licensee determined that procedural guidance for closure of corrective actions caused confusion, which led to the inadvertent closures of the described Apparent Cause Evaluations. Procedural guidance allows for closure of cause evaluations to other cause evaluations if the condition is a duplicate condition. The licensee had previously addressed the underlying technical issue as the result of a separate ACE, so the equipment impact was insignificant. The licensee entered this issue into the corrective action program as Palo Verde Action Request 3565255.
* It was noted that there was virtually no guidance in procedure 01DP-0AP12, "Palo Verde Action Request Processing," Appendix E, "Condition Classification Instruction" concerning the classification and prioritization of PVARS related to the emergency plan. The existing guidance contains a reference to screening conditions that affect licensing basis documents such as the emergency plan as significant, however inspectors identified one example (CRDR 3296869) where incorrectly made protective action recommendations during a training evolution were screened as adverse, despite the fact that they are a critical attribute in NRC evaluations of the licensee emergency plan response.
* Corrective action documents associated with NRC findings were not clearly marked as such, and NRC findings were not always subject to apparent cause evaluations. While Palo Verde procedures currently allow this practice it is noteworthy that the licensee has received two notices of violation during the assessment period associated with a failure to restore compliance from a previous NRC identified noncited violation.
* The team identified one example of an inappropriately cancelled prompt operability (POD) request where the Senior Reactor operator (SRO) and engineering decided to cancel a POD request even though the procedure required the underlying issue to be resolved in order to cancel a POD request.
 
The inspection team observed multiple screening processes associated with the licensee's corrective action program including the action request review committee, the condition review group, and the corrective action review board. During these processes the team observed rigorous technical discussions and challenges to conclusions, and felt that these were effective efforts to improve issue screening and prioritization (c) Assessment - Effectiveness of Corrective Actions Although in the majority of cases corrective actions appeared to be effective, the inspection team identified a number of examples where corrective actions were ineffective or absent and determined that this area is the licensee's biggest challenge in maintaining an effective corrective action program. Examples of inadequate corrective actions included:
* A corrective action associated with the 2009 fuel oil transfer pump failure consisted of actions to perform meggering of the wiring insulation as well as a visual inspection of the motor termination box. This action was closed to work orders that did not always complete the visual inspections as required by the corrective actions. This was a contributing factor in the 2010 demand failure because it could have detected the water intrusion before it caused electrical problems during pump operations. This was a violation of corrective action program procedures which resulted in a failure to detect water in the motor termination box. It is possible that if these inspections had been performed as directed the 2010 fuel oil transfer pump failure might not have occurred.
* A review of operations standing orders revealed a heat trace on safety injection piping that had been broken since the spring of 2007. The purpose of the heat trace was to prevent the piping from the Reactor Makeup Water Tank (RWT) to the safety injection pumps from freezing. The broken heat trace had been entered into the licensee's corrective action program and a work order to repair it had been generated, however the corrective maintenance was not performed in a timely manner. The work was deferred from online to outage then back to online. The standing order required operators to check the temperature of the line when outside temperatures were below freezing. Though the overall impact to safety was minor due to operating experience that showed that the affected piping would stay warm when outside temperatures were freezing, the impact to operators was not considered, and the condition adverse to quality was left unrepaired for over 3 years.
* Inspectors noted further examples of untimely corrective actions associated with the spray pond chemistry addition system. These adverse conditions had little to no impact on nuclear safety however they did present a burden to operators as well as an industrial safety hazard. In one instance, a PVAR was generated in June, 2010 to clean the acid residue on the spray pond acid skid. A corrective action was generated to create a routine maintenance action to clean the skid, and this action was closed to a work order to perform the same. As of December, 2010, when the NRC inspection team arrived onsite the original acid residue had not yet been cleaned up.
 
Inspectors interviewed the management team responsible for implementing the licensee's procedure improvement process, and recognized the licensee's investment of time and resources as a positive initiative to improve in this area. The team was impressed with the overall scope of the project as well as its flexibility to allow immediate procedural changes and fixes as well as planned upgrades to the site's library of procedures. It was noted in at least one focus group discussion that procedures are improving but there is still much to accomplish in this area.
 
Overall, inspectors acknowledge the efforts to improve in various areas, but the team felt that there is a challenge in the area of corrective actions based on the number of items observed that fell short of addressing the underlying issue either due to timeliness, inaction, or inadequate action.
 
b. Assessment of the Use of Operating Experience (1)    Inspection Scope The team examined the licensee's program for reviewing industry operating experience, including reviewing the governing procedure and self-assessments.
 
A sample size of 22 operating experience condition report/disposition records that had been issued during the assessment period were reviewed to assess whether the licensee had appropriately evaluated the notification for relevance to the facility. The team also reviewed a number of Root and Apparent Cause Evaluations as well as various Licensee Event Reports that covered the assessment period to verify if the licensee had appropriately included industry-operating experience.
 
(2)    Assessment Overall, the team determined that the licensee evaluated and utilized industry operating experience, but noted that the licensee could improve in using industry operating experience to prevent similar events onsite. This was based on reviewing a sample of 35 industry operating experience documents. The team concluded that the licensee was evaluating for industry operating experience by reviewing generic industry guidance. The team also concluded that the licensee identifies weaknesses in specific operating experience evaluations when performing root and apparent cause evaluations as well as documenting licensee event reports.
 
The team noted that root and apparent cause evaluations were required in order to evaluate whether internal or external operating experience was available associated with the event or failure being examined, and whether the evaluation and actions to address those items had been effective. Additionally, all root cause evaluations reviewed included an assessment as to whether the issue being evaluated had potential application to other similar components or plants. Several exceptions were noted where recent evaluations identified relevant operating experience, which had been ineffectively addressed.
* The licensee is currently evaluating a deficiency related to the emergency diesel generator fuel oil transfer pumps; however, the site has had repetitive occurrences of pump failures. Industry operating experience exists concerning water intrusion into submerged cables, which is part of the pathway for water intrusion which resulted in pump trips.
* Significant (Sig) CRDR 3417311 identifies an example of poor use of operating experience in that spray pond missile hazards continue to challenge Palo Verde's ultimate heat sink. The site received a non-cited violation due to violation of requirements, but the site did not implement effective corrective actions. This condition report discusses the weakness of using internal operating experience, which prevented the site from restoring compliance and resulted in issuance of a Notice of Violation.
* Sig CRDR 3303334 identifies another example of poor use of operating experience in that condensate storage tank temperature monitoring failed to be controlled. The site received a non-cited violation due to design control and continued to be deficient. This condition report discusses the weakness of using internal and external operating experience, which prevented the site from restoring compliance and resulted in issuance of a Notice of Violation.
* LER 1-2009-001 for unanalyzed RWT recirculation alignment root cause analysis was documented in CRDR 3287805. The evaluation determined that industry operating experience existed and presented an opportunity to identify and mitigate the consequence of the violation.
* NCV 2009006-05 identified that the licensee failed to adequately evaluate operating experience related to inverters.
 
The examples above show signs of weakness in converting operating experience into effective preventive measures. Palo Verde appears to be identifying the potential causes of events, however, the licensee has not always implemented the actions to address and trend these as conditions adverse to quality.
 
c. Assessment of Self-Assessments and Audits (1) Inspection Scope The team reviewed a sample size of 32 licensee self-assessments and audits to assess whether the licensee was regularly identifying performance trends and effectively addressing them. The team reviewed audit reports to assess the effectiveness of assessments in specific areas. The team evaluated the use of self- and third-party assessments, the role of the quality assurance department, and the role of the performance improvement group related to licensee performance. The specific self-assessment documents reviewed are listed in the Attachment.
 
        (2) Assessment The team concluded that the licensee had an effective self-assessment process. The team observed that Palo Verde Nuclear Generating Station's management was involved in developing the topics and objectives of self-assessments. The team observed that the assignment of the assessment team included members with the proper skills and experience to ensure an effective self-assessment was conducted, and the team members included individuals from outside organizations. Inspectors observed that the licensee was very effective in identifying issues. The team observed that certain licensee organizations performed several self-assessments within a short period of time, going above and beyond established requirements. For example, the operations department has performed 10 benchmarking trips and self-assessments in the past two years. The team also noted that a self-assessment had caught Part 21 screening issues and improved the licensee's warehouse discrepancy notice procedure to prevent future missed screenings of component discrepancies.
 
d. Assessment of Safety Conscious Work Environment (1) Inspection Scope During the week of November 29, the inspection team reviewed the last safety culture assessment for the site and conducted seven safety culture focus groups involving approximately 70 individuals. The interviewees represented four functional organizations and ranged across permanent employees and contractors. No supervisors were allowed to attend the focus group discussions. The team conducted these interviews to assess whether conditions existed that would challenge the establishment of a safety conscious work environment at Palo Verde Nuclear Generating Station.
 
    (2) Assessment The team found that three of the four work groups represented had a strong safety conscious work environment (SCWE) in that workers felt comfortable raising safety concerns without fear of retaliation. However, the team identified weaknesses in the safety conscious work environment in Operations. Specifically, the team found that although there were many individuals in Operations who felt comfortable raising safety concerns without fear of retaliation, there were also some individuals who expressed the perception (to varying degrees) that they would or might be retaliated against for raising certain safety concerns. This was not isolated to a single focus group; rather, the team observed elements of a SCWE weakness in each of the three focus groups conducted with licensed and non-licensed operators. Importantly, the team found that the individuals who felt they could be retaliated against stated they would still raise their issues using one of the avenues available to them (e.g., raising them to certain shift managers or to the NRC).
 
Operators in all three focus groups raised concerns about equipment reliability and change management. Concerns about equipment reliability and change management were also expressed in other focus groups, but to a lesser extent. The operators expressed that unless equipment problems involve a technical specification or plant operation, it can take a long time for the problem to get resolved. As a result of continually raising some equipment issues, several of the more experienced operators do not believe their opinions are valued.
 
Some operators stated that they would not raise safety issues up to certain supervisors or managers if they suspected the supervisor or manager disagreed with them. In addition, operators stated they might not use certain avenues available to them for fear of retaliation. For example, some individuals stated that they would not raise certain safety issues with certain supervisors and managers, and some stated they viewed it as another arm of management and not as an independent entity who would objectively review their concerns. The majority of the operators stated they were comfortable raising issues to the NRC however there were some that were concerned that they might be retaliated against if plant management found out. As a result, the team concluded that some operators were not comfortable using all avenues available to them for raising safety concerns. Nevertheless, all operators stated they would raise safety concerns using one avenue or another.
 
In conclusion, the team found that most of the work groups interviewed had a strong safety conscious work environment in that workers felt comfortable raising safety concerns without fear of retaliation. However, the team identified weaknesses in the safety conscious work environment in the operations functional group. In response to the team's finding, the licensee planned to further evaluate the safety conscious work environment in operations and develop corrective actions to address the issue. The licensee planned to expand its actions to other work groups if it identified weaknesses in those areas.
 
e. Specific Issues Identified During This Inspection (1) Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps.
 
=====Introduction.=====
The inspectors identified a Green cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action", for the failure of the licensee to correct a significant condition adverse to quality associated with the emergency diesel generator fuel oil transfer pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections.
 
=====Description.=====
On September 15, 2010 operations personnel started a monthly emergency diesel generator surveillance per Procedure 40ST-9DG01, "Diesel Generator A Test."  During the surveillance, operations personnel received an alarm in the control room indicating trouble with the fuel oil transfer pump. It was discovered that the supply breaker for the fuel oil transfer pump was open, interrupting power to the pump.
 
Each emergency diesel generator has its own fuel oil transfer system which consists of one fuel oil storage tank located in a vault, one diesel fuel oil transfer pump located inside the fuel oil storage tank, and one fuel oil day tank. The fuel oil transfer pump takes suction from the fuel oil storage tank and pumps fuel oil to the fuel oil day tank. The fuel oil day tank then supplies fuel to the emergency diesel generators via gravity drain. Operations personnel declared emergency diesel generator Train A inoperable and maintenance personnel began troubleshooting. During troubleshooting activities, water was found in the motor termination box which houses electrical connections for the fuel oil transfer pump. The presence of water was determined to be the cause of a phase-to-ground fault resulting in a pump trip. The Train B emergency diesel generator was started and completed a one hour loaded surveillance run to comply with technical specifications to ensure operability of the opposite train. An extent of condition review found the Train B emergency diesel generator fuel oil transfer pump motor termination box filled with corrosion products, indicative of water intrusion into the box. Inspectors were present to observe the discovery of corrosion products in the motor termination box. Subsequent megger testing found the resistance of the cable insulation to be reduced by two orders of magnitude, from 1 Gohm at installation 1 year prior to the present reading of 5 Mohm, and below inservice limits of 50 Mohm established by Specification 13-EN-306, "Installation Specification for Cable Splicing and Terminations."  Water intrusion into the motor termination box was the same significant condition adverse to quality present and identified during a failure of the Train B emergency diesel generator fuel oil transfer pump in April 2009. This significant condition adverse to quality was not corrected prior to its recurrence.
 
On April 22, 2009, the Unit 2 emergency diesel generator Train B fuel oil transfer pump failed surveillance test Procedure 73ST-9DF01, "Diesel Fuel Oil Pump Inservice Test," due to a supply breaker opening and interrupting power to the pump. Operations personnel declared the Train B emergency diesel generator inoperable. The April 2009 event was considered a significant condition adverse to quality, and was classified as such using Procedure 01DP-AP12, "Palo Verde Action Request Processing" Appendix E. A root cause evaluation was performed under CRDR 3317532. The cause was determined to be water intrusion into the emergency diesel generator fuel oil transfer pump motor termination box which resulted in corrosion and failure of the pump during its surveillance. Subsequently, the root cause analysis identified inadequate maintenance practices as the cause of the water intrusion and did not identify the problem with standing water in the underground cable conduit, even though it had been initially identified in 2004. Maintenance practices were thought to be inadequate to identify and prevent water intrusion, as such, maintenance in response to previous water intrusion into the fuel oil vaults of Units 2 and 3 in December 2004 was thought to be the source of the water intrusion for the April 2009 event. A self-revealing noncited violation, NCV 05000529/2009004-02, was issued for failure to take corrective actions to correct a condition adverse to quality associated with water intrusion into the emergency diesel generator fuel oil transfer pump motor termination box, in response to inadequate corrective actions from the December 2004 water intrusion event.
 
Corrective actions to prevent recurrence of the significant condition adverse to quality, as determined in April 2009, were to perform visual inspections of the motor termination boxes and test insulation resistance of the motor power cable. The corrective actions credited with the completion of this task did not always perform a visual inspection as required. During the root cause for the most recent September 2010 event, the water intrusion path was identified. The water intrusion path consisted of standing water in the underground cable conduit leaking into degraded cable jacketing, creating a path to the motor termination box. The degraded cable jacketing was initially identified in 2004, and this water intrusion path was likely active during the April 2009 event. Corrective actions are to change the design of the cable and conduit and replace it with a cable qualified for water submergence. In the interim, splices have been placed in the cabling located in the junction box to prevent any water from migrating to the motor termination box.
 
=====Analysis.=====
The performance deficiency associated with this finding was the failure of the licensee to correct a significant condition adverse to quality and prevent recurrence. The finding is more than minor because it affects the equipment performance attribute of the Mitigating Systems Cornerstone and affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was determined to require a Phase 2 and Phase 3 analysis by a senior reactor analyst because the finding resulted in an actual loss of safety function of a single train for greater than its technical specification allowed outage time. A Region IV senior reactor analyst performed a Phase 2 significance determination using the pre-solved worksheet from the "Risk Informed Inspection Notebook for the Palo Verde Nuclear Generating Station," Revision 2.01a. The analyst assumed an exposure period of one year. The finding was potentially Yellow, which warranted further review. The senior reactor analyst subsequently performed a bounding Phase 3 significance determination, which determined that since the Delta-CDF was less than 1E-6 and the Delta- LERF was not a significant contributor to risk, this finding was of very low safety significance, Green. The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump. Since most of this same equipment remained available, the components helped to mitigate the significance of the finding.
 
The finding had a cross-cutting aspect in the area of Problem Identification and Resolution associated with the corrective action program component because the licensee failed to thoroughly evaluate problems such that the resolutions address causes and extent of condition, as necessary. [P.1.(c)]
 
=====Enforcement.=====
Title 10 CFR, Part 50, Appendix B, Criterion XVI, "Corrective Action," states, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. Contrary to the above, from April 2009 to September 2010, the licensee failed to assure that the cause of the significant condition adverse to quality was determined and that corrective action was taken to preclude repetition. Specifically, the licensee failed to correct a water intrusion path to the motor termination box for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections. As a corrective action, splices have been placed in the cabling to prevent water from reaching the motor terminations until a design change for the affected cabling can be implemented. A Phase 3 analysis determined the finding to be of very low safety significance and it has been entered into the licensee's corrective action program as CRDR 3529151. Due to the licensee's failure to restore compliance within a reasonable period of time after the violation was identified in NRC Inspection Report NCV 05000529/2009004-0. This violation is being cited as a Notice of Violation consistent with Section 2.3.2 of the NRC Enforcement Policy. VIO 05000529/2010008-01 "Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps"
{{a|4OA3}}
==4OA3 Follow-up of Events and Notices of Enforcement Discretion==
{{IP sample|IP=IP 71153}}
a. Event Report Reviews
 
  (1) Inspection Scope The inspectors reviewed the below listed licensee event report and related documents to assess:  (1) the accuracy of the licensee event report; (2) the appropriateness of corrective actions; (3) violations of requirements; and (4) generic issues.
 
  (2) Findings and Observations (Closed) Licensee Event Report 05000528/2010-003-00, Technical Specification Violation - Loss of Containment Building Equipment Hatch Closure Capability On May 8, 2010, during the Unit 1 refueling outage, the licensee discovered that the containment building equipment hatch was not capable of being fully closed while core alterations were in progress on May 2, 2010. Core alterations were in progress on two occasions totaling approximately four hours and forty-seven minutes. The cause was determined to be due to an inadequate post maintenance test following maintenance on both the east and west hatch hoist upper limit switches. The licensee adjusted these switches and the hatch was closed and secured. Inspectors reviewed this issue and documented a Green noncited violation of Technical Specification 5.4.1, "Procedures," in Section 1R12 of NRC Inspection Report 05000528;529;530/2010003. The licensee documented the inadequate post maintenance test in Palo Verde Action Request 3478220 and Condition Report Disposition Request 3431177. Inspectors reviewed the root cause evaluation and the licensee event report and determined that no additional violations of NRC requirements exist. This licensee event report is closed.
 
{{a|4OA5}}
==4OA5 Other Activities==
 
(Closed) Notice of Violation (VIO) 05000528,529,530/2009005-01, "Failure to Establish Adequate Procedures to Control Potential Tornado Borne Missile Hazards Near the Essential Spray Ponds."  Inspectors reviewed the Root Cause Evaluation and Corrective Actions associated with this Notice of Violation. Inspectors reviewed commitments made in the NOV response letter dated March 11, 2010 and actions taken to comply with commitments and found them to be adequate. This Notice of Violation is closed.
 
{{a|OA6}}
==OA6 Meetings==
 
===Exit Meeting Summary===
 
On December 17, 2010 the team presented the inspection results to Mr. R. Bement, Vice President of Nuclear Operations, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.
 
=SUPPLEMENTAL INFORMATION=
 
==KEY POINTS OF CONTACT==
 
===Licensee Personnel===
: [[contact::B. Berles]], Nuclear Engineering Department Leader, Component Performance Engineering
: [[contact::J. Bungard]], RP Technical Services Section Leader, Radiological Engineering
: [[contact::R. Bramlett]], Nuclear Security Programs Department Leader, Emergency Services Programs
: [[contact::C. Clark]], Control Room Supervisor, Shift Technical Advisors
: [[contact::J. Copsey]], Employee Concerns Director, Employee Concerns
: [[contact::K. Chavet]], Regulatory Affairs Senior Consultant, Regulatory Affairs
: [[contact::J. Dotson]], Field Services Technician, Emergency Preparedness
: [[contact::D. Elkington]], Regulatory Affairs Senior Consultant, Regulatory Affairs
: [[contact::A. Hartwig]], Engineering Section Leader, Instrumentation and Control Design
: [[contact::M. Heider]], Nuclear Engineering Department Leader, Procurement and Obsolescence
: [[contact::M. Hypse]], Technical Management Assistant, Design Electrical / I&C
: [[contact::T. Hook]], Engineering Section Leader, PRA
: [[contact::C. Karlson]], Engineering Section Leader, Design Electrical
: [[contact::M. Karbassian]], Nuclear Engineering Director, CDBR Group
: [[contact::P. Koss]], Chemistry Unit Section Leader, Chemistry Work Management
: [[contact::F. Lake]], Performance Improvement Team Department Leader, CAP 2A
: [[contact::J. Livorsi]], Employee Concerns Senior Consultant, Employee Concerns
: [[contact::L. Leavitt]], Performance Improvement Section Leader, CAP 2A
: [[contact::W. Liu]], Senior Engineer, Transient Analysis
: [[contact::D. Mims]], VP Regulatory Affairs and Plant Improvement, Regulatory Affairs / Plant Improvement 
: [[contact::D. Myatt]], Nuclear Maintenance Section Leader, Employee Concerns
: [[contact::H. Mckaig]], Nuclear Engineering Department Leader, Systems Engineering 
: [[contact::M. McGhee]], Operations Support Department Leader, Shift Technical Advisors
: [[contact::M. Muhs]], Work Management Department Leader, Work Management Outage
: [[contact::P. McSparran]], Nuclear Training Department Leader, Total Operations Training
: [[contact::R. Meeden]], Engineering Section Leader, Design Mechanical NSSS
: [[contact::T. McCloud]], Performance Improvement Section Leader, Operating Experience
: [[contact::E. O'Neill]], TMA Management, Nuclear Assurance Administration
: [[contact::F. Oreshack]], Regulatory Affairs Consultant, Regulatory Affairs
: [[contact::S. Pobst]], Engineering Section Leader, Maintenance Rule
: [[contact::B. Routolo]], Radiation Monitoring Section Leader, RP RMS/RP Initial Training
: [[contact::J. Rodriguez]], Engineer II, Compliance
: [[contact::M. Renfroe]], TMA Management, Plant manager Admin
: [[contact::E. Sterling]], Nuclear Assurance Department leader, NAD Engineering and Support Admin.
: [[contact::J. Shannon]], Engineering Section Leader, Design Civil
: [[contact::J. Sontchi]], Nuclear Training Department leader, Tech/Mtce Training Admin
: [[contact::K. Schrecker]], Engineering Section Leader, System Engineering - BOP
: [[contact::M. Shea]], Director ImPACT, Safety Culture
: [[contact::R. Stroud]], Licensing Section Leader, Licensing
: [[contact::S. Sawtschenko]], Emergency Planning Program Department Leader, Emergency Preparedness
: [[contact::D. Wheeler]], Performance Improvement Team Department Leader, CAP 1 Admin
: [[contact::M. Webb]], Compliance Section Leader, Compliance 
  - 19 - Enclosure
: [[contact::T. Weber]], Nuclear Regulatory Affairs Department Leader, Nuclear Regulatory Admin
: [[contact::G. Zuniga]], Nuclear Maintenance Team Leader, Maintenance Admin A
===NRC Personnel===
: [[contact::N. Okeefe]], RIV DRS
: [[contact::G. Replogle]], RIV DRS
: [[contact::E. Ruesch]], RIV DRS
 
  - 20 - Attachment
 
==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED==
 
===Opened===
: 05000529/2010008-01 VIO Failure to Correct and Prevent Recurrence of a Significant Condition Adverse to Quality
Associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps
 
===Closed===
: 05000528-2010-003-00 LER 2010-003-00 Technical Specification Violation - Loss of Containment Building Equipment Hatch Closure Capability
: 05000528,529,530/2009005-01 VIO
: Failure to Establish Adequate Procedures to Control Potential Tornado Borne Missile Hazards Near the Essential Spray Ponds


  - 21 - Attachment
In accordance with NRC enforcement policy, the violation is listed below: 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action" requires, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected.
==LIST OF DOCUMENTS REVIEWED==
==Section 4OA2: Identification and Resolution of Problems==


===Procedures===
In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition.
: NUMBER TITLE REVISION / DATE 01PR-0AP04 Corrective Action Program 5 90DP-0IP10 Condition Reporting 47 01DP-0AP12 Palo Verde Action Request Processing 14
: 40DP-9OP26 Operations PVAR Processing and Operability Determination / Functional Assessment
: 90DP-0IP12 Root Cause CRDR Evaluation 5 70DP-0MR01 Maintenance Rule 30 90DP-0IP14
: Adverse CRDR Evaluation 4 01DP-0FI01 Management of Nuclear Projects 5 12DP-0MC08 Control of Purchasing Material and Equipment 21
: 74DP-9CY04 Systems Chemistry Specifications 64 81TD-0EE10 Design Change Process 24 74OP-9SP04 Essential Spray Pond Chemical Addition System Human-Machine Interface Operation
: 40OP-9SP-04 Spray Pond Chemical Addition System Train B 33 40OP-9SP-03 Spray Pond Chemical Addition System Train A 34 01DP-0XX-01 Control and Monitoring of Potential Tornado Borne Missiles 1 75DP-0RP02 Radioactive Contamination Control 12
: EPIP-04 Emergency Operations Facility Actions 51 87DP-0CC08 Control of Vendor Documentation 18
: 01DP-0EM09 Employee Concerns Program 2 40DP-9OP15 Operator Challenges and Discrepancy Tracking 26 40OP-9PC07 Miscellaneous Fuel Pool Operations 57 40AL-9RK3A Panel B03A Alarm Responses 27 40AO-9ZZ06 Loss of Instrument Air 32
: 01DP-0AP01-01 Procedure Preparer's Administrative Guideline 4 01DP-0AP01-02 Procedure Reviewer's Administrative Guideline 2 01DP-0AP01 Procedure Process 43 01DP-0AP21 NATM Procedure Replacement Project 0 70DP-0RA01 Shutdown Risk Assessments 35
: 40OP-9ZZ23 Outage
: GOP 59 90DP-OIP13
: Apparent Cause CRDR Evaluation 5 65DP-0QQ01 Industry Operating Experience Review 25 40DP-9OP26 Operations PVAR Processing and Operability Determination/Functional Assessment
: 90DP-OIP15 Review CRDR Evaluation 15 01DP-0AP16 PVNGS Self Assessments and Benchmarking 7
: 60DP-0QQ02 Trend Analysis and Coding 22 40OP-9DG01 Emergency Diesel Generator A 66 
  - 22 - Attachment Procedures
: NUMBER TITLE REVISION / DATE 40ST-9DG01 Emergency Diesel Generator A Test 40 73ST-9DF01 Diesel Fuel Oil Transfer Pump Inservice Test 22 40OP-9DG02 Emergency Diesel Generator B 62
: 70DP-0RA05
: Assessment and Management of Risk when Performing Maintenance in modes 1 and 2
: 73ST-9DF01 Diesel Fuel Oil Transfer Pump Inservice Test 22 93DP-0LC07 10
: CFR 50.59 and 72.48 Screenings and Evaluations 21 93DP-0LC17 10
: CFR 50.59 and 72.48 Guidance Manual 5 
===Condition Report===
: Action Items
: CRAI 3112430
: CRAI 3182155
: CRAI 3182163
: CRAI 3182175
: CRAI 3182179
: CRAI 3182185
: CRAI 3182198
: CRAI 3183978
: CRAI 3183981
: CRAI 3337954
: CRAI 3488855
: CRAI 3337957
: CRAI 3476074
: CRAI 3486807
: CRAI 3149172
: CRAI 3149165
: CRAI 3149057
: CRAI 3297439
: CRAI 3297444
: CRAI 3367740
: CRAI 3307571
: CRAI 3307579
: CRAI 3309913
: CRAI 3340171
: CRAI 3477407
: CRAI 2941932
: CRAI 3407034
: CRAI 3407039
: CRAI 2988522
: CRAI 3032686
: CRAI 3075733
: CRAI 3400753
: CRAI 3270067
: CRAI 3348269
: CRAI 3140933
: CRAI 3333356
: CRAI 3337943
: CRAI 3288652
: CRAI 3333353
: CRAI 3047302
: CRAI 3437919
: CRAI 3445238
: CRAI 3392785
: CRAI 3270067
: CRAI 3295068
: CRAI 3295070
: CRAI 3295078
: CRAI 3327046
: CRAI 3330347
: CRAI 3350208
: CRAI 3420003
: CRAI 3420005
: CRAI 3334253
: CRAI 3298139
: CRAI 3298141
: CRAI 3479994
: CRAI 3487566
: CRAI 3127427
: CRAI 3293601
: CRAI 3294085
: CRAI 3112414
: CRAI 3112425
: CRAI 3212753
: CRAI 3421079
: CRAI 3297448
: CRAI 3243265
: CRAI 3381398
: CRAI 3243265
: CRAI 3381398
: CRAI 3258233
: CRAI 3301301
: CRAI 3318967
: CRAI 3300928
: CRAI 3300889
: CRAI 3300892
: CRAI 3300912
: CRAI 3560968
: CRAI 3419280
: CRAI 2919416
: CRAI 2919417
: CRAI 2771662
: CRAI 2785586
: CRAI 3030118
: CRAI 3333020
: CRAI 3380080
: CRAI 3333028
: CRAI 3333032
: CRAI 3452548
: CRAI 3342549
: CRAI 2902572
: CRAI 3563117
: CRAI 3560693
: CRAI 3563121
: CRAI 3563122
: CRAI 3560705
: CRAI 3333033
: CRAI 3342549
: CRAI 3563060
: CRAI 3560747
: CRAI 3560727
: Palo Verde Action Requests
: PVAR 3171090
: PVAR 3484106
: PVAR 3103044
: PVAR 3283865
: PVAR 3327597
: PVAR 3302093
: PVAR 3567358
: PVAR 3491342
: PVAR 3492005
: PVAR 3296740
: PVAR 3303150
: PVAR 3292077
: PVAR 3185240
: PVAR 3055680
: PVAR 2968212
: PVAR 3124491
: PVAR 3302472
: PVAR 3419941
: PVAR 3419732
: PVAR 3563820
: PVAR 3511452
: PVAR 3527166
: PVAR 3529476
: PVAR 3425898
: PVAR 3529475
: PVAR 3562840
: PVAR 3209329
: PVAR 3209031
: PVAR 3055297
: PVAR 3055294
: PVAR 3274141
: PVAR 3566562
: PVAR 3009375
: PVAR 3567849
: PVAR 3567337
: PVAR 3475479
: PVAR 3009375
: PVAR 3562636
: PVAR 3562621
: PVAR 3560617 
  - 23 - Attachment 
===Condition Report===
: Disposition Requests
: CRDR 3171340
: CRDR 3139278
: CRDR 3139279
: CRDR 3105139
: CRDR 3308660
: CRDR 3476071
: CRDR 3417311
: CRDR 3382436
: CRDR 3125620
: CRDR 3456438
: CRDR 3479992
: CRDR 3279750
: CRDR 3282706
: CRDR 3311298
: CRDR 3282707
: CRDR 3283802
: CRDR 3340170
: CRDR 3310925
: CRDR 3293411
: CRDR 3282708
: CRDR 3484837
: CRDR 3466318
: CRDR 3451431
: CRDR 3048866
: CRDR 2897810
: CRDR 3470763
: CRDR 3394172
: CRDR 3393609
: CRDR 3297924
: CRDR 3173844
: CRDR 3314991
: CRDR 3383919
: CRDR 3358222
: CRDR 3288651
: CRDR 3211958
: CRDR 3434085
: CRDR 3111432
: CRDR 3398042
: CRDR 3296869
: CRDR 3446100
: CRDR 3297229
: CRDR 3296421
: CRDR 3292700
: CRDR 3434624
: CRDR 3280774
: CRDR 3304057
: CRDR 3219742
: CRDR 3202957
: CRDR 3560585
: CRDR 3281353
: CRDR 3289353
: CRDR 3336555
: CRDR 3411547
: CRDR3411547
: CRDR 3283802
: CRDR 3185716
: CRDR 3229998
: CRDR 3282704
: CRDR 3312554
: CRDR 3383898
: CRDR 3211958
: CRDR 3246250
: CRDR 3255331
: CRDR 3279838
: CRDR 3285918
: CRDR 3296259
: CRDR 3301459
: CRDR 3304518
: CRDR 3299343
: CRDR 3301540
: CRDR 3315776
: CRDR 3323052
: CRDR 3337682
: CRDR 3287805
: CRDR 3418378
: CRDR 3303334
: CRDR 3343933
: CRDR 3434665
: CRDR 3481401
: CRDR 3485725
: CRDR 3411074
: CRDR 3411033
: CRDR 3411161
: CRDR 3343933
: CRDR 3401304
: CRDR 3411031
: CRDR 3311158
: CRDR 3411155
: CRDR 3411152
: CRDR 3487756
: CRDR 3405785
: CRDR 3481402
: CRDR 2761657
: CRDR 3012697
: CRDR 31202747
: CRDR 3529151
: CRDR 2882166
: CRDR 3517526
: CRDR 3303334
: CRDR 3347523
: CRDR 3287805
: CRDR 3086170
: CRDR 3317532
: CRDR 3418378
: CRDR 2882166
: CRDR 3390784
: CRDR 3431177
: CRDR 3425538
: CRDR 3426364
: CRDR 3562875
: CRDR 3192389
: CRDR 3059185
===Work Orders===
: 236820
: 3291430
: 3291434
: 3339386
: 3097215
: 3334856
: 2944206
: 3057308
: 2967863
: 2934353
: 3304408
: 3388525
: 3527172
: 3534552
: 3534548
: 3558553
: 3529375
: 3527183
: 2785583
: 3316981
: 3539192
: 3529213
: 3529215
: 3529223
: 3529237
: 26319
: 3317320
: 2910651
: 3540782
: 3426319
: Audits 2010-002 Chemistry Audit Report 2009-010 Operations Audit Report
: 2009-009 Corrective Action Audit Report 2010-001 Emergency Preparedness Audit Report 2009-003 Special Processes, Inservice Inspection & Testing Audit Report 2009-004 Material Control/Procurement Audit Report 2009-005 Fire Protection Audit Report 2009-007 Design Control Audit Report 2010-007 Technical Specifications & Administrative Controls Audit Report
: 2010-008 Radiation Protection Audit Report 2010-004 Maintenance And Refueling Outage Activities Audit Report 
  - 24 - Attachment 2009-006 Document Control Audit Report 2009-011 Verification Process Audit Report 2010-003
: Maintenance Rule Audit Report 2010-006 Training and Qualification Audit Report
: Self Assessments
: 278891
: 3287745
: 3332917
: 3424019
: 3188986
: 3456721
: 3456891
: 3531574
: 3511365
: 3350944
: 3525944
: 3511217
: 351109
: 3525944
: 3511217
: 3456930
: 3362726
: 3420895 
: Drawing Numbers
: Title
: Revision
: 2-M-DGP-001 Diesel Generator System Rev 52 02-E-DFB-001
: Elementary Diagram Diesel Fuel Oil & Transfer System Rev 5 13-E-ZVU-0014 Underground Electrical Installation Rev 15 03-E-ZYU-010 Diesel Storage Tank Conduit Plan and Section Rev. 7 12-E-ZYU-009 Diesel Storage Tank Conduit Plan & Section Rev. 12 
===Miscellaneous Documents===
: NUMBER TITLE REVISION / DATE
: EWR
: 3339327 DG B Trip on Lube Oil Low Pressure Turbo During Cooldown
: EJT12C00102 Electrical Maintenance Training Program Polar Crane JITT 9/23/2009 NGW16-L-0001-13 Radworker Dressout Practical August 19, 2008 NGW01-C-0001-36 Radiation Worker Training March 4, 2010 Audit Report 2010-01 Emergency Preparedness March 2, 2010 N/A Performance Improvement Department Qualification Requirements and Training Program Description
: 008 N/A Effectiveness Review SIIP Task 6.1.1.c CRAI
: 3032686 September, 2009
: LER 50-528/2010-003-00 Technical Specification - Loss of Containment Building Equipment Hatch Closure Capability July 7, 2010 S-09-0039 10
: CFR 50.59 Screening for DFWO
: 3286021 0
: SP-485 Industrial Safety Assessment In Support of EWR
: 3491766 0 (CRAI 3127427) Engineering Evaluation of EDG Fluid Leakage and Potential Operability Concerns
: EWR 3283182 EDG Slow Start 0 
  - 25 - Attachment
: ODP-7 Technical Specifications 9 ECP02 Employee Concerns Program Guideline 10 N/A Unit 1 Control Room Deficiency Log Dec 15, 2010 MP1 Site Corrective Maintenance Metric Summary Nov 2010 MP2 Site Elective Maintenance Metric Summary Nov 2010 OP1 Site Operational Focus Indicator Summary Nov 2010
: N/A Site top 10 technical Issues Summary Dec 2010 N/A Active Standing Orders Dec 2, 2010 N/A Units 1,2,3 Night Orders Nov 2010 N/A NATM Procedure Replacement Project Schedule 2011 N/A NATM Procedure Replacement Project Schedule 2010 N/A CRG Daily Agenda
: Dec 1, 2010 N/A ARRC Daily Agenda Dec 2, 2010 N/A CARB Agenda Nov 30, 2010 13-VTD-G080-0137 General Electric CR120B 600 Volt Industrial Relay - Series June 11, 1992 P.O.
: 500514424 Element Assembly, Lube Oil Thermostatic Valve; Cooper-Bessember Part Number 2-05V-419-109 Nov 15, 2007 U3R15 R8 10-
: Unit 3 R15 PRA model Nov 10, 2010 13-EN-0301 Installation Specification for Electrical Cables in Conduit and Duct Banks Rev 5
: 13-EN-0306 Installation Specification for Cabling Splices and Terminations Rev 11 N/A Updated Final Safety Analysis Report, Section 9.5 Rev 11 13-EM-057 600V Control Cables
: Site Health Reports January through December
: 2009
: Site Health Reports January through December
: 2010
: NRC Information Notice 2007-01
: Regulatory Guide 1.137 Fuel-Oil Systems For Standby Diesel Generators


  - 26 - Attachment Information Request October 12, 2010 Biennial Problem Identification and Resolution Inspection Palo Verde Nuclear Generating Station Inspection Report Number 05000528/529/530-2010008
Contrary to the above, from April 2009 through September 2010, the licensee failed to correct a significant condition adverse to quality and implement adequate corrective actions to preclude repetition.
: This inspection will cover the period from March 1, 2009, through December 17, 2010.
: All requested information should be limited to this period unless otherwise specified.
: To the extent possible, the requested information should be provided electronically in Adobe PDF or Microsoft Office format.
: Lists of documents should be provided in Microsoft Excel or a similar sortable format.
: A supplemental information request will likely be sent during the week of November 9, 2010.
: Please provide the following no later than November 2, 2010.
: 1. Document Lists Note: for these summary lists, please include the document/reference number, the document title or a description of the issue, initiation date, and current status.
a. Summary list of all corrective action documents related to significant conditions adverse to quality that were opened, closed, or evaluated during the period
b. Summary list of all corrective action documents related to conditions adverse to quality that were opened or closed during the period
c. Summary lists of all corrective action documents which were upgraded or downgraded in priority/significance during the period
d. Summary list of all corrective action documents that subsume or "roll up" one or more smaller issues for the period
e. Summary lists of operator workarounds, engineering review requests and/or operability evaluations, temporary modifications, and control room and safety system deficiencies opened, closed, or evaluated during the period
f. Summary list of plant safety issues raised or addressed by the Employee Concerns Program (or equivalent)


g. Summary list of all Apparent Cause Evaluations completed during the period
Specifically, the licensee failed to correct a water intrusion path to the Unit 2 motor termination boxes for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections and a pump trip.This Notice of Violation is associated with a Green Significance Determination Process finding.I Enclosure 2 Reply to Notice of Violation EA-1 1-006 Arizona Public Service (APS) concurs with the violation.
h. Summary list of all Root Cause Evaluations planned or in progress but not complete at the end of the period
: 2. Full Documents, with Attachments
a. Root Cause Evaluations completed during the period
b. Quality assurance audits performed during the period 
  - 27 - Attachment
c. All audits/surveillances performed during the period of the Corrective Action Program, of individual corrective actions, and of cause evaluations
d. Corrective action activity reports, functional area self-assessments, and non-NRC third party assessments completed during the period (do not include INPO assessments)
e. Corrective action documents generated during the period for the following:
i. NCV's and Violations issued


ii. LER's submitted
The NRC Integrated Inspection Report 05000528/2010008, 05000529/20109008, and 05000530/2010008, noted that APS failed to restore compliance of the Unit 2 emergency diesel generator fuel oil transfer pumps (DFOTP) within a reasonable time after the previous violation was identified in NCV 05000529/2009-004-002.
f. Corrective action documents generated for the following (for those that were evaluated but determined not to be applicable, provide a summary list):
i. NRC Information Notices, Bulletins, and Generic Letters issued or evaluated during the period


ii. Part 21 reports issued or evaluated during the period iii. Vendor safety information letters (or equivalent) issued or evaluated during the period  iv. Other external events and/or Operating Experience evaluated for applicability during the period
The investigation concluded that the previous investigation did not recognize the possibility for water to flow inside the control cable's outer jacket to cause an electrical fault inside the motor termination box of the DFOTP.The Reason For The Violation The cause for failing to take adequate actions to correct the water intrusion path to the Unit 2 motor termination box for the emergency diesel generator fuel oil transfer pump (DFOTP) was that the investigation from the previous event did not identify an internal water path through a degraded cable outer jacket to the motor termination box. Subsequent investigation determined that the probable cause for the degradation of the cable outer jacket which allowed the water path to develop was the presence of an unknown chemical contaminant on the cable jacket. Corrective actions from the previous investigation focused on an electrical short to ground inside the motor termination box as a result of moisture and corrosion inside the motor termination box. The moisture source was thought to have been caused by water traveling through the conduit around the cables installed in the conduit. The previous investigation did not recognize the possibility for water to flow inside of the control cable's outer jacket to the motor termination box.The Corrective Steps That Have Been Taken And The Results Achieved As an immediate corrective action, the two conductor control cables and three conductor power cables from the cable pull boxes in the DFOTP vault to the motor termination boxes for the six site DFOTPs were replaced.
g. Corrective action documents generated for the following:
i. Emergency planning drills and tabletop exercises performed during the period ii. Maintenance preventable functional failures which occurred or were evaluated during the period iii. Adverse trends in equipment, processes, procedures, or programs which were evaluated during the period iv. Action items generated or addressed by plant safety review committees during the period
: 3. Logs and Reports
a. Corrective action performance trending/tracking information generated during the period and broken down by functional organization
b. Corrective action effectiveness review reports generated during the period


c. Current system health reports or similar information
The control and power cables were spliced to interrupt the drain path through the cable outer jacket around the cables and prevent possible future water drainage into the motor termination box. Additionally, boroscopic examinations were performed for the accessible sections of cable between the diesel building and the DFOTP 1 Enclosure 2 Reply to Notice of Violation EA-1 1-006 vault. This section of cable for the Unit 2 train A DFOTP was replaced due to the degraded condition of its outer jacket. No additional cables in the areas inspected boroscopically needed to be replaced.
d. Radiation protection event logs during the period 
  - 28 - Attachment
e. Security event logs and security incidents during the period (sensitive information can be provided by hard copy during first week on site)
f. Employee Concern Program (or equivalent) logs (sensitive information can be provided by hard copy during first week on site)
g. List of Training deficiencies, requests for training improvements, and simulator deficiencies for the period
: 4. Procedures
a. Corrective action program procedures, to include initiation and evaluation procedures, operability determination procedures, apparent and root cause evaluation/determination procedures, and any other procedures which implement the corrective action program.
b. Quality Assurance program procedures
c. Employee Concerns Program (or equivalent) procedures
d. Procedures which implement/maintain a Safety Conscious Work Environment
: 5. Other  a. List of risk significant components and systems


b. Organization charts for plant staff and long-term/permanent contractors Note:  "Corrective action documents" refers to condition reports, notifications, action requests, cause evaluations, and/or other similar documents, as applicable.
Finally, the motor termination boxes for the Unit 2 trains A and B DFOTP were replaced due to corrosion.
: This information should be uploaded on the Certrec IMS website no later than November 2, 2010.
: In addition, all electronic documents should be loaded onto a CD or DVD and sent via overnight carrier to:
: U.S. NRC Region
: IV 612 E. Lamar Blvd. Suite 400 Arlington,
: TX 76011-4125
: Attn: Harry Freeman Please note that the NRC is not currently able to accept electronic documents on thumb drives or other similar digital media.


  - 29 - Attachment Palo Verde Fuel Oil Transfer Pump Failure Risk Assessment The senior resident inspector performed the initial significance determination for the fuel oil transfer pump failure using the NRC Inspection Manual 0609, Attachment 0609.04, "Phase 1 -
The Units 1 and 3 motor termination boxes were inspected and exhibited no evidence of corrosion or moisture intrusion.
: Initial Screening and Characterization of Findings."
: The finding screened to a Phase 2 significance determination because it involved a loss of one train of safety related equipment for greater than its technical specification allowed outage time.
: A Region IV senior reactor analyst performed a Phase 2 significance determination using the pre-solved worksheet from the "Risk Informed Inspection Notebook for the Palo Verde Nuclear Generating Station," Revision 2.01a.
: The analyst assumed an exposure period of one year.
: The finding was potentially Yellow, which warranted further review.
: The senior reactor analyst subsequently performed a bounding Phase 3 significance determination, which is provided below.
: Internal Events:
: The analyst used the Palo Verde SPAR Model, Saphire 8, Revision 8.15, dated August 21, 2010 and assumed a truncation limit of 1.0E-13.
: Only Unit-1 was modeled, but it could be used for any one of the three units.
: Unit 1 terminology is used.
: The fuel oil transfer pump failed on September 15, 2010.
: The last time that the pump operated satisfactorily was August 18, 2010.
: Since the exact time of inoperability was not known, the analyst assumed a T/2 mission time of 14 days.
: To refine the estimate further, the analyst noted that the fuel oil day tank could have provided at least 3.0 hours of fuel.
: Therefore, with the fuel oil transfer pump failed, the emergency diesel generator could have operated for at least 3.0 hours.
: NUREG/CR 6890, Volume 1, "Analysis of Loss of Offsite Power Events: 1986-2004" specified a loss of offsite power non-recovery probability of 2.15E-1.
: That means, following a loss of offsite power initiating event, there is a
: 79.5% chance that offsite power will be recovered before the day tank expires.
: The analyst adjusted the loss of offsite power frequency to be consistent with these assumptions.
: The base loss of offsite power frequency for Palo Verde was 3.59 E-2/year.
: The 3.0-hour non-recovery of offsite power was 2.15E-1.
: Therefore, the frequency for a loss of offsite power event that cannot be recovered in 3.0 hours was 3.59E-2 * 2.15E-1 = 7.7E-3/year.
: This was the new loss of offsite power initiating frequency.
: Resetting event time t=0 to 3.0 hours following the event required that the recovery factors for offsite power be adjusted.
: Because the new loss of offsite power frequency included the assumption that offsite power was not recovered in the first 3.0 hours, the analyst calculated new nonrecovery values for offsite power given that the event of interest started at 3.0 hours.
: The analyst used the following standard statistical equation for the calculation:
: P(BIA) = P(A*B)/P(A), where:
: P(BIA) = probability that offsite power will not be recovered given that it was not recovered at t=3.0 hours.


  - 30 - Attachment P(A) = nonrecovery probability of offsite power at t=3.0 hours.
These actions eliminate the potential water flow path to the DFOTP motor termination box through the inside of the cable's outer jacket.The Corrective Steps That Will Be Taken To Avoid Further Violations In addition to the actions described above, the following design modification, specification changes and extent of condition evaluation will be performed." Implement a Design Change Request (DCR) to replace the underground duct banks and associated cabling from all three EDG buildings to the six site DFOST vaults with direct buried submersible cables." Revise installation specification 13-EN-0300 "Installation Specification for Electrical Cable in Cable Trays" and 13-EN-0301 "Installation Specification for Electrical Cables in Conduit and Duct Banks" to identify compatible chemicals for cleaning cables." Revise installation specification 13-EN-0306 "Installation Specification for Cable Splicing and Termination" to add a requirement to install a cable splice or other means of blocking the flow of water internal to the outer jacket of cables where an electrical component is at a location such that a hydrostatic head may be developed with respect to the cables.* Evaluate the need for cable splices for critical components and components that support key safety or operational functions, that are located in areas where water may be transported through the outer jacket of the cables.2 Enclosure 2 Reply to Notice of Violation EA-1 1-006 Additionally, Palo Verde will continue to monitor the EDG DFOTP cables as part of the Palo Verde preventive maintenance program to ensure the actions taken are effective to prevent water flow to the DFOTP motor termination box.The Date When Full Compliance Will Be Achieved Full compliance was achieved on September 23, 2010, with the replacement of the two conductor control cables and three conductor power cables for the Unit 2 trains A and B DFOTP. During replacement these cables were spliced to interrupt the drain path and prevent possible future water drainage into the motor termination box through a path internal to the jackets around the cables.3
: Equal to 2.149E-1.
: P(B) = SPAR nonrecovery of offsite power at time = x Since B is a subset of A and the two probabilities are dependent, P(A*B) = P(B).
: The equation then reduced to:
: P(BIA) = P(B)/(PA)
: For example, for the 2-hour SPAR sequences, the offsite power nonrecovery probability at 5.0 hours, given that offsite power was not recovered at 3.0 hours is: 
: P(BlA) = 1.265E-1/2.149E-1 = 5.866E-1 
: The following table represents adjustments the analyst made to the SPAR model to accommodate these changes:
: Offsite Power Recovery Period Normal Offsite Power Non-Recovery Value Adjusted Offsite Power Non- Recovery Value (i.e.
+3 hours) Notes: 1.0 Hour 5.303E-1 7.278E-1
: 2.0Hours
: 3.181E-1 5.886E-1 Example cited above 3.0 hours 2.149E-1 4.484E-1
: 4.0 hours 1.566E-1 3.806E-1
: 6.0 hours 9.637E-2 2.741E-1
: 8.0 hours 6.718E-2 2.173E-1
: 10.0 hours 5.070E-2 1.801E-1
: 15.0 hours 3.079E-2 1.233E-1
: 24 hours 1.79E-2 8.33E-2 
: The analyst performed two calculations, a "base case," and a "current case."
: The base case assumed that the train A emergency diesel generator did not fail because of the performance deficiency.
: The current case assumed that diesel did fail (at 3.0 hours).
: Nominal recoveries were allowed.
: The analyst set the train A emergency diesel generator "failure to run" basic event to True, which meant that a common cause failure of the train B emergency diesel generator was possible.
: The analyst then solved only the loss of offsite power sequences for the base case and current case.
: For a one year exposure period, the base case conditional core damage probability was 9.2E-7
and the current case (with the failure) probability was 1.2E-5.
: The incremental conditional core damage probability was 1.1E-5.
: For a 14 day exposure period, the delta-CDF for internal event initiators was:
: Delta-CDFinternal = 1.1E-5 * 14/365 = 4.2E-7/year 
  - 31 - Attachment External Events:
: External events of interest included seismic induced loss of offsite power events and fire events that could result in a loss of offsite power.
: Seismic:
: The Analyst used the NRC's "Risk Assessment of Operational Events Handbook," Revision 1.01 to estimate the delta-CDF contribution from seismic events.
: Appendix 1, Table 1, "Frequencies of Seismically-Induced Loss of Offsite Power Events," stated that the seismic induced loss of offsite power frequency for Palo Verde was 5.37E-5.
: Seismic initiated losses of offsite power are considered non-recoverable.
: The analyst use the noted SPAR model and calculated the conditional core damage probability (CCDP) for a non-recoverable loss of offsite power initiating event with a failed emergency diesel generator.
: The analyst allowed the nominal emergency diesel generator recoveries to occur.
: The day tank provided 3.0 hours of run time before the emergency diesel would have failed.
: Operators would have had approximately 3.0 hours to troubleshoot the condition before failure occurred.
: The CCDP was 1.4E-3.
: The delta-CDF for a 14 day exposure period was:
: Delta-CDFseismic = (3.5E-5*1.4E-3)*14/365 = 1.9E-9
: Fires:
: The fires of interest involved those that could have caused a loss of offsite power to Unit 2.
: The analyst used the "Palo Verde Nuclear Generating Station, Units 1, 2, and 3 Individual Plant Examination of External Events (IPEEE) for Severe Accident Vulnerabilities," dated June 30, 1995 to evaluate the risk increase from fire initiators.
: The IPEEE identified three fire compartments, where a fire in one of those compartments could cause a loss of offsite power.
: Those compartments included:
: 5A - Train A ESF switchgear room 5B - Train B ESF switchgear room
- Control room The IPEEE concluded that a loss of offsite power was not a significant risk in the control room (because of cable routing and limited exposure to affected circuits), unless the fire was substantial and warranted a control room evacuation.
: In these cases, operators would rely on train B components to affect a safe shutdown.
: Since the performance deficiency affected train A, it did not result in a quantitative change to the core damage frequency for control room fires.
: Fires in fire compartment 5A were assumed to render the train A components inoperable.
: Therefore, a fire in this area did not result in a quantitative change to the core damage frequency.
: Fires in fire compartment 5B had the potential to cause a loss of offsite power coupled with the loss of train B components.
: However, the analyst noted that the affected offsite power feeds had isolation breakers, outside of the fire area that should open to isolate offsite power from the affected bus if a short did occur.
: If offsite power was lost to train
: A (because a breaker did not open), operators could easily isolate the fault and restore 
  - 32 - Attachment offsite power to the unit within the 3.0 hours dictated by the day tank capacity.
: The analyst qualitatively screened this fire area from further consideration.
: The analyst noted that the actual risk associated with fires was not zero but, due to limitations in the IPEEE process, no quantitative result could be practicably obtained.
: Nonetheless, the analyst qualitatively determined that the increase to the core damage frequency was very small.
: Internal and External Events:
: The delta-CDF for the performance deficiency was:
: Delta-CDF = Delta-CDFinternal + Delta-CDFseismic + Delta-CDF
fires          = 4.2E-7 + 1.9E-9 + 0 = 4.2E-7/year The licensee performed a similar calculation, at the request of the analyst, and calculated the Delta-CDF to be approximately 1.2E-7.
: Large Early Release Frequency (LERF):
: Using
: IMC 0609 Appendix H, the SRA determined that this was a Type A finding for a large dry containment.
: For pressurized water reactor plants with large dry containments, only findings related to accident categories intersystem loss of coolant accidents or steam generator tube ruptures have the potential to impact LERF.
: Since this finding is not related to these two types of accidents, the analysts concluded that LERF was not a significant contributor to the risk associated with this finding.
: Conclusion:
: Since the Delta-CDF was less than 1E-6 and the Delta- LERF was not a significant contributor to risk, this finding was of very low safety significance, Green.
: The dominant cutsets included a loss of offsite power initiating event, failure to align the turbine driven generator and failures of the turbine driven auxiliary feedwater pump.
: Since most of this same equipment remained available, the components helped to mitigate the significance of the finding.
}}
}}

Revision as of 21:52, 18 September 2018

Palo Verde Nuclear Generating Station Units 1, 2 and 3 - NRC Problem Identification and Resolution Inspection Report 05000528-10-008, 05000529-10-008, 05000530-10-008, and Notice of Violation, Dated January 28, 2011
ML110670169
Person / Time
Site: Palo Verde  Arizona Public Service icon.png
Issue date: 02/28/2011
From: Mims D C
Arizona Public Service Co
To:
Document Control Desk, Office of Nuclear Reactor Regulation
References
102-06326-DCM/DFH, EA-11-006 IR-10-008
Download: ML110670169 (6)
v • d • e


Text

10 CFR 2.201 EA-11-006 A':AO A subsidiary of Pinnacle West Capital Corporation Dwight C. Mims Mail Station 7605 Palo Verde Nuclear Senior Vice President Tel. 623-393-5403 P.O. Box 52034 Generating Station Regulatory Affairs and Oversight Fax 623-393-6077 Phoenix, Arizona 85072-2034 102-06326-DCM/DFH February 28, 2011 ATTN: Document Control Desk U.S. Nuclear Regulatory Commission Washington, DC 20555-0001 Reference:

Palo Verde Nuclear Generating Station Units 1, 2 and 3 NRC Problem Identification and Resolution Inspection Report 05000528/2010008, 05000529/2010008, 05000530/2010008, and Notice of Violation, dated January 28, 2011

Dear Sirs:

Subject: Palo Verde Nuclear Generating Station (PVNGS)Units 1, 2 and 3 Docket Nos. STN 50-528, 50-529, and 50-530 Reply to Notice of Violation EA-11-006 In the above referenced letter to Arizona Public Service (APS), the NRC identified that APS failed to correct a water intrusion path to the Unit 2 motor termination boxes for the emergency diesel generator fuel oil transfer pumps and implement adequate corrective actions to preclude repetition.

Pursuant to the requirements of 10 CFR 2.201 and Notice of Violation (NOV) EA-1 1-006, attached to the above referenced letter, APS hereby submits its reply to the NOV.Enclosure 1 to this letter contains a restatement of the violation.

Enclosure 2 contains the APS reply to the NOV.No commitments are being made to the NRC by this letter.A member of the STDARS (Strategic Teaming and Resource Sharing) Alliance ZZ7rJII-.Callaway * Comanche Peak * Diablo Canyon -Palo Verde -San Onofre -South Texas -Wolf Creek i ATTN: Document Control Desk U.S. Nuclear Regulatory Commission Reply to Notice of Violation EA-1 1-006 Page 2 If you have any questions, please contact Marianne Webb, Regulatory Affairs, Compliance Section Leader, at (623) 393-5730.

Sincerely,DCM/MNW/DFH/gat

Enclosures:

1. Restatement of Notice of Violation EA-1 1-006 2. Reply to Notice of Violation EA-1 1-006 cc: E. E. Collins Jr.J. R. Hall L. K. Gibson M. A. Brown NRC Region IV Regional Administrator NRC NRR Senior Project Manager -(electronic

/ paper)NRC NRR Project Manager (electronic

/ paper)NRC Senior Resident Inspector for PVNGS Enclosure 1 Restatement of Notice of Violation EA-1 1-006 During an NRC inspection conducted on November 29, 2010 through December 17, 2010, a violation of NRC requirements was identified.

In accordance with NRC enforcement policy, the violation is listed below: 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action" requires, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected.

In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition.

Contrary to the above, from April 2009 through September 2010, the licensee failed to correct a significant condition adverse to quality and implement adequate corrective actions to preclude repetition.

Specifically, the licensee failed to correct a water intrusion path to the Unit 2 motor termination boxes for the emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections and a pump trip.This Notice of Violation is associated with a Green Significance Determination Process finding.I Enclosure 2 Reply to Notice of Violation EA-1 1-006 Arizona Public Service (APS) concurs with the violation.

The NRC Integrated Inspection Report 05000528/2010008, 05000529/20109008, and 05000530/2010008, noted that APS failed to restore compliance of the Unit 2 emergency diesel generator fuel oil transfer pumps (DFOTP) within a reasonable time after the previous violation was identified in NCV 05000529/2009-004-002.

The investigation concluded that the previous investigation did not recognize the possibility for water to flow inside the control cable's outer jacket to cause an electrical fault inside the motor termination box of the DFOTP.The Reason For The Violation The cause for failing to take adequate actions to correct the water intrusion path to the Unit 2 motor termination box for the emergency diesel generator fuel oil transfer pump (DFOTP) was that the investigation from the previous event did not identify an internal water path through a degraded cable outer jacket to the motor termination box. Subsequent investigation determined that the probable cause for the degradation of the cable outer jacket which allowed the water path to develop was the presence of an unknown chemical contaminant on the cable jacket. Corrective actions from the previous investigation focused on an electrical short to ground inside the motor termination box as a result of moisture and corrosion inside the motor termination box. The moisture source was thought to have been caused by water traveling through the conduit around the cables installed in the conduit. The previous investigation did not recognize the possibility for water to flow inside of the control cable's outer jacket to the motor termination box.The Corrective Steps That Have Been Taken And The Results Achieved As an immediate corrective action, the two conductor control cables and three conductor power cables from the cable pull boxes in the DFOTP vault to the motor termination boxes for the six site DFOTPs were replaced.

The control and power cables were spliced to interrupt the drain path through the cable outer jacket around the cables and prevent possible future water drainage into the motor termination box. Additionally, boroscopic examinations were performed for the accessible sections of cable between the diesel building and the DFOTP 1 Enclosure 2 Reply to Notice of Violation EA-1 1-006 vault. This section of cable for the Unit 2 train A DFOTP was replaced due to the degraded condition of its outer jacket. No additional cables in the areas inspected boroscopically needed to be replaced.

Finally, the motor termination boxes for the Unit 2 trains A and B DFOTP were replaced due to corrosion.

The Units 1 and 3 motor termination boxes were inspected and exhibited no evidence of corrosion or moisture intrusion.

These actions eliminate the potential water flow path to the DFOTP motor termination box through the inside of the cable's outer jacket.The Corrective Steps That Will Be Taken To Avoid Further Violations In addition to the actions described above, the following design modification, specification changes and extent of condition evaluation will be performed." Implement a Design Change Request (DCR) to replace the underground duct banks and associated cabling from all three EDG buildings to the six site DFOST vaults with direct buried submersible cables." Revise installation specification 13-EN-0300 "Installation Specification for Electrical Cable in Cable Trays" and 13-EN-0301 "Installation Specification for Electrical Cables in Conduit and Duct Banks" to identify compatible chemicals for cleaning cables." Revise installation specification 13-EN-0306 "Installation Specification for Cable Splicing and Termination" to add a requirement to install a cable splice or other means of blocking the flow of water internal to the outer jacket of cables where an electrical component is at a location such that a hydrostatic head may be developed with respect to the cables.* Evaluate the need for cable splices for critical components and components that support key safety or operational functions, that are located in areas where water may be transported through the outer jacket of the cables.2 Enclosure 2 Reply to Notice of Violation EA-1 1-006 Additionally, Palo Verde will continue to monitor the EDG DFOTP cables as part of the Palo Verde preventive maintenance program to ensure the actions taken are effective to prevent water flow to the DFOTP motor termination box.The Date When Full Compliance Will Be Achieved Full compliance was achieved on September 23, 2010, with the replacement of the two conductor control cables and three conductor power cables for the Unit 2 trains A and B DFOTP. During replacement these cables were spliced to interrupt the drain path and prevent possible future water drainage into the motor termination box through a path internal to the jackets around the cables.3

Retrieved from "https://kanterella.com/w/index.php?title=IR_05000528/2010008&oldid=1052451"