IR 05000528/2012007
| ML12349A184 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 12/13/2012 |
| From: | Ray Kellar Division of Reactor Safety IV |
| To: | Edington R Arizona Public Service Co |
| References | |
| IR-12-007 | |
| Download: ML12349A184 (19) | |
Text
UNITE D S TATE S NUC LEAR RE GULATOR Y C OMMI S SI ON ber 13, 2012
SUBJECT:
PALO VERDE NUCLEAR GENERATING STATION - NRC BIENNIAL PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT 05000528/2012007, 05000529/2012007, AND 05000530/2012007
Dear Mr. Edington:
On November 9, 2012, the U. S. Nuclear Regulatory Commission (NRC) completed a Problem Identification and Resolution biennial inspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3. The enclosed report documents the inspection results, which were discussed on November 9, 2012, with Dwight Mims, Senior Vice President, Nuclear Regulatory and Oversight, and other members of your staff.
The inspection examined activities conducted under your license as they relate to identification and resolution of problems, safety and compliance with the Commissions rules and regulations and with the conditions of your operating license. The team reviewed selected procedures and records, observed activities, and interviewed personnel. The team also interviewed a representative sample of personnel regarding the condition of your safety conscious work environment.
Based on the inspection sample, the inspection team concluded that the implementation of the corrective action program and overall performance related to identifying, evaluating, and resolving problems at Palo Verde was effective. Licensee identified problems were entered into the corrective action program at a low threshold. Problems were effectively prioritized and evaluated commensurate with the safety significance of the problems and corrective actions were effectively implemented in a timely manner. Corrective actions were effectively implemented in a timely manner commensurate with their importance to safety and addressed the identified causes of problems. Lessons learned from industry operating experience were effectively reviewed and applied when appropriate. Audits and self-assessments were generally used to identify problems and appropriate actions. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web-site at www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Ray L. Kellar, P.E., Chief Technical Support Branch Division of Reactor Safety Dockets: 05000528, 05000529, 05000530 Licenses: NPF-41, NPF-51, NPF-74 Enclosure: Inspection Report 05000528/2012007, 05000529/2012007 and 05000530/2012007 w/Attachments:
1. Supplemental Information 2. Information Request - August 22, 2012
SUMMARY
OF ISSUES
IR 05000528/2012007, 05000529/2012007, 05000530/2012007; October 22 through November 9, 2012; Palo Verde Nuclear Generating Station; Biennial Baseline Inspection of the Identification and Resolution of Problems.
The inspection was conducted by a regional senior reactor inspector, two regional reactor inspectors, and a resident inspector. No findings of significance were identified during this inspection.
Identification and Resolution of Problems The team reviewed approximately 300 condition reports, work orders, engineering evaluations, root and apparent cause evaluations, and other supporting documentation to determine if problems were being properly identified, characterized, and entered into the corrective action program for evaluation and resolution. The team reviewed a sample of system health reports, self-assessments, trending reports and metrics, and various other documents related to the corrective action program. The team found that licensee was generally effective at identifying problems and putting them into the corrective action program; however, there were a few instances identified during the assessment period where the licensee had missed identification of problems. The licensee was also generally effective in prioritizing the extent to which individual problems would be evaluated and in establishing schedules for implementing corrective actions notwithstanding three instances identified early in the assessment period where the licensee failed to perform prompt operability assessments. These failures do not appear to represent current performance. The licensees corrective action process was generally found to be effective in documenting and tracking problems to resolution although it was somewhat disjointed due to having a separate processes for documenting the identification, evaluation, and correcting problems. Corrective actions were generally implemented in a timely manner.
Licensee audits and assessments were found to be effective and highlighted areas of ineffective corrective actions similar to weaknesses identified by the team. Operating experience usage was also found to be effective. Self assessment results adequately identified problems. On the basis of focus group interviews and an independent safety culture survey, workers at the site felt free to raise safety concerns using the corrective action program, their management and chain of command, and to the NRC without fear of retaliation.
NRC-Identified and Self-Revealing Findings
No findings of significance were identified.
Licensee-Identified Violations
None
REPORT DETAILS
OTHER ACTIVITIES (OA)
4OA2 Problem Identification and Resolution
The team based the following conclusions on a sample of the corrective action documents that were initiated in the assessment period, which ranged from December 17, 2010, to the end of the on-site portion of the inspection on November 9, 2012.
.1 Assessment of the Corrective Action Program Effectiveness
a. Inspection Scope
The team reviewed approximately 300 Palo Verde Action Requests (PVARs), Condition Report/Disposition Requests (CRDRs), and Condition Report Action Items (CRAIs) and associated root cause, apparent cause, and direct cause evaluations, that had been issued between December 2010 and November 2012 to determine if problems were being properly identified, characterized, and entered into the corrective action program for evaluation and resolution. Concerns enter the licensees corrective action process as a PVAR which include all manner of issues and concerns, both safety-related and non-safety-related. During the assessment period, the licensee initiated 50,492 PVARs of which 11,314 or approximately 22% were classified as conditions adverse to quality.
The team reviewed a sample of system health reports, operability determinations, self-assessments, trending reports and metrics, and various other documents related to the corrective action program. The team evaluated the licensees efforts in establishing the scope of problems by reviewing selected logs, work requests, self-assessments results, audits, system health reports, action plans, and results from surveillance tests and preventive maintenance tasks. The team reviewed work requests and attended the licensees daily management review committee meetings - Condition Review Group (CRG) to assess the reporting threshold, prioritization efforts, and significance determination process, as well as observing the interfaces with the operability assessment and work control processes when applicable. The teams review included verifying the licensee considered the full extent of cause and extent of condition for problems, as well as how the licensee assessed generic implications and previous occurrences. The team assessed the timeliness and effectiveness of corrective actions, completed or planned, and looked for additional examples of similar problems. The team conducted interviews with plant personnel to identify other processes that may exist where problems may be identified and addressed outside the corrective action program.
The team also reviewed corrective action documents that addressed past NRC-identified violations to ensure that the corrective action addressed the issues as described in the inspection reports. The inspectors reviewed a sample of corrective actions closed to other corrective action documents to ensure that corrective actions were still appropriate and timely.
The team considered risk insights from both the NRCs and Palo Verdes risk assessments to focus the sample selection and plant tours on risk significant systems and components. The team selected the following risk significant systems: essential chill water, emergency diesel generator, safety and non-safety related inverters, and emergency core cooling pumps. The samples reviewed by the team focused on, but were not limited to, these systems. The team also expanded their review to include five years of evaluations involving the essential chill water system to determine whether problems were being effectively addressed. The team conducted a walkdown of this system to assess whether problems were identified and entered into the corrective action program.
b.
Assessments 1. Assessment - Effectiveness of Problem Identification The team concluded that the licensee identified issues and adverse conditions in accordance with the licensees corrective action program guidance and NRC requirements. The team noted that licensee personnel had a very low threshold for entering issues into PVARs (corrective action program) as evidenced by the more than 50 thousand PVARs issued during the two year review cycle. While there were three findings identified during the assessment period and two similar observations during the inspection in which the licensee had failed to identify conditions adverse to quality, the team concluded that the license was generally identifying problems at a low threshold.
These examples include:
- The licensee failed to identify a weakness during a biennial exercise (non-cited violation 2011003-04) in that the Technical Support Center did not understand the radiological release path and had developed ineffective mitigation strategies based on their inaccurate understanding. The licensee failed to identify a performance issue completely and accurately.
- The licensee failed to promptly identify and correct a condition adverse to quality (non-cited violation 2012002-05) in that more frequent biocide additions to the essential chilled water systems resulted in significant bacterial off gassing and voiding in the systems in all three units. The licensee failed to trend and assess information from the corrective action program and other assessments to identify this common cause problem.
- The licensee failed to follow corrective action program procedures (non-cited violation 2012002-01) to generate a Palo Verde Action Request when significant delays in completing maintenance on safety related components occurred.
- Two additional examples were identified during the inspection where the licensee failed to initiate a PVAR contrary to Procedure 01PR0AP04. These examples included failing to document the failure of an essential chill water chiller to load in October 2012 and failing to document problems identified with the bench setup for calibrating an inverter circuit board. In both cases, the licensee had been investigating other conditions adverse to quality. Once identified by the NRC, the
licensee documented these in their corrective action program as PVARS. These examples were considered to be minor because the failures to document these conditions did not result in any safety consequences. These failures to comply with Procedure 01PR0AP04 constitute minor violations that are not subject to enforcement action in accordance with the NRCs Enforcement Policy.
2. Assessment - Effectiveness of Prioritization and Evaluation of Issues The team concluded that the licensee was generally effective in the prioritization and evaluation of conditions adverse to quality during this assessment period. The team reviewed corrective action documents that involved operability reviews to assess the quality, timeliness, and prioritization of operability assessments. While the team noted three findings had been documented during the assessment period where the licensee had failed to promptly evaluate the operability of safety-related systems with identified deficiencies, these findings were identified during the first part of the assessment period and no additional findings were identified during the inspection. The team concluded that operability assessments were generally completed in an appropriate manner.
- The licensee failed to promptly evaluate the operability of the essential chill water system (non-cited violation 2011004-02) after identifying gas voids in the system.
- The licensee failed to promptly evaluate the operability of the diesel fuel oil transfer system (non-cited violation 2011003-02) following identification that the system leakage test method did not conform to ASME Code Section XI testing requirements.
- The licensee failed to promptly evaluate the operability of safety-related structures, systems and components (non-cited violation 2011002-02) following discovery of a degraded and nonconforming condition associated with a potential manufacturing defect in K-600S 480 VAC Class 1E circuit breakers.
The team monitored the licensees action request review committee and the corrective action review board. The team found that the licensee was effectively reviewing and prioritizing conditions adverse to quality.
3. Assessment - Effectiveness of Corrective Action Program
Overall, the team concluded that the licensee had an effective corrective action program where conditions adverse to quality were promptly identified, prioritized, evaluated, and corrected in a timely manner commensurate to safety significance. The team noted that the licensees corrective action process was somewhat disjointed due to having a separate processes and numbering systems for documenting the identification, the evaluation, and the correction of problems. The team noted that the licensee had made notable progress in reducing the backlog of issues and reducing the average time to address condition reports.
Over the assessment period, several findings were documented associated with the adequacy of the corrective action program although the team concluded that this was not indicative of program weaknesses. Some of the findings include:
- The licensee failed to promptly correct a condition adverse to quality (non-cited violation 2010005-04) in that between July 2008 and November 2010, corrective actions for high vibrations in the Unit 3 essential cooling water system train A room cooler blower failed to promptly address the incorrect shaft dimensions at the bearing shaft interface.
- The licensee failed to perform functionality assessment for safety-related buildings (non-cited violation 2012002-07) after identifying a potential for insufficient drainage for safety related building roofs and failed to assess the non-conforming condition with the current licensing basis.
- The licensee failed to promptly identify and correct a condition adverse to quality (non-cited violation 2011004-03) associated with an inadequate plant operating procedure used to operate the essential chiller at low load conditions. As a result, the chiller was declared inoperable six times between July 27, 2011, and September 5, 2011.
- The licensee failed to correct and prevent recurrence of a significant condition adverse to quality (violation 2010008-01) associated with the Emergency Diesel Generator Fuel Oil Transfer Pumps. Specifically, from April 2009 to September 2010, the licensee failed to correct a water intrusion path to the motor termination box for the Unit 2 emergency diesel generator fuel oil transfer pumps, resulting in degraded electrical connections.
.2 Assessment of the Use of Operating Experience
a. Inspection Scope
The team examined the licensee's program for reviewing industry operating experience, including reviewing the governing procedure and self assessments. A sample size of 12 operating experience notifications that had been issued during the assessment period were reviewed to assess whether the licensee had appropriately evaluated the notification for relevance to the facility. The team then examined whether the licensee had entered those items into their corrective action program and assigned actions to address the issues. The team reviewed a sample of root cause evaluations and corrective action documents to verify if the licensee had appropriately included industry-operating experience.
b. Assessment Overall, the team determined that the licensee was adequately evaluating industry operating experience for relevance to the facility. The licensee had generally entered applicable items in the corrective action program in accordance with station procedures.
However, during the assessment period, findings were documented associated with the licensees inadequate incorporation of operating experience (both internal and external).
These include:
- Failure of reactor coolant pump motor junction box (finding 2011005-03) due to inadequate work instructions. As a result, water leakage from an open nuclear
cooling system vent valve entered the enclosure, resulting in a ground fault on the energized electrical conductors, explosion, and subsequent declaration of an Unusual Event. While the licensee had previous experience with water penetration in electrical enclosures following maintenance, this operating experience had not been incorporated into revised work instructions.
- The licensee failed to promptly identify and correct a condition adverse to quality (non-cited violation 2011004-03) associated with an inadequate plant operating procedure used to operate the essential chiller at low load conditions. As a result, the chiller was declared inoperable six times between July 27, 2011, and September 5, 2011, due to low oil level caused by oil migration into the refrigerant, a known phenomenon that occurs during low load conditions.
- The licensee failed to adequately classify and evaluate conditions adverse to quality (non-cited violation 2011004-01) involving the installation of an automatic voltage regulator card in an emergency diesel generator that was potentially defective and had been previously reported under Part 21.
.3 Assessment of Self-Assessments and Audits
a. Inspection Scope
The team reviewed a sample of five licensee self-assessments, surveillances, and audits to assess whether the licensee was regularly identifying performance trends and effectively addressing them. The team reviewed audit reports to assess the effectiveness of assessments in specific areas. The team evaluated the use of self- and third party assessments, the role of the quality assurance department, and the role of the performance improvement group related to licensee performance. The specific self-assessment documents reviewed are listed in the Attachment.
b. Assessment The team found that the internal self-assessments and audits were generally thorough, detailed, indepth and critical. The team found that the effectiveness of corrective actions initiated to address self-assessment findings was not always effective. The licensee acknowledged that they had also identified this and had initiated actions to perform effectiveness reviews for self-assessment corrective actions earlier this year. The team acknowledged this and concluded that it there was insufficient data to assess whether the licensees effectiveness reviews would reduce the number of repeat findings documented.
.4 Assessment of Safety-Conscious Work Environment
a. Inspection Scope
The team conducted five focus group interviews with typically 8 - 10 individuals per group. The focus groups consisted of workers from operations, security, engineering, instrumentation and controls, and health physics. Individuals were randomly selected to assure representative outcomes for the interviews. The inspection team also conducted
individual interviews. The interviewees represented various functional organizations and ranged across staff, and supervisor levels. The team conducted these interviews to assess whether conditions existed that would challenge the establishment of a safety conscious work environment at Palo Verde Nuclear Generating Station. The team also reviewed the most recent safety culture survey results conducted by an independent organization in 2010.
b. Assessment Based upon the results of these interviews and survey results, the team concluded that the licensee had established a safety-conscious work environment where individuals felt free to raise safety concerns both to the licensee and the NRC without fear of retaliation.
None of the individuals interviewed knew of anyone who had suffered retaliation for having raised safety concerns and all indicated that they felt comfortable raising safety concern to their supervisor, the corrective actions program, to the employee concerns program, to the NRC, or using their chain of command although they indicated that they had not had the need to raise concerns beyond their supervisor or the corrective action program. Responses to questions and topics during the focus group sessions did not reveal any sense that safety was not the highest priority. All organizations indicated that the work environment had changes significantly over the past several years.
.5 Specific Issues Identified During This Inspection
None
4OA6 Meetings
Exit Meeting Summary
On November 9, 2012, the team presented the inspection results to Dwight Mims, Senior Vice President, Nuclear Regulatory and Oversight, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.
4OA7 Licensee-Identified Violations
None
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
A. Amoroso Electrical Instrumentation and Controls Section Leader
G. Andrews Assistant Plant Manager
G. Archambault Information Technology Supervisor
- R. Barnes Director, Nuclear Regulatory Affairs
R. Bramlett Security Programs
M. Brannin ISI Program Manager
C. Coles Nuclear Security Division
D. Coxon Operations Department Leader
B. Doyle Senior Engineer
- E. Dutton Director, Nuclear Assurance
D. Elkinton Regulatory Affairs Consult Sr.
M. Fladager Department Leader Work Management
C. Goff Nuclear Section Leader
D. Harrison Maintenance Section Leader
A. Hartwig Electrical I&C Design Engineering Department Leader
M. Heider Department Lead PM Program
B. Kershaw Engineering Cause Analysis
K. Keyes Coordinator Senior
D. Leech Section Leader Nuclear Assurance
B. Logue IT Manager
N. Lossing Electrical Engineering Section Leader
R. Maner Senior Consulting Instrumentation and Controls Engineer
M. McGhee Manager Operations Support
L. McKinney Security Operations
F. Oreshack Regulatory Affairs Consultant
J. Rodriguez Engineer III
L. Sewell Site Rigging Material Coordinator
J. Skrtch Department Leader Work Management
C. Smith Auxiliary Equipment Operator
J. Sontchi Department Leader Nuclear Training
B. Thiele Program Engineering Department Leader
T. Tidyman PM Program Project Manager
- D. Wheeler Acting Director, Performance Improvement
R. Wilferd Department Leader Nuclear Engineering
G. Zuniga Maintenance Team Leader
NRC Personnel
T. Brown Senior Resident Inspector
M. Baquera Resident Inspector
Attachment 1
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
None
Opened and Closed
None
Closed
None
Discussed
None