ML080320562
| ML080320562 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 02/01/2008 |
| From: | Collins E Region 4 Administrator |
| To: | Edington R Arizona Public Service Co |
| References | |
| IR-07-012 | |
| Download: ML080320562 (162) | |
See also: IR 05000528/2007012
Text
UNITED STATES
NU CLEAR REGU LATOR Y C O M M I SSI O N
R E GI ON I V
611 R YAN PLAZA D R I V E, SU I TE 400
AR LIN GTON , TEXAS 76011-4005
February 1, 2008
Randall K. Edington,
Executive Vice President Nuclear
and Chief Nuclear Officer
Arizona Public Service Company
P.O. Box 52034
Phoenix, AZ 85072-2034
SUBJECT: PALO VERDE NUCLEAR GENERATING STATION - NRC SUPPLEMENTAL 95003
INSPECTION REPORT 05000528/2007012, 05000529/2007012, AND
Dear Mr. Edington:
On December 19, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection
at your Palo Verde Nuclear Generating Station (PVNGS), Units 1, 2, and 3, facility. The inspection
was conducted in accordance with the guidance contained in NRC Inspection Manual Chapter
(IMC) 0305, Operating Reactor Assessment Program and Inspection Procedure (IP) 95003,
"Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple Degraded Cornerstones,
Multiple Yellow Inputs, or One Red Input," and was performed in response to your facility's
designation as having a Repetitive Degraded Cornerstone, as defined by the NRC's reactor
oversight process. The enclosed report documents the inspection findings, which were discussed
on December 19, 2007, with you and other members of your staff.
In our Annual Assessment Letter dated March 2, 2007, we informed you that PVNGS Unit 3 was
placed in the Multiple/Repetitive Degraded Cornerstone Column (Column IV) of the NRC's Action
Matrix. In accordance with IMC 0305, this decision was made on the basis of two separate safety
significant inspection findings (one Yellow and one White) in the Mitigating Systems cornerstone.
The Yellow finding, open since the fourth quarter 2004, involved a significant section of containment
sump safety injection piping that was void of water at all three PVNGS units. The White finding,
open since the fourth quarter 2006, involved two failures of the Unit 3, Train A emergency diesel
generator. This inspection evaluated the extent of condition of the performance issues, and the
adequacy of the safety culture at PVNGS.
The results of our inspection indicate that your facility is being operated safely. However, the team
identified numerous performance deficiencies that were additional examples of the organizational
and programmatic weaknesses that the NRC had previously identified. Despite previous attempts,
PVNGS had been unsuccessful in implementing changes that result in sustained improvement in
safety system reliability, human performance, problem identification and resolution, the quality of
engineering work products, and oversight of station activities by operations personnel. The
inspection and recent PVNGS safety culture self-assessment activities also identified degradations
in the safety culture of the facility. The team identified weaknesses in organizational characteristics
and attitudes associated with ten of the NRCs thirteen safety culture components. The
Arizona Public Service Company -2-
weaknesses were apparent across several functional groups at the site. This is of concern because
it indicates that, as an overriding priority, nuclear plant safety issues had not always received the
attention warranted by their significance.
The team validated that the root and contributing causes for the performance deficiencies at Palo
Verde included: (1) leaders did not establish, communicate, and enforce standards and
expectations for performance or hold individuals accountable to those standards; (2) the corrective
action program, operating experience, self assessments, and benchmarking did not drive individual
and station performance improvement; (3) responsibility, accountability, and authority for nuclear
safety were not well defined or understood; (4) individual behaviors that demonstrate nuclear safety
principles were not consistently applied; (5) management was not receptive to organizational issues
identified during investigations; (6) change management activities did not anticipate unintended
consequences and did not clearly define and communicate changes to station personnel; and (7)
oversight groups did not provide specific and meaningful interventions to correct declining
performance.
As stated in the June 21, 2007, Confirmatory Action Letter, and subsequently revised with NRC
approval by your letter dated November 28, 2007, you submitted an improvement plan to the NRC
on December 31, 2007. Following the NRCs review of the plan, we will issue a revised
Confirmatory Action Letter including the minimum actions believed necessary to improve
performance and sustain performance improvement. The NRC will also conduct periodic
performance improvement public meetings and inspections until PVNGS demonstrates sustained
performance improvement.
The inspection examined activities conducted under your licenses as they relate to safety and
compliance with the Commission's rules and regulations and with the conditions of your licenses.
The team reviewed selected procedures and records, observed activities, and interviewed
personnel. A listing of the documents requested by the team for review during the inspection is
available electronically in the NRCs document system (ADAMS) as ML080250295.
The report documents numerous performance deficiencies resulting in 18 NRC identified findings.
The findings represent performance deficiencies in all 7 inspection cornerstones and 10 of the 13
safety culture components. Sixteen of these findings were evaluated under the significance
determination process as having very low safety significance (Green). One finding involving the
failure to update the Final Safety Analysis Report impacted the regulatory process and was
assessed in accordance with the NRC Enforcement Policy. Because of the very low safety
significance of these violations and because they were entered into your corrective action program,
the NRC is treating these findings as noncited violations consistent with Section VI.A of the NRC
Enforcement Policy. The significance of one finding (failure to implement corrective actions for a
risk significant planning standard in the emergency preparedness cornerstone) is being separately
evaluated by the NRC. Additionally, licensee-identified violations which were determined to be of
very low safety significance are listed in this report. If you contest these noncited violations, you
should provide a response within 30 days of the date of this inspection report, with the basis for
your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk,
Washington DC 20555-0001; with copies to the Regional Administrator, U.S. Nuclear Regulatory
Commission Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas 76011-4005; the
Arizona Public Service Company -3-
Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington DC 20555-
0001; and the NRC Resident Inspector at the Palo Verde Nuclear Generating Station, Units 1, 2,
and 3, facility.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its
enclosure, and your response (if any) will be made available electronically for public inspection
in the NRC Public Document Room or from the Publicly Available Records (PARS) component
of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at
http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Elmo E. Collins
Regional Administrator
Dockets: 50-528
50-529
50-530
Licenses: NPF-41
Enclosure:
NRC Inspection Report 05000528/2007012, 05000529/2007012, and 05000530/2007012
w/Attachment: Supplemental Information
cc w/Enclosure:
Steve Olea
Arizona Corporation Commission
1200 W. Washington Street
Phoenix, AZ 85007
Douglas K. Porter, Senior Counsel
Southern California Edison Company
Law Department, Generation Resources
P.O. Box 800
Rosemead, CA 91770
Chairman
Maricopa County Board of Supervisors
301 W. Jefferson, 10th Floor
Phoenix, AZ 85003
Arizona Public Service Company -4-
Aubrey V. Godwin, Director
Arizona Radiation Regulatory Agency
4814 South 40 Street
Phoenix, AZ 85040
Scott Bauer, Director
Regulatory Affairs
Palo Verde Nuclear Generating Station
Mail Station 7636
P.O. Box 52034
Phoenix, AZ 85072-2034
Mr. Dwight C. Mims
Vice President, Regulatory Affairs and
Performance Improvement
Palo Verde Nuclear Generating Station
Mail Station 7636
P.O. Box 52034
Phoenix, AZ 85072-2034
Jeffrey T. Weikert
Assistant General Counsel
El Paso Electric Company
Mail Location 167
123 W. Mills
El Paso, TX 79901
Eric J. Tharp
Director of Generation
Los Angeles Department of Water & Power
Southern California Public Power Authority
P.O. Box 51111, Room 1255
Los Angeles, CA 90051-5700
John Taylor
Public Service Company of New Mexico
2401 Aztec NE, MS Z110
Albuquerque, NM 87107-4224
Geoffrey M. Cook
Southern California Edison Company
5000 Pacific Coast Hwy, Bldg. D21
San Clemente, CA 92672
Robert Henry
Salt River Project
6504 East Thomas Road
Scottsdale, AZ 85251
Arizona Public Service Company -5-
Brian Almon
Public Utility Commission
William B. Travis Building
P.O. Box 13326
1701 North Congress Avenue
Austin, TX 78701-3326
Karen O' Regan
Environmental Program Manager
City of Phoenix
Office of Environmental Programs
200 West Washington Street
Phoenix, AZ 85003
Matthew Benac
Assistant Vice President
Nuclear & Generation Services
El Paso Electric Company
340 East Palm Lane, Suite 310
Phoenix, AZ 85004
Chief, Radiological Emergency Preparedness Section
National Preparedness Directorate
Technological Hazards Division
Department of Homeland Security
1111 Broadway, Suite 1200
Oakland, CA 94607-4052
Arizona Public Service Company -6-
Electronic distribution by RIV:
Regional Administrator (EEC)
DRP Director (DDC)
DRS Director (RJC1)
DRS Deputy Director (ACC)
Senior Resident Inspector (GXW2)
Branch Chief, DRP/D (TWP)
Senior Project Engineer, DRP/D (GEW)
Team Leader, DRP/TSS (CJP)
RITS Coordinator (MSH3)
V. Dricks, PAO (VLD)
D. Pelton, OEDO RIV Coordinator (DLP1)
ROPreports
SUNSI Review Completed: TWP ADAMS: X Yes No Initials: TWP
X Publicly Available Non-Publicly Available Sensitive X Non-Sensitive
R:\_REACTORS\_PV\2007\PV2007-012RP-TWP.doc
RI:SRA RIV:RI RIII:RE RI:SRI RII:SHP RIII:RI
CGCahill MPCatts BJose SMSchneider HJGepford RLSmith
E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/
12/26/07 12/19/07 12/20/07 12/19/07 01/03/08 12/20/07
RIII:RI RIV:SRI RIV:SRI RII:SRI RIII:PE RIV:RE
MAWilk JFDrake SDCochrum SAWalker ARBarker MRBloodgood
E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/
12/18/07 01/07/08 12/20/07 12/19/07 12/21/07 12/26/07
NRR:SHFA RI:OE NRR:HFS OE:ES NRR:SHFA RIV:SRI
VBarnes BCHaagensen MJKeefe JCai DRDesaulniers CCOsterholtz
E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/ E-TWP /RA/
12/24/07 12/27/07 12/20/07 12/20/07 01/04/08 12/24/07
RIV:EPI:DRS NSIR NSIR RIV:SPE:DRP/D RIV:C:DRP/D RIV:DD:DRP
PJElkmann REKahler KWilliams GEWerner TWPruett AVegel
E-TWP /RA/ E-TWP /RA/ T-TWP /RA/ E-TWP /RA/ /RA/ /RA/
01/09/08 01/09/08 01/09/08 12/26/07 01/25/08 01/26/08
RIV:D:DRP RIV:RA
DDChamberlain EECollins
/RA/ /RA/
01/25/08 02/01/08
OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax
U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Dockets: 50-528, 50-529, 50-530
Licenses: NPF-41, NPF-51, NPF-74
Report: 05000528/2007012, 05000529/2007012, 05000530/2007012
Licensee: Arizona Public Service Company
Facility: Palo Verde Nuclear Generating Station, Units 1, 2, and 3
Location: 5951 S. Wintersburg Road
Tonopah, Arizona
Dates: April 3 through December 19, 2007
Team Members: T. Pruett, IP 95003 Team Leader; Chief, Project Branch D
Division of Reactor Projects, Region IV
G. Werner, IP 95003 Assistant Team Leader; Senior Project Engineer,
Region IV
S. Gillum, Secretary, Region IV
Substantive Crosscutting Issues Group
M. Schneider, IP 95003 Group Leader; Senior Resident Inspector,
Region I
H. Gepford, Senior Health Physicist, Region II
M. Wilk, Resident Inspector, Region III
R. Smith, Resident Inspector, Region III
J. Drake, Senior Reactor Inspector, Region IV
R. Kahler, Team Leader, Office of Nuclear Security and Incident
Response
P. Elkmann, Emergency Preparedness Inspector, Region IV
Maintenance and Testing Group
S. Cochrum, IP 95003 Group Leader; Senior Resident Inspector,
Region IV
A. Barker, Project Engineer, Region III
S. Walker, Senior Reactor Inspector, Region II
M. Bloodgood, Reactor Engineer, Region IV
Engineering Group
C. Cahill, IP 95003 Group Leader; Senior Reactor Analyst, Region I
B. Jose, Reactor Engineer, Region III
M. Catts, Resident Inspector, Region IV
M. Villaran, Brookhaven National Laboratory, Contractor
-1- Enclosure
Safety Culture Group
V. Barnes, IP 95003 Group Leader; Senior Human Factors Analyst,
Office of Nuclear Regulatory Research
V. Mehrhoff, Secretary, Las Vegas Site Office
J. Cai, Enforcement Specialist, Office of Enforcement
M. Keefe, Human Factors Specialist, Office of New Reactors
C. Osterholtz, Senior Resident Inspector, Region IV
B. Haagensen, Operations Engineer, Region I
Accompanied By: D. Desaulniers, Senior Human Factors Specialist, Office of Nuclear
Reactor Regulation
K. Martin, Human Factors Engineer, Office of Nuclear Reactor
Regulation
M. Barrientos, Nuclear Safety Council, Spain
Approved By: Dwight Chamberlain, Director
Division of Reactor Projects
-2- Enclosure
CONTENTS
EXECUTIVE SUMMARY ............................................................................................................. 7
SUMMARY OF FINDINGS ......................................................................................................... 10
REPORT DETAILS ..................................................................................................................... 21
1 PERFORMANCE HISTORY ............................................................................................... 21
2 SITE INTEGRATED BUSINESS PLAN (SIBP) AND SITE INTEGRATED IMPROVEMENT
PLAN (SIIP) ........................................................................................................................ 23
3 COLLECTIVE SIGNIFICANCE REVIEW............................................................................. 29
4 NRC METHODOLOGY AND DIAGNOSTIC ASSESSMENT .............................................. 32
5 REACTOR SAFETY STRATEGIC PERFORMANCE ARENA ............................................ 34
5.1 Licensee Controls for Identifying, Assessing, and Correcting Performance
Deficiencies.............................................................................................................. 34
b.1 Failure to Implement Operability Determination Process for Bechtel
Nonconformance Reports
b.2 Failure to Implement Operability Determination Process for Action
Tracking System (ACT)
b.3 Failure to Implement Operability Determination Process for Spray
Pond Missile Hazards
b.4 Failure to Evaluate Abnormally High Lead Levels in Low Pressure
Safety Injection Pump Bearing Oil
b.5 Failure to Implement the Operability Determination Process on
Unit 2 Essential Cooling Water Heat Exchanger 'A' Sleeve Adhesive
b.6 Failure to Implement the Operability Determination Process on
the Unit 2 Essential Cooling Water Heat Exchanger A Tube Leak
b.7 Observations and Minor Noncited Violations Involving Licensee
Controls for Identifying, Assessing, and Correcting Performance
Deficiencies
b.7.1 Corrective Action Program Implementation
b.7.2 Problem Identification and Resolution Root Cause Report
b.7.3 Action Request Review Committee
b.7.4 Backlog Review
b.7.5 Self Assessments
5.2 Design52
b.1 Failure to Implement Adequate Design Controls for Condensate
Storage Tank Temperature
b.2 Inadequate Installation of Fire Sprinklers
b.3 Failure to Enter Environmental Qualification Self Assessment
Deficiencies into the Corrective Action Program
-3- Enclosure
b.4 Failure to Implement Corrective Actions for Operating Experience
Involving the Turbine Driven Auxiliary Feedwater Pump Trip and Throttle
Valve
b.5 Observations and Minor Noncited Violations Involving Design
b.5.1 High Pressure Safety Injection Pump Bearing Modification
5.3 Human Performance ................................................................................................ 61
b.1 Observations and Minor Noncited Violations Involving Human
Performance
b.1.1 Human Performance Root Cause Report
b.1.2 Main Control Room Observations
5.4 Procedure Quality .................................................................................................... 65
b.1 Observations and Minor Noncited Violations Involving Procedure Quality
b.1.1 Procedure Issues
5.5 Equipment Performance .......................................................................................... 66
b.1 Failure to Evaluate Performance Monitoring Criteria for Auxiliary
Feedwater System
b.2 Failure to Control Nonconforming Target Rock Reed Switches
b.3 Failure to Meet the Requirements of Technical Specifications
Surveillance Requirement 3.6.6.6
b.4 Failure to Meet the Requirements of Technical Specifications
Surveillance Requirement 3.0.3
b.5 Untimely Corrective Actions for Submerged Safety Related Cables
b.6 Failure to Properly Evaluate the Extent of Condition of 4160V and 480V
Motor Issues
b.7 Observations and Minor Noncited Violations Involving Equipment
Performance
b.7.1 Environmental Qualification Program
5.6 Configuration Control ............................................................................................... 80
5.6.1 Effectiveness of Corrective Actions ......................................................... 80
b.1 Failure to Implement Corrective Actions for Borg-Warner Check
Valves
5.6.2 Selected System Walkdown .................................................................... 83
b.1 Failure to Maintain Control of Transient Combustibles
b.2 Failure to Install Emergency Lighting in Containment
b.3 Incorrect Installation of Temporary Shielding
b.4 Observations and Minor Violations Involving Selected System
Walkdown
b.4.1 Inadequate Seismic Scaffolding Procedures
5.6.3 Work Control Process.............................................................................. 88
b.1 Failure to Adequately Manage Risk for Switchyard Activities
b.2 Observations and Minor Noncited Violations Involving Work
Control Process
b.2.1 Failure to Properly Document Temporary Modifications
b.2.2 Inadequate Shutdown Risk Assessments
-4- Enclosure
5.6.4 Control of Fission Barriers ....................................................................... 94
b.1 Incorrect Rigging for Personnel Air Lock Door
5.6.5 Review of Individual Plant Examination................................................... 96
5.6.6 Human Performance................................................................................ 96
b. Observations and Minor Noncited Violations Involving Human
Performance
b.1 Inadequate Procedure for Adjustment of Polar Crane Switch
5.6.7 Design .97
b.1 Failure to Maintain Configuration of Pressurizer Instrument
Condensing Pot Support Brackets
b.2 Observations and Minor Violations Involving Design
b.2.1 Lack of Design Control for Breaker Modification
5.6.8 Problem Identification and Resolution .................................................... .99
b.1 Failure to Evaluate Adverse Condition for the Emergency Diesel
Generators
b.2 Failure to Identify and Correct a Non-Conforming Condition
of Post Accident Monitoring Instrumentation Recorders
5.6.9 Equipment Performance ........................................................................ 104
b.1 Failure to Establish Maintenance Rule Goals for the Safety
Injection System
5.7 Emergency Response and Preparedness ............................................................ 105
b.1 Failure to Correct Weakness Associated with Risk Signifivant Planning
Standard 10 CFR 50.47(b)(4)
b.2 Inability to Implement Emergency Action Levels (EALs)
b.3 Observations and Minor Noncited Violations Involving Emergency
Response and Preparedness
b.3.1 Failure to Notify Offsite Agencies of Emergency Action Level
Changes
b.3.2 Failure to Train Emergency Planners
6 RADIATION SAFETY STRATEGIC PERFORMANCE ARENA ........................................ 114
6.1 Occupational Radiation Safety .......................................................................... 114
b.1 Inadequate Briefings on Radiological Conditions
b.2 Observations and Minor Violations Involving Occupational Radiation
Safety
b.2.1 Failure to Conduct Appropriate Radiological Surveys
6.2 Public Radiation Safety ..................................................................................... 116
b.1 Failure to Periodically Update the Final Safety Analysis Report
7 SAFEGUARDS STRATEGIC PERFORMANCE AREA..................................................... 118
-5- Enclosure
8 SAFETY CULTURE ........................................................................................................... 118
8.1 Evaluation of Licensees Independent Safety Culture Assessment .................. 118
8.2 NRC Independent Safety Culture Assessment ................................................. 124
b.1 Decision-Making
b.2 Organizational Change Management
b.3 Resources
b.4 Continuous Learning Environment
b.5 Accountability
b.6 Corrective Action Program
b.7 Work Practices
b.8 Work Control
b.9 Operating Experience
b.10 Self and Independent Assessments
b.11 Environment for Raising Concerns
b.12 Preventing, Detecting, and Mitigating Perceptions of Retaliation
b.13 Safety Policies
9 REVIEW OF YELLOW FINDING - CONTAINMENT SUMP VOIDING .............................. 147
10 REVIEW OF WHITE FINDING - DIESEL GENERATOR K-1 RELAY FAILURE ............... 148
11 LICENSEE-IDENTIFIED VIOLATIONS ............................................................................. 150
12 MANAGEMENT MEETINGS ............................................................................................. 151
ATTACHMENT: SUPPLEMENTAL INFORMATION................................................................ A-1
KEY POINTS OF CONTACT .................................................................................................... A-1
ITEMS OPENED AND CLOSED .............................................................................................. A-2
LIST OF ACRONYMS USED.................................................................................................... A-3
-6- Enclosure
Executive Summary
Palo Verde performance had declined since 2003. The team determined that Palo Verde is
safe for continued operation even though several longstanding performance concerns were
identified.
The root and contributing causes associated with declining performance included: (1) leaders
did not establish, communicate, and enforce standards and expectations for performance or
hold individuals accountable to those standards; (2) the corrective action program, operating
experience, self assessments, and benchmarking did not drive individual and station
performance improvement; (3) responsibility, accountability, and authority for nuclear safety
were not well defined or understood; (4) Individual behaviors that demonstrate nuclear safety
principles were not consistently applied; (5) management was not receptive to organizational
issues identified during investigations; (6) change management activities did not anticipate
unintended consequences and did not clearly define and communicate changes to station
personnel; and (7) oversight groups did not provide specific and meaningful interventions to
correct declining performance.
Multiple substantive crosscutting aspects associated with problem identification and resolution
have existed since 2004. Corrective actions continue to remain ineffective in sustaining
improving performance as noted by effectiveness reviews, external industry reviews, and NRC
inspections. The team determined that personnel often recognized appropriate problem
identification and resolution fundamentals and behaviors when interviewed; however, this
knowledge and understanding of expectations was not consistently demonstrated.
A number of weak or non-existent operability evaluations of degraded conditions affecting
safety-related equipment were identified. A lack of understanding of the need to assess
operability for some conditions adverse to quality and a lack of knowledge or skills necessary to
conduct an operability assessment were apparent. This is a continuing weakness at Palo Verde
and impacts nuclear safety margins. The inability to consistently perform operability
determinations formed part of the NRCs basis for leaving open the Yellow finding involving
voiding of the emergency core cooling suction piping in all three units.
Operating experience opportunities were missed, ignored or misapplied. A lack of technical
rigor was cited in component design basis reviews and self assessments with respect to the
application of operating experience. The station did not appear to have a sense of the
importance and benefits of a strong operating experience program. The failure to incorporate
operating experience into daily activities is an open issue from the Yellow finding. In addition,
the failure to effectively utilize operating experience contributed to several performance
deficiencies identified by the team.
Self-assessments performed by Palo Verde personnel lacked depth and did not always
effectively specify or implement corrective actions. As a result, the self-assessment program
seldom resulted in improved organizational performance. Self-assessment corrective actions
were not always tracked nor were corrective action documents always written to track the
expected actions. The team noted self-assessments conducted by a mix of Palo Verde and
industry personnel led to more meaningful results.
Multiple substantive crosscutting aspects associated with human performance have existed
since 2004. Corrective actions continue to remain ineffective in sustaining consistent
performance improvement as noted by effectiveness reviews, external industry reviews, and
-7- Enclosure
NRC inspections. Human performance concerns observed during this inspection included
weaknesses in implementing the operability determination program, failures to follow
procedures, failures to implement human error prevention tools, inadequate procedures, and
inconsistent implementation of fundamental control room behaviors. The licensees
effectiveness review for human performance concluded that corrective actions were not well
defined and there were no actions for implementation, monitoring, reinforcement, adjustment, or
transfer of human performance changes. Furthermore, the corrective actions were either not
fully implemented or not implemented as intended.
Knowledge gaps and a lack of an effective emergency response training program were
identified by the licensee and team. Because of ineffective corrective actions for emergency
preparedness deficiencies, emergency action levels could not be implemented for a Site Area
Emergency, an Alert, and a Notice of Unusual Event. In response to the emergency
preparedness deficiencies, the licensee instituted actions to augment the emergency response
organization by assigning six managers, specially trained on emergency action level
classification, to the shift rotation.
The licensees third-party safety culture assessment applied a multi-method approach to
conduct the safety culture assessment, including a survey, behavioral observations, interviews,
and document reviews. Two third-party teams performed their assessment activities in parallel,
but compared, contrasted, and reconciled their findings to ensure they provided integrated
assessment results to the licensee. The NRC did note that the multiple methods approach
provided a comprehensive understanding of the onsite safety culture, whereas a stand-alone
survey would not have provided sufficient information. The result of the NRC teams evaluation
of site safety culture was consistent with the third-party assessment.
Site personnel described past decision making as being governed by the goals of reducing
costs in preparation for deregulation and cost containment, unless the decisions involved
meeting regulatory requirements or ensuring continued operations. The sites reengineering
effort in the early 1990s, focused on streamlining work processes, reducing staff size, reducing
operating and maintenance costs, and allocating decision-making authority to those closest to
the work. These cumulative reductions contributed to the increase in equipment failures, plant
events, and other performance problems at the site.
Past efforts to reduce staff through attrition and the increasing rate of retirements in the aging
workforce have contributed to a reduction in the availability of qualified personnel at the site.
The reductions had the effect of requiring licensed personnel to routinely work overtime. The
team also observed that senior maintenance and engineering department personnel were
retiring at an accelerating rate. The licensees ability to replace senior personnel and ensure
knowledge transfer has been limited by past weaknesses in recruiting and hiring efforts and
reductions in the number of training staff. Replacement of senior staff with inexperienced
personnel has increased human error rates and hampered improvement efforts.
The team identified weaknesses in organizational characteristics and attitudes associated with
10 of the NRC's 13 safety culture components, as detailed in Section 06.07 of Inspection
Manual Chapter 0305 "Operating Reactor Assessment Program. The most significant
weaknesses were identified in the safety culture components of accountability, the licensee's
corrective action program, decision-making, and work practices. The team noted that these
-8- Enclosure
weaknesses were widespread among functional groups across the organization. Organizational
characteristics and attitudes were adequate in the safety culture components of safety policies;
the environment for raising concerns; and preventing, detecting, and mitigating perceptions of
retaliation.
-9- Enclosure
SUMMARY OF FINDINGS
IR 05000528/2007012, 05000529/2007012, 05000530/2007012; 04/03/07 - 12/19/07; Palo
Verde Nuclear Generating Station, Units 1, 2, and 3; Inspection Procedure 95003,
Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple Degraded
Cornerstones, Multiple Yellow Inputs, or One Red Input.
This report covered a 9-month period of inspection by personnel in all four NRC Regional
Offices and from Headquarters, one contractor, and an observer from the Spanish Nuclear
Safety Council. The inspection identified numerous performance deficiencies that resulted in 15
noncited violations, 1 finding, 1 Severity Level IV violation, and 1 apparent violation with
significance to be determined. The significance of most findings is indicated by their color
(Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance
Determination Process." Findings for which the significance determination process does not
apply may be Green or be assigned a severity level after NRC management's review. The
NRCs program for overseeing the safe operation of commercial nuclear power reactors is
described in NUREG-1649, "Reactor Oversight Process," Revision 3, dated July 2000.
- Green. The team identified a noncited violation of Technical Specification 5.4.1.d for the
failure of fire protection personnel to follow Procedure 14DP-0FP33, "Control of Transient
Combustibles," Revision 15. Specifically, the team identified that on the 70 elevation of the
Auxiliary Building (Radiation Protection Remote Monitoring Station) and in the Unit 3
containment, there were transient combustibles being stored without the proper evaluation
and required permits. This issue was entered into the corrective action program as Palo
Verde Action Request 3071785.
The finding is considered more than minor because storing unanalyzed material could result
in the potential to exceed combustible limits and is associated with an increase in the
likelihood of an initiating event. Using Inspection Manual Chapter 0609, Significance
Determination Process, Appendix F, Fire Protection Significance Determination Process,
this issue affected the Fire Prevention and Administrative Controls Category. In this case
the stored materials required a permit per the licensees procedure; however, the area was
attended, fire detection and suppression was available, and the amounts did not exceed the
loading calculation to the point of changing the loading classification. Therefore, this finding
is considered of Low Degradation and had very low safety significance. The cause of this
finding has crosscutting aspects associated with work practices in the human performance
area because: (1) the licensee failed to communicate human error prevention techniques
such that work activities were performed safely (H.4.(a)), and (2) the licensee did not
effectively communicate expectations regarding procedural compliance (H.4.(b)). The cause
of this finding is also related to the safety culture component of accountability in that fire
protection personnel failed to demonstrate a proper safety focus and reinforce safety
principles among their peers (O.1.(c)). (Section 5.6.2.b.1)
- Green. The team identified a noncited violation of 10 CFR 50.65(a)(4) for the failure to
adequately assess the increase in risk and effectively implement risk mitigation actions for
maintenance activities in the switchyard. Specifically, the switchyard was not being
protected by controlling access and movement as required and the risk modeling did not
include all work being performed. The Unit 1 shift manager and the switchyard coordinator
- 10 - Enclosure
were unaware of the movement of multiple vehicles and pieces of equipment in or near
restricted areas and not all maintenance was included in the schedule provided to the
switchyard coordinator for risk review. This issue was entered into the licensees corrective
action program as Palo Verde Action Request 3078392.
This finding is greater than minor because the licensees risk assessment failed to consider
maintenance activities that could increase the likelihood of initiating events such as work in
the switchyard and failed to effectively manage compensatory measures. Inspection
Manual Chapter 0609, Significance Determination Process, Appendix K, Maintenance
Risk Assessment and Risk Management Significance Determination Process, was used to
assess the significance. Using data from the licensees probabilistic risk assessment, a
NRC Region IV senior reactor analyst calculated the risk deficit. Based on the magnitude of
the calculated risk deficit being less than 1E-6/year, this finding is determined to be of very
low safety significance. The cause of this finding has crosscutting aspects associated with
work control of the human performance area in that the licensee did not appropriately
coordinate switchyard activities incorporating risk insights (H.3.(a)) and did not communicate
with each other during activities in which coordination is necessary to assure plant and
human performance (H.3.(b)). (Section 5.6.3.b.1)
- Green. The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V,
"Instructions, Procedures, and Drawings," with eight examples for the failure of the licensee
to adequately evaluate degraded and unanalyzed conditions to support operability decision
making between May 2006 and October 26, 2007. The team noted a significant number of
weak or non-existent operability evaluations of degraded conditions affecting safety-related
equipment. There was a lack of understanding of the need to assess operability for some
conditions adverse to quality and a lack of knowledge or skills necessary to conduct quality
operability assessments. The examples of the violation involved two instances of conditions
adverse to quality documented in databases outside of the corrective action program,
missile hazards near the essential spray pond, two issues effecting essential cooling water
system heat exchangers, 480V and 4160V motor terminations, oil leaks on the emergency
diesel generators, and high lead content in a Unit 3 low pressure safety injection pump.
Each of the individual technical issues was entered into the licensees corrective action
program.
The examples associated with this finding are greater than minor because they were
associated with the mitigating systems cornerstone attribute of equipment performance and
affected the cornerstone objective of ensuring the availability and reliability of systems that
respond to initiating events to prevent undesirable consequences. Using the Inspection
Manual Chapter 0609, "Significance Determination Process," Phase 1 Worksheets, the
examples associated with this finding are determined to have very low safety significance
since they only affected the mitigating systems cornerstone and did not represent a loss of
system safety function. The causes of the examples of this finding have crosscutting
aspects associated with decision making of the human performance area in that operations
and engineering personnel: (1) did not make safety significant decisions using a systematic
process (H.1.(a)), and (2) failed to use conservative assumptions for operability decision-
making when evaluating degraded and nonconforming conditions (H.1.(b)). The causes of
the examples of this finding also have crosscutting aspects associated with evaluation and
corrective action of the problem identification and resolution area in that licensee personnel:
(1) did not assess conditions adverse to quality for impacts to the operability of safety-
- 11 - Enclosure
related equipment (P.1.(c), and (2) did not address safety issues in a timely manner P.1.(d)).
The causes of the examples of this finding also related to the safety culture component of
accountability in that workers and managers failed to demonstrate a proper safety focus and
reinforce safety principles (O.1.(b) and O.1.(c)). (Multiple Sections)
- Green. The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion
XVI, Corrective Action, with six examples for the failure of the licensee to identify, evaluate,
or correct conditions adverse to quality between 1988 and October 10, 2007. The corrective
actions implemented by the licensee to address the substantive human performance and
problem identification and resolution crosscutting issues were ineffective in sustaining
performance improvement as noted by licensee self assessments, external industry reviews,
and NRC inspections. The team also identified several examples of poor and inconsistent
implementation of corrective action program behaviors. The examples of the violation
involved not entering the use of unqualified tape in containment in the corrective action
process, evaluating the condition, or taking timely actions to remove the tape from all three
units; not identifying, evaluating, or implementing timely corrective actions associated with
operating experience applicable to the auxiliary feedwater pump trip and throttle valve; not
implementing timely corrective actions for water intrusion and flooding of underground
manholes and cable vaults; inadequate evaluation for nonconforming Target Rock reed
switches; not evaluating and correcting a degraded condition with post accident monitoring
instrument chart recorders, and not correcting a degraded/nonconforming condition
associated with 3 inch Borg-Warner check valves. Each of the individual technical issues
was entered into the licensees corrective action program.
The examples associated with this finding are greater than minor because they were
associated with the mitigating systems cornerstone attribute of equipment performance and
affected the cornerstone objective of ensuring the availability and reliability of systems that
respond to initiating events to prevent undesirable consequences. Using the Inspection
Manual Chapter 0609, "Significance Determination Process," Phase 1 Worksheets, the
examples associated with this finding are determined to have very low safety significance
since they only affected the mitigating systems cornerstone and did not represent a loss of
system safety function. The causes of the examples of this finding have crosscutting
aspects associated with decision making of the human performance area in that operations
and engineering personnel failed to use conservative assumptions for operability decision-
making when evaluating degraded and nonconforming conditions (H.1.(b)). The causes of
the examples of this finding have crosscutting aspects associated with: (1) corrective
actions of the problem identification and resolution area because the licensee failed to
evaluate previous issues such that resolutions addressed all conditions affecting operability
(P.1.(c)), (2) operating experience of the problem identification and resolution area in that
engineering personnel failed to ensure implementation and institutionalization of operating
experience through changes to station processes, procedures, equipment, and training
programs (P.2.(b)), and (3) self assessment of the problem identification and resolution area
in that the licensee did not follow their benchmarking and self assessment guide to ensure
findings were evaluated in their corrective action program (P.3.(c)). The causes of the
examples of this finding also related to the safety culture component of accountability in that
workforce and management personnel failed to demonstrate a proper safety focus and
reinforce safety principles (O.1.(b) and O.1.(c)). (Multiple Sections)
- Green. The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion III,
"Design Control," for the failure to translate design basis requirements into procedures to
ensure the plant is operated within its design basis. Specifically, between 1985 and
- 12 - Enclosure
October 2007, the maximum condensate storage tank temperature requirements did not
include the effect of recirculated hot condensate water from the main condenser. The issue
was entered into the corrective action program as Palo Verde Action Request 3073243.
This finding is greater than minor because it was associated with the mitigating systems
cornerstone attribute of equipment performance and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the Inspection Manual Chapter 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have very low
safety significance since it only affected the mitigating systems cornerstone and did not
represent a loss of system safety function. The cause of this finding has crosscutting
aspects associated with corrective action of the problem identification and resolution area in
that engineering personnel did not assess conditions adverse to quality for impacts to the
operability of safety related equipment (P.1.(c)). (Section 5.2.b.1)
- Green. The team identified a noncited violation of License Condition 2.C(6) for the failure to
install sprinkler heads in accordance with the fire protection program. Specifically, on
October 2, 2007, the team identified several upright fire sprinkler heads in the auxiliary
building that were incorrectly installed in a downward orientation. This issue was entered
into the corrective action program as Palo Verde Action Request 3073824.
This finding is greater than minor because it was associated with the mitigating systems
cornerstone attribute of external factors and affected the cornerstone objective of ensuring
the availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the Inspection Manual Chapter 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to require additional
evaluation under Inspection Manual Chapter 0609, Appendix F, Fire Protection Significance
Determination Process, because it was associated with the suppression element of
defense-in-depth. Since the installed configuration of the sprinkler heads represented a low
degradation of the fire suppression system, in accordance with Section 1.3.1, of Inspection
Manual Chapter 0609, Appendix F, the issue was determined to have very low safety
significance. (Section 5.2.b.2)
- Green. The team identified a noncited violation of 10 CFR 50.65(a)(2) for the failure of
maintenance rule and engineering personnel to demonstrate that the performance or
condition of structures, systems, or components was being effectively controlled through
appropriate preventive maintenance to ensure systems or components remained capable of
performing their intended function. Specifically, between April and October 2007, an
inadequate evaluation of maintenance rule performance criteria was performed and, even
though the Unit 2 auxiliary feedwater Train A had exceeded its maintenance rule
10 CFR 50.65(a)(2) performance criteria, no goal setting and monitoring was performed as
required by 10 CFR 50.65(a)(1) of the maintenance rule. This issue was entered into the
corrective action program as Palo Verde Action Request 3075907.
This finding is greater than minor because it was associated with the mitigating systems
cornerstone attribute of equipment performance and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the Inspection Manual Chapter 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have very low
safety significance since it only affected the mitigating systems cornerstone and did not
represent a loss of system safety function. The cause of this finding has crosscutting
- 13 - Enclosure
aspects associated with self assessments of the problem identification and resolution area
in that maintenance rule and engineering personnel failed to perform self assessments that
were comprehensive, appropriately objective, and self-critical (P.3.(a)). The cause of this
finding has crosscutting aspects associated with decision-making of the human performance
area in that engineering personnel failed to make safety-significant or risk-significant
decisions using a systematic process (H.1.(a)). The cause of this finding is also related to
the safety culture component of accountability in that management did not reinforce safety
standards and display behaviors that reflected safety as an overriding priority (O.1.(b)).
(Section 5.5.b.1)
- Green. The team identified a finding for the failure of maintenance personnel to install
emergency lighting in containment in support of the refueling outage per repetitive
maintenance work Order 2935399 and work Instruction WSL 24436. As a result, work
began in the Unit 3 containment with no emergency lighting installed and no egress
contingency plan for a loss of containment lighting. This issue was entered into the
corrective action program as Palo Verde Action Request 3070783.
This finding is considered more than minor because if left uncorrected, a failure to install
emergency lighting could hamper emergency response activities in the containment or
complicate emergency egress from the containment. Using the Inspection Manual Chapter
0609, "Significance Determination Process," Appendix M, Significance Determination
Process Using Qualitative Criteria, the finding is determined to be of very low safety
significance because emergency lighting was necessary for personnel safety and personnel
were expected to carry flashlights when responding to events. The cause of the finding has
crosscutting aspects associated with work control of the human performance area in that
maintenance personnel failed to properly plan the emergency lighting installation work by
incorporating contingencies in case the work was not completed in the appropriate
timeframe (H.3.(a)). The cause of this finding is also related to the safety culture component
of accountability in that management personnel failed to reinforce safety standards and
display behaviors that reflected safety as an overriding priority (O.1.(b)). (Section 5.6.2.b.2)
- Green. The team identified a noncited violation of Technical Specification 5.4.1.a for the
failure of radiation protection personnel to follow procedures for installing temporary
shielding at the 87 foot elevation of the auxiliary building west penetration room.
Specifically, temporary shielding (Package A-87-10) was installed in direct contact and
across the Train A low pressure safety injection pressure instrument sensing line. However,
a piping stress analysis was not performed as required by Procedure 75RP-9RP25,
Temporary Shielding, Revision 9. This issue was entered into the corrective action
program as Palo Verde Action Requests 3071468 and 3072224.
This finding is greater than minor because it was associated with the mitigating systems
cornerstone attribute of configuration control and affected the cornerstone objective to
ensure the availability and capability of systems to respond to initiating events. Using the
Inspection Manual Chapter 0609, "Significance Determination Process," Phase 1
Worksheets, this finding is determined to be of very low safety significance because the
condition did not result in an actual loss of safety function, and did not screen as risk
significant or contribute to external event initiated core damage sequences since it did not
involve a loss or degradation of equipment designed to mitigate a seismic event. This
finding has crosscutting aspects associated with the work practices component of the
human performance area because the licensee did not effectively use human error
- 14 - Enclosure
prevention techniques such as self checking and proper documentation of activities for the
shielding installation (H.4.(a)). (Section 5.6.2.b.3)
- Green. The team identified a noncited violation of 10 CFR 50.65, for the failure of
engineering personnel to establish goals and monitor the performance of the safety injection
system. Specifically, on March 22, 2007, engineering personnel failed to establish goals to
properly monitor system performance, or provide a technical justification to demonstrate that
monitoring under 10 CFR 50.65(a)(1) was not required for the safety injection system
following the system changing status from 10 CFR 50.65(a)(2) to 10 CFR 50.65(a)(1). This
issue was entered into the corrective action program as Palo Verde Action Requests
3074255 and 3076699.
This finding is greater than minor because it was associated with the mitigating systems
cornerstone attribute of equipment performance and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the Inspection Manual Chapter 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have very low
safety significance since there was no loss of safety function. The cause of this finding has
crosscutting aspects associated with: (1) corrective actions of the problem identification and
resolution area in that engineering personnel failed to take appropriate actions to address
safety issues and adverse trends in a timely manner (P.1.(d)), and (2) self assessment of
the problem identification and resolution area in that engineering personnel did not perform
self assessments that were comprehensive, objective, and self critical (P.3.(a)).
(Section 5.6.9.b.1)
- Green. The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V,
"Instructions, Procedures and Drawings," for the failure of maintenance and engineering
personnel to maintain proper configuration of the support brackets for the pressurizer
condensate pots in accordance with design drawings. Specifically, on October 2, 2007, the
team identified that the support bracket U-bolts were not tight against the condensate pot
piping, jam nuts were not installed on the U-bolts, and jacking bolts were not in full contact
with the pressurizer vessel. The support brackets minimize lateral motion during a seismic
event. This issue was entered into the corrective action program as Palo Verde Action
Requests PVAR 3070805 and 3075704.
This finding is greater than minor because it was associated with the mitigating systems
cornerstone attribute of equipment performance and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the Inspection Manual Chapter 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have very low
safety significance since it only affected the mitigating systems cornerstone and did not
represent a loss of system safety function. This finding has crosscutting aspects associated
with the work practices component of the human performance area because maintenance
personnel did not effectively use human error prevention techniques such as self checking
and proper documentation of activities for the installation of the support bracket (H.4.(a)).
(Section 5.6.7.b.1)
- 15 - Enclosure
- Green. The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V,
Instructions, Procedures, and Drawings, for the failure of maintenance personnel to
properly rig the Unit 3 100 foot elevation inner personnel airlock door in accordance with
engineering drawings. Specifically, the suspended rigging was completed with the
inappropriate placement of wire rope slings over two locking pins resulting in an unanalyzed
force being applied to the doors operating mechanism. This issue was entered into the
corrective action program as Palo Verde Action Request 3086057.
The finding is greater than minor because it could become a more significant safety concern
if left uncorrected in that the applied suspended force on the bronze bushing and the doors
operating mechanism, which were not designed for vertical loading, could degrade the
personnel airlock door sealing capability. This finding can not be evaluated by the
significance determination process because Inspection Manual Chapter 0609, "Significance
Determination Process," Appendix A, "Determining the Significance of Reactor Inspection
Findings for At-Power Situations," and Appendix G, "Shutdown Operations Significance
Determination Process," do not apply to the door for the plant conditions that existed during
the event. This finding affects the barrier integrity cornerstone and is determined to be of
very low safety significance by NRC management review using Inspection Manual Chapter
0609, Appendix M, "Significance Determination Process Using Qualitative Criteria," because
it was a deficiency that did not result in the actual breach of the containment barrier. The
cause of this finding has crosscutting aspects associated with the work practices aspect of
the human performance area in that maintenance personnel failed to provide adequate
oversight of work activities (H.4.(c)). (Section 5.6.4.b.1)
- Green. The team identified a noncited violation of Technical Specification Surveillance
Requirement 3.6.6.6, for the failure to verify that each containment spray nozzle was
unobstructed. Specifically, the last completed surveillance test conducted on each unit,
identified that one nozzle in each unit was obstructed and that the nozzles were not retested
in accordance with the approved retest requirement. This issue was entered into the
corrective action program as Palo Verde Action Requests 3075026, 3075059, 3068647 and,
3048511.
The finding is more than minor because it affected the configuration control attribute of the
barrier integrity cornerstone, and affected the associated cornerstone objective to provide
reasonable assurance that physical design barriers protect the public from radionuclide
releases caused by accidents or events. Using the Inspection Manual Chapter 0609,
"Significance Determination Process," Phase 1 Worksheets, the finding is determined to be
of very low safety significance because it did not involve an actual reduction in defense-in-
depth for the atmospheric pressure control function of the reactor containment.
(Section 5.5.b.3)
- Green. The team identified a noncited violation of Technical Specification Surveillance
Requirement 3.0.3 for the failure of operations personnel to conduct an assessment and
manage the risk for a missed surveillance test. On September 27, 2007, the team identified
that the requirements for testing the containment spray nozzles in Units 1, 2, and, 3 did not
meet Technical Specifications Surveillance Requirement 3.6.6.6. Operations personnel did
not enter Technical Specification Surveillance Requirement 3.0.3 until prompted by the team
on October 30, 2007. This issue was entered into the corrective action program as Palo
Verde Action Request 3085708.
- 16 - Enclosure
The finding is determined to be more than minor because it affected the configuration
control attribute of the barrier integrity cornerstone, and affected the associated cornerstone
objective to provide reasonable assurance that physical design barriers protect the public
from radionuclide releases caused by accidents or events.
Using the Inspection Manual Chapter 0609, "Significance Determination Process," Phase 1
Worksheets, the finding is determined to have very low safety significance because it did not
involve an actual reduction in defense-in-depth for the atmospheric pressure control function
of the reactor containment. The cause of this finding has crosscutting aspects associated
with work practices of the human performance area in that operations personnel failed to
ensure supervisory and management oversight of work activities that resulted in a missed
Technical Specification surveillance requirement (H.4(c)). The cause of this finding is also
related to the safety culture component of accountability in that operations personnel failed
to demonstrate a proper safety focus and reinforce safety principles (O.1.(c)).
(Section 6.2.b.1)
- TBD. The team identified an apparent violation of 10 CFR 50.54(q) and 10 CFR Part 50,
Appendix E.IV.F.2.g, with the significance yet to be determined, for the licensees failure to
correct an identified risk significant planning standard weakness between May 2, 2007 and
October 28, 2007. Specifically, the licensee failed to implement adequate corrective actions
for identified weaknesses in the ability to correctly make a Site Area Emergency declaration
for a steam generator tube rupture event. This issue was entered into the licensees
correction action program as Palo Verde Action Request 3083911.
The team determined that the inability to consistently implement an Emergency Action Level
was a performance deficiency within the licensees control. This finding is more than minor
because it was associated with the Emergency Preparedness attribute of emergency
response organization performance and affected the cornerstone objective to implement
adequate measures to protect the health and safety of the public because the inability to
properly recognize and classify an emergency condition affects the licensees ability to
implement adequate protective measures. This finding was evaluated using the Emergency
Preparedness Significance Determination Process and was preliminarily determined to be of
low to moderate safety significance because it was a failure to comply with NRC
requirements; it was an issue associated with the requirements of Appendix E of
10 CFR Part 50; it was not an issue with a risk significant planning standard as described in
Manual Chapter 0609, Significance Determination Process, Appendix B, Emergency
Preparedness Significance Determination Process, Section 2.0; and it was a functional
failure of the requirements of Appendix E IV.F.2.g because the licensee failed to correct a
weakness associated with Risk Significant Planning Standard 10 CFR 50.47(b)(4). The
cause of this finding has crosscutting aspects associated with the corrective action aspect of
the problem identification and resolution area in that the licensee failed to thoroughly
evaluate problems such that resolutions ensured correcting problems (P.1.(c)). The cause
of this finding was also related to the safety culture component of accountability in that the
licensee failed to demonstrate a proper safety focus and reinforce safety principles (O.1.(c)).
(Section 5.7.b.1)
- Green. The team identified a Green noncited violation of 10 CFR 50.54(q) and
§50.47(b)(4), for the failure of the licensee to be able to implement Emergency Action Levels
3-12 and 7-1. Specifically, area radiation Monitor RU-18 could not be utilized in the vicinity
- 17 - Enclosure
of the remote shutdown panels and therefore, the emergency classification associated with
Emergency Action Level 3-12 could not be declared at the Alert level as required in
Procedure EPIP-99, EPIP Standard Appendices. In addition, the licensee improperly
overclassified Emergency Action Level 7-1 as an Alert when presented conditions
warranting a classification of a Notification of Unusual Event. Specifically, the licensee did
not develop a procedure to enable personnel to differentiate between an aircraft and an
airliner and therefore, the proper emergency classifications could not be consistently
determined. This finding was entered into the licensees corrective action program as
Condition Report Disposition Requests 3071570, 3071572, and 3085175.
The team determined that the inability to implement Emergency Action Levels was a
performance deficiency. The finding was more than minor because it was associated with
the Emergency Preparedness attribute of procedure quality and could affect the cornerstone
objective associated with the licensees ability to correctly classify an emergency condition
which would affect the licensees ability to implement adequate measures to protect the
health and safety of the public. Using the Manual Chapter 0609, "Significance
Determination Process," Appendix B, Emergency Preparedness Significance Determination
Process, the finding was determined to have very low safety significance because the
licensee would be unable to declare one Emergency Action Level at the Alert and one
Emergency Action Level at the Notification of Unusual Event level. The cause of this finding
had crosscutting aspects associated with the corrective action of the problem identification
and resolution area in that the licensee had previous opportunities to identify the
deficiencies (P.1.(a)). (Section 5.7.b.2)
- Green. The team identified a noncited violation of 10 CFR 19.12, Instructions to Workers,
for the failure of radiation protection personnel to provide adequate information regarding
radiological conditions and precautions to minimize exposure during pre-job briefs.
Specifically, on October 1 and 3, 2007, radiation protection personnel did not adequately
inform workers of radiological conditions and precautions to minimize exposure during
radiological briefings. This issue was entered into the corrective action program as Palo
Verde Action Request 3070507 and 3071940.
The finding is greater than minor because it is associated with the Occupational Radiation
Safety Cornerstone attribute of programs and process and affected the cornerstone
objective of ensuring the adequate protection of the workers health and safety from
exposure to radiation during routine operations. Using Inspection Manual Chapter 0609,
Significance Determination Process, Appendix C, Occupational Radiation Safety
Significance Determination Process, the finding was determined to be of very low safety
significance because it was not an as low as is reasonably achievable issue, there was not
an overexposure or substantial potential for an overexposure, and the ability to assess dose
was not compromised. The cause of this finding has crosscutting aspects associated with
decision making in the human performance area in that radiation protection personnel failed
to communicate decisions, and the basis for decisions, to personnel who had a need to
know the information (H.1.(c)). This finding also has a safety culture component aspect of
accountability in that radiation protection personnel did not demonstrate a proper safety
focus or reinforce safety principles among peers when conducting pre-job briefings (O.1.(c)).
(Section 6.1.b.1)
- 18 - Enclosure
- SLIV. The team identified a Severity Level IV noncited violation of 10 CFR 50.71(e) for the
failure of the licensee to periodically update the Final Safety Analysis Report (UFSAR) with
all changes made in the facility or procedures. Specifically, in 2002, radiation protection and
operations personnel changed the operation of the total dissolved solids holdup tanks from
that described in the Updated Final Safety Analysis Report (UFSAR) and did not submit an
update to the NRC. This issue was entered into the licensees corrective action program as
Palo Verde Action Request 3075089.
This issue is being treated as traditional enforcement because the failure to update the Final
Safety Analysis Report has the potential to impact the NRCs ability to perform its regulatory
function. The finding is characterized as a Severity Level IV violation because the
erroneous information was not used to make an unacceptable change to the facility or
procedures. The finding is of very low safety significance because the change in operation
of the total dissolved solids holdup tanks did not result in an increase in the likelihood of a
release of radioactive material. The cause of this finding has a crosscutting aspect
associated with resources in the human performance area in that the licensee failed to
ensure that personnel and equipment were available and adequate to maintain radiological
safety by minimization of long-standing equipment issues (H.2.(a)). (Section 6.2.b.1)
- N/A. The team identified a minor violation of the Palo Verde Physical Security Plan,
associated with the calculation of group work hours. This issue was entered into the
licensees corrective action program as Palo Verde Action Request 3078227. The details of
the finding can be found in Inspection Report 05000528; 05000529; 05000530/2007402.
Miscellaneous
- N/A. The team noted that the licensee had not completed corrective actions and
effectiveness reviews associated with the root and contributing causes for the July 2004,
Yellow finding involving the voiding of emergency core cooling system piping in all three
units. The cause of the failure to implement effective corrective actions was related to the
safety culture component of organizational change management in that, licensee personnel
ceased to implement corrective actions and effectiveness reviews when the existing
management team members assumed that the activities would be integrated into other
station processes following the arrival of a new senior management team. (Section 9.0)
- N/A. The team identified continuing human performance issues at Palo Verde consistent
with previously identified issues discussed in End-of-Cycle and Mid-cycle letters since 2005.
Specifically, human performance concerns observed during this inspection included
weaknesses in implementing the operability determination process, failures to follow
procedures, failures to implement human performance tools, and inadequate procedures. In
addition, a number of engineering issues reflected a lack of technical rigor in resolving
complex issues. The team noted a lack of adherence to basic radiological work practices
and inconsistent implementation of control room behaviors. The team also identified that
the licensees training department had been inconsistent in supporting site improvement.
Although a human performance root cause investigation had been conducted, corrective
actions were not effective in sustaining performance improvement. (Multiple Sections)
- 19 - Enclosure
- N/A. Multiple substantive crosscutting aspects associated with problem identification and
resolution (PI&R) have existed since 2004. Corrective actions continue to remain ineffective
in sustaining improving performance as noted by effectiveness reviews, external industry
reviews, and NRC inspections. The licensees corrective action program was complicated
and cumbersome. Licensee personnel recognized the attributes of problem identification,
evaluation, and resolution when interviewed; however, the knowledge and understanding
was not consistently demonstrated to the NRC during the inspection. (Multiple Sections)
- N/A. The team noted that the licensee's third-party safety culture assessment was
adequate to provide the licensee with the information necessary to develop
appropriate corrective actions for safety culture weaknesses. The results of the NRC's
independent safety culture assessment validated the results of the licensee's third-party
safety culture assessment.
The team identified weaknesses in organizational characteristics and attitudes associated
with 10 of the NRC's 13 safety culture components, as detailed in Section 06.07 of
Inspection Manual Chapter 0305 "Operating Reactor Assessment Program." The most
significant weaknesses were identified in the safety culture components of accountability,
the corrective action program, decision-making, resources, self assessments, and work
practices. The team noted that these weaknesses were widespread among functional
groups across the organization. Organizational characteristics and attitudes were adequate
in the safety culture components of safety policies; the environment for raising concerns;
and preventing, detecting, and mitigating perceptions of retaliation. (Sections 8.1 and 8.2)
Licensee-Identified Violations
Violations of very low safety significance which were identified by the licensee have been
reviewed by the team. Corrective actions taken or planned by the licensee have been entered
into the licensee's corrective action program. These violations and corrective actions are listed
in Section 11 of this report.
- 20 - Enclosure
REPORT DETAILS
1 PERFORMANCE HISTORY
On March 2, 2007, the NRC issued the Annual Assessment Letter, which documented the
results of the annual performance review for the Palo Verde Nuclear Generating Station
(PVNGS), including the decision to perform a supplemental inspection at PVNGS, using
Inspection Procedure (IP) 95003, Supplemental Inspection for Repetitive Degraded
Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs, or One Red Input.
PVNGS Unit 3 was placed in the Multiple/Repetitive Degraded Cornerstone column
(Column 4) of the NRCs Action Matrix, effective in the fourth Quarter 2006. In accordance
with NRC Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment
Program, the decision to place Unit 3 in Column 4 was made on the basis of the definition
of a Repetitive Degraded Cornerstone in that there were two separate safety significant
inspection findings (one Yellow and one White) in the Mitigating Systems cornerstone and
the cornerstone had been degraded for more than four quarters.
Unit 3 was placed in Column 4 based on two findings: (1) a White finding (issued
February 21, 2007) for inadequate maintenance and corrective actions involving the K-1
electrical relay on a Unit 3 emergency diesel generator (EDG); and (2) a Yellow finding
(issued April 8, 2005) involving voiding in the suction line for the emergency core cooling
system (ECCS) pumps in all three units.
As a result of the Yellow finding, the NRC completed the IP 95002, Inspection For One
Degraded Cornerstone Or Any Three White Inputs In a Strategic Performance Area,
supplemental inspection in December 2005. The associated inspection report dated
January 27, 2006, closed the Severity Level III violation of 10 CFR 50.59 and kept the
Yellow finding open. The Yellow finding remained open because the licensees corrective
actions were not fully developed, were narrowly focused, and their implementation was not
effective. In August 2006, the NRC completed a second IP 95002 supplemental inspection.
The associated report dated November 11, 2006, documented that the Yellow finding could
not be closed because the corrective actions to address problems with questioning attitude,
technical rigor, and operability determinations (ODs) were not fully effective. In addition,
measures and metrics to monitor performance improvement had not been developed and
the licensee did not have an effective program for using operating experience (OE).
Throughout 2006, the licensee continued to have performance problems that challenged the
operation of all three units in the following areas: (1) equipment reliability; (2) human
performance; and (3) problem identification and resolution (PI&R). Two special inspections
were conducted in June and September of 2006. The June 2006 special inspection
reviewed concerns regarding spray pond chemistry control and a reduction in heat
exchanger performance for key safety systems. This inspection resulted in the issuance of
five noncited violations of very low risk significance (Green). The September 2006 special
inspection reviewed concerns with the failure of the K-1 electrical relay on the Unit 3 Train A
EDG. This inspection resulted in the issuance of a White finding. The causes for the
findings associated with both of these inspections were similar to the programmatic issues
associated with the 2005 Yellow finding and included: a lack of technical rigor in performing
evaluations and incomplete consideration of the extent of problems when they were
identified.
- 21 - Enclosure
The licensees performance also warranted the issuance of several substantive crosscutting
PI&R and human performance aspects in March 2005. The substantive crosscutting
aspects continue to remain open because of a failure to implement changes that would
result in sustainable performance improvement.
The licensee initiated an integrated performance improvement plan in the fourth quarter of
2005. Their improvement plan was ineffective and performance problems continued
throughout 2006 and into 2007. Factors associated with the lack of performance
improvement included:
- Fixing symptoms and not addressing the root causes of problems,
- Not performing a thorough review of issues,
- Accepting incomplete answers and actions,
- Failing to question the impact of actions,
- Incomplete ODs, and
- Inadequate corrective action program (CAP) implementation.
On June 21, 2007, the NRC issued Confirmatory Action Letter (CAL) 4-07-004, which
required PVNGS to perform additional actions to address their decline in performance.
Specifically, the licensee was required to:
1. Complete actions to address the root and contributing causes identified in evaluations
for the Yellow finding associated with the voided containment sump suction piping for all
three units, and the White finding associated with the Unit 3 Train A emergency diesel
generator electrical relay problems.
2. Complete corrective actions that will result in sustained improved performance in the
crosscutting areas of human performance and PI&R.
3. Complete an independent (third party) safety culture assessment by
September 15, 2007.
4. Incorporate the results of their in-depth evaluations and their safety culture assessment
described in Item 3 above into a modified improvement plan.
5. Submit the portions of the modified improvement plan that impact the Reactor Safety
strategic performance area, including safety culture improvement initiatives by
November 30, 2007.
On September 4, 2007, the licensee submitted a letter to the NRC indicating the
independent safety culture assessment had been completed. The NRCs review of the
safety culture assessment is documented in Section 8.1.
On November 28, 2007, the licensee submitted a letter to the NRC requesting an extension
to the submittal date of the modified improvement plan. The plan was being developed
during the IP 95003 inspection, and was therefore only partially reviewed in October 2007.
The licensee submitted the plan to the NRC on December 31, 2007.
- 22 - Enclosure
2 SITE INTEGRATED BUSINESS PLAN (SIBP) AND SITE INTEGRATED IMPROVEMENT
PLAN (SIIP)
Overview
Because the improvement plan was not complete at the time of the IP 95003 inspection, the
appropriateness, timeliness, and effectiveness of the corrective actions to address the root
and contributing causes, as well as other identified problems, could not be fully evaluated.
The team determined that additional NRC inspections of the modified improvement plan will
need to be conducted before an assessment can be completed.
a. Inspection Scope
The team performed a review of the SIBP and SIIP in accordance with IP 95003,
Sections 02.02.a - 02.02.e. This assessment of the improvement plan was
accomplished by reviewing numerous documents including, in part, root cause
evaluations, apparent cause evaluations, self assessments, condition report/disposition
requests (CRDRs), Palo Verde Action Requests (PVARs), condition report action items
(CRAIs), problem development statements (PDSs), fundamental overall problems
(FOPs), effectiveness reviews, the improvement plan database, and the Improved
Performance and Cultural Transformation (ImPACT) database. The team: (1) reviewed
the procedures for completing the ImPACT project and improvement plan; (2) assessed
the scope of the ImPACT project; (3) reviewed PVARs generated as a result of ImPACT
activities; (4) assessed the ability to cross reference data between the various
documents used to develop the improvement plan; (5) reviewed PDSs for adequacy and
for outstanding technical issues; (6) sampled completed improvement plan corrective
actions to determine timeliness, completion of actions, and measures of effectiveness;
(7) determined if corrective actions identified in ImPACT documents were included in the
improvement plan and the CAP at the appropriate priority level; (8) assessed the
resource loading of the improvement plan; (9) assessed the significance of overdue
action items; and (10) reviewed various background documents for areas that were not
included in the improvement plan.
b. Observations and Findings
Introduction. The team identified several observations associated with the development
of the improvement plan. Since the licensee did not submit the improvement plan to the
NRC before the IP 95003 inspection commenced, the team only reviewed a draft version
of the SIBP/SIIP.
Description. The SIBP and SIIP were developed and controlled by
Procedure 01DP-0AC06, Site Integrated Business Plan (SIBP)/Site Integrated
Improvement Plan (SIIP) Process, Revision 1. Revision 0 of this procedure was issued
in September 2007, which was well after the May 2007 start of the improvement plan
efforts. The SIBP plan is a database program that was developed using Microsoft
Access. This database was designed to track the implementation and completion of
actions contained within the SIBP. The actual corrective actions associated with the
improvement plan, which were CRAIs, were contained in the Site Work Management
System (SWMS) database that is used to track CAP documents. The SIBP included a
subset of corrective actions known as the SIIP. The SIIP contains corrective actions
associated with the ImPACT process, NRC CAL, PVNGS safety culture assessment, IP
- 23 - Enclosure
95001 issues (White finding for inadequate maintenance and corrective actions involving
the K-1 electrical relay on a Unit 3 EDG), IP 95002 issues (Yellow finding involving
voiding in the suction line for the ECCS pumps in all three units), and the substantive
crosscutting issues for human performance and PI&R. The SIIP is expected to be the
modified improvement plan described in Item 5 of the CAL.
The ImPACT process (see figure below) consisted of a series of assessment steps
including ImPACT procedures, checklist findings, PDSs, and FOPs. Checklists were
used to document the results of ImPACT assessments. PDSs were used to collate
related findings from individual assessment activities and then those findings were
grouped into FOPs. After developing the FOPs, the licensee used one or more of the
following tools to identify casual factors by conducting root cause evaluations, apparent
cause evaluations, self-assessments, or effectiveness reviews. Action plans were then
developed, analyzed, prioritized, and incorporated into the SIBP and SIIP. The team
concluded that the ImPACT process successfully identified the performance concerns at
Palo Verde in need of corrective actions.
95003 Inspection Topical Areas FOPs
Module and Selected Fundamental Overall Problems
MORE
ImPACT Feedback Collective
Assessment Plan Evaluation
PVARs
Input CAs,
CHECKLISTS
Historical PDSs Data Base CAPRs &
- KART Effectiveness
Data - IA &CPD Multiple Fault Codes
Reviews
- Focused
Review Assessments
Site Integrated Improvement Plan
PVARs
Roll-Up
Process
- 24 - Enclosure
The ImPACT process reviewed the following seven areas:
- Historical Data Review: reviewed and analyzed over 4000 documents since 2001
including, in part, significant internal and external assessments of performance at
PVNGS, NRC inspection reports, NRC assessment letters, licensee event reports
(LERs), maintenance rule functional failures, trends, various corrective action
documents, and unplanned downpowers.
- Key Attribute Review Team (KART): evaluated the emergency diesel generator (EDG)
and safety injection systems while focusing on the adequacy of programs and processes
for design, human performance, procedure quality, equipment performance,
configuration control, and emergency response organization readiness. The Key
Attribute Review also included the inspection attributes of NRC IP 95003.
- Identifying, Assessing, and Correcting Performance Deficiencies Review: evaluated the
effectiveness of corrective actions associated with significant performance deficiencies,
audits and assessments, resource allocation, performance goals, employee concerns
program, technical resolution programs (e.g., differing professional opinions), and use of
industry information.
- Focused Assessments Review: evaluated specific areas of known weaknesses and
significant change for the 1989 NRC Diagnostic Assessment, the licensees re-
engineering program, PI&R crosscutting assessment, human performance crosscutting
assessment, and performance improvement plan effectiveness assessment.
- Safety Culture Assessment Review: utilized two independent third party teams that
reviewed the safety culture for the site. This item met the safety culture requirements of
NRC IP 95003 and the NRC Confirmatory Action Letter.
- Recirculation Actuation System and K-1 Relay Review: assessed the root causes,
appropriateness of corrective actions, effectiveness of corrective actions, and
measurements of success.
- Collective Evaluation and Action Plans Development: performed an evaluation of the
failures and deficiencies associated with the above six evaluations. This final process
was done to identify the causes for the performance problems and then develop
corrective actions necessary to improve performance.
As of October 1, 2007, the SIBP consisted of 20 building blocks with 5 building blocks that
were designed to always be included in the business plan. These five building blocks
include plant equipment, people, CAP, safety, and knowledge/training. The other 15
building blocks can change as progress and improvement is made on an individual block
and other issues arise which require improvement and corrective action. These blocks are
depicted in the above figure and include, in part, oversight, work management,
programs/processes, procedures, and emergency preparedness. Within each building block
there were one or more initiatives, with a total of 152 initiatives. Each initiative contained
numerous tasks. At the time of the inspection, there were 1609 tasks (each task had a
CRAI) in the SIBP, of which 357 were a part of the SIIP.
- 25 - Enclosure
As of November 1, 2007, there were 339 tasks (CRAIs) closed in SWMS, with only 12 tasks
having completed improvement plan closure packages. Procedure 01DP-0AC06, required
that each task be closed and that the closure review process use a graded approach based
on the category of the task or priority of the CRAI. The team observed a Closure Review
Board meeting on October 31, 2007, and reviewed the October 24, 2007, Closure Review
Board meeting minutes. During the October 31, 2007, meeting, only two tasks were
reviewed and both were rejected because objective evidence of the actions being completed
and sustainability of the actions were not demonstrated or included as part of the package.
The meeting minutes described seven tasks being reviewed for closure of which five were
closed, one was rejected, and one was tabled (supporting information was not included with
the closure package). The team determined that the closure review of the individual tasks
was in accordance with Procedure 01DP-0AC06. The team did note that the contractors
who attended the meeting were driving the Closure Review Board members to higher levels
of accountability and making sure the process was followed; however, the Closure Review
Board was still in the process of establishing repeatable standards.
The team identified the following observations associated with the development of the SIIP:
- The root and contributing causes for each of the FOP root causes were attributed to a
lack of management oversight, leadership, and accountability. Many of the improvement
plan tasks contained little or no detail as to how the specific tasks were to be
implemented. No additional details were available on the criteria/goals that the tasks
should meet,the development schedule, or the resource needs. For example:
1. The root cause for CRDR 3048835, Operational Focus, attributed the problems to
senior management not establishing and enforcing expectations. The evaluation did
not investigate the operations department ability to lead and the appropriateness
and implementation of the current standards of conduct.
2. CRAI 3064362 was initiated to develop a leadership model that established a vision,
mission, values, and behaviors. This corrective action was the main action in
numerous root cause evaluations that was designed to prevent recurrence of various
performance problems that resulted in PVNGS being placed into Column 4 of the
NRC Action Matrix. The description contained in the improvement plan and SWMS
for CRAI 3064362 stated, Benchmark and develop a leadership/management model
that establishes the vision, mission, values and expected behaviors for each of the
problem areas identified by the ImPACT team and the additional areas noted above.
Additionally, the management model should address ownership, the Palo Verde core
fundamental areas (Plant Equipment, People, CAP, Safety, and
Knowledge/Training), a mechanism for continuous monitoring and improvement, and
metrics to measure effectiveness. This CRAI, with a due date of June 2008,
contained no further details as to how to achieve this corrective action.
3. CRAIs 3063852, 3075713, and 3075649, identified corrective actions that were not
specific or measurable as stated in Section 17 of Root Cause Investigation Manual
for Significant CRDRs. The three CRAIs discussed corrective actions to implement
a Management Review Meeting process, develop and implement a
leadership/management model, and establish a site-wide emphasis and alignment
on the core mission and on the core fundamental focus areas. However, the CRAIs
did not include specific details and/or measurable actions.
- 26 - Enclosure
- The team determined that for most cases, actions were included in both the CAP and
the SIIP. However, two items were not found in the improvement plan: 1) CRAI
3076878, Develop, coordinate, and implement a campaign to establish and reinforce
the position that Engineering is the design authority of the site, was one of the
corrective actions to address CRDR 3048865, Design Control and Configuration
Management Weaknesses; and 2) from the independent safety culture assessment,
CRAI 3090979 was an action to include safety conscious work environment (SCWE)
expectations in the contracts for PVNGS contractors.
- Most of the initiatives contained tasks to either develop or modify existing metrics in
order to measure progress. However, most of the new or modified metrics that the team
reviewed were not fully developed. As with corrective actions to address the root
causes, the actions to develop metrics were high level and contained few details. It was
unclear how CRAI 3064372, Develop and utilize metrics to ensure Palo Verde uses the
CAP, training, operating experience, self-assessments/benchmarking, and independent
oversight activities to establish a continuous learning environment, will address the
contributing cause of ineffective implementation of those programs to drive
improvements in individual and station performance as described in the CRDR 3048836,
Organizational Effectiveness root cause report.
- Effectiveness review descriptions were broad, and the criteria provided ambiguous
information on acceptability. For example, CRAIs 3064491 and 3075832 stated that the
interim and final effectiveness reviews can be closed once the following are met: 1) site
performance indicators reflected acceptable performance or overall site improvement;
2) the independent assessment determined that actions were effective, specifically that
Palo Verde had established, communicated, and reinforced standards specific to each of
the focus areas in the leadership/management model and that accountability is
adequately addressed; and 3) overall responses from the safety culture survey indicated
an improving trend. The team did not identify specific criteria that will be used to
determine the effectiveness of the corrective actions (e.g., what constitutes overall site
improvement or an improving trend).
- CRAIs 3063112, develop and implement a site-wide communication strategy, and
3063852, implement a Management Review Meeting process, were coded as Priority 3;
however, the improvement plan had the CRAIs listed as corrective actions to prevent
recurrence which should have been coded Priority 2 as specified by
Procedure 01DP-0AC06. Licensee personnel indicated that they were already aware of
these two examples and had documented these differences, as well as other differences
for CRAI due dates, priorities, and text descriptions on PVAR 3083805, dated
October 26, 2007. As of November 2, 2007, the licensee had identified 35 CRAIs whose
priority codes did not match the improvement plan classification.
- As of November 2, 2007, the licensee had not resource loaded the SIBP/SIIP.
Nevertheless, over 1100 of the 1609 tasks (from November 2007 to December 2008)
were scheduled to be completed by December 31, 2008. This schedule did not appear
to be achievable based on the large number of tasks that have to be closed over the
next 12 months along with the large backlog of work activities that currently exist.
Numerous issues with corrective action due dates were identified by the team, including:
- 27 - Enclosure
1. CRAIs associated with CRDR 3048835, operational focus root cause, had out of
sequence due dates. CRAI 3065021, which was to develop a site indicator for
operational focus, had a due date of January 31, 2008. CRAIs 3062174, 3062184
and 3062188 were written to train leaders on the establishment and proper use of
performance indicators; however, this action had a due date of October 27, 2008,
well after the development of the operational focus indicator.
2. CRAI 3038014 was to conduct a site wide stand-down in order to communicate CAP
fundamentals to all PVNGS personnel. This corrective action item was initiated on
July 9, 2007, and had a due date of December 28, 2007. The team determined that
this action was untimely considering that the Unit 3 outage started
September 29, 2007, and the continuation of CAP weaknesses demonstrated at
3. The PI&R root cause in CRAI 3037453 initiated on July 6, 2007, was to conduct a
self-assessment of the OD program by June 30, 2008. The team considered this
action untimely given the continued problems with the implementation of this
program for the past several years, as well as numerous OD issues identified during
this inspection.
4. The SIIP contained actions that had due dates significantly different from what was
initially specified for the root and contributing cause corrective actions. The team
was concerned that the corrective actions were untimely, especially for the
substantive crosscutting areas of human performance and PI&R, where performance
had not appreciably improved. After incorporation into the SIIP, all of the following
CRAIs had their due dates extended for more than a year from the originally
scheduled completion date: 1) CRAI 3015013, Facilitate implementation of
programmatic actions to improve procedure use and adherence, as well as improve
procedure quality, had the due date changed from October 1, 2007, to
October 1, 2008; 2) CRAI 2936516 was written to evaluate human performance
integration with key work processes. This CRAI was due to be completed December
31, 2007, but was changed to March 15, 2009; 3) CRAI 2941720 was written to
develop a process to add operating experience to work packages. This CRAI was
due to be completed by June 1, 2007, but was changed to December 31, 2008;
4) CRAI 2941718 was written to make operating experience search engines more
available and easier to use. This CRAI was due to be completed by June 1, 2007,
but was changed to December 28, 2008; and 5) CRAI 3038038 was a corrective
action to provide training for all advocates in their responsibilities for quality CAP
implementation with a due date of November 30, 2007. When the action was
incorporated into the SIBP as Task 3.3.3.d, the action was changed to Establish a
process to provide training for all Advocates with the same due date. Actual
training of the advocates is in Action 6.3.1.b (CRAI 3032702) with a due date of
March 15, 2009.
- Limited reviews were completed by the licensee on past work products to look for
mistakes that could have a potential impact on plant equipment and a corresponding
reduction in safety. The team was concerned that a historical review of most
programs/processes work products, including the CAP, had not been conducted and
was not included as an action in the SIIP.
- 28 - Enclosure
- Observations associated with the incorporation of safety culture insights into the
improvement plan are referenced in Section 8.1, under the heading titled, Licensee
Analysis and Corrective Actions. The observations included weaknesses in
resource/staffing levels, a lack of links between corrective actions associated with safety
culture and the SIIP, and ongoing incorporation of safety culture assessment findings
and recommendations into the SIIP.
3 COLLECTIVE SIGNIFICANCE REVIEW
Collective Review of Root and Contributing Causes
The team compared the results from the inspection to the root cause analyses performed by
the licensee and information docketed from previous NRC inspections. The team concluded
that the licensees root and contributing causes bounded the performance deficiencies
identified during the ImPACT review and the NRC IP 95003 inspection. The licensee
identified numerous root and contributing causes for the performance deficiencies. The
following is a summation of the key root and contributing causes applicable to most of the
licensees investigations: (1) leaders did not establish, communicate, and enforce standards
and expectations for performance or hold individuals accountable to those standards; (2) the
corrective action program, operating experience, self assessments, and benchmarking did
not drive individual and station performance improvement; (3) responsibility, accountability,
and authority for nuclear safety were not well defined or understood; (4) individual behaviors
that demonstrate nuclear safety principles were not consistently applied; (5) management
was not receptive to organizational issues identified during investigations; (6) change
management activities did not anticipate unintended consequences and did not clearly
define and communicate changes to station personnel; and (7) Oversight groups did not
provide specific and meaningful interventions to correct declining performance.
Collective Review of Risk
The team completed an assessment of the collective risk associated with the IP 95003
findings. The team was supported by senior reactor analysts from NRC Region IV and
headquarters during the risk assessment. Three methods were used: (1) an adjustment to
the human error probabilities in the Palo Verde Standardized Plant Analysis Risk (SPAR)
model, Revision 3.31, (2) assignment of risk results to each finding screened as having very
low safety significance, and (3) a qualitative assessment using the NRC IMC 0305,
Operating Reactor Assessment Program, criteria for determining if oversight of a licensee
should be performed under NRC Manual Chapter 0350, Oversight of Reactor Facilities in
Shutdown Condition due to Significant Performance and/or Operational Concerns. The
team concluded that Palo Verde was safe for continued operation even though a
degradation in safety performance had occurred and there were several longstanding
performance concerns.
Palo Verde SPAR Model
Palo Verde had documented substantive crosscutting issues in human performance and
PI&R since March 2005 NRC Annual Assessment Letter. Given the duration of the
substantive crosscutting issues, the analyst used approved significance determination tools
to estimate the effect that this condition had on the risk of operating the plant. The primary
source document used in this effort was the SPAR-H Human Reliability Analysis Method,
NUREG/CR-6883 (SPAR-H).
- 29 - Enclosure
The analyst used the Palo Verde SPAR model, Revision 3.31, dated June 18, 2007. The
model was updated to correct errors where the SPAR-H calculator did not account for
dependencies when three or more negative performance shaping factors (PSFs) were
judged to affect the human error probability (HEP) for a human action basic event. This had
the effect of lowering some of the HEPs in the base model.
The analyst assumed that the condition of poor work practices existed for at least one year,
consistent with the exposure time limits of the significance determination process, and that
the condition affected all of the human actions included in the SPAR model equally, with the
exception of offsite power recovery actions (which were deemed to be controlled mostly by
outside influences). Using the SPAR-H Worksheets for action steps at power, a PSF
penalty for poor work practices was assumed, which assigns a multiplier of 5.0 for the
likelihood of failure. For basic events where there were less than 3 negative PSFs, this
resulted in the HEP being increased by a factor of 5.0. For cases where three or more PSFs
existed, the factor of increase was less than 5.0. Although offsite power recovery actions
were left unchanged, the non-recovery probabilities for recovery of a diesel generator, which
use actuarial data in lieu of the SPAR-H method, were increased by a factor of 10 percent.
The base core damage frequency (CDF) was 8.989E-6/year. Application of the 5.0 PSF for
poor work practices resulted in a total CDF of 4.605E-5/year, or a delta-CDF of
3.706E-5/year.
This result accounts only for internal initiating events and does not consider the additional
risk associated with seismic, fire, or other external initiators, nor does it account for the risk
associated with shutdown conditions. Typically, external initiating events approximately
equal the risk associated with internal initiators. Using the above results, and assuming that
poor work practices would affect the recovery from external initiators to the same extent as
for internal initiators, the total baseline CDF would be 1.798E-5/year. The total CDF
associated with poor work practices would be 9.210E-5/year, and the delta-CDF would be
7.412E-5/year.
In accordance with IMC 0609, Appendix H, Containment Integrity Significance
Determination Process, for a large, dry containment, the large early release frequency
(LERF) is significant only with respect to steam generator tube ruptures and intersystem loss
of coolant accidents (ISLOCAs). Employing the same assumptions used in the CDF
calculation for the effect of poor work practices, the results for ISLOCAs and steam
generator tube ruptures are as follows:
Base CDF = 5.025E-7/year
Work Practices CDF = 5.371E-6/year
Delta CDF = 4.868E-6/year
The LERF fraction for both ISLOCAs and steam generator tube ruptures is 1.0. Therefore
the delta LERF is also equal to 4.868E-6/year. The significance bands for LERF are one
order of magnitude lower than those corresponding to CDF. Consequently, for the case of
poor work practices, the LERF significance is the same as the CDF significance.
The result is below the Regulatory Guide 1.174, An Approach to Using Probabilistic Risk
Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis,
limitations for a maximum total plant CDF. Regulatory Guide 1.174 makes use of the NRCs
Safety Goal Policy Statement in evaluating increases in CDF and LERF. The safety goals
- 30 - Enclosure
define an acceptable level of risk that is a small fraction (0.1 percent) of the other risks to
which the public is exposed. Regulatory Guide 1.174 specified that, if there is an indication
that the total CDF may be considerably higher than 1E-4/year or 1E-5/year for LERF, the
focus should be on finding ways to decrease the risk. The team noted that the total
collective risk did not exceed the Regulatory Guide 1.174 upper limits.
Several defense in depth layers of protection are provided to protect the public and the
environment from potential events. These include the integrity of the physical structure of
the plant and its systems, the automatic initiation capabilities of the safety-related systems,
the proceduralized operator manual actions to start equipment and initiate systems, and the
ability of plant operators and technicians to restore, repair, or replace equipment as
necessary. Poor work practices can degrade any of these defense-in-depth layers of
protection, but would mostly cause a loss of efficiency and precision in the operators ability
to take important manual actions, as well as the ability of the plant staff to restore non-
functioning equipment. The team determined that there had been a reduction in defense in
depth features because of the degradation of the CAP and human performance safety
culture concerns; however, the reduction was not sufficient to result in an unsafe condition.
Collective Assessment of IP 95003 Findings
Inspection Manual Chapter 0609, Significance Determination Process, utilizes a counting
rule to assess the significance of a performance deficiency. Using the Phase 2 plant
specific worksheets, core damage sequences are assigned a range of numeric values.
Three sequences with the same numeric result are treated with the next lower value
(e.g., three sequences with an 8 would be treated as one 7). For the purpose of the
collective review, the team assigned a significance determination process result of 8 for all
findings screened as Green during the Phase 1 process. The counting rule was then used
to determine the collective risk. This result was combined with any numerical results
obtained as part of a Phase 3 SDP evaluation for an inspection finding. The emergency
preparedness finding was assigned a value of 3.3E-6. Fifteen examples of findings were
screened as Green during the Phase 1 SDP process (this included findings screened using
IMC 0609, Significant Determination Process, Appendix M, Significance Determination
Process using Qualitative Criteria). Using the counting rule, this equates to a result of one
5. The team applied a CDF value of 3.3E-5/year from the counting rule result. The results
from the Phase 3 SDP evaluation for the switchyard finding was 5.0E-7/year. The combined
result was a CDF of 3.68E-5/year.
If all of the significant findings since 2004 were included, the total result would be between a
range of 4.69E-5/year to 8.79E-5/year. This includes a range of 5.7E-6/year to 4.6E-5/year
for the Yellow finding and 10 CFR 50.59 Severity Level III violation, an assigned value of
3.3E-6/year for the Emergency Preparedness Plan Change Severity Level III violation, and a
range of 1.1E-6/year to 1.8E-6/year for the White finding. Both cases are below the
Regulatory Guide 1.174 limitations for a maximum total plant CDF.
Qualitative Assessment Using Manual Chapter 0305 Criteria
Manual Chapter 0305 uses three criteria to assess the applicability of Manual Chapter
0350. The teams assessment of the Manual Chapter 0305 criteria are as follows:
1. Multiple significant violations of the facilitys license, Technical Specifications,
regulations, or orders.
- 31 - Enclosure
Multiple significant violations (greater than green for SDP findings or greater than
Severity Level IV for non-SDP findings) have not recently occurred. Specifically, a
Severity Level III violation of 10 CFR 50.59 and a Yellow finding related to the
containment sump voiding issue occurred in 2004; a Severity Level III violation for the
failure to obtain prior NRC approval for an emergency plan change was issued in 2005;
and a White finding for the failure of an emergency diesel generator was issued in 2006.
In consideration of this attribute, the team reviewed significant violations identified since
2004, as well as the potentially significant emergency preparedness and overtime
findings identified during the IP 95003 inspection. The team concluded that while there
had been multiple significant findings dating back to 2004, the current assessment cycle
did not have any significant findings. If the emergency preparedness and overtime
findings are determined to be greater than Green (significant), they will be the only
significant items identified during 2007. As such, this criterion would still not be met.
2. Loss of confidence in the licensees ability to maintain and operate the facility in
accordance with the design basis (e.g., multiple safety significant examples where the
facility was determined to be outside of its design basis, either due to inappropriate
modifications, the unavailability of design basis information, inadequate configuration
management, or the demonstrated lack of an effective problem identification and
resolution program).
The team determined that while the licensees CAP is complicated and cumbersome, the
CAP contained the basic elements of an effective program. Licensee personnel
recognized the attributes of problem identification, evaluation, and resolution when
interviewed; however, the knowledge and understanding was not consistently
demonstrated to the NRC during the IP 95003 inspection. Nevertheless, multiple
significant examples of problems with the design basis have not been identified;
therefore, this criterion was not met.
3. A pattern of failure of licensee management controls to effectively address previous
significant concerns to prevent recurrence.
A substantial degradation of the CAP has occurred. There have been repetitive failures
in management controls to improve human performance and problem identification and
resolution. There have also been several repetitive occurrences of risk important
equipment failures (auxiliary feedwater Target Rock steam admission valves,
emergency diesel generator fuel and lube oil filters, safety injection system check
valves, and essential cooling water heat exchanger fouling). The licensee has not had a
recurrence of voided piping or emergency diesel generator K-1 relay failures following
the issuance of the Yellow and White findings. Because the repetitive occurrences were
determined to be of very low safety significance, this criterion was not met.
4 NRC METHODOLOGY AND DIAGNOSTIC ASSESSMENT
The intent of IP 95003 is to allow the NRC to obtain a comprehensive understanding of the
depth and breadth of safety, organizational, and performance issues at facilities where data
indicate the potential for serious performance degradation. The objectives of the IP 95003
inspection are to:
- 32 - Enclosure
(1) provide additional information to be used in deciding whether the continued operation of
the facility is acceptable and whether additional regulatory actions are necessary to
arrest declining performance;
(2) provide an independent assessment of the extent of risk significant issues to aid in the
NRCs current assessment that an acceptable margin of safety exists;
(3) independently evaluate the adequacy of facility programs and processes used to
identify, evaluate, and correct performance issues;
(4) independently evaluate the adequacy of programs and processes in the affected
strategic performance areas;
(5) provide insight into the overall root and contributing causes of identified performance
deficiencies;
(6) determine if the NRC oversight process provided sufficient warning of significant
reductions in safety; and
(7) independently assess the licensee safety culture and assess their evaluation of safety
culture.
A multi-disciplinary team conducted the inspection over the course of approximately nine
months, with a total of five weeks of onsite inspection effort. The inspection implemented
the applicable portions of IP 95003 necessary to assess the extent of performance problems
that led to the licensees entry into Column 4 of the NRCs Action Matrix, including the safety
culture contributions to the performance problems, as well as the licensees corrective action
plan. The team performed an independent diagnostic review of numerous programs and
processes with an emphasis on the reactor safety strategic performance areas. This
provided the NRC with a comprehensive understanding of the depth and breadth of safety,
organizational, and performance issues at PVNGS, in addition to the insights already gained
from the IP 95002 inspections conducted in 2005 and 2006.
The team selected the containment spray system and the turbine driven auxiliary feedwater
pump, high pressure safety injection pump, low pressure safety injection pump, and
essential spray pond pumps. The selection of these components was based on the impact
of component failure on large early release frequency and the completion of a detailed
design review being completed by the licensee as part of their component design basis
review. The team performed a review of the work performed on these components which
involved multiple licensee organizations, including operations, maintenance, engineering,
quality assurance, and management. With respect to these components, the team review
included, as applicable, permanent and temporary design modifications (including
implemented, planned, and cancelled modifications), procedure and drawing changes, ODs,
operator work arounds, configuration control, maintenance, root and apparent cause
evaluations, and various corrective action documents. Additionally, the team reviewed
PVNGS programs and processes associated with human performance and PI&R.
- 33 - Enclosure
5 REACTOR SAFETY STRATEGIC PERFORMANCE AREA
5.1 Licensee Controls for Identifying, Assessing, and Correcting Performance Deficiencies
The licensee had multiple substantive crosscutting aspects associated with human
performance and PI&R. Since 2004, the corrective actions implemented by the licensee
had yet to sustain performance improvement as noted by licensee self assessments,
external industry reviews, and NRC inspections. The team noted that licensee personnel
often recognized appropriate CAP fundamentals and expected behaviors when
interviewed; however, this knowledge and understanding of the program expectations was
not consistently demonstrated. The team noted several examples of poor and inconsistent
implementation of safety culture aspects associated with PI&R. Specifically:
- Licensee personnel did not recognize the need to initiate a PVAR, the licensees
corrective action document form, when a degraded condition was identified by the
team. This particular behavior improved during the conduct of the inspection in
response to the teams repeated questioning of licensee personnel on whether a
PVAR was appropriate for NRC identified issues. The team noted that consistent re-
enforcement of expectations was needed to ensure PVARs would continue to be
initiated following the teams departure.
- The team noted that a licensee component design basis review (CDBR) team
(consisting largely of contractor personnel) was documenting issues that challenged
the design basis at an appropriately low threshold. In contrast, Palo Verde
engineering personnel considered these issues below the PVAR threshold or that the
problems entered were not issues at all. This demonstrated a continuing lack of
understanding on the part of Palo Verde engineering personnel of the level at which
conditions adverse to quality should be documented in the CAP.
- The team noted a significant number of weak or non-existent operability
determinations of degraded conditions affecting safety-related equipment, indicating
an apparent lack of understanding of the need to assess operability for conditions
adverse to quality and a lack of knowledge or skills necessary to conduct an operability
assessment. This is a continuing weakness in the implementation of the CAP at Palo
Verde and had a direct impact on maintaining nuclear safety margins. The inability to
consistently perform ODs formed part of the NRCs basis for leaving open the Yellow
finding involving voiding of the ECCS suction piping in all three units. Improvement in
the operations and engineering departments are required for Palo Verde to effectively
evaluate degraded conditions affecting safe plant operation.
- The team noted that a significant backlog review was required due to the large number
of databases (at least 37) that existed outside of the corrective action process. The
team identified that at least two databases existing outside of the recognized CAP
contained conditions adverse to quality that had not been assessed for operability.
The Action Tracking System (ACT) database and the Bechtel non-conformance
reporting (NCR) database both contained conditions adverse to quality that were not
evaluated for operability impacts until prompted by the team. Licensee personnel
subsequently reviewed the databases and additional conditions adverse to quality that
required operability assessments were identified. The placement of conditions
adverse to quality in systems outside the CAP hindered the ability of operations
- 34 - Enclosure
personnel to assess challenges to the operability of structures, systems, and
components (SSCs).
- The team identified that when conditions adverse to quality were recognized or
evaluated within the CAP, the need to evaluate the extent of condition or impact to the
other units was not always recognized.
- The team concluded that self-assessments completed by Palo Verde personnel lacked
depth and did not effectively specify or implement corrective actions. As a result, the
self-assessment program seldom resulted in improved organizational performance.
The team did note one training self-assessment that had been recently conducted
which had more depth and contained insightful observations. The team noted that this
self-assessment was conducted by a mix of Palo Verde and non-Palo Verde personnel
which may have led to the more meaningful self-assessment.
- The teams evaluation of root cause analyses determined that the analyses of
problems did not consistently specify complete or adequate corrective actions, or
establish timely corrective actions for significant conditions adverse to quality.
- The team identified that the licensee had difficulty determining the status or completion
of corrective actions taken in response to significant issues. This was most apparent
when licensee personnel could not effectively respond to a team request to
communicate the status of corrective actions related to the Yellow finding for voiding of
ECCS suction piping. The licensee could not effectively determine the completion
status of these corrective actions nor had the actions been effectively evaluated for
resolution of the issues. The licensees IP 95002 Readiness/Effectiveness Report
stated that the 95002 focus areas, Seem to have been administratively forgotten. In
addition, the team noted that an ImPACT Checklist intending to evaluate the status of
the Yellow finding, identified several problems; however, not all of the problems had
CAP actions written to address the identified issues.
- Licensee personnel were assigned corrective actions for significant conditions adverse
to quality; however, processes were not consistently implemented to ensure corrective
actions were completed or that effectiveness reviews of these actions were completed.
The team identified that corrective actions taken in response to significant conditions
adverse to quality were sometimes closed prior to completion of the corrective action.
This sometimes occurred when a significant action was closed to another document,
which was subsequently closed prior to the completion of the action. In the past, the
licensee used an unsuccessful approach that relied on individual management team
members to verify significant corrective actions were complete and to evaluate their
effectiveness. More recently, the licensee instituted a Closure Review Board process
to assess completion of significant corrective actions and to assess their effectiveness.
The team acknowledged that a management team review could be more successful in
assuring the completion and effectiveness of corrective actions.
a. Inspection Scope
The team evaluated whether the licensees CAP was sufficient to prevent further
declines in safety that could result in unsafe operation. Specifically, the team
reviewed: (1) licensee investigations, evaluations, and corrective actions taken in
response to significant conditions adverse to quality; (2) audits and assessments
- 35 - Enclosure
conducted by the Nuclear Assurance Department, self-assessments by organizations,
and external evaluations and assessments; (3) the effectiveness of the licensees use
of operating experience and industry information for previously documented
performance issues; (4) historical and current resource allocations, as well as the
current backlog and existing operator work-arounds; (5) the business plan to
determine if licensee performance goals were congruent with corrective actions
needed to address performance issues; (6) the employee concerns program as well as
a significant number of focus group discussions with a cross-section of the licensees
workforce; and (7) the licensees programs and processes in place to support
improvement suggestions by employees and to provide employees feedback on issues
they had identified.
b. Findings and Observations
b.1 Failure to Implement Operability Determination Process for Bechtel
Nonconformance Reports
Introduction. The team identified an example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures and
Drawings, for the failure of the licensee to follow procedures to evaluate
conditions adverse to quality for impacts on the operability of safety-related
equipment.
Description. On October 4, 2007, the team met with the licensee to discuss the
quality assurance program requirements agreed to between the licensee and
Bechtel for the conduct of the Unit 3 steam generator replacement outage, and
how Bechtel nonconformance reports (NCRs) generated during this activity were
reviewed by the licensee. The discussion was held in response to the teams
identification of a condition adverse to quality associated with the rigging of the
containment personnel airlock (PAL) door.
On October 6, 2007, the team questioned the CAP manager on how Bechtel
NCRs were reviewed by the licensee for potential impacts to the operability of
safety-related equipment. The team noted that a formal process to review NCRs
for immediate operability did not exist. As a result of the teams questioning, the
CAP manager initiated actions to review the NCR database. As a result, two
NCRs were identified which documented conditions adverse to quality that
affected safety-related equipment. Specifically, a piping support affecting
shutdown cooling heat exchanger Train A had been inadvertently removed by
Bechtel and an NCR was written to document the problem. No PVAR was
generated and as a result, no operability assessment of the degraded condition
was conducted. Shutdown cooling heat exchanger Train A was declared
inoperable until an engineering evaluation determined the missing support did
not affect operability. A second Bechtel NCR was then identified that
documented the inadvertent removal of steam generator weldment. This
condition was subsequently determined not to affect operability of safety-related
equipment.
On October 8, 2007, the licensee generated a night order that required all NCRs
generated by Bechtel to have PVARs written to assure operability assessments
of conditions adverse to quality were conducted.
- 36 - Enclosure
Analysis. The failure to implement the OD process for conditions adverse to
quality identified in the Bechtel NCR database was a performance deficiency.
The finding is greater than minor because it was associated with the equipment
performance attribute of the mitigating systems cornerstone and affected the
associated cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. Using the IMC 0609, Significance Determination Process,
Phase 1 Worksheets, the finding is determined to have very low safety
significance (Green) because it only affected the mitigating systems cornerstone,
and did not result in the loss of safety function. The cause of this finding had
crosscutting aspects associated with decision-making of the human performance
area in that licensee personnel did not make safety-significant or risk-significant
decisions using a systematic process (H.1.(a)). This finding also had a safety
culture component aspect in the area of accountability in that management did
not reinforce safety standards associated with the need to perform operability
assessments (O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, Instructions,
Procedures, and Drawings, requires that activities affecting quality be prescribed
by instructions, procedures, or drawings, and be accomplished in accordance
with those instructions, procedures, and drawings. The assessment of
operability of safety-related equipment needed to mitigate accidents was an
activity affecting quality and was implemented by Procedure 40DP-9OP26,
Operability Determination and Functional Assessment, Revision 18. Procedure
40DP-9OP26, Step 3.1.1, stated the OD process was entered upon discovery of
circumstances where operability of any SSC described in the Technical
Specifications was called into question upon discovery of a degraded,
nonconforming, or credible unanalyzed condition. Contrary to the above,
between October 4 and 6, 2007, licensee personnel failed to enter the OD
process upon discovery of circumstances where the operability of a component
described in the Technical Specifications was called into question. Specifically,
the removal of a shutdown cooling heat exchanger support and the removal of
steam generator weldment were not evaluated for operability impacts to safety-
related equipment. Because this finding is of very low safety significance and
had been entered into the CAP as PVAR 3072732, this violation is being treated
as an NCV, consistent with Section VI.A of the Enforcement Policy: NCV
05000528, 05000529,05000530/2007012-01, eight examples of the Failure to
Implement Operability Determination Process. This was the first of eight
examples associated with the licensees failure to properly implement the OD
program.
b.2 Failure to Implement Operability Determination Process for ACTs
Introduction. The team identified a second example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures and
Drawings, for the failure of licensee personnel to follow procedures to evaluate
conditions adverse to quality for degraded or non-conforming conditions that
Description. On June 22, 2007, the Palo Verde ImPACT team documented that
the ACT database contained conditions adverse to quality and that the, Entire
- 37 - Enclosure
ACT database needed to be scrubbed to identify all discrepancies. On
August 29, 2007, the team requested the status of the ACT database scrub, to
determine whether additional conditions adverse to quality were identified in the
ACT database since the June 22, 2007, roll-up, and whether these and the
previous conditions identified on June 22, 2007, had been evaluated by a
licensed senior reactor operator (SRO) for degraded or non-conforming
conditions that would require ODs or FAs. The ImPACT team determined that
additional conditions adverse to quality had been identified and that a PVAR had
been generated; however, neither the previously identified ACT issues nor the
more recently identified ACT issues had been assessed individually for OD or FA
requirements as discussed in Procedure 01DP-0AP12, Palo Verde Action
Request Processing, Revision 3. An SRO evaluated the initial PVAR
documenting the ACTs and determined that no impact to plant safety existed, but
did not complete a review of each individual ACT in question. Subsequent to the
teams questioning, an SRO reviewed each ACT that documented a condition
adverse to quality. The ImPACT team subsequently informed the NRC team on
September 4, 2007, that none of the conditions adverse to quality identified in the
ACT database required further evaluation.
Analysis. The failure to implement the PVAR process for conditions adverse to
quality identified in the ACT database was a performance deficiency. The finding
is greater than minor because it is associated with the equipment performance
attribute of the mitigating systems cornerstone and affects the associated
cornerstone objective to ensure the availability, reliability, and capability of
systems that respond to initiating events to prevent undesirable circumstances.
Using the IMC 0609, Significance Determination Process, Phase 1 Worksheets,
the finding is determined to have very low safety significance (Green) because it
only affected the mitigating systems cornerstone and each of the ACT database
conditions adverse to quality were subsequently determined not to result in a loss
of safety function. The cause of this finding had crosscutting aspects associated
with decision-making of the human performance area in that licensee personnel
did not make safety-significant or risk-significant decisions using a systematic
process (H.1.(a)). The cause of the finding is also related to the safety culture
component of accountability in that management failed to reinforce safety
standards and display behavior that reflect safety as an overriding priority
(O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, Instructions,
Procedures, and Drawings, requires that activities affecting quality be prescribed
by instructions, procedures, or drawings, and be accomplished in accordance
with those instructions, procedures, and drawings. The evaluation of the need to
forward degraded or non-conforming conditions documented in PVARs to the
control room for OD or FAs was an activity affecting quality implemented by
Procedure 01DP-0AP12. Procedure 01DP-0AP12 required that a SRO evaluate
PVAR issues to determine whether a degraded or non-conforming condition
exists in an SSC subject to the OD or FA process. Contrary to the above,
between June 22 and September 4, 2007, licensee personnel did not assess
individual conditions adverse to quality documented in ACTs and attached to a
PVAR for the need to conduct an OD or FA. This example is of very low safety
significance and had been entered into the CAP as PVAR 3057126 and CRDR
- 38 - Enclosure
3058751. This was the second of eight examples associated with the licensees
failure to properly implement the OD program.
b.3 Failure to Implement Operability Determination Process for Spray Pond Missile
Hazards
Introduction. The team identified a third example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and
Drawings, for the failure of licensee personnel to follow procedures to evaluate
conditions adverse to quality for impacts on the operability of safety-related
equipment.
Description. On August 29, 2007, the team conducted an external walkdown of
Unit 1 with licensee personnel and identified approximately 20 unsecured metal
bars (severe weather missile hazards) near the Unit 1 essential spray pond
(ESP). Following prompting by the team, the licensee generated PVAR 3057285
on August 30, 2007, to address this condition.
The ESPs function as the ultimate heat sink. Spray headers, located above the
surface of the ESPs, are used to maintain design temperature within safety
analysis assumptions. There are no missile hazard ESP design features to
protect the spray headers from airborne missiles and, as a result, they are
vulnerable to airborne missiles generated during a high wind event. Procedure
81DP-0ZY01, Control of Potential Tornado Borne Missiles in the Outside Areas,
Revision 2, Section 1.1 stated the purpose of the procedure was to establish
administrative controls for using and storing items in outside areas so the risk of
losing the ESPs was within acceptable limits. Procedure 81DP-0ZY01, Appendix
E, Tornado Missile Density Criteria (Zones 1-14), identified the average density
limit at four missiles per 10,000 square feet (sqft) within a defined area around
the ESP. The unsecured transient missiles identified by the team were within
this defined area.
On August 30, 2007, a civil engineer conducted a tour of the area. PVAR
3057285 stated that the engineer determined that there was no operational
impact on the spray pond headers because the condition did not exceed the
operability basis of 4 missiles/sqft. This PVAR incorrectly referenced the
guidance from Procedure 81DP-0ZY01, did not address the fact that there were
more than 4 missiles, and contained no operations shift manager assessment of
the impact to the Unit 1 ESPs.
Procedure 40DP-9OP26, Revision 18, Operability Determination and Functional
Assessment, Section 3.1.1 stated that the OD process was entered upon
discovery of circumstances where operability of any SSC described in the
Technical Specifications was called into question upon discovery of a degraded,
nonconforming, or credible unanalyzed condition. The team noted that the
licensee did not enter the OD process on August 29, 2007, upon discovery of an
unanalyzed condition (unsecured, transient missiles near the Unit 1 ESP).
Procedure 40DP-9OP26, Section 1.3 stated that the immediate OD was
performed based on the best information available to on-shift personnel within a
relatively short time, typically on the order of two hours. In this case, neither
- 39 - Enclosure
engineering nor operations personnel notified the control room of the condition
when PVAR 3057285 was generated. Instead a work control SRO reviewed
PVAR 3057285 on August 31, 2007, and determined that a degraded condition
no longer existed because PVAR 3057285 stated the 20 transient missiles were
being removed and an analysis was completed satisfactorily.
Procedure 40DP-9OP26, Section 2.1 stated that the shift manager (SM) was
responsible for the OD decision. In this case, the Unit 1 SM was not notified of
the condition. PVAR 3057285 noted that the Unit 1 shift technical advisor, a non-
licensed operator, was notified of the civil engineering evaluation completed on
August 30, 2007, and that the 20 unsecured transient missiles would be removed
by August 31, 2007. However, the shift manager was not informed and no
assessment of operability was conducted.
Analysis. The failure to implement the OD process to assess the impact of the
unsecured, transient missiles on the operability of the Unit 1 ESP was a
performance deficiency. The finding is greater than minor because it is
associated with the external factors attribute of the mitigating systems
cornerstone, and impacted the cornerstone objective of ensuring the availability,
reliability, and capability of the ultimate heat sink to respond to initiating events.
Using the IMC 0609, Significance Determination Process, Phase 1 Worksheets,
the finding is determined to have very low safety significance (Green) because
the finding did not involve the loss of a safety function due to a severe weather
initiating event. The cause of this finding had crosscutting aspects associated
with decision making in the human performance area in that operations and
engineering personnel failed to use conservative assumptions for operability
decision-making when evaluating degraded and nonconforming conditions
(H.1.(b)). This finding also had a safety culture component aspect associated
with accountability in that workforce did not demonstrate a proper safety focus
and reinforce safety principles among peers (O.1.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures, and Drawings, requires that activities affecting quality be prescribed
by instructions, procedures, or drawings, and be accomplished in accordance
with those instructions, procedures, or drawings. The assessment of operability
of the Unit 1 ESP was an activity affecting quality and implemented by Procedure
40DP-9OP26. Procedure 40DP-90P26, Step 3.1.1 stated the OD process was
entered upon discovery of circumstances where operability of any SSC described
in the Technical Specifications was called into question upon discovery of a
degraded, nonconforming, or credible unanalyzed condition. Contrary to the
above, between August 29 and 31, 2007, licensee personnel failed to enter the
OD process upon discovery of circumstances where the operability of a
component described in the Technical Specifications was called into question.
Specifically, operations personnel did not implement the OD process described in
Procedure 40DP-9OP26 during the period from discovery of the issue to the
removal of the missiles from the ESP area. This was the third of eight examples
of the NCV associated with the failure to implement the OD program. This
example was of very low safety significance (Green) and documented in the
licensees CAP as PVAR 3057285.
- 40 - Enclosure
b.4 Failure to Evaluate Abnormally High Lead Levels in Low Pressure Safety
Injection Pump Bearing Oil
Introduction. The team identified a fourth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and
Drawings, for the failure of engineering personnel to determine the cause of an
abnormally high lead content in the Unit 3 low pressure safety injection (LPSI)
Pump Train B upper motor coupling bearing oil, to establish periodic monitoring
requirements, or to establish a lead content threshold upon which to take further
action on a degrading condition.
Description. On October 10, 2007, the team reviewed the OD associated with
the Unit 3 Train B LPSI Pump high lead levels (258 parts per million (ppm)),
which had existed in the upper motor coupling bearing oil since May 2006. This
coupling bearing was installed on all six LPSI pumps between 1995 and 2000.
The other five LPSI pumps at the site had not exhibited this condition and had oil
sample results of less than 1 ppm lead. The OD for this issue was documented
in CRDR 2896417.
During the initial investigation in May 2006, the Unit 3 Train B LPSI Pump
bearing oil was drained, flushed, and refilled with oil from a separate source. The
oil samples from the upper motor coupling bearing continued to show abnormally
high levels (242 ppm) of lead. The engineering evaluation concluded that there
should be no component materials in the pump assembly that contain lead.
Maintenance personnel determined that the parts used during the modification
were of the same type used for the other 5 LPSI pump modifications, whose
current oil samples showed lead levels to be less than 1 ppm. Oil chemistry
analysis determined that the lead particulates were relatively small and did not
detect any abnormal bearing wear metals. Also, the LPSI Pump Train B vibration
data remained within normal limits. On this basis, the licensee concluded the
Train B LPSI pump was operable and discontinued their investigation into the
cause of the high lead condition.
Engineering personnel determined that the expected lead content for the motor
coupling oil should be less than 1 ppm. The industry standard used in
determining precursor failure criteria assumed the oil environment contained less
than 10 ppm of lead content. The actual condition of the Unit 3 Train B LPSI
pump upper motor coupling bearing was approximately 242 ppm following the
drain, flush, and refill of the oil reservoir.
Procedure 40DP-9OP26, Revision 18, Section 1.3 stated that if a condition was
determined operable but degraded/nonconforming, then a PVAR will pursue the
appropriate corrective actions. The OD performed in May 2006 did not
determine a cause for this existing condition, did not develop a monitoring plan,
and did not develop a plan to take actions at predetermined thresholds in the
event of a further degradation in lead levels. In response to the teams
questions, the licensee initiated CRDR 3079670 on October 19, 2007, to
determine the source of the lead particles in the Unit 3 Train B LPSI upper motor
coupling bearing oil.
- 41 - Enclosure
Analysis. The failure to take measures to evaluate conditions adverse to quality,
to establish a monitoring program, or to establish a threshold of when to take
actions for a degrading condition was a performance deficiency. The finding is
greater than minor because it was associated with the equipment performance
attribute of the mitigating systems cornerstone, and impacted the cornerstone
objective of ensuring the availability, reliability, and capability of the LPSI system
to respond to initiating events to prevent undesirable consequences. Using the
IMC 0609, Significance Determination Process, Phase 1 Worksheets, the
finding is determined to have very low safety significance (Green) because the
finding did not result in an actual loss of Technical Specification equipment for
greater than the allowed outage time. The cause of this finding had crosscutting
aspects associated with corrective actions of the PI&R area because the licensee
failed to take appropriate corrective actions to address safety issues and adverse
trends in a timely manner (P.1.(d)). The cause of the finding was also related to
the safety culture component of accountability in that management failed to
reinforce safety standards and display behavior that reflected safety as an
overriding priority (O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures, and Drawings, requires that activities affecting quality shall be
prescribed by instructions, procedures, or drawings, and shall be accomplished
in accordance with those instructions, procedures, and drawings. The
assessment of operability of safety-related equipment needed to mitigate
accidents was an activity affecting quality, and was implemented by
Procedure 40DP-9OP26. Section 1.3 stated that if a condition was determined
operable but degraded/nonconforming, then a PVAR will pursue the appropriate
corrective actions. Contrary to this, between May 2006 and October 19, 2007,
the licensee did not initiate a PVAR or CRDR to pursue the appropriate actions
for a high lead content in the Unit 3 train B LPSI pump. Specifically, the licensee
had not determined the cause of abnormally high lead levels in the Unit 3 Train B
LPSI motor coupling bearing oil, did not establish a monitoring plan, and did not
establish thresholds to take additional actions upon a degrading condition. This
was the fourth of eight examples associated with the NCV involving inadequate
implementation of the OD program. This example was of very low safety
significance (Green) and was documented in the licensees CAP as PVAR
3075442.
b.5 Failure to Implement the Operability Determination Process on Unit 2 Essential
Cooling Water Heat Exchanger A Sleeve Adhesive
Introduction. The team identified a fifth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and
Drawings," for the failure of operations and engineering personnel to adequately
evaluate degraded and unanalyzed conditions to support operability decision
making associated with the Unit 2 essential cooling water (EW) Heat Exchanger
Train A epoxy sleeve adhesive degradation and leak. Specifically, on
October 23, 2007, operations and engineering personnel failed to consider all
relevant information to perform an adequate OD when evaluating Unit 2 EW Heat
Exchanger Train A sleeve adhesive under chemistry conditions associated with
the ESP system fouling identified in 2006.
- 42 - Enclosure
Description. The Unit 2 Train A EW Heat Exchanger developed a leak as noted
by elevated chlorides from the ESP into the EW system on June 27, 2007.
During a short notice outage on October 16, 2007, eddy current tests were
performed to determine and repair the source of the leak. Three tubes were
identified to be leaking, with location Row 2, Tube 26, found to have a leak
underneath the tube sleeve. After the source of the leak was identified,
operations and engineering personnel failed to validate the qualification of the
epoxy with respect to chemistry conditions associated with ESP fouling identified
in 2006. The epoxy was used to seal the EW heat exchanger tube sleeves into
the heat exchanger. All of the Unit 2 EW Heat Exchanger Train A tubes were
sleeved using the epoxy adhesive under limited design change package
2LM-EW-036. Unit 2 was the only unit to have sleeves inserted into the EW heat
exchanger tubes.
The leak was determined to be underneath the tube sleeve. The sleeve
adhesive was used to seal the sleeves to the heat exchanger tubes and to
prevent potentially corrosive water from causing leaks under the tube sleeves. In
response to the teams questions, the licensee initiated CRAI 3081800 on
October 23, 2007, to determine whether the sleeve adhesive was a potential leak
path under the Unit 2 EW Heat Exchanger Train A tube sleeves. However; no
OD of the condition was conducted.
The team reviewed Design Change Package 2LM-EW-036 and Combustion
Engineering Report TR-MCC-315, and determined the adhesive was tested
under design assumptions indicative of 1993 plant conditions. The adhesive was
not verified to perform under the chemistry conditions associated with the ESP
fouling concerns identified in 2006. ESP fouling came to the NRC's attention as
a result of unusual temperatures noted during a surveillance test of EDG 2B
conducted on May 17, 2006. The NRCs review was documented in NRC
Inspection Report 05000528, 05000529, 05000530/2006011. Significant
CRDR 2897810 documented changes made to ESP chemistry after the fouling
was identified, but no evaluation was documented on the potential effects of ESP
chemistry on the adhesive.
Procedure 40DP-9OP26, Step 3.1.1, stated that the OD process was entered
upon discovery of circumstances where operability of any SSCs described in
Technical Specifications was called into question upon discovery of a degraded,
nonconforming, or credible unanalyzed condition. Since a CRAI was written
without identification that a degraded or unanalyzed condition existed, the
adhesive concern did not receive an OD as required by Procedure 40DP-9OP26.
Per Procedure 01DP-0AP12, Palo Verde Action Request Processing, Revision
1, if additional work mechanisms changed the original degraded/non-conforming
evaluation, then the PVAR should be amended so that another degraded/non-
conforming evaluation can be performed.
After the team further questioned operations and engineering personnel,
PVAR 3083892 was initiated on October 26, 2007, and an immediate OD was
completed. The immediate OD evaluated the qualification of the adhesive used
to seal the U2 EW heat exchangers with respect to ESP fouling chemistry
conditions. Operations determined a reasonable expectation of operability of the
EW heat exchangers existed based on testing of the adhesive, no existing leaks
- 43 - Enclosure
under the remaining tube sleeves, and chemistry samples confirming no current
ESP leakage into the EW system.
Analysis. The performance deficiency associated with this finding was the failure
of operations and engineering personnel to adequately evaluate degraded and
unanalyzed conditions to support operability decision making associated with the
Unit 2 EW Heat Exchanger Train A epoxy sleeve adhesive degradation and leak.
This finding is greater than minor because it is associated with the mitigating
systems cornerstone attribute of equipment performance and affects the
cornerstone objective of ensuring the availability and reliability of systems that
respond to initiating events to prevent undesirable consequences. Using the
IMC 0609, "Significance Determination Process," Phase 1 Worksheets, the
finding is determined to have very low safety significance (Green) since it only
affected the mitigating systems cornerstone and did not represent a loss of
system safety function. The cause of this finding had crosscutting aspects
associated with decision making of the human performance area in that
operations and engineering personnel failed to use conservative assumptions for
operability decision-making when evaluating degraded and nonconforming
conditions (H.1.(b)). The cause of this finding was also related to the safety
culture component of accountability in that operations and engineering personnel
failed to demonstrate a proper safety focus and reinforce safety principles
(O.1.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures and Drawings," requires that activities affecting quality shall be
prescribed by instructions, procedures, or drawings, and shall be accomplished
in accordance with those instructions, procedures, and drawings. The
assessment of operability of safety-related equipment needed to mitigate
accidents was an activity affecting quality, and was implemented by
Procedure 40DP-9OP26. Procedure 40DP-9OP26, Step 3.1.1, stated the OD
process was entered upon discovery of circumstances where the operability of
any SSCs described in Technical Specifications was called into question upon
discovery of a degraded, nonconforming, or credible unanalyzed condition.
Contrary to the above, between October 23 and 26, 2007, operations and
engineering personnel failed to enter the OD process upon the discovery of
circumstances where the operability of a component described in Technical
Specifications was called into question. Specifically, operations and engineering
personnel failed to consider all relevant information to perform an adequate OD
when evaluating the Unit 2 EW Heat Exchanger Train A sleeve adhesive under
chemistry conditions associated with ESP fouling identified in 2006. This was the
fifth of eight examples of the NCV associated with inadequate OD program
implementation. This example was of very low safety significance and had been
entered into the CAP as PVAR 3083892.
b.6 Failure to Implement the Operability Determination Process on the Unit 2
Essential Cooling Water Heat Exchanger A Tube Leak
Introduction. The team identified a sixth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and
Drawings," for the failure of operations and engineering personnel to adequately
evaluate degraded and nonconforming conditions associated with a Unit 2 EW
- 44 - Enclosure
Heat Exchanger Train A tube leak. Specifically, between June 27 and
October 4, 2007, operations and engineering personnel failed to consider all
relevant information to perform an adequate OD when evaluating the Unit 2 EW
Heat Exchanger Train A tube leak.
Description. Unit 2 EW Heat Exchanger Train A developed a leak as seen by
elevated chloride concentrations in the EW system from the ESP system.
PVAR 3033604 was initiated on June 27, 2007. A control room review was
performed and the Unit 2 EW Heat Exchanger Train A tube leak was determined
to be bounded for leak rate and chloride concentration by a similar condition that
occurred on the Unit 3 EW Heat Exchanger Train B on June 28, 2001, where
operations personnel determined the condition did not impact operability.
A prompt OD was performed on June 29, 2007, in PVAR 3033604. The prompt
OD determined there was no impact on operability based on the heat exchanger
having adequate structural integrity, thermal performance, and spray pond
inventory. Thermal performance was determined to not be impacted by the leak
since Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance
Design Bases Analysis," Revision 8, assumed up to 257 of the 2575 tubes could
be plugged and only 30 tubes were currently plugged.
The team reviewed Calculation 13-MC-SP-0307 and determined that the
calculation assumed zero leakage of the heat exchanger tubes. Further, the
team determined the control room review and prompt OD only evaluated
chemistry concerns with respect to chloride concentrations. The team reviewed
Specification 74DP-9CY04, "Systems Chemistry Specifications," Revision 51,
and determined that other chemical constituents that are usually in the ESP
system were not evaluated for their effects on the EW system. These
constituents included dispersant, calcium hardness, and phosphate. The team
also noted that the prompt OD did not have acceptance criteria for when leakage
or chemistry parameters would render the Unit 2 EW Heat Exchanger Train A
The team determined operations personnel should have performed an immediate
OD on October 4, 2007, when the team questioned the validity of the initial OD.
Procedure 40DP-9OP26, Step 3.1.1, stated that the OD process was entered
upon discovery of circumstances where operability of any SSC described in the
Technical Specifications was called into question upon discovery of a degraded,
nonconforming, or credible unanalyzed condition.
After questioning by the team, PVAR 3033604 was redirected to the control room
for another immediate OD review on October 4, 2007. The immediate OD and
subsequent evaluation determined the current leak rate was 2.6 gallons per hour
and established a maximum acceptable leak rate of 3.3 gallons per hour, to
ensure chemistry parameters remained within specification in the EW system.
The evaluation also determined that the leak rate would not affect the structural
integrity or the heat removal design function based on the small size of the leak.
On October 16, 2007, the licensee plugged the leaking tubes.
Analysis. The performance deficiency associated with this finding was the failure
of operations and engineering personnel to adequately evaluate degraded and
- 45 - Enclosure
nonconforming conditions to support operability decision making associated with
the Unit 2 EW Heat Exchanger Train A tube leak. This finding is greater than
minor because it is associated with the mitigating systems cornerstone attribute
of equipment performance and affected the cornerstone objective of ensuring the
availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the IMC 0609, "Significance Determination
Process," Phase 1 Worksheets, the finding is determined to have very low safety
significance (Green) since it only affected the mitigating systems cornerstone and
did not represent a loss of system safety function. The cause of this finding had
crosscutting aspects associated with decision making in the human performance
area in that operations and engineering personnel failed to use conservative
assumptions for operability decision-making when evaluating degraded and
nonconforming conditions (H.1.(b)). The cause of this finding was also related to
the safety culture component of accountability in that operations and engineering
personnel failed to demonstrate a proper safety focus and reinforce safety
principles (O.1.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures and Drawings," requires that activities affecting quality be prescribed
by instructions, procedures, or drawings, and be accomplished in accordance
with those instructions, procedures, and drawings. The assessment of
operability of safety-related equipment needed to mitigate accidents was an
activity affecting quality, and was implemented by Procedure 40DP-9OP26,
Revision 18. Procedure 40DP-9OP26, Step 3.1.1, stated the OD process was
entered upon discovery of circumstances where operability of any SSC described
in the Technical Specifications was called into question upon discovery of a
degraded, nonconforming, or credible unanalyzed condition. Contrary to the
above, between June 27 and October 4, 2007, operations and engineering
personnel failed to enter the OD process upon discovery of circumstances where
the operability of a component described in the Technical Specifications was
called into question. Specifically, operations and engineering personnel failed to
consider all relevant information to perform an adequate OD when evaluating the
Unit 2 EW Heat Exchanger Train A tube leak. This was the sixth of eight
examples associated with the NCV involving inadequate implementation of the
OD program. This example was of very low safety significance and had been
entered into the CAP as PVAR 3033604.
b.7 Observations and Minor Noncited Violations Involving Licensee Controls for
Identifying, Assessing, and Correcting Performance Deficiencies
b.7.1 Corrective Action Program Implementation
Description: The team reviewed CAP implementation and identified the
following minor issues/observations:
During the week of October 2, 2007, the team noted that licensee
personnel consistently failed to recognize conditions under which a PVAR
would be required to document an adverse condition. Licensee
personnel believed they needed to ensure that a degraded condition was
a condition adverse to quality before they would consider initiating a
PVAR. Throughout the inspection, the team continued to prompt the
- 46 - Enclosure
licensee on initiating PVARs. The team noted improved performance by
licensee personnel late in the inspection. However, the team could not
conclude whether this was an artifact of the team being onsite or whether
this would result in sustained improvement.
The team noted that the CDBR team documented issues at an
appropriate threshold. However, the team also noted that engineering
personnel incorrectly considered that the CDBR team was entering issues
into the PVAR process that they considered below threshold or not worthy
of review.
The team reviewed the quality of ODs to evaluate the effect of degraded
conditions on safety-related equipment. The team also reviewed
degraded and nonconforming conditions for which the licensee had not
conducted any assessment of operability. In addition to the examples
discussed on ODs in this report, the team noted a generally poor
understanding of the insights necessary to conduct operability
assessments of degraded conditions and a failure to recognize the need
to conduct an operability evaluation. The team also noted failures to
recognize the need to conduct an extent of condition review for identified
degraded conditions. Poor operability assessments and program
implementation have been a longstanding concern at Palo Verde.
The team identified that corrective actions for conditions adverse to
quality were not always timely or were not completed. For example, the
team identified that corrective actions to train personnel on apparent
cause evaluations, which was a concern during the December 2006 NRC
PI&R inspection, were still not completed in November of 2007. The
licensee believed corrective actions to conduct 10 CFR 50.59 training for
chemistry personnel were completed in November of 2006. However, the
team determined that some chemistry personnel had not attended the
required training and even though CRAI 2942350 was closed.
On October 4 and 9, 2007, the team observed Corrective Action Review
Board (CARB) meetings and noted the following observations; the CARB
meeting was frequently interrupted, management personnel did not
appear prepared for or dedicated to the CARB meeting and frequently left
the meeting to answer cell phone and pager calls, the quorum was lost
when the minimum number of managers required was not maintained as
personnel left the meeting, and the meeting was cut short or cancelled
due to the number of distractions or due to other meetings considered to
have a higher priority. The team noted that the CARB members did not
challenge the disruptive behaviors and did not hold themselves
accountable for their participation in the meeting.
b.7.2 Problem Identification and Resolution Root Cause Report
Description: The team reviewed the PI&R Root Cause Report issued in
August 2007. The team noted the following weaknesses in the PI&R
report:
- 47 - Enclosure
On July 9, 2007, the licensee initiated CRAI 3038014, a corrective action
to prevent recurrence (CAPR), for the root cause of the failure to correct
continued poor accountability behaviors with implementation of the CAP.
As of November 2, 2007, CRAI 3038014 was not completed. The PI&R
root cause CAPR was to conduct a site wide stand-down in order to
communicate CAP fundamentals to station personnel, managers, and
supervisors. The PI&R root cause report CAPR defined the fundamentals
that needed to be communicated and specified the forum in which to
communicate the fundamentals (site wide stand-down); however, the
assigned CAPR completion date was December 28, 2007. The team
noted that this action was untimely considering that the Unit 3 refueling
outage was scheduled to start in October 2007. The team did note that
limited CAP discussions were conducted by site senior management
during weekly video presentations leading up to the Unit 3 refueling
outage; however, the discussions did not include all of the CAP
fundamentals described in the PI&R root cause.
The team noted that the PI&R root cause report discussed the lack of
Specific, Measurable, Achievable, Reasonable, and Timely (SMART)
corrective action criteria in CAP procedures and prior root cause reports.
The team recognized that the PI&R root cause report contained CRAI
3038040 to identify SMART criteria in the condition reporting procedure
and in the root cause evaluation manual. However, the teams review of
the corrective actions identified in the PI&R root cause report noted a
similar lack of SMART criteria (CRDR 3071645) in the PI&R root cause
report corrective actions. In general, the team noted that the PI&R root
cause report corrective actions (e.g., communication of CAP standards
and fundamentals) were not timely in consideration of the existing
weaknesses in the CAP. Also, the team noted that the continuing
problems identified with the OD process that have been identified by the
NRC over the last several years, and which continued to occur during this
inspection, were not discussed in any great detail in the PI&R root cause.
The only PI&R root cause report corrective action related to this program
was to conduct a self-assessment of the OD program by June 30, 2008.
The team did not consider this action timely given the problems identified
with the implementation of the OD process.
The team noted that the PI&R root cause report described the CAP as
comprising the PVAR, CRDR, corrective maintenance program,
engineering deficiency work process, OD and FA evaluations, and the
warehouse discrepancy notice program. However, the PI&R root cause
report did not recognize that the existence of this many tracking systems
had contributed to the complexity of the licensees CAP; thereby, creating
vulnerabilities to CAP implementation. This is consistent with the results
of interviews conducted during the inspection which identified that
licensee personnel did not see a difference between their multiple
database process and the more prevalent nuclear industry one form
process. In addition, the PI&R root cause did not recognize the existence
of other tracking systems (such as the ACT and Bechtel NCR databases)
which potentially included multiple unrecognized conditions adverse to
quality outside of the defined CAP.
- 48 - Enclosure
In March 2005, Palo Verde initiated significant CRDR 2780286 to perform
a root cause investigation of the substantive crosscutting issues in PI&R.
The identified root cause was management behaviors, in that they did not
hold themselves and others to high standards relative to the CAP. The
CAP substantive crosscutting area self assessment performed in
preparation for the ImPACT in 2007 determined that a new root cause
analysis did not need to be conducted, primarily because significant
CRDR 3015327 was already in progress to determine why the corrective
actions from CRDR 2780286 had not been effective. The identified root
cause in CRDR 3015327 was inadequate personnel and organizational
accountability. The evaluation determined that many of the CAPRs and
corrective actions implemented by CRDR 2780286 were conceptual,
poorly conceived, and did not follow the SMART model. Consequently,
they were not effectively implemented. Examples included CRAI
2828390 (revise the Palo Verde Business Plan to reflect the CAP as a
strategic focus area), CRAI 2828392 (develop improved CAP metrics),
and CRAI 2828404 (revise the Palo Verde expectations and standards
booklet to include the CAP). The team determined that the ineffective
corrective actions from CRDR 2780286 had not been incorporated into
the SIBP/SIIP. The team reviewed the SIBP/SIIP and determined that the
corrective actions for CRDR 3015327 had been incorporated. Because
the SIBP/SIIP was in draft form, and many of the proposed actions had
not yet been implemented, the team was unable to evaluate whether the
actions will be effective in correcting the PI&R issues the site is
experiencing.
The team reviewed a number of other root cause reports and noted
similar issues including; the failure to identify all contributing causes, the
failure to specify SMART corrective actions, a lack of timely corrective
actions, an inability to track the completion of or determine the status of
corrective actions taken in response to significant conditions adverse to
quality, and the closure of corrective actions taken in response to
significant conditions adverse to quality that had not been implemented or
completed.
b.7.3 Action Request Review Committee
Description: The team attended several Action Request Review
Committee (ARRC) meetings. The ARRC was established following the
implementation of the PVAR process to review and disposition each
PVAR to implement an effective CAP. The team noted the following
weaknesses in the conduct of the ARRC activities:
- The team noted that the ARRC members frequently debated whether
a given condition documented in a PVAR was actually an adverse
condition. One ARRC member commented that if the subject
condition was considered adverse, Then we would have hundreds of
adverse conditions. The team noted that an adverse condition
should be judged as adverse based on its characteristics, not whether
it would subsequently result in a high number of adverse conditions
being documented.
- 49 - Enclosure
- The team observed that ARRC members would call personnel in the
field to resolve a degraded condition and would then close the PVAR
to actions taken. The team noted that this had the appearance of the
ARRC acting as first line supervisors to correct conditions adverse to
quality rather than as a multi-discipline team to review and disposition
PVARs for corrective actions by responsible organizations.
- ARRC members were observed to be rewriting PVARS rather than
returning them to the initiating organization. This prevented the
initiating organization from learning from the lack of a complete PVAR
description and precluded the originating organization (i.e., the
organization in the know) from providing the most accurate
information regarding the condition.
- An ARRC member was observed to be overly biased against a PVAR
that he considered should not have been written and stated to the
group that he would handle this particular PVAR, and that he would
tell the originator that this was not a problem. It was apparent to the
team that the originator would receive negative feedback on the
generation of this PVAR from the ARRC member rather than allowing
the PVAR process to evaluate and resolve the condition. The team
also noted that the other ARRC members did not intercede, allowing
this negative behavior to continue.
- The ARRC could determine no corrective actions were necessary by
designating a Review CRDR with no actions needed. The ARRC
also appeared to be conducting evaluations and specifying corrective
actions for PVAR issues. The team noted that this could put the
ARRC in the position of specifying corrective actions rather than
dispositioning PVARs to the responsible organization for review and
created a vulnerability to bypassing organizational processes for
evaluating conditions adverse to quality.
The team determined that the management oversight provided to the
ARRC, a relatively new review committee, was insufficient given the
number and depth of NRC observed concerns. The team discussed
these ARRC observations with the Performance Improvement and CAP
managers. In response to these concerns, the licensee initiated PVAR
3072299 and an ARRC improvement strategy was generated. The
ARRC Charter was revised, some ARRC members were reassigned, new
members were designated, and briefings were conducted with ARRC
members on the vision and expectations of the ARRC. The team noted
some improvement following these actions; however, the team also noted
some of the poor behaviors were repeated during subsequent ARRC
sessions.
- 50 - Enclosure
b.7.4 Backlog Review
Description: The team reviewed the licensees efforts in defining and
evaluating the existing backlog and had the following observations:
The team noted that there were over 250 OD backlog entries. The
characterization of this many ODs as part of a backlog could be confusing
since open ODs generally documented current degraded or
nonconforming equipment conditions that had been evaluated as not
affecting the ability of equipment to meet intended safety functions, but
that had not yet been corrected. At Palo Verde, ODs were kept open,
even if full qualification was restored, until all associated corrective
actions had been completed. The team noted that this approach may
dilute the significance of how issues documented under the OD process
were viewed and could confuse the organization and impact the ability to
effectively evaluate the aggregate impact of degraded and nonconforming
conditions on plant equipment.
The licensees backlog review team identified that items in the activity
tracking (AT) database had a low priority review need because ATs, did
not perform physical work. The team identified that some AT entries
appeared to perform physical work, such as AT work order (WO) 220774,
which required vibration readings to be taken on plant equipment.
Following the teams observations, the backlog review team reassessed
their decision not to review ATs. On October 31, 2007, the licensee
identified approximately 54 out of 3901 AT WOs that appeared to perform
physical work. The licensee determined that several of the items should
not have been entered into the AT database. No degraded or non-
conforming conditions were identified which would have affected safety-
related or other plant equipment. The team noted that the decision to not
review ATs assumed proper implementation of licensee programs and
processes and that prior decisions were valid. The apparent
unwillingness of licensee personnel to question decisions made during a
period of declining performance was a significant vulnerability for the
licensee. As noted during the SIBP/SIIP review, the licensee had not
developed any actions to evaluate the legitimacy of past decisions.
PVAR 3074083, CRDR 3079482, and CRAI 3079483 were generated to
document this issue.
The team discussed the status of the ACT database review with the
backlog review team. The backlog review team indicated that they were
nearing completion and that they were verifying whether the ACTs of
concern were in fact conditions adverse to quality. The team noted that
the backlog review team appeared to be spending an inordinate amount
of time verifying whether they considered a given ACT concern to be an
issue adverse to quality rather than initiating a PVAR and letting the CAP
determine the significance and required corrective actions.
- 51 - Enclosure
b.7.5 Self Assessments
Description: The team reviewed a number of self-assessments and had
the following observations:
A significant number of self-assessments conducted by Palo Verde
personnel lacked depth and did not challenge the assessed organization.
The recommendation for the November 2006 decision-making self-
assessment was vague because it only requested an Operational
Decision Making Instruction (ODMI) review and provided no further
details on current ODMI weaknesses. The only recommendation from the
December 2005 Operational Decision Making self-assessment was to
combine two procedures. The March 2007 work management self-
assessment concluded that the assessment needed to be re-performed
later in 2007 and provided no other insights. The self-assessment of the
maintenance rule program did not recognize that unavailability and
reliability performance criteria could not be validated and that numerous
systems had non-conservative performance criteria.
Self-assessment corrective actions were not always tracked nor did they
always have PVARs written to document the expected corrective actions.
The December 2006 leadership self-assessment recommended the
initiation of a mentoring program that was later postponed several
months. The decision was influenced by the upcoming change in senior
management. Deficiencies described in the assessment of the safety
injection system and environmental qualification assessments were not
entered into the CAP.
In one case, the team noted that a recent training assessment appeared
to be more probing and insightful. The team observed that the makeup of
the training self-assessment team included a mix of licensee and industry
personnel which may have led to the better assessment product when the
experiences of industry personnel were used.
5.2 Design
Weak engineering program and process implementation had been a continuing problem at
Palo Verde. The team noted numerous instances of design errors and omissions, and an
overall lack of technical rigor. Specifically:
- The team noted that the CDBR effort was effective in identifying design issues. The
composition of the group included both site engineering and contractor support. The
success of this effort could be attributed to the broader perspective that the group had
due to the external contractor support. Although the CDBR effort had identified issues
at the appropriate threshold, the team noted instances in which issues entered into the
CAP were not appropriately addressed. The team also noted that a cumulative impact
review of all of the CDBR issues for a particular system or component could further
reduce the available margin.
- The team noted several design documents had inadequate or unverified design
assumptions. For example, Calculation13-MC-SP-306, "MINET Hydraulic Analysis of
- 52 - Enclosure
SP System," Revision 4, stated values for essential spray pond net positive suction
head and submergence requirements to prevent vortexing, but the values were for
generic pump design and did not ensure operation under the specific PVNGS design
basis conditions, such as worst case ESP temperature.
- The team noted that the engineering organization lacked a consistent questioning
attitude. Reviews and evaluations often addressed the simplest or primary causes
only. Extent of condition reviews, operability evaluations, and conditions dealing with
off-normal operations were frequently not well documented. When questioned by the
team, engineering personnel needed to perform further evaluation and documentation
to support the technical position.
a. Inspection Scope
The team reviewed licensing and design basis documents for safety injection, ESP,
and auxiliary feedwater (AF) systems, including the UFSAR, calculations, engineering
analyses, system descriptions, CDBR reports, and self assessments to determine the
functional requirements of the systems for normal, abnormal, and accident conditions.
The team reviewed a sample of risk-significant plant modifications for the selected
systems, including those that involved vendor supplied products and services to
determine whether the changes had an adverse impact on the ability of the systems to
perform their design basis functions and determine whether the changes would result
in an unexpected initiating event. During this review, the team evaluated the
effectiveness of the licensee in controlling design and licensing information, in
providing necessary calculations to support plant changes, and in developing and
implementing thorough post-modification testing procedures. The team assessed the
adequacy of the licenses engineering products in evaluating applicable system and
support system design attributes and regulatory requirements.
The team conducted general walkdowns of the selected systems and components.
Recent changes to plant maintenance and operating procedures were reviewed to
ensure that they did not result in inadvertent design changes to the systems. For
procedures that involved design changes, the team ensured that the change was
subjected to the appropriate design change processes, including a review in
accordance with 10 CFR 50.59, Changes, Tests, and Experiments. The team also
reviewed a sample of PVARs to assess the effectiveness of corrective actions for
deficiencies involving design activities. Additionally, the team reviewed a sample of
engineering training programs to verify that training programs were consistent with the
current design.
b. Findings and Observations
b.1 Failure to Implement Adequate Design Controls for Condensate Storage Tank
Temperature
Introduction. The team identified a Green NCV of 10 CFR Part 50, Appendix B,
Criterion III, "Design Control," for the failure of engineering personnel to translate
design basis maximum condensate storage tank (CST) temperature
requirements into procedures to ensure the plant is operated within its design
basis.
- 53 - Enclosure
Description. On October 4, 2007, the team questioned engineering personnel
with regards to the control of the maximum CST temperature. A CST maximum
temperature of 120°F was used in Calculation 13-MC-CT-0205, "Condensate
Storage Tank," Revision 4, Calculation 13-MC-CT-0307, "CST Minimum Level
Setpoint," Revision 4, and Calculation 13-MC-AF-0309, "AF Hydraulic
Calculation for Q-Trains," Revision 7, to ensure sufficient CST volume and net
positive suction head for the AF pumps during a design basis accident. Neither
operations nor maintenance and testing personnel took routine recordings of
CST temperature, the parameter was not monitored by Technical Specifications,
and no alarm existed for high CST temperature to ensure operation within the
design basis maximum temperature of 120°F.
The 120°F CST maximum temperature was based on summertime ambient
weather conditions affecting water temperature. The team noted that hotwell
condensate from the main condenser was rejected to the CST during startup,
shutdown, and on a high hotwell level. When the hotwell was rejected to the
CST, the potential existed to exceed the 120°F maximum temperature limit
because the condensate average temperature during July and August 2007 was
130°F.
Following the teams questions on control of CST temperature, engineering
personnel initiated PVAR 3073243. Operations personnel determined this
condition was not a degraded or nonconforming condition, and an immediate OD
was not performed due to current ambient temperatures being significantly lower
than the maximum tank temperature, and due to establishing compensatory
measures through a night order on October 11, 2007. The night order identified
the deficiencies in monitoring CST temperature and directed operations
personnel to take CST temperature readings once per shift, and contact system
engineering personnel if temperature exceeded a lower administrative limit of
110°F.
On November 13, 2006, PVAR 2949167 was written to evaluate how AF pump
heat load contributions were not considered in determining maximum CST
temperature. The team determined that the failure to consider other inputs that
could raise CST temperature during the licensees review of PVAR 2949167 was
a missed opportunity.
Analysis. The performance deficiency associated with this finding was the failure
of engineering personnel to adequately translate the design basis CST maximum
temperature requirements into applicable procedures. This finding is greater
than minor because it is associated with the mitigating systems cornerstone
attribute of equipment performance and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events
to prevent undesirable consequences. Using the IMC 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have
very low safety significance since it only affected the mitigating systems
cornerstone and did not represent a loss of system safety function. The cause of
this finding had crosscutting aspects associated with corrective action of the
PI&R area in that engineering personnel failed to thoroughly evaluate problems
such that resolutions ensured that the problems were resolved. (P.1.(c)).
- 54 - Enclosure
Enforcement. 10 CFR Part 50, Appendix B, Criterion III, Design Control,
requires, in part, that the design basis for SSCs be translated into specifications,
drawings, procedures, and instructions. Contrary to the above, since 1985,
engineering personnel failed to correctly translate design basis information into
specifications, drawings, procedures, and instructions. Specifically, engineering
personnel failed to translate design basis maximum CST temperature
requirements into procedures to ensure the plant is operated within its design
basis. This example was of very low safety significance and was entered into the
CAP as PVAR 3073243, this violation was treated as an NCV consistent with
Section VI.A of the Enforcement Policy: NCV 05000528, 05000529,5000530/2007012-02, Failure to Implement Adequate Design Controls.
b.2 Inadequate Installation of Fire Sprinklers
Introduction. The team identified a Green NCV of License Condition 2.C(6) for
the failure to install sprinkler heads in accordance with the FP program.
Specifically, on October 2, 2007, the team identified several upright fire sprinkler
heads in the auxiliary building that were incorrectly installed in a pendent or
downward orientation.
Description. During walkdowns of the Unit 3 auxiliary building high pressure
safety injection Train A pump room, the team identified that a FP sprinkler was
installed in the wrong orientation. The sprinkler was located in a drop line for
coverage below a heating ventilation and air conditioning unit and above cable
Tray 3EZACCATCBA. The sprinkler head was an upright style; however, the
sprinkler head was installed in a downward orientation. The team also identified
that the sprinkler head in an alcove area on the 40 foot elevation of the LPSI
pump room was installed in the incorrect orientation.
The team questioned engineering personnel on the orientation of these sprinkler
heads. License Condition 2.C(6), "Fire Protection Program," stated that the
licensee shall implement and maintain in effect all provisions of the approved FP
program as described in the UFSAR for the facility, as supplemented and
amended, and as approved in the Safety Evaluation Report (SER) through
Supplement 11, subject to the following provision: the licensee may make
changes to the approved FP program without prior approval of the Commission
only if those changes would not adversely affect the ability to achieve and
maintain safe shutdown in the event of a fire.
UFSAR Section 9.5.1.2.1.F stated that automatic preaction sprinklers,
hydraulically designed using National Fire Protection Association (NFPA)
Pamphlet No. 13 (1976) as guidance, are provided to protect the areas so
indicated in Table 9.5-1. Each automatic preaction system contains piping
supervised by service air and fusible link sprinkler heads arranged such that flow
densities meet the guidelines of the American Nuclear Insurer, and also NFPA
Pamphlet No. 13 (1976). NFPA Pamphlet No. 13 (1976) Section 3-15.2.2 stated
that the character of the discharge of sprinklers is such that it is necessary to use
two distinct designs, one approved for the upright and the other approved for the
pendent position.
- 55 - Enclosure
The team determined the three listed upright type sprinkler heads were found
installed in a downward position. In the installed configuration, there was no
testing to demonstrate that sprinklers would be capable of achieving the required
flow or densities. Engineering personnel initiated PVAR 3073824 to address
these issues.
Analysis. The performance deficiency associated with this finding was the failure
to install sprinkler heads in accordance with the FP program. This finding is
greater than minor because it was associated with the mitigating systems
cornerstone attribute of external factors and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events
to prevent undesirable consequences. Using the IMC 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding was determined to
require additional evaluation under Manual Chapter 0609, Appendix F, "Fire
Protection Significant Determination Process," because it was associated with
the suppression element of defense-in-depth. Since the installation of the
sprinkler heads represented a low degradation of the fire suppression system, in
accordance with Section 1.3.1 of IMC 0609, Appendix F, the finding is
determined to have very low safety significance.
Enforcement. License Condition 2.C(6), Fire Protection Program, stated that
the licensee shall implement and maintain in effect all provisions of the approved
fire protection program as described in the Final Safety Analysis Report for the
facility, as supplemented and amended, and as approved in the safety evaluation
report through Supplement 11, subject to the following provision: the licensee
may make changes to the approved fire protection program without prior
approval of the Commission only if those changes would not adversely affect the
ability to achieve and maintain safe shutdown in the event of a fire. UFSAR
Section 9.5.1.2.1.F stated that automatic preaction sprinklers, hydraulically
designed using NFPA Pamphlet No. 13 (1976) as guidance, are provided to
protect the areas so indicated in Table 9.5-1. Each automatic preaction system
contains piping supervised by service air and fusible link sprinkler heads
arranged such that flow densities meet the guidelines of the American Nuclear
Insurer, and also NFPA Pamphlet No. 13 (1976). NFPA Pamphlet No. 13 (1976)
Section 3-15.2.2 stated that the character of the discharge of sprinklers is such
that it is necessary to use two distinct designs, one approved for the upright and
the other approved for the pendent position. Contrary to the above, as of
October 2, 2007, three listed upright type sprinkler heads were found in the
untested pendent position. Because the finding was of very low safety
significance and was entered into the CAP as PVAR 3072557, this violation was
treated as an NCV, consistent with Section VI.A of the Enforcement Policy:
NCV 05000530/2007012-03, "Inadequate Installation of Fire Sprinklers.
b.3 Failure to Enter Environmental Qualification (EQ) Self Assessment Deficiencies
into the Corrective Action Program
Introduction. The team identified an example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, for the failure of
engineering personnel to promptly identify and correct a significant condition
adverse to quality described in an environmental qualification self assessment
- 56 - Enclosure
report. Specifically, the licensee had not evaluated or removed unqualified tape
used to repair Anaconda conduit from the containment buildings.
Description. EQ Self Assessment No. 2957427, issued July 2, 2007, found that
Engineering Change Evaluation (ECE), ECE-ZZ-A143, Anaconda Degraded
Sealtite Repair Material, Scotch 33 Tape, Revision 1, was used as a basis for
the prompt OD for degraded Anaconda Sealtite flexible conduit (CRDRs
2940338, 2940354, and 2940359). The ECE did not address the worst-case in-
containment radiation dose. Under the worst case radiation levels, the tape was
calculated to be exposed to the combined normal, accident gamma, and accident
beta of over 300 Mrad. However, the ECE only evaluated the tape up to
radiation levels of 100 Mrad. Although the condition was identified in the self
assessment, it was not entered into the CAP and evaluated as a condition
adverse to quality. Based on concerns raised by the team, PVAR 3073528 was
written to evaluate why an adverse condition was not dispositioned properly in
the CAP and to evaluate the extent of condition for other issues in the EQ self
assessment.
The team was also concerned that the failure of the tape during an accident
could also result in the failure of the repaired flexible conduit. The additional
debris caused by this condition would contribute to containment sump loading. In
response, engineering personnel initiated PVAR 3071831, to evaluate the
potential impact of the additional tape and conduit sheathing loading on the
containment sump. Since Unit 3 was in a refueling outage at the time of
discovery and not impacted by the condition, engineering personnel evaluated
the impact of current operability on Units 1 and 2. Approximately six months
prior to the NRC team identifying the concern, Palo Verde replaced the Unit 1
sump strainers. The new Unit 1 strainers size was increased from 210 square
feet to 3142 square feet. Since Unit 2 was still configured with the smaller
strainers, engineering personnel evaluated this as the bounding condition. In
their evaluation, engineering personnel estimated that there would be
approximately 45 square feet of additional loading on the containment sump
strainers and concluded that there was still adequate margin for operation.
Subsequent to this evaluation, Unit 1 experienced a forced outage. On
October 26, 2007, as part of work Order 3034098, the licensee conducted a
containment walkdown to quantify and remove susceptible tape and flexible
conduit in containment. The licensee estimated that there was in excess of 600
square feet of combined tape and conduit that had not been accounted for in the
sump loading analysis and initiated PVAR 3083224, to evaluate the condition.
The licensee concluded that with the larger strainers, the additional loading
would have little impact.
The licensee conducted additional analyses to evaluate the past operability of the
strainers in the Unit 1 containment. The licensee evaluated the realistic radiation
dose that the 639 square feet of tape and conduit outside the bio-shield wall
would be exposed to and determined that it was substantially below the qualified
rating of 100 Mrads. Specifically, the realistic accident total integrated dose (TID)
within containment (wetted or dry but not submerged) during a loss-of-coolant-
accident was calculated to be approximately one-fifth of the TID values reported
in the bounding calculation of record 13-NC-ZC-105, Revision 9, or 58 Mrads.
- 57 - Enclosure
The 148 square feet of tape and flex conduit material found within the bio-shield
in Unit 1 also exceeded previous estimates. Generally, material within this zone
was more of a concern for containment sump strainer loading because it was
assumed that all material within the high energy break zone of influence would
be destroyed and potentially transported to the sump. Consistent with the
approach used for assessment of other potential debris source terms,
engineering personnel conducted a review of the tapes physical properties and
established that the specific gravity for the tape was approximately 1.3.
Therefore, the debris generated within the bio-shield wall may be transported out
of the steam generator compartment, but would have sufficient time to settle prior
to realignment of the ECCS pump suctions to the containment sump.
Additionally, most, if not all, of the material deposited outside the steam
generator compartment would remain submerged and in place since the
maximum flow velocities in and around this area were below the minimum
velocity required for incipient motion of the debris.
The team determined that since the actual TID was less than the qualification
rating for the tape outside the bio-shield wall, it would likely maintain its integrity
and not fail as a result of realistic radiation exposure. In addition, for conditions
in which the additional materials could be susceptible to high energy line break
effects, the specific characteristics of the material, transport velocities, and actual
location precluded any significant challenge to the containment sump loading
assumptions.
Analysis. The performance deficiency associated with this finding was the failure
to enter a condition adverse to quality into the CAP. This finding is greater than
minor because it is associated with the mitigating systems cornerstone attribute
of equipment performance and affected the cornerstone objective of ensuring the
availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the IMC 0609, "Significance Determination
Process," Phase 1 Worksheets, the finding is determined to have very low safety
significance (Green) since it only affected the mitigating systems cornerstone and
did not represent a loss of system safety function. The cause of this finding had
crosscutting aspects associated with self assessment of the PI&R area in that the
licensee did not follow their benchmarking and self assessment guide to ensure
findings were evaluated in the CAP (P.3(c)). The cause of the finding was also
related to the safety culture component of accountability in that management
failed to reinforce safety standards and display behavior that reflected safety as
an overriding priority (O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action,
requires that measures be established to assure that conditions adverse to
quality are promptly identified and corrected. Contrary to the above, between
July 2 and October 4, 2007, the licensee did not assure that conditions adverse
to quality were promptly identified and corrected. Specifically, conditions adverse
to quality identified in EQ Self Assessment No. 2957427 were not entered into
the CAP or corrected in a timely manner. Because the finding was of very low
safety significance and was entered into the CAP as PVARs 3073528, 3071831,
and 3083224, this violation was treated as an NCV, consistent with Section VI.A
of the Enforcement Policy: NCV: 05000528, 05000529,05000530/2007012-04,
Six Examples of the Failure to Implement Corrective Action Program
- 58 - Enclosure
Requirements. This was the first of six examples of the failure to implement the
corrective action program requirements.
b.4 Failure to Implement Corrective Actions for Operating Experience Involving the
Turbine Driven Auxiliary Feedwater Pump Trip and Throttle Valve
Introduction. The team identified a second example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action," for the failure of
engineering personnel to identify, evaluate, and correct degraded and
nonconforming conditions associated with OE applicable to the AF pump trip and
throttle valve (T&TV). Specifically, between February 8 and October 2, 2007,
engineering personnel did not enter applicable OE on the mechanical overspeed
trip mechanism for the AF turbine T&TV into the CAP.
Description. On February 8, 2007, system engineering reviewed industry OE
from South Texas (OE24167) and Saint Lucie (OE24002) in order to determine
the applicability to Palo Verde. The OE described failures of the turbine driven
AF pump T&TV's mechanical overspeed trip mechanism to trip on demand due
to rust forming on mating surfaces between the trip-hook and latch-up lever.
System engineering determined this OE was applicable to PVNGS and that
current preventative maintenance (PM) tests would not detect this failure.
On February 8, 2007, engineering personnel initiated ACT 3046427 to
incorporate force measurements needed to trip the T&TV into the existing
overspeed trip linkage PM tests. The OE review was documented in the January
to June 2007, AF system health report. The team determined engineering
personnel should have entered Procedure 65DP-0QQ01, "Industry Operating
Experience Review," Revision 13, which stated that ACTs can be used to track
industry OE when related actions are not corrective or adverse in nature. The
team questioned whether OE that was determined to be applicable to the site
and where current PMs could not detect the failure should be entered into the
CAP, not the ACT process. After further review by engineering personnel, the
licensee determined that a PVAR should have been written instead of an ACT,
and an OD should have been performed.
The assessment of operability of safety-related equipment needed to mitigate
accidents was an activity affecting quality, and was implemented by
Procedure 40DP-9OP26, "Operability Determination and Functional
Assessment," Revision 18. Procedure 40DP-9OP26, Step 3.1.1, stated that the
OD process is entered upon discovery of circumstances where operability of any
SSCs described in Technical Specifications is called into question upon
discovery of a degraded, nonconforming, or credible unanalyzed condition.
Since an ACT was written instead of a PVAR, the OE on the AF Pumps T&TVs
did not receive an OD as required by Procedure 40DP-9OP26.
In response to the teams observations, on October 2, 2007, engineering initiated
PVAR 3070597 to address the potential for the turbine driven AF pump T&TV's
mechanical overspeed trip mechanism to fail to trip on demand due to rust
forming on mating surfaces between the trip-hook and latch-up lever. Operations
personnel performed an immediate OD and noted that a reasonable expectation
of operability existed because the T&TVs were in a less harsh environment than
- 59 - Enclosure
Saint Lucie and South Texas and had not experienced the rust problems seen at
those facilities. The licensee changed ACT 3046427 to CRAI 3072364 to ensure
the item was entered into the CAP. CRAI 3072364 was initiated to include steps
in work order WSL245709 to ensure the T&TV trip levers trip at a value less than
25 pounds force, as specified in (EPRI) Manual, "Terry Turbine Maintenance
Guide AFW Application." Engineering management also provided additional
training to engineering personnel on the differences between when to initiate an
ACT and when to initiate a PVAR.
Analysis. The performance deficiency associated with this finding was the failure
of engineering personnel to adequately evaluate degraded and nonconforming
conditions to support operability decision making associated with OE applicable
to AF Pump T&TV. This finding is greater than minor because it is associated
with the mitigating systems cornerstone attribute of equipment performance and
affects the cornerstone objective of ensuring the availability and reliability of
systems that respond to initiating events to prevent undesirable consequences.
Using the Manual Chapter 0609, "Significance Determination Process," Phase 1
Worksheets, the finding is determined to have very low safety significance
(Green) since it only affected the mitigating systems cornerstone and did not
represent a loss of system safety function. The cause of this finding had
crosscutting aspects associated with OE of the PI&R area in that engineering
personnel failed to ensure implementation and institutionalization of OE through
changes to station processes, procedures, equipment, and training programs
(P.2.(b)). The cause of this finding was also related to the safety culture
component of accountability in that engineering personnel failed to demonstrate
a proper safety focus and reinforce safety principles (O.1.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action,"
requires, in part, that measures be established to ensure that conditions adverse
to quality are promptly identified and corrected. Contrary to this, between
February 8 and October 2, 2007, engineering personnel failed to ensure that
conditions adverse to quality were promptly identified and corrected. Specifically,
engineering personnel failed to enter applicable OE on the mechanical
overspeed trip mechanism for the AF pump T&TV into the CAP. As a result,
testing to demonstrate the functionality of the overspeed trip mechanism was not
performed and an operability assessment of the degraded and nonconforming
condition was not completed. This was the second example of the NCV involving
failure to implement the CAP requirements. This finding was of very low safety
significance and was entered into the CAP as PVAR 3070597.
b.5 Observations and Minor Violations Involving Design
b.5.1 High Pressure Safety Injection Pump Bearing Modification
Description. The team identified an observation associated with a lack of
technical rigor during the development of a modification associated with
the high pressure safety injection (HPSI) pumps. Work Order (WO)
2972259 consisted of a temporary modification to lower the oiler height
on the Unit 3 HPSI pump bearings. As a result of this modification, the
pump bearing was no longer in a constant oil bath during long periods of
shutdown, when the residual oil in the bearing may drain away. WO
- 60 - Enclosure
2972259, HPSI Bubbler, Attachment 1, stated that for the new oil
configuration, "With the absence of the flooded condition and the
presence of the residual oil within the bearing, Flowserve did not
anticipate any significant bearing degradation resulting from idle periods
of up to and including three months." The team questioned how this
configuration constraint was incorporated into operating procedures. As a
result of the teams questioning, the licensee conducted a review of
procedures and found that they did not incorporate any guidance or
precautions dealing with the pumps being idle for up to three months.
The review for the temporary modification did not specify any concerns in
this area and did not resolve the concern of a pump being idle for more
than three months. The licensee entered this issue into their CAP as
PVAR 3069219.
5.3 Human Performance
The team identified continuing human performance issues at Palo Verde consistent with
previously identified issues discussed in End of Cycle and Mid-cycle letters since 2005.
Specifically, human performance concerns observed during this inspection included
weaknesses in implementing the OD process, failures to follow procedures, failures to
implement human performance tools, and inadequate procedures. In addition, a
significant number of engineering issues reflected a lack of technical rigor in resolving
complex issues. The team noted a lack of adherence to basic radiological work practices
and inconsistent implementation of control room behaviors. The team identified that the
licensees training department had been inconsistent in supporting site improvement.
Although a human performance root cause investigation had been conducted, corrective
actions to date had not been effective in improving human performance. These continuing
human performance deficiencies indicated that corrective actions to resolve the
substantive crosscutting issues had not been successful in sustaining performance
improvement.
a. Inspection Scope
The team evaluated the effectiveness of how Palo Verde personnel identified,
evaluated, and corrected deficiencies involving human performance. The team
evaluated training by reviewing instructional procedures and material, conducting
interviews with training department personnel, observing classes, and job performance
measure (JPM) evaluations, reviewing nuclear assurance department audits, and
reviewing training department self assessments. The team evaluated the work control
process by reviewing procedures, conducting interviews with work control personnel
and work control SROs, and observing outage control center and online work control
center activities. The team conducted a review of substantive human performance
crosscutting aspects and a review of the human performance crosscutting aspects
identified in the findings discussed in this report. Finally, the team conducted
emergency planning performance drills with a sampling of SRO, Technical Support
Center, and Emergency Operations Facility Emergency Directors to assess their ability
to implement the Emergency Plan (EP).
b. Findings and Observations
b.1 Observations and Minor Violations Involving Human Performance
- 61 - Enclosure
b.1.1 Human Performance Root Cause Report
Description: The team reviewed the human performance root cause
report issued in September 2005 and effectiveness reviews completed in
August 2007. The team noted the following weaknesses:
- The September 2005 human performance root cause report identified
that the Palo Verde organization did not demonstrate ownership and
leadership of the human performance culture. The root cause report
stated, Palo Verde Management does not emphasize that excellence
in human performance will result in excellence in plant performance,
and Leaders sometimes model behaviors inconsistent with site
expectations. These statements indicated that the Palo Verde
management team may not have understood what behaviors
contributed to an excellent human performance culture. Also, the
August 2007 effectiveness review of CRAI 2830264 for decision
making stated, The evaluation concluded that there is a lack of an
organizational definition on what constitutes a decision making error
and the behaviors of questioning attitude and technical rigor are not
well defined or understood. The team noted that understanding and
defining the expected behaviors that contribute to an excellent human
performance culture were needed to achieve the desired culture
change.
- In March 2005, the licensee initiated CRDR 2780273 to perform a root
cause investigation of the substantive crosscutting issues in human
performance. Although the August - September 2007 human
performance self-assessment performed in preparation for the
ImPACT review in 2007 determined that the root cause initiated in
CRDR 2780273 was ineffective in identifying the root cause, a
subsequent effectiveness review performed under CRAI 3033705 in
August 2007, determined that the root cause (i.e., the Palo Verde
organization does not demonstrate ownership and leadership of the
human performance culture) was correctly identified. The licensee
supported this conclusion based on subsequent CRDR evaluations
that used streaming analyses, fault tree analyses, common cause
analyses, and human performance models. The effectiveness review
also concluded that a new root cause determination was not
necessary because the root causes had been correctly identified, and
common cause analyses and/or streaming analyses had been
recently performed for industrial safety, clock reset events, and
decision making errors. Additionally, Building Block 6, Human
Performance/Continuous Learning, for the SIBP/SIIP had been
developed. The effectiveness review concluded that the corrective
actions for CRDR 2780273 were not well-defined and there were no
actions for implementation, monitoring, reinforcement, adjustment, or
transfer of human performance ownership change. Furthermore, the
corrective actions were either not fully implemented or not
implemented as intended. During review of the SIBP/SIIP, the team
noted that none of these corrective actions for CRDR 2780273 had
been incorporated into Building Block 6.
- 62 - Enclosure
- The team reviewed apparent cause and root cause evaluations
addressing human performance issues to determine whether the
licensees conclusion that the root cause analysis for CRDR 2780273
was correct. These included CRDR 2994589 (Human Performance
Department Clock Reset Events ACE Report), CRDR 2994593
(Continuous and Reference Procedure Use and Adherence
Department Clock Resets ACE Report), CRDR 2936096 (2006 Site
Clock Reset and Significant Event Stream Analysis), CRDR 3011305
(Industrial Safety Events Common Cause Analysis), CRDR 3008308
(Decision Making Errors from 1/1/06-3/30/07 ACE Report), CRDR
3031159 (2007 Human Performance Site clock Reset Events ACE
Report), and Significant CRDR 3048800 (Industrial Safety
Performance Weakness). The identified causes for CRDRs 2994589,
2994593, and 3031159 were the same; failures in human
performance tool use, leadership oversight, knowledge/skills, and
procedure quality. Of these causes, only leadership oversight and
procedure quality were addressed by CRDR 2780273. Because the
identified contributing causes in CRDR 2780273 included
management not setting/reinforcing clear standards and expectations,
the team concluded that the workforce was unfamiliar with the use of,
and expectation to use, human performance tools such as stopping
when unsure. Discussions with licensee personnel involved in the
apparent cause analyses of department clock resets revealed that it
was common for workers to not be aware of an expectation to stop
before proceeding when procedure quality problems were
encountered. The team verified that corrective actions from these
additionally reviewed CRDRs related to human performance had been
incorporated into the SIBP/SIIP. The team also reviewed CRDR
2928806 which was initiated to track actions in the human
performance crosscutting issue closure plan. CRDR 2928806
contained 75 actions which were included in Building Block 6 of the
SIBP/SIIP. Because the SIBP/SIIP was still in draft form, and many of
the proposed actions had not yet been implemented, the team was
unable to evaluate whether the actions will be effective in correcting
the human performance issues the site was experiencing.
b.1.2 Main Control Room Observations
Description: The team conducted control room observations in all three
units. The team observed Unit 3 for 28 hours3.240741e-4 days <br />0.00778 hours <br />4.62963e-5 weeks <br />1.0654e-5 months <br /> (October 4 and 5), Unit 1
for 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> (October 9), and Unit 2 for 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> (October 11). During
these observations the team observed turnovers between crews, control
room briefs, response to control room alarms, and performance of
control room duties. The team noted the following weaknesses in
control room behaviors:
- On October 4, 2007, the team observed the off going and oncoming
shift managers (SMs) conducting turnover in the Unit 3 Control
Room. The oncoming SM did not use the SM turnover sheet and
there was no discussion between the two SMs of a new night order
- 63 - Enclosure
concerning an Emergency Action Level clarification issued the
previous evening. The team waited until completion of the turnover
to inquire if there were any new night orders. At that point, the off
going SM provided a turnover of the new night order.
- Operations personnel were inconsistent in the use of 3-way
communications. The third part of the 3-way communications was
either not performed or was conducted by body language. Certain
crews demonstrated a higher standard than others. This
demonstrated inconsistency across the operations organization in
the use of 3-way communications.
- Control room personnel did not demonstrate a consistent manner in
declaring expected alarms. Site procedures allowed expected
alarms not to be declared if it is agreed to prior to the test/evolution.
When this methodology was agreed upon, it was not followed
consistently by the control room operators.
- The team noted that there was no methodology in place to identify
who was assigned as the Control Room Supervisor (CRS). On
October 4, 2007, during a turnover brief, the CRS was announced,
but during other briefs this was not done. The team also noted the
lack of a formal announcement by the CRS when leaving the at the
controls (ATC) area and the lack of a formal turnover to another on
shift SRO for control room oversight. On October 9, 2007, the team
observed that while the SM was out of the control room, the CRS
stepped out of the ATC area and the CRS did not inform the control
room of his whereabouts. The team noted this was in compliance
with Procedure 40DP-9OP02, Conduct of Shift Operations,
requirements which defined the control room as the entire 140 level
of the auxiliary building. During this time, there was no command
SRO in the ATC area. This did not provide effective SRO oversight
of control room activities and did not promote a high standard for
control room oversight.
- Crew briefs were not consistently announced by the briefer, nor did
all attendees respond by stating, Ready, as described by site
procedure. Some briefs were interrupted by plant manipulation
requests and in one case a medical emergency. During these
interruptions, the briefs continued while a reactor operator and the
CRS responded to the requests.
- In reviewing control room logs for October 4 and 5, 2007, the team
noted that the Unit 3 shutdown cooling (SDC) Train A inoperability
issue was in two different control room logs used by the SM and the
CRS. One log was used for Limiting Condition for Operations
entries and the other for ODs. The Unit 3 SDC Train A inoperability
times contained in each log were different, which made it difficult to
recover the event timeline.
- 64 - Enclosure
- During the Unit 1 observation on October 11, 2007, the team
determined that the control room was unaware that utility vehicles
were conducting work within the onsite Salt River Project (SRP)
switchyard. The licensee did not track switchyard work in the
respective control room nor did they routinely apply risk
management features to their risk profile.
- Peer checking was inconsistent. On October 4, 2007, the team
noted that a peer checker was not paying attention (eyes diverted in
another direction) as he was providing a peer check to an operator
performing system manipulations. In another example, the peer
checker did not respond verbally about equipment being started.
- The licensee used jumpers to achieve a black board status (a state
in which there are no lighted false or non-impacting alarms on the
control room panels). The licensee had approximately eight
jumpers installed between all 3 units for greater than a year that had
been used to achieve black board status.
5.4 Procedure Quality
Poor procedure quality has been a continuing problem at Palo Verde. The root cause
analysis for the substantive crosscutting issues in human performance documented in
CRDR 2780273 identified that non-conservative decisions were made because of
inadequate procedural guidance and/or poor anticipation of system and human interaction
during procedure and document development. The root cause report also identified that
cognitive decisions were made to not follow procedures because personnel were not able
to follow the procedure as written. During this inspection, the team noted continuing
examples of poor procedure quality indicating that prior corrective actions had not been
completely effective.
a. Inspection Scope
The team reviewed a sampling of procedures to determine whether inadequate
procedures contribute to initiating events, improper mitigating system operation, poor
maintenance or testing, or inadequate emergency and abnormal operations response.
Specifically, the team assessed the effectiveness of corrective actions taken for
procedure quality issues, evaluated the adequacy of the procedure development and
revision processes, and reviewed a sampling of Emergency Planning Implementing
Procedure (EPIP) changes to determine if the EPIP change process was adequate to
correct EPIP related deficiencies and maintain EP commitments.
b. Findings and Observations
b.1 Observations and Minor Violations Involving Procedure Quality
b.1.1 Procedure Issues
Description: The team noted examples of poor procedure quality during
this inspection, including:
- 65 - Enclosure
- Emergency Operating Procedures (EOPs) written for the operation of
AF allowed operation outside of the design basis. For example, the
procedure for using AF for cold shutdown allowed a cooldown rate of
100°F per hour; however, the design basis for AF limits the cooldown
rate to 70°F per hour.
- The team noted that a procedure used to set the limit switch on the
polar crane was based on handwritten engineering notes that did not
have a second verification performed. Furthermore, the notes were
not attached to the procedure. Since the WO was incorrectly
annotated as a non-quality package, it was not maintained and all the
information, including the engineering notes, were discarded after
completion of the work. The licensee subsequently requested copies
of the documents from the team to recreate the record.
- The team noted that the head lift procedure included handwritten
calculations and email communications, but did not include references
to the drawings used to verify proper heights and that no tolerances
were specified for the height measurements. In addition, a sign-off
step involving a cautionary statement was located two steps after the
caution was applicable.
- The team noted numerous weaknesses in EPIPs. For example,
EPIP-03, Technical Support Center Actions, did not provide direction
on appropriate actions to implement when radiation Monitor RU-13A
was out of service. This radiation monitor was used to evaluate the
habitability of the Technical Support Center. Other examples of
emergency preparedness procedure weaknesses are discussed in
Section 5.7 of this report.
5.5 Equipment Performance
Long standing equipment performance issues have challenged the site. Engineering
programs and processes required to reliably track and trend systems important to safety
and reliable operations were often weak. Specifically:
- The team noted that system engineers generally did not understand the implementing
requirements of the maintenance rule (MR) program. Specifically, system trending
was not consistent, establishment and maintenance of performance criteria was not
well understood, and the training of system engineers was not sufficient to ensure that
the program was consistently implemented.
The team noted weaknesses in the evaluation of operating experience relied upon to
maintain adequate plant performance. For example, since 1988, engineering
personnel had not adequately evaluated and inspected pre-1983 Target Rock reed
switches in response to OE. Consequently, the licensee was unaware of a pre-1983
reed switch, that did not conform to requirements, had been installed in Unit 2 safety-
related solenoid operated valve (SOV) 2JRCEHV0403 (Reactor Vessel Seal Drain
Valve to Reactor Drain Tank).
- 66 - Enclosure
- The team identified on September 27, 2007, that the requirements for testing the CS
nozzles in Units 1, 2, and, 3 did not meet TSSR 3.6.6.6. Operations personnel did not
enter TSSR 3.0.3 until prompted by the team on October 30, 2007.
- The team noted that several long standing degraded conditions were not aggressively
pursued by the licensee. Noteworthy examples include cable vault flooding, ESP
material condition, AF system performance, and safety injection system performance.
a. Inspection Scope
The team reviewed various engineering related issues for the selected systems
(containment spray, turbine driven AF pump, ESP pumps, HPSI pumps, and LPSI
pumps) to evaluate the licensees effectiveness in identifying the causes and extent of
equipment problems, as well as developing and implementing corrective actions.
Additionally, a review of the implementation of the EQ program was conducted. The
team reviewed equipment performance related documents, observed inspection
activities, and conducted plant tours to assess the effectiveness of the licensee in
entering equipment performance issues into the CAP. The team also reviewed open
PVARs and corrective maintenance WOs for the selected systems to assess their
potential impact on operability.
The team reviewed surveillance and post-maintenance tests to assess the
effectiveness of the licensee in specifying appropriate acceptance criteria and to
determine whether the licensees controls to restore equipment to operation following
testing and maintenance were effective. For example, the team reviewed the
licensees program and procedures used to test containment sump butterfly valves to
ensure that the ECCS piping was filled with water as required by Technical
Specifications.
The team reviewed selected EQ preventive maintenance activities for the selected
systems to assess program adequacy and to determine whether the design document,
vendor manual, and generic communication information were appropriately
incorporated into the maintenance program.
The team conducted interviews with licensee personnel, including engineering and
procurement personnel, who had an input into maintenance-related activities, to
determine how the system was operated, whether that operation conflicted with the
intended safety function, and whether engineering input was at an appropriate level to
ensure safe and reliable plant operation.
The team evaluated line organization, quality assurance, external audits, and
assessments to determine whether the licensee had demonstrated the capability to
identify performance issues before they resulted in actual events of undesired
consequence. The team reviewed the licensees management support to the audit
and assessment process, as evidenced by staffing of the quality assurance
organization, responsiveness to audit and assessment findings, and contributions of
the quality organization to improvements in licensee activities.
- 67 - Enclosure
b. Observations and Findings
b.1 Failure to Evaluate Performance Monitoring Criteria for Auxiliary Feedwater
System
Introduction. The team identified a Green NCV of 10 CFR 50.65, "Requirements
for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants," for the
failure of MR and engineering personnel to demonstrate that the performance or
condition of SSCs was being effectively controlled through the performance of
appropriate preventive maintenance to ensure the SSCs remain capable of
performing their intended function. Specifically, between April and October 2007,
an inadequate evaluation of MR performance criteria (PC) was performed. As a
result, Unit 2 AF Train A exceeded the 10 CFR 50.65(a)(2) PC, and goal setting,
and monitoring was not performed as required by 10 CFR 50.65(a)(1).
Description. The team reviewed the MR PC for the AF system to verify that the
performance and condition of SSCs was being controlled through the
performance of appropriate preventive maintenance to ensure the AF system
was capable of performing its intended function.
The team questioned MR and engineering personnel on the establishment and
evaluation of MR unavailability and reliability PC for the AF system. Maintenance
Rule and engineering personnel discussed the AF system health report for
January 1, 2007 through June 30, 2007, which provided unavailability and
reliability PC for the AF system. During interviews with MR and system
engineering personnel, the team was unable to identify the roles and
responsibilities, as well as the ownership of establishing and maintaining PC for
the AF, CS, and ESP systems. Further, no documentation existed to validate
that unavailability and reliability were appropriately balanced through the
establishment of accurate PC.
The team reviewed Procedure 70DP-0MR01, "Maintenance Rule," Revision 16.
Step 3.3.2.4 stated that, "Performance criteria will be established such that there
would not be an unacceptable increase in plant risk as measured by Core
Damage Frequency (CDF) when SSC performance is at or near the performance
criteria limit." The team questioned MR personnel to determine what an
acceptable increase in plant risk would be to establish PC. MR personnel
determined an increase in CDF of 1E-6 per year from the baseline CDF, as
described in Study 13-NS-C025R004, "Risk-Informed Performance Criteria,"
Revision 4, would be appropriate for establishing PC. However, Step 3.3.2.4 did
not provide explicit direction to consider this CDF criterion.
The team requested PC data for unavailability and reliability of the AF system
considering the change in CDF criteria from Study 13-NS-C025R004. The
allowed unavailability PC used in the AF system health report for
January 1, 2007, through June 30, 2007, was 1.60 percent while the change in
CDF criteria from Study 13-NS-C025R004 would have only allowed an
unavailability PC of 1.16 percent.
The team questioned MR personnel as to the validity of the PC in the AF system
health report. On October 12, 2007, MR personnel initiated PVAR 3075907 to
- 68 - Enclosure
evaluate the AF system unavailability and reliability PC. PVAR 3075907 created
an action plan to reconstitute the PC for any system where the PC was greater
than the value documented in Study 13-NS-C025R004. Maintenance Rule
personnel reevaluated the PC in a white paper attached to PVAR 3075907 and
determined that 22 systems had non-conservative PC for either unavailability or
reliability or both.
Procedure 70DP-0MR01, Step 3.5.2.3, also stated that if goal setting is
determined to be necessary, then the SSC will be moved from
10 CFR 50.65(a)(2) to 10 CFR 50.65(a)(1), PC will be monitored, goal setting will
be established, and management attention will be focused on the poorly
performing SSC. Maintenance Rule and engineering personnel failed to move
Unit 2 AF Train A from 10 CFR 50.65(a)(2) to 10 CFR 50.65(a)(1) status in
April 2007, to ensure heightened monitoring and goal setting for the system. In
accordance with the new PC, Unit 2 AF Train A should have been moved from
10 CFR 50.65(a)(2) status to 10 CFR 50.65(a)(1) status due to exceeding
unavailability criteria. The MR expert panel met on October 12, 2007, and
determined Unit 2 AF Train A should have been placed in 10 CFR 50.65(a)(1)
status in April 2007 when unavailability exceeded 1.16 percent.
On October 10, 2007, MR personnel initiated PVAR 3074255 to evaluate the
adequacy of Procedure 70DP-0MR01 with regard to determining PC.
Maintenance Rule personnel also initiated PVAR 3076699 on October 15, 2007,
to reiterate an understanding of the ownership and responsibilities of system
engineers with respect to managing the MR PC.
The team reviewed the Palo Verde "Periodic Assessment of Maintenance Rule
Program," July 2005 through December 2006, assessment. Maintenance Rule
personnel reviewed system engineering inputs to the periodic assessments
including a review of 10 CFR 50.65(a)(2) systems performance criteria. This
periodic assessment did not identify any problems with PC exceeding the values
documented in Study 13-NS-C025R004. The team determined that the annual
assessment was a missed opportunity to identify the non-conservative
performance criteria.
Analysis. The performance deficiency associated with this finding was the failure
of MR and engineering personnel to demonstrate that the performance or
condition of SSCs was being effectively controlled through the performance of
appropriate preventive maintenance for Unit 2 AF Train A. This finding is greater
than minor because it is associated with the mitigating systems cornerstone
attribute of equipment performance and affects the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events
to prevent undesirable consequences. Using the Manual Chapter 0609,
"Significance Determination Process," Phase 1 Worksheets, the finding is
determined to have very low safety significance (Green) since it only affected the
mitigating systems cornerstone and did not represent a loss of system safety
function. The cause of this finding had crosscutting aspects associated with self
assessments of the PI&R area in that MR and engineering personnel failed to
perform self assessments that were comprehensive, appropriately objective, and
self-critical (P.3.(a)). The cause of this finding had crosscutting aspects
associated with decision-making of the human performance area in that
- 69 - Enclosure
engineering personnel failed to make safety-significant or risk-significant
decisions using a systematic process (H.1.(a)). The cause of the finding was
also related to the safety culture component of accountability in that management
failed to reinforce safety standards and display behavior that reflected safety as
an overriding priority (O.1.(b)).
Enforcement. 10 CFR 50.65(a)(1) requires, in part, that the licensee monitor the
performance or condition of SSCs against licensee-established goals, in a
manner sufficient to provide reasonable assurance that such SSCs are capable
of fulfilling their intended functions. 10 CFR 50.65(a)(2) requires, that monitoring
as specified in 10 CFR 50.65(a)(1) is not required where it has been
demonstrated that the performance or condition of a SSC is being effectively
controlled through the performance of appropriate preventive maintenance, such
that the SSC remains capable of performing its intended function. Contrary to
the above, from April to October 2007, MR and engineering personnel failed to
demonstrate that performance of Unit 2 AF Train A was being effectively
controlled through appropriate scheduled maintenance. Specifically, an
inadequate evaluation of MR performance criteria was performed and, as a
result, Unit 2 AF Train A exceeded its 10 CFR 50.65(a)(2) PC and goal setting
and monitoring was not performed as required by 10 CFR 50.65(a)(1). Because
the finding was of very low safety significance and was entered into the CAP as
PVAR 3075907, this violation was treated as a NCV, consistent with Section VI.A
of the Enforcement Policy: NCV 05000529/2007012-05, "Failure to Implement
Maintenance Rule Requirements for Auxiliary Feedwater.
b.2 Failure to Control Nonconforming Target Rock Reed Switches
Introduction. The team identified a third example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action," for the failure of
engineering personnel to evaluate and correct the installation of nonconforming
Target Rock reed switches. Between 1988 and October 10, 2007, engineering
personnel had not adequately evaluated and inspected pre-1983 Target Rock
reed switches in response to OE. Consequently, the licensee was unaware that
a pre-1983 reed switch, that did not conform to requirements, had been installed
in Unit 2 safety-related solenoid operated valve (SOV) 2JRCEHV0403 (reactor
vessel seal drain valve to reactor drain tank).
Description. Operating Experience (OE) on Target Rock reed switches,
manufactured before 1983 with Part Number 100967-1, was originally reviewed
at PVNGS in 1988 to determine if any of these reed switches were installed in
the plant. The reed switches had deterioration of the lead wire insulation, that
cracked when the wires were flexed during maintenance or handling. Some of
the cracks occurred at the terminal blocks while tensioning the terminal block
fasteners. This degradation can cause a short to ground of the exposed wires
resulting in dual position indication, blown fuses, or inadvertent opening of the
valves.
The original disposition closed the OE to the PVNGS Generic Letter 91-15,
"Operating Experience Feedback Report, Solenoid-Operated Valve Problems at
U.S. Reactors, SOV Program. During a review by the CDBR, the licensee
determined that a formal SOV program did not exist, and the OE had been
- 70 - Enclosure
closed without a thorough evaluation. The CDBR team wrote PVAR 2959880 on
January 12, 2007, and determined no degraded or non-conforming condition
existed without performing a review to determine if any of these reed switches
were installed in the plant.
CRAI 2960705 was initiated on January 19, 2007, to evaluate the availability and
current use of the reed switches. The CRAI determined no pre-1983 Target
Rock reed switches were available or in use in the plant and no further action on
the OE was required. However, CRAI 2960705 also determined that six reed
switches were installed in the plant that had not been inspected, reworked, or
replaced. Three of the six were located inside containment, with one being
safety related and two being quality augmented. The other three were located in
the auxiliary building.
The team questioned engineering personnel about the conclusion of the CRAI
that no pre-1983 reed switches were installed in the plant and that no further
action was required. The team also questioned the CRAI 2960705 conclusion
that none of these reed switches were installed in the plant since the CDBR
evaluation stated one safety related reed switch had not been inspected,
reworked, or replaced. After further review by the licensee, it was determined
that one Target Rock reed switch, made before 1983, was installed in safety-
related Valve SOV 2JRCEHV0403. Valve SOV 2JRCEHV0403 provides
isolation for the reactor vessel o-ring to maintain a boundary to fission product
release.
On October 10, 2007, PVAR 2959880 was redirected to the control room for an
immediate OD/FA. Operations personnel determined that all other pre-1983
Target Rock reed switches had been inspected or had no design basis safety
function. Engineering personnel determined Valve SOV 2JRCEHV0403
remained functional because the length of time in service with no failures
indicated Valve SOV 2JRCEHV0403 was not susceptible to cracking and that no
cracking had occurred. In addition, Valve SOV 2JRCEHV0403 had no history of
being reworked, replaced, or inspected, so the integrity of the reed switch had
not been challenged. A corrective maintenance WO was generated per
PVAR 2959880 to inspect Valve SOV 2JRCEHV0403.
Analysis. The performance deficiency associated with this finding was the failure
of engineering personnel to evaluate and correct a condition adverse to quality
involving the installation of nonconforming Target Rock reed switches. The
finding is greater than minor because it is associated with the equipment
performance cornerstone attribute of the initiating event cornerstone and affects
the associated cornerstone objective to limit the likelihood of those events that
upset plant stability and challenge critical safety functions during shutdown as
well as power operations. Using the IMC 0609, "Significance Determination
Process," Phase 1 Worksheets, the finding is determined to have very low safety
significance (Green) because assuming the worst case degradation, the finding
would not result in exceeding the Technical Specification limit for reactor coolant
system leakage because a redundant valve existed in series with
SOV 2JRCEHV0403. The cause of this finding had crosscutting aspects
associated with OE of the PI&R area in that operations and engineering
personnel failed to ensure implementation and institutionalization of OE through
- 71 - Enclosure
changes to station processes, procedures, equipment, and training programs
(P.2.(b)). The cause of this finding was also related to the safety culture
component of accountability in that operations and engineering personnel failed
to demonstrate a proper safety focus and reinforce safety principles (O.1.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action,
requires, in part, that measures be established to assure that conditions adverse
to quality are promptly identified and corrected. Contrary to the above, between
1988 and October 10, 2007, engineering personnel failed to ensure that
conditions adverse to quality were promptly identified and corrected. Specifically,
in response to OE issued in 1988, the licensee did not identify and correct the
installation of a pre-1983 Target Rock reed switch in Unit 2 safety-related
SOV 2JRCEHV0403. This was the third example of the NCV involving the failure
to implement CAP requirements. This finding was of very low safety significance
and was entered into the CAP as PVAR 2959880.
b.3 Failure to Meet the Requirements of Technical Specifications Surveillance
Requirement 3.6.6.6
Introduction. The team identified a Green NCV of Technical Specification
Surveillance Requirement (TSSR) 3.6.6.6 for the failure of operations personnel
to verify that each containment spray (CS) nozzle was unobstructed.
Specifically, the last completed surveillance test conducted on each unit
identified that one nozzle in each unit was obstructed and that the nozzles were
not tested in accordance with the approved retest requirement.
Description. The team reviewed Procedure 73ST-9SI02, Containment Spray
Nozzle Air Test, Revision 5, completed on Unit 3 in April 27, 2000, to verify that
the CS nozzles were not obstructed. The surveillance test aligns warmed
compressed air to the spray headers and then verifies that the nozzles are
unobstructed either through use of an infrared camera to observe the nozzles or
by visually observing movement of streamers attached to the nozzle. If a nozzle
is determined to be obstructed, Section 10.1 of 73ST-9SI02, stated that
corrective actions must be taken and the nozzle retested to verify flow prior to
entry into Mode 4. During the test on Unit 3, Nozzle 3PSIAL429 was found to be
obstructed. CRDR 117284 was initiated to evaluate the condition and clear the
blockage. The surveillance test log indicates that the blockage was cleared;
however, there was no evidence to indicate that the nozzle was retested in
accordance with the surveillance test requirement.
As a follow-up to the extent of condition, the team also reviewed the surveillance
test results for Units 1 and 2. Procedure 73ST-9SI02, Revision 5, was partially
completed for Unit 1 on July 12, 2001. During that test, Nozzle 1PSIAL433 was
plugged. Work Order 2380383 was initiated to clear the blockage. Upon review
of the test results the licensee determined that two additional nozzles were not
tested. These two nozzles were later retested on October 21, 2002. However,
there is no evidence to indicate that blocked Nozzle 1PSIAL433 was retested in
accordance with the surveillance test requirement. Procedure 73ST-9SI02,
Revision 6, was completed for Unit 2 on April 12, 2002. The test discovered that
Nozzle 2PSIBL419 was obstructed. Work Order 2797713 was initiated to clean
- 72 - Enclosure
and replace the nozzle. Again, there was no evidence to indicate that the
blocked nozzle was retested in accordance with the surveillance test
requirement.
Following the teams questioning, PVARs 3075026, 3075059 and 3068647 were
initiated to document that during performance of Procedure 73ST-9SI02 in Units
1, 2 and 3 respectively, corrective maintenance was performed to clean a nozzle
that was observed to be obstructed. In each case, a WO was written to inspect
and clean the nozzle. Based on this the licensee concluded that there was no
immediate impact on operability.
Analysis. The performance deficiency associated with this finding was the failure
to meet the requirements of TSSR 3.6.6.6. The finding is determined to be more
than minor because it affected the configuration control attribute of the barrier
integrity cornerstone, and affected the associated cornerstone objective to
provide reasonable assurance that physical design barriers protect the public
from radionuclide releases caused by accidents or events. Using the IMC 0609,
"Significance Determination Process," Phase 1 Worksheets, the finding is
determined to have very low safety significance (Green) because it did not
involve an actual reduction in defense-in-depth for the atmospheric pressure
control function of the reactor containment.
Enforcement. TSSR 3.6.6.6 required that the CS nozzles be verified free of
obstructions. Contrary to the above, as of April 11, 2000, for Unit 3,
March 22, 2002, for Unit 2, and April 13, 2001, for Unit 1, the licensee did not
verify CS nozzles were free of obstructions through the conduct of surveillance
testing. Specifically, Units 1, 2, and 3 each had a blocked CS nozzle during the
performance of Procedure 73ST-9SI02; however, retests were not conducted
following corrective maintenance. Because of the very low safety significance of
the issue and because the issue was entered into the licensees CAP as PVARs
3075026, 3075059, 3068647, and 3048511, the issue was treated as an NCV
consistent with Section VI.A.1 of the NRC Enforcement Policy: NCV 05000528,
05000529,05000530/2007012-06, Failure to Meet the Requirements of
Technical Specification Surveillance Requirement 3.6.6.6.
b.4 Failure to Meet the Requirements of Technical Specifications Surveillance
Requirement 3.0.3
Introduction. The team identified a Green NCV of TSSR 3.0.3 for the failure of
operations personnel to conduct an assessment and manage the risk for a
missed surveillance test. Specifically, on September 27, 2007, the team
identified that the requirements for testing the CS nozzles in Units 1, 2, and, 3 did
not meet TSSR 3.6.6.6. Operations personnel did not enter TSSR 3.0.3 until
prompted by the team on October 30, 2007.
Description. On September 27, 2007, the team identified that the requirements
for testing the CS nozzles (described above) in Units 1, 2, and, 3 did not meet
TSSR 3.6.6.6. The licensee initially entered the condition into their CAP as
PVAR 3068647. On October 18, 2007, the licensee was pursuing approval from
the Plant Review Committee to credit the work orders that removed the blockage
from the nozzles as equivalent to the retest specified Procedure 73ST-9SI02,
- 73 - Enclosure
"Containment Spray Nozzle Air Test," Revision 5, Section 10.1. Although the
Plant Review Committee did not act on this request, they had the opportunity to
recognize that the surveillance requirements had not been met and the
requirement for a missed surveillance test had not been invoked.
Upon further prompting by the team, the licensee entered TSSR 3.0.3 for Units 1
and 2 on October 30, 2007. Since Unit 3 was shutdown, the requirements of
TSSR 3.6.6.6 were not applicable and therefore TSSR 3.0.3 was not required to
be entered. Engineering personnel initiated PVAR 3085708 to address these
issues.
Analysis. The performance deficiency associated with this finding was the failure
of operations personnel to conduct an assessment and manage the risk for a
missed surveillance test in accordance with TSSR 3.0.3. The finding is
determined to be more than minor because it affected the configuration control
attribute of the barrier integrity cornerstone, and affected the associated
cornerstone objective to provide reasonable assurance that physical design
barriers protect the public from radionuclide releases caused by accidents or
events. Using the IMC 0609, "Significance Determination Process," Phase 1
Worksheets, the finding is determined to have very low safety significance
because it did not involve an actual reduction in defense-in-depth for the
atmospheric pressure control function of the reactor containment. The cause of
this finding had crosscutting aspects associated with work practices of the human
performance area in that operations personnel failed to ensure supervisory and
management oversight of work activities that resulted in a missed TSSR
(H.4.(c)). The cause of this finding was also related to the safety culture
component of accountability in that operations personnel failed to demonstrate a
proper safety focus and reinforce safety principles (O.1.(c)).
Enforcement. TSSR 3.0.3, requires that a risk evaluation be performed for any
surveillance delayed greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> and the risk impact be managed.
Contrary to the above, between September 27, 2007, and October 30, 2007,
operations personnel failed to perform a risk evaluation and manage the impact
of risk for a delayed surveillance test. Specifically, the team identified that the
requirements for testing the CS nozzles in Units 1, 2, and, 3 did not meet TSSR
3.6.6.6. Operations personnel did not enter TSSR 3.0.3 for Units 1 and 2 until
prompted by the team on October 30, 2007. Because of the very low safety
significance of the issue and because the issue was entered into the CAP as
PVAR 3085708, the issue was treated as an NCV, consistent with Section VI.A
of the Enforcement Policy: NCV 05000528,05000529/2007012-07, Failure to
Meet the Requirements of Technical Specifications Surveillance Requirement
b.5 Untimely Corrective Actions for Submerged Safety Related Cables
Introduction. The team identified a fourth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, for the failure of
operations and engineering personnel to take timely corrective actions for
conditions adverse to quality involving water intrusion and flooding of
underground manholes and cable vaults. Specifically, since 1996, water
- 74 - Enclosure
intrusion and flooding of underground manholes and cable vaults had been a
recurrent problem affecting electric cables and cable splices for safety-related,
non-safety-related, and security systems.
Description. Since 1996, water intrusion and flooding of underground manholes
and cable vaults had been a recurrent problem affecting electric cables and cable
splices for safety-related, non-safety-related, and security systems. Operations
and engineering personnel initiated CRDR 2407009, CRDR 2784074, and CRAI
2800511 to address these issues.
In October 2007, the team observed the pump-out and inspection of non-safety
related manhole (KMA07) that contained a faulted power cable affecting security
equipment. The cable had been submerged when it failed. Approximately 15
feet of water was pumped from the manhole in order to allow access to the
damaged cable. The team noted that duct banks connecting to adjacent
manholes were approximately 6 feet from the bottom of the manhole vault and
could have served as a potential conduit for the water intrusion. The team
observed water dripping from the ends of a splice on another cable in the
manhole that had been repaired from a previous failure. The team noted that
neither safety related nor non-safety related electric cables and cable splices, in
these underground cable runs, were qualified for continuous submergence.
The team reviewed repeated efforts to address the extent and cause of water
intrusion into underground vaults described in CRAI 2425879, CRAI 2429470,
CRDR 2882166, and CRAI 2919409. The team also reviewed the root cause
investigation, documented in CRDR 2784074, for the Unit 1 spray pond
degraded cable splice failure on March 23, 2005. The team determined that the
root cause analyses failed to address that power cables, not just cable splices,
are susceptible to degradation and failure when submerged for extended periods.
The team also determined that past corrective actions have not been effective in
eliminating underground manhole and cable vault flooding, or cable failures due
to submergence.
The team reviewed a standing order that required the inspection of manholes
that are susceptible to water intrusion following a rainfall of greater than 0.3
inches within a 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> period. The team determined there were no formal
administrative controls in place to initiate this inspection. The inspection was not
incorporated into station procedures to assure that the process was reviewed,
documented, approved and, administratively controlled.
The team also determined that the OD for Unit 1 Spray Pond Pump 1MSPB01,
documented in CRDR 2784074, relied on inspection of manhole
1EZV06BKEM04 after a rainfall of greater than 0.3 inches to ensure that the
power cable splice stayed dry. The 0.3 inch rainfall number was arbitrarily
chosen by examining rainfall history at the site and selecting a value that would
result in about 4 to 5 rainfall-based inspections per year. The team determined
that there was no technical data or root cause analysis that indicated excessive
rainfall was the primary cause of the flooding problem in the electrical manholes
and underground cable systems, and not water from another source.
- 75 - Enclosure
The team noted that, in addition to site specific experiences, a substantial
amount of external OE had been provided to the station. The licensee's
evaluation of Generic Letter (GL) 2007-01, "Inaccessible or Underground Power
Cable Failures that Disable Accident Mitigation Systems or Cause Plant
Transients," was not technically rigorous nor comprehensive since it did not
address failures associated with cable splices. Additionally Information Notice
2002-12, "Submerged Safety-Related Electrical Cables," was closed on March
29, 2002, by reference to CRDR 2407009. CRDR 2407009 evaluated cables in
manholes in response to a 2001 manhole flooding and cable submergence event
and established a long term plan to deal with water intrusion. CRDR 2407009
remained open and had not been effective in addressing the root causes of the
manhole water intrusion problem nor in implementing effective corrective action
as evidenced by the U1 Spray Pond B degraded cable splice failure on
March 23, 2005, and the non-safety manhole flooding and 12.5kV cable failure
observed during this inspection.
Analysis. The performance deficiency associated with this finding was the failure
of operations and engineering personnel to take timely corrective action for
conditions adverse to quality involving water intrusion and flooding of
underground manholes and cable vaults. This finding is greater than minor
because it is associated with the mitigating systems cornerstone attribute of
equipment performance and affects the cornerstone objective of ensuring the
availability and reliability of systems that respond to initiating events to prevent
undesirable consequences. Using the IMC 0609, "Significance Determination
Process," Phase 1 Worksheets, the finding is determined to have very low safety
significance since it only affected the mitigating systems cornerstone and did not
represent a loss of system safety function. The cause of this finding had
crosscutting aspects associated with decision making of the human performance
area in that operations and engineering personnel failed to use conservative
assumptions for operability decision-making when evaluating degraded and
nonconforming conditions (H.1.(b)). The cause of the finding was also related to
the safety culture component of accountability in that management failed to
reinforce safety standards and display behavior that reflected safety as an
overriding priority (O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action,
requires in part, that measures shall be established to ensure that conditions
adverse to quality are promptly identified and corrected. Contrary to the above,
since 1996, operations and engineering personnel failed to take timely corrective
actions for conditions adverse to quality involving water intrusion and flooding of
underground manholes and cable vaults. Specifically, water intrusion and
flooding of underground manholes and cable vaults had been a recurrent
problem affecting electric cables and cable splices for safety-related, non-safety-
related, and security systems. This was the fourth example involving the failure
to implement the CAP. This example was of very low safety significance and
was entered into the CAP as PVAR 3072557.
- 76 - Enclosure
b.6 Failure to Properly Evaluate the Extent of Condition of 4160 V and 480 V Motor
Issues
Introduction. The team identified a seventh example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and
Drawings," for the failure of operations and engineering personnel to adequately
evaluate degraded and unanalyzed conditions to support ODs associated with
CS and LPSI motor lug issues. Specifically, since April 2005, CRDR 2841653
noted that the extent of condition review required by CRDR 2790388, was
complete for the CS and LPSI motor issues, but identified that the condition may
be transportable to other 4160V and 480V motors. However, no evaluation of
additional 4160V and 480V motors was conducted.
Description. Between April and October 2005, there were several CRDRs
documenting loose lugs, improper crimping, and broken motor lead strands on
the CS and LPSI pumps on all three units. The licensee performed technical and
operability evaluations associated with these conditions in CRDR 2968639. On
February 8, 2007, the licensee initiated CRDR 2973072 to address several
process issues associated with the disposition of the CS and LPSI motor lug
issue.
On October 25, 2005, the licensee initiated CRDR 2841653, which identified that
loose lugs, improper crimping, and broken motor lead strands may be
transportable to other 4160V and 480V motors. The evaluation in CRDR
2973072, stated that although the originator of the CRDR believed the issues
were transportable to other 4160V and 480V motors, it was impractical to open
the terminations on each and every 4160V and 480V motors. Engineering and
operations personnel decided to address the rest of the stations motor
terminations as they were removed and re-terminated as part of regularly
scheduled maintenance. No specific corrective action or work-tracking
mechanism was specified to ensure that the inspections were performed.
The team determined operations should have entered Procedure 40DP-9OP26,
"Operability Determinations and Functional Assessment," Revision 18.
Procedure 40DP-9OP26, Step 3.3.5 stated that if other plant conditions or
disassembly is required, then the extent of condition should be addressed by the
CAP, where work mechanisms can be developed and scheduled as appropriate
based on the safety significance. Operations personnel failed to schedule work
mechanisms to ensure the extent of condition on other 4160V and 480V motors
was addressed. On October 24, 2007, engineering personnel initiated
PVAR 3082645 to address this issue.
Analysis. The performance deficiency associated with this finding was the failure
of operations and engineering personnel to adequately evaluate degraded and
unanalyzed conditions to support operability decision making associated with CS
and LPSI motor issues. This finding is greater than minor because it is
associated with the mitigating systems cornerstone attribute of equipment
performance and affects the cornerstone objective of ensuring the availability and
reliability of systems that respond to initiating events to prevent undesirable
consequences. Using the IMC 0609, "Significance Determination Process,"
Phase 1 Worksheets, the finding is determined to have very low safety
- 77 - Enclosure
significance (Green) since it only affected the mitigating systems cornerstone and
did not represent a loss of system safety function. The cause of this finding had
crosscutting aspects associated with corrective actions of the PI&R area in that
operations and engineering personnel failed to take corrective actions to address
safety issues and adverse conditions in a timely manner (P.1.(d)). The cause of
the finding was also related to the safety culture component of accountability in
that management failed to reinforce safety standards and display behavior that
reflected safety as an overriding priority (O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, Instructions,
Procedures and Drawings, requires that activities affecting quality be prescribed
by instructions, procedures, or drawings, and be accomplished in accordance
with those instructions, procedures, and drawings. The assessment of
operability of safety-related equipment needed to mitigate accidents was an
activity affecting quality and was implemented by Procedure 40DP-9OP26,
Operability Determination and Functional Assessment, Revision 18. Step 3.3.5
stated that if other plant conditions or disassembly is required, then the extent of
condition should be addressed by the CAP, where work mechanisms can be
developed and scheduled as appropriate based on the safety significance.
Contrary to the above, since April 2005, engineering personnel failed to ensure
work mechanisms were developed and scheduled to determine the extent of
condition of motor termination degradations. Specifically, operations and
engineering personnel failed to adequately evaluate the extent of condition for
4160 V and 480 V motor lug issues, including loose lugs, improper crimping, and
broken motor lead strands. This is the seventh of 8 examples associated with
the NCV involving failure to implement the OD program. This example was of
very low safety significance and was entered into the CAP as PVAR 3082645.
b.7 Observations and Minor Violations Involving Equipment Performance
b.7.1 Environmental Qualification Program
Description. The existing EQ group responsibility is focused on the EQ
requirements of 10 CFR 50.49 for electrical equipment important to safety
in harsh environments and seismic qualification. Responsibility for EQ
requirements outside of these areas falls upon procurement engineering,
the warehouse and supply chain group, maintenance engineering, and
design engineering. When interviewed, these groups stated several of
their members had previous EQ experience, but that their personnel did
not receive any formal EQ training. Consequently, there was no single
group with overall responsibility for the full range of environmental and
seismic qualification requirements.
The formal mechanical EQ program was deleted from the EQ program
based on a position paper entitled, The Elimination Of The Mechanical
EQ Program, prepared by Tenera in 1994. This study stated that
continued compliance with 10 CFR Part 50, Appendix A, Criterion 4,
Environmental and Dynamic Effects Design Basis, will be maintained by
the procurement program, that had in place detailed and sophisticated
controls of all materials in mechanical equipment to confirm the ability of
equipment and components to perform their required functions in harsh
- 78 - Enclosure
environments; and the maintenance program, that will monitor, trend, and
correct equipment aging for mechanical equipment. However, as
mentioned previously, these groups stated that although several of their
members had previous EQ experience, their personnel do not receive any
formal EQ training.
The fragmented approach to the various aspects of EQ requirements
relied heavily on the EQ awareness and knowledge of the persons in the
groups responsible for implementing the EQ requirements of
10 CFR 50.49 and 10 CFR Part 50, Appendix A, Criterion 4. Examples of
how this EQ program approach and the lack of formal training in groups
required to implement EQ requirements led to problems in the EQ area
included the following:
- During a Unit 3 plant walkdown the team identified minor
discrepancies in the installation configuration of ASCO solenoid
valves on the Unit 3 atmospheric dump valves. The configuration
discrepancy had no impact on the function of the components. In
their investigation of the discrepancies, the licensee identified that
there was no existing design control in place for mechanical
components requiring EQ (PVAR 3079739).
- During a July 1, 2005, review of preventive maintenance for charging
pump motors, the licensee noted that EQ-required lubrication
activities had been stopped in 1998. The condition was documented
in CRDR 2811528 on June 27, 2005, and the activities were re-
verified. Although the condition did not impact the ability of the
equipment to function, this illustrated a lack of communication and
coordination between various site organizations and the EQ program.
- During a Unit 3 containment walkdown, the team observed that the
outer polymer sheath covering for flexible conduit connectors in
numerous equipment locations was cracked, split, and separating
away from the underlying flexible metal conduit. Three different types
of repairs were performed on several degraded flexible conduit
sheaths: wrapping with black electrical tape, application of room
temperature vulcanization sealant at the ends of the sheath that
remained on the flex conduit after other sections had broken away,
and wrapping with a fiberglass tape. As a result of this observation,
the licensee initiated PVAR 3079739 to evaluate this deficiency in the
design control process.
- Water intrusion and flooding of underground manholes and cable
vaults had been a recurrent problem affecting electric cables and
cable splices for safety-related, non-safety-related, and security
systems. Electric cables and cable splices in these underground
cable runs were not qualified for continuous submergence.
- During an October 26, 2006, review of the routine tasks associated
with the EQ requirements for the GL 89-10, "Safety Related Motor
- 79 - Enclosure
Operated Valve Testing and Surveillance," motor operated valves, the
licensee identified that repetitive tasks did not reflect the correct
frequency and initiated CRDR 2936445. Specifically, the work
descriptions for the maintenance activities did not adequately note the
EQ requirements.
The team reviewed the results of Self Assessment No. 2957427,
Equipment Qualification Program, and CRDR 3048870, Engineering
Programs, Appendix B, Equipment EQ Program Review, and found that
the reviews generally identified performance issues at the appropriate
level. However, the team found that lax procedural ties to other plant
organizations were symptomatic of the fragmentation and organizational
weakness in the treatment of the full range of EQ issues.
In summary, EQ program weaknesses were attributed to: insufficient
staffing; a fragmented approach to the EQ program implementation with
no single group with overall responsibility for the full range of
environmental and seismic qualification requirements; and no formal EQ
training for groups responsible for implementing the EQ requirements of
10 CFR 50.49 and NRC general design criteria.
5.6 Configuration Control
5.6.1 Effectiveness of Corrective Actions
The team concluded that corrective actions to address adverse conditions regarding
configuration control were generally effective. The team reviewed a sample of planned and
installed modifications, as well as unapproved and cancelled modifications, to ensure that
changes to equipment were effectively controlled and implemented. The team noted the
licensees program was adequate in implementing corrective actions related to changes in
the plant. However, there were some weaknesses identified with modifications that were
tracked in the licensees database. The potential existed for scheduled modifications to
inadvertently appear on the cancelled or unapproved list. This caused confusion in
determining the status of a specific modification. The team also identified weaknesses in
the thoroughness of performing evaluations regarding changes, or modifications to the
plant that may be outside of the licensing and design bases.
a. Inspection Scope
The team assessed whether corrective actions which affected configuration control
were effective because the loss of configuration control of risk-significant systems or
equipment could lead to the initiation of a reactor transient and/or compromise
mitigation capability. The team reviewed several corrective action documents, WOs,
system health reports, assessments, and audits, as well as conducted interviews of
licensee personnel, in order to adequately assess the effectiveness of corrective
actions for deficiencies involving configuration control. The team reviewed selected
ODs and modifications to verify if a loss of configuration control of risk-significant
systems or equipment which led, or potentially led, to the initiation of a reactor
transient and/or compromised the systems mitigation capability.
- 80 - Enclosure
b. Observations and Findings
b.1 Failure to Implement Corrective Actions for Borg-Warner Check Valves
Introduction. The team identified a fifth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action," for the failure of
maintenance and engineering personnel to promptly correct a
degraded/nonconforming condition associated with a Part 21 notification related
to 3 inch Borg-Warner check valves. Specifically, the licensee did not perform a
disassembly and inspection of Valve 1PSIEV123, HPSI header containment
penetration check valve, during the Unit 1 R13 refueling outage for a 2001 Part
21 corrective action. The failure to perform the maintenance resulted in the
failure of 1PSIEV123 in July 2007, and the continued degradation of additional
safety injection system check valves.
Description. On August 23, 2001, the licensee received Part 21 2001-27-0 on
Borg-Warner Flowserve check valves which expanded the scope of the original
Part 21 notification issued in 1993 to include all 3 and 4 inch Borg-Warner swing
check valves of any pressure class. The condition described in the original Part
21 report was a potential failure of Borg-Warner valves to go fully closed due to
the valve disk becoming lodged under the lip of the valves seat. The licensee
assumed that CRDR 2332280 initiated on October 23, 2000, already performed
the required evaluations for this issue and thus no action was taken.
On January 26, 2007, mechanical engineering determined that not all Borg-
Warner check valves had been evaluated by CRDR 2332280 and generated
PVAR 2963565, coded as degraded/nonconforming, to address the 2001 Borg-
Warner Part 21 notification. This PVAR identified valves that were more critical
due to the potential for having a nonconformance issue, and the last reassembly
being implemented before Procedure 31MT-9ZZ17, "Borg-Warner Check Valve
Disassembly and Assembly, was developed. This list included Valve
1PSIEV123. On February 2, 2007, the ARRC initiated CRDR 2965988 to
complete the necessary action for the 2001 Part 21 notification. CRDR 2965988
was closed after addressing the 2001 Part 21 evaluation without any action taken
to address the degraded/nonconforming conditions of the check valves.
On May 2, 2007, Significant CRDR 2930774, "Failure of LPSI Injection Check
Valve 1PSIEV134," was issued following the failure of another Borg-Warner
check valve. The valve failed because of excessive friction in the disc to seat
landing zone, spherical bearing and swing arm bore, and the spherical bearing
and disc/stud raised weld. This corrective action document was issued to
change the extent of cause to apply to the weld size, gap measurements and
stiffness issues to all Borg-Warner valves, including the 3 and 4 inch valves, and
revise Procedure 31MT-9ZZ17, "Borg-Warner Check Valve Disassembly and
Assembly," to incorporate new Borg-Warner assembly information and
clearances.
On May 19, 2007, the licensee did not perform Procedure 73ST-9SI05, "Leak
Test of HPSI/LPSI Containment Isolation Check Valves," Section 8.2, Revision
21, on 1PSIEV123. Procedure 73ST-9SI05, Section 7.6, stated, in part, that a
typical refueling outage involves performance of Sections 8.1 through 8.4 during
- 81 - Enclosure
plant shutdown, and then retest of individual valves during the startup if work was
performed on any valve during the outage. However, during the Unit 1 R13
refueling outage the licensee did not perform the leak tests during the plant
shutdown. This was further affected by the maintenance on Valve 1PSIEV123
being removed from the outage schedule on June 24, 2007, because of a
perceived parts issue by supply chain services. The parts required for the
maintenance were actually staged on May 24, 2007. The licensee failed to
properly code the WO as degraded/nonconforming which allowed for the
maintenance to be cancelled without an OD or FA. Completion of the scheduled
maintenance would have provided another chance to identify the
degraded/nonconforming condition.
On July 5, 2007, Valve 1PSIEV123 failed during performance of Procedure
73ST-9SI05, "Leak Test of HPSI/LPSI Containment Isolation Check Valves,"
Revision 21, and was declared inoperable. The valve failure was because of
binding in the spherical bearing due to excessive wear between the hinge arm
and spherical bearing. The valve also exhibited excessive washer to hinge arm
gap and indications of disc to stud weld interference.
Analysis. The performance deficiency associated with this finding was the failure
of maintenance and engineering personnel to promptly correct a
degraded/nonconforming condition associated with a Part 21 notification related
to 3 inch Borg-Warner check valves. The finding is more than minor because it is
associated with the equipment performance attribute of the mitigating systems
cornerstone and affected the associated cornerstone objective to ensure the
availability, reliability, and capability of systems that respond to initiating events
to prevent undesirable consequences. Using the Manual Chapter 0609,
"Significance Determination Process," Phase 1 Worksheets, the finding is
determined to have very low safety significance (Green) because the condition
only affected the mitigating systems cornerstone and did not result in the actual
loss of safety function to any component, train, or system. The cause of this
finding had crosscutting aspects associated with OE of the PI&R area in that
maintenance and engineering personnel failed to ensure implementation and
institutionalization of OE through changes to station processes, procedures,
equipment, and training programs (P.2.(b)). The cause of the finding was also
related to the safety culture component of accountability in that management
failed to reinforce safety standards and display behavior that reflected safety as
an overriding priority (O.1.(b)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action,"
states, in part, that measure shall be established to assure that conditions
adverse to quality are promptly identified and corrected. Contrary to the above,
the licensee failed to promptly correct a degraded/nonconforming condition
associated with a Part 21 notification related to 3 inch Borg-Warner check valves
and site specific OE, resulting in the failure of Valve 1PSIEV123 while in Mode 3
on July 5, 2007. This was the fifth example of the NCV involving the failure to
implement the CAP. This example is of very low safety significance and was
entered into the CAP as CRDR 3038601.
- 82 - Enclosure
5.6.2 Selected System Walkdown
The team determined that the LPSI and CS systems were in good material condition, and
system components were found in the expected positions. Equipment labels, hangers and
supports, and environmental conditions were adequately maintained. There were no
observed system leakage points that would degrade the system function. General
housekeeping practices were found to be adequate; however, the team did identify several
issues regarding a lack of control of transient combustibles. No significant deficiencies
with regards to configuration control for the selected systems were identified. The team
did identify several examples that demonstrated a weakness with the licensee maintaining
an adequate condition of less risk significant systems, incorrectly installed scaffolding,
equipment tagging, and fire protection features.
a. Inspection Scope
The team performed a walkdown of general plant areas, and accessible portions of the
LPSI and the CS systems for Units 1 and 2, in order to verify the licensee maintained
adequate configuration control of risk significant systems. The team reviewed design
documents, plant drawings, and system procedures to verify actual plant conditions
were consistent with as-built requirements. In addition, the team reviewed applicable
temporary modifications to ensure proper installation in accordance with the design
information. The team also performed observations of components and surrounding
plant areas for the selected systems to identify additional equipment conditions and
items that might degrade system performance.
b. Observations and Findings
b.1 Failure to Maintain Control of Transient Combustibles
Introduction. The team identified two examples of a Green NCV of Technical
Specification 5.4.1.d for the failure of Fire Protection (FP) personnel to follow
procedures for the control of transient combustibles. Specifically, the team
identified that on the 70 foot elevation of the auxiliary building (radiation
protection (RP) remote monitoring station) and in the Unit 3 containment building,
there were transient combustibles being stored without a proper evaluation or the
required permits.
Description. During a walkdown of auxiliary building 70 foot elevation (RP
remote monitoring station) on October 1, 2007, the team noted a large amount of
transient combustibles (rolls of large plastic bags, large rolls of paper, etc)
being stored in the area. The team requested the transient combustible control
permit (TCCP) for the stored materials. Upon further inspection, the team
determined that the licensee did not evaluate the mass quantities of material that
were being stored in the area per Procedure 14DP-0FP33, "Control of Transient
Combustibles," Revision 15, and that the licensee did not have a TCCP for the
additional combustibles. The team noted that the excess combustible material
should have been identified during fire watch walkdowns when verifying the
requirements for the RP remote monitoring station TCCP were being met.
During a walkdown of the Unit 3 containment building on October 2, 2007, the
team noted a large amount of transient combustibles being stored in the area.
- 83 - Enclosure
The team requested the TCCP for the stored materials. During interviews with
the program owners, the team was informed that containment was exempt from
the TCCP program. The team was provided a licensee evaluation that stated
issuing permits during the refueling outage for the containment would, Create a
bottleneck and impact work scheduling. Upon further review of the TCCP
program, the team identified that licensee procedures did not exempt
containment from the TCCP program. Specifically, Procedure 14DP-0FP33,
"Control of Transient Combustibles," Revision 15, stated that all levels and all
areas of the containment building required permits for transient materials,
including treated wood scaffolding.
Analysis. The failure to control transient combustibles in accordance with the FP
program requirements was a performance deficiency. The finding is more than
minor because storing unanalyzed combustibles results in the potential to exceed
combustible limits and may increase in the likelihood of an initiating event.
Additionally, this finding represented degradation in the FP defense-in-depth
strategy in that the licensee did not recognize that bulk materials were being
stored in the area in support of the outage. Without proper evaluation, this
storage increased the likelihood of a transient fire. Using the Manual Chapter
0609, "Significance Determination Process," Appendix F, "Fire Protection
Significance Determination Process," this issue affected the Fire Prevention and
Administrative Controls Category. The stored materials required a permit per the
licensees procedure; however, the area was attended, fire detection and
suppression was available, and the amounts did not exceed the loading
calculation to the point of changing loading classification. Therefore, this finding
is considered of low degradation and is determined to have very low safety
significance (Green). The cause of this finding had crosscutting aspects
associated with work practices of the human performance area in that the
licensee failed to communicate human error prevention techniques such that
work activities were performed safely (H.4.(a)). The cause of this finding had
crosscutting aspects associated with work practices of the human performance
area in that the licensee did not effectively communicate expectations regarding
procedural compliance (H.4(b)). The cause of this finding was also related to the
safety culture component of accountability in that FP personnel failed to
demonstrate a proper safety focus and reinforce safety principles among their
peers (O.1.(c)).
Enforcement. Technical Specification 5.4.1.d, states, in part, that written
procedures shall be established, implemented, and maintained for FP program
implementation. Procedure 14DP-0FP33, "Control of Transient Combustibles,"
Revision 15, stated in part that transient combustibles being stored in the
Auxiliary Building and Containment Building in support of maintenance (outage)
activities are required to have a permit. Contrary to the above, between
August 23, 2007, and October 5, 2007, the licensee failed to have a proper
permit for all of the stored materials in the RP remote monitoring station.
Specifically, Fire Area 37A had transient combustibles stored with no associated
permit. Additionally, between September 29, 2007, and October 10, 2007, the
licensee failed to have a proper permit for all of the stored transient materials in
the containment building. Specifically, Fire Areas 63, 66, and 67 had transient
combustibles stored with no associated permit. Because this finding was of very
low safety significance and was entered into the CAP as PVARs 3071785,
- 84 - Enclosure
3072224, and 3072260, this violation was treated as a NCV, consistent with
section VI.A of the NRC Enforcement Policy: NCV 05000530/2007012-08, "Two
Examples of a Failure to Maintain Control of Transient Combustibles."
b.2 Failure to Install Emergency Lighting in Containment
Introduction. The team identified a Green finding for the failure of maintenance
personnel to install emergency lighting in containment in support of the Unit 3
refueling outage per repetitive maintenance WO 2935399 and work instruction
WSL 24436. As a result, work began in the Unit 3 containment with no
emergency lighting installed and no egress contingency plan for a loss of
containment lighting.
Description. During a walkdown of the Unit 3 containment on October 2, 2007,
the team identified that emergency lighting units did not have the batteries
installed. Upon further research, the team found the licensee removed
emergency lighting batteries in containment while at power to preserve the
availability and reliability of the batteries. The batteries were to be reinstalled for
outage support; however, the licensees work instructions did not prescribe when
the batteries needed to be re-installed (prior to commencing work). As a result of
the inadequate procedural guidance, work commenced in the Unit 3 containment
building on September 29, 2007, without having completed the emergency
lighting battery installation. Additionally, the licensee did not have a contingency
plan for personnel in the event normal power to containment lighting was lost.
Analysis. The performance deficiency associated with this finding was the failure
of maintenance personnel to have an adequate procedure for installing
emergency lighting in containment and not including appropriate acceptance
criteria for determining that the activity had been satisfactorily accomplished.
This finding is considered more than minor because it is associated with the
Mitigating Systems Cornerstone attribute of procedural quality and if left
uncorrected, a failure to install emergency lighting could hamper emergency
response activities in the containment or complicate emergency egress from the
containment. Using the IMC 0609, "Significance Determination Process,"
Appendix M, Significance Determination Process Using Qualitative Criteria, the
finding is determined to have very low safety significance because emergency
lighting was necessary for personnel safety and personnel were expected to
carry flashlights when responding to events. The cause of the finding has
crosscutting aspects associated with work control of the human performance
area in that maintenance personnel failed to properly plan the emergency lighting
installation work by incorporating contingencies in case the work was not
completed in the appropriate timeframe (H.3.(a)). The cause of this finding was
also related to the safety culture component of accountability in that management
personnel failed to reinforce safety standards and display behavior that reflected
safety as an overriding priority (O.1.(b)).
Enforcement. No violation of regulatory requirements occurred. The team
determined that the finding did not represent a noncompliance, because the
failure to install the emergency lighting or adequately evaluate the condition
occurred on a non-safety-related system. The finding was of very low safety
significance and the issue was entered into the CAP under PVAR 3070783.
- 85 - Enclosure
FIN: 05000530/2007012-09, Failure to Install Emergency Lighting in
Containment Prior to Work Commencement.
b.3 Incorrect Installation of Temporary Shielding
Introduction. The team identified a Green NCV of TS 5.4.1a for the failure of RP
personnel to follow procedures for installing temporary shielding in the 87 foot
auxiliary building west penetration room.
Description. During a walkdown of the auxiliary building 87 foot elevation on
October 3, 2007, the team observed temporary shielding Package A-87-10
installed near Train A LPSI piping. Upon further inspection, it was noted that the
shielding was in direct contact and installed across the Train A LPSI instrument
sensing line. The shielding had been erected per WO 2955341on
September 5, 2007, to reduce dose during the Unit 3 refueling outage.
Procedure 75RP-9RP25, Temporary Shielding, Revision 9, stated, in part, that
if shielding is to be installed on piping systems which are declared operable, a
piping stress analysis must be performed and cited in Specification 13-CN-0211,
Installation Specification for Temporary Shielding for the Palo Verde Nuclear
Generation Station Units 1, 2, & 3, Revision 9. Temporary shielding Evaluation
07-017 and installation Specification 13-CN-0211 had evaluated the shielding
installation near large bore LPSI piping with no evaluations or operability
concerns noted. WO 2955341 stated that the shielding was installed per
specification requirements. However, neither the temporary shielding evaluation,
the temporary shielding package, nor the installation specification addressed or
evaluated the shielding installed in contact with and over the LPSI instrument
sensing line.
After reviewing the procedures for temporary shielding installation, the team
contacted RP personnel and questioned the seismic qualification of the LPSI
pressure instrument sensing line. The licensee immediately rearranged the
shielding blankets to eliminate the contact with the instrument line. Engineering
concluded that the condition could have caused the line to fail during a
postulated design basis seismic event. No loss of safety function occurred since
the other LPSI train was not affected.
Analysis. The team determined that the licensees failure to correctly install
temporary shielding was a performance deficiency. This finding is greater than
minor because it is associated with the mitigating systems cornerstone attribute
of configuration control and affected the cornerstone objective to ensure
availability and capability of systems to respond to initiating events. Using the
IMC 0609, "Significance Determination Process," Phase 1 Worksheets, this
finding is determined to have very low safety significance (Green) because the
condition did not result in an actual loss of safety function and did not screen as
risk significant or contribute to external event initiated core damage sequences
since it did not involve a loss or degradation of equipment designed to mitigate a
seismic event. The cause of this finding had a crosscutting aspect associated
with work practices of the human performance area in that the licensee did not
effectively use human error prevention techniques such as self checking and
proper documentation of activities for the shielding installation (H.4.(a)).
- 86 - Enclosure
Enforcement. Technical Specification 5.4.1.a requires that written procedures be
established, implemented, and maintained covering the activities specified in
Appendix A, "Typical Procedures for Pressurized Water Reactors and Boiling
Water Reactors," of Regulatory Guide 1.33, "Quality Assurance Program
Requirements (Operations)," dated February 1978. Regulatory Guide 1.33,
Appendix A, Section 9a, requires maintenance that can affect safety-related
equipment be properly preplanned and performed in accordance with written
instructions appropriate to the circumstances. Procedure 75RP-9RP25,
Temporary Shielding, Revision 9, stated in part, that if shielding is to be
installed on piping systems which are declared operable, a piping stress analysis
must be performed and cited in Specification 13-CN-0211, Installation
Specification for Temporary Shielding for the Palo Verde Nuclear Generation
Station Units 1, 2, & 3. Contrary to this, between September 5, 2007, and
October 3, 2007, the licensee installed temporary shielding in contact with the
Train A LPSI instrument sensing line, and a piping stress analysis was not
performed. Because the finding was of very low safety significance and was
entered into the CAP as PVARs 3071468 and 3072224, this violation was treated
as an NCV, consistent with Section VI.A of the Enforcement Policy: NCV
05000530/2007012-10, "Failure to Follow Procedures for Temporary Shielding
Installation."
b.4 Observations and Minor Violations Involving Selected System Walkdown
b.4.1 Inadequate Seismic Scaffolding Procedures
Technical Specification 5.4.1.a requires that written procedures be
established, implemented, and maintained covering the activities
specified in Appendix A, "Typical Procedures for Pressurized Water
Reactors and Boiling Water Reactors" of Regulatory Guide 1.33, "Quality
Assurance Program Requirements (Operations)," dated February 1978.
Regulatory Guide 1.33, Appendix A, Section 9a, requires maintenance
that can affect safety-related equipment be properly preplanned and
performed in accordance with written instructions appropriate to the
circumstances. Contrary to this, as of October 8, 2007, the licensee did
not have adequate procedures or written instructions for maintenance that
affects safety related equipment. Specifically, Procedure 30DP-9WP11,
"Scaffolding Instructions," Revision 18, did not specify clearance
requirements for scaffolding installed near risk important non-safety
components that have a potential to impact safety related equipment.
Scaffolding was erected with an approximately one half inch clearance
between the CS pump room Train A flooding level switch. A failure of the
level switch could impact the operability of the CS system during a room
flooding event. The finding is determined to be minor because the
inadequate procedure did not have any actual safety significance. The
finding was of very low safety significance and was entered into the CAP
as PVARs 3073777 and 3071468. This performance deficiency is being
documented because of the insights associated with inadequate
procedures and recurring scaffolding concerns.
- 87 - Enclosure
5.6.3 Work Control Process
The team identified several weaknesses involving the licensees work control process,
including the following areas: adequate risk management of maintenance activities,
effective control of main control room deficiencies, prioritization of work with consideration
to environmental qualification, adherence to and effectiveness of controls for transient
combustibles and hot work, and thoroughness of pre-job briefings. Specifically:
- The team observed several control room and work control activities to verify the
licensees controls for independent verification were adequate, including the EDG
standby readiness testing and an EW system tagout. No significant discrepancies
were observed during these activities. The team did note an event on
October 26, 2007, when an incorrect breaker was manipulated because the workers
were at the wrong unit. The individuals recognized the mistake and returned the
breaker to the as-found position; however, did not immediately notify the control room.
Once the control room became aware of the event, all site wide maintenance work was
stopped to reinforce independent verification practices and expectations.
- The team identified several examples of inadequate risk management regarding
shutdown activities and switchyard activities. The team identified a lack of effective
communication between the switchyard owners, Salt River Project, and the licensee.
Maintenance activities in the switchyard, which could increase the risk of an initiating
event, were not thoroughly scheduled and integrated with on-site work activities. In
response to the team's findings, the licensee implemented immediate and long term
corrective actions to address risk management of switchyard maintenance activities.
The team also observed two minor examples of inadequate shutdown risk
assessments performed by the licensee which further demonstrated a weakness with
the licensees understanding of risk management.
- The team noted there were several means of tracking control room deficiencies
including: control room deficiency log, jumpered alarm log, lit annunciator log, and
multiple operator workaround logs. The team identified that the pens were removed
on some strip charts required for post accident monitoring instrumentation. The charts
were tagged as being degraded and requiring maintenance; however, it was not
recognized by the control room operators that this rendered the instrumentation
- The team inspected the prioritization of maintenance activities as it relates to EQ to
verify if equipment was being effectively maintained and not subject to environmental
degradation. The team identified an inability of the licensee to maintain the cable
vaults void of water and the use of unqualified tape in containment.
- There were several incidents during the Unit 3 refueling outage involving hot work.
The licensee conducted two stand-downs in response to multiple small fires caused by
hot work activities. None of the fires were significant enough to warrant an emergency
declaration; however, the incidents supported the team's assessment that there
appeared to be lack of effective control and communication of expectations regarding
administrative controls for hot work and the control of transient combustibles. The
licensee did not consistently adhere to the procedures in place for controlling and
evaluating temporary and long term storage of transient combustibles throughout the
- 88 - Enclosure
plant. Ownership and accountability responsibilities for the control of transient
materials was fragmented between FP engineers, operations, and the site fire
department.
a. Inspection Scope
The team conducted a review of the backlog of corrective and preventive maintenance
activities to determine if the work control process used risk-insights during planning
and scheduling of maintenance and surveillance testing activities and the control of
emergent work. The team conducted interviews of licensee personnel, reviewed work
packages, and work control and maintenance procedures in order to assess the
adequacy of the licensee's efforts to integrate maintenance to minimize equipment
unavailability, establish effective communication and coordination, and address plant
performance deficiencies. The team reviewed the licensees policies to assess if the
licensee adequately considered the need for planned contingencies, compensatory
actions, and abort criteria when scheduling and executing work. The team reviewed
the performance history for selected systems and components and compared it to the
design basis to verify the licensee made conservative assumptions when scheduling
and performing work. The team also reviewed the following: long-term (typically
greater than six months) tagouts, caution and danger tags, disabled control room
annunciators and instruments, control room deficiencies, operator work-arounds and
other equipment deficiency tracking systems, to assess the significance of these
conditions.
b. Observations and Findings
b.1 Failure to Adequately Manage Risk for Switchyard Activities
Introduction. The team identified a Green NCV of 10 CFR 50.65(a)(4) for the
failure to adequately assess the increase in risk and effectively implement risk
mitigation actions for maintenance activities in the switchyard (SWYD).
Description. On October 11, 2007, the team observed several personnel and
pieces of equipment moving about the switchyard and noticed postings that
stated, in part, to contact the Unit 1 shift manager (SM) for entry into the SWYD.
While the activities appeared to be positioning of materials and equipment, the
team was unable to determine if any work was being conducted. The team
contacted the Unit 1 SM who stated that he was not aware of work in the SWYD
and that no one had contacted him for entry into the SWYD. The team then
contacted the SWYD coordinator and was informed that work on PL-942 and
PL-928 525kV breakers was being performed but he had failed to inform the Unit
1 SM. The team reviewed the risk assessment for the SWYD work and noted it
was revised to include the breaker work being performed. During discussions
with the licensees risk analyst and SWYD coordinator about the control and
modeling of work in the SWYD, it was noted that the risk model only accounts for
certain breakers and relays, and does not independently model equipment or
personnel traffic in the SWYD since that was considered in the modeling of the
work. It was also noted that routine relay planned maintenance (PM) and
equipment movement is not included on the schedule provided to the coordinator
and may not be included in the risk assessment. The SWYD coordinator stated
that equipment traffic was communicated to him and that the risk was managed
- 89 - Enclosure
by scheduling the work, controlling access to the SWYD via the Unit 1 SM, and
restricting equipment to designated lanes and areas in the SWYD.
On October 24, 2007, the team, accompanied by the SWYD coordinator and a
Transmission/Generation Operations (TGO) SWYD foreman, performed a
walkdown of the SWYD to observe breaker work. The team noticed multiple
trucks, pieces of equipment, and personnel moving around the SWYD that were
not involved with the breaker work. The team asked the TGO SWYD foreman
about the additional traffic, he stated that this was considered normal and that his
crew of 3-10 personnel works almost every day in the SWYD performing
maintenance. Procedure 40DP-9OP34, Switchyard Administrative Control,
Revision 16, Step 2.7 stated, in part, that all personnel entering the switchyard
shall notify the Unit 1 Shift Manager. When asked about contacting the Unit 1
SM prior to entering the SWYD, he stated that his supervisor coordinated any
work with the SWYD coordinator but was not aware of the need to contact the
Unit 1 SM for access to the SWYD. During the walkdown, the team also
observed a truck outside the designated traffic lanes and noted multiple tire
tracks and a man lift inside a restricted access area were no work was being
performed. The SWYD coordinator stated he was unaware of all of the
equipment traffic occurring in the SWYD.
The team noted that the SWYD was not being protected by controlling access
and movement as required and that the risk modeling did not include all work
being performed. The Unit 1 SM and SWYD coordinator were unaware of the
movement of multiple vehicles and pieces of equipment in or near restricted
areas nor is this included in the risk model. Additionally, routine relay PMs and
maintenance was not included on the schedule provided to the SWYD
coordinator for risk review.
The team noted that OE existed related to switchyard work, including vehicles in
the switchyard, potential impact of switchyard work on offsite power, and taking
into consideration all switchyard work when calculating risk in accordance with
10 CFR 50.65. Based on the amount of OE and the importance of offsite power
in relation to risk, the licensee should have incorporated more controls to
manage work in the switchyard and factored that work into the risk assessment
process. In particular:
- Information Notice 90-25, Loss of Vital AC Power with Subsequent Reactor
Coolant System Heat-up, described an event that occurred when a truck
backed into a support column for a feeder line in the switchyard resulting in a
loss of power to the vital buses.
- Regulatory Issue Summary 2004-005, Grid Reliability and the Impact on
Plant Risk and the Operability of Offsite Power, describes calculating risk
associated with 10 CFR 50.65(a)(4), including the impact of switchyard
maintenance on the operability of offsite power sources.
- Temporary Instructions 2515/156, Offsite Power System Operational
Readiness, and 2515/163, Operational Readiness of Offsite Power, both
described the potential impact of switchyard maintenance on offsite power
sources.
- 90 - Enclosure
- Generic Letter 2006-02, Grid Reliability and the Impact on Plant Risk and the
Operability of Offsite Power, describes the need for effective coordination of
switchyard maintenance and the need to assess risk for switchyard activities.
Analysis. The failure to integrate all SWYD work into the risk assessment and
implement effective risk management actions to assess and manage the risk was
a performance deficiency. This finding is greater than minor because the
licensees risk assessment failed to consider maintenance activities that could
increase the likelihood of initiating events such as work in the SWYD and failed
to effectively manage compensatory measures. Inspection Manual Chapter
0609, Appendix K, Maintenance Risk Assessment and Risk Management
Significance Determination Process, was used to assess the significance. The
senior risk analyst made the following assumptions:
1. In accordance with IMC 0609, Appendix K, the significance of this finding was
numerically equal to the incremental core damage probability deficit (ICDPD),
or the difference between the ICCDP calculated by the licensee and the
ICCDP that would have been calculated had the SWYD work been properly
incorporated within the on-line risk monitor.
2. The exposure period for the finding was one year. The finding included both
at-power and shutdown conditions.
3. Three initiating events were postulated to be caused by human error
associated with general work in the SWYD: loss of offsite power (LOOP),
partial loss of offsite power, and turbine trip/reactor trip.
4. There was insufficient data at Palo Verde to estimate the frequency of
switchyard-centered LOOPs (none have occurred in the 20 years of
operation). Therefore, industry data were used to estimate this value.
5. The frequency of LOOP events caused by SWYD human error events was
derived from NUREG/CR6890, Reevaluation of Station Blackout Risk at
Nuclear Power Stations, Analysis of Loss of Offsite Power Events:
1986-2004.
A bounding assumption was made that the baseline LOOP and transient initiating
event frequencies in the licensees risk monitor do not include consideration of
data related to human error in the SWYD. Although this was not the actual
situation, it simplified the analysis and produces a result that can be used to
define an upper bound to the significance (which could be refined later if
necessary). Therefore, based on this assumption, the baseline was zero and the
risk deficit was equal to the expected rate of events caused by SWYD work
multiplied by the conditional core damage probability (CCDP) of the event as
quantified in the Palo Verde SPAR model, Revision 3.31. The CCDP of a LOOP
event was determined to be 4.332E-5. Using industry data, LOOP event
frequencies caused by SWYD work were determined to be 0.0016/year for at-
power and 0.0042/year for shutdown conditions during a typical calendar year.
The at-power frequency was doubled to account for an increased presence of
workers in the Palo Verde SWYD. The average CCDP for a shutdown LOOP
was determined by doubling the at-power CCDP. The resulting delta-CDF was
- 91 - Enclosure
5.0E-7/year. The risk effect of partial LOOPs and transients caused by SWYD
work was determined to be insignificant for this analysis. Neither external events
nor large early release contributed to the risk of the finding. Based on the
magnitude of the calculated risk being less than 1E-6/year, this finding is
determined to have very low safety significance (Green). The cause of this
finding had crosscutting aspects associated with work control of the human
performance area in that the licensee failed to plan work activities incorporating
risk insights (H.3.(a)). The cause of this finding had crosscutting aspects
associated with work control of the human performance area in that the licensee
failed to appropriately communicate work activities (H.3.(b)).
Enforcement. 10 CFR Part 50.65(a)(4), states in part, that before performing
maintenance activities (including but not limited to surveillance, post-
maintenance testing, and corrective and preventive maintenance), the licensee
shall assess and manage the increase in risk that may result from the proposed
maintenance activities. Contrary to this, between October 11 and 24, 2007, the
licensee failed to adequately assess and manage the increase in risk.
Specifically, the licensee failed to include all work being performed in the risk
assessment and fully implement risk management actions to protect the SWYD.
Because the finding was of very low safety significance and was entered into the
CAP as PVAR 3078392, this violation was treated as an NCV, consistent with
Section VI.A of the Enforcement Policy: NCV 05000528, 05000529,05000530/2007012-11, Inadequate Implementation of Risk Management
Actions and Risk Assessment for the Switchyard.
b.2 Observations and Minor Violations Involving Work Control Processes
b.2.1 Failure to Properly Document Temporary Modifications
The team identified a minor violation of Technical Specification 5.4.1.a for
the failure of operations and maintenance personnel to follow Procedure
81DP-0DC17, "Temporary Modification Control," Revision 20. Procedure
81DP-0DC17 required, in part, that: 1) upon completion of the
installation, a copy of the temporary modification procedure/work order
pages shall be given to the control room, and 2) upon receiving a copy of
the procedure/work order, the SM, control room supervisor, or authorized
designee shall log the temporary modification into a temporary
modification book or computer spread sheet. Contrary to this, on
October 15, 2007, the team identified that temporary modifications
installed to support the Class 1E Bus E-PBA-S03 and Non Class 1E Bus
NAN-S02 outages on Unit 3, were not accounted for in the temporary
modification book and the procedures/work orders were not being given
to the control room in accordance with procedural guidance. This finding
was entered into the licensee's CAP as PVAR 3076979. Using IMC
0612, Appendix E, "Examples of Minor Issues," this finding was
determined to be minor because this was an insignificant procedural error
and there were no safety consequences. This performance deficiency is
being documented because of insights associated with procedure
compliance and conduct of operations.
- 92 - Enclosure
b.2.2 Inadequate Shutdown Risk Assessments
The team identified two minor examples of improperly performed
shutdown risk assessments for Units 2 and 3 performed by the shift
technical advisors (STAs).
- At 6:58 p.m. on October 4, 2007, the site entered a severe thunder
storm warning. The STA was called back to the Unit 3 control room to
re-evaluate the risk assessment due to this emergent condition. The
STA used the control room posted risk assessment as a tool to
determine if the risk to the current plant conditions had changed due
to the severe weather. The STA incorrectly determined that part two
of the shutdown risk assessment identified severe weather as a high
risk to electrical resources and inventory control. The STA then
marked the two identified areas as increased risk to yellow from
green. When questioned by the team as to why inventory control risk
had increased as well as electrical resources, the STA acknowledged
he had made an error and inventory control should not have been
increased to yellow risk. The STA corrected the error for inventory
control and downgraded the risk to green. The licensee generated
PVAR 3072733 to document this issue.
October 11, 2007. The team noted that the shutdown risk
assessment did not include the availability of the Train A EDG in the
shutdown risk assessment. The shutdown risk assessment was
evaluated as the EDG being unavailable placing the unit in the
incorrect yellow risk category for electrical resources. When the team
questioned the STA about the Train A EDG status; the STA was not
aware the SM had declared the EDG available. Procedure
70DP-0RA01, Shutdown Risk Assessment, Section 3.1 required the
STA to provide actual plant conditions for determining the plant
shutdown risk profile. Contrary to this, the STA failed to correctly
evaluate risk for the electrical resources and placed the Unit in a
yellow risk status when it should have been in a green risk status. A
contributing cause to this incorrect shutdown risk assessment was the
lack of timely information being made available to all control room
staff members in reference to the status of the Train A EDG.
Using IMC 0612, Appendix E, "Examples of Minor Issues," these
examples were determined to be minor because they were an
insignificant procedural error and there were no safety consequences.
The performance deficiencies are being documented because of insights
associated with control room behaviors and maintenance rule
implementation.
- 93 - Enclosure
5.6.4 Control of Fission Barriers
The team determined that the programs outlining configuration control of components and
equipment related to fission product barriers were adequate. During a walkdown of
containment, the team noted discrepancies with rigging of the personnel air lock (PAL)
door that had the potential to impact the functionality of the PAL door.
a. Inspection Scope
The team observed a selected portion of the containment isolation lineup to
independently verify whether valves, dampers, and airlock doors were being properly
controlled in accordance with the licensing and design bases. The team reviewed
plant drawings and system procedures to verify that selected components were in their
required positions. The team conducted interviews and reviewed the licensees
policies to assess whether the programs and controls (tracking systems) in place for
maintaining knowledge of the configuration of the fission product barriers including:
containment leakage monitoring and tracking, containment isolation device operability
(valves, blank flanges), and reactor coolant leak-rate calculation and monitoring were
adequate. The team also observed selected containment isolation tests to
independently verify whether the valves were being properly controlled in accordance
with 10 CFR Part 50, Appendix J, "Primary Reactor Containment Leakage Testing for
Water-Cooled Power Reactors," and local leak rate testing programs.
b. Observation and Findings
b.1 Incorrect Rigging for Personnel Air Lock Door
Introduction. The team identified a Green NCV of 10 CFR Part 50, Appendix B,
Criterion V, "Instructions, Procedures, and Drawings," for the failure of
maintenance personnel to follow procedures to rig the Unit 3 100 foot elevation
inner PAL door. Specifically, the suspended rigging was completed with the
inappropriate placement of the wire rope slings over two of the locking pins
resulting in an unanalyzed force being applied to the doors operating
mechanism.
Description. On October 2, 2007, during a walk down of the Unit 3 containment,
an inappropriate rigging configuration of the Unit 3 100 foot elevation inner PAL
door was identified. The team questioned the Bechtel rigging engineer on the
placement of the wire rope slings over the locking pins of the door. The Bechtel
rigging engineer explained the tension forces developed for the basket rigging
configuration of the door, but did not provide any additional supporting
information to address the teams questions. On October 2, 2007, Bechtel
generated NCR 25030-U3-035 to document that the rigging configuration was
not completed in accordance with Bechtel Drawing U3-FSK-C-022. Specifically,
the shackles shown on drawing Section D-D, Item 9, were installed inverted and
the slings shown on drawing Section D-D, Item 11, were installed over the
existing door closure pin instead of behind the pin. The licensee generated
PVAR 3070843 to document that the PAL door rigging installation was in error.
The teams review of Procedure VTD-T966-0001,Section XIII, Maintenance, on
lubrication, identified that the door latch pin guides each have bronze bushings.
- 94 - Enclosure
The bronze bushing in the door latch pin guide was not a fixed support. The
identification of a bushing that was not designed for vertical loading invalidated
the Bechtel engineering evaluation bounding assumption that the configuration
was in cantilever loading. The licensee generated PVAR 3086057 to document
that the PVAR 3070843 and NCR 25030-U3-035 responses were not adequate,
and that there was potential bushing damage.
Analysis. The performance deficiency associated with this finding was the failure
of maintenance personnel to rig the Unit 3 100 foot elevation inner PAL door in
accordance with WO 2688885, and the subsequent failure to adequately
evaluate any potential impacts from the unanalyzed rigging configuration. The
finding is greater than minor because it would become a more significant safety
concern if left uncorrected in that the applied suspended force on the bronze
bushing and the doors operating mechanism, which were not designed for
vertical loading, could degrade the PAL door sealing capability. This finding
could not be evaluated by the significance determination process because
IMC 0609, "Significance Determination Process," Appendix A, "Determining the
Significance of Reactor Inspection Findings for At-Power Situations," and
Appendix G, "Shutdown Operations Significance Determination Process," did not
apply to the PAL door for the plant conditions that existed during the event. This
finding affects the barrier integrity cornerstone and is determined to have very
low safety significance (Green) by NRC management review using the IMC 0609,
"Significance Determination Process," Appendix M, "Significance Determination
Process Using Qualitative Criteria," because it is a deficiency that did not result in
the actual breach of the containment barrier. The cause of this finding had
crosscutting aspects associated with work practices of the human performance
area in that maintenance personnel failed to provide adequate oversight of work
activities, including contractors, such that nuclear safety was supported (H.4.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures and Drawings," requires, in part, that activities affecting quality shall
be accomplished in accordance with prescribed instructions, procedures, and
drawings. Contrary to the above, Bechtel construction workers failed to rig the
Unit 3 100 foot elevation inner personnel air lock door per Bechtel Drawing
U3-FSK-C-022 and Work Order 2688885. Specifically, the suspended rigging
was completed with the inappropriate placement of the wire rope slings over two
of the locking pins resulting in an unanalyzed force being applied to the doors
operating mechanism. The slings were required to be placed under the locking
pins, not over. Because this violation was of very low safety significance and
was entered into the corrective action program as PVAR 3086057, the issue was
treated as a NCV consistent with Section VI.A.1 of the NRC Enforcement Policy
NCV 05000530/2007012-12, Incorrect Rigging for Personnel Air Lock Door.
- 95 - Enclosure
5.6.5 Review of Individual Plant Examination
a. Inspection Scope
The inspection team reviewed the results of the plant specific Individual Plant
Examination relative to selected systems to determine if the Individual Plant
Examination is being maintained to reflect actual system conditions regarding system
capability and reliability.
b. Observations and Findings
No findings or observations were identified.
5.6.6 Human Performance
a. Inspection Scope
The team observed several maintenance related work activities to determine if Palo
Verde personnel effectively identified, evaluated, and corrected deficiencies involving
human performance. The team observed pre-job briefings, clearance order activities,
and work performance.
b. Findings and Observations
b.1 Observations and Minor Violations Involving Human Performance
b.1.1 Inadequate Procedure for Adjustment of Polar Crane Limit Switch
Technical Specification 5.4.1.a, requires, in part, that written procedures
be established, implemented, and maintained covering the activities
specified in Appendix A, "Typical Procedures for Pressurized Water
Reactors and Boiling Water Reactors," of Regulatory Guide 1.33, "Quality
Assurance Program Requirements (Operations)," dated February 1978.
Regulatory Guide 1.33, Appendix A, Section 9a, requires maintenance
that can affect safety-related equipment be properly preplanned and
performed in accordance with written instructions, documented
instructions and drawings appropriate to the circumstances. Contrary to
this, on October 9, 2007, the licensee performed maintenance without the
appropriate instructions and drawings resulting in a failure to retain quality
related documents and an incorrect evaluation of maintenance results.
Specifically, on October 10, 2007, the team identified that WO 3068693
did not contain appropriate direction for the setting of the 18 foot
maximum limit switch position for the Unit 3 polar crane main hoist
resulting in the electrical technicians documenting a height of 18 foot
0.375 inches when the actual height was 17 foot 6.375 inches. Using
IMC 0612, Appendix E, "Examples of Minor Issues," this finding was
determined to be minor because this was an insignificant procedural error
and there were no safety consequences. This finding was of very low
safety significance and was entered into the CAP as PVARs 3073911,
3074132 and 3086770. This performance deficiency is being
documented because of insights associated with inadequate procedures.
- 96 - Enclosure
5.6.7 Design
a. Inspection Scope
The team conducted general walkdowns of the containment and auxiliary buildings
and reviewed current component configuration, material condition, and equipment
status. The team also reviewed a sample of PVARs and CRDRs to assess the
effectiveness of corrective actions for deficiencies involving design activities. During
the walkdown and review the team noted discrepancies with pressurizer instrument
brackets and breaker modifications.
b. Observations and Findings
b.1 Failure to Maintain Configuration Control of Pressurizer Instrument Condensing
Pot Support Brackets
Introduction. The team identified a NCV of 10 CFR Part 50, Appendix B,
Criterion V, "Instructions, Procedures and Drawings," for the failure of
maintenance and engineering personnel to maintain proper configuration of the
support brackets for the pressurizer condensate pots in accordance with design
drawings. Specifically, on October 2, 2007, the team identified that the support
bracket U-bolts were not tight against the condensate pot piping, jam nuts were
not installed on the U-bolts, and jacking bolts were not in full contact with the
pressurizer vessel. The support brackets minimize lateral motion during a
seismic event.
Description. On October 11, 2007, the team conducted a containment walkdown
and observed that the support brackets for Valves 3PRCCV204 and
3PRCAV206, (pressurizer instrumentation root valves), had different
configurations. The licensee evaluated the brackets and determined that they
were not configured in accordance with design Drawings 13-J-ZZS-0080,
"Condensing Pot Support Details," and 13-J-ZZS-0081, "Condensing Pot Support
Details Pressurizer. The design drawings stated to field tighten the jacking bolt
stud to the pressurizer vessel hand tight, then add jam nuts; and the U-bolts to
be field tightened to obtain zero clearance around the pipe, then secured with a
jam nut. The bracket for Valve 3PRCCV204 had both U-bolts in full contact with
the pipe and 3 of the 4 jack bolt studs in contact with the pressurizer vessel. The
bracket for Valve 3PRCAV206 had 1 of 2 U-bolts in full contact with the pipe and
3 of the 4 jack bolt studs in contact with the pressurizer vessel. The licensee
entered the issue into the CAP as PVAR 3075704 and generated CRDR
3078397 and corrective maintenance WO 3076022 to resolve the deficiency.
On October 13, 2007, the licensee performed an OD which determined that
based on Calculation 13-MC-ZZ-0037, "Evaluation of Double U-Bolts Used as an
Anchor Restraint," only 1 of 2 U-bolts was required to maintain the design
function of the support; and Calculation 13-MC-RC-501, "RCS - Pressurizer
Surge Line," indicated that there was margin in the design to transfer the load to
the remaining jack bolt studs. Civil engineering determined that the incorrect
support configuration was acceptable without an adverse effect on the subject
pipe stresses and pipe support design. PVAR 3075704 identified a need to
- 97 - Enclosure
review the potential transportability to the other units and similar valves around
the pressurizer using this hanger design.
On October 30, 2007, the team visually inspected the support brackets for
Pressurizer Instrument Root Valves 3PRCDV205 and 3PRCBV207. The team
identified that 2 of 4 jack bolts on Valves 3PRCDV205 and 3PRCBV207 were not
in contact with the pressurizer vessel in accordance with design Drawings
13-J-ZZS-0080 and 13-J-ZZS-0081. The team noted that the original immediate
OD stated that there was a margin in the design to transfer the load to the
remaining 3 of 4 jack bolts still in contact with the pressurizer vessel, not when 2
of 4 jack bolts were not in contact. Civil engineering personnel evaluated the
effect of 2 jack bolts not being in contact with the pressurizer vessel and
determined that this condition was acceptable without an adverse affect to the
subject pipe stresses and pipe support design/evaluation.
On November 5, 2007, the licensee completed WO 3076022 to correct the
deficiencies identified in the support brackets associated with Valves
3PRCCV204, 3PRCDV205, 3PRCAV206, and 3PRCBV207, restoring the
support brackets in accordance with design drawings.
On November 6, 2007, the team visually inspected the support brackets for Valve
3PRCDV205 and 3PRCBV207 and identified that the bracket for Valve
3PRCDV205 was missing the jam nuts for the U-bolt farthest from the
pressurizer vessel. WO 3076022 indicated that the bracket U-bolts were
restored to the appropriate configuration and verified by civil engineering on
November 3, 2007. This issue was entered into the CAP as PVAR 3089364.
Analysis. The performance deficiency associated with this finding was the failure
of maintenance and engineering personnel to maintain proper configuration of
the support brackets on Valves 3PRCCV204, 3PRCDV205, 3PRCAV206, and
3PRCBV207 in accordance with the design drawings. This finding is greater
than minor because it is associated with the mitigating systems cornerstone
attribute of equipment performance and affected the cornerstone objective of
ensuring the availability and reliability of systems that respond to initiating events
to prevent undesirable consequences. Using the Manual Chapter 0609,
"Significance Determination Process," Phase 1 Worksheets, the finding is
determined to have very low safety significance (Green) since it only affected the
mitigating systems cornerstone and did not represent a loss of system safety
function. This finding had crosscutting aspects associated with the work
practices component of the human performance area because maintenance
personnel did not effectively use human error prevention techniques such as self
checking and proper documentation of activities for the installation of the support
bracket (H.4.(a)).
Enforcement. 10 CFR Part 50, Criterion V, "Instructions, Procedures and
Drawings," requires, in part, that activities affecting quality shall be prescribed by
documented instructions, procedures, or drawings and shall be accomplished in
accordance with these instructions, procedures, or drawings. Contrary to this,
since 2003, maintenance personnel did not ensure that Unit 3's support brackets
for Valves 3PRCCV204, 3PRCDV205, 3PRCAV206 and 3PRCBV207 were
configured and maintained in accordance with design drawings 13-J-ZZS-080
- 98 - Enclosure
and 13-J-ZZS-081. Specifically, the support bracket U-bolts were not tight
against the pipe, jam nuts were not installed on the U-bolts, and jacking bolts
were not in full contact with the pressurizer vessel. Because the finding was of
very low safety significance and was entered into the CAP as PVAR 3070805
and 3075704, this violation was treated as an NCV consistent with Section VI.A
of the Enforcement Policy: NCV 05000530/2007012-13, "Failure to Maintain
Configuration Control of Pressurizer Instrument Condensing Pot Support
Brackets."
b.2 Observations and Minor Violations Involving Design
b.2.1 Lack of Design Control for Breaker Modification
The team identified a minor finding for the failure of engineering
personnel to maintain design control measures for a temporary electrical
power modification per Procedure 01DP-0CC01, "Configuration Control,"
Revision 0. The team identified that a modification to install 70 amp
breakers in place of 60 amp breakers for temporary power used during
the outage to power instrument air and breathing air was placed on the
cancelled modifications list. After questioning engineering personnel, the
team determined the modification was cancelled before full
implementation. Plant drawings were updated for Unit 3 to reflect a 70
amp breaker installation. No changes to drawings were made for Units 1
and 2. During a plant walkdown, the team discovered all 60 amp
breakers were installed in each of the three units. The licensee was
unaware the modification was on the cancelled modifications list and
records indicated the modification had been completed in October 1993.
This finding was determined to be of very low safety significance because
the cancelled modification was for temporary power for instrument air and
breathing air and did not affect any safety related equipment. The
licensee placed the issue into their CAP as PVAR 3068451.
5.6.8 Problem Identification & Resolution
a. Inspection Scope
The team conducted general walkdowns of the containment and auxiliary buildings.
The team reviewed current component configuration, material condition, and
equipment status. The team also reviewed a sample of PVARs and CRDRs to assess
the effectiveness of corrective actions for degraded and unanalyzed conditions. The
team ensured that licensee evaluations of, and corrective actions to, significant
performance deficiencies have been sufficient to correct the deficiencies and prevent
recurrence.
b. Observations and Findings
b.1 Failure to Evaluate Adverse Condition for the Emergency Diesel Generators
Introduction. The team identified an eighth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and
- 99 - Enclosure
Drawings," for the failure of operations and engineering personnel to adequately
evaluate degraded and unanalyzed conditions to support operability decision
making associated with EDG leaks.
Description. On October 2, 2007, the team conducted a walk down of the Unit 3
EDGs. During the walk down, several puddles of oil and surfaces wet with fluids
were identified. The observations were shared with the licensee who stated that
CRDR 2914886 initiated August 1, 2006, addressed the issue of lube oil leaks.
In response to the teams observations, maintenance personnel conducted
additional walk downs of the Unit 3 EDGs to make an assessment of any new
leaks.
The team reviewed the evaluation for CRDR 2914886 which stated that,
"Engineering, operations, and maintenance were aware of the several small oil
leaks but no program existed to quantify the leakage, nor had an evaluation of
the aggregate impact been performed." The team also reviewed CRAI 2979205
completed on June 6, 2007, that contained an engineering evaluation of the
maximum allowable leak rate for diesel lube oil of 0.5 gallons per hour (gph) was
acceptable. This was based on the lube oil burn rate of 1.0 gph such that a total
net lube oil consumption rate of 1.5 gph for seven days would not exceed the
Technical Specification bases. Additionally, the team reviewed engineering white
paper, "EDG Fluid Leakage and Operability," issued December 1, 2006. The
white paper listed several areas that were known to leak and gave some general
guidance on leak locations that would be of operational concern. The guidance
also listed several leak locations that were considered nuisance leaks and that
minor drips or weeps were not an operability concern. However, no definition of
what quantity of leakage would be considered minor or nuisance was provided.
The team reviewed the EDG fluid leakage database used to track leaks that are
being monitored. The database listed the source, WOs written, and internal
engineering severity rankings. Engineering classified all of the identified leaks as
minor with varying severity rankings. The licensee concluded that none of the
individual leaks would challenge the operability of the EDGs. Concerned that the
total aggregate of all of the leaks may exceed the allowed leak rate, the team
questioned operability based on the number and location of leaks.
Procedure 40DP-9OP26, "Operability Determination and Functional
Assessment," Revision 18, Step 3.1.1, stated, in part, that the OD process is
entered upon discovery of circumstances where operability of any SSCs
described in Technical Specifications is called into question upon discovery of a
degraded, nonconforming, or credible unanalyzed condition. However,
engineering personnel stated that only individual leaks greater than 0.5 gph
would be of concern for operability and performing a quantitative evaluation or
aggregating all the oil leaks would be too difficult. Engineering personnel
acknowledged that it would be beneficial to determine if the total oil leak rate
exceeded 0.5 gph.
The team performed walkdowns to determine if additional leaks existed. Based
on transportability of oil and poor EDG housekeeping, the team was unable to
determine if leaks, other than the leaks listed in the EDG fluid leakage database,
existed. While none of the individual leaks identified were determined to
challenge the operability of the EDGs (each was less than 0.5 gph), the team
- 100 - Enclosure
expressed their concern about the adequacy of the licensee's program to identify
individual leak rates and track the aggregate leak rates of the EDGs to ensure
that material condition issues would not create a challenge to operability.
Analysis. The performance deficiency associated with this finding was the failure
of operations and engineering personnel to adequately evaluate degraded and
unanalyzed conditions to support operability decision making. This finding is
greater than minor because it would become a more significant safety concern if
left uncorrected in that unanalyzed conditions could challenge the operability of
the EDGs. The finding affected the mitigating systems cornerstone. Using the
IMC 0609, "Significance Determination Process," Phase 1 Worksheets, the
finding is determined to have very low safety significance (Green) because the
finding did not result in the actual loss of safety function. The cause of this
finding had a crosscutting aspect associated with corrective action of the PI&R
area in that the licensee did not thoroughly evaluate previous EDG leaks such
that the resolutions addressed all conditions affecting operability (P.1.(c)).
Enforcement. 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures and Drawings," requires that activities affecting quality shall be
prescribed by instructions, procedures, or drawings, and shall be accomplished
in accordance with those instructions, procedures, and drawings. The
assessment of operability of safety-related equipment needed to mitigate
accidents was an activity affecting quality, and was implemented by
Procedure 40DP-9OP26, "Operability Determination and Functional
Assessment," Revision 18. Procedure 40DP-9OP26, Step 3.1.1, stated, in part,
that the OD process is entered upon discovery of circumstances where
operability of any SSC described in the Technical Specifications is called into
question upon discovery of a degraded, nonconforming, or credible unanalyzed
condition. Contrary to the above, between August 1, 2006 and October 2, 2007,
operations and engineering personnel failed to enter the OD process upon
discovery of circumstances where the operability of a component was called into
question. Specifically, operations and engineering personnel failed to consider
all relevant information to perform an adequate OD when evaluating aggregate
EDG lube oil leaks. This was the eighth example of the NCV involving the failure
to implement the OD program. This example was of very low safety significance
and was entered into the licensees CAP as PVAR 3073559.
b.2 Failure to Identify and Correct a Non-Conforming Condition of Post-Accident
Monitoring Instrumentation Recorders
Introduction. The team identified a sixth example of the Green NCV of
10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action," for the failure to
promptly correct a nonconforming condition that resulted in the inoperability of
several post accident monitoring (PAM) chart recorders.
Description. On October 10, 2007, the team conducted a Unit 3 control room
walk down and observed that several PAM chart recorders had significant ink
bleeding on the paper roll and that pens had been removed from several
instruments. Operations personnel stated that this was normal due to the design
of the pens, that the bleeding rendered the affected chart recorders unusable for
historical trending, and that if the bleeding was severe enough they would pull
- 101 - Enclosure
the pen from the chart recorder. The team questioned the operability of the PAM
chart recorders if the trend plots were unusable or if the pens were pulled. The
team was referred to CRDR 2629437, initiated on August 8, 2003, that indicated
there were no immediate operability concerns, even with the trend data not
usable, because the paper scales of the chart recorders were not calibrated.
During the review of CRDR 2629437, the team noted that the evaluation stated
that no cause could be determined and that the only corrective action was to
track the cause determination and solution implementation. No corrective
actions were identified for removing the pens from the PAM chart recorders.
Based on this cause evaluation, the licensee initiated CRAI 2637936 on
September 28, 2003, to replace the instruments.
On March 9, 2005, during procurement engineerings review of the issue, an
engineer questioned the original operability determination contained in CRDR
2629437, stating that UFSAR Table 1.8-1, "PVNGS Compliance with Regulatory
Guide 1.97 (Revision 2) Requirements," listed chart recorders that are required
for compliance with Regulatory Guide 1.97, Instrumentation for Light-Water-
Cooled Nuclear Power Plants to Assess Plant and Environs Conditions During
and Following an Accident, Revision 2. Regulatory Guide 1.97 states, in part,
where direct and immediate trend or transient information is essential for
operator information or action, recording should be continuously available on
dedicated recorders. CRAI 2790230 was issued to perform another operability
evaluation of the chart recorders. However, this action was not taken until
April 11, 2005, approximately 21 months after the initial concern and over a
month after procurement engineering questioned the original operability
evaluation. Again, this second operability evaluation determined that no
immediate operability concerns existed since there were no surveillance
requirements for the recorders and the Technical Specification basis did not
specifically address the recorder as part of a required PAM channel. The team
noted that both operability evaluations failed to address the UFSAR requirements
for compliance with Regulatory Guide 1.97.
On October 24, 2007, the team conducted additional walk downs of the Units 1
and 2 control rooms. The Unit 1 control room had several recorders with
moderate chart bleeding and two with pens removed. The team noted that the
Unit 2 control room had two recorders with moderate ink bleeding. The team
again questioned the licensee about PAM instrument operability based on the
UFSAR Table 1.8-1 listing of chart recorders that are required for compliance
with Regulatory Guide 1.97. Operations again provided the basis contained in
CRAI 2790230 for continued operability of the chart recorders.
On October 29, 2007, after additional discussions about operability with PVNGS
senior management, the licensee recognized that two chart recorders in the
Unit 1 control room had pens removed. Senior management immediately
directed operations personnel to install the pens and made operations aware of
the requirements to maintain pens in the recorders. During the Unit 2 walk down,
senior management discovered that operations personnel had minimized the ink
bleeding on the chart recorders by removing about half of the ink from the pens.
This interim corrective action was not shared with the other units or documented
in the CAP.
- 102 - Enclosure
On October 29, 2007, the licensee initiated PVAR 3086251. PVAR 3086251
indicated that the recorders were required for trending and recording. All the
recorders were verified to have pens installed and a night order was written to
alert operations personnel about this condition. The night order required
operations personnel to declare the PAM instrument channel inoperable if the
recording function was not available for any reason (including blotching or
bleeding).
The inspectors concluded that the licensee had failed to review the licensing
basis for the PAM chart recorders and failed to implement corrective actions to
maintain the functionality of the instruments. This condition involved multiple
safety and non-safety related recorders that were in a non-conforming condition
for an unspecified period with no controls or compensatory actions in place.
Analysis. The performance deficiency associated with this finding involved the
failure to identify an inadequate operability evaluation and the failure to promptly
correct a non-conforming condition that resulted in the inoperability of PAM chart
recorders. The finding is greater than minor because it would become a more
significant safety concern if left uncorrected in that safety-related equipment that
was not maintained in a qualified condition may not be available to perform its
safety function under certain accident conditions. The finding affected the
mitigating systems cornerstone. Using the IMC 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have
very low safety significance because it did not result in a complete loss of system
safety function. The cause of this finding had crosscutting aspects associated
with corrective actions of the PI&R area in that the licensee did not thoroughly
evaluate previous issues such that the resolutions addressed all conditions
affecting operability (P.1.(c)). The cause of the finding was also related to the
safety culture component of accountability in that management failed to reinforce
safety standards and display behavior that reflected safety as an overriding
priority (O.1.(b)).
Enforcement. 10 CFR Part 50, Criterion XVI, "Corrective Action," requires, in
part, that measures shall be established to assure that conditions adverse to
quality, such as failures, malfunctions, deficiencies, deviations, defective material
and equipment, and nonconformance, are promptly identified and corrected.
Contrary to this, from August 8, 2003, to October 29, 2007, operations personnel
did not promptly identify and correct conditions adverse to quality. Specifically,
licensee personnel unknowingly rendered chart recorders for PAM
instrumentation inoperable by removing the ink pens and failed to take prompt
corrective actions to restore operability of PAM instrument chart recorders. This
was the sixth example of the failure to implement the CAP. This example was of
very low safety significance and was entered into the CAP as CRDR 3088033.
- 103 - Enclosure
5.6.9 Equipment Performance
a. Inspection Scope
The team reviewed the operational performance of selected safety systems to verify
their capability of performing the intended safety functions. The team assessed the
effectiveness of corrective actions for deficiencies involving equipment performance,
including equipment designated for increased monitoring via implementation of the
Maintenance Rule. The team also ensured that the licensee has effectively
implemented programs for control and evaluation of surveillance testing, calibration,
and post-maintenance testing.
b. Observations and Findings
b.1 Failure to Establish Maintenance Rule Goals for the Safety Injection System
Introduction. The team identified a Green NCV of 10 CFR 50.65 for the failure of
engineering personnel to establish goals and monitor the performance of the
safety injection system. Specifically, as of March 22, 2007, engineering
personnel failed to establish goals to properly monitor system performance, or
provide a technical justification to demonstrate that monitoring under
10 CFR 50.65(a)(1) was not required for the safety injection system following the
system changing status from 10 CFR 50.65(a)(2) to 10 CFR 50.65(a)(1).
Description. On October 25, 2007, following the teams request
for 10 CFR 50.65(a)(1) action plans for several risk significant systems, it was
discovered that the licensee had reclassified the safety injection system from 10
CFR 50.65(a)(2) status to 10 CFR 50.65(a)(1) status because of unacceptable
unavailability. Specifically, the HPSI pumps had experienced unavailability
issues and sporadic reliability issues for the last three years. However,
engineering personnel did not establish goals to properly monitor system
performance, or provide a technical justification to demonstrate that monitoring
under 10 CFR 50.65(a)(1) was not required. As a result of the team's questions,
the licensee initiated actions to establish goals and monitoring for the safety
injection system. The team noted that this concern was not identified during the
licensees annual maintenance rule program assessment.
Analysis. The performance deficiency associated with this finding was the failure
of engineering personnel to properly establish goals and monitor system
performance; and provide technical justification for not establishing goals for the
safety injection system. This finding is greater than minor because it was
associated with the mitigating systems cornerstone attribute of equipment
performance and affected the cornerstone objective of ensuring the availability
and reliability of systems that respond to initiating events to prevent undesirable
consequences. Using the IMC 0609, "Significance Determination Process,"
Phase 1 Worksheets, the team concluded the finding is of very low safety
significance (Green) because there was no design deficiency, and the finding did
not represent an actual loss of a safety function. The cause of this finding had
crosscutting aspects associated with corrective action of PI&R area in that
engineering personnel failed to take appropriate actions to address safety issues
and adverse trends in a timely manner (P.1.(d)). The cause of this finding had
- 104 - Enclosure
crosscutting aspects associated with self assessments of the PI&R area in that
engineering personnel failed to perform self assessments that were
comprehensive, appropriately objective, and self-critical (P.3.(a)).
Enforcement. 10 CFR 50.65(a)(1) states, in part, that the performance or
condition of systems shall be monitored against established goals, to provide
reasonable assurance that the systems are capable of performing their intended
functions. 10 CFR 50.65(a)(2) requires, in part, that monitoring as specified in
paragraph 10 CFR 50.65(a)(1) is not required where it had been demonstrated
that the performance or condition of a system was being effectively controlled
through the performance of appropriate preventive maintenance such that the
system remained capable of performing its intended function. Contrary to the
above, Between March 22 and October 25, 2007, the licensee failed to establish
goals and monitor the performance of the safety injection system to provide
reasonable assurance that the system was capable of performing its intended
function. Specifically, the licensee determined that the performance of the safety
injection system was such that it was necessary to monitor system performance
against established goals under 10 CFR 50.65(a)(1), yet failed to establish goals
and/or monitor the performance of the system against such goals. Because this
finding is of very low safety significance and had been entered into the CAP as
PVARs 3074255 and 3076699, this violation is being treated as an NCV,
consistent with Section V1.A of the Enforcement Policy: NCV 05000528;
05000529;05000530/2007012-14, Failure to Implement Maintenance Rule
Requirements for the High Pressure Safety Injection System.
5.7 Emergency Preparedness and Response
The team had not originally planned an in-depth review of the Emergency Response
Strategic Performance Area. However, between October 1 and 12, 2007, the team
identified significant issues with the licensees ability to correctly classify an emergency
condition and/or determine a Protective Action Recommendation (PAR). Between
October 29 and November 2, 2007, emergency planning specialists from both NRC
Region IV and Headquarters were added to the team to conduct a more detailed
emergency response assessment. Further review by the team noted significant
knowledge gaps associated with emergency classifications and PARs, and a failure to
correct identified weaknesses. On October 28, 2007, in response to the problems
identified by the team, the licensee instituted corrective actions to augment the emergency
response organization (ERO) by assigning 6 managers, specially trained on EAL
classification, to the shift rotation until additional training could be provided to the
remaining ERO members. The team determined that this interim measure should be
effective in improving EAL implementation. Nevertheless, significant improvement in
emergency response program knowledge, and correction of emergency plan weaknesses
was warranted.
a. Inspection Scope
The team conducted a limited assessment of the ability of licensee personnel to
activate the ERO augmentation of on-shift personnel. The team assessed the
effectiveness of prior corrective actions involving ERO deficiencies. Although, no ERO
drills were conducted or reviewed during this evaluation, the team reviewed
emergency response facilities, planned on-shift emergency response, and augmented
- 105 - Enclosure
emergency response staffing. The team selected 10 members of the ERO and tested
their ability to implement EAL event classifications. Since the Emergency
Preparedness Cornerstone was not degraded, IP Attachment 95003.01, Emergency
Preparedness, was not conducted.
b. Observations and Findings
b.1 Failure to Correct a Risk Significant Planning Standard
Introduction: The team identified an apparent violation with the significance to be
determined for the licensees failure to correct an identified risk significant
planning standard weakness from May 2, 2007, through October 28, 2007. The
finding had a potential safety significance of White.
Description: 10 CFR Part 50, Appendix E.IV.F.2.g., requires, in part, that any
deficiencies identified as a result of training, exercises, or drills be corrected.
Between May 2 and October 28, 2007, the licensee failed to implement adequate
corrective actions for identified deficiencies which impacted a risk significant
planning standard associated with the ability to make EAL declarations.
Background:
For a steam generator tube rupture (SGTR), with a 200 gpm primary/secondary
leak, valid reactor vessel level monitoring system (RVLMS) level < 21 percent
plenum level, and the use of automatic depressurization valves (ADVs) with the
secondary plant stabilized, EPIP-99, EPIP Standard Appendices, Table 1,
Fission Product Barrier Reference (Modes 1-4), specified the following EAL
classification.
- 106 - Enclosure
FUEL CLAD RCS BARRIER CONTAINMENT
BARRIER BARRIER
POTENTIAL POTENTIAL LOSS LOSS
LOSS LOSS
Valid RVLMS SGTR > 44 SGTR >132 gpm Release of
currently or gpm with a prolonged contaminated
previously < 21 (EAL 1-7) release of secondary side to
percent plenum contaminated atmosphere (i.e., S/G
(EAL 1-2) secondary coolant safety or ADV) with
occurring from the S/G primary to
ruptured S/G to the secondary leakage >
environment (See Technical Specification
Limitations in allowable limits
Section 1) (EAL 1-14)
(EAL 1-7)
APPLY THE CRITERIA ABOVE TO THE CONDITIONS BELOW
UNUSUAL ALERT SITE AREA GENERAL
EVENT EMERGENCY EMERGENCY
Any loss OR Any loss OR Loss of both fuel Loss of any two
any potential any potential clad and RCS barriers
loss of loss of either Or And
containment fuel clad or Potential loss of Potential loss of a third
reactor coolant both fuel clad and barrier
Or
Potential loss of
either fuel clad or
RCS and loss of
any additional
barrier
EPIP-99, Section 1, Precautions and Limitations, Step 1.7, stated, Used in the
context of a steam generator tube rupture as stated in the Fission Product Barrier
EAL [1-7], a "prolonged release of contaminated secondary coolant"
encompasses a main steam line break, feedwater line break, stuck open steam
generator safety and/or atmospheric dump valve(s), and plant cooldown (i.e., to
Mode 5) while steaming the affected steam generator to atmosphere. The team
noted that for the associated EAL JPMs, the licensee was using the ADVs to
stabilize the secondary plant (a plant cooldown was not in progress). The correct
emergency classification was a Site Area Emergency based on the following
conditions: SGTR >44 gpm resulting in a potential loss of the RCS barrier;
RVLMS <21 percent resulting in a potential loss of the fuel clad barrier; and a
release of contaminated secondary side to atmosphere through the ADVs with
primary to secondary leakage exceeding Technical Specification limits resulting
in a loss of containment barrier.
- 107 - Enclosure
Training Requirement:
Licensed Operator Continuing Training (LOCT) Program Description,
Revision 31, required SROs responsible to fill ERO positions to maintain
emergency preparedness proficiency by receiving annual training to meet EP
training requirements as specified in Section 8.1.1.2, Specialized Training for
Key Emergency Organization Personnel, of EPIP-59, Emergency Planning
Training Program Description. EPIP 59 further defined the necessary training to
maintain emergency preparedness proficiency for onshift emergency
coordinators, which included all of the control room supervisors and SMs.
PVNGS Emergency Plan, Revision 36, Section 3.0 stated, in part, that, the
Emergency Plan was based upon NRC and Federal Emergency Management
Agency (FEMA) guidance as contained in NUREG-0654 (FEMA-REP-1), Criteria
for Preparation and Evaluation of Radiological Emergency Response Plans and
Preparedness in Support of Nuclear Power Plants, Revision 1. NUREG-0654,
Section N, stated, in part, that, periodic drills will be conducted to develop and
maintain key skills, and deficiencies identified as a result of drills will be
corrected. NUREG-0654 further defined a drill as a supervised instruction period
aimed at testing, developing, and maintaining skills.
Operator Licensing Requalification Program EP Classification Failures:
As part of LOCT Cycle 3 (April 3 - May 4, 2007), the licensee included
JPM EP009-CR-002, "Direct the Emergency Response as the Emergency
Coordinator," as part of their training to maintain emergency preparedness
readiness. This JPM consisted of a SGTR event with the following conditions:
200 gpm primary/secondary leakage, valid RVLMS level < 21 percent plenum
level, and the use of automatic depressurization valves to control steam
generator pressure. The evaluation standard (expected trainee response), which
was incorrect for this event, was a General Emergency based on, Loss of any
two barriers AND Potential Loss of a third barrier. The incorrect classification
resulted from the misapplication of EPIP-99, Section 1, Precautions and
Limitations, Step 1.7. JPM EP009-CR-002 identified the EAL classification as a
General Emergency because of an incorrect assumption that under the
described conditions a prolonged release was occurring, when the definition of
prolonged release did not apply (see above description).
From April 4 through May 2, 2007, 10 SROs were given this JPM and were
asked to identify the EAL classification. Nine of 10 SROs classified a General
Emergency, while one classified a Site Area Emergency. On May 2, the SRO
who classified the event as a Site Area Emergency identified that the evaluation
standard was incorrect because under the presented conditions only one barrier
was lost (Containment, use of automatic depressurization valves) and two
potentially lost barriers (RVLMS level < 21 percent plenum, SGTR > 44 gpm).
Under these conditions the correct classification was a Site Area Emergency
(see above table). After discussing this with the emergency planning personnel,
the instructors determined that this event should have been classified as a Site
Area Emergency and the 9 SROs that classified the event as a General
Emergency were given immediate remedial training (per the Training
Supervisor). However, the licensee failed to enter the incorrect evaluation
- 108 - Enclosure
standard into either the CAP or the training deficiency program and no additional
training was given to the other ERO personnel responsible for classifying events.
The licensee did not remove JPM EP009-CR-002 from the training bank or make
any corrections to the JPM.
Initial Exam EP Classification Failures:
JPM SA-5 (identical to JPM-EP009-CR-002) was administered during an initial
license examination on July 27, 2007. The evaluation criteria incorrectly
specified the classification as a General Emergency. Two of the five SRO
candidates classified the event as a Site Area Emergency, while the other three
classified the event as a General Emergency. An evaluation of the JPM was
conducted that day by training personnel. An EPIP training instructor recognized
that the misclassification issue involved the same concern from the JPM that was
given in LOCT Cycle 3. On July 27, 2007, the licensee entered the
misclassification issue into the CAP as CRDR 3046233, Incorrect Interpretation
of Event Conditions During the Creation of and Administration of an NRC Exam
JPM, and conducted an apparent cause evaluation. The apparent cause
evaluation was completed on August 31, 2007. As of October 5, 2007, no
training had been conducted on what constituted a prolonged release and the
proper classification for SGTR events. The team noted that training on this
particular SGTR event was not scheduled to be completed until
November 30, 2007.
The licensee identified three apparent causes of the performance deficiency:
(1) a lack of knowledge/understanding on the specific conditions of EAL 1-7;
(2) insufficient use of the Limitations in Section 1 referenced in the EAL 1-7
description box in Table 1 of EPIP-99; and (3) insufficient use of the technical
bases in Appendix P of EPIP-99. The team determined these apparent causes
stemmed from inadequate training, in that SROs were given generalized initial
and continuing training on EALs and were not provided systematic training on the
entry conditions and basis for individual EALs to ensure their understanding of
entry conditions.
IP 95003 Emergency Plan (EP) Classification Failures:
As a result of the incorrect EAL classifications during the operator licensing initial
exam in July 2007, the team selected JPM EP009-CR-002 to test the ability of
ERO personnel to properly classify a SGTR event and to verify that the licensee
had taken actions to correct the knowledge deficiencies associated with the
SGTR EAL classification. The team was unaware of the additional failures
associated with this JPM during LOCT Cycle 3 training. The team administered
JPM EP009-CR-002, to one SRO. The JPM contained the exact same
conditions as described above: 200 gpm primary/secondary leak, valid RVLMS
level < 21 percent plenum level, and the use of ADVs to control steam generator
pressure. The SRO incorrectly classified the event as a General Emergency
verses a Site Area Emergency.
Due to this additional failure, the licensee implemented immediate corrective
actions to provide intensive training to six managers and assigned them to shift
rotations, beginning on October 28, 2007, to assist ERO personnel in making
- 109 - Enclosure
EAL declarations. These six managers were to remain on-shift until the licensee
completed their review of the other EALs and provided training to the remainder
of the applicable ERO positions. Between October 9 and November 16, 2007,
the licensee did provide specific training on EAL 1-7 to the applicable ERO
positions.
Analysis: The team determined that the failure to correct an identified risk
significant planning standard weakness was a performance deficiency. This
finding was more than minor because it was associated with the Emergency
Preparedness attribute of response organization performance and could affect
the cornerstone objective to implement adequate measures to protect the health
and safety of the public because of the licensees inability to properly classify an
emergency condition. This finding was evaluated using the Emergency
Preparedness SDP and was preliminarily determined to be of low to moderate
safety significance because it was a failure to comply with NRC requirements; it
was an issue associated with the requirements of Appendix E of 10 CFR Part 50;
it was not an issue with a risk significant planning standard as described in
Manual Chapter 0609, Appendix B, Section 2.0; and it was a functional failure of
the requirements of Appendix E IV.F.2.g because the licensee failed to correct a
weakness associated with Risk Significant Planning Standard
10 CFR 50.47(b)(4). The cause of this finding had crosscutting aspects
associated with corrective action of the PI&R area in that the licensee failed to
thoroughly evaluate problems such that resolutions ensured that the problems
were resolved (P.1.(c)). The cause of this finding was also related to the safety
culture component of accountability in that the licensee failed to demonstrate a
proper safety focus and reinforce safety principles (O.1.(c)).
Enforcement: 10 CFR 50.54(q) states in part, that, a licensee authorized to
possess and operate a nuclear power reactor shall follow and maintain in effect
emergency plans which meet the standards in §50.47(b) and the requirements in
10 CFR Part 50, Appendix E. 10 CFR Part 50, Appendix E, Section IV.F.2.g,
states, in part, that all training shall provide formal critiques in order to identify
deficient areas. Any deficiencies that are identified shall be corrected.
Contrary to the above, between May 2, 2007, and October 28, 2007, the licensee
failed to correct identified deficiencies pertaining to the ability to correctly
implement EALs for one Site Area Emergency classification associated with a
SGTR event. Specifically, the deficiency involved licensee personnel being
unable to consistently implement EAL 1-7 associated with a SGTR which
resulted in an over classification of a Site Area Emergency as a General
Emergency. The issue associated with EAL implementation was entered into the
licensees correction action program as PVAR 3083911. Pending determination
of the findings final safety significance, this finding was identified as Apparent
Violation (AV) 05000528, 05000529,0500030/2007012-15, Failure to Correct a
Risk Significant Planning Standard.
b.2 Inability to Implement Emergency Action Levels (EALs)
Introduction: The team identified a Green NCV for the failure to correctly
implement two EALs as required by 10 CFR 50.54(q) and 10 CFR 50.47(b)(4).
Specifically, between January 2006 and October 2007 the licensee was not able
- 110 - Enclosure
to implement one EAL at the Alert level and over-classified one Notification of
Unusual Event EAL at the Alert level.
Description: The team identified a performance deficiency related to the
licensees inability to ensure implementation of EALs associated with an
aircraft/airliner attack threat and remote shutdown panel area high radiation
levels.
Aircraft/Airliner Threat
In January 2006 the licensee added EAL 7-1 in response to NRC Bulletin 2005-002, dated July 18, 2005. The EAL was associated with an aircraft and
airliner attack threat. The EAL action was defined as follows:
- EAL 7-1 required declaration of an Unusual Event when the NRC notified
PVNGS of an aircraft threat greater than 30 minutes away.
On October 4 and 5, 2007, the team administered one JPM associated with the
aircraft and airliner attack threat, EAL 7-1, to two licensee ECs. The first EC
classified the postulated conditions as an Alert, when the correct classification for
the JPM condition was a Notification of Unusual Event. Licensee management
informed the NRC staff that they would not evaluate the EC for the application of
EAL 7-1 when the JPM was administered to the second EC because they
recognized that they were unable to implement the EAL with existing procedures
and guidance available to the ECs. The team determined that the licensee would
be unable to properly classify this EAL during an actual threat because the
licensee failed to develop implementing procedures for classifying an
aircraft/airliner attack threat.
Procedure EPIP-99, EPIP Standard Appendices, Appendix P, Emergency
Action Level Technical Bases, Revision 15, stated in part, that an airliner was
based on the size of aircraft as defined in the site-specific procedure developed
for response to airborne threats. The team noted that EPIP-99, Revision 15, did
not define an airliner. In response to the teams observation, the licensee issued
EPIP-99, Appendix P, Revision 16, on October 11, 2007, to include the definition
of an airliner as a large aircraft with the potential for causing significant damage
to the plant. The licensee documented the aircraft/airliner EAL classification
findings in PVAR 3070849.
Remote Shutdown Panels
Procedure EPIP-99, EPIP Standard Appendices, Revision 15, EAL 3-12
required an Alert to be declared when radiation levels at the remote shutdown
panels exceeded 5000 mrem/hr as indicated on area radiation Monitor RU-18.
The purpose of this EAL was to identify conditions that could impede the
operation of systems required to establish and/or maintain cold shutdown plant
conditions. The team determined that area radiation Monitor RU-18 was located
inside the control room envelope, on the 140 foot elevation, while the remote
shutdown panels are located one level below, on the 100 foot elevation. The
team determined that area radiation monitors were not installed in the vicinity of
the remote shutdown panels and that area radiation Monitor RU-18 could not be
- 111 - Enclosure
monitored from and did not represent the radiological conditions at the remote
shutdown panels. Therefore, the licensee could not determine the radiation
levels at the remote shutdown panels with radiation Monitor RU-18 and could not
properly classify an Alert condition based on high radiation levels in the area. On
July 13, 1994, this EAL was modified to meet guidance contained within
NUMARC/NESP-007, Methodology for Development of Emergency Action
Levels, Revision 2, and at that time, EAL 3-12 was added to include radiation
readings at the remote shutdown panel. The licensee documented the inability to
declare an Alert based on EAL 3-12 in PVAR 3073229.
Analysis: The team determined that the inability to implement EALs was a
performance deficiency within the licensees ability to foresee and control. The
finding was more than minor because it was associated with the Emergency
Preparedness attribute of procedure quality, and could affect the cornerstone
objective of implementing adequate measures to protect the health and safety of
the public, if the licensee cannot promptly recognize an emergency condition.
Using the IMC 0609, "Significance Determination Process," Appendix B,
Emergency Preparedness Significance Determination Process, the finding was
determined to have a very low safety significance (Green) because the licensee
could be unable to declare one EAL at the Alert and one EAL at the Notification
of Unusual Event level. The cause of this finding had crosscutting aspects
associated with the corrective action of the PI&R area in that the licensee had
previous opportunities to identify the deficiencies (P.1.(a)).
Enforcement: 10 CFR 50.54(q) states, in part, that a licensee authorized to
possess and operate a nuclear power reactor shall follow and maintain in effect
emergency plans which meet the standards in §50.47(b) and the requirements in
10 CFR Part 50, Appendix E. Risk Significant Planning Standard §50.47(b)(4),
states, in part, that a standard emergency classification and action level scheme
shall be used. 10 CFR Part 50, Appendix E, IV(B), states, in part, that the means
for determining the magnitude of and assessing the impact of the release of
radioactive materials shall be described and the EALs shall be based on in-plant
conditions and instrumentation. Contrary to the above, from July 1994 until
October 2007, the licensee failed to have the ability to implement EAL 3-12 at the
Alert level. Specifically, area radiation Monitor RU-18 could not be monitored
from the remote shutdown panels and therefore, the emergency classification
could not be declared as required in Procedure EPIP-99. In addition, from
January 2006 until October 2007, the licensee failed to have the ability to
implement EAL 7-1 resulting in the over-classification of a Notification of Unusual
Event. Specifically, the licensee did not develop a procedure to enable
personnel to define an airliner and therefore, the proper emergency
classifications could not be declared. Because this finding was of very low safety
significance and was entered into the CAP as PVARs 3073229 and 3070849,
this violation was treated as an NCV, consistent with Section VI.A of the
Enforcement Policy: NCV 05000528, 05000529,0500030/2007012-16, Inability
to Implement Emergency Action Levels.
- 112 - Enclosure
b.3 Observations and Minor Violations Involving Emergency Response and
Preparedness
b.3.1 Failure to Notify Offsite Agencies of Emergency Action Level (EAL)
Changes
The team identified a minor violation of 10 CFR 50.54(q) which requires in
part, that, licensees follow and maintain emergency plans which meet the
standards in §50.47(b) and Appendix E. Palo Verdes Emergency Plan,
Section 5.1, Revision 37, stated in part, that, EAL changes would be
discussed and agreed upon with state and county governmental
authorities. Contrary to the above, between January 2005 and
October 2007, the licensee made changes to the EALs without discussing
and obtaining the prior approval of state and county governmental
authorities. The team determined that following a change to
10 CFR Part 50, Appendix E, IV(B), which permitted a licensee to
discontinue the practice of obtaining the prior approval of offsite agencies
for EAL changes under the authority of 10 CFR 50.54(q), the licensee
implemented the change, without changing the requirements of the
Emergency Plan. Using IMC 0612, Appendix E, Examples of Minor
Issues, this finding was determined to be minor because it was similar to
Example 2.d. in that there was no regulatory requirement requiring
approval of EAL changes from offsite agencies and there was no impact
on public health and safety. The performance deficiency was entered into
the licensees corrective action system as PVAR 3085397. This
performance deficiency is being documented because of insights
associated with emergency preparedness concerns.
b.3.2 Failure to Train Emergency Planners
10 CFR 50.54(q) states, in part, that a licensee authorized to possess and
operate a nuclear power reactor shall follow and maintain in effect
emergency plans which meet the standards in §50.47(b).
10 CFR 50.47(b)(16) states in part, that, responsibilities for plan
development and review and for distribution of emergency plans be
established, and planners are properly trained. EPIP-59, Emergency
Planning Training Program Description, Section 1.7.1, stated, Training
for PVNGS Emergency Planning staff is conducted via the completion of
a required reading list and/or other training and includes participation in
industry sponsored emergency planning symposia and workshops.
Contrary to the above, prior to October 2007, not all emergency planners
participated in industry symposia and workshops. Specifically, for one
emergency planner, the licensee was unable to provide documentation or
determine that the individual had ever attended symposia or workshops.
Using IMC 0612, Appendix E, Examples of Minor Issues, this
performance deficiency was determined to be minor since it was similar to
the Example 4.h. in that there were other planners whose qualifications
were current. The performance deficiency was entered into the licensees
corrective action system as PVAR 3086481. This performance deficiency
is being documented because of insights associated with emergency
preparedness concerns.
- 113 - Enclosure
6 RADIATION SAFETY STRATEGIC PERFORMANCE AREA
6.1 Occupational Radiation Safety
A review of radiological work practices was conducted in conjunction with other site
activities that were reviewed in more detail. A number of observations were noted which
identified failures to implement radiological worker expectations and failures to follow
radiological procedures. Areas of note included: the failure to conduct personal
contamination monitoring by radiological workers in the presence of posted signs,
out-of-date surveys, using out-of-date survey information to conduct briefings, and
incomplete radiological briefings. Though this was not a significant focus of the teams
activities, the number of adverse observations indicate improvement is warranted in
implementation of the occupational radiation safety program at Palo Verde.
a. Inspection Scope:
The team did not conduct an in-depth review of the occupational radiation safety
program; however, observations relevant to this Radiation Safety Strategic
Performance Area were collected and assessed to provide insights into Palo Verdes
performance. Work site observations and the results of plant tours, including
radiologically controlled areas, were evaluated to determine if applicable radiological
program procedures were adequately implemented, including worker radiation
exposure controls, radiation work permits, implementation of as low as reasonably
achievable (ALARA) concepts, and effectiveness of work planning, coordination,
implementation, and lessons learned. In addition, the team reviewed a sample of
radiological facilities, equipment, and radiation monitoring instrumentation. Information
relevant to this area was collected during tours of shutdown and operating units
including tours of radiologically controlled areas, the Unit 3 containment, and other
plant areas that contained radioactive material storage areas. Interviews with
radiological protection managers, supervisors, and workers were conducted to provide
additional insights into this performance area. Finally, the contribution of radiological
worker human performance issues identified over the course of this inspection were
assessed to determine if these issues were adequately investigated, evaluated, and
resolved.
b. Observations and Findings:
b.1 Inadequate Briefings on Radiological Conditions
Introduction. The team identified a Green NCV of 10 CFR 19.12 for the failure of
RP personnel to provide adequate information regarding radiological conditions
and precautions to minimize exposure during pre-job briefs.
Description. During select pre-job briefs performed between October 1 and
October 3, 2007, RP personnel failed to provide accurate information regarding
the radiological conditions commensurate with the hazard. For a Unit 3
containment entry briefing that did not involve entry into high radiation areas on
October 1, 2007, dose rate information was communicated by RP personnel
using elevation drawings and pointing to different locations and verbally stating
Aless than 2 mrem/hr, elevated (with no actual dose rates specified), or AHRA
[high radiation area], which your REP [radiation exposure permit] does not allow.@
- 114 - Enclosure
The elevation drawings used for the briefing were not radiological surveys and
contained no dose rate data. In addition, the expected contamination levels were
not reviewed and the RP person giving the briefing did not know if the 80 foot
elevation had been released. Furthermore, although it was the first entry for the
radiological workers, the expected response to dose and dose rate alarms was
not discussed, the expectation to check the electronic dosimeter every 15
minutes was not mentioned, and the electronic dosimeter setpoints were not
reviewed.
During a briefing at the RP control point on the 70 foot elevation of the Unit 3
auxiliary building on October 1, 2007, it was stated there were no high radiation
areas in the Train A CS room, based on information contained in the posted
radiation survey. While performing a walkdown of the room, the team identified a
posted and barricaded high radiation area. Subsequently, the team noted that a
number of the radiation survey maps at the 70 foot RP control point used for the
briefing were out of date, including the survey for the Train A CS room. The
licensee initiated PVAR 3070507 with the action to replace the survey maps with
the most recent version. However, the posted survey maps at the RP control
point for the Train A charging pump room and the 140 foot hot lab were out of
date when used for a briefing on October 3, 2007.
Analysis. The failure of RP personnel to adequately inform workers of the
radiological conditions in the Unit 3 containment and auxiliary building was
determined to be a performance deficiency. This finding is greater than minor
because it is associated with the Occupational Radiation Safety Cornerstone
attribute of program and process and affected the cornerstone objective of
ensuring the adequate protection of the worker health and safety from exposure
to radiation during routine operations. The finding was determined to be of very
low safety significance (Green) because it was not an ALARA issue, there was
not an overexposure or substantial potential for an overexposure, and the ability
to assess dose was not compromised. The cause of the finding had crosscutting
aspects associated with decision making of the human performance area in that
RP personnel performing briefings failed to communicate decisions, and the
basis for decisions, to personnel who had need to know the information to
perform work safely (H.1.(c)). The cause of this finding was also related to the
safety culture component of accountability in that RP personnel failed to
demonstrate a proper safety focus and reinforce safety principles (O.1.(c)).
Enforcement. 10 CFR 19.12 requires, in part, that all individuals who in the
course of employment are likely to receive in a year an occupational dose in
excess of 100 mrem be kept informed of the transfer or use of radioactive
material and in precautions to minimize exposure. Contrary to these
requirements, on October 1 and 3, 2007, RP personnel did not adequately inform
workers of radiological conditions and precautions to minimize exposure during
radiological briefings. Specifically, RP personnel failed to adequately inform
workers of the radiological conditions and precautions/procedures to minimize
exposure in the Unit 3 containment and auxiliary building so that the workers
could take the necessary precautions to minimize exposure. Because the finding
was of very low safety significance and had been entered into the licensee's CAP
- 115 - Enclosure
as PVARs 3070507 and 3071940, this violation was treated as an NCV
consistent with Section VI.A of the NRC Enforcement Policy:
NCV 05000530/2007012-17, Inadequate Briefings on Radiological Conditions.
b.2 Observations and Minor Violations Involving Occupational Radiation Safety
b.2.1 Failure to Conduct Appropriate Radiological Surveys
The team identified a minor violation of 10 CFR 20.1501(a) which
requires, in part, that each licensee make or cause to be made surveys
that may be necessary to comply with regulations in this part, and are
reasonable under the circumstances to evaluate the magnitude and
extent of radiation levels, concentration/quantities of radioactive material,
and the potential radiological hazards. Contrary to the above, on October
1 and 2, 2007, licensee personnel failed to make or cause to be made
surveys to ensure compliance with 10 CFR 20.1201. Specifically, the
team observed radiological workers failing to complete personnel
contamination monitoring surveys in Unit 3 and the 70 foot auxiliary
building and 140 foot fuel building, as specified by signs posted adjacent
to the respective monitoring stations. Using IMC 0612, Appendix B,
Issue Screening, this finding was minor because the survey was an
administrative requirement and there was no unexpected contamination.
The performance deficiency was entered into the CAP as PVARs
3070009 and 3072066. This performance deficiency is being
documented because of insights associated with implementation of RP
program and accountability of management personnel.
Selected aspects of the public radiation safety program were reviewed including; (1) a
sampling of plant facilities, equipment, and instrumentation for radioactive effluent
monitoring, (2) a sampling of procedures affecting the processing, control and discharge of
radioactive effluents, and (3) a sampling of training and qualifications of personnel
involved in radioactive waste and effluent processing. Performance issues identified in
this area related to failures to operate liquid radiological waste tanks in accordance with
station procedures and the UFSAR.
a. Inspection Scope
The team did not conduct an in-depth review of the Public Radiation Safety program;
however, a sampling of program effluent monitoring equipment and radioactive
material controls was evaluated. Unit 3 radiological waste systems were walked down
and valve alignments were compared to system drawing requirements; observations
during site tours and radiological work activities were evaluated against program
requirements. Interviews with managers, supervisors, engineers, and radiological
workers were conducted. Radiological waste system procedures, applicable sections
of the UFSAR, the Offsite Dose Calculation Manual, the Radiological Environmental
Monitoring Report, the 2006 Annual Radioactive Effluent Release Report, radiation
protection self-assessments, and CAP documents were reviewed. In addition, the
Units 1, 2, and 3 radiological waste tank farms were walked down and the operation of
radiological waste systems (total dissolved solids and recycle monitor tanks) were
- 116 - Enclosure
evaluated. The above activities provided insight into the assessment of plant facilities,
equipment, and radiological instrumentation intended for public radiation safety. In
addition, the team used feedback from these activities to evaluate the implementation
of public radiation safety programs and processes, and to evaluate how any observed
human performance issues affected the public radiation safety area.
b. Observations and Findings
b.1 Failure to Periodically Update the Final Safety Analysis Report
Introduction: The team identified a Severity Level IV NCV of 10 CFR 50.71(e) for
the failure of the licensee to periodically update the UFSAR with all changes
made in the facility or procedures.
Description: While conducting a review of the Unit 2 liquid radiological waste
system, the team found that the system was not being operated in accordance
with the description provided in the UFSAR. Specifically, evaporator concentrate
was being pumped to one of the high total dissolved solids (TDS) holdup tanks
rather than the concentrate monitor tanks as specified in Section 11.2.2 of the
The licensee stated that the Unit 2 concentrate monitor system had been out of
service since 2002. The teams review of corrective action documents related to
the system determined that the concentrate monitor tanks were not being used
because of equipment/maintenance issues with the concentrate monitor system.
The UFSAR stated in Section 11.2.2.4.1.2, that flow from the high TDS holdup
tank can be terminated or diverted to an alternate path by operator action based
on evaporator or holdup pump malfunction, high-pressure drop across the
adsorption bed or ion exchangers, an exhausted resin bed, or when the
radiological waste section leader determines it is necessary. The UFSAR did not
specify the alternate flow path nor the allowed duration. The team concluded
that operating outside of the UFSAR design basis for approximately 5 years was
not the intent of UFSAR Section 11.2.2.4.1.2.
Analysis: The team determined that the failure to update the UFSAR to reflect
changes made to the facility was a performance deficiency. This issue was
subject to traditional enforcement because it had the potential for impacting the
NRCs ability to perform its regulatory function. The finding is characterized as a
Severity Level IV violation because the erroneous information in the UFSAR was
not used to make an unacceptable change to the facility or procedures. The
cause of this finding had a crosscutting aspect associated with resources of the
human performance area in that the licensee failed to ensure that personnel and
equipment were available and adequate to maintain radiological safety by
minimization of long-standing equipment issues (H.2.(a)).
Enforcement: 10 CFR 50.71(e) requires that the licensee periodically update the
USFAR with all changes made in the facility or procedures. Contrary to the
above, in 2002 the licensee made a change to the facility and procedures as
described in the UFSAR and failed to update the UFSAR. Specifically, the
licensee began operating the Unit 2 liquid radiological waste system in a manner
different than that specified by UFSAR when they commenced pumping
- 117 - Enclosure
evaporator concentrate to the high TDS holdup tanks rather than the concentrate
monitor tanks as specified in UFSAR Section 11.2.2. The failure to update the
UFSAR was characterized as a Severity Level IV violation. The finding was of
very low safety significance because the change in operation of the total
dissolved solids holdup tanks did not result in an increase in the likelihood of a
release of radioactive material. This issue was entered in the licensees CAP as
PVAR 3075089. This violation was treated as an NCV, consistent with Section
VI.A.1 of the NRC Enforcement Policy: NCV 05000529/2007012-18, Failure to
Periodically Update the Updated Final Safety Analysis Report.
7 SAFEGUARDS STRATEGIC PERFORMANCE AREA
7.1 Safeguards Strategic Performance Area
The team did not conduct an in-depth review of the Safeguards Strategic Performance
Area; however, the team conducted tours of site physical protection areas and evaluated
their attributes and performed spot checks of security equipment. In addition, the team
interviewed security personnel to determine if latent organizational or security equipment
issues exist at Palo Verde. The team also observed the owner controlled area and
protected area access control process. One finding associated with the calculation of
group work hours was identified. The finding is discussed in NRC Inspection Report
05000528, 05000529, 05000530/2007402.
8 SAFETY CULTURE
8.1 Evaluation of the Licensees Independent Safety Culture Assessment
The team determined that the licensees third-party safety culture assessment was
adequate to provide the licensee with the information necessary to develop appropriate
corrective actions for safety culture weaknesses, although limitations in the interpretability
of the survey tool decreased its usefulness to the licensee. Without the many write-in
comments provided by the survey participants, the licensee may not have been able to
use the survey results to develop specific corrective action plans. The results of the
NRCs independent safety culture assessment validated the results of the licensees third-
party safety culture assessment.
a. Inspection Scope
Consistent with inspection requirements in Section 02.07 of IP 95003, the team
evaluated the licensees safety culture assessment to determine whether: (1) the
assessment was comprehensive, (2) the assessment team members were
independent and qualified, (3) the assessment was methodologically sound, (4) the
data collected supported the conclusions derived from the assessment, and (5) the
licensees corrective actions in response to the assessment findings were likely to be
effective.
The team met with licensee representatives and one of the licensees safety culture
assessment contractors (Synergy) at NRC Headquarters in Rockville, Maryland, on
March 14, 2007, to discuss the independent safety culture assessment. The team also
reviewed the licensees plans for conducting the safety culture assessment, the
resumes of the personnel who conducted the assessment and analyzed the data, and
- 118 - Enclosure
the survey instrument and interview guides. Team members and the NRC resident
inspectors observed the administration of the survey on six different occasions
between April 15 - 25, 2007, to verify that the instructions provided to survey
participants were consistent and did not introduce the potential for response biases.
During the week of June 18-21, 2007, the team completed an onsite review of the
preliminary results from the safety culture assessment and conducted interviews with
licensee personnel and members of the assessment team to better understand their
methods to aid in interpreting the preliminary results. In addition, conference calls with
the licensee and Synergy were held on June 27, 2007, and July 26, 2007, to discuss
the measurement properties of the survey instrument and the statistical analyses of
the survey data. During the weeks of October 1-12, 2007, and October 29-
November 2, 2007, the team solicited feedback on the safety culture assessment
during individual and group interviews with site personnel and evaluated the licensees
corrective action plans for addressing identified safety culture weaknesses.
b. Observations
Comprehensiveness
The team concluded that the safety culture assessment provided the licensee with the
information necessary to: (1) develop appropriate corrective actions for the identified
safety culture weaknesses and (2) take actions to maintain the sites safety culture
strengths.
Two teams with different areas of emphasis, using complementary methods,
conducted the assessment. One team, the Independent Safety Culture Performance
Evaluation Team (ISCPET), focused on the effectiveness of the sites policies,
programs, processes, and procedures in establishing that nuclear plant safety issues
receive the attention warranted by their significance. This team conducted interviews,
document reviews, and behavioral observations to obtain information. A second team
focused on the site workforces attitudes and perceptions related to the extent to which
nuclear plant safety issues receive attention. This team, Synergy, collected
information for the assessment by administering a site-wide safety culture survey
augmented by follow-up interviews with site personnel. The combined activities of the
assessment teams addressed all levels of site and corporate management, obtained
safety culture survey responses from approximately 80 percent of the Palo Verde
workforce including contractors, and sampled organizational characteristics and
attitudes related to each of the 13 safety culture components identified in Section
06.07 of NRC IMC 0305, Operating Reactor Assessment Program.
Independence and Qualifications
The team concluded that the licensees safety culture assessment was conducted
independently and that the assessment teams members were qualified. Although
licensee personnel administered the safety culture survey, the NRC teams
observations of survey administration and focus group interviews with Palo Verde staff
indicated that the independence of the effort was not compromised. Licensee
personnel administering the survey followed the instructions provided by the
assessment team and implemented adequate methods for collecting completed
surveys to ensure participants believed their responses would remain anonymous and
confidential.
- 119 - Enclosure
The NRC team verified that the licensees safety culture assessment teams had
unrestricted access to information and opportunities to interview the individuals
necessary to complete the assessment.
The NRC team verified that the assessment teams were composed of individuals with
a knowledge of nuclear safety culture and the topics they were assigned to assess.
The licensee ensured that Synergy subcontracted with an independent professional
survey research firm, Westat, to assist in analyzing the statistical properties of the
survey instrument and the survey results. The additional analyses performed by
Westat enhanced the interpretability of the survey portion of the safety culture
assessment.
Assessment Methods
The team concluded that the methods used to perform the assessment were
appropriate, although some weaknesses in the safety culture survey were identified.
Multi-method approach. The NRC team verified that the assessment teams applied a
multi-method approach to conduct the safety culture assessment, including a survey,
behavioral observations, interviews, and document reviews. Sample sizes for applying
each method obtained representative information, and the teams behavioral
observation and interview guides did not bias the assessment results. The teams
performed their assessment activities in parallel, but compared, contrasted, and
reconciled their findings to ensure they provided integrated assessment results to the
licensee. The NRC teams review of the preliminary results from the teams confirmed
that the large majority of their results were consistent and required little additional data
gathering to reconcile contrasting results.
Survey tool. The team concluded that the safety culture survey appropriately screened
for workforce attitudes and that the most useful information was contained in the write-
in comments provided by the participants. Over half of those participating in the
survey provided write-in comments. The write-in comments provided more detailed
information related to safety culture strengths and weaknesses at the site, and
enhanced the overall usefulness of the results. The NRC team verified that Synergy
had appropriately grouped the write-in comments to identify the recurring safety culture
themes.
Site personnel who participated in the survey and were interviewed by the NRC team
believed that the anonymity of their responses had been maintained and that the
survey gave them an opportunity to express their views on important issues at the site.
None of the participants interviewed reported feeling any pressure to respond to the
survey questions.
Survey participants interviewed by the NRC expressed reservations about the length
of the survey (i.e., they perceived it to be too long and repetitive) and indicated that the
construction of some survey items made it difficult to respond. For example, some
items asked participants to respond with respect to both their managers and
supervisors. Interviewees stated they had difficulty in responding to these items
because their perceptions of their supervisors differed from perceptions of their
managers. The team identified additional examples of survey items that addressed
multiple topics within a single item, which is inconsistent with standard survey design
- 120 - Enclosure
techniques described in IP 95003, Enclosure F, Guidance for Evaluating Safety
Culture Surveys. Licensee personnel who were developing improvement plans also
reported similar interpretation difficulties. Synergy indicated that they did not pilot-test
the safety culture survey on a representative sample of Palo Verde survey participants
before the survey was administered. The team concluded that the licensee may have
been able to make better use of the results had these items been addressed before
administering the survey.
The team noted a low response rate from security personnel on the survey
(approximately 40 percent participated), compared to other functional groups at the
site. Synergy indicated that this response rate is characteristic of security groups at
other sites and results from (1) a perception among security personnel that the survey
items are less relevant to their jobs than to other jobs at nuclear facilities and
(2) typical difficulties in arranging to administer the survey to security personnel
because of shift schedules. The team noted that shift scheduling issues did not
adversely affect response rates from other functional groups, such as operations, and
verified that all security personnel had an opportunity to participate. During focus
groups, the NRC verified that security personnel who took the survey believed the
items were more relevant to the crafts, consistent with Synergys experience at other
sites. Interviews indicated that security personnel believed the effort of taking the
survey would not be worthwhile because it would not result in positive changes related
to staffing and overtime. The team determined that the failure to include items directly
relevant to the security function or adjust existing items to be more clearly relevant to
the security function was a weakness in the survey tool. The team noted that Synergy
and licensee personnel followed-up on the low response rate with individual interviews
to more clearly understand the security groups safety culture concerns.
Survey analyses. Based on the NRC teams review of the statistical analyses of the
survey data performed by Westat, the team concluded that the survey results were of
limited effectiveness in differentiating between functional groups at the site that may
have localized safety culture issues. Statistically significant differences were found
only between the functional group with the most positive responses on the survey and
the group with the most negative results. Therefore, Synergy relied more heavily on
the write-in comments and interview results to discriminate among functional groups.
Based on their review, Synergy identified 12 priority groups in need of particular
attention. The NRC team determined that the recommendation to focus on these 12
groups may be narrowly focused given the similarities in the safety culture issues
raised in the write-in comments from all of the groups.
The NRC team reviewed the survey data analyses performed by Westat and
determined that the survey met standard survey design requirements for internal
consistency. The write-in comments, the results of Synergys and the licensees
follow-up interviews, the ISCPET review, and the NRCs independent safety culture
assessment indicated that the survey tool provided adequate information related to
safety culture attitudes at Palo Verde.
Third-party assessment conclusions
The team concluded that the results and conclusions of the assessment were
consistent with the data collected. The team also noted that the themes identified from
the assessment were very similar to the results of licensee safety culture assessments
- 121 - Enclosure
performed in 2004 and 2005. Responses to the 2007 survey items were more
negative than responses to the 2005 survey, and write-in comments on the 2007
survey were both more extensive and more negative in tone than the write-in
comments from 2005. The issues raised by site personnel in each of these
assessments were consistent and were discussed by site personnel in progressively
stronger terms. This trend suggests that corrective actions were not effective in
sustaining improvement following the 2004 and 2005 safety culture assessments.
Licensee analysis and corrective actions
The team concluded that individual findings and recommendations from the safety
culture assessment were appropriately reviewed by the licensee to identify corrective
actions. The licensee had not finished developing corrective actions at the time of the
inspection; therefore, the team could not evaluate the completeness and effectiveness
of the planned corrective actions.
The licensee addressed the results of the safety culture assessment using several
methods. These methods included Employee Concerns Program (ECP) actions to
respond to some write-in comments, establishment of a Safety Culture Team (SCT),
development of safety culture improvement plans for the 12 functional groups
identified by Synergy, and efforts to develop site-wide safety culture improvement
plans.
ECP actions. ECP staff reviewed the write-in comments from the survey for any
instances in which a comment implied or reported perceptions of retaliation for raising
concerns. Using information collected from the survey, the ECP identified the work
groups of approximately 9 cases, but made no attempt to identify individuals who had
submitted the comments in order to maintain their anonymity and confidentiality. The
ECP manager provided an overview to the team of how each case was investigated
and dispositioned. The team concluded that the handling of the comments was
appropriate.
SCT actions. The licensee established the SCT to facilitate the development,
communication, and implementation of actions to improve safety culture. The SCT
tasked the managers of the 12 functional groups to develop improvement plans. The
SCT provided the managers their groups survey scores, write-in comments, and other
relevant information from the assessment, and directed the managers to communicate
the survey results and develop improvement plans. The SCT worked with the
managers to plan their communications with their groups, provided individual and
organizational consulting to the managers in developing their improvement plans, and
were responsible for tracking implementation and effectiveness of the plans. Senior
management met with each manager to review the improvement plans. The NRC
team also reviewed the improvement plans, observed meetings between senior
management and the managers, and conducted individual interviews with the
managers to obtain their views of the process. The team concluded that the safety
culture improvement plans for the 12 groups were appropriate.
The SCT also provided safety culture assessment results to other managers at the site
in September 2007, with a request for the managers to meet with staff to discuss the
results, and develop any necessary improvement plans. In addition, the SCT
requested the managers review the results for their work groups and determine
- 122 - Enclosure
whether any immediate improvement actions were necessary before the start of the
Unit 3 steam generator replacement outage. The SCT requested the managers
complete their meetings by the end of October 2007, but did not require that any
improvement plans be entered into the CAP for tracking to completion. At the time of
the inspection, the SCT did not plan to monitor implementation of the managers
dissemination of the assessment results or development of improvement plans.
This approach for non-priority groups was consistent with the licensees process for
responding to the results from the 2005 safety culture assessment. About half of the
frontline participants in the NRCs focus groups had not yet met with managers to
receive detailed information about the assessment results or participate in developing
improvement plans. The team noted that a failure to communicate specific results
from a survey and develop improvement plans may discourage personnel from
participating in future surveys. In addition, because the statistical differences between
functional groups on the survey responses were not significant, this approach may not
ensure improvement in other groups that could have safety culture issues.
Site-wide actions. The SCT informed the team that they intended to address safety
culture weaknesses identified through the assessment with site-wide improvement
actions. The SCT performed streaming analyses on: (1) the areas for improvement
identified by the Synergy survey and follow-up interviews; (2) the summary of the
write-in comments from the survey; and (3) the areas for improvement identified by the
ISCPET. The analyses identified drivers and contributing causes for each of the
areas, which were then consolidated into a set of overall key drivers. These key
drivers were: (1) individual accountability and ownership; (2) clarity and
communication of overall priorities and strategies; (3) quality of leadership and
management; (4) receptivity to employee input; (5) change management, and (6) site
programs and processes. The NRC team determined that the key drivers captured the
issues from the licensees safety culture assessment.
The licensees corrective actions to address the safety culture drivers were primarily
high-level actions referenced from several ImPACT Root Cause Evaluations. For
example, to address individual accountability and ownership, the SCT corrective
actions referenced actions being taken under the Organizational Effectiveness Root
Cause Evaluation, including developing an accountability model (CRAI 3075803),
implementing a management review meeting process (CRAI 3063852), developing a
leadership/management model (CRAI 3082328), and developing a site-wide
communication strategy (CRAI 3063112). The corrective actions from the ImPACT
Root Cause Evaluations were either recorded in the Site Integrated Business Plan
(SIBP) or were in the process of being added at the time of the inspection. The SCT
also described plans to establish mechanisms for tracking, measuring, and assessing
the effectiveness of the corrective actions to address the key drivers. Based on the
level of detail available, the NRC team was unable to assess the effectiveness of the
corrective actions or the SCTs plans.
Verification of completeness. The SCT performed a detailed review of the findings,
recommendations, and write-in comments from the safety culture assessment teams
and compared them with SIBP tasks and existing CAP items. For issues that were not
in the SIBP or CAP, the SCT initiated additional actions. For example, one of the
findings from the ISCPET and Fundamental Overall Problem 9, Organizational
- 123 - Enclosure
Effectiveness, was a need to establish safety conscious work environment (SCWE)
expectations for contractors and incorporate them into their contracts. The SCT
initiated CRAI 3090979 on November 9, 2007, to address this action.
In addition, for actions that were described at a general level in the SIBP or CAP, the
SCT issued or planned to take additional actions to ensure findings and
recommendations from the safety culture assessment were addressed. For example,
the SCT initiated CRAI 3082328 to verify that the communication strategy being
developed under CRAI 3063112 (related CRDR 3048836, Organizational
Effectiveness root cause) included actions to motivate site personnel to understand
and take responsibility for improving current levels of performance. Another example
was CRAI 3082469, which was to verify that the formal process for change
management being developed under CRAI 3064376 (related CRDR 3048836),
required solicitation of employee input in appropriate cases. The SCT identified
several issues from the licensees safety culture assessment that were not addressed
by existing actions, and planned to enter those into the CAP.
The NRC team noted that the actions that were referenced in the CRAIs owned by the
SCT did not have a link back to the safety culture improvement efforts. For example,
CRAI 3082469 to develop the process for change management, which was in the
SIBP, did not have a link back to CRAI 3082469 to ensure the change management
process solicits input from employees as appropriate. With this structure, the SCT had
the responsibility to communicate with the action owner, initiate involvement, and
ensure the products met the specifics stated. The action owner, however, did not have
any responsibility to ensure the product addressed specific findings from the safety
culture assessment. This one-way linkage created the potential for the action owners
to not fully consider the safety culture assessment findings when developing and
implementing corrective actions.
8.2 NRC Independent Safety Culture Assessment
The team identified weaknesses in organizational characteristics and attitudes associated
with 10 of the NRCs 13 safety culture components, as detailed in Section 06.07
of Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program.
The most notable weaknesses were identified in the safety culture components related to
decision-making, organizational change management, resources, the licensees corrective
action program, accountability, operational experience, self assessments, and work
practices. The observed weaknesses were widespread among functional groups across
the organization, involving operations, engineering, maintenance, radiation protection, and
corrective action program personnel. Organizational characteristics and attitudes were
acceptable in the safety culture components of safety policies; the environment for raising
concerns; and preventing, detecting, and mitigating perceptions of retaliation. The team
concluded that although the safety culture has degraded at the site, Palo Verdes existing
safety culture supports continued safe operation.
a. Inspection Scope
Consistent with the inspection requirements in Sections 02.08 and 02.09 of IP 95003,
the team conducted an independent assessment of the licensees safety culture. The
- 124 - Enclosure
purposes of this assessment were to (1) inform the NRCs assessment of the
contributors to degraded performance in the affected Strategic Performance Areas and
(2) validate the licensees third-party safety culture assessment.
The team relied on document reviews, individual and group interviews, and behavioral
observations to conduct the assessment. The team assessed safety culture attitudes
by conducting 125 individual interviews and 34 focus groups with an average of 8
participants in each group, for an approximate total of 400 safety culture-specific
interviews over the course of the inspection. These interviews involved personnel from
the majority of functional groups at the site and at each management level affecting
the organization, including Arizona Public Service (APS) corporate and owner
personnel, former senior site managers, and an Arizona Corporate Commission (ACC)
staff member. The team also assessed safety culture-related behaviors during plant
tours, system walkdowns, control room and outage control center observations, and
observations of site meetings and pre-job briefings. The team assessed the licensees
organizational characteristics with respect to each safety culture component using at
least two data-collection methods. The data-collection methods were implemented by
at least two inspectors. The team also integrated the safety culture insights from the
inspection findings into the overall assessment of the safety culture at Palo Verde.
b. Observations and Findings
b.1 Decision-making
The team identified past decisions that continue to adversely affect site
performance as well as ongoing weaknesses in some site decision-making
processes. Results of the NRCs safety culture assessment indicated that the
majority of Palo Verde personnel interviewed perceived that cost reduction efforts
inadvertently created an environment in which nuclear safety was degraded.
Most of the site personnel interviewed described decision-making as being
primarily governed by the goals of reducing costs in preparation for deregulation
and cost containment, unless the decisions involved meeting new regulatory
requirements or ensuring continued production (e.g., steam generator
replacements). Site personnel provided numerous examples of decisions related
to the erosion of nuclear and industrial safety margins; failures to maintain
adequate levels of qualified staff to implement programs, processes and
procedures; failures to replace or upgrade out-dated or degrading equipment; a
lack of preventative maintenance; and untimely repairs.
Impact of Deregulation. During the early 1990s, the ACC determined that APS
should deregulate its generation assets, including Palo Verde, and separate
these assets to enter into a commercially competitive retail electricity market. In
anticipation of a deregulated retail market, APS implemented cost reductions with
a goal of decreasing retail rates by approximately 30 percent. The cost
reductions were implemented by reducing staffing levels through reductions in
force and an extended hiring freeze, and by cutting operations and maintenance
(O&M) budgets by 10 percent per year across the board. Senior management
believed that this reduction could be completed without degrading nuclear safety
by eliminating the inefficiencies in processes and workflow. By 1998, total
expenditures (O&M + capital) at Palo Verde had been cut by 35 percent from
1992 levels.
- 125 - Enclosure
The consequences of the cost reductions combined with the effects of plant
aging, contributed to an increase in unplanned outage time and equipment
failures. In 2000, after nine consecutive years of across-the-board O&M cost
reductions, O&M expenditures began increasing. By 2006, O&M costs had
increased by 64 percent from their low point in 2000 and were 21 percent higher
than 1992 baseline levels.
Palo Verde replaced steam generators and initiated plans to replace the reactor
vessel pressure heads in all three units. This caused capital expenditures to
increase by a factor of 5 from 1996 to 2005. The increase in combined O&M and
capital expenditures between 1998 and 2005 was 85 percent and was
attributable to both capital expenditures on major improvement projects as well
as increased O&M costs associated with declining performance.
Interviews with site personnel and document reviews indicated that during the
period of 2000 to 2007, cost-containment pressure increased. Licensee
personnel stated high priority modifications were cancelled or deferred, the
backlog of preventive maintenance deferrals increased, aging equipment was not
replaced, tools and equipment needed to perform simple tasks were not repaired
or replaced, training staff was reduced, training materials were not updated,
benchmarking efforts and external training opportunities were curtailed, and
procedures were not updated or maintained. These cumulative reductions
contributed to the increase in equipment failures, plant events, and other
performance problems at the site.
The licensee continued to lose qualified staff in the line organizations
(e.g., operations, engineering, maintenance) during this period as Palo Verdes
workforce began to retire or personnel took other jobs. Further, experienced
people were shifted to support large capital projects, such as the main turbine
and steam generator replacements, or the improvement projects necessitated by
Palo Verdes declining performance. These personnel were not replaced in the
line organizations, which exacerbated the lack of support for operations,
maintenance, engineering work, and improvement projects at the station.
During interviews, corporate personnel stated that they had lost touch with site
operations over the five years preceding Palo Verdes entrance into Column 4 of
the NRCs action matrix, and were unaware that cost-containment efforts were
adversely affecting performance. A complicating factor was that corporate
management allowed multiple lines of communication with the site to be closed
off. Virtually all significant non-financial assessments of site performance flowed
to the corporate organization through a single communication channel at the site.
From the corporate perspective, APS was appropriately investing a steadily
increasing amount of resources to protect the Palo Verde asset. Senior onsite
management believed that the site had to become more efficient and more
productive in order to establish competitive rates and maintain safety. The site
leadership was determined to avoid problems with cyclic performance by
maintaining sustainable budgets while addressing latent problems.
- 126 - Enclosure
In 2005, the ACC reversed the original decision to deregulate. In April 2007, the
ACC approved the first APS base rate increase in 14 years and implemented a
process whereby APS was reimbursed for increased fuel costs.
Licensee response. Management at the most senior corporate levels has taken
steps to enhance decision-making processes affecting nuclear safety at Palo
Verde. For example, to ensure that Board and owner decision-making is more
fully informed, the composition of Palo Verdes off-site safety review committee
has been changed and the committee has an avenue to report directly to the
Board of APS rather than to the site vice president/chief nuclear officer
(SVP/CNO). Additionally, the Nuclear Oversight Committee provides a second
source of information by directly reporting to the Board and APS corporate
executives. At the time of the inspection, Board members were making more
frequent visits to the site to meet with frontline and other personnel, and owner
representatives were regularly observing site decision-making meetings.
During the first quarter of 2007, APS hired a new SVP/CNO who has a clear
focus on nuclear safety and is knowledgeable of current industry practices and
standards. The new SVP/CNO assembled a team of similarly knowledgeable
and experienced managers in key senior management positions to improve site
decision-making and performance. During NRC safety culture interviews, station
personnel cited examples of visible decisions made by the new senior
management team within the past few months that they perceived as initial
indications of an increased emphasis on nuclear safety. These decisions
included the development and scheduling of departmental top 10 lists of
equipment repairs, extending a refueling outage to correct some longstanding
equipment deficiencies, and authorization to hire new staff or contractors.
APS has increased the current O&M budget to address the backlog of issues.
Corporate and site management indicated that the resources needed to sustain
improvement at Palo Verde will be provided.
Continuing challenges. With the exception of operations personnel and some
mid-level managers who have been interacting with members of the new senior
management team, most site personnel interviewed by the NRC reported that
they had yet to see or experience a significant change in the decision-making
patterns that affected their individual work groups.
Consistent with this perception were the NRC teams observations that decision-
making at lower levels in the organization had not yet become fully aligned with
station managements expectations. Although corrective actions have been
formulated and some were beginning to be implemented to enhance station
decision-making, the licensee did not consistently make safety-significant or risk-
significant decisions using a systematic process that ensured safety is
maintained. For example, as previously discussed, the licensees process for
making operability determinations has not ensured that (1) all degraded
equipment conditions that may require an operability determination are identified,
(2) SROs are provided the technical information necessary to make timely
operability determinations, and (3) the technical information that is provided is
sufficiently rigorous to support decisions that ensure safety is maintained.
- 127 - Enclosure
Licensee safety culture assessment. The team determined that the licensees
third-party safety culture assessment had adequately captured these issues.
b.2 Organizational Change Management
Results of the NRCs safety culture assessment indicated that (1) the licensee
was continuing to experience adverse consequences from previous poorly
managed change efforts and (2) organizational change management continues
to be a significant challenge.
A key organizational change that impacted Palo Verdes performance was the
sites reengineering effort in the early 1990s, which focused on streamlining
work processes, reducing staff to reduce O&M costs, and allocating decision-
making authority to those closest to the work (Checklist #FA-4, Reengineering
Checklist). Palo Verde management undertook the reengineering effort to
position the organization for the anticipated deregulation. Reengineering was a
popular and successful management approach undertaken by other companies
during this time period. This effort was based on a best-selling book by Hammer
and Champy entitled Reengineering the Corporation published in 1993.
Fundamental to this approach was the premise that productivity gains will
naturally follow as processes are streamlined and wasteful steps are eliminated.
The productivity gains should translate directly to cost reductions. However,
budget and staff reductions first require a commensurate increase in worker
productivity in order to match the estimated resource supply and demand.
The actions taken to reduce staff and costs from 1992 to 1998 enhanced cost
competitiveness in response to the pending deregulation. However, the
reengineering effort did not sustain the desired productivity and performance
improvements. The goal to achieve sustained cost reductions was not met
because of several factors, including flaws in how the reengineering effort was
implemented, failures to recognize unanticipated consequences, and failures to
make adjustments when unintended consequences occurred.
Productivity methods and tools. The licensee focused on cost reductions without
a commensurate effort to provide the workforce with productivity-enhancing
methods and tools. Interviewees perceived that past senior management did not
want to invest current resources to save future resources. Interviewees believed
that past senior management approached the productivity problem by first cutting
staff and budgets, and then demanding that middle management find new and
creative ways of enhancing productivity. This approach did not include investing
in the processes or technology that might have enabled the desired productivity
improvements.
For example, the CAP was structured around SWMS, a commercial software
database. Palo Verde procured this software application but did not also
purchase the optional interfacing application package that was more intuitive and
would have more readily facilitated linking of CRDRs, CRAIs and other related
CAP documents. As a result, gaining proficiency with the SWMS database
required extensive training and effort to master the software. The consensus
from interviews and focus groups was that many of the workers had not spent the
- 128 - Enclosure
time to become proficient because SWMS was too complex. As a result,
personnel continued using multiple problem identification and corrective action
tracking databases they had developed before SWMS was implemented and that
were tailored to their unique needs. The team noted that there were at least 37
separate problem identification and action tracking databases in use at the site at
the time of the inspection. Fragmenting the action tracking systems into separate
databases that were not linked prevented site management from being able to
monitor problems effectively and trend the status of corrective actions. This
fragmentation masked the true extent of the backlogs and made cross-
department prioritization of corrective actions difficult and time consuming.
Palo Verde financial management processes also did not support productivity
improvements, such as effective planning to fund emergent work. A consistent
theme from interviews with mid-management personnel was that department
budgets were considered to be inviolate (i.e., department budgets could not be
overrun and unbudgeted emergent work generally had to be funded from existing
line items). Specifically, when important equipment failed, middle management
was required to find the funds to repair the equipment from within their own
departmental budgets. These unplanned repairs often required that other key
department projects had to be deferred, reduced in scope, or cancelled in order
to fund the emergent repairs. Important projects in one department would be
delayed due to emergent work while other less important projects in other
departments were executed because they were funded under a different
department or group budget.
This weakness in financial management processes contributed to the increase in
the stations backlog. The lack of integration of budget priorities allowed some
low priority projects to be executed while higher priority projects were cancelled
or deferred. Some managers reportedly resorted to padding their budgets to
fund emergent work while others attempted to accurately estimate each budget
line item. Those who padded their budgets had the funds to support both
planned and emergent work, while those who attempted to comply with the spirit
of the formal budgeting process ran short of funds to complete planned work by
the end of the year.
Streamlining. The effort to streamline processes and procedures at the site was
initially effective, as indicated by the decade during which Palo Verde received
favorable NRC and industry assessments. Interviewees described many
examples of efficiencies that were achieved from reducing the number of
management levels in different functional groups and empowering individuals
and teams at lower levels of the organization to solve problems.
The streamlining effort also resulted in the elimination of clear lines of authority,
roles, and responsibilities for programs and processes, which were replaced by
informal, and typically undocumented or poorly documented, methods of
decision-making. Interviewees described the streamlined processes as relying
on expert power. They believed they were effective because of the knowledge
and skills of the staff, many of whom had joined the organization during
construction and start-up. When technical knowledge was required to make a
decision or solve a problem, personnel knew who on the staff had the necessary
- 129 - Enclosure
expertise and could access it with a phone call. One interviewee described the
resulting methods of accomplishing work as management by friendship.
The sites streamlined processes began to falter as qualified personnel left the
site or were moved into other positions. Experienced personnel who left a work
group took their knowledge with them. Their expertise was not systematically
captured in site documentation or training programs with the result that overall
organizational effectiveness was reduced.
Staff reductions and reassignments. Middle management and frontline
personnel interviewed by the team consistently reported that the loss or
reassignment of qualified staff from the line organizations (e.g., operations,
engineering, and maintenance) contributed to the sites declining safety
performance. Attrition actually reduced staff to approximately 2000 full-time
licensee personnel by 2001. An internal licensee staffing study in 2002
recommended increased hiring of operations and engineering personnel. The
study showed that this action was necessary because of projected workforce
attrition from retirements, job migration, and the length of time required for new
hires to become fully qualified. The study recommended that the effort to hire
and train new personnel should begin no later than 2004 to preclude significant
shortages of qualified staff. The licensee initiated the Legacy Engineer
program to recruit and train recently graduated engineering personnel, but did
not otherwise implement the recommended aggressive hiring strategy.
Reductions in standards and technical rigor. Interviewees indicated that the
reduced availability of qualified personnel in the line organizations, the loss of
organizational formality and expert knowledge, and increased cost-containment
pressure, as both the workload and annual expenditures (both O&M and capital)
began to increase combined to influence site personnel to reduce standards and
the technical rigor of their work. Interviewees reported finding new ways to meet
management expectations to expedite or defer work in order to contain costs.
However, when it was not possible to find ways to complete necessary work
more productively, interviewees reported that they sometimes resorted to cutting
corners, reducing technical rigor, and reducing the total effort spent on jobs.
Consequently, technical standards in some groups began to slip and quality
suffered. Interviewees also indicated that management accepted less technical
rigor or a lack of product quality as a necessary compromise to meet deadlines
or keep equipment operating. According to site personnel, the sites
streamlined processes were inadequate barriers to prevent such compromises
and over time, the organizations standards degraded as compromises became
more common.
Licensee response. The team concluded that the new senior management
understands the extent of the changes required to reverse the adverse effects of
the past reengineering and cost-containment efforts and has appropriately
prioritized the necessary changes. The team noted that the licensee was
revising the sites financial planning processes; planned to enhance the SWMS
interface; had published and disseminated standards to clarify expectations for
technical rigor and quality work to the line organizations; had begun to implement
- 130 - Enclosure
a program for funding and expediting minor modifications and repairs at the time
of the inspection; and was taking steps to recruit new staff and enhance training
programs to qualify the new hires.
Continuing challenges. The team observed that managements efforts to engage
the workforce in implementing the needed changes were not yet fully effective.
Frontline personnel interviewed by the team were not aware of many of the
changes that management was planning or had made, which, over time, would
resolve some of the staffs more significant concerns, particularly with respect to
hiring and training new personnel.
In addition, the large majority of interviewees stated that they were willing to
make changes to improve performance, but, other than being encouraged to
write PVARs, were seeking direction and information about how they, as
individuals, could play a part in turning the site around. After their early
successes with empowerment under the reengineering initiative, this mature
workforce perceived themselves as an untapped resource for improving
performance in their work groups that management has ignored over the past
five years. Only the interviewees from the operations department were clear
about the new managements expectations for their role as the sites leaders.
In other cases, interviewees were experiencing changes but did not fully
understand or accept the bases for the changes. For example, some specialty
maintenance personnel interviewed were recently reassigned to begin cross-
training in other disciplines. These staff recalled a similar effort in the early
1990s that was undertaken as part of the reengineering initiative, then later
cancelled because it caused the specialty staffs primary skills to degrade, and
reduced rather than enhanced staff competence overall. It was unclear to these
interviewees why management was again pursuing a cross-training effort.
The team observed that the licensee had identified the communication
challenges associated with change management at the site, including: the need
to enhance two-way communication between the frontline and management to
ensure that changes are implemented as intended, do not have unintended
consequences, and minimize resistance to change. The team noted that the
licensee was initiating the development of departmental communication plans to
include effectiveness measures during the inspection.
Licensee safety culture assessment. The team determined that the licensees
third-party safety culture assessment had adequately captured these issues.
b.3 Resources
The results of the NRCs independent safety culture assessment indicated that
past resource allocation decisions have challenged nuclear safety at Palo Verde.
Cost-containment efforts caused or contributed to a reduction in the availability of
qualified personnel, procedures that have not been upgraded or maintained, and
degraded facilities and equipment.
Staffing, qualifications, and work hours. The licensee reduced staffing at the site
through reductions in force and attrition over the past 15 years. The team
- 131 - Enclosure
concluded that the availability of qualified staff in key departments was reduced
to levels that impacted the licensees ability to simultaneously: (1) respond to the
high amounts of emergent work and unplanned outages, (2) plan for and execute
2 refueling outages each year, (3) reduce growing backlogs, (4) train and qualify
new hires, and (5) complete implementation of multiple programs and processes
to improve site performance. The team noted that improving the staffing issues
and performance issues are challenged by: (1) the relatively long periods
required to fully qualify new staff in key disciplines (ranging from 2 to 6 years);
(2) challenges in recruiting personnel; (3) limited training resources; and (4) the
increasing rate of attrition from retirements.
Operations
Introduction: The team identified an unresolved item (URI) associated with
Technical Specification 5.2.2.d. for the routine use of heavy amounts of overtime
for operations personnel.
Description: Interviews with frontline personnel and managers in operations
indicated that shortages of licensed operators and operator training personnel
were perceived to be the most significant issue facing the operations
organization. Interviewees reported that the licensed operator training pipeline
was interrupted several times after 2000 with a resulting net loss of 20 licensed
operators by 2007 (see chart below). This loss occurred concurrently with a
reduction from 6 operator shifts to 5 self-relieving shifts (i.e., shift crews that
have sufficient numbers of personnel to ensure that regulatory and administrative
control room staffing requirements can be met without overtime or assigning a
member of another shift crew to cover for an individuals absence). The
continued loss of operators reduced shift staffing to a point where 13 of 15 shifts
were not self-relieving. This meant that most control room shifts did not have a
sufficient number of operators to make up for a temporary absence or permanent
loss of either a reactor operator (RO) or SRO. The reductions had the effect of
requiring personnel to work additional overtime and limited most licensed
operators activities to standing watch in the control room. Interviewees indicated
that career advancement opportunities for licensed operators were limited
because of pressures to maintain shift crews; thereby, limiting the ability of
licensed operators to integrate an operations perspective into other site activities.
The team reviewed operations payroll data that summarized the cumulative
regular and overtime hours for each operations department position and
calculated the annual overtime rate for select positions. Since 2003, overtime, as
a percent of regular hours worked, has increased steadily and substantively for
control room and auxiliary operators. The team noted that the increase in
overtime rates for operations department positions appeared to be largely the
result of a decrease in staffing, rather than the result of an increase in the total
number of person-hours expended.
Specifically, from 2003 through 2006, the total number of hours worked annually
by personnel in the control room supervisor (CRS), SRO, RO, and auxiliary
operator (AO) positions remained relatively constant, or decreased, while the
percentage of those total hours that were worked as overtime increased. As a
- 132 - Enclosure
result, the payroll data indicated that the licensee increasingly relied on the use
of overtime to provide the person-hours necessary to operate the three units.
Technical Specification 5.2.2.d requires administrative procedures to be
developed and implemented to limit the working hours of unit staff that perform
safety-related functions (e.g., licensed SROs, licensed ROs, radiation protection
technicians, auxiliary operators and key maintenance personnel). The Technical
Specifications further requires that the controls shall include guidelines on
working hours that ensure adequate shift coverage shall be maintained without
routine heavy use of overtime. Pending the completion of a review of the actual
work hours by operations personnel, this issue is identified as URI 05000528,
05000529,05000530/2007012-19, Routine Heavy Use of Overtime.
Maintenance. Interviews with maintenance personnel did not indicate that
overtime was a particular concern. Staffing and qualifications were consistent
areas of concern among those interviewed. Some individuals described the
staffing issue as huge, adding that with low staffing the attitude has become, I
will do it however I can. Many of the comments were focused on the increasing
loss of experienced and qualified personnel. They indicated that although an
apprentice or other new hire represents a pair of hands, so that it may appear
that staffing levels are adequate, their knowledge and skills do not replace those
of a senior technician who has retired. They also stated that training and
supervising new hires, many of whom have not worked in an industrial
environment before, also increased their workload.
The team reviewed maintenance department staffing levels since 2003 and
found that the total number of maintenance staff has remained relatively stable
during this period. However, consistent with the interviewees perceptions of the
loss of senior staff, the team also noted that 125 maintenance personnel (about
23 percent of the departments staff) have retired or left the site since 2000, 48 of
whom left in the 18-month period preceding the inspection. Overtime levels also
increased markedly from their levels during the 2003 through 2004 time period as
workload from emergent work has increased.
Maintenance Department Overtime
Annual Averages for Years 2003 through 2007
Year 2003 2004 2005 2006 2007
Total
546 546 542 545 524
Staff*
Overtime 10.4% 10.1% 15.8% 18.5% 17.9%**
- Estimate based on total department staff during September of year shown.
- Estimate based on monthly overtime rates for January through September 2007.
The teams review of an Apparent Cause Evaluation (ACE) Report, Analysis of
Maintenance Organization Performance 2003 Present, Event Date:
March 1, 2007, (CRDR 3039642), indicated that the increase in maintenance
organization overtime was related to an increase in the maintenance organization
human performance error rate. The report states, The current materiel issues of
the plant require more and more frequent overtime, which has shifted the
performance of the maintenance organization in a negative direction. The
organization generally performs at an error occurrence rate of 4/10000 hours or
- 133 - Enclosure
less when overtime worked is 5000 hours0.0579 days <br />1.389 hours <br />0.00827 weeks <br />0.0019 months <br /> or less. When overtime worked
exceeds 5000 hours0.0579 days <br />1.389 hours <br />0.00827 weeks <br />0.0019 months <br /> the error-occurrence rate changes to 5.5/10000 hours or
worse. Second, after overtime begins to escalate and longer periods of overtime
are experienced a cumulative effect on error-occurrences becomes apparent.
These two observations may be indications of overload and fatigue.
In addition to describing an association between overtime and maintenance
human performance, the report provided some additional validation of the
concerns expressed by maintenance personnel regarding the experience level of
the staff. Specifically, the report described an analysis of human performance,
overtime, and worker experience levels in the electrical maintenance shop and
states, The Electrical Maintenance shop is not the only work group showing
evidence of this condition, but the indications are more pronounced and easier to
illustrate What is evident is that the increased error occurrence rate caused by
overtime demand is exacerbated by the decreasing level of station experience
within the organization.
Engineering. Interviews with personnel in the engineering organization indicated
that overtime was not generally perceived as excessive or a particular area of
concern. Staffing and qualifications were significant concerns for the engineering
personnel interviewed, and were described by some as the biggest issue facing
the engineering organization. Although many interviewees acknowledged that
Palo Verde had made significant efforts to hire additional engineering staff, they
were concerned that given the extended time period required to train engineers,
the effort to hire and train new personnel (i.e., the Legacy Program) was not
started soon enough to effectively support transfer of the expert knowledge held
by the many senior engineers who will soon be eligible for retirement.
The team reviewed a summary of engineering organization payroll data from
January 2003 through September 2007. The review indicated that staffing
numbers had remained stable from 2003 through 2005 and then began
increasing substantively beginning in June 2006. However, consistent with the
interviewees perceptions of the loss of senior staff, the team also noted that 102
engineering personnel (or about one-third of the departments staff) have retired
or left the site since 2000, 46 (or about half) of whom left in the 18-month period
preceding the inspection. Recorded overtime rates during this period peaked in
2006 at 8.4 percent, although the team noted that the majority of engineering
personnel are classified as exempt and do not record overtime hours.
Engineering Department Staff and Overtime
for Years 2003 through 2007
Year 2003 2004 2005 2006 2007
Total
331 337 335 366 410
Staff*
Overtime 4.5% 4.0% 6.3% 8.4% 5.9%**
- Estimate based on total department staff during September of year shown.
- Estimate based upon monthly overtime rates for January through September 2007.
Other groups and interactive effects. Interviewees from other functional groups
at frontline and mid-management levels also consistently reported inadequate
levels of qualified staff to support the current workload, including the procedures
- 134 - Enclosure
and standards group, work management, radiation protection, chemistry,
business operations, performance improvement, quality assurance, and the
training and human resources groups.
Because little hiring outside of APS occurred between 1993 and 2004, the
human resources workload associated with recruiting and hiring was negligible
and human resources staff did not develop recruiting skills. Interviewees stated
that any active recruiting for open positions was carried out by line managers and
supervisors, typically by friendship when possible. Interviewees reported that
when friendship was insufficient, positions would sometimes remain open for
years. If an individual was identified to be hired, competing demands on human
resources staff often delayed completing the hiring process. The result for the
line organizations was that the workload associated with the unfilled positions
became the responsibility of the remaining staff for extended periods of time, or
was simply not addressed.
The licensee also permitted the number of qualified training personnel to decline.
When an individual left a training position, the position either was eliminated or
was difficult to fill because the line organizations could not afford to move
personnel into the training positions. As a result, when new staff or contractors
were hired and needed training to become fully qualified for their positions, the
training resources were not available to qualify them in a timely manner.
Interviewees reported numerous examples of staff in chemistry, radiation
protection, security, maintenance, and engineering that could not perform all of
the tasks required for their positions without supervision, over extended periods
of time, because there were insufficient training personnel to provide the required
training.
The procedures and standards group was created in late October 2006, to
centralize responsibility for maintenance and operations procedures, in response
to procedure-related site performance problems. The original staffing plan for the
group had eight vacancies, three of which were to be filled by hiring people
external to APS. In addition, the group hired nine contractors for a project to
enhance maintenance procedures. Because of difficulties in filling the open
positions and a growing backlog of procedure change requests, the maintenance
procedure improvement project was deferred and the contractors were assigned
to address the backlog. This action met the groups need for procedure writers
who were knowledgeable of maintenance practices. However, because of the
staffing limitations in the operations department discussed above, the group was
unable to recruit Palo Verde operations personnel to fill the in-house positions
and was seeking to hire experienced operators from other sites.
Licensee response to staffing and qualifications issues. The team noted that the
new senior managers have implemented an aggressive plan to recruit, hire, and
train new staff to overcome the current shortages and prepare for staff
retirements. In November 2007, the licensee had 226 open positions and was
actively seeking staff from outside of APS with the requisite skills and knowledge
of current industry standards and practices. Personnel to fill 50 of those open
positions had been identified and were expected to begin work at the site in
December 2007. In addition, the licensee had approximately doubled the
number of Legacy Program engineers, maintenance apprentices, and junior
- 135 - Enclosure
staff in other disciplines. Positions for new instructors have been authorized.
The licensee is also augmenting many staff capabilities with additional skilled
contractor personnel.
Since arriving at Palo Verde, senior managements highest priority has been to
recruit and train large numbers of operator candidates, including candidates for
non-licensed operator positions and instant SROs. The human resources
department recently hired an experienced nuclear recruiter to assist in the hiring
of personnel. In addition, the licensee hired four new operations training
instructors and was considering alternative approaches to increase training
instructors. During the inspection, senior management elected to advance the
schedule for a class for non-licensed operator candidates by five months. The
licensee also increased authorized staffing levels for the operations department
to 333 positions.
To maintain a more stable level of staffing within the security department, the
licensee was increasing the frequency of the security training academy to twice
per year and posting a continuously open vacancy announcement to establish a
training pipeline for security officers. The licensee was also considering
alternative methods to improve the retention of security personnel.
The licensee was taking steps to reduce barriers to recruiting, hiring, and
retaining staff. For example, APS had previously implemented a policy to
achieve compensation parity between engineers at Palo Verde and in the non-
nuclear business units of APS. This change caused several Palo Verde
engineers to take other, non-nuclear positions within APS to reduce stress or
shorten their commutes. Senior management worked with corporate decision-
makers to revise the policy and reduce the attrition of skilled engineers from the
site. The licensee has also authorized hiring and retention bonuses for targeted
skill sets and is offering reimbursement for relocation costs to some new hires.
Procedures and documentation. Interviewees uniformly indicated that station
procedures, work instructions, drawings, and other documentation necessary to
perform work were: (1) difficult to follow, (2) unnecessarily complicated, and
(3) sometimes inaccurate, incomplete, or inconsistent with regulatory and other
applicable requirements. Many procedures have become outdated over time.
Although these documentation deficiencies have been identified by the NRC and
the licensee as important contributing causes for Palo Verdes performance
decline, the team noted that licensee actions to correct this problem had been
ineffective in sustaining performance improvement.
The team observed that the licensees processes for managing procedures and
other critical documentation continued to be fragmented among various
organizations across the site. At the time of the inspection, the licensee had
identified the need for, but had not yet developed a comprehensive, integrated
approach to address the full scope of site-wide documentation deficiencies
(CRDR 3079100 - Programmatic Weaknesses in PV Programs, procedures, and
processes - ImPACT FOP 11 and safety culture, Apparent Cause Evaluation
Report, October 2007).
- 136 - Enclosure
The licensee had not determined whether to initiate a wholesale upgrade to its
existing maintenance and operating procedures to bring them up to current
industry standards or continue to address individual procedural deficiencies. As
previously discussed, the procedures and standards group initiated a project to
enhance maintenance procedures by ensuring the procedures incorporated
human factors good practices. However, the project was stopped and the
resources diverted when the backlog of procedure change requests began
increasing in 2007 as a result of management efforts to reinforce procedure use
and adherence expectations. Interviewees indicated that preliminary results of
the enhancement project were less than satisfactory to the procedure users, who
had been hoping for complete procedure rewrites. The team noted that the
availability of qualified staff in the maintenance and operations organizations may
not have supported the technical reviews and procedure validation activities that
a wholesale upgrade project would require.
Interviews also indicated that licensee personnel were aware of the implications
of the changing workforce at the site (i.e., increasing numbers of less
experienced staff) on the level of detail and usability of the sites documentation,
but have not developed a plan to address the issue. The deficiencies in current
procedures and work instructions were described as particularly problematic by
the less experienced personnel interviewed. These interviewees commented
that procedures and other documentation were not helpful as training tools, were
not written in plain language that could be understood without step-by-step
translation from a senior staff person, and that the level of detail in the
procedures was frequently inadequate for them to understand how to perform the
task. Because procedures and documentation were of limited usefulness to the
less experienced interviewees, these individuals were particularly concerned
about the loss of expert knowledge and guidance they rely on when senior
members of their work groups retire.
Facilities and equipment. Examples of longstanding degraded equipment
conditions identified by the team include, in part, Borg Warner check valves, post
accident monitoring chart recorders, radioactive waste systems, Target Rock
solenoid valves, and cable vault flooding. In addition, interviewees provided
numerous examples of degraded or inadequate facilities and equipment that they
described as challenging their ability to perform work effectively. Examples
included work spaces that were not air conditioned, being denied heat protection
when working outside during the summer, bird droppings in work spaces, frayed
and decaying safety harnesses, outdated and unreliable software, instruments
and test equipment that cannot be repaired because parts are no longer
available, security personnel being required to use personal vehicles to patrol
because there were an inadequate number of site vehicles, temporary power
and ventilation systems in workspaces that have been in-place for years, training
spaces too small to accommodate class sizes, inadequate access to desks,
computers and telephones, and inadequacies in the availability of simple items,
such as chairs, stools, shop cabinets, hand tools, or lockers for storing personal
belongings. Interviewees reported that they had raised these needs to their
supervisors, documented them in the CAP, but had been unsuccessful in
resolving the issues over long periods of time. The team concluded that the
staffs longstanding inability to resolve such issues contributed to the apparent
tolerance for degraded conditions the team has observed. The team also noted
- 137 - Enclosure
that new management was taking steps to address some of these concerns with
mechanisms such as the departmental Top 10 lists and the safety culture
improvement plans for some work groups.
Continuing challenges. Corporate and senior site management personnel have
repeatedly affirmed that the resources are available to address these issues.
The team noted that the licensees ability to make a rapid improvement in overall
site performance may be hampered by limitations in the availability of qualified
staff and that previous performance improvement efforts were partly ineffective
for similar reasons. Although senior management is taking aggressive steps to
augment staff capabilities, the productivity of inexperienced personnel will likely
be challenged until the improvement is made in programs, processes, and
procedures.
Licensee safety culture assessment. The team determined that the licensees
third-party safety culture assessment adequately captured these issues.
b.4 Continuous Learning Environment
The team determined that Palo Verde has not established a continuous learning
environment. Results of the licensees self-assessments, the licensees third-
party safety culture assessments, and the results of the NRCs safety culture
assessment concurred that the site had become insular over the past 15 years.
As a result of cost-containment efforts, the licensee curtailed benchmarking and
external training opportunities, the few new personnel who were hired between
1994 and 2003 were drawn from inside of APS, and internal training resources
were cut. Palo Verde personnel had little exposure to new practices and rising
standards in the nuclear industry.
Palo Verdes success in the 1990s created an attitude of arrogance, according
to many interviews. Interviewees reported this as another reason they stopped
sending people to other utilities on benchmarking trips or for training
opportunities. They saw themselves as a world-class nuclear plant that did not
need to learn from others. Interviewees indicated that this attitude had hampered
previous improvement efforts and led staff to dismiss information about current
industry practices and standards from new hires and contractors with broader
knowledge.
At the time of the inspection, the team did not identify any evidence that
personnel were resistant to new ideas or feedback on means to improve
individual and site performance. Interviewees were aware of planned
benchmarking activities and perceived that benchmarking was necessary to fully
understand and be able to implement new expectations and standards. As one
operator stated, I dont know what an operations-led organization looks like.
However, because of high workload levels, some interviewees predicted that
many of the planned benchmarking activities would be cancelled or curtailed.
Based on past experiences, some believed that lessons learned from
benchmarking activities would not result in improvements at the site because
they would be judged by management to be unnecessary enhancements that
would just add to the work groups workload, when workload was already
excessive.
- 138 - Enclosure
Many interviewees also expressed the desire for more technical training. This
was particularly true of the engineering groups. Focus group participants and
individual interviewees were generally dissatisfied with the technical training they
received because it had become solely focused on maintaining qualifications
rather than enhancing knowledge and skills. Interviewees attributed the
perceived training deficiencies to staffing shortages in the training function and
restricted resources allocated to training. Some newer employees reported that
they had elected to supplement the training they received from the organization
by using personal funds to travel to conferences, attend seminars, or take
classes because management would not pay for these activities.
Frontline and supervisory personnel and most middle managers interviewed
believed that knowledge transfer was one of the more important challenges
facing the site. Frontline and supervisory staff perceived that: (1) site
procedures are particularly difficult for new hires to understand and follow and
they were not aware of any plans to revise the procedures to make them more
usable by new employees; (2) there have been limitations in the quality of
training materials and the training provided to new employees that did not
adequately prepare them for work in the field; (3) hiring plans within their work
groups did not appear to take into account the length of time required for new
employees to become fully qualified and effective in their jobs; and (4) the hiring
plans did not take into account the additional workload that mentoring new staff
imposes on the senior staff. The interviewees indicated that the consequences
they experienced from the perceived inadequacies in ensuring knowledge
transfer have included an increase in human errors in job performance and on-
the-job injuries from inexperienced employees who are unfamiliar with an
industrial environment, as well as increased difficulty in managing current
workloads. The interviewees perceived that these problems have further
contributed to the sites backlogs.
Licensee response. In addition to accelerating the hiring of new staff and training
personnel, the licensee was beginning to address the knowledge transfer
challenges. The human resources department had developed a tool to aid
managers in planning for the upcoming retirements in their work groups. Human
resources had also developed and recently pilot-tested a knowledge
management assessment tool to aid managers in understanding the scope of
knowledge those personnel who were retiring would take with them. The tool
could be used to identify new-employee training needs. The licensee has also
retrained line managers in the systematic approach to training to improve their
ability to ensure that training programs are effective. Senior management has
also established the expectation with middle management that they, rather than
the training department, own and are therefore responsible for the quality of
training provided to their work groups.
Continuing challenges. The overhead costs associated with transitioning to an
effective continuous learning organization are formidable. Adding and training a
large number of new personnel, while at the same time increasing the work
output from the existing workforce, will require personnel to do more than just
work harder. Substantial productivity increases will be necessary to sustain this
environment in the long-term. Site productivity will also be challenged by the
expected loss of experienced personnel.
- 139 - Enclosure
Licensee safety culture assessment. The team determined that the licensees
third-party safety culture assessment adequately captured these issues, but did
not fully explore their implications.
b.5 Accountability
The team observed that a positive consequence of the sites reengineering effort
was to create a strong sense of empowerment, individual responsibility for site
performance, and pride in the site within the workforce. This sense of ownership
was evident in: (1) the number of individuals who provided detailed write-in
comments on the licensees safety culture surveys in 2005 and 2007 (over half of
the respondents on the latter); (2) the personnel who called the NRCs
confidential hotline established for the inspection to request an interview simply
to ensure that the team had their insights regarding the reasons for the
performance decline at Palo Verde and what is needed to improve; (3) the many
statements by focus group participants that they had been raising concerns
about degrading site performance and offering improvement suggestions to
management as early as 2001/2002, as documented in CRDRs, white papers, or
PVARs provided to the team; and (4) the demonstrated willingness of personnel
during the inspection to challenge ARRC decisions and submit repeat PVARs to
attempt to ensure that their concerns were fully understood and classified
appropriately. However, as previously described, a similar number of focus
group participants expressed frustration that they were not fully aware of site
performance improvement plans or how they could make an individual
contribution.
When the team raised the issue of accountability in focus groups, personnel
expressed a strong willingness to be held accountable for individual and site
performance but were frustrated by what they perceived as the failure of past
senior management and some of their middle-managers to be accountable to
them. The context for these comments was generally in relation to having the
resources to fix equipment and procedures, obtain training, replace personnel
who had left their work groups, and the ability to perform work to their standards
without excessive schedule or cost-containment pressures or interference with
their views of the right way to perform a task. Several individuals reported that
they had used the recently disseminated standards and expectations and
industry safety culture principles booklets to challenge management decisions or
actions they perceived as being inconsistent with the goals expressed in the
documents.
Interviewees also discussed the difficulties of holding co-workers accountable in
the face of the many long-standing personal and professional relationships they
have developed at the site and in the community (20 years or more among the
majority of the workforce). Interviewees discussed the barriers to challenging the
work products and behavior of long-term colleagues who have become close
friends when those work products or behaviors were professionally
unacceptable. Some personnel self-reported the choice to accept inadequate
work products and behavior to avoid conflict in these close relationships.
Conversely, interviewees also noted the long-standing adverse effects of past
interpersonal conflicts that had not been resolved. In these instances,
interviewees described conscious efforts to avoid interacting with the individuals
- 140 - Enclosure
with whom they had previous conflicts. The team noted that these conflict-
avoidant behaviors contributed to the observed siloing (i.e., lack of cooperation)
between some functional groups, as well as the failure of staff to hold one
another accountable for meeting their own and the new managements
standards. However, during the inspection, several interviewees reported that
they were changing their conflict-avoidant behavior to support the need for
performance improvement. These individuals described incidents in which they
had personally rejected work products from other organizations that did not meet
their standards and worked with the other organization to provide an acceptable
product.
The team determined that the behavior of site personnel did not consistently
reflect the strong, positive attitudes they expressed regarding their willingness to
hold themselves accountable as well as to be held accountable by management.
The examples of human performance deficiencies described earlier in this report
indicated that personnel had not yet internalized senior managements new
standards and expectations in individual behavior.
Licensee safety culture assessment. The team determined that the licensees
safety culture assessment adequately captured these issues.
b.6 Corrective Action Program
The team identified several concerns in the corrective action safety culture
component associated with problem identification, evaluation, and effective
corrective actions. This safety culture component was assessed primarily
through direct inspection activities.
Specific problem identification concerns during this inspection involved
implementation of emergency action levels, the emergency exercise critique
process, and solenoid valve performance in the auxiliary feedwater system. As
previously discussed in this report, the team identified an apparent reluctance or
inability among some personnel to identify issues as conditions adverse to quality
without prompting. The team determined that this reluctance or inability was a
safety culture weakness.
Specific problem evaluation concerns during this inspection involved condensate
storage tank temperatures, scaffolding procedures, post-accident monitoring
instruments, emergency diesel generator oil leaks, emergency action levels,
operability determinations, and the conduct of the corrective action review board
and ARRC. The team noted that the licensees problem evaluations lacked
depth and rigor and were generally inconsistent with current industry standards
and practices. The team determined that the observed lack of depth and rigor
was a safety culture weakness.
Specific corrective action concerns during this inspection involved high lead
levels in a low pressure safety injection pump bearing, 4160 and 480V motor
terminations, establishment of maintenance rule criteria, and multiple databases
to track deficient conditions. The team noted that corrective actions for these
issues had not been completed or had not been effective, which the team
determined represented a safety culture weakness.
- 141 - Enclosure
Multiple substantive crosscutting aspects associated with problem identification,
evaluation and resolution have existed since 2004. Corrective actions have
continued to be ineffective in improving performance as noted by effectiveness
reviews, external industry reviews, and NRC inspections.
The team determined that the licensees CAP, while complicated and
cumbersome, contained the basic elements of an effective program. Licensee
personnel often recognized appropriate problem identification, evaluation and
resolution fundamentals and behaviors when interviewed; however, this
knowledge and understanding of expectations was not consistently demonstrated
in meetings or in the field over the course of the inspection.
Licensee response: The licensees plan to improve the corrective action program
was incomplete at the time of the inspection. However, the draft plan available
for review addressed the majority of the teams concerns.
Licensee safety culture assessment. The team determined that the licensees
safety culture assessment adequately captured these issues.
b.7 Work Practices
The team identified several concerns in the work practices area. This safety
culture component was primarily assessed through direct inspection activities.
Work practice human performance concerns observed during this inspection
included: (1) Poor human error prevention techniques involving transient
combustibles in the containment building and temporary shielding installation;
(2) poor procedure compliance findings involving transient combustibles in the
auxiliary building and radiological surveys; (3) inadequate management oversight
for findings involving compliance with Technical Specification Surveillance
Requirement 3.0.3, and rigging of the Unit 3 air lock door; and (4) operations
personnel conduct of operations weaknesses, including turnovers, three-way
communications, alarm response, crew briefs, control room logs, and oversight of
switchyard activities.
Work practice concerns have also been a longstanding issue and performance
improvement actions have not sustained improvement as noted by effectiveness
reviews, external industry reviews, and NRC inspections. In particular, the
licensees effectiveness review for human performance concluded that corrective
actions were not well defined and there were no actions for implementation,
monitoring, reinforcement, adjustment, or for managing the transfer of
responsibility for human performance program changes. Furthermore, the
corrective actions for past human performance problems were not fully
implemented.
Interviews indicated that some personnel had begun implementing new work
practice standards and expectations. For example, several interviewees
described recent incidents during which they had stopped work in the face of
uncertainty (e.g., an incorrect procedure or work order instructions that did not
apply to the specific job) or what they perceived to be unsafe job conditions.
However, the team noted that these and other desirable work practices were not
yet consistently implemented by site personnel.
- 142 - Enclosure
b.8 Work Control
The team identified several concerns in the work control area. This safety culture
component was primarily assessed through direct inspection activities. Work
control human performance concerns observed during this inspection included
weaknesses in communications between fire protection, operations, engineering,
and maintenance, which contributed to findings associated with transient
combustible material controls, switchyard maintenance activities, establishment
of compensatory measures for incorrectly installed sprinklers, establishing
performance criteria for plant systems, and installing emergency lighting in
containment.
Work control concerns have been a longstanding issue and performance
improvement actions have not sustained improvement as noted by effectiveness
reviews, external industry reviews, and NRC inspections. In particular, the
licensees effectiveness review for human performance concluded that corrective
actions were not well defined and there were no actions for implementation,
monitoring, reinforcement, adjustment, or transfer of human performance
ownership change. Furthermore, the corrective actions were either not fully
implemented or not implemented as intended.
b.9 Operating Experience
The team identified several concerns in the OE area. This safety culture
component was assessed through direct inspection activities. OE opportunities
were frequently missed, ignored or misapplied. A lack of technical rigor was
frequently cited in component design basis reviews and self assessments with
respect to the application of OE. The station did not appear to have a sense of
the importance and benefits of a strong OE program. The failure to incorporate
OE into daily activities is an open issue from the Yellow finding. In addition, the
failure to effectively use OE contributed to several performance deficiencies
identified by the team. Specific examples of ineffective use of OE during the
inspection involved AF TT&V, Target Rock reed switches, Borg Warner check
valves, and switchyard maintenance activities.
b.10 Self and Independent Assessments
The team identified several concerns with self assessments. This safety culture
component was assessed through direct inspection activities. Self-assessments
conducted by Palo Verde personnel often lacked depth and did not effectively
specify or implement corrective actions. As a result, the self-assessment
program seldom resulted in improved organizational performance. Self-
assessment corrective actions were not always tracked nor were corrective
action documents always written to track the expected actions. The team noted
that self assessments conducted by a mix of Palo Verde and industry personnel
led to more meaningful results.
Specific examples of poor self assessment implementation involved vague
recommendations in the November 2006 operational decision-making self-
assessment; the March 2007 work management self-assessment concluded only
that the assessment needed to be re-performed later in 2007; the self-
- 143 - Enclosure
assessment of the maintenance rule program did not recognize that unavailability
and reliability performance criteria could not be validated, that numerous systems
had non-conservative performance criteria, and that switchyard risk reviews were
not consistently performed; and deficiencies from the assessment of the safety
injection system and the assessment of the environmental qualification program
were not entered into the CAP.
b.11 Environment for Raising Concerns
The team determined that the environment for raising concerns was healthy.
None of the licensee employees interviewed by the team indicated they were
hesitant to raise nuclear safety issues and about 25 percent of those interviewed
gave examples of occasions where they had willingly raised an issue multiple
times. These included occasions when the individuals believed that the CAP had
failed to prioritize an issue appropriately or had not timely or effectively resolved
an issue. The large majority of interviewees perceived that their managers were
receptive to concerns and willing to address them, although they also reported
frustration with the organizations ineffectiveness at resolving longstanding issues
such as obtaining replacements for out-dated equipment, completing repairs on
equipment within an acceptable timeframe, and delays in hiring and qualifying
personnel in time to replace those who had left their work groups or the site.
The team identified very few examples of recent incidents or perceptions of
retaliation for raising safety concerns. Some interviewees described isolated
examples of past incidents that created a perception of retaliation but the
licensee had effectively mitigated those perceptions.
Almost all of the interviewees stated that if they were not satisfied with the
response from their immediate supervisor, they would feel free to escalate the
concern. The interviewees uniformly described positive experiences when
bringing issues to their supervisors and could name several other avenues for
raising concerns. The majority of interviewees explained that approaching their
supervisors and using the CAP to raise concerns had been generally effective to
communicate the concerns (although less effective in resolving them), and
therefore, they have not had the need to use other alternative avenues.
The team noted some differences in the willingness of contractors to raise
concerns compared to licensee employees. About 5 percent of the contractors in
the focus groups stated that they had not been trained in how to write a PVAR or
expressed reluctance to doing so for fear of being viewed as a troublemaker.
Consistent with these perceptions, the Employee Concerns Program (ECP) had
received several concerns involving contractor personnel in the month before the
team arrived on site. In response to those concerns, the licensee reinforced
expectations for maintaining a safety conscious work environment (SCWE) in all
contract organizations. The ECP sent a letter describing the appropriate SCWE
duties and obligations to each contract organization, which became a part of the
contracts terms and conditions. In addition, senior management took steps to
integrate contractor supervisors and managers into alignment and other
meetings to better communicate SCWE expectations.
- 144 - Enclosure
b.12 Preventing, Detecting, and Mitigating Perceptions of Retaliation
Palo Verde had an Integrated Issues Resolution Process (IIRP) comprising the
ECP, the Differing Professional Opinions (DPO) Program, the Management
Issues Tracking Resolution (MITR) program, and the PVAR. A fifth, recently
implemented corporate-level program, called EthicsPoint, was available for
raising ethical concerns or conflicts with the corporate code of conduct, although
no-one at the site had used EthicsPoint since it was implemented in early 2007.
The combined IIRP included these five alternative avenues for raising concerns
at Palo Verde.
Employee Concerns Program. Most individuals interviewed by the team were
aware of the ECP. Interviews indicated that a few groups, primarily contractors,
had not heard of the ECP or received any information or training about the
program. Many interviewees did not have personal experience with the ECP
because they had not needed to use the program. The majority of those
interviewed said that they would raise an issue through their chain-of-command
first, and if that didnt work they would take their concern to the companys DPO
program instead of using the ECP. The inspection team identified a
misconception about the purpose of the ECP among many of the staff
interviewed. The most common view was that the ECP is to be used for human
resource (HR) issues, which the licensee normally processes through the MITR
program, rather than for nuclear safety concerns. When the inspection team
discussed this issue with the ECP manager, she indicated that she was aware of
the issue and believed this misconception may exist because she formerly was
the HR manager. Some interviewees thought that the ECP was not objective
because it was linked to senior management. Also, several interviewees told the
team that they did not trust the ECP, but were unable to give examples to
support the distrust. Personnel interviewed who had used the ECP in the past
indicated that the experience was positive, and that they would not hesitate to
use the ECP again if needed. No interviewees were aware of any breaches of
confidentiality.
The team reviewed 36 ECP files from 2007 related to SCWE issues. The team
determined that the concerns had been reviewed thoroughly and dispositioned
appropriately.
The ECP manager had received approval from senior management to conduct
extensive benchmarking at other nuclear facilities. This effort has been funded in
the 2008 budget. The ECP manager planned to contact the ECP managers at
several other sites to obtain information about how other programs write reports,
apply policies and guidelines, and advertise their programs. The effort will
include reviewing performance indicators and methods for using the programs
metrics to better educate management about resolution of issues. One of the
other areas to be pursued is how other sites integrate safety issues into their
CAPs without compromising confidentiality.
The ECP manager was actively working to increase the awareness of the
program by making the program more visible at the site. The ECP manager had
recently hired two new ECP investigators, and was planning to hire a third with
- 145 - Enclosure
greater technical knowledge to better ensure that each concern is assessed
appropriately. Interviewees indicated that the ECP staff was well known, well
liked, and approachable.
The ECP was developing a plan to re-market the program. Since its integration
into the IIRP, the ECP has lost some of its identity to the Palo Verde staff. The
ECP manager planned to work with the communications department to develop a
new way to communicate the purpose of the ECP without losing integration with
the IIRP.
Differing Professional Opinions Program. The DPO program was an avenue for
resolving technical disagreements between staff members. The process
required an independent third party with appropriate technical knowledge to
review both sides of the issue and negotiate an acceptable resolution to the
problem. After the review is complete, both parties have the option to agree with
the resolution. If there is no agreement, the initiator may choose to escalate the
issue to the senior management team for resolution where the final decision will
be made by the site vice president/CNO. The team reviewed seven recently
closed DPO files and concluded that the DPO process was effective.
Management Issues Tracking Resolution Process. The MITR process was
designed to resolve personnel issues arising between management and staff and
was managed by the HR department. As previously mentioned, the team noted
some confusion among the staff as to the purpose of this process and took
personnel issues or concerns to the ECP more frequently than to HR. Many of
those interviewed had never heard of the MITR process. The team reviewed all
MITR files from 2007 and determined that the issues had been investigated and
resolved effectively.
Retaliation and the Disciplinary Review Board. Approximately 98 percent of the
interviewees stated that they had not experienced, nor heard of any issues of
retaliation, harassment, intimidation or discrimination at Palo Verde. Some
interviewees expressed concern that new accountability standards for industrial
safety might lead to future perceptions of retaliation, but the team noted that the
licensee was working to quell those impressions.
The licensees Disciplinary Review Board (DRB) screened disciplinary actions for
evidence of retaliation. The team reviewed several examples of the DRBs
efforts to ensure that controversial terminations were not viewed by staff as being
retaliatory. One example was a case where an individual had been terminated
because of a fitness-for-duty (FFD) violation. Management worked with the line
organization to explain the FFD process and the reasons why an employee might
be fired for violating FFD standards. This communication successfully diffused
the rumors surrounding this particular termination.
At the time of the inspection, the DRB did not review actions involving contractor
personnel, but the ECP and HR were assessing the need to expand the scope of
the program. Both organizations were benchmarking disciplinary review
processes at other sites to better understand how Palo Verde can revise its own
process to include contractor actions and ensure that all disciplinary actions are
thoroughly reviewed for perceptions of retaliation.
- 146 - Enclosure
Employee Dispute Resolution Process. The Employee Dispute Resolution
(EDR) process was a corporate-level program that allows an employee to dispute
a disciplinary action. There were three steps in the process. The first step
requires the individual to present the dispute and request resolution from his or
her direct supervisor. If the employee does not agree with the supervisors
response, the employee can appeal the issue to the HR manager. At this second
step, the HR manager assigns a representative to investigate the dispute and
propose a solution that is acceptable to both the employee and supervisor.
When a disciplinary action or termination takes place, or if the result of Step 2 is
not acceptable to the individual, he or she has a choice to request a review of the
action taken by either the APS Corporate Vice President of HR or from a review
panel. Employees may dispute the nature or severity of the impending discipline.
During the review, the management team will try to ensure that the employee is
able to openly discuss their opposition to the action. Once the review takes
place, the decision to change the disciplinary action must be made within 10
working days.
The teams review of the EDR process indicated that the EDR generally reduced
the level of discipline applied, but there were no terminations that had been
reversed as a result of the process. The team determined that this process was
effective in resolving employee disputes involving disciplinary actions.
b.13 Safety Policies
The team concluded that Palo Verdes safety policies and training related to
safety culture and the safety conscious work environment were appropriate.
Interviews indicated that the new senior management team was generally
perceived as believable in their emphasis on nuclear safety and as walking the
talk. Focus group participants who had exercised the new senior managers
invitations to send an email or other communication regarding concerns or
suggestions commented favorably that their issues were taken seriously and, in
most cases, resulted in action. Consistent with the results of the licensees
safety culture assessment, the NRC team determined that most personnel
interviewed were cautiously optimistic that the new senior management team
can be trusted to improve performance at the site.
9.0 REVIEW OF YELLOW FINDING - CONTAINMENT SUMP VOIDING
Before commencing the inspection, the licensee informed the NRC that they were not
prepared to support a closure review of the corrective actions associated with the Yellow
finding. Consequently, the team only reviewed the licensees progress in addressing the
Yellow findings performance concerns.
The team identified that the licensee was unable to effectively track the completion of
corrective actions associated with the two NRC IP 95002 supplemental inspections and
had not evaluated the effectiveness of corrective actions taken for this item. The inability
of the licensee to resolve the Yellow finding performance deficiencies contributed to
several of the violations documented in this report.
- 147 - Enclosure
a. Inspection Scope
The team reviewed the status of the implementation and completion of corrective
actions associated with a Yellow finding previously issued to Palo Verde regarding the
voiding of ECCS piping in all three units. The team evaluated the results of previous
NRC IP 95002 inspections related to this finding, as well as prior Palo Verde
performance improvement plans and corrective action plans. The team also reviewed
a recent audit of these corrective actions conducted by Palo Verde.
b. Observations and Findings
On October 24, 2007, the team reviewed the July 2004 Yellow finding to determine if
the associated corrective actions had been completed and if they had been assessed
by the licensee as effective. The root cause analysis for the Yellow finding identified
several deficiencies which were segregated into 10 focus areas. These 10 focus
areas were assigned to individual licensee managers or focus area owners. The
December 12, 2005, and the October 11, 2006, NRC IP 95002 inspections determined
that the corrective actions for these deficiencies were not completed. The areas of
concern involved questioning attitude, technical rigor, technical review, the
establishment of performance measures and metrics, and the use of OE. PVNGS
responded to the NRC in a November 16, 2006, letter detailing further commitments in
completing these corrective actions by March 30, 2007.
In June 2007, the licensee completed an IP 95002 effectiveness review and concluded
that they had not maintained current documentation of the project which precluded an
accurate status assessment of the corrective actions. The checklist used for this
effectiveness review described several reasons for non-completion of the corrective
actions, including: improper alignment of the corrective action to the root cause;
corrective actions not assigned as CRDRs and CRAIs (which did not allow for
assessment of corrective action completion); CRDR and CRAI completion dates being
extended several months past original due dates; and the lack of metrics to measure
effectiveness (originally scheduled for completion by December 1, 2006). An
additional issue was that no effectiveness reviews (effectiveness reviews of
engineering products was originally scheduled to be complete by February 1, 2007)
were conducted to insure proper closure of corrective actions.
Following discussions with the licensee, the team determined that in early 2007, when
it was known that Unit 3 was entering Column 4, the focus area owners assumed that
the IP 95002 corrective actions would be integrated into the IP 95003 process. During
this period a new senior management team was arriving and it was assumed by the
focus area owners that a new plan would be developed for site improvement. As a
result, the IP 95002 corrective actions were administratively forgotten as stated in the
evaluation report and PVAR 3030058, which identified this deficiency on
June 19, 2007. The licensee initiated CRDR 3031092 to resolve their inability to
address the Yellow finding performance concerns.
10 REVIEW OF WHITE FINDING - EMERGENCY DIESEL GENERATOR K-1 RELAY
Prior to the performance of the inspection, the licensee indicated they had not completed
the effectiveness reviews of the root causes and corrective actions associated with the K-1
relay failure. Consequently, the team reviewed the licensees progress and did not
- 148 - Enclosure
complete an assessment using IP 95001. A subsequent inspection will be completed
using IP 95001 as part of the NRCs review of the items described in the Confirmatory
Action Letter dated June 21, 2007.
a. Inspection Scope
The team reviewed the licensees assessment of the White finding associated with the
Unit 3 K-1 relay to assure that the root causes and contributing causes of the risk
significant performance issues were understood. In addition, the team reviewed the
extent of condition and corrective actions to verify that they were sufficient to address
the root causes and contributing causes, and to prevent recurrence.
Specifically, CRDR 2926830, Unit 3 Diesel Generator K1 Contactor Repeat Failure,
Revision 3, dated September 20, 2007, was reviewed using the guidance provided in
IP 95001. CRDR 2926830, incorporated the results of the Palo Verde ImPACT Team
review to correct inadequacies in the previous revision of the root cause investigation.
In addition, Revision 2 of this document was reviewed, along with APS
Correspondence 102-05626-CDM/SAB/JAP/CJS from D. Mauldin to US NRC, dated
January 9, 2007, responding to NRC Inspection Report 05000528; 05000529;
05000530/2006012 and a draft copy of the K-1 Relay Issue Problem Development
Sheet, dated July 26, 2007, used by the licensee to evaluate and address the
inadequacies in earlier root cause investigations of this problem.
b. Observations and Findings
The team considered the technical analysis provided in the root cause investigation
analysis to be adequate. However, the team observed several examples where the
investigation could have been more technically rigorous or the investigators should
have had a more questioning attitude. Specifically:
1) Root Cause 1 stated that the K1 relay was treated as a single replaceable
component; however, there were no design documents or drawings of this safety-
related relay found in the PVNGS nuclear records as stated in the Overview of K1
Contactor History.
2) The discovery, during troubleshooting, of variations of straight and bent actuator
arms without corroborating drawings may have indicated that field modifications
had been made at some time in the past and thus may have invalidated the original
equipment qualification.
3) The decision to "adjust and field straighten" the actuator arms may have
invalidated the equipment qualification. Metal fatigue and spring compression
issues are mentioned; however, other qualification issues such as seismic
qualification were not.
4) The report was not rigorous in documenting the extent of condition. Specifically,
the K1 relay condition could have also existed in the other two units at the same
time; thereby, having an impact on plant risk at the other two units. The Safety
Significance section, as it was written, potentially indicated a lack of appreciation
by the licensee of the impact that the inoperability of safety systems and
components had on plant risk.
- 149 - Enclosure
11 LICENSEE-IDENTIFIED VIOLATIONS
The following violations of very low significance (Green) were identified by the licensee
and are violations of NRC requirements which meet the criteria of Section VI.A of the
NRC Enforcement Policy, NUREG-1600, for being dispositioned as NCVs:
a. Technical Specification 5.4.1.a requires written procedures to be established,
implemented, and maintained covering the activities specified in Regulatory
Guide 1.33, Quality Assurance Program Requirements (Operations), dated
February 1978. Regulatory Guide 1.33, Appendix A, Item 1.l, Plant Fire Protection
Program, requires, in part, procedures for plant fire protection. Procedures
14DP-0FP34, Fire Watch Duties, and 14DP-0FP36, Hot Work Permit, stated that in
the event of a fire, notify the fire department by calling the site emergency extension
(i.e., contact security who contacts the control room). Contrary to this requirement,
personnel notified the site fire department via the normal fire department extension
vice the site emergency extension following a small fire in Unit 3 on October 5, 2007.
This resulted in the control room not being notified of the fire until several hours after
the fire started, which impacted the ability of the SM to implement the EAL assessment
process. The licensee subsequently determined that no EAL classification would have
been required since the fire lasted less than five minutes. The licensee entered this
item into the CAP as PVARs 3071922 and 3071994. This finding was determined to
be of very low safety significance because it did not result in a missed emergency
classification.
b. 10 CFR Part 50, Appendix B, Criteria XVI, Corrective Action, requires the licensee to
take appropriate and timely corrective action for conditions adverse to quality. The
inspectors reviewed CRAI 2942350 that addressed training for chemistry personnel on
changes to the 10 CFR 50.59 Guidance Manual. Some Chemistry personnel had not
attended the training and the CRAI was closed as complete. This corrective action
was in response to the ESP chemistry issues which resulted in the fouling of the EDG
heat exchanger in 2006. The licensee did review procedures that were revised by
these personnel that had not attended this training. The licensee performed an extent
of condition and found one individual that was not qualified on applicability
determinations had performed applicability determinations with supervisor permission
because they thought the individual was qualified to perform applicability
determinations after attending chemistry training in November 2006. Additionally, as a
follow up to PVAR 3009064, dated May 4, 2007, the team reviewed an additional nine
CRDRs reported in 2005, four CRDRs in 2006 and eleven CRDRs as of
October 5, 2007 related to personnel performing safety-related and non-safety-related
activities without proper qualifications. No items of significance were identified. This
event was documented in the licensees CAP as PVARs 3073306 and 3082659. This
finding is of very low safety significance because the licensee concluded that the
procedures that were changed and the tasks that were performed did not contain
significant errors and had not resulted in the need to perform an evaluation for
applicability determinations.
c. Technical Specification 5.4.1.a requires, in part, that written procedures be
established, implemented, and maintained covering the activities specified in Appendix
A of Regulatory Guide 1.33, "Quality Assurance Program Requirements (Operations),"
dated February 1978. Regulatory Guide 1.33, Appendix A, Section 9a, requires
maintenance that can affect safety-related equipment be properly preplanned and
- 150 - Enclosure
12 MANAGEMENT MEETINGS
On December 19, 2007, a public meeting was held to present the results of the inspection
to Mr. R. Edington, Senior Vice President, Nuclear, and other members of the licensees
staff. The licensee acknowledged the inspection results. Proprietary information was
reviewed during the inspection. The proprietary information was returned to the licensee
and was not included in this inspection report.
On December 19, 2007, a public meeting was conducted following the IP 95003 exit
meeting to discuss the licensees performance improvement initiatives.
- 151 - Enclosure
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
G. Andrews, Director, Performance Improvement
S. Bauer, Director, Regulatory Affairs
R. Bement, Vice President, Nuclear Operations
P. Borchert, Director, Operations
P. Brandjes, Department Leader, Maintenance
R. Buzard, Senior Consultant, Regulatory Affairs
D. Carnes, Director, Nuclear Assurance
P. Carpenter, Department Leader, Operations
R. Cavalieri, Director, Outages
K. Chavet, Senior Consultant, Regulatory Affairs
D. Coxon, Unit Department Leader, Operations
R. Edington, Senior Vice President, Nuclear
D. Elkington, Consultant, Regulatory Affairs
J. Gaffney, Director, Radiation Protection
T. Gray, Department Leader, Radiation Protection
K. Graham, Department Leader, Fuel Services
M. Grigsby, Unit Department Leader, Operations
M. Grissom, Section Leader, Reactor Engineering
J. Hesser, Vice President, Engineering
M. Karbasian, Director, Engineering
D. Marks, Section Leader, Regulatory Affairs
S. McKinney, Department Leader, Operations Support
J. Mellody, Department Leader, PV Communications
E. ONeil, Department leader, Emergency Preparedness
M. Radspinner, Section Leader, Systems Engineering
T. Radtke, General Manager, Emergency Services and Support
H. Ridenour, Director, Maintenance
F. Riedel, Director, Nuclear Training Department
M. Shea, Director, ImPACT Team
E. Shouse, Representative, EPE
M. Sontag, Department Leader, Performance Improvement
D. Straka, Senior Consultant, Regulatory Affairs
J. Taylor, Unit Department Leader, Operations
D. Vogt, Section Leader, OPS STA
T. Weber, Section Leader, Regulatory Affairs
J. Wood, Department Leader, Nuclear Training Department
NRC Personnel
M. Runyan, Senior Reactor Analyst
A-1 Attachment
Items Opened and Closed
Item Number Type Description
05000528; 05000529; NCV Eight Examples of the Failure to Implement the
05000530/2007012-01 Operability Determination Process
05000528; 05000529; NCV Failure to Implement Adequate Design Controls for
05000530/2007012-02 Condensate Storage Temp.05000530/2007012-03 NCV Inadequate Installation of Fire Sprinklers
05000528; 05000529; NCV Six Examples of a Failure to Implement the
05000530/2007012-04 Corrective Action Program Requirements
05000528; 05000529; NCV Failure to Evaluate Performance Monitoring Criteria
05000530/2007012-05 for Auxiliary Feedwater System
05000528; 05000529; NCV Failure to Meet Technical Specification Surveillance
05000530/2007012-06 Requirement 3.6.6.6
05000528; NCV Failure to Meet Technical Specification Surveillance
05000529/2007012-07 Requirement 3.0.3
05000530/2007012-08 NCV Two Examples of a Failure to Maintain Control of
Transient Combustibles05000530/2007012-09 FIN Failure to Install Emergency Lighting in Containment
Prior to Work Commencement
05000530/2007012-10 NCV Failure to Follow Procedures for Temporary
Shielding Installation
05000528; 05000529; NCV Inadequate Implementation of Risk Management
05000530/2007012-11 Actions and Risk Assessments for the Switchyard
05000530/2007012-12 NCV Incorrect Rigging of Personal Airlock Door
05000530/2007012-13 NCV Failure to Maintain Configuration Control of
Pressurizer Instrument Condensing Pot Support
Brackets
05000528; 05000529 NCV Failure to Implement Maintenance Rule
05000530/2007012-14 Requirements for the High Pressure Safety Injection
System
05000528; 05000529; NCV Inability to Implement Emergency Action Levels05000530/2007012-16
05000530/2007012-17 NCV Inadequate Briefings of Radiological Conditions05000529/2007012-18 NCV Failure to Periodically Update the Updated Final
Safety Analysis Report
Items Opened
05000528; 05000529; AV Failure to Correct a Risk Significant Planning
05000530/2007012-15 Standard
05000528; 05000529; URI Routine Heavy Use of Overtime
A-2 Attachment
List of Acronyms
ACC Arizona Corporate Commission
ACT Action Tracking System
ADAMS Agencywide Documents Access and Management System
ADV atmospheric dump valve
ALARA as low as reasonably achievable
ARRC Action Request Review Committee
AT activity tracking
ATC at-the-controls
CAL Confirmatory Action Letter
CAP corrective action program
CAPR corrective action to prevent recurrence
CARB corrective action review board
CCDP conditional core damage probability
CDBR component design basis review
CDF core damage frequency
CFR Code of Federal Regulations
CRAI condition report action item
CRDR condition report/disposition request
CRS control room supervisor
CST condensate storage tank
DPO differing professional opinion
DRB Disciplinary Review Board
EAL emergency action level
EC emergency coordinator
ECP Employee Concerns Program
ECCS emergency core cooling system
ECE engineering change evaluation
ED emergency director
EDG emergency diesel generator
EDR Employee Dispute Resolution
EOP emergency operating procedure
EPIP Emergency Plan Implementing Procedure
EQ environmental qualification
ESP essential spray pond
EW essential cooling water
FA functional assessment
FFD fitness-for-duty
FOP fundamental overall problem
FP fire protection
GPH gallons per hour
HEP human error probability
HPSI high pressure safety injection
A-3 Attachment
IIRP Integrated Issues Resolution Process
IMC Inspection Manual Chapter
ImPACT improved performance and cultural transformation
ISCPET Independent Safety Culture Performance Evaluation Team
ISLOCA intersystem loss of coolant accident
IP Inspection Procedure
KART key attribute review team
LER Licensee Event Report
LERF large early release frequency
LPSI low pressure safety injection
LOCT licensed operation cycle training
LOOP loss of offsite power
MITR Management Issues Tracking Resolution
MR maintenance rule
NCR nonconformance report
NPSH net positive suction head
NRC U.S. Nuclear Regulatory Commission
O&M Operations & Maintenance
ODMI operational decision making instruction
OE operating experience
PAL personnel airlock
PAR Protective Action Recommendation
PC performance criteria
PDS problem development statement
PI&R problem identification and resolution
PM preventative maintenance
PPM parts per million
PSF performance shaping factor
PSIA pounds per square inch absolute
PVAR Palo Verde action request
PVNGS Palo Verde Nuclear Generating Station
RP radiation protection
RVLMS reactor vessel level monitoring system
SCWE safety conscious work environment
SRP Salt River Project
SGTR steam generator tube rupture
SIBP Site Integrated Business Plan
SIIP Site Integrated Improvement Plan
SM shift manager
SMART specific, measurable, achievable, reasonable, and timely
SPAR Standardized Plant Analysis Risk
SOV solenoid operated valve
SQFT square foot
SRO senior reactor operator
SSC structures, systems, and components
A-4 Attachment
STA shift technical advisor
SWMS site work management system
SWYD switchyard
TCCP transient combustible controls permit
TDS total dissolved solids
TID total integrated dose
TGO transmission/generation operations
T&TV trip and throttle valve
TS Technical Specification
TSSR Technical Specification Surveillance Requirement
UFSAR Updated Final Safety Analysis Report
URI unresolved item
WO work order
A-5 Attachment