ML063320538

Response to NRC Followup Supplemental Inspection Report; 05000528/2006010, 05000529/2006010; 05000530/2006010

ML063320538
Person / Time
Site:	Palo Verde
Issue date:	11/16/2006
From:	James M. Levine Arizona Public Service Co
To:	Mallett B Region 4 Administrator
References
102-05594-JML/SAB/JAP/DJS/DCE, IR-06-010
Download: ML063320538 (33)

Text

PAl A subsidiary of Pinnacle West Capital Corporation James M. Levine Mail Station 7602 Palo Verde Nuclear Executive Vice President Tel (623) 393-5300 PO Box 52034 Generating Station Generation Fax (623) 393-6077 Phoenix, Arizona 85072-2034 102-05594-JM LISAB/JAP/DJS/DC E November 16, 2006 Dr. B. S. Mallett Regional Administrator, Region IV U. S. Nuclear Regulatory Commission 611 Ryan Plaza Dr., Suite 400 Arlington, TX 76011-4005

Subject:

Palo Verde Nuclear Generating Station (PVNGS)Units 1, 2, and 3 Docket Nos: 50-528, 50-529, 50-530 Response to NRC Followup Supplemental Inspection Report (05000528/2006010; 05000529/2006010; 05000530/2006010)

The intent of this letter is to document the actions that APS is taking to address the remaining open items, as described in the NRC Followup Supplemental Inspection Report (NRC Inspection Report 50-528/529/530/2006-010), dated October 11, 2006.The report discussed the results of the NRC inspection using Inspection Procedure 95002 for the facility's Yellow finding. APS has modified its action plan to address comments from the NRC inspection report as well as our own concerns as we continue to monitor and adjust the plan in response to results.The inspection report discussed the results of the ten focus areas that the inspection team reviewed.

The inspection report identified that, with respect to the Yellow finding, nine of those areas are closed. APS will continue improvements in each of the nine areas and will monitor the effectiveness of those actions. The closed focus areas include: Focus Area 1, "Procedures Did Not Contain Necessary Requirements" Focus Area 2, "Lack of Specific Provisions in the Design and Licensing Basis" Focus Area 4, "Inadequate Communication of Design Information" I I U. S. Nuclear Regulatory Commission Dr. B. S. Mallett Response to NRC Followup Supplemental Inspection Report dated October 11, 2006 Page 2 Focus Area 5, "Inadequate Problem Identification and Resolution" Focus Area 6, "Limited or Weak Operating Experience Program" Focus Area 7, "Limited Experience and Training" Focus Area 8, "Limited Resources" Focus Area 9, "Limited Nuclear Assurance Department Oversight" Focus Area 10, "Limited Procedural Guidance" One focus area remaine'd open, requiring additional action: Focus Area 3, "Lack of Questioning Attitude and Technical Rigor of Individuals." APS agrees that corrective actions for concerns involving questioning attitude, technical rigor, and technical review have not been fully effective.

Performance measures and metrics to monitor the corrective actions that have been taken in this area had not been fully developed and, therefore, had not been effective in identifying needed adjustments to the corrective actions. These remaining issues and our actions to address them are listed in detail in the enclosure.

We are confident these actions when implemented will address our and the NRC's remaining concerns for this area and will support closure of the Yellow finding.Two focus areas, while closed for the purposes of the 95002 inspection, also require additional actions:* Focus Area 5, "Inadequate Problem Identification and Resolution." While corrective actions specific to the Yellow finding have been completed, we understand that improvement in the corrective action program is still needed. It is our understanding that future NRC reviews associated with this area will be handled under the PI & R substantive crosscutting issue. The Corrective Action Area of our Performance Improvement Plan will include the actions to address these corrective action program deficiencies." Focus Area 6, "Limited or Weak Operating Experience Program." The inspection team concluded that corrective actions did not incorporate routine use of operating experience (OE) in emergent activities.

Further NRC review of corrective actions related to OE in daily activities will be performed during NRC assessment of Focus Area 3. Our actions to address these concerns are incorporated into the enclosed action plan.As discussed in the inspection report, during the September 7, 2006 public meeting, APS indicated that success measures would be developed for NRC review prior to our request for an additional NRC assessment of the Yellow finding.

U. S. Nuclear Regulatory Commission Dr. B. S. Mallett Response to NRC Followup Supplemental Inspection Report dated October 11, 2006 Page 3 The actions described in the enclosure provide the success measures for focus areas 3 and 6. The measures are currently in various stages of implementation and completion, as noted.APS will notify the NRC when these measures have been implemented sufficiently and show the desired results, as defined in the enclosed plan. At that time, we will request a followup inspection.

The listed completion dates are for completion of the related action items and do not represent dates by which we expect to have achieved the results required to ask for a followup inspection.

The actions described in this letter represent corrective action plans; they are not considered to be regulatory commitments.

Should you have any further questions, please contact Craig Seaman at (623) 393-5421.Sincerely, 1A JMLISAB/JAP/DJS/DCE/gt

Enclosure:

0 95002 Inspection Closure Action Plan cc: B, S. Mallett M. B. Fields G. G. Warnick NRC Region IV Regional Administrator NRC NRR Project Manager NRC Senior Resident Inspector for PVNGS ENCLOSURE 95002 Inspection Closure Action Plan 95002 Inspection Closure Action Plan Action Item 1: Revise Operational Decision Making guidance (ODP- 16)Problem Statement:

Station personnel are missing opportunities to use the Operational Decision Making process for degraded conditions.

Action Plan Goal: To develop additional tools for Operational Decision Making and increase the use and effectiveness of ODP-16.Actions Previously Taken: (a) Benchmarked current guidance contained in Operations ODP- 16 (Principles for Effective Operational Decision-Making and the use of Management Review Teams) with guidance from other utilities (Exelon and Entergy).(b) Implemented an interim change to procedure ODP-16 based on benchmarking results.Current Status: Decision Making Errors (DME) metric indicates that additional actions are required.Actions to be taken: (a) The following actions will be taken to improve the use of ODMI process: E Include an assessment for emergent issues in the Shift Manager Turnover (SMTO)0 Include an assessment for off-normal alignment in the Crew Turnover sheet.m Include an assessment for shift activities in the crew briefing template.0 Brief the Operations staff on the resultant Operations Department Practice changes.0 Perform formal training in License Operator Continuing Training (LOCT) on the Operational Decision Making process. Training to include review of case studies.Metric Intent: To measure the number of Decisions Making Errors (DME) for Operations, Engineering, Maintenance, and site as captured via the Corrective Action Program (CRDRs).Events captured in the metric that occurred pre-2006 are referred to as "Latent" and are tracked for informational purposes.Metrics: Site metric for Decision Making Errors 1. Decision Making Errors -Operations

2. Decision Making Errors -Engineering

3. Decision Making Errors -Maintenance

4. Decision.Making Errors -Palo Verde site 5. Decision Making Errors -Latent Effectiveness Review: Perform a self-assessment on the effectiveness of the ODMI process by 03/30/07.Due Date: Complete actions by 03/30/07.Page 1 of 29 11/16/06 Metric Description Decision Making Errors Operations, Engineering, Maintenance, Palo Verde Site, Latent Criteria:

The following are inputs to the overall DME metric for each group listed above.1. Lack of technical rigor o A deficient product or document due to technical inaccuracies because all the facts were not gathered, the facts were incorrect, and/or the wrong conclusions were made. This includes "narrow focus" approaches to problems or solutions.

2. Failure to recognize hazard, error, or deficiency o A failure to recognize a deficient or abnormal condition

-unawareness.

3. Lack of questioning attitude o Information received or observed and is contrary to the expected or the norm. Some amount of knowledge-based analysis is performed, but is inadequate.

Usually characterized as mindset, tunnel vision, and wrong assumptions.

4. Proceeding in the face of uncertainty o Confronted with an abnormal situation or information and no gathering of facts, conclusions, and/or decision making are apparent and action is taken. Characterized as overly optimistic, overconfident, etc.5. Non-conservative decision-making o All the facts are gathered and weighed, but the wrong or non-conservative decision is made between two or more possible solutions (e.g., production over safety margin). Characterized as "can do" attitude, inappropriate attitude, etc.Data ,oal: o Site DME Metric goal is management directed.o Department DME Metric goal is trend only.o Latent DME Metric is information only.Data comes from: Corrective Action Program database.Page 2 of 29 11/16/06 Decision Making Errors Operations Monthly Number of Errors 10 9 8 7 6 5 4 3 2 1 0 I1 Good Jan Feb Mar Apr May Jun Jul Aug Sep 2006 Oct Nov Dec Page 3 of 29 11/16/06 Decision Making Errors Engineering Monthly Number of Errors 10 9 8 7 6 5 4 3 2 1 0 I Good Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2006 Page 4 of 29 11/16/06 Decision Making Errors Maintenance 10 9 8 7 6 5 4 3 2 1 0 I1 Good Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2006 Page 5 of 29 11/16/06 Decision Making Errors Palo Verde Site 10 9 8 7 6 5 4 3 2 1 0 Good Jan Feb Mar Apr May Jun Jul Aug Sep 2006 Oct Nov Dec Page 6 of 29 11/16/06 Decision Making Errors Latent Monthly Number of Errors 10 9 8 7 6 5 4 3 2 1 0 I Good Jan Feb Mar Apr May Jun Jul 2006 Aug Sep Oct- Nov Dec Page 7 of 29 11/16/06 95002 Inspection Closure Action Plan Action Item 2: Monitoring Operability Determination (OD) quality Problem Statement:

Quality of Operability Determinations does not always meet site expectations.

Action Plan Goal: To ensure that Operability Determinations meet procedural requirements.

Actions Previously Taken: (a) Established an OD Quality Review Board with Engineering, NAD, Performance Improvement and Training representation.(b) Included results of Engineering Product Review Board (EPRB) reviews in the OD Quality metric.Current Status: Metrics indicate inconsistent use of the OD program. This will be addressed by the following actions: Actions to be Taken: (a) Incorporate changes to 40DP-9OP26 (Operability Determination procedure) to support new action request process roll-out.(b) Implement changes to the OD Quality Review Board: " Develop a charter for the OD Quality Review Board" Include qualifications for the OD Quality Review Board to include Shift Manager experience and an off-site/independent individual (c) Change OD Timeliness metric to include corrective action due dates.Metric Intent: To measure the quality of Operability Determinations.

Metrics: 1. Operability Determination Quality 2. Operability Determination Timeliness Effectiveness Review: Perform a self-assessment on the effectiveness of the OD quality process by 03/30/07.Due Date: Complete actions by 03/30/07.Page 8 of 29 11/16/06 Metric Description Operability Determination Quality Criteria: 1. Self identify potential challenges to operability o Number of issues identified by oversight agency.o Number of corrective action documents the CRDR Review Committee sent for control room review because initiator and leader failed to identify Control Room review required.o Number of work documents not sent to Control Room that were subsequently determined to be degraded / non-conforming tech specifications systems, structures and components during weekly review by work control Senior Reactor Operator.2. Timely Identification to Operations of Operability Issues o Corrective action documents that were not processed/identified to the Shift Manager within the same shift as discovery of the degraded / non-conforming condition affecting technical specifications systems, structures and components.

o Work documents that were not identified to Shift Manager within same shift for degraded/ non-conforming conditions affecting tech specifications systems, structures and components.

3. Inadequate tech information or communication of information for the IOD o Does not follow requirements of section 3.2.2 of the Operability Determination procedure as a minimum.o Immediate Operability Determinations (IOD) not sent to responsible Engineering leader for review when Prompt Operability Determination (POD) is not requested..o IOD contains inaccurate information.

o IOD does not provide sufficient justification and a POD is not requested (Shift Technical Advisor / Engineering leader review)4. Inadequate tech information or communication of information for the POD o Does not follow requirements for POD as described in section 3.5 of OD procedure.

o Nuclear Assurance Department

/ Shift Technical Advisor section leader determines POD is inadequate.

o Engineering Product Review Board determines POD is "unacceptable".

5. Shift Manager Review of POD o Shift Manager concurrence with POD conclusion is documented in the POD.o Simple Quality Verification and Validation of the POD product should have caught an error but did not.Data goal: Metric goal is "trend only" and management directed.Data comes from: Corrective Action Program database.Page 9 of 29 11/16/06 Operability Determination Quality Number of Operability Determinations that did not meet expectations for Quality 10 9 8 7 6 5 4 3 2 1 0'I Good Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2006 Page 10 of 29 11/16/06 Metric Description Operability Determination Timeliness Criteria:

Corrective Action(s) for each Operability Determination (OD) will be completed in accordance with the schedule established by the Director of Operations.

Data goal: Number of ODs that have corrective actions that have exceeded the schedule date.Data comes from: Corrective Action Program database.Page 11 of 29 11/16/06 Operability Determination Timeliness Number of Operability Determinations that did not meet expectations for Timeliness 10 9 8 7 6 5 4 3 2 1 0.I Good Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2006 Page 12 of 29 11/16/06 95002 Inspection Closure Action Plan Action Item 3: Establish reviews of Engineering work products to improve technical adequacy, rigor and questioning attitude Problem Statement:

Historical issues indicate a need to monitor the quality of Engineering products and errors in process implementation, including Operability Determinations.

Action Plan Goal: To see a decrease in the number of Engineering decision-making errors and Engineering products that did not meet expectations per the Engineering Products Review Board (EPRB).Actions Previously Taken: (a) Engineering human performance tools were developed from industry benchmarking.(b) Engineering and leaders have been trained on tool usage.(c) An EPRB was established and has performed quarterly quality and technical accuracy reviews of selected Engineering work products.Current Status: Engineering does not have a formal trouble shooting guide, and EPRB currently meets quarterly.

Actions to be Taken: (a) An Engineering tool is being developed to add formality and consistency to the troubleshooting process.(b) The EPRB function is being modified to meet on a monthly basis and to become a senior Engineering leadership review/oversight group for engineering work products and human performance.

A new human performance review board will continue reviews of Engineering work products.(c) EPRB feedback on process implementation and quality of work products is provided to Engineering management, and used to revise expectations and training to improve performance.(d) Each of the four Engineering departments assesses and monitors decision-making errors (DMEs) for their department.

Metric Intent: To monitor work product quality and decision-making errors (DME).Metrics: Engineering DME metric.1. Decision-Making Errors -Engineering.

Refer to Action Item 1.2. Engineering Product Review Board Results. (Metric display is under development).

Effectiveness Review: Perform a self-assessment of the effectiveness of engineering products process by 02/01/07.Due Date: Complete actions by 02/01/07.Page 13 of 29 11/16/06:

• Metric Description Decision-Making Errors -Engineering Engineering department decision-making errors are compiled and reflected monthly.Refer to Action Item 1 Page 14 of 29 11/16/06

{ DRAFT)Metric Description Engineering Product Review Board Results Criteria:

A minimum of 20 engineering products will be reviewed monthly. Engineering product types subject to sampling will be defined.1.2.3.4.5.Product meets expectations Comment(s) made, no changes required Product must be changed -CRDR issued Margin impacted, no plant impact -CRDR issued Margin and plant impacted -CRDR issued Data goal: 90% or more meet criteria 1 & 2 Data comes from: Monthly Engineering Product Review Board results.=..DRAFT&.

Page 15 of 29 11/16/06 Engineering Product Review Board Results Metric Display is Under Development Page 16 of 29 11/16/06 95002 Inspection Closure Action.Plan Action Item 4: Improve use of Operating Experience Problem Statement:

Operating Experience (OE) is not consistently used for emergent issues.Action Plan Goal: To reduce plant events through the use of Operating Experience thereby improving plant performance and enhancing safety.Actions Previously Taken: (a) Completed a self assessment of the OE program.(b) Document high-tiered OE in the Corrective Action Program.(c) Developed checklist for high-tiered OE.(d) Perform independent reviews of high-tiered OE evaluations.

Current Status: Low-level OE is inconsistently used for emergent issues.Action to be Taken: (a) Incorporate station and industry Operating Experience into daily meetings and safety-human performance meetings.(b) Publish Operating Experience Outage books prior to each refueling outage. These will outline internal and external Operating Experience and the behaviors to prevent occurrence.(c) Revise Standard and Expectations Books to include specific expectations and desired behaviors for the use of Operating Experience.(d) Implement a station graded approach for the use of OE and reverse pre-job briefing process for normal and emergent work.(e) Add OE to work packages for normal and emergent work.(f) Make search engines more available to employees to make searching for OE easier.Metric Intent: To measure Palo Verde's use of OE on emergent issues.Metrics: Operation Experience (OE)1. OE Use Observations

2. Inadequate Use of OE Effectiveness Review: Conduct effectiveness review/self-assessment to include how/when OE is being used.Due Date: Complete actions by 06/01/07.Page 17 of 29 11/16/06 Metric Description OE Use Observations Criteria:

Percentage of field observations in which OE was appropriately utilized or discussed.

Data 2oal: Percentage of observations in which OE use or discussion meets or exceeds 90 percent of the opportunities.

Goal is set by management.

Data comes from: Performance Improvement Team, field, and management observation programs.Page 18 of 29 11/16/06 OE Use Observations Percentage of Observations that incorporated Adequate OE Percent 100 90 80 70 60 50 40 30 20 10 0 I Good Jan Feb Mar Apr May Jun Jul 2006 Aug Sep Oct Nov Dec Page 19 of 29 11/16/06 Metric Description Inadequate Use of OE Criteria:

Inadequate use/evaluation/implementation/etc.

of Operating Experience

-either Industry OE or Palo Verde OE. Specifically

-inadequate use/evaluation/implementation, etc. of industry or Palo Verde OE that could have prevented a Palo Verde site or department clock reset.Data 2oal: For "trend only" on an interim basis.Data comes from: A review of Apparent Cause Evaluation and Significant CRDR Evaluation events looking for missed opportunities for Operating Experience use.Page 20 of 29 11/16/06 Inadequate Use of OE Number of events that could have been prevented by using available OE Events 10 9 8 7 6 5 4 3 2 1 0'I Good Feb Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2006 Page 21 of 29 11/16/06 95002 Inspection Closure Action Plan Action item 5: Establish a formal plant walkdown process Problem Statement:

Identification of degraded / non-conforming conditions has not been consistently achieved.Action Plan Goal: To ensure that the appropriate plant personnel, such as Auxiliary Operators (AOs), plant engineers and system engineers, identify degraded / non-conforming systems, structures, or components.

Actions Previously Taken: (a) Implemented a program to walk down safety-significant systems weekly (as defined in procedure 79DP-9ZZ02) on a 12-week schedule.

This walkdown is performed by Operations and Engineering.

Current Status: Formalizing the program and the training to ensure that appropriate plant personnel identify these types of conditions.

Actions to be Taken: (a) Create a "stand-alone" procedure to walk down safety-significant systems weekly on a 12-week schedule." Add instructions to identify personnel safety issues and other common walkdown areas of interest." Develop a plant walkdown pre-job brief to ensure consistency of issues identified in the walkdowns." Include Maintenance Fix It Now (FIN) team in the weekly walkdowns." Coordinate weekly schedule with Maintenance and Engineering.(b) Provide plant walkdown classroom training for AOs.(c) Develop and administer a practical demonstration of plant walkdowns to the AOs.(d) Provide additional plant Walkdown training to improve general knowledge of walkdown expectations for STAs, FIN, Maintenance Engineers and System Engineers.(e) Develop a metric to monitor performance.

Metric Intent: To measure the number of degraded / non-conforming issues identified by the walkdown teams and NAD.Metrics: 1. Safety-Significant Systems Walkdowns

-Operations

/ Engineering Walkdown Teams.2. Safety-Significant Systems Walkdowns

-Nuclear Assurance Department Walkdown Teams.Effectiveness Review: Perform an effectiveness review of plant walkdowns.

Due Date: Complete actions by 03/30/07.Page 22 of 29 11/16/06 Metric Description Safety-Sihnificant Systems Walkdowns

-Operations

/ Engineering Walkdown Teams Criteria:

Degraded / non-conforming conditions identified by Operations

/ Engineering walkdown teams.Data Goal: Progressively fewer degraded / non-conforming conditions identified over time -management directed goal.Data comes from: Operations

/ Engineering walkdowns and resultant corrective action document(s).

Page 23 of 29 11/16/06 Safety-Significant Systems Walkdowns Operations

/ Engineering Walkdown Teams Number of Degraded / Non-Conforming Conditions Identified 10 9 8 7 6 5 4 3 2 1 0 I Good Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2006 Page 24 of 29 11/16/06 Metric Description Safety-Significant Systems Walkdowns

-Nuclear Assurance Department Walkdown Teams Criteria:

Degraded / non-conforming conditions, identified by Nuclear Assurance Department (NAD), not previously identified by Operations

/ Engineering walkdown teams.Data Goal: Progressively fewer degraded / non-conforming conditions identified over time -management directed goal.Data comes from: NAD walkdowns and resultant corrective action document(s).

Page 25 of 29 11/16/06 Safety-Significant Systems Walkdowns Nuclear Assurance Department Walkdown Teams 10 9 8 7 6 5 4 3 2 1 0 I1 Good Jan Feb Mar Apr May Jun Jul 2006 Aug Sep Oct Nov Dec Page 26 of 29 11/16/06 95002 Inspection Closure Action Plan Action Item 6: Palo Verde independent review of plant walkdown results Problem Statement:

The walkdown process is new and Nuclear Assurance Department (NAD)oversight is needed to ensure identification of degraded / non-conforming conditions by plant walkdown teams.Action Plan Goal: To see a decrease in the number of degraded / non-conforming systems, structures, or components identified by NAD and not previously noted by the line or from the formal integrated walkdown teams.Actions Previously Taken: (a) Utilizing the NAD Find It Now (FIN) team to conduct independent walkdowns, after the weekly plant walkdown team.Current Status: Performing walkdowns.

Actions to be Taken: 1. Observe and evaluate training described in Action Item 5.Metric Intent: To measure the number of degraded / non-conforming systems, structures, or components identified by NAD and not previously noted by the line from the formal integrated walkdown teams.Metric: Refer to Action Item 5 for metric.Effectiveness Review: Continuing monitoring will be conducted.

Due Date: Complete actions by 03/30/07.Page 27 of 29 11/16/06 95002 Inspection Closure Action Plan Action Item 7: Perform a review of 95002 metrics (10 Focus Areas) to ensure compliance with the metric guideline Problem Statement:

A review of the 95002 metrics /performance indicators has indicated that a consistent methodology was not used to develop metrics. Some metrics were not effective.

Action Plan Goal: To make 95002 metrics consistent and effective.

Actions Previously Taken: (a) Performed a review of each 95002 Focus Area metric.(b) Adjusted metric with appropriate management review and approval.(c) Conducted independent review of 95002 metrics.Current Status: Review of the 95002 metrics indicates that three metrics require adjustment.

Actions to be Taken: 1. Adjust the three metrics.Due Date: Complete actions by 12/1/06.Page 28 of 29 11/16/06 95002 Inspection Closure Action Plan 95002 Inspection Readiness Review Evaluate the readiness for a 95002 inspection.

The criteria are: (a) Actions above are completed or progressing satisfactorily, (b) Metrics and data streams that supply them have been independently verified to accurately portray actual performance (c) Metrics are either satisfactory or improving.

Monitor and adjust actions will be taken for performance that is declining or not improving.

Actions to be Taken (a) Develop effectiveness and inspection readiness plan.(b) Assemble a team to conduct a readiness review, which should include two industry individuals.(c) Evaluate completed readiness review results and determine if additional actions are required.Metrics/Measures:

Closure of 95002 NRC finding.Due Date: To be determined based on Metric results.Page 29 of 29 11/16/06