ML061010702

Response to Annual Assessment Letter, NRC IR 05000528-06-001, 05000529-06-01 & 05000530-06-001

ML061010702
Person / Time
Site:	Palo Verde
Issue date:	04/04/2006
From:	James M. Levine Arizona Public Service Co
To:	Document Control Desk, NRC Region 4
References
102-05456-JML/SAB/JAP/DJS IR-06-001
Download: ML061010702 (17)

Text

A subsidiary of Pinnacle Ifest Capital Corporation James M. Levine Mail Station 7602 Palo Verde Nuclear Executive Vice President Tel (623) 393-5300 PO Box 52034 Generating S ation Generation Fax (623) 393-6077 Phoenix, Arizona 85072-2034 102-05456-JMUSAB/JAP/DJS April 4, 2006 U. S. Nuclear Regulatory Commission ATTN: Document Control Desk Washington, DC 20555-0001

Subject:

Palo Verde Nuclear Generating Station (PVNGS)

Units: 1, 2, and 3 Dockets: 50-528, 50-529, 50-530 Response to Annual Assessment Letter NRC Inspection Report 05000528,2006001; 05000529/2006001; 05000530,2:006001)

Arizona Public Service Company (APS) has reviewed the Annual Assessment Letter (NRC Inspection Report 50-528/529/530/2006-01), dated March 2, 2006. The Annual Assessment Letter identified continuing substantive crosscutting issues in the areas of human performance and problem identification and resolution. Three common themes associated with human performance and three common themes associated with problem identification and resolution were specifically discussed.

APS agrees that these issues were present during the 2005 assessment period, and were also items identified in the 2004 assessment period.

APS has implemented an integrated improvement plan at Palo Verde which consists of the Business Plan, the Performance Improvement Plan, Departmental Improvement Plans and the Employee Incentive Plan. APS has evaluated the various aspects of the substantive crosscutting issues and has implemented action plans to address these issues and, specifically, the six common themes. The actions described herein are from A member of the STARS (Strategic Teaming and Resource Sharing) Alliance Callaway

• Comanche Peak

• Diablo Canyon

• Palo Verde

• South Texas Project

• Wolf Creek zWO t

U. S. Nuclear Regulatory Commission ATTN: Document Control Desk Respcnse to Annual Assessment Letter dated March 2, 2006 Page 2 variou s sections of the integrated improvement plan. We believe the integrated effect of the actions will improve performance in these areas. Progress is being measured by established performance indicators and adjustments will be made to the actions, as needed, until the desired performance is achieved. Initial results of the performance monitoring show signs of improved performance but many of the actions are just beginning to take hold and more run time and monitoring is needed to get conclusive results. APS is committed to achieving high levels of performance in both of these areas. In assessing the effect of the actions described herein, it is imperative that they be evaluated within the context of the entire integrated improvement plan. Each of the six common themes from the two substantive cross cutting issues is addressed in the enclosure to this letter.

The actions described in this letter represent the performance improvement plan action plans and are not considered to be regulatory commitments.

Should you have any further questions, please contact Craig Seaman at (623) 393-5421.

Sincerely, JML/SAB/JAP/DJS/gt

Enclosure:

Response to Substantive Cross-Cutting Issues Discussed in the NRC Annual Assessment Letter, dated March 2, 2006 cc:

B. S. Mallett NRC Region IV Regional Administrator M. B. Fields NRC NRR Project Manager G. G. Warnick NRC Senior Resident Inspector for PVNGS

ENCLOSURE Response to Substantive Crosscutting issues Discussed in NRC Annual Assessment Letter, dated March 2, 2006

1. The Failure of Personnel to Follow Procedures Probable Causes:

1)

Cognitive decisions were made to not follow procedures. Users rationalized decisions to depart from procedural direction.

2)

Critical activities were being performed without consistently consulting written guidance (e.g., workers relying on skill-of-the-craft or believing they know the procedure and, therefore, do not need to consult it). Criteria for determination and designation of level-of-use were not well understood or applied.

3)

Personnel in some cases did not recognize procedure non-compliance while performing a task.

4)

Standards and expectations for procedure use were not always clearly understood or reinforced.

Corrective Actions Taken

1)

PVNGS established site expectations on procedure use and adherence.

These management expectations provide a consistent definition on the various levels of use for procedures. Management expectations were communicated to site personnel in the Palo Verde Standards and Expectations handbook, in the leaders standards and expectations handbook, in procedure 01 DP-OAPO9 (Procedure Use and Adherence),

and in multiple reinforcing communications including leaders meetings and all-hands meetings. Standards and expectations were also emphasized during site wide Procedure Use and Adherence training.

2)

The standards and expectations were added to each employee's year-end; performance evaluation and are being included in the 2006 individual performance plans so that one-on-one communication of the expectations occurs.

3)

Each procedure was assigned a level of use designation to verify activities are correctly performed of either:

a.

Continuous Use-the procedure is used "in-hand," with steps completed as provided for in the document.

b.

Reference-reviewed prior to or referred to periodically during performance of activities.

c.

Information-refer to as needed based on frequency of use, individual knowledge and proficiency.

2

d.

Combined Use-use each procedure section according to the level identified (some procedures contain multiple usage levels).

4)

Revisions were made to the standard prevent events questions used in pre-job briefings to specifically include asking what document describes the task that is about to be performed. This question focuses the worker's attention on the procedure or work instructions for the task.

5)

One of the areas identified for emphasis in the leader observation program, implemented as the "+/- Logbook" in February 2006, has been procedure use and adherence.

6)

Monthly procedure stand-down meetings are being held by work groups which are designed to reinforce expectations of procedure use and adherence as well as provide an opportunity for employees to review procedures, gain better understanding of the technical information and associated procedure use, and identify needed improvements.

7)

In order to demonstrate the importance that senior leadership places on high level performance in this area, procedure use and adherence errors are reviewed by the Senior Leadership team at the daily senior leadership team meeting. Each error is also discussed with a senior manager by the entire reporting chain of individuals down to the person who committed the error. Where appropriate, non-compliance with procedures has resulted in employee discipline. These discussions have also led to the identification of procedure quality issues that are then corrected.

8)

Palo Verde implemented an organizational change that established a Performance Improvement Team to address human performance deficiencies. Included in this structure was the use of Performance Advocates (PA) within each organization responsible for facilitating performance improvement initiatives for the station. The PA's have been given Human Performance training and will serve as their organization's Human Performance expert.

Actlions to be Taken

1)

As the human performance department begins to roll-out its new processes, they and the performance advocates will begin more focused observations of site work. This will include consistency in the identification and evaluation of procedure use and adherence errors. This effort is scheduled to start in April 2006.

3

2)

To improve use of procedures, a search engine tool is being implemented for the site to enable easier identification of applicable procedures. This action is scheduled for completion in June 2006.

3)

A new Human Performance Simulator will be implemented in three phases. Phase one of this simulator is a tool designed to exercise both procedure use and adherence skills as well as self-checking and leader observation skills. Phase one is scheduled to be completed in the second quarter of 2006. A schedule is being developed for phases two and three.

4)

Potential human performance events will be evaluated by a Human Performance Review Board using both a culpability matrix and organizational weakness review in accordance with the station Human Performance Policy. This process will be in place by May 2006.

5)

Continued monitoring and evaluation by the Performance Improvement Team of the Site Monthly Trend Report and the Human Performance Health Report is ongoing to develop trending information and subsequent additional corrective actions, as needed.

Results Achieved to Date Site Monthly Trend Report (MTR) goals for 2006 were implemented which include a performance indicator for procedure use and adherence errors. To underscore the importance of reducing these errors, this goal has been included in the 2006 employee year-end incentive plan as well as the Unit 3 refueling outage incentive plan. Based on initial data, there appears to be an improving trend but there has been insufficient run time to make any definitive conclusions.

Qualitatively, observation program reports have shown 'procedure use and adherence' as one of the most frequently noted weaknesses but it was also noted as one of the most improved areas in the February report.

2. The Use of Inadequate Procedures Probable Causes

1)

In some cases, there has been inadequate procedural guidance and/or poor anticipation of system and human interaction during procedure and document development.

4

d.

Combined Use-use each procedure section according to the level identified (some procedures contain multiple usage levels).

6)

Procedure use and adherence, and identifying procedure problems (stop and correct the procedure when problems are identified) are focus areas for leader observations and interactions.

Actions to be Taken

1)

The SWMS Action Tracking (ACTs) tool will be implemented as the single management tool to identify and track the status of procedure change requests. This action will be completed in the 2nd quarter 2006.

2)

Operations, radiation protection, chemistry and maintenance have developed procedure improvement plans. Implementation of these plans will be completed by the end of the 3rd quarter 2006.

3)

The on-shift procedure change process will be improved to enable correction of inadequate procedures on the spot. This action is scheduled for completion by September 2006.

4)

The procedure process will be benchmarked against industry best practice plants to identify best practice performance indicators and reports for the procedure process, including an indicator for timeliness of feedback to the initiator. This action will be completed in September 2006.

5)

A procedure change request backlog indicator is being developed to provide an assessment of the health of the procedure change process.

This action will be completed in the 2nd quarter 2006.

6)

Continuous monitoring and evaluation of the Site Monthly Trend Report and the Human Performance Health Report is ongoing to develop trending information and subsequent additional corrective actions.

7)

A single site procedure standards organization will be established by the end of the 3' quarter 2006.

Results Achieved to Date A Human Performance Health Report was also developed that includes specific performance indicators according to the INPO Human Performance Model with a

'Procedure Adequacy' indicator as a barrier to error. This indicator is populated from the Condition Report database which tracks those events/conditions which have a cause attributed to an inadequate procedure. Based on initial data, there 6

appears to be an improving trend but there has been insufficient run time to make any definitive conclusions.

3. Ineffective Interactions Between Engineering and Operations Personnel When Assessing Degraded and Nonconforming Conditions Probable Causes

1)

Engineering and operations personnel did not consistently demonstrate sufficient knowledge of key concepts important to operability determinations, such as reasonable expectation, mounting tide of evidence, and indeterminate state. As a result, personnel in some cases did not communicate in a timely manner, did not communicate the information needed or did not ask questions to assure that adequate communications had occurred.

2)

Poor anticipation of system and human interaction during procedure and document development also resulted in an ineffective engineering to operations interface.

3)

Implementation of the Operability Determination procedure was inconsistent.

Corrective Actions Taken

1)

The Operability Determination (OD) and Functional Assessment procedure has been revised. The improvements to this procedure include more formal guidance for engineering input and timeliness. A discovery checklist has been developed for use as a tool for problem identification and proper characterization. This revision also incorporated NRC RIS 2005-020, Revision to Guidance Formerly Contained in NRC Generic Letter 91-18, "Information to Licensees Regarding Two NRC Inspection Manual Sections on Resolution of Degraded and Non-Conforming Conditions and on Operability." Initial overview training has been performed on the revised procedure and the use of the discovery checklist.

2)

Site wide procedure adherence training was completed in September 2005 that specifically covered issues with adherence to the corrective action and operability determination processes.

3)

A "Recirculation Actuation Sump (RAS) Case Study" training course was developed and provided to Operations and Engineering personnel. The 7

course objectives included a focus on the missed opportunities for discovering the unanalyzed condition, the need to have a questioning attitude, and use of the Qualification, Validation, and Verification (QV&V) process as a prevent event tool.

4)

The station electronic corrective action work flow was modified to automatically route condition report documents (when a control room block is checked) to the control room. The electronic work flow was also revised to allow the routing of Engineering Deficiency (ENG-DF) work orders to the control room staff as appropriate for the opportunity to perform and document Operability Determinations.

5)

An assessment, with industry participation, of the OD process was completed in February 2006 which identified additional enhancements to the process. These improvements have been incorporated into the OD procedure.

6)

A set of Engineering Department Guidelines (EDGs) have been developed to promote a graded approach to engineering rigor. These EDGs provide guidance on the appropriate degree of rigor to be applied based on the risk significance of the issue or engineering product. These documents are currently being used as reference material during engineering pre-job briefs.

7)

Monthly Operations/Engineering interface meetings have been established. The purpose of these meetings is to allow the Directors and Department Leaders from both organizations to discuss issues and focus attention on priorities, allocate resources as warranted, and promote open communications of concerns and issues for management resolution.

8)

Additionally, table top exercises for issues requiring ODs have been conducted between engineering and operations to strengthen this process.

Actions to be Taken

1)

A self-assessment will be performed of the quality and thoroughness of Nuclear Engineering's developed inputs and evaluations used to support implementation of the operability determination process. This action is scheduled for completion in September 2006.

2)

Additional in depth training on conditions adverse to quality including the recently implemented NRC RIS 2005-20 and revised OD procedure will be provided to selected engineering personnel, STAs, and shift managers in May and June 2006.

8

Results Achieved to Date A t:tal of 7 ODs have been performed using the new discovery checklist. The immediate operability determination has in all cases been completed within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. The detail the discovery checklist has added to the process has provided the Shift Managers with better information for making decisions and has resulted in better teamwork between Operations and Engineering. Operations and Engineering personnel have seen improvement in working together as a team, communicating face to face, identifying issues, and bringing issues to the Operations team earlier in the process.

4. Inadequate Identification of Nonconforming Conditions Probable Causes

1)

Programs and processes designed to identify nonconforming conditions were not always effectively implemented.

2)

The SWMS work process associated with identification of a non-conforming condition was flawed. The Engineering Deficiency Work Order (ENG-DF) was not identified as a process that required the user to apply degraded or nonconforming condition codes.

3)

Implementation of the Corrective Action Program (CAP) at Palo Verde did not effectively address identification of equipment condition issues.

Although the program recognized the need to identify conditions adverse to quality, identification of equipment issues was not consistently done through the CAP.

Corrective Actions Taken

1)

Clarification was provided to the Work Control - Senior Reactor Operators (WC-SROs) of the correct application of the work order codes for degraded and non-conforming conditions. A review of open work orders was conducted and verified correct coding was applied. The effectiveness of this corrective action is being monitored by an on-going review of work orders to ensure the work order coding is consistently and correctly applied. This review is conducted twice each month by the Operations work control SROs.

2)

The SWMS definitions for Degraded and Non-conforming conditions were corrected to ensure that the descriptions were consistent with those provided in the station procedures. Additionally, procedures were revised to provide consistent alignment and ensure compliance with RIS 2005-20.

9

3)

A revision to the SWMS design was completed and currently requires the use of the degraded and nonconforming condition coding to ENG-DF work orders. This process change was discussed with the Work Control (WC)-

SROs to ensure proper application of the codes and understanding that the ENG-DF work orders were within the Corrective Action Program.

4)

The overall scope of the Palo Verde Corrective Action Program was validated to ensure compliance with regulatory requirements. A matrix was developed to document the relationship between these regulatory requirements and the implementing procedures. The validation identified El number of areas needing enhancements which have been entered into the corrective action program for resolution.

5)

Training was provided to leaders on their responsibilities in the corrective action program to ensure problems were fixed including identification, classification, timeliness and closures of actions.

6)

A performance indicator and goal has been developed to monitor self-initiation of condition reports and the goals have been tied to the 2006 year-end employee incentive program.

7)

A performance indicator has been established to identify and fix at least two trends per month from the corrective action program.

Actions to be Taken

1)

Provide additional SWMS codes for the definition of material conditions that don't meet the NRC RIS 2005-20 criteria for degraded and nonconforming conditions. This action is scheduled to be completed in July 2006.

2)

A program level procedure is being written and will be issued to clearly define the relationship of CAP processes including those used to resolve equipment related issues. This procedure will define roles and responsibilities for program implementation. This procedure will be issued in the 2nd quarter 2006.

3)

Following the implementation of the above CAP procedure, the individual processes will be revised to be aligned to the new CAP procedure. This action will be completed in the 2nd quarter 2006.

10

Results Achieved to Date The ongoing review of work orders, performed twice monthly, indicates that improvements in the work order coding accuracy are being achieved. The need for any subsequent corrective actions to address the proper coding of work orders will be determined from the results of the effectiveness monitoring.

Site performance indicators show improved leader involvement during condition report initiation. This is evident by the following:

Site meeting goals of Condition Reports being self-identified or owners being contacted and accepting the condition report at initiation (for February 72%) which is an indicator of being self-critical, demonstrating ownership of the corrective action program, and improving timeliness and quality of fixing problems; and Adverse Condition Reports being issued closed or completed indicating a focus on promptly fixing problems (for February 38%).

There was an increase in condition reports initiated in 2005 compared to 2004. Nearly 1200 more condition reports were initiated in 2005.

5. Inadequate Evaluation of Conditions Adverse to Quality by Operations and Engineering personnel Probable Causes

1)

Critical information and key messages were not consistently or completely communicated in formal Condition Report evaluations and problem resolutions. Engineering evaluations sometimes lacked rigor and quality and evaluations had a narrow focus on problem evaluation and extent of condition.

2)

The timeliness of notifying the Operations control room staff of conditions adverse to quality was inconsistent.

3)

The conduct of Special Evaluations impacted the timely classification and assignment of Condition Reports by the Condition Report Review Committee (CRC).

11

Corrective Actions Taken

1)

Training was provided to site leaders on their responsibilities (including identification, classification, timely reviews, and closures) to improve the quality and content of Condition Report evaluations.

2)

A "RAS Case Study" was developed and provided to Operations and Engineering personnel. The course objectives focused on the missed opportunities for discovering unanalyzed conditions, the need to have a questioning attitude, and use of the Qualification, Validation, and Verification process as a prevent event tool.

3)

The station electronic corrective action work flow was modified to automatically route condition report documents (when a control room block is checked) to the control room. The electronic work flow was also revised to allow the routing of ENG-DF work orders to the control room staff as appropriate for the opportunity to perform and document Operability Determinations.

4)

Changes in the Condition Report Review Committee (CRC) role were implemented that provided additional focus to ensure Condition Reports were appropriately evaluated by control room personnel. These actions, along with management expectations, were included in the site's procedure for the Condition Report process.

5)

The Condition Report procedure changes and discovery checklist provided improved guidance for notification of the control room for degraded or nonconforming conditions that may impact safety function.

The Condition Reporting procedure was also revised to provide an improved definition for Degraded and Nonconforming Conditions and the associated required actions.

6)

The Operability Determination (OD) and Functional Assessment procedure has been revised. The improvements to this procedure include more formal guidance for engineering input, timeliness, and include a discovery checklist for use as a tool for problem identification and proper characterization. This revision also incorporated NRC RIS 2005-020.

Initial training has been performed on the revised procedure and the use of the discovery checklist.

7)

Continued monitoring and evaluation of the implemented actions is ongoing to develop trending information and subsequent additional corrective actions, if appropriate.

12

Actions to be Taken

1)

Training course "Response to Conditions Adverse to Quality" will be provided to engineering personnel and Shift Technical Advisors (STAs) in May 2006.

2)

Changes to the Condition Reporting process will be implemented to improve the initial classification timeliness of Condition Reports. This is scheduled to be completed by the end of April 2006.

Results Achieved to Date Site Monthly Trend Report (MTR) goals for 2006 were implemented to focus on timeliness and leadership involvement with issue resolution during Condition Report initiation. The 2006 MTR performance indicators are demonstrating the effectiveness of the training provided and the timeliness is improving. Site performance indicators show improved leader involvement during condition report initiation. This is evident by the following:

Site meeting goals of Condition Reports being self-identified or owners being contacted and accepting the condition report at initiation (for February 72%) which is an indicator of being self-critical, demonstrating ownership of the corrective action program, and improving timeliness and quality of fixing problems; and Adverse Condition Reports being issued closed or completed indicating a focus on promptly fixing problems (for February 38%).

The Condition Report backlog reduction plan has effectively reduced the total number of pre-2006 Condition Report evaluations that are over the site goal by over 50%. The number of evaluations over 30 days old for 2006 condition reports is under the goal of 10 or less for January and February.

A monthly comprehensive CAP Health Report was developed to measure and provide management feedback on multiple aspects of the Condition Reporting process. As part of health monitoring, the Performance Improvement Team performs Condition Report quality reviews on a sample of items closed during the previous month. Based on initial data, there appears to be an improving trend but there has been insufficient run time to make any definitive conclusions.

13

6. Inadequate and Ineffective Corrective Actions Probable Causes

1)

Management failed to hold themselves and others to high standards relative to the successful resolution of Conditions Adverse to Quality and correcting Degraded and Nonconforming Conditions. Palo Verde staff did not effectively use the CAP as a means for continuous improvements and were not held accountable for effective problem resolution. Individual and separate implementing processes resulted in weak management ownership and oversight of the CAP program. The Corrective Action Review Board (CARB) did not provide active oversight of this process.

2)

There was an inconsistent application of Engineering (ENG) work order types by the Engineering department personnel. Some degraded and nonconforming conditions were identified using the ENG Modification work order (ENG-DM) process instead of the ENG Deficiency Process ENG-DF. When the appropriate ENG-DF process was used, the SWMS work flow design did not route them to the control room staff for the opportunity to perform and document Operability Determinations.

3)

Some Work Order Corrective Actions in the Condition Report process were inappropriately milestoned, leading to untimely corrective actions and a reduction in management oversight and prioritization.

4)

Work orders that identify degraded and non conforming conditions were inappropriately canceled without correction of the condition.

5)

A significant backlog of Condition Report evaluations and corrective actions burdened the Palo Verde staff, making recognition and timely response difficult to emergent Conditions Adverse to Quality.

Corrective Actions Taken

1)

The CARB charter was revised to improve Senior Management involvement and CAP oversight. The CARB is now chaired by a site General Manager and the meeting frequency has been increased from the previously scheduled monthly meeting to daily meetings. These changes to the CARB have been implemented to provide timely feedback for significant Condition Report evaluations and Condition Report classification and assignment activities.

2)

A Performance Improvement Department was established to focus the station on actions necessary to return to excellence. Ownership for the 14

cancellations. Procedure changes were implemented to prohibit closing Condition Reports with any corrective actions incomplete.

8) Trend codes were developed to enable monitoring of Condition Reports resulting from a lack of questioning attitude/technical rigor or failures to adequately implement the Qualification, Validation, and Verification process. This data will provide the Engineering and Operations departments with measurements of effectiveness for the actions taken.

These trend codes have just recently been set up and data collection is in progress.

Actions to be Taken

1)

Implementation of a quality review process for the remaining portions of the Corrective Action Program is in progress. These actions are scheduled to be completed in May 2006.

2)

Complete alignment of work order degraded and non conforming condition coding and work order backlog identification between Operations and Work Management with procedural revisions and training as appropriate is in progress. These actions are scheduled to be completed in September 2006.

3)

PVNGS will revise the process for milestoned Work Order Corrective Actions. The CAP Health Report will include metrics for increased visibility and management attention to milestoned Condition Reports and Corrective Actions. The scheduled completion date is July 2006.

4)

Expectations for problem statements, evaluations and corrective actions will be developed and proceduralized. This action will be completed in June 2006.

5)

Continued monitoring and evaluation of the implemented actions is ongoing to develop trending information and subsequent additional corrective actions, as necessary.

Results Achieved to Date The Condition Report backlog reduction plan has effectively reduced the total number of pre-2006 Condition Report evaluations that are over the site goal by over 50%.

A monthly comprehensive CAP Health Report was developed to measure and provide management feedback on multiple aspects of the Condition Reporting 16

process. As part of health monitoring, the Performance Improvement Team performs Condition Report quality reviews on a sample of items closed during the previous month. Based on initial data, there appears to be an improving trend bul there has been insufficient run time to make any definitive conclusions.

17