IR 05000528/2007007: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(StriderTol Bot change)
Line 1: Line 1:
{{Adams
#REDIRECT [[IR 05000354/1985027]]
| number = ML071170509
| issue date = 04/24/2007
| title = IR 05000528-07-007, 05000529-07-007, 05000530-07-007, on 01/22/2007 - 03/20/2007, Palo Verde, Units 1, 2, and 3, NRC Problem Identification and Resolution
| author name = Smith L
| author affiliation = NRC/RGN-IV/DRS
| addressee name = Edington R
| addressee affiliation = Arizona Public Service Co
| docket = 05000528, 05000529, 05000530
| license number = NPF-041, NPF-051, NPF-074
| contact person =
| document report number = IR-07-007
| document type = Inspection Report, Letter
| page count = 42
}}
 
{{IR-Nav| site = 05000528 | year = 2007 | report number = 007 }}
 
=Text=
{{#Wiki_filter:ril 24, 2007
 
==SUBJECT:==
PALO VERDE, NUCLEAR GENERATING STATION UNITS 1, 2 AND 3 - NRC PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT 05000528/2007007; 05000529/2007007; 05000530/2007007
 
==Dear Mr. Edington:==
On March 20, 2007, the U S Nuclear Regulatory Commission (NRC) completed a team inspection at the Palo Verde Nuclear Generating Station, Units 1, 2, and 3. The enclosed report documents the inspection findings, which were discussed in the final exit meeting on [[Exit meeting date::March 20, 2007]], with Mr. B. Bement, Vice President, Nuclear Operations, and Mr. D. Mims, Vice President, Regulatory Affairs and Performance Improvement, and other members of your staff.
 
This inspection was an examination of activities conducted under your license as they relate to the identification and resolution of problems, and compliance with the Commission's rules and regulations and the conditions of your operating license. Within these areas, the inspection involved examination of selected procedures and representative records, observations of activities, and interviews with personnel.
 
The team reviewed approximately 190 action requests and work orders, associated root and apparent cause evaluations, and other supporting documents. The team reviewed cross-cutting aspects of NRC and licensee-identified findings and interviewed personnel regarding the condition of a safety conscious work environment at the Palo Verde Nuclear Generating Station.
 
You have made numerous changes to the corrective action program and some improvement was evident, but some of the changes were not yet fully effective. Greater management involvement is improving the consistency of how you address problems, but it is also creating a key bottleneck that delays fixing problems. Problems involving operability questions were getting to the control room more consistently, but examples where the operability implications of problems were not being recognized continued to be identified by NRC inspectors. Problems continue to exist in the quality of problem descriptions, significance determinations, and technical rigor of evaluations. The timeliness of cause evaluations were improving slowly, but were still several times longer than station goals and industry standards.
 
Arizona Public Service Company  -2-Palo Verde continues to have an large number of latent equipment issues. Longstanding equipment material condition problems exist which have received limited assessments and get added to the backlog with routine priority. The NRC continues to identify examples where the significance was underestimated by your staff and were not being addressed with the timeliness commensurate with the actual safety significance until the NRC became involved.
 
The team noted that significant challenges have been created because there are large backlogs of work affecting work control, maintenance support, and a variety of engineering activities. These backlogs are affecting the sites ability to address problems in a timely manner. It is apparent that these backlogs have built up over a period of years with the knowledge of management.
 
Interviews with site workers indicated that a safety-conscious work environment exists at Palo Verde, and that workers had an improved confidence in the strength of the safety culture.
 
However, there was less confidence that routine priority issues will get addressed in a timely manner.
 
On the basis of the sample selected for review, there were two findings of very low safety significance. One of these involved a violation of NRC requirements. The team concluded that problems were properly identified, evaluated, and resolved within the problem identification and resolution programs. The violation is being treated as a noncited violation because it was of very low safety significance and was entered in your corrective action program consistent with Section VI.A of the Enforcement Policy. If you contest the violation or the significance of the violation, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U. S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001, with copies to the Regional Administrator, U. S.
 
Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas, 76011; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Palo Verde Nuclear Station.
 
In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection. In the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web-site at http://www.nrc.gov/NRC/ADAMS/Index.html (the Public Electronic Reading Room).
 
Sincerely,
/RA/
Linda J. Smith, Chief Engineering Branch 2 Division of Reactor Safety Docket Nos.: 50-528, 50-529, 50-530 License Nos.: NPF-41, NPF-51, NPF-74
 
Arizona Public Service Company -3-
 
===Enclosure:===
Inspection Report 05000528, 05000529, 0500530/2007007 w/Attachment: Supplemental Information
 
REGION IV==
Dockets No.: 50-528, 50-529, 50-530 License No.: NPF-41, NPF-51, NPF-74 Report No.: 05000528/2007007; 0500529/2007007; 0500530/2007007 Licensee: Arizona Public Service Company Facility: Palo Verde, Nuclear Generating Station Units 1, 2,and 3 Location: 5951 S. Wintersburg Rd.
 
Tonopah, Arizona Dates: January 22 through March 20, 2007 Inspectors: N. O'Keefe, Senior Reactor Inspector (Team Leader)
J. Melfi, Resident Inspector J. Drake, Operations Engineer S. Alferink, Reactor Inspector G. Werner, Senior Project Engineer Accompanied By: B. Correll, Reactor Inspector Approved By: Linda J. Smith, Chief Engineering Branch 2 Division of Reactor Safety-1-  Enclosure
 
=SUMMARY=
OF ISSUES
 
IR 05000528; 05000529; 05000530/2007007; 01/22/2007 - 03/20/2007; Palo Verde, Nuclear Generating Station Units 1, 2, and 3; Identification and Resolution of Problems; Operability Evaluations, Maintenance Effectiveness.
 
The inspection was conducted by four region based inspectors and one resident inspector.
 
Two findings were identified during this inspection, one of which was a violation. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG1649, Reactor Oversight Process, Revision 3, dated July 2000.
 
Identification and Resolution of Problems The team concluded that the thresholds for identifying and classifying issues were appropriately low, although several instances were identified where new aspects to complex problems were identified but not broken out and addressed properly.
 
Numerous changes were made to the corrective action program and some improvement was evident, but some of the changes were not yet fully effective. The new Palo Verde Action Request was introduced, and senior managers were assigned to determine which actions were required in order to improve the consistency of problem treatment.
 
Problems involving operability questions were getting to control room operators more consistently, but NRC inspectors continued to identify operability concerns that were missed by the licensee. However, having the Action Request Review Committee review all problem reports created a bottleneck in the process, creating delays in getting problems from the identification to a working stage. Problems continue to exist in the quality of problem description and significance determination. The timeliness of problem cause evaluations were improving due to management attention, but were still several times longer than station goals and industry standards.
 
Palo Verde Nuclear Generating Station continued to have a large number of latent equipment issues. Numerous longstanding material conditions exist which have received limited assessments and get added to the backlog with routine priority. The NRC continued to identify examples where the significance was underestimated by the licensee and were not being addressed with the timeliness commensurate with the actual safety significance until the NRC gets involved.
 
The team noted that significant challenges have been created because there are large backlogs of work affecting work control, maintenance support, and a variety of engineering activities. These backlogs are affecting the sites ability to address problems in a timely manner. It is apparent that these backlogs have built up over a period of years with the knowledge of management.
 
The Nuclear Assurance Department was active in the internal oversight role and focused on current performance problems, issuing reports that provided useful assessments. Other self-assessments reviewed were frequently narrow in scope and of limited depth. Interviews with site workers indicated that a safety-conscious work environment exists at Palo Verde Nuclear Generating Station, and that workers had an improved confidence in the strength of the safety culture. However, there was less confidence that routine priority issues will get addressed in a timely manner.
 
===NRC-Identified===
and Self-revealing Findings
 
===Cornerstone: Mitigating Systems===
: '''Green.'''
A noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions,
Procedures and Drawings," with two examples was identified for two inadequate operability evaluations. Prompt operability determinations in CRDRs 2941494 and 2303499 incorrectly concluded that High Pressure Safety Injection Pumps 2A and 3A, respectively, could meet their mission time with existing oil leakage from the bearings.
 
The team concluded that these evaluations relied upon unverified and incorrect assumptions and non-conservative volumes. The apparent cause evaluation for the leakage identified contributing causes that were common to all pumps, but the operability of the other pumps was not assessed. The team identified a history of small oil leaks in high pressure safety injection pumps since 2000, but the licensee was unaware of this trend. Subsequent testing confirmed that five of the six high pressure safety injection pumps had oil leakage which would not allow running those pumps for the full mission time, but sufficient oil was available to run for at least 94 days. This finding was determined to have cross-cutting aspects in the human performance area of decision-making, because the licensee did not use conservative assumptions and demonstrate that the proposed course of action was safe.
 
Failure to adequately evaluate and correct oil leakage in High Pressure Safety Injection Pumps 2A and 3A, and failure to assess the extent of condition for similar pumps, was a performance deficiency. The finding was more than minor because it affected the equipment performance attribute of the mitigating systems cornerstone objective of ensuring the availability and reliability of a system that responds to initiating events.
 
This finding screened as Green during Phase 1 of the significance determination process because it did not involve a loss of safety function. This issue was entered into the corrective action program under Condition Report/Disposition Report 2973682.
 
      (Section 4OA2.e.1)
: '''Green.'''
A finding was identified for failure to schedule and perform preventive maintenance tasks that were in the preventive maintenance change process. The team identified that a backlog of over 2500 preventive maintenance changes existed which resulted in these preventive maintenance tasks not being scheduled or performed, potentially challenging completion within the specified frequency. The team found 438 examples of preventive maintenance tasks that were overdue, and an additional 2113 that had no due date assigned yet. This program was used to revise both safety-related and non-safety preventive maintenance tasks. Because these preventive maintenance tasks were in the change process, the tasks were not scheduled or tracked in a way that would show when they became overdue. This was contrary to Procedure 30DP-9MP08, Preventive Maintenance Program, Revision 17, which required that no preventive maintenance on operational equipment shall pass that late date without an approved deferral which will address a technical justification for the identified issue. This finding had human performance cross-cutting aspects associated with resources because the large backlog of preventive maintenance tasks was contrary to maintaining long-term equipment reliability.
 
Failure to track, schedule, and perform preventive maintenance activities within their specified frequencies in accordance with their preventive maintenance program was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, it could become a more significant safety concern in that the lack of preventive maintenance would affect the reliability of plant equipment which could impact the initiating events or mitigating systems cornerstones. Because of the large number of preventive maintenance tasks (over 2500) in this category, the team reviewed a sample of 79 tasks associated with safety-related or quality-class components to assess the significance. The team did not identify any examples of overdue safety-related tasks. Based on the lack of risk significant examples and the fact that this finding is not suitable for significance determination process evaluation, this issue was reviewed by NRC management and was determined to be a finding of very low safety significance. This issue was entered into the corrective action program under Palo Verde Action Request 2970076. (Section 4OA2.e.2)
 
=REPORT DETAILS=
 
==OTHER ACTIVITIES (OA)==
{{a|4OA2}}
==4OA2 Identification and Resolution of Problems==
 
The team based the following conclusions, in part, on all issues that were identified in the assessment period, which ranged from January 1, 2006, to the end of the inspection on March 20, 2007. The issues are divided into two groups. Current Issues includes problems identified during the assessment period where at least one performance deficiency occurred during the assessment period. Historical Issues includes issues that were identified during the assessment period where all the performance deficiencies occurred outside the assessment period.
 
a.
 
ASSESSMENT OF CORRECTIVE ACTION PROGRAM EFFECTIVENESS
: (1) Inspection Scope The team reviewed items selected across the seven cornerstones to determine if problems were being properly identified, characterized, and entered into the corrective action program (CAP) for evaluation and resolution. The team performed field walkdowns of selected systems and equipment to inspect for deficiencies that should have been entered in the CAP. The team also observed control room operations and reviewed operator logs and station work orders to ensure conditions adverse to quality were being entered into the CAP. Additionally, the team reviewed a sample of self assessments, trending reports, system health reports, and various other documents related to the CAP.
 
The team interviewed station personnel, attended Action Request Review Committee and Corrective Action Review Board meetings, and evaluated corrective action documentation to determine the licensees threshold for entering problems in their CAP.
 
In addition, the team reviewed the licensees evaluation of selected industry operating experience information, including operator event reports, NRC generic letters and information notices, and generic vendor notifications to ensure that issues applicable to Palo Verde Nuclear Generating Station were appropriately addressed.
: (2) Assessment
: (a) Assessment - Effectiveness of Problem Identification The team reviewed the numerous changes that have been made to the CAP. The team found that the source of the changes were both corrective actions from NRC and self-assessment findings, as well as from benchmarking other sites. The majority of these changes appeared appropriate and were implemented as intended. In general, the process changes were effectively addressing problem areas, although it was too soon to assess the effectiveness of changes in improving the quality of many of the products of the CAP.
 
In December 2006, the licensee initiated a single entry point corrective action process called the Palo Verde Action Request (PVAR). All PVARs were routed for an operability impact review by licensed operators, then to the Action Request Review Committee (ARRC) for disposition. These changes allowed for standardization of assignments because senior supervisory personnel on the ARRC made the assignments. The changes also reduced the time it took for some problems to get an operability review and ensured that possible operability issues did not bypass licensed operators.
 
The new PVAR process included some improvements, such as requiring equipment identification numbers to be included, but also had some problems to be resolved. The latter included not being able to print hard copies of PVARs. Also, the PVAR was administratively closed as soon as ARRC made classifications and assignments, which made it cumbersome to later change the significance if additional information was discovered.
 
Since the PVAR process had been in use for less than two months when the inspection began, the team was not able to determine its effectiveness. However discussions with various licensee employees indicated the new PVAR system was more user friendly.
 
The team also noted that the licensee has three procedures in place that describe various portions of the licensees CAP. The team found this to be confusing and found that some portions of the three CAP procedures appear to somewhat contradict each other. For example, Procedure 01DP-0AP10, Corrective Action Program, Step 1.2.1 stated, This procedure does not apply to personnel or processes at the Water Reclamation Facility (WRF). This was contrary to Procedures 90DP-0IP10, Condition Reporting, and 01DP-0AP12, Palo Verde Action Request Processing.
 
The team noted an overall improvement in problem identification. A steady increase in number of problems entered into the CAP annually was apparent over the last three years. However, there were still some specific problem areas in problem reporting. The team noted and previous NRC inspection reports documented, numerous instances where new aspects to problems already being addressed did not always get broken out and reported to management in the CAP process so they can get addressed specifically. Also, there were examples where the licensee failed to identify the extent of condition repeat problems.
 
The following examples of inadequate problem identification were identified during this review period:
* Noncited Violation 2006004-005 documented the failure to identify degraded material conditions affecting emergency diesel generators (EDGs) and enter them into the corrective action program. (Condition Report/Disposition Requests (CRDR) 2914886) (Current Issue)
* Violation 2006012-02 documented the failure to identify and correct the cause of erratic relay performance in Emergency Diesel Generator (EDG) 3A, resulting in and unreliable EDG. (CRDR 2926830) (Current Issue)
* Noncited Violation 2006011-06 documented two examples of failure to translate spray pond design assumptions into plant procedures. The licensee calculated the peak spray pond temperature during an accident without accounting for the impact of sediment and sludge on the pond bottoms. Approximately 4 inches of sediment and sludge were present in each pond at the time of the violation. The licensee was aware of the sediment buildup, but failed to enter this condition into the corrective action program or formally evaluate or track it. Also, the water available to cool safety related components during a design basis accident did not account for any leakage from the pond. The team identified that the licensee failed to inspect almost the entire portion of the ponds that contained water because they considered those surfaces inaccessible. (CRDRs 2906671 and 2910912) (Current Issue)
* Noncited Violation 2006002-02 documented the failure to provide required training to emergency response personnel. There was a prior opportunity to have identified this problem, but the evaluation in CRDR 2667913 was inadequate.
 
      (Current Issue)
* The licensee waived performance of Preventive Maintenance (PM) task 029620 for the security diesel generator startup battery when they could not verify the basis for the acceptance criteria. The team noted that the due date for the engineering evaluation (Condition Report Action Item 2953779) to verify the PM acceptance criteria was extended beyond the revised expiration date for this PM.
 
The licensee was aware of this, but they did not write a PVAR to document it.
 
      (Current Issue)
The team assessed the effectiveness of the licensees identification of material condition of important plant systems. The licensee was conducting formal team walkdowns of the top twelve risk-significant systems (approximately one per week). The team reviewed the results, including the independent walkdowns by Nuclear Assurance Department personnel, and found that they were identifying problems appropriately. The NRC team selected a sample of risk-significant systems which were not part of the licensees weekly walkdowns for inspection. These systems were walked down in company with the cognizant system or maintenance engineer, and any problems noted were checked to see if they were known and entered into the CAP.
 
Portions of the following systems were inspected:
* high pressure safety injection system
* gas turbine generators
* chemical and volume control
* safety-related battery systems
* containment spray system
* condensate system
* main turbine control oil system
* switchyard systems The team concluded that the material condition of the systems inspected was generally good. The system and maintenance engineers were knowledgeable of the condition of
 
their systems, and the deficiencies which were observed were mostly entered into the CAP. Some minor material condition problems were noted.
: (b) Assessment - Effectiveness of Prioritization and Evaluation of Issues Process changes to the corrective action program appeared to be improving the screening process so that new problems were entered into the appropriate portions of the CAP. The ARRC was observed to be standardizing the process. However, the team noted that guidance for assessing the significance of a problem was limited, unclear, and lacking some important considerations. The team also noted that the ARRC did not have an effective feedback mechanism for checking assumptions made when assigning actions and significance that are typical in industry. Effectively, the only mechanism available to the ARRC to gain information needed for making a decision was to place the PVAR on hold while the information was obtained, delaying the start of corrective actions.
 
The team observed several ARRC meetings and selected a sample of PVARs for more detailed review. The team noted that each day, some of the PVAR descriptions were inadequate for the ARRC members to be able to determine what the problem was or what was needed to address the problem. In discussions with ARRC members, it was apparent that this was an ongoing problem which was not being addressed in a systematic manner.
 
The ARRC classified each PVAR and assigned a significance. The team noted that this process was not clearly defined in the program, and may not have been yielding consistent results. In particular, discussion were observed where the ARRC made assumptions about the impact or extent of condition of a problem in assigning significance, but did not follow up to verify the assumptions were correct. Also, the repetitiveness of a problem was not part of assigning significance. These conditions were important considerations because cause evaluations were only assigned to the more significant problems; a repetitive failure might indicate that the previous corrective actions did not address the cause, so a cause evaluation would be needed.
 
The team concluded that requiring the ARRC to review all problems was creating a bottleneck in the process. The team was concerned that the administrative process for the ARRC review was cumbersome and caused delays. Because ARRC made action and owner assignments, only the highest priority work was allowed to be assigned prior to the ARRC completing its review. The cutoff for new PVARs going to ARRC was noon the previous work day. While this allowed ARRC members to spend the afternoon preparing for the next days meeting, it also meant that their meetings were not reviewing issues in a timely manner. This was particularly true for items identified over a weekend; from noon on a Friday until noon on the following Tuesday, all problems documented in a PVAR would have to wait until Wednesday morning to be dispositioned by the ARRC. The team considered this to be nontypical of industry practice and untimely. The team concluded that the focus on quantity was affecting quality, although the net effect was an improvement.
 
The Corrective Action Review Board (CARB), comprised of more senior managers than the ARRC, provided daily quality checks of the ARRC PVAR reviews. The team observed that the CARB was aggressive in setting a high standard and routinely returned PVARs to the ARRC for reconsideration of priority and classification.
 
The team noted that the Site Work Management System (SWMS), which was the software program for the CAP, was very segmented and difficult to use. In interviews, it was clear that site users took years to become proficient with its use, and constant use was necessary to maintain proficiency. Trending was not possible within SWMS, and had to be done by a separate program. This required manually loading information from the CRDR portion of SWMS into a data base called Trend-O-Matic. The team noted that a significant portion of equipment performance history resided in the work order and engineering evaluation sections of SWMS, but this information was not copied over to where it could be accessed by Trend-O-Matic. Therefore, the trending capability of the CAP was significantly limited. The team considered these limitations to be related to the ARRCs limited consideration of repetitiveness of problems, since the discussions involving repetitive failures were limited to examples where the ARRC members memories were the source of information.
 
The team reviewed the December 2006 Corrective Action and Human Performance Program Health Report. Of the 18 performance indicators, nine were Red (Significant Weakness), and one was Yellow (Improvement Needed). Of particular note was that for 2006, the licensee performed 46 root cause evaluations and 315 apparent cause evaluations. The team reviewed the licensees classification of these issues and found them to be appropriately classified. However, this large number of issues significant enough to require root cause or apparent cause evaluations (including engineering failure analyses) was challenging the technical and supervisory personnel who performed and reviewed them. These numbers were significantly higher than was typical at other sites.
 
The licensee discontinued its previous practice of performing cause evaluations for problems classified as routine in order to free up resources for evaluating more significant problems. These problems were now being treated as broke-fix. This practice was common in the nuclear industry. However, without a trending mechanism, this created the possibility of not being able to recognize the extent of condition for repeat or widespread problems.
 
Palo Verde Nuclear Generating Station continued to have an unusually large number of latent equipment issues. These typically involved longstanding conditions that had received a limited assessment, enough to reach a conclusion that the problem did not clearly involve a condition which caused equipment to be inoperable. Based on this limited assessment, these issues were added to the backlog and assigned routine-priority. These problems have typically not been addressed with the timeliness commensurate with the safety significance until the NRC got involved. The cumulative effects of these degraded conditions was not being assessed. Further, despite recent improvements in the quality of evaluations, these latent issues have not been effectively identified and reassessed using the improved processes. Examples of latent equipment issues identified during the review period included:
* Noncited Violation 2006011-07 documented multiple examples of inadequate operability assessments over a period of years involving degraded essential cooling water heat exchangers and emergency diesel generator heat exchangers.
 
Technical organizations were not always involving operations personnel with questions affecting operability. Current capability was compared to existing conditions rather than the more limiting design basis conditions. The impact of continued degradation was not considered in some cases. Some examples were outside the evaluation period, but significant examples occurred during this evaluation period. (CRDRs 2918892, 2901815 and 2898237) (Current Issue)
* Noncited Violation 2006003-05 documented the failure to assess the impact to safety related equipment for drain hose manifold boxes for years, which created a potential for flooding in the ECCS pump room sumps. (CRDRs 2918892, 2901815, and 2898237) (Current Issue)
* Cracks in the accessible portions of the concrete spray pond liners triggered the identification that a preventive maintenance requirement to clean and inspect the spray pond was never implemented. About 4 inches of sediment was known to exist on each of the spray pond bottoms. Informal, undocumented assessments caused the licensee to conclude that up to 18 inches of sediment could be acceptable before affecting operability, but this did not consider the effect displacing water would have on the peak pond temperature during a design basis accident. (CRDR 2910912) (Current Issue)
* The team identified that running High Pressure Safety Injection Pumps 2A and 3B caused high room temperature alarms because the motor fans blow hot air onto the temperature sensors. Though known to exist for a long time, this issue was documented recently in PVAR 2964632, which suggested an enhancement to move the sensor. The licensee had lived with the problem because they had evaluated bulk room temperature as reaching about 101 degrees F, which was well below the limit of 120 degrees F to maintain pump operability. The team concluded operators would be unable to identify the unexpected condition once the alarm setpoint was reached. (Current Issue)
* The team reviewed air flow problems identified in the essential ventilation system for each EDG train. The licensee identified that there was no periodic testing to verify sufficient air flow to cool the engines, but when testing was performed, air flow was considerably below the required minimum for design conditions in all trains. The focus of the licensees efforts were to justify a lower flow rate as acceptable, rather than to find and correct the cause of degraded fan performance. A conclusion was reached that the existing flow was enough, and little effort was put into determining the cause of the significant degradation. Low priority was being given to implementing the corrective actions to improve fan performance. (CRDR 2850999) (Current Issue)
The team noted that the licensee had been inappropriately prioritizing maintenance work orders. The guidance for prioritizing work orders was appropriate, but supervisors outside of the work control group routinely requested that high priority be given to specific jobs, contrary to the guidance. This sometimes had the effect of bumping
 
scheduled work with emergent work that had not been fully planned or without parts being available. Operations management initiated a review of all work assigned higher than routine priority in late November 2006, and downgraded about 60 percent of them out of the high priority category. Interviews indicated that many site personnel believed that individuals had to lobby to get work assigned as high priority because there were insufficient resources to get routine work done in a timely manner.
 
The team noted that significant challenges have been created because there are large backlogs of work affecting work control, maintenance support, and a variety of engineering activities. These backlogs are affecting the sites ability to address problems in a timely manner. There were large backlogs of work affecting work control, maintenance planning, and a variety of engineering activities.
 
The team attempted to identify and evaluate the impact of work backlogs. Significant backlogs existed in programs affecting work control, maintenance support, and a variety of engineering activities. Through interviews with workers and supervisors, it was apparent that these backlogs built up because there were insufficient resources to keep up with the workload, and that active efforts to prioritize the existing work resulted in the lower priority work building up. Further, senior management indicated that long-term workforce reductions through a policy of attrition had exasperated this condition. While hiring had recently begun, it appeared that the hiring levels were targeted at restoring pre-attrition levels, which would not effectively support reducing backlogs. The teams review indicated that most of the backlog contents were low priority work that has been there for years, though some of the examples were things that did need to be fixed. For example, a large number of work planning items exist with due dates in the year 2025.
 
This corresponded to the last year the plants were licensed to operate. Selecting a due date in this manner ensured that the work would not become overdue, effectively hiding these issues.
 
The licensee was in the process of adopting a new program for managing the lifecyle of plant equipment. Their initial focus was to reduce maintenance on nonkey components, or eliminate it entirely by classifying equipment as run to failure. The team noted that this involved a lot of work without a corresponding safety benefit. The team noted this conflict, and the licensee stated that many of the individuals performing these maintenance reviews were the same people needed to work off these backlogs.
 
The following examples illustrated impacts caused by failing to perform routine work in a timely manner:
* The team assessed corrective maintenance items that were to be cancelled. The licensee had previously had problems inappropriately cancelling work that needed to be done. The licensee committed to having a shift manager review and approve cancelling such work items. However, while a review and approval queue had been created, shift managers had not been performing these reviews.
 
Between November 2006 and the time of this inspection, a backlog of almost 2600 corrective maintenance items had built up, with the effective result that they were not being performed, which was no different than cancelling them. Thus, the team concluded that this corrective action was ineffective. Palo Verde Action Request 2960102 identified this issue. (Current Issue)
* The team identified three examples where the licensee had not completed the steps necessary to abandon plant equipment that was no longer needed. These included the containment spray system spray additive subsystem (DMWOs 217202 and 219236), an EDG room spray cooling system (DMWO 2315467), and the post-accident sampling system (DMWOs 2336963, 2529758 and 2529755). Enough of the work was done to make it safe to ignore these systems, but the rest of the work was assigned a low priority and added to the backlog without due dates. These examples were entered into the corrective action program under PVAR 2968631. (DMWOs 217102, 219236, 2315467, 2336963, 2529755, 2529758) (Current Issue)
* The team identified that drawing revisions were not made until some undefined number of approved changes had accumulated. Three examples of drawings that were not being maintained current were identified by the team and entered into the corrective action program under PVAR 2968805. The changes in these examples involved at least five approved changes per drawing, and most of the changes had been approved for 3 to 10 years. Also, the team identified that the temporary modification procedure did not require promptly updating control room drawings. Instead, there was a proceduralized workaround requiring users to check for approved changes before relying on a drawing. The team concluded that this was impractical, particularly if more than one change was involved.
 
      (Current Issue)
* The team reviewed the impact of the licensees commitment to perform reviews of work scope library (WSL) documents prior to use. This was corrective action for having discontinued periodic reviews. The team concluded that these reviews were mostly getting done before they are needed, although several examples of the late review impacting the scheduled work date were identified. The team found a few examples where work was done in the grace period because of a WSL review. (Current Issue)
Operability Assessments The team evaluated the elements of the CAP that contributed to identifying the need for operability assessments, and how those assessments were performed and approved.
 
The team reviewed completed examples of immediate operability determinations (IODs)and prompt operability determinations (PODs). The team interviewed personnel who performed screening and approval of operability issues (shift managers, shift technical advisors, and work control senior reactor operators). These operations personnel indicated that changes to the process were mostly positive, although confusion was introduced in some key areas. The team noted that the licensee required operations personnel to make an initial determination whether a PVAR involved a degraded or non-conforming condition. These words have specific meaning in NRC operability assessment guidance, but the licensee was using them to trigger actions in both the operability assessment and work control programs. This resulted in the terms having broader meanings than intended in the operability assessment process and confusion in some cases among those assigned responsibility to make those determinations. The team noted that making a determination of whether a system, structure, or
 
component (SSC) was degraded or nonconforming was normally one of the last steps of an operability determination because it required research to determine the requirements and commitments which the SSC was supposed to satisfy. Therefore, the team considered that it was potentially inappropriate and confusing to make this assessment before formally entering the operabilty determination process.
 
Problems involving operability questions were observed to be getting to the control room more consistently, but examples where the operability implications of a problem were not recognized continue to occur. Also, operators were becoming more active in questioning engineering positions, but the examples below indicate that this was still a problem.
 
The team noted that the licensee made improvements in the level of documentation used to support operability determinations. Also, the licensee recently dedicated several engineers to work with operations on operability determinations. These on-call engineers helped standardize the process and the finished product, and improved the ability to locate the necessary resources to perform PODs. However, the team noted that this change was mostly showing improvement for operability questions initiated from operations personnel; examples continued to exist where problems that potentially involved operability impacts were recognized in engineering but were not promptly brought to operations, delaying timely evaluations. Corrective actions to address untimely action by engineering in this area have not been effective.
 
The quality of operability assessments was somewhat improved, but remained inconsistent. Most significantly, the NRC continued to identify examples where the operability assessment was narrow and did not identify all of the impacted functions, limiting conditions, or extent of condition.
 
Examples of inadequate or untimely technical evaluations involving operability questions included:
* Violation 2006012-03 documented the two examples of failure to perform operability determinations associated with an EDG 3A relay. (CRDRs 2928389 and 2940558) (Current Issue)
* The team identified that the licensee had not adequately evaluated the ability of smaller, slower speed motors used as replacements for the Containment Spray Pump 3A and Low Pressure Safety Injection Pump 1A motors before installing them in the plant. When the NRC pointed out the inadequate evaluation for the containment spray pump motor, it took the licensee 6 months to identify that the same situation existed in another application, even though the motors were identical and purchased at the same time. (CRDRs 2870352 and 2932177)
        (Current Issue)
* In April 2006, CRDR 2820810 identified that HPSI Pumps 1A and 1B had unapproved bearings installed since October 2002, which was a nonconforming condition. Engineering was assigned actions to perform a parts equivalency evaluation, but the control room operators were not informed nor was an operability assessment performed until September 27, 2006. (CRDR 2928315)
 
  (Current Issue)
* An example of an inadequate technical evaluation was identified which incorrectly concluded that both trains of low pressure safety injection valves were not rendered inoperable when the injection header was over pressurized. The control room log was annotated to withdraw the Technical Specification 3.0.3 entry. The evaluation was based on engineering judgement which was not supported by the available facts. (CRDR 2892697) (Current Issue)
* Team members walking down the HPSI system identified that HPSI Suction Isolation Valve Actuator SIB-V402 in Unit 2 was not properly supported. This heavy valve was fastened to a deck plate, and when that deck plate was walked on, both the deck plate and the valve wobbled. The team identified this to the support team, but the seismic qualification was not promptly assessed. (Current Issue)
* The team questioned the impact of oil leakage observed in the HPSI Pump 2A (discussed in Section 4OA2.e.1) at the beginning of the inspection, but engineering did not raise this operability concern to operations until the evening before the exit meeting. Both the original and revised POD contained inappropriate assumptions that were not challenged until the team questioned them. Testing that was subsequently performed disproved these assumptions.
 
  (Current Issue)
* Upon the discovery that calculations did not account for any spray pond leakage, inspectors questioned the operability of the emergency spray ponds. This was compounded by the finding that the licensee was not performing adequate inspections of pond integrity. The licensee performed an inadequate operability determination based on structural integrity and lack of evidence of gross leakage, despite an operations assessment that indicates that only about 5 gpm leakage would be enough to render the ponds inoperable. (CRDR 2906671) (Current Issue)
* Noncited Violation 2006011-07 documented multiple examples of inadequate operability assessments involving degraded essential cooling water heat exchangers and emergency diesel generator heat exchangers. Key support organizations were not always involving operations personnel with questions affecting operability. Operability was compared to existing conditions rather than the more limiting design basis conditions. Further degradation was not considered in some cases. Some examples were outside the evaluation period, but significant examples occurred during this evaluation period. (CRDRs 2918892, 2901815 and 2898237) (Current Issue)
* Noncited Violation 2006003-05 documented the failure to assess the impact to safety related equipment for drain hose manifold boxes, which created a potential for flooding. The boxes allowed repeated wetting and degradation of sump level switches which were intended to start a sump pump and alert operators of an unusual amount of water in the ECCS pump room sumps. (CRDRs 2918892, 2901815, and 2898237) (Current Issue)
* Noncited Violation 2006003-06 documented two examples of failure to evaluate the impact of foreign material on operability of the shutdown cooling system.
 
        (CRDRs 2902258 and 2892737) (Current Issue)
* The team identified an example of an incomplete and untimely evaluation because the licensee performed only part of an evaluation for the use of ultra-low sulfur fuel in EDGs. Parts of the evaluation were based on limited information, which was not verified and updated in the operability assessment. The licensee deferred the evaluation of other diesels (fire pump, security diesel, gas turbine generator) until April 2007. This fuel has been in the system since June 2006.
 
        (CRDR 2928626) (Current Issue)
Root Cause and Apparent Cause Evaluations As discussed above, the number of cause evaluations is very high compared to typical site. The team concluded that root cause and apparent cause evaluations were being assigned appropriately, but the high number of issues seemed to be a consequence of having a large number of latent equipment issues and human performance issues. The high number of cause evaluations of all types continued to negatively impact the timeliness of these evaluations. Completion times for cause evaluations continued to significantly exceed the 30 day goal (averaging 100-120 days), although this was observed to be improving slowly.
 
The quality of cause evaluations was observed to be improving. Root cause evaluations were noted to improve significantly, although this was initially accompanied by reduced timeliness. In contrast, apparent cause evaluation quality was inconsistent. The team noted that the licensee had devoted most of the corrective actions to improving the quality of root cause evaluations, and that, unlike most plants, no training or qualifications were required for a person to perform an apparent cause evaluation.
 
The team noted that the licensees assessment of extent of condition for problems was inconsistent. It appeared that there was not clear guidance on when to evaluate extent of condition and how much effort was expected. Also, the limited ability to perform CAP trending impacted the quality of the result when it was performed.
 
In December 2006, completion of root cause evaluations, Engineering Root Cause Failure Analysis (ERCFA) Level 2 evaluations, and apparent cause evaluations required an average of 105, 123, and 137 days, respectively. As a comparison to the previous PI&R inspection conducted in January 2006, the average completion time for root, ERCFA Level 2, and apparent cause evaluations were 125, 172, and 77 days respectively. The licensee had a goal of 30 days for the evaluations to be completed.
 
The team concluded that the licensee continued to have significant challenges associated with timely completion of root and ERCFA Level 2 cause evaluations, which subsequently delayed corrective action implementation. In discussions with the CAP owner, many of the delays are associated with engineering because of the complexity of the issues and the inability to rapidly determine a root cause.
: (c) Assessment - Effectiveness of Corrective Actions
: (1) Inspection Scope The inspectors reviewed plant records, primarily CRDRs and work orders, to verify that corrective actions related to identified problems were developed and implemented, including corrective actions to address common cause or generic concerns.
 
Additionally, the inspectors reviewed a sample of CRDRs that addressed past NRC identified violations for each cornerstone to ensure that the corrective actions adequately addressed the issues as described in the inspection reports. The inspectors also reviewed a sample of corrective actions closed to other CRDRs, work orders, or tracking programs to ensure that corrective actions were still appropriate and timely.
: (2) Assessment The team concluded that corrective actions to address adverse conditions were generally effective. This was particularly true for the more significant problems, where there was increased management involvement. However, problems of a more routine nature did not always get effective corrective actions. The team identified a number of examples of untimely or ineffective corrective actions:
* An example of a nonrigorous cause evaluation and untimely corrective action was identified for degraded air flow in the EDG essential ventilation system. Air flow was considerably below the required minimum for design conditions in all trains.
 
Engineering focused on justifying a lower flow rate as acceptable, rather than to find and correct the cause of degraded fan performance. (Current Issue)
* An example of narrow corrective actions was identified for a prior violation for not assessing the impact to safety related equipment for drain hose manifold boxes, which created a potential for flooding. The licensee addressed drain manifolds only in the rooms described in the violation. (PVAR 2968089) (Current Issue)
* An example of ineffective corrective actions was identified for not correcting a problem with inappropriately closing corrective maintenance (CM) items. The licensee committed to having a shift manager review all CM items prior to closing.
 
However, 2595 work orders, half of them CMs, were building up in an electronic in-box since November 2006, and none had ever been reviewed. (PVAR 2960102) (Current Issue)
* Noncited Violation 2006003-03 documented inadequate corrective actions to preclude repetition of a significant condition adverse to quality when a submersible robot was sucked into a pump. (CRDR 2885213) (Current Issue)
* Violation 2006012-02 documented the failure to identify and correct the cause of erratic relay performance in EDG 3A, resulting in an unreliable EDG.
 
      (CRDR 2926830) (Current Issue)
* Noncited Violation 2006004-01 documented inadequate corrective action to preclude water intrusion and corrosion of underground piping. (CRDRs 2885972, 2880283, and 2902572) (Current Issue)
* Noncited Violation 2006011-03 documented the failure to assess changes to the chemistry control program intended to prevent fouling of heat exchangers cooled by the spray pond system. While the licensee identified that chemistry control program changes had been made without assessing the impact to safety per 10 CFR 50.59 in January 2006, the licensee had taken no action to assess the extent of condition or to correct the problem by the time the NRC identified this violation in July 2006. (CRDR 2902498) (Current Issue)
* Noncited Violation 2006011-04 documented inadequate corrective action for degraded heat exchanger performance over a period of years. Chemistry control changes starting in 1994 made progressive changes which resulted in the over-addition of chemicals which fouled heat exchanger tubes and system piping.
 
System performance test results and internal heat exchanger inspections documented slimy buildup and significantly degraded performance. However, the licensee failed to recognize this as a significant condition adverse to quality and failed to assess the cause until May 2006, when the NRC performed a special inspection. While portions of this performance deficiency occurred prior to this evaluation period, significant portions occurred during this period.
 
      (CRDR 2897810) (Current Issue)
* Condition Report/Disposition Report 2624427 was initiated in response to the pressurizer spray valve failure that resulted in a manual reactor trip on July 29, 2003. The NRC issued Noncited Violation 05000529/2004006-03 for procedure deficiencies revealed during the event. The licensee determined that one of the root causes was that the spray valve control design was not tolerant to single failure. The corrective action was to design and install a solenoid valve that would allow the operators to bleed off control air, shutting the valve. The inspectors noted that even though this corrective action had been designated as Priority 2 (high priority) with a due date of August 11, 2004, no design work had been started as of January 2007. In discussions, the team was told that these design changes were no longer considered necessary, but the reasons had not been documented and the actions had not been closed out. (Current Issue)
The licensee implemented improvements to increase the number and quality of effectiveness reviews for completed corrective actions. Also, Nuclear Assurance Department (NAD) personnel were performing routine samples of corrective action effectiveness. These efforts were identifying problems with corrective action effectiveness, although actions to address the findings generically were primarily limited to performing more and better effectiveness reviews,
 
The team reviewed the licensees efforts to reduce the number of PM deferrals. The licensee had previously been deferring significant numbers of PMs with low-level approval. The PM deferral process was changed in September 2006. The new process increased the level of management approval needed for a PM deferral and requires engineering involvement. The actions taken by the licensee to reduce the number of PM deferrals appear to be effective, as virtually no deferred PMs existed.
 
Section 4OA2.e.2 discusses PM changes which were not being tracked to ensure the changes were completed in time to support completing the maintenance within the intended PM frequency. In contrast, PM deferrals were a mechanism to approve not performing the PM.
 
b.
 
ASSESSMENT OF THE USE OF OPERATING EXPERIENCE (OE)
: (1) Inspection Scope The team examined the licensee's program for reviewing industry operating experience.
 
A number of operating experience notification documents (NRC Bulletins, Information Notices, Generic Letters, Part 21 reports, Licensee Event Reports, vendor notifications, etc.) that had been issued during the assessment period were selected to verify whether the licensee had appropriately evaluated the notification for relevance to the facility.
 
The team also examined whether the licensee had entered those items into their corrective action program. The team reviewed a sample of root cause evaluations and significant action requests to verify if the licensee had appropriately evaluated relevant industry operating experience.
: (2) Assessment During this evaluation period, the licensee implemented numerous enhancements to the Operating Experience Program, including: line manager review for selected OE evaluation reports; periodic effectiveness reviews for Significant Operating Experience Reports; training on the use of OE and available search tools; and improved guidance for evaluating operating experience documents. The team noted that the licensee had also been effectively expanding the use of operating experience in routine activities.
 
Both internal and external operating experience was being incorporated into lessons learned for training and pre-job briefs.
 
The team noted that root and apparent cause evaluations were increasingly evaluating whether internal or external operating experience was available associated with the event or failure being examined, and whether the evaluation and actions to address those items had been effective. Several recent root cause evaluations were effective in identifying relevant operating experience which had been ineffectively addressed. The team did not identify any additional examples.
 
The licensee uses Procedure 65DP-0QQ01, Industry Operating Experience Review, Revision 12 as guidance when evaluating industry experience. For the operating experience samples selected by the team, the inspectors found that the licensees review was satisfactory. The inspectors found that the licensee generally met the requirements for an industry review program. In May 2006, the licensee performed an operating experience program self assessment and identified areas that could be
 
improved. For example, the licensee identified an issue with timeliness of operating experience reviews. The inspectors noted that CRDRs 2953152, 2953751, and 2953766 identified that some industry experience was not reviewed in a timely manner.
 
Also, the people assigned to perform the technical evaluations were not trained or qualified to perform those roles, they are just designated to do all OE reviews for their work group.
 
c.
 
ASSESSMENT OF SELF-ASSESSMENTS AND AUDITS
: (1) Inspection Scope The inspectors reviewed a number of licensee self and independent assessments and audits to assess whether the licensee was regularly identifying performance trends and effectively addressing them. The team also evaluated the role of the NAD in their quality assurance function of internal oversight, particularly in relation to the licensees effort to improve station performance. The team reviewed a sample of audit and surveillance reports and interviewed auditors and supervisors in NAD. The specific self-assessment documents reviewed are listed in the Attachment.
: (2) Assessment The team evaluated the role of the NAD in their quality assurance function of internal oversight. This role had a heightened importance in the licensees effort to improve station performance. The team observed that NAD was active in an oversight role, and focused their attention to areas that have been identified as needing increased oversight and improvement. It was apparent that NAD had the support of station management, and had been receiving requests from line managers to assess specific areas. Nuclear Assurance Department maintained an active list of Station Quality Issues, a focus list of issues affecting quality that NAD wants station management to address. The team noted that the issues were each supported by a document describing the problems, examples, impacts, and need for corrective actions. These issues appeared to be getting appropriate attention from station management.
 
The team reviewed a sample of audits and surveillances performed by NAD. A sample of NAD personnel who contributed to those reports were also interviewed. Based on these samples, the team concluded that NAD personnel were being probing in their oversight roles, and effectively documented their findings. Their audit and surveillance reports were detailed, and provided good assessments. The standard of performance frequently involved external references, such as industry best practices. Findings were clearly stated and placed into context, and were linked to the source and the corrective action document addressing the issue.
 
Based on interviews with NAD personnel and personnel from other site organizations, it appeared that NAD had an effective working relationship with the station line organizations. In particular, NAD personnel interviewed all provided examples of recent requests from line managers to have NAD observe performance in certain focus areas.
 
The team noted that line organizations at Palo Verde Nuclear Generating Station were less effective in performing self-assessments. The team reviewed a sample of line
 
organization self-assessments. The topics in this sample tended to be very narrowly focused. Most of the reports sampled were compliance-based rather than comparing to industry best-practices or other external references, limiting the effectiveness of the assessment and the value to the organization.
 
d.
 
ASSESSMENT OF SAFETY CONSCIOUS WORK ENVIRONMENT
: (1) Inspection Scope The inspectors interviewed 35 individuals from different departments representing a cross section of functional organizations, including supervisory and nonsupervisory personnel. These interviews assessed whether conditions existed that would challenge the establishment of a safety conscious work environment. The inspectors also reviewed the results of the "2005 Nuclear Safety Culture Assessment" conducted by Synergy Consulting Services.
: (2) Assessment The inspectors concluded that a safety conscious work environment exists at the Palo Verde Nuclear Generating Station. Employees felt free to enter issues into the CAP, as well as raise safety concerns to their supervision, the Employee Concerns Program, and the NRC. Improvement was apparent from these interviews in some areas identified as concerns during the 2005 Nuclear Safety Culture Assessment. Individuals were familiar with the CAP, and had used the process to report and correct problems. Additionally, many interviewees believed changes to the CAP were improving the process, and supported the improvements.
 
Some of the interviewees expressed a concern with the timeliness of corrective actions.
 
For safety significant issues, there was confidence that the issue would be addressed.
 
However, for issues classified as routine priority, there was less confidence that those issues would be ultimately resolved because of lack of resources. Despite some hiring, many licensee personnel believed that backlogs of routine work exceeded the resources available.
 
e.
 
SPECIFIC ISSUES
 
===.1 Inadequate Technical Evaluation of HPSI Pump Bearing Oil Leaks===
 
====a. Inspection Scope====
The team reviewed the history of oil leaks from high pressure safety injection pump bearings and the licensees evaluations and corrective actions. In particular, the team interviewed engineers and operators involved in the operability evaluations for oil leaks from the HPSI Pump 2A in November, 2006. Pump and bearing design information, industry literature on design and lubrication of this type of bearings, and maintenance history documents were reviewed, as well as operability evaluations and apparent cause evaluations. When the licensee conducted testing to assess oil leakage from each HPSI pump, the team reviewed the test methods and results, as well as the scope of modifications to prevent leakage.
 
====b. Findings====
 
=====Introduction.=====
A noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures and Drawings," with two examples was identified for two inadequate operability evaluations. Prompt operability determinations in CRDRs 2941494 and 2303499 incorrectly concluded that HPSI Pumps 2A and 3A, respectively, could meet their mission time with existing oil leakage from the bearings.
 
The team concluded that these evaluations relied upon unverified and incorrect assumptions and non-conservative volumes. The apparent cause evaluation for the leakage identified contributing causes that were common to all pumps, but the operability of the other pumps were not assessed. The team identified a history of small oil leaks in HPSI pumps since 2000, but the licensee was unaware of this trend.
 
Subsequent testing confirmed that five of the six high pressure safety injection pumps had oil leakage which would not allow running those pumps for the full mission time of 180 days, but sufficient oil was available to run for at least 94 days.
 
=====Description.=====
The team evaluated the licensees response to oil leakage from the bearings following maintenance on HPSI Pump 2A on November 4, 2006. This leakage had exceeded criteria previously established for the maximum acceptable leak rate in a previous evaluation, as documented in CRDR 2303499.
 
Following pump maintenance, the licensee identified elevated vibration and bearing oil leakage on the outboard pump bearing. Over the next week, the licensee reworked the bearings several times. On November 12, 2006, vibration was considered acceptable, though still somewhat elevated. Oil leakage from the outboard bearing started about one hour after the pump was started, was measured at three drops/minute, and then stopped after 2-3 hours.
 
The team concluded that the licensees technical evaluation was inadequate to demonstrate operability for this mission time. Specifically, the evaluation concluded that oil level was being maintained too high inside the bearing housing, which was expected to cause leakage until oil level was reduced to the optimum level, and then leakage would completely stop. The team pointed out that the licensee did not verify that this theory was correct, or identify evidence to support the assumption that the oil remaining when leakage stopped was adequate to cool and lubricate the bearing during long term operation. Further, since the other five HPSI pumps onsite were set up essentially identically, and because normal HPSI pump runs were shorter than the time needed to see the leakage start, it was possible that other pumps had oil leakage that was not detected. The team also pointed out that the operability determination had established a maximum acceptable leak rate that was based on a nonconservatively high initial oil volume. Finally, the team identified that the apparent cause evaluation identified several contributing causes, but this condition was not classified as degraded or nonconforming, and no corrective actions were identified to address these problems.
 
The licensee concluded that the problem was a combination of the following problems:
* The static oil level setting in bearing housing was too high, allowing the ball bearing to splash more oil and increase heating and friction. This was undesirable with a slinger ring installed.
* The oilers were located on the non-preferred side of the bearing housing, so as to allow dynamic effects that further raised the oil level during pump starts.
* The actual oiler settings were higher than intended on five HPSI pumps, probably because the setup methodology introduced errors.
 
The team identified that the licensees apparent cause evaluation for leakage from HPSI Pump 2A in November 2006 had concluded that the oilers were located on the nonpreferred side of the bearings, allowing the rotation of the shaft to draw oil out of the oiler due to dynamic effects, particularly at pump start. However, the team noted that HPSI Pump 2A was the only HPSI pump with the oiler on the preferred side where dynamic effects were much smaller. Therefore, this element of the apparent cause was incorrect.
 
In response to the teams concerns, the licensee conducted testing of HPSI Pump 2A on February 16, 2007. The outboard bearing started leaking three drops/hour after about an hour, but stopped after about eight hours. However, the inboard bearing leaked at three drops/hour and the leak did not stop or slow. Adjusting the oiler to control at a lower level did not change the leakage. The licensee shut down Unit 2 on February 19, 2007, to comply with Technical Specification 3.5.3, Condition C, because they did not have confidence that they understood the cause of the leakage sufficiently to declare the pump operable.
 
In responding to the leakage from HPSI Pump 2A on February 16-19, 2007, the licensee modified the pump bearings to locate the oiler tap on the bottom in order to remove the dynamic effects. The single 4 oz oiler was replaced with a pair of 8 oz oilers, and the setpoint was lowered to just below the bearing race. This did not reduce the leak rate, so the flinger was adjusted to reduce the gap, which stopped the leakage.
 
Between February 19 and 27, 2007, testing was performed on each of the remaining HPSI pumps. The following summarizes the results:
Pump            Condition                                            Oiler Modified 1A              No leakage observed.
 
Scheduled for 3/07 1B              Inboard bearing leaked 6 drops/hour.
 
Yes 2A              Inboard bearing leaked 3 drops/hour.
 
Yes Outboard bearing leaked after 1 hour of run time and stopped after 3 hours.
 
2B              Inboard bearing leaked 3 drops/hour.
 
Yes Outboard bearing leaked after 1 hour of run time and stopped after 3 hours.
 
3A              Inboard bearing leaked 6 drops/hour.
 
Yes 3B              Inboard bearing leaked 2 drops/hour.
 
Yes
 
The licensee stated that they had no documentation to indicate they were previously aware of any bearing oil leakage on these pumps except the outboard bearing on the Pump 2A in November 2006. However, in reviewing CAP documents on HPSI pump history since 2000, the team determined that small bearing oil leaks on inboard and outboard bearing and incorrect oiler settings were mentioned in multiple maintenance documents but not always written into CRDRs. Other instances were attributed to overfilling of the bearing reservoir. For example, on September 8, 2005, CRDR 2831011 documented high vibration and high oil level in HPSI Pump 2B, with leakage at both ends of the inboard bearing. The oiler was found to be set 1/4 inch too high, and level was an additional 1/2 inch above that setpoint due to repeated starting and stopping of the pump during postmaintenance testing. This was an example of not recognizing the extent of condition of a problem.
 
The team also noted that the licensee had observed higher than normal vibrations in Pumps 2B (in Alert range since December 12, 2006) and 2A (in Alert range on November 4, 2006 and November 11, 2006, and still elevated but somewhat below Alert range since that time). The cause was not determined and no corrective actions were taken, although increased monitoring was performed. It was now believed this was also caused by high oil level.
 
In response to the team questioning the accuracy of the settings, the licensee identified that the maintenance procedure used to set the oilers introduced error, which had contributed to some of the settings being even higher than intended. This was corrected by scribing the correct level on the outside of the bearing housing.
 
Subsequent to pump testing, the licensee concluded that the six month mission time previously being applied to the HPSI pumps was incorrectly interpreted from their license basis. A 30 day period was actually the correct mission time. The team determined that this was consistent with the current NRC interpretation. However, this new information did not affect the performance deficiencies that occurred prior to that recognition.
 
=====Analysis.=====
Failure to adequately evaluate and correct oil leakage in HPSI Pumps 2A and 3A, and failure to assess the extent of condition for similar pumps, was a performance deficiency. The finding was more than minor because it affected the equipment performance attribute of the mitigating systems cornerstone objective of ensuring the availability and reliability of a system that responds to initiating events.
 
Based on the testing conducted in February 16-27, 2007, oil leakage from inboard bearings existed in five HPSI pumps. The leak rate due to this condition was determined by the licensee to be insensitive to oil level, so it was conservatively assumed by the team that leakage would continue at the same rate until the slinger ring could no longer lubricate the bearing, and the pump would subsequently fail. Using the most limiting leak rate observed (6 drops per hour, or 0.168 ml per hour) and conservative estimates of the oil which could be lost prior to losing effectiveness of the slinger ring (379 ml), failure would not occur for at least 94 days. This was determined to have very low safety significance in a Phase 1 significance determination because the finding did not represent an actual loss of safety function. The finding did not affect low pressure safety injection pumps or containment spray pumps, which would remain
 
available to perform the long term decay heat removal function, and the lengthy time the HPSI pumps would remain available would ensure the plant would be in a cold shutdown condition where low pressure systems could be used.
 
This finding was determined to have cross-cutting aspects in the human performance area of decision-making, because the licensee did not use conservative assumptions and demonstrate that the proposed course of action was safe.
 
=====Enforcement.=====
Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion V, "Instructions, Procedures and Drawings," requires that activities affecting quality shall be prescribed by instructions, procedures, or drawings, and shall be accomplished in accordance with those instructions, procedures, and drawings. The assessment of operability of safety-related equipment needed to mitigate accidents was an activity affecting quality. Regulatory Issue Summary 2005-20 provided technical guidance on operability determinations, which was incorporated into implementing Procedure 40DP-9OP26, "Operability Determination and Functional Assessment,"
Revision 18. These documents require in part that an operability determination should assess the effect or potential effect of the degraded or nonconforming condition on th affected structure, system, or components ability to perform the safety functions specified by its design, including mission time. Contrary to this, two examples of inadequate operability determinations were identified because incorrect and nonconservative assumptions were used to assess the ability of HPSI pumps to meet their full mission time with existing oil leakage.
* In May 2000, an operability determination documented in CRDR 2303499 concluded that HPSI pumps with bearing oil leakage up to 2 ml/day would remain operable because this leakage would allow sufficient oil to remain to support operation through the 180 day mission time. This conclusion was used betweeen May 2000 and November 2006 as an operability limit when assessing oil leakage in HPSI pumps. This evaluation was inadequate because it relied upon an initial volume which was nonconservative and assumed a linear leak rate. When these issues were challenged by the team, the licensee concluded that these were inappropriate assumptions in light of the presumed loss mechanism. The revised operability assessment significantly reduced the allowable leak rate to less than had been observed on several occasions.
* The November 2006 operability determination documented in CRDR 2941496 concluded that the cause of the HPSI pumps leaking oil from the bearings was related to an initially high oil level. The leakage was assumed to eventually cause the level to lower to the point where it would no longer cause leakage. It was further assumed that the level at which the leakage would stop would be sufficient to cool and lubricate the bearing during long-term operation. When the team challenged these assumptions, the licensee performed testing which showed that leakage would not stop when oil level was lowered to the optimum level but leakage did not stop as had been assumed. Based on this, the basis for the operability determination in CRDR 2941496 was incorrect.
 
Failure to adequately evaluate operability constituted a violation of Criterion V. This issue was documented in CRDRs 2941494, 2303499, and 2973682. This violation is
 
being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000529; 05000530/2007007-01, Two Examples of Failure to Properly Assess Operability for HPSI Oil Leaks.
 
===.2 Preventive Maintenance Change Backlog Was Not Tracking Due Dates===
 
====a. Inspection Scope====
The team reviewed activities associated with scheduling, tracking, deferring, and revising preventive maintenance items. This included reviewing procedures, self-assessments, and corrective action documents associated with PM deferrals and PMs located outside of the normal scheduling and tracking system.
 
The team reviewed improvements made to the process for deferring PMs. The team discussed recent and planned changes to the PM deferral process with the program owner. The team also reviewed one self-assessment, two benchmarking reports, and corrective action documents related to the PM deferral process.
 
The team reviewed the Site Work Management System (SWMS) database for PMs that were outside the normal scheduling and tracking system. In particular, the team examined PMs that were either in an unapproved status or were in an approved status with no scheduled expiration date. The team discussed the data in the database with the PM program owner and representatives from work management.
 
====b. Findings====
Introduction A Green finding was identified for failure to schedule and perform PM tasks that were in the PM change process. The team identified that a backlog of over 2500 PM changes existed which resulted in these PMs not being scheduled or performed, potentially challenging completion within the specified frequency. The team found 438 examples of PMs that had passed their due dates and an additional 2113 that had no due date calculated.
 
Description The licensee had 438 PM tasks in their change program that were in an unapproved status yet had due dates. An additional 2113 PM documents were found to exist in the change process with no due date assigned. Of the PM tasks with expiration dates, 79 were past their 125 percent grace period at the end of this inspection and 53 of these were associated with safety-related or quality augmented equipment.
 
Many of these tasks were found to be in the process to be cancelled. However, the team noted a number of PM tasks which were still intended to be performed. When a PM task was placed in the change process, it was placed in unapproved status to ensure it would not be performed before it was revised. Some of these PM tasks were intended to revise some aspect (e.g. change periodicity, work instructions, or parts) and return to approved status within the current periodicity, however no tracking was done to assure this. For example, the licensee replaced instrument air compressors. These components were not safety-related, but they were important to safety. Preventive Maintenance Tasks 126833, 126834, and 126835 to replace oil filters and grease motor bearings on the new instrument air compressors were revised, but the revision was not
 
completed until after missing the intended periodicity. Also, PM Tasks 033426, 031696, and 031698 to change the oil on the gaseous radwaste pumps were revised too late to meet the intended periodicity (PVAR 2966783). These pumps were also not safety-related, but performed an important support function.
 
Because these PM tasks were in the PM change process, the tasks were not being scheduled nor were they tracked in a way that would show when they became delinquent. Procedure 30DP-9MP08, Preventive Maintenance Program, Revision 17, required that no PM on operational equipment shall pass that late date without an approved deferral which will address a technical justification for the identified issue.
 
Preventive Maintenance Tasks 126833,126834, 126835, 033426, 031696, and 031698 were past their late date and did not have an approved deferral which addressed a technical justification for the delay. Thus, the licensee failed to schedule and perform PM activities in accordance with their PM program requirements.
 
=====Analysis.=====
The team determined that the licensees failure to track, schedule, and perform PM activities within their specified frequencies in accordance with their PM program was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, it could become a more significant safety concern in that the lack of preventive maintenance would affect the reliability of plant equipment which could impact the initiating events or mitigating systems cornerstones. This program was used to revise both safety-related and nonsafety preventive maintenance items. Because of the large number of PMs (over 2500) in this category, the team reviewed a sample of 79 overdue PMs associated with safety-related or quality-class components to assess the significance. The team did not identify any examples of overdue safety-related PMs. Based on the lack of risk significant and the fact that this finding is not suitable for SDP evaluation, this issue was reviewed by NRC management and was determined to be a finding of very low safety significance.
 
This finding had human performance cross-cutting aspects associated with resources because the large backlog of PMs was contrary to maintaining long-term equipment reliability.
 
=====Enforcement.=====
No violation of a regulatory requirements was identified because no specific examples of missed preventive maintenance on safety-related equipment were identified. Therefore, this is being treated as a finding:
FIN 05000528;529;530/2007007-02, Preventive Maintenance Change Backlog Was Not Tracking Due Dates. This issue was entered into the corrective action program under PVAR 2970076.
 
{{a|4OA6}}
==4OA6 Exit Meeting==
 
On February 9, 2007, the inspection findings were discussed with Mr. D. Mauldin and Mr. B. Bement, and other members of your staff, who acknowledged the findings.
 
On March 20, 2007, a final telephonic exit was conducted with Mr. B. Bement, Vice President, Nuclear Operations, and Mr. D. Mims, Vice President, Regulatory Affairs and Performance Improvement, and other members of your staff, who acknowledged the findings. This meeting was held to present the final characterization of the findings.
 
The team confirmed that proprietary information was handled in accordance with NRC policy and was returned to the licensee.
 
ATTACHMENT:
 
=SUPPLEMENTAL INFORMATION=
 
==KEY POINTS OF CONTACT==
 
===Licensee personnel===
: [[contact::G. Andrews]], Department Leader, System Engineering
: [[contact::S. Bauer]], Department Leader, Regulatory Affairs
: [[contact::B. Bement]], Vice President, Nuclear Operations
: [[contact::K. Chavet]], Senior Consultant, Regulatory Affairs
: [[contact::P. Borchert]], Director, Operations
: [[contact::J. Boulanger]], Section Leader, Nuclear Maintenance
: [[contact::R. Buzard]], Senior Consultant, Regulatory Affairs
: [[contact::D. Carnes]], Director, Nuclear Assurance
: [[contact::P. Carpenter]], Unit Department Leader, Operations
: [[contact::C. Churchman]], Director, Engineering
: [[contact::E. Dutton]], Department Leader, Nuclear Assurance Department
: [[contact::M. Grissom]], Section Leader, Engineering
: [[contact::R. Henry]], Site Representative, Salt River Project
: [[contact::D. Kissinger]], Engineer, Engineering Operations Support
: [[contact::L. Leavitt]], Performance Improvement Team
: [[contact::D. Leech]], Department Leader, Performance Improvement Team
: [[contact::D. Mauldin]], Vice President, Engineering
: [[contact::J. Mellody]], Department Leader, Communications
: [[contact::D. Mims]], Vice President, Regulatory Affairs and Performance Improvement
: [[contact::M. Perito]], Plant Manager, Nuclear Operations
: [[contact::J. Proctor]], Section Leader, Regulatory Affairs - Compliance
: [[contact::T. Radtke]], General Manager, Emergency Services and Support
: [[contact::R. Randalls]], Director, Nuclear Engineering Designs and Technical Services
: [[contact::F. Riedel]], Director, Nuclear Training Department
: [[contact::J. Scott]], Section Leader, Nuclear Assurance
: [[contact::M. Shea]], Director, Maintenance
: [[contact::E. Shouse]], Representative, El Paso Electric
: [[contact::D. Straka]], Senior Consultant, Regulatory Affairs
: [[contact::J. Taylor]], Nuclear Project Manager, Public Service of New Mexico
: [[contact::D. Vogt]], Section Leader, Operations Shift Technical Advisor
: [[contact::T. Weber]], Section Leader, Regulatory Affairs
NRC
: [[contact::M. Runyan]], Senior Reactor Analyst, Region IV
: [[contact::B. Wolfgang]], Component Performance and Testing Branch, NRR
Attachment
 
==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED==
 
Opened or
===Discussed===
 
None.
 
===Opened and Closed===
: 05000528;529/2007007-01              NCV        Two Examples of Failure to Properly Assess Operability for HPSI Oil Leaks. (Section 4OA2.e.1)
: 05000528;529;530/2007007-02          FIN        Preventive Maintenance Change Backlog Was Not Tracking Due Dates. (Section 4OA2.e.2)
 
==LIST OF DOCUMENTS REVIEWED==
 
}}

Revision as of 08:23, 18 June 2020

Redirect to:

Retrieved from "https://kanterella.com/w/index.php?title=IR_05000528/2007007&oldid=2835213"