IR 05000255/2007003

IR 05000255-07-003(DRP), on 02/05/2007 - 02/27/2007; Palisades Nuclear Plant; Supplemental Inspection IP 95001 - Mitigating Systems Cornerstone

ML071000609
Person / Time
Site:	Palisades
Issue date:	04/10/2007
From:	Satorius M Division Reactor Projects III
To:	Harden P Nuclear Management Co
References
IR-07-003
Download: ML071000609 (25)

Text

April 10, 2007

SUBJECT:

PALISADES NUCLEAR PLANT NRC SUPPLEMENTAL INSPECTION REPORT NO. 05000255/2007003(DRP)

Dear Mr. Harden:

On February 27, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection at your Palisades Nuclear Plant. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on February 27, 2007, with you and other members of your staff. The NRC was informed of your readiness for the inspection on January 11, 2007.

As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001, Inspection For One or Two White Inputs in a Strategic Performance Area. The purpose of the inspection was to examine the causes for and actions taken related to the Mitigating Systems Performance Indicator (MSPI) for High Pressure Safety Injection crossing the threshold from Green to White for the second quarter of 2006. This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, and to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes and to prevent recurrence. The inspection consisted of selected examination of representative records, interviews with personnel, and field walk downs.

Based on the results of this supplemental inspection, two NRC-identified findings of very low safety significance were identified, both of which involved violations of NRC requirements.

However, because these violations were of very low safety significance and because they were entered into your corrective action program, the NRC is treating the issues as Non-Cited Violations (NCV) in accordance with Section VI.A.1 of the NRC's Enforcement Policy. The inspectors determined that the cause evaluations were generally adequate. However, the inspectors identified several weaknesses in the thoroughness and quality of the cause evaluations including deficiencies in the extent of condition and extent of cause reviews. We plan to review your actions to address these weaknesses further during the next biennial Problem Identification and Resolution Inspection.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure, and your response (if any), will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRC's document system (ADAMS), which is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Mark A. Satorius, Director Division of Reactor Projects Docket No. 50-255 License No. DPR-20 Enclosure:

Inspection Report 05000255/2007003 w/Attachment: Supplemental Information cc w/encl:

M. Sellman, President and Chief Executive Officer R. Fenech, Senior Vice President, Nuclear Fossil and Hydro Operations D. Cooper, Senior Vice President and Chief Nuclear Officer L. Lahti, Manager, Regulatory Affairs J. Rogoff, Vice President, Counsel and Secretary A. Udrys, Esquire, Consumers Energy Company S. Wawro, Director of Nuclear Assets, Consumers Energy Company Supervisor, Covert Township Office of the Governor State Liaison Office, State of Michigan L. Brandon, Michigan Department of Environmental Quality -

Waste and Hazardous Materials Division

SUMMARY OF FINDINGS

IR 05000255/2007003(DRP); 02/05/2007 - 02/27/2007; Palisades Nuclear Plant; Supplemental

Inspection IP 95001 - Mitigating Systems Cornerstone.

This report covered a supplemental inspection conducted by a resident and a regional inspector. Two findings of very low safety significance were identified, both of which were associated with a Non-Cited Violation (NCV). The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609,

"Significance Determination Process (SDP)." Findings for which the SDP does not apply may be "Green" or be assigned a severity level after Nuclear Regulatory Commission (NRC)management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 3, dated July 2000.

The NRC performed this inspection in accordance with Inspection Procedure 95001,

Inspection for One or Two White Inputs in a Strategic Performance Area. This inspection was conducted in response to a White Mitigating Systems Performance Indicator (MSPI) associated with the High Pressure Safety Injection (HPSI) system. The licensee performed two root cause evaluations pertinent to the White MSPI. In addition, the licensee performed four apparent cause evaluations for component failures contributing to the White MSPI. The licensees evaluations identified weaknesses in design, maintenance, and root cause analysis.

Based on the results of this inspection, the inspectors determined that the cause evaluations at the conclusion of the inspection were generally adequate. However, the inspectors identified several weaknesses in the thoroughness and quality of the cause evaluations, including deficiencies in the extent of condition and extent of cause reviews. Implementation of the licensees corrective actions to address these weaknesses will be reviewed during a future inspection. The licensee corrected the extent of condition and extent of cause issues identified during the inspection; therefore, the inspectors concluded that the licensee had performed an adequate root cause evaluation and developed corrective actions to address the concerns associated with the various failures in the HPSI system.

Given the licensees acceptable performance in addressing the HPSI MSPI, the White performance indicator will only be considered in assessing plant performance for the quarter in which the licensee reported the White MSPI, in accordance with the guidance in IMC 0305,

Operating Reactor Assessment Program.

A.

Inspector-Identified and Self-Revealed Findings

Cornerstone: Mitigating Systems

Green.

The inspectors identified an NCV of 10 CFR 50, Appendix B, Criterion V,

Instructions, Procedures, and Drawings, for the failure to adequately quarantine a component for failure analysis as required by the licensees procedures. The licensee discarded a valve body, which resulted in the failure to complete a corrective action assigned in an associated root cause evaluation. The finding was associated with the work practices component of the human performance cross-cutting area because licensee personnel failed to use appropriate human error prevention techniques to ensure the valve body was effectively quarantined. After the issue was identified by the NRC, the licensee entered the issue into their corrective action program as Action Requests (ARs) 01076153 and 01076213.

This finding was determined to be more than minor based on a review of the list of more than minor issues in IMC 0612, Appendix E, in that the valve body was irretrievably lost.

Additionally, if left uncorrected, the failure to quarantine items could become a more significant safety concern since the failure to do so could impede the identification of root and/or contributing causes for conditions adverse to quality and prevent the implementation of appropriate corrective actions. The finding was of very low safety significance because the finding was not a design or qualification deficiency resulting in a loss of function per Generic Letter 91-18; did not represent an actual loss of safety function of a system or the loss of safety function of a train of equipment; and was not potentially risk significant due to a seismic, fire, flooding, or severe weather initiating event. (Section 02.02.a.1)

Green.

The inspectors identified an NCV of 10 CFR Part 50, Appendix B, Criterion XVI,

Corrective Action, for failure to assure that conditions adverse to quality were promptly corrected. Specifically, the inspectors concluded that the licensee failed to develop adequate actions to correct conditions adverse to quality identified during root cause evaluation activities for a valve failure on March 29, 2006. This finding had a cross-cutting aspect in the corrective action program component of the problem identification and resolution area because licensee personnel failed to promptly perform an adequate extent of condition for the valve failure. The licensee entered this performance deficiency into the corrective action program as AR 01076287 for resolution.

The finding was more than minor because, if left uncorrected, future conditions adverse to quality would not be fully evaluated or corrected. The inspectors assessed the significance of this finding as very low safety significance because, upon completing an adequate extent of condition review, no additional examples of improperly supported equipment were identified. (Section 02.02.d.1)

Licensee-Identified Violations

None.

REPORT DETAILS

INSPECTION SCOPE

The purpose of this supplemental inspection was to assess the licensees evaluation associated with a performance indicator that crossed the threshold from Green to White in the Mitigating Systems Cornerstone of the Reactor Safety Strategic Performance Area. Specifically, the licensee experienced five High Pressure Safety Injection (HPSI) Mitigating Systems Performance Indicator (MSPI) failures within a three year period. The cumulative effect of these failures was to cause the HPSI MSPI to cross the threshold from Green to White in the second quarter of calendar year 2006.

The inspectors reviewed the licensees actions associated with these five events, reviewed plant procedures and other documents, and conducted interviews with licensee personnel to ensure that the root and contributing causes of the events were identified, understood, and appropriate corrective actions were initiated. The licensee performed a root cause evaluation (RCE) for one of the five failures and a combined common cause evaluation of the five failures to identify performance and process issues that led to the White Performance Indicator. The inspectors reviewed the common cause evaluation to ensure the licensee had adequately addressed the causal factors for the White Performance Indicator.

01.01 Background Information Between 2003 and 2006, multiple failures of components within the HPSI system resulted in the MSPI crossing the Green/White threshold for the second quarter of 2006. The MSPI returned to Green the following quarter after the oldest failure rolled out of the indicator window. The licensee performed two RCEs specific to the White MSPI. The first focused on the failure of the left train HPSI subcooling valve, CV-3070, and was designated as RCE 01021152. The second RCE evaluated the aggregate issues causing the MSPI to turn White and was designated as RCE 01039162, Revision 5. In addition, as individual failures occurred, the licensee performed apparent cause evaluations. A total of five failures occurred between 2003 and 2006 that led to the White MSPI. The failures were:

• CV-3070 failure to stroke in August 2003 and March 2006

• HPSI injection valve MO-3064 failure to stroke in May and June 2005

• Improperly installed Swagelock fitting failed in January 2006 The licensees evaluations identified that both failures of CV-3070 were due to improper supports of the valve. In addition, the failures of MO-3064 were both due to an improperly installed pivot yoke in the valves breaker.

The licensee initially prepared RCE 01039162, Revision 5, and RCE 01021152 for the issues associated with the White MSPI and the CV-3070 failure. However, after making preparations for the 95001 inspection, the licensee noted inadequacies in both RCEs and decided that the weaknesses in both RCEs would be addressed by creating Revision 6 of RCE 01039162. Even though corrections for the deficiencies in RCE 01021152 were included in Revision 6 of RCE 01039162, the inspectors reviewed both root cause evaluations collectively to determine the adequacy of the licensees analysis for the White MSPI.

The licensee informed the inspectors on January 11, 2007, that Revision 6 of RCE 01039162 would be ready for inspection the week of February 5, 2007. The NRC began the inspection on February 5, 2007, and identified weaknesses in Revision 6 of RCE 01039162. Despite the additional effort by the licensee, the extent of condition and extent of cause reviews for the HPSI valve CV-3070 failure remained inadequate. The licensee committed to complete additional extent of condition and extent of cause reviews for the HPSI valve failure. This additional effort was documented in Revision 7 of RCE 01039162. To allow for completion of the additional actions to be documented in Revision 7, the inspectors delayed completion of the inspection until February 27. When the inspectors returned to evaluate Revision 7 of RCE 01039162, they determined it adequately addressed the concerns associated with the extent of condition and extent of cause reviews, and included a more comprehensive set of corrective actions.1 EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.

Determine that the evaluation identifies who (i.e., licensee, self-revealing, or NRC) and under what conditions the issue was identified.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke The licensee concluded that CV-3070 failed during a maintenance/surveillance activity and that the failure was self-revealed. The inspectors reviewed the root cause evaluation, corrective action program (CAP) documents, and prior NRC inspection reports and agreed with the licensees assessment.

RCE 01039162, Rev. 6, HPSI Equipment Issues The licensee identified that each of the five individual failures were self-revealed and documented the circumstances of each failure. The evaluation also identified that collectively the failures resulted in a White MSPI that led to the need for the collective root cause evaluation. The inspectors reviewed the RCE, CAPs associated with the individual failures, and prior NRC inspection reports discussing each failure. The inspectors agreed with the licensees conclusion that the findings were self-revealed and the circumstances under which they were identified.

b.

Determine that the evaluation documents how long the issue existed, and prior opportunities for identification.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke The licensee developed a timeline associated with the HPSI failure. Based on the inspectors review of the source documents used to construct the timeline, the inspectors agreed with the licensees assessment that the condition leading to the 1 Revision 7 of RCE 01039162 was written to correct inadequacies with the extent of condition and extent of cause reviews in Revision 6. Therefore, review of Revision 7 only appears in sections related to extent of condition, extent of cause, and corrective actions.

failure occurred in 1992, when the licensee found a bent and loose hanger which was subsequently removed in 1994. In addition, the licensee determined prior opportunities existed in 1992 during a safety related piping re-verification project (SRPRP), in 1994 when the hanger was removed, in 1995 when an engineering action request related to the hanger was initiated, and in 2003 when the valve failed to stroke.

Although the licensee developed an adequate timeline, the licensee failed to fully explore earlier opportunities for identification. For example, although the licensee identified several instances that could have identified the missing support and corrected the issue prior to the valve failure, the licensee did not interview personnel associated with the activities. The inspectors readily identified several personnel still on site with applicable knowledge that could have been interviewed. Furthermore, the licensee determined that the failure in 2003 could be attributed to personnel or programs that had changed significantly since the failure and thus did not warrant review to determine why the troubleshooting associated with that failure did not identify the missing support. The inspectors determined that such a conclusion was inappropriate because many of the people involved with the 2003 failure were still employed with the licensee and recent examples of poor troubleshooting existed. Thus, while the licensee identified prior opportunities to identify the issue, the licensee failed to determine why these opportunities did not identify the issue.

Noting these deficiencies, the inspectors reviewed historical corrective action documents, engineering evaluations, apparent cause evaluations, design change documents, and other documents. Through interviews and document reviews, the inspectors concluded that although these deficiencies existed with this evaluation, they did not substantially affect the root or contributing causes as detailed in RCE 01039162, Revision 6, as described below. With consideration of the improvements in this area contained in RCE 01039162, the inspectors concluded this item was adequately discussed. However, at the time the licensee completed and approved RCE 01021152, the broader Root Cause Evaluation to review HPSI equipment issues had not been initiated.

RCE 01039162, Rev. 6, HPSI Equipment Issues This evaluation discusses each of the five HPSI equipment failures that collectively led to the MSPI indicator turning White. While RCE 01021152 did not adequately evaluate why the issue was not previously discovered and therefore, failed to take any corrective actions in that regard, this evaluation corrected that deficiency. For example, the licensee determined ineffective troubleshooting to be a major contributor to the White MSPI. One of the corrective actions to prevent recurrence was the implementation of a significantly different and improved troubleshooting process. The inspectors agreed with the licensees assessment; therefore, the inspectors concluded that effective implementation of this corrective action would substantially address one of the causes for the White MSPI.

c.

Determine that the evaluation documents the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke This evaluation does not adequately discuss the plant-specific risk consequences; however, RCE 01039162, Revision 6 includes the risk consequences for the failure of CV-3070. RCE 01021152 identifies that the valve failure resulted from improper design control. The inspectors compared the licensees conclusion with prior NRC findings and determined the conclusions were consistent with each other.

RCE 01039162, Rev. 6, HPSI Equipment Issues This evaluation includes a quantitative analysis of core damage frequency with each of the failures associated with this issue. In all cases, the licensee determined that the issues were of very low safety significance. The conclusions contained in the evaluation were consistent with the safety significance assigned to the findings contained in previous NRC inspection reports. In addition, the evaluation identified compliance issues consistent with issues identified in previous NRC inspection reports. Therefore, the inspectors concluded that the licensee adequately characterized the risk significance and compliance concerns.

02.02 Root Cause and Extent of Condition and Extent of Cause Evaluation a.

Determine that the problem was evaluated using a systematic method(s) to identify root cause(s) and contributing cause(s).

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke Although the licensee used systematic methods for this evaluation, the inspectors noted that the results were generally of poor quality and with less detail than what would normally be expected, commensurate with the significance of the issue. The licensees evaluation stated that the following methods were used to arrive at the root cause:

creating a time line, review of historical documents, interviews, failure modes analysis, and a why staircase. Of those methods, the only one that the inspectors found to be adequately implemented was the time line. Examples to support the inspectors conclusions are as follows:

• Review of historical documents The licensee reviewed specification change 93-068, the document that removed the CV-3070 support, and found no issues. However, the inspectors learned that the root cause team did not adequately review the specification change program in place at the time to ensure it was both the appropriate process to be used and that the process was properly followed. Instead, the licensee relied on staff recollections of what the program required and did not review the actual procedures in place at the time. In response to the inspectors questions, the licensee subsequently reviewed the process that was in place at the time and found no additional issues.

The licensee reviewed prior maintenance on CV-3070 but did not review the details of what actually was done. The licensee limited the review of maintenance to a review of the titles of work orders and a review of the summary of work performed. The inspectors selected and reviewed two work orders associated with air operated valve (AOV) repair. Work Order 24412938, completed on July 11, 1995, refurbished the air operator for CV-3070. The procedure followed for reinstalling the operator to the valve yoke required mechanics to lift the operator with a rigging device, install and hand tighten the air operator mounting bolts and nuts, and then remove the rigging prior to torquing the mounting bolts. In this sequence, the untorqued bolts could allow the operator to misalign with the valve, thus contributing to the valve failure.

However, after reviewing as found valve data and pictures from the March 29, 2006, failure, the inspectors concluded that inadequate torquing of those bolts was less likely to be the cause than the missing hanger.

• Interviews The inspectors determined that licensee efforts in this area were minimal in scope. The licensee could not provide the inspectors with either interview notes or a list of people interviewed as part of this root cause evaluation. The evaluation explicitly stated that it did not assess the underlying programmatic or organizational conditions that resulted in the evaluations being done and accepted at that time. Much of the personnel involved with those evaluations are no longer at Palisades or NMC. As discussed in Section 02.01.b above, the inspectors were able to interview engineers who were employed at Palisades during all relevant time periods as well as an engineer directly involved with the SRPRP. Further, the inspectors easily learned that at least four other people with direct, relevant knowledge of the issue were still employed at Palisades.

None were interviewed by the root cause team.

• Failure Modes Analysis The licensee did not include possible failure modes such as improper maintenance or defective internal valve components in the scope of the review.

Although the root cause team did consider a bent stem, warped valve disk, and inadequate sealing surface, all of which could have contributed to the failure, the root cause evaluation was complete and approved approximately three months before troubleshooting activities to examine the stem straightness, disk flatness, and valve seat blue check were completed. As described in Section 02.02.a.1 below, the licensee could not complete a seat blue check of the disks in the valve as planned because the valve body was inadvertently discarded. When the inspectors questioned the licensee as to the status of those troubleshooting activities, the root cause team leader was unaware of their status. The licensee was, therefore, unable to explain to the inspectors why, for example, a warped valve disc or a bent valve stem from improper maintenance was not either a root or contributing cause until the licensee reviewed the results of those troubleshooting activities during this inspection.

• Why Staircase Method The licensees use of the why staircase concluded that CV-3070 failed to open because the hanger removal did not consider the effect on valve operation; however, the licensee did not question why the effect on valve operation was not evaluated at the time. The inspectors noted that the failure could have been from a lack of appropriate technical expertise, an inadequate design change process, or an inadequate vendor technical manual, none of which was explored in RCE 01021152.

Notwithstanding these weaknesses in the evaluation, the inspectors concluded that the licensee arrived at the correct root cause in RCE 01039162, Revision 6, as described below. Because of this, the inspectors determined that this item was adequately discussed.

(1) Valve Body Inadvertently Discarded Due to Ineffective Quarantine

Introduction:

The inspectors identified a finding of very low safety significance (Green)and a Non-Cited Violation of 10 CFR 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the failure to adequately quarantine a component for failure analysis as required by the licensees procedures. The inspectors identified that the licensee failed to implement procedural requirements regarding the quarantine of components for troubleshooting activities.

Description:

Following the failure of CV-3070 on March 29, 2006, the licensee prepared AR 01021152 and performed a root cause evaluation in accordance with their corrective action program as described in Procedure FP-PA-ARP-01. The licensee approved the RCE on June 21, 2006, with several corrective actions including future completion of a valve seat blue check on the removed components. This action, along with several others, was assigned to help support or refute the identified root cause. However, when the licensee attempted to complete this corrective action, they were unable to locate the removed valve body and determined it had been discarded.

The inspectors learned that licensee management was unaware of the discarded valve body because it was not entered into the corrective action process. The licensee subsequently entered the issue of the failed quarantine into their corrective action program as AR 01076153. Additionally, the licensee wrote AR 01076213 to review why the issue was not entered into the corrective action program when it was discovered in June 2006. Based on information reviewed, photographs taken of the disassembled valve, and discussions with the licensee, the inspectors determined that while the blue check would have provided additional confidence in the identified root cause, there was adequate assurance that the licensee identified the technical cause of the valve failure.

Therefore, the inspectors concluded that the failure to quarantine the valve body did not substantially impact the ability to determine the technical root cause of the valve failure.

Analysis:

The inspectors determined the failure to quarantine the degraded valve body associated with AR 01021152 was a performance deficiency that warranted a significance evaluation in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening. This finding was determined to be more than minor because it was similar to an example of more than minor issues in Appendix E in that the valve body was irretrievably lost. Additionally, if left uncorrected, the failure to properly quarantine items could become a more significant safety concern since the failure to do so could impede the identification of root and/or contributing causes for conditions adverse to quality and prevent the implementation of appropriate corrective actions. The finding was associated with the work practices component of the human performance cross-cutting area because licensee personnel failed to use appropriate human error prevention techniques to ensure the valve body was effectively quarantined.

The inspectors determined the finding impacted the human performance attribute of the Mitigating Systems Cornerstone. Using IMC 0609, Appendix A, Determining the Significance of Reactor Inspection Findings for At-Power Situations, Attachment 1, SDP Phase 1 Screening Worksheet for IE, MS, and B Cornerstones, the inspectors determined the finding was not a design qualification deficiency resulting in a loss of function per Generic Letter 91-18, did not represent an actual loss of safety function of a system or train of equipment, and was not potentially risk-significant due to a seismic, fire, flooding, or severe weather initiating event. Therefore, the finding was considered to be of very low safety significance (Green).

Enforcement:

10 CFR 50, Appendix B, Criterion V, requires, in part, that activities affecting quality shall be prescribed and accomplished by procedures appropriate to the circumstances. Procedure FP-PA-ARP-01, Revision 12, CAP Action Request Process, is a licensee procedure that is used to meet the requirements of 10 CFR 50, Appendix B, Criterion XVI. FP-PA-ARP-01 requires, in part, that the originator of an Action Request SHALL take the necessary actions to preserve any evidence or damaged equipment to minimize the loss of information that may help determine the cause of the problem. Contrary to the above, the licensee failed to adequately preserve the valve body from a HPSI Sub-Cooling valve, and the valve body was inadvertently discarded. As a result, the licensee was unable to complete corrective actions assigned in the RCE associated with AR 01021152 to gain additional information to support or refute the identified root cause. This issue has been entered into the licensees corrective action program as AR 01076153. Because this finding was of very low safety significance and because it was entered into the licensees corrective action program, this violation is being treated as a Non-Cited Violation (NCV 05000255/2007003-01)consistent with Section VI.A of the NRC Enforcement Policy.

RCE 01039162, Rev. 6, HPSI Equipment Issues The licensee utilized several systematic root cause methodologies in this evaluation including data analysis, event and causal factors charts, and barrier analysis. The inspectors determined that the methodologies were appropriate to the common cause issue and generally well-developed. The substantial improvement in the use and quality of the methodologies compared to those used in RCE 01021152 provided the inspectors with assurance that appropriate root causes were determined.

The inspectors noted that this evaluation also failed to properly review details of previous CV-3070 maintenance. The licensee was also unable to provide a list of interviewees for this evaluation. Therefore, in order to ensure that the licensee did not overlook significant causal factors, the inspectors reviewed documents and interviewed licensee personnel. The information gathered by the inspectors did not contradict the licensees conclusion; therefore, the inspectors concluded that the licensee had identified a reasonable set of root and contributing causes.

b.

Determine that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke Because of the deficiencies discussed above, the inspectors concluded that this evaluation was not conducted to a level of detail commensurate with the issue.

However, because this deficiency was corrected with the completion of RCE 01039162, Revision 6, the inspectors concluded this item was adequately discussed.

RCE 01039162, Rev. 6, HPSI Equipment Issues The inspectors concluded this evaluation was generally conducted to a level of detail commensurate with the significance of the problem. For example, the licensee identified a fundamental weakness in their troubleshooting process as well as several potential weaknesses in both their AOV and motor operated valve (MOV) programs as a result of this evaluation. The inspectors reviewed the background documents and agreed with these conclusions. However, because the licensee utilized this revision to correct deficiencies in RCE 01021152, the inspectors concluded that the failure to properly review previous CV-3070 maintenance or to interview people directly involved with the hanger being removed constituted a weakness in this evaluation. Despite these weaknesses, the inspectors concluded they were not significant because they had no impact on the licensees ability to arrive at the proper root and contributing causes.

Therefore, the inspectors concluded this item was adequately discussed.

c.

Determine that the root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke The evaluation discussed relevant prior internal and external operating experience. As discussed in Section 02.01.b, the inspectors concluded this item was adequately discussed.

RCE 01039162, Rev. 6, HPSI Equipment Issues The evaluation discussed relevant prior internal and external operating experience. As discussed in Section 02.01.b, the inspectors concluded this item was adequately discussed.

d.

Determine that the root cause evaluation addresses the extent of condition and the extent of cause of the problem.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke The inspectors concluded that the licensee failed to perform an adequate extent of condition and an adequate extent of cause. For this extent of condition, the licensee relied on a search of the corrective action database to identify any other degraded valves. Although the licensee stated that some plant walk downs were completed and valve traces reviewed, the licensee did not have documentation to support these activities. Through discussions with licensee personnel, the inspectors concluded that these efforts were not comprehensive. The inspectors determined that a review of the corrective action database would not be an effective method of identifying if plant equipment lacked adequate supports; this method relies on degradation of components to alert operators to a deficiency. After discussing the issue with the licensee, the licensee performed an extensive extent of condition and extent of cause review, which was documented in RCE 01039162, Revision 7. In the expanded review, the licensee performed physical walk downs of the facility, reviewed valve traces, plant drawings, maintenance records, and reviewed all hanger modifications performed during the SRPRP. The inspectors reviewed the licensees documentation of this effort and performed limited plant walk downs. Although not a part of this evaluation, the licensee performed a latent modification review where several previous plant modifications were reviewed to identify other potential deficiencies. Deficiencies identified during this review were entered into the licensees corrective action program. The licensee considered this effort as an extent of cause review.

By the end of the scheduled inspection week, February 9, 2007, the licensee was unable to furnish sufficient documentation to show that an adequate extent of condition and extent of cause was performed. However, because the inspectors believed that an adequate extent of condition and extent of cause could be accomplished in a short time frame, the inspectors postponed the closure of this inspection until February 27, 2007.

The licensee developed a revised extent of condition and extent of cause plan and completed those actions. Because the licensee performed an adequate extent of cause and extent of condition as documented in RCE 01039162, Revision 7, the inspectors concluded this item was adequately discussed.

(1) Inadequate Extent of Condition for HPSI Valve Failure

Introduction:

The inspectors identified a Green NCV of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, for failure to assure that conditions adverse to quality were promptly corrected. Specifically, the inspectors concluded that the licensee failed to develop actions to correct conditions adverse to quality identified during Root Cause Evaluation activities for a HPSI valve failure in March 2006.

Description:

During routine maintenance on March 29, 2006, Palisades staff identified that CV-3070 would not open and declared the associated train of HPSI inoperable.

Since the valve could not be repaired in the specified action time, the licensee completed a plant shutdown. The licensee completed a root cause evaluation and determined the root cause to be the failure to thoroughly evaluate the valve operation and provide adequate support to the valve actuator. The horizontally-mounted air actuator weighed approximately 265 pounds and was originally installed with a hanger supporting its weight. While performing a safety-related piping reverification project in the early 1990s, the licensee removed the support based primarily on a flawed evaluation of the necessity of the support.

In performing the root cause evaluation in June 2006, the licensee concluded that the unsupported actuator caused the valve shaft to bow which caused the shaft to contact the backseat and scored the shaft. The licensee replaced the valve and installed a hanger to properly support the actuator. The licensees documented extent of condition review in the root cause evaluation was a search of the corrective action program database. The inspectors were concerned that such a search did not constitute an appropriate extent of condition review. The licensee stated that they reviewed the valve diagnostic data for many safety-related and risk significant valves but could not provide the inspectors with any supporting documentation. The inspectors learned that only a limited and undocumented plant walk down was performed to identify other similarly mounted actuators.

The inspectors concluded the deficient mounting resulted from the lack of an appropriate evaluation to support a hanger modification. At the time of the inspection, the licensee had not determined if similar modifications existed. The inspectors recognized that removing a support could have adverse effects on many more components other than just heavy horizontally mounted air actuators such as vertical valves, pumps, or piping systems. Therefore, the inspectors concluded that the licensees original extent of condition efforts that only evaluated horizontally oriented AOVs were not adequate to appropriately determine the extent of condition and take associated corrective actions. The licensee acknowledged this deficiency, entered the issue into their corrective action program as AR 01076287, and issued a corrective action to review a sample of previous hanger/support removals.

Analysis:

The inspectors concluded that the failure to perform an adequate extent of condition review for a condition adverse to quality was a performance deficiency warranting a significance determination. The inspectors concluded that the performance deficiency was more than minor because it is directly related to the Reactor Safety Cornerstone objective of limiting the likelihood of challenging critical safety functions.

The inspectors concluded that the failure to perform an adequate extent of condition could adversely affect equipment performance by allowing similar conditions to self-reveal. The inspectors reviewed Appendix B to IMC 0612 and determined that this finding was required to be evaluated by the Significance Determination Process due to its impact on the Mitigating Systems Cornerstone objective of ensuring the operability, availability, reliability, or function of systems that respond to initiating events to prevent undesirable consequences. The potentially impacted systems included component cooling water and safety injection. The inspectors assessed the significance of this finding as very low safety significance (Green) because although additional discrepancies were found during the licensees review, none rendered the associated systems inoperable. This finding had a cross-cutting aspect in the corrective action program component of the problem identification and resolution area because licensee personnel failed to promptly perform an adequate extent of condition for the valve failure.

Enforcement:

10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, states, in part, measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. Contrary to the above, the licensee failed to identify other equipment that could have been affected by similarly inadequate engineering reviews of support removals as identified in Root Cause Evaluation 01021152. Because this failure to comply with 10 CFR Part 50, Appendix B, Criterion XVI, is of very low safety significance and has been entered into the licensees corrective action program as AR 01076287, this violation is being treated as an NCV, consistent with Section VI.A of the Enforcement Policy (NCV 05000255/2007003-02). Corrective actions for this NCV included a revised extent of condition review that was adequately performed.

RCE 01039162, Rev. 6, HPSI Equipment Issues The inspectors concluded that neither the extent of condition nor the extent of cause as documented in this evaluation was adequate with respect to the CV-3070 valve failure.

As part of the extent of condition and extent of cause with respect to the White MSPI issue, the licensee reviewed other MSPI systems to determine the extent to which improper licensee actions were affecting the margin to the Green-White threshold. In addition, the licensee planned to review their AOV and MOV programs to determine the extent to which deficiencies were affecting other valves. Other extent of condition and extent of cause reviews are discussed previously in this section.

RCE 01039162, Rev. 7, HPSI Equipment Issues The licensee performed maintenance and diagnostic reviews of all relevant valves, both air and motor operated. Field walk downs were performed, where appropriate, to verify the adequacy of the supports. The licensee reviewed all support modifications as part of the SRPRP and found four instances where a support modification or removal could have affected a nearby component, all of which were valves. Upon further review, none of the four valves were found to have been adversely impacted by the support modification. No other similarly installed valves were found to show signs of the same failure mechanism as CV-3070.

The licensee documented their revised extent of condition and extent of cause in this evaluation, which the inspectors reviewed. The inspectors performed independent field walk downs and document reviews to verify the adequacy of the actions taken. The inspectors concluded that the licensees efforts had sufficiently improved in this area such that it was adequately addressed.

e.

Determine that the root cause evaluation, extent of condition, and extent of cause appropriately considered the safety culture components as described in IMC 0305.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke The requirement in inspection procedure 95001 to review safety culture components went into effect on July 1, 2006. This evaluation was approved in June 2006, and did not include consideration of safety culture aspects. Therefore, this inspection requirement is not applicable to this evaluation.

RCE 01039162, Rev. 6, HPSI Equipment Issues The evaluation discusses each of the required safety culture aspects in sufficient detail; however, the inspectors identified weaknesses in the methods that the licensee used to arrive at their conclusions, although the inspectors generally agreed with the licensees conclusions. For example, this evaluation states that, the evaluation team reviewed the HPSI event reports listed in Attachment 1 of this report and found that none are attributed to inadequate work control [or work practices]. However, as described in Section 02.02.a of this report, the licensee did not review maintenance records in sufficient detail to properly arrive at this conclusion. The inspectors therefore reviewed a sample of previous CV-3070 maintenance, and did not identify any issues that contradicted the licensees conclusion.

Additionally, as described in Section 02.02.a of this report, the scope of the licensees interviews also appeared to be of insufficient depth and breadth to arrive at the conclusion that none of the events were attributable to a negative environment for raising a nuclear safety issue. The inspectors interviewed five engineers, all with varying lengths of employment at Palisades dating back to the late 1980's. None of the interviewees provided any information to the inspectors that contradicted the licensees conclusions. Therefore, the inspectors concurred with the licensees conclusions.

02.03 Corrective Actions a.

Determine that appropriate corrective action(s) are/were specified for each root/contributing cause or that there is/was an evaluation that no actions are/were necessary.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke Immediate corrective actions included replacing the valve and providing an actuator support. Because the evaluation was not completed to a level commensurate with the significance of the issue, appropriate comprehensive corrective actions were not developed nor implemented (see NCV discussed in Section 02.02.d.1). However, because additional information in RCE 01039162, Revisions 6 and 7 adequately addressed this deficiency, the inspectors concluded this item was sufficiently addressed by the end of the inspection.

RCE 01039162, Rev. 7, HPSI Equipment Issues The inspectors concluded that the corrective actions taken were generally appropriate to the identified root or contributing cause and that each cause had appropriate corrective actions. The inspectors concluded that this item was adequately addressed by the end of the inspection.

b.

Determine that the corrective actions have been prioritized with consideration of the risk significance and regulatory compliance.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke As discussed in Section 02.03.a, appropriate comprehensive corrective actions were not developed and, therefore, not prioritized. The inspectors noted, however, that certain troubleshooting efforts as documented in Section 02.02.a, were actually documented as corrective actions. Those actions, the results of which could have been used to either confirm or refute the identified root cause, were not scheduled to be performed until approximately three months after the root cause evaluation was approved. The inspectors considered this to be an example of ineffective prioritization. However, because RCE 01039162, Revisions 6 and 7, developed and appropriately prioritized the corrective actions, the inspectors concluded that this item was adequately addressed.

RCE 01039162, Rev. 7, HPSI Equipment Issues The inspectors reviewed the corrective actions identified in this evaluation and determined they were effectively prioritized with respect to risk and regulatory compliance. The inspectors concluded this item was adequately addressed.

c.

Determine that a schedule has been established for implementing and completing the corrective actions.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke As discussed in Section 02.03.a, appropriate comprehensive corrective actions were not developed and, therefore, not scheduled. However, because RCE 01039162, Revisions 6 and 7, corrected this deficiency, the inspectors concluded this item was adequately addressed.

RCE 01039162, Rev. 7, HPSI Equipment Issues The inspectors concluded the licensee developed a schedule to implement the corrective actions and, in some instances, planned corrective actions were aggressively scheduled. The inspectors noted that the most significant corrective actions had already been completed prior to this inspection. Therefore, the inspectors concluded this item was adequately addressed.

d.

Determine that quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence.

RCE 01021152, CV-3070 HPSI P-66B Sub-Cooling Valve Will Not Stroke As discussed in Section 02.03.a, appropriate comprehensive corrective actions were not developed and, therefore, not assessed. However, because RCE 01039162, Revision 6, corrected this deficiency, the inspectors concluded this item was adequately addressed.

RCE 01039162, Rev. 7, HPSI Equipment Issues The inspectors concluded the licensee had in place appropriate measures of success for the identified corrective actions such as an effectiveness review. Because no concerns were identified with this area, the inspectors concluded this item was adequately addressed.

MANAGEMENT MEETINGS

Exit Meeting Summary

On February 9, 2007, the inspectors presented their issues and observations to Mr. Paul Harden and other members of licensee staff in an interim exit meeting. After additional on-site inspection, including review of the licensees revised extent of condition efforts on February 27, 2007, the inspectors presented their issues and observations to Mr. Paul Harden and other members of licensee staff at the exit meeting. The licensee acknowledged the issues presented.

Proprietary information was reviewed during the inspection and was handled in accordance with NRC policy. The inspectors verified with the licensee that all proprietary material examined during the inspection was returned to the licensee.

Regulatory Performance Meeting On February 27, 2007, as part of the exit meeting associated with the 95001 inspection, the NRC met with the licensee to discuss their performance in accordance with Section 06.05.a.1 of IMC 0305. During this meeting, the NRC and licensee discussed the issues related to the White MSPI that resulted in Palisades being placed in the Regulatory Response Column of the Action Matrix. This discussion included the causes, corrective actions, extent of condition, extent of cause, and other planned licensee actions.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

P. Harden, Site Vice President

G. Hettel, Plant Manager

B. Berles, Systems Engineering Manager

J. Broschak, Engineering Director

T. Fouty, Engineering Programs and Analysis Supervisor

L. Lahti, Regulatory Affairs Manager

P. Russell, Engineering Programs Manager

B. Van Wagner, Engineering Programs Supervisor

Nuclear Regulatory Commission

C. Lipa, Branch Chief

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

05000255/2007003-01 NCV Valve Body Inadvertently Discarded Due to Ineffective Quarantine (Section 02.02.a.1)

05000255/2007003-02 NCV Inadequate Extent of Condition for HPSI Valve Failure (Section 02.02.d.1)

Discussed

None.