IR 05000454/2011016
Download: ML112770516
Text
October 4, 2011
EA-11-014 Mr. Michael Senior Vice President, Exelon Generation Company, LLC President and Chief Nuclear Officer (CNO), Exelon Nuclear 4300 Winfield Road Warrenville, IL 60555
SUBJECT: BYRON STATION, UNIT 2 - NRC INSPECTION PROCEDURE 95001 SUPPLEMENTAL INSPECTION REPORT 05000455/2011016
Dear Mr. Pacilio:
On August 26, 2011, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection pursuant to Inspection Procedure 95001 at your Byron Station, Unit 2. The enclosed report documents the inspection results, which were discussed during an exit and regulatory performance meeting on August 26, 2011, with Mr. T. Tulon and other members of your staff. As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure (IP) 95001, "Inspection for One or Two White Inputs in a Strategic Performance Area." The purpose of the inspection was to examine the causes for, and actions taken related to a finding having low to moderate safety significance (i.e., White) at Byron Station, Unit 2. The finding involved the failure to ensure that a flange connection on the upper lube oil cooler of the 2A Emergency Diesel Generator (EDG) was correctly torqued following maintenance. This led to the 2A EDG being required to be shut down when a significant oil leak developed during routine monthly surveillance testing on November 17, 2010. This issue was documented previously in NRC Inspection Report 05000455/2011011. The NRC staff was informed by your letter dated July 14, 2011, of your readiness for this inspection. This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the event resulting in the White finding were understood, to independently assess the extent of condition and extent of cause, and to provide assurance that the corrective actions for the risk-significant performance issues were sufficient to address the root causes and contributing causes to prevent recurrence. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The inspector reviewed selected procedures and records and interviewed personnel. The inspector determined that your root cause evaluation was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event. The inspector also concluded that you identified reasonable and appropriate corrective actions for each root and contributing cause and that the corrective actions appeared to be prioritized commensurate with the safety significance of the issues. Several observations regarding specific aspects of your root cause evaluation and corrective actions that warrant additional consideration by your staff were also identified.
Based on your overall acceptable performance in addressing the White finding that was the subject of this inspection, in accordance with the guidance in Inspection Manual Chapter (IMC) 0305, "Operating Reactor Assessment Program," the White finding will only be considered in assessing plant performance for a total of four quarters (i.e. through the fourth quarter of 2011). Based on the results of this inspection, one NRC-identified finding of very low safety significance (Green) was identified. The finding did not involve a violation of NRC requirements. If you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region III, and the NRC Resident Inspectors at the Byron Station. The information you provide will be considered in accordance with IMC 0305.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any), will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System (PARS) component of NRC's Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/readingrm/ adams.html (the Public Electronic Reading Room).
Sincerely,/RA/ Eric R. Duncan, Chief Branch 3 Division of Reactor Projects Docket No. 50-455 License No. NPF-66
Enclosure:
Inspection Report 05000455/2011016
w/Attachment:
Supplemental Information cc w/encl: Distribution via ListServ Enclosure U. S. NUCLEAR REGULATORY COMMISSION REGION III Docket No: 50-455 License No: NPF-66 Report No: 05000455/2011016 Licensee: Exelon Generation Company, LLC Facility: Byron Station, Unit 2 Location: Byron, Illinois Dates: August 22, 2011, through August 26, 2011 Inspectors: T. Taylor, Resident Inspector, Palisades Approved by: E. Duncan, Chief Branch 3 Division of Reactor Projects
1 Enclosure
SUMMARY OF FINDINGS
IR 05000455/2011016; 08/22/2011 - 08/26/2011; Byron Station, Unit 2; Supplemental Inspection - Inspection Procedure 95001. This supplemental inspection was performed by the Palisades Resident Inspector. One Green finding was identified by the inspector. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, "Significance Determination Process" (SDP). Assigned cross-cutting aspects were determined using IMC 0310, "Components Within the Cross-Cutting Areas." Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006. This supplemental inspection was performed in accordance with Inspection Procedure (IP) 95001, "Inspection for One or Two White Inputs in a Strategic Performance Area," to assess the licensee's root cause evaluation, extent of condition and extent of cause review, and corrective actions for a finding that involved the failure to ensure that a flange connection on the upper lube oil cooler of the 2A Emergency Diesel Generator (EDG) was correctly torqued following maintenance. This led to the 2A EDG being required to be shut down when a significant oil leak developed during routine monthly surveillance testing on November 17, 2010. This finding was previously characterized as having low to moderate safety significance (i.e., White) in an NRC letter dated March 14, 2011, which finalized the preliminary assessment of the finding documented in NRC Inspection Report 05000455/2011011. During this inspection, the inspector determined that the licensee's root cause evaluation was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event. The inspector also concluded that the licensee identified reasonable and appropriate corrective actions for each root and contributing cause and that the corrective actions appeared to be prioritized commensurate with the safety significance of the issues. The licensee identified one root cause and one contributing cause in their evaluation. The root cause was that there was no formal structured process in place to ensure that Electric Power Research Institute (EPRI) documents were reviewed to capture good work practices. The contributing cause was that there was inadequate procedural direction on the assembly of multiple joint configurations. The inspector noted the following observations that warranted additional consideration by the licensee: The extent of cause evaluation was narrow in scope. As discussed above, the root cause identified by the licensee was the absence of a formal, structured process to ensure that EPRI documents were reviewed to capture good work practices. The licensee searched for other EPRI documents within the leak sealing technology area, but did not broaden the scope to identify other potentially vulnerable processes or programs. This observation was the subject of an NRC-identified finding detailed in this report. The details of the effectiveness review for the corrective action to prevent recurrence (CAPR) had not been developed prior to the inspection. While the inspector was able to review the general tools that were planned to perform the effectiveness review (i.e. the focused area self-assessment procedure and effectiveness review procedures), the specific plan and metrics had not been developed. The licensee generated an issue report (IR) to address this issue. A revision to one of the licensee's procedures to address the contributing cause did not provide specific criteria to prompt additional actions by maintenance workers. As a result, this action could be subject to improper interpretation by maintenance personnel conducting work in the field. Given the licensee's acceptable performance in addressing the 2A EDG lube oil cooler leak, the White finding associated with this issue will only be considered in assessing plant performance for a total of four quarters (i.e. through the fourth quarter of 2011) in accordance with the guidance in IMC 0305, "Operating Reactor Assessment Program."
A. NRC-Identified
and Self-Revealed Findings
Cornerstone: Mitigating Systems
- Green.
The inspector identified a finding of very low safety significance when licensee personnel failed to perform an adequate extent of cause review in the root cause evaluation for the 2A EDG lube oil cooler leak. Specifically, the root cause evaluation identified that the root cause for the White finding was the absence of a formal, structured process to ensure that EPRI documents were reviewed to capture good work practices. However, the extent of cause review performed by the licensee was narrow in scope and did not include other potentially vulnerable programs other than that which affected the EDG lube oil cooler (i.e. the leakage reduction series publications). The licensee entered this issue into their corrective action program in an effort to define an appropriate scope for a supplemental extent of cause evaluation effort. The inspector concluded the finding was more than minor because if left uncorrected it could become a more significant safety concern. Specifically, the licensee's stated root cause of not having a formal process in place to incorporate EPRI documents from the Sealing Technology and Plant Leakage Reduction Series, which led to an inoperable EDG, could also impact other programs or processes. However, the potential impact of the identified root cause on other programs or processes were not reviewed as part of the licensee's extent of cause review effort. The inspector determined the finding could be evaluated using the SDP in accordance with IMC 0609, "Significance Determination Process," Attachment 4, "Phase I - Initial Screening and Characterization of Findings," Table 4a for the Mitigating Systems Cornerstone and answered "No" to the Mitigating Systems Cornerstone questions. Therefore, the finding screened as having very low safety significance (Green). The finding had an associated cross-cutting aspect in the Self and Independent Assessments component of the Problem Identification and Resolution cross-cutting area, because the licensee's assessment on the readiness for the NRC Supplemental Inspection failed to recognize the weakness in the extent of cause discussion (P.3(a)).
B. Licensee-Identified Violations
None.
REPORT DETAILS
OTHER ACTIVITIES
4OA4 Supplemental Inspection
.01 Inspection Scope
This inspection was conducted in accordance with Inspection Procedure (IP) 95001, "Inspection for One or Two White Inputs in a Strategic Performance Area," to assess the licensee's evaluation of one inspection finding of low to moderate safety significance (i.e., White) in the Mitigating Systems Cornerstone. The inspection objectives were to: Provide assurance that the root causes and contributing causes of risk-significant performance issues were understood; Provide assurance that the extent of condition and extent of cause of risk-significant issues were identified; and Provide assurance that the licensee's corrective actions to risk-significant performance issues were or will be sufficient to address the root causes and contributing causes, and to prevent recurrence. By letter dated March 14, 2011, the NRC communicated the final significance determination for a finding having low to moderate safety significance (i.e., White), with an associated violation of NRC requirements at Byron Station, Unit 2. The finding involved the failure to ensure that a flange connection on the upper lube oil cooler of the 2A Emergency Diesel Generator (EDG) was correctly torqued following maintenance. This led to the 2A EDG being required to be shut down when a significant oil leak developed during routine monthly surveillance testing on November 17, 2010. The NRC reviewed the circumstances that led to the finding and the licensee's response to the event during a Maintenance Effectiveness inspection completed on February 7, 2011. The details of the performance issues and the preliminary results of the NRC's significance evaluation were documented in NRC Inspection Report 05000455/2011011. Byron Station, Unit 2, entered the Regulatory Response column of the NRC's Reactor Oversight Process (ROP) Action Matrix in the first quarter of 2011 based on the White inspection finding. On July 14, 2011, the licensee notified the NRC that applicable corrective actions for the finding had either been completed or initiated, and that it was ready for the NRC to conduct this supplemental inspection to review its evaluation of the causes and the actions taken to address the White finding. In preparation for the inspection, the licensee performed Root Cause Evaluation (RCE) 1166626, Revision 1, to identify the root and contributing causes for the White finding. The inspector reviewed the licensee's RCE in addition to other evaluations conducted in support and as a result of the RCE. The inspector reviewed corrective actions that were taken or planned to address the identified causes. The inspector also held discussions with licensee personnel to ensure that the root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and preclude repetition.
.02 Evaluation of the Inspection Requirements
02.01 Problem Identification a. Determine that the evaluation documented who identified the issue (i.e., license-identified, self-revealed, or NRC-identified) and under what conditions the issue was identified. The inspector determined that the licensee's RCE adequately described the conditions of this self-revealed event. On November 17, 2010, the licensee was conducting routine monthly surveillance testing of the 2A EDG. While the EDG was at full load, the equipment operator noted a significant lube oil leak on the upper lube oil cooler. Investigation into the event by the licensee revealed that most of the bolting associated with the stationary head-to-shell connection on the cooler (the connection where the leak occurred) had loosened significantly from the specified torque in the work instruction that last attached the head to the shell. The RCE determined that this connection was last disassembled in January 2010 to repair a leaking tube in the heat exchanger. The inspector requested a list of work orders associated with the lube oil coolers and conducted interviews to validate that maintenance on the cooler had not been performed since January 17, 2010. The RCE documented several surveillance testing activities for the 2A EDG between January 17, 2010, and the failed monthly surveillance test on November 17, 2010, with no known abnormalities indicating a degraded head-to-shell connection. Minor leakage had been observed by equipment operators from various other components during some of the previous surveillance tests, but nothing that would indicate degradation of the head-to-shell connection.
b. Determine that the evaluation documented how long the issue existed and prior opportunities for identification. The inspector determined that the licensee's RCE adequately documented how long the degraded head-to-shell connection existed and whether or not there were prior opportunities for identification.
As stated previously, the inspector reviewed the work order database for other maintenance activities that could have affected the flange connection from the time maintenance was completed on January 17, 2010, until the 2A EDG was required to be shut down when a significant oil leak developed during routine monthly surveillance testing on November 17, 2010. Additionally, the inspector observed a routine surveillance test and noted that operations personnel and the system engineer performed detailed walkdowns of the EDG and its associated support systems while it was running. The inspector agreed with the findings in the RCE that there were no other reasonable opportunities for prior detection of the degraded connection. c. Determine that the evaluation documented the plant-specific risk consequences, as applicable, and compliance concerns associated with the issue. The inspector determined that the RCE adequately documented the plant-specific risk consequences and compliance concerns associated with the event.
The inspector reviewed the risk assessment documented in the RCE. The licensee's risk assessment calculated two delta-CDF [core damage frequency] outcomes from the event. One was a bounding value of 9.4E-6 resulting from an assumed inoperability date of May 19, 2010. This reflected a point-in-time where the 2A EDG had accumulated 28.9 hours1.041667e-4 days <br />0.0025 hours <br />1.488095e-5 weeks <br />3.4245e-6 months <br /> of run time prior to the failure in November 2010 (Probabilistic Risk Assessment (PRA) mission time is 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />). Additionally, the licensee calculated a value of 7.6E-6 delta-CDF when using an accumulated run time of 23.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> prior to the failure on June 23, 2010 (which was closer to the PRA mission time). The inspector discussed the results with the Region III Senior Risk Analyst (SRA) who developed the Phase 3 risk analysis result associated with this event and that was discussed in NRC Inspection Report 05000455/2011011. The NRC calculated a value of 2.9E-6 delta-CDF based on an assumed inoperability date of March 24, 2010. In discussions with the SRA, the inspector learned that there were slight differences in how run time was credited between the licensee's and NRC's assessment. The SRA credited partial run times while the licensee assumed the EDG was not available at all starting at the assumed inoperability date. The SRA methodology reflected an increased probability to restore offsite power during applicable accident sequences as compared to the licensee's approach. Additionally, the SRA only credited times at which the EDG was at full load, while the licensee credited all running times. The SRA was not concerned with the slight differences in methodology, and the licensee stated they had worked with and discussed the various issues with the SRA when performing their PRA assessment. Based on this, the inspector concluded that the licensee adequately documented the plant-specific risk consequences of the event. The inspector also reviewed the Notice of Violation (ML110740619) and the Licensee Event Report (LER) submittals associated with the event. Additionally, the inspector reviewed the regulatory compliance issues documented in the licensee's RCE. The inspector also reviewed NRC Inspection Report 05000455/2011011, which documented the then-preliminary White finding for the lube oil leak. The inspector concluded that the compliance concerns were adequately addressed by the licensee, as described in the Notice of Violation. No other compliance issues were identified.
d. Findings
No findings were identified. 02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a. Determine that the problem was evaluated using a systematic methodology to identify the root and contributing causes. The inspector determined that the RCE adequately applied systematic methods in evaluating the issue in order to identify root causes and contributing causes. In its root cause analysis, the licensee utilized the TapRooT system to identify root causes and contributing causes for the event. The inspector reviewed the TapRooT process and assumptions made by the licensee. Additionally, the inspector noted two other approaches used during the analysis: Event and Causal Factor Charting and Why Staircases.
b. Determine that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem. The inspector determined that the RCE was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event. The licensee identified one root cause and one contributing cause in their evaluation. The root cause was that there was no formal structured process in place to ensure that Electric Power Research Institute (EPRI) documents were reviewed to capture good work practices. The contributing cause was that there was inadequate procedural direction on the assembly of multiple joint configurations. The inspector was informed by the licensee's root cause team that the original intent was to cite the procedural inadequacy as the root cause of the issue. However, by employing the Why Staircase systematic method one step further, the licensee sought an answer as to why maintenance personnel did not have adequate procedural guidance. The root cause team stated that this extra "why" question led them to conclude that there was relevant operating experience (OE) published by EPRI in 2001 that if it had been incorporated into maintenance procedures, would have precluded this event. Additionally, other EPRI guidance was discovered within the Leakage Sealing and Technology Series which could have helped preclude the leak. The licensee concluded that their operating experience program was not structured such that EPRI documents would receive adequate screening. The inspector reviewed the supporting documentation for the licensee's root cause and had numerous discussions with members of the maintenance department and root cause team leader to understand why other causes were ruled out. Specifically, the inspector was concerned with possible deficiencies in the work planning process and/or mechanical maintenance training program. This was driven by the fact that in the course of their investigation, the licensee had documented that in January 2010, the gasket material for the 2A EDG lube oil cooler had been changed from what had previously been used and that this gasket had then been mispositioned by the workers during installation in January 2010. Additionally, while onsite, the inspector learned of a common cause analysis being conducted by the licensee to address an adverse trend in maintenance rework issues. A majority of the rework issues centered around poor mechanical maintenance work practices, including several examples of work resulting in bolted connection leakage.
The licensee's investigation into the event included an independent evaluation to determine potential causes by the engineering company MPR Associates, Inc. MPR's report was reviewed by the inspector. After the leak, the licensee identified that 3 of the 16 bolts on the stationary head-to-shell connection had become loose and most of the remaining bolts had significantly lower measured torque values than what was documented for the work performed in January 2010. The MPR report cited several possible causes for the leak. One concerned the mispositioning of the gasket on the head-to-shell connection. MPR calculated the resultant maximum stress on the gasket. Utilizing data from the gasket vendor, MPR concluded that although the misalignment created a localized stress higher than vendor recommendations, it was still below the point at which gasket breakdown would occur.
One possible cause was related to the new type of gasket material chosen for installation in January 2010 (Garlock 9900 vice JM Clipper 961). The inspector reviewed the engineering change associated with the material in question, MPR's conclusions, and the specification sheet for the material and determined that the change in gasket material did not contribute to the leakage from the cooler. Another potential cause cited by MPR was misalignment of the cooling water outlet nozzle with the lower cooler water inlet nozzle. The stationary head which bolted onto the heat exchanger shell had two additional flanged connections to route cooling water from the lower lube oil cooler, through the upper lube oil cooler, and into the return header. Soon after the tube bundle and gasket were replaced to address the leak in November 2010, the licensee noted that the replacement gasket had started to extrude from the joint. The licensee had returned to using the original gasket design. At this point, the equipment had been satisfactorily pressure tested, but was still awaiting complete restoration to operable status. The licensee decided to disassemble the cooler again and replace the gasket. While disassembling the cooler, the licensee found that the bolting on the head-to-shell connection was only wrench tight. Based on discussions with the licensee, the period of time from assembly to realizing the bolting had loosened was estimated to be less than a day. The licensee took measurements on the lower flange connection off of the head and discovered a flange-to-flange offset of approximately 0.125 inches as well as an angular offset between the flange faces (i.e. out of parallel). MPR concluded that while tightening this connection, the misalignment could have applied forces on the already-bolted head-to-shell connection which resulted in a loosening of the bolts (a condition now observed twice).
As part of the repair efforts in November 2010, a new tube bundle was installed. Measurements were taken on the old (January 2010) and new tube bundles. The width of the old bundle was approximately 0.023 inches less than the new bundle, which would result in an estimated lower flange misalignment of approximately 0.100 inches when the cooler was reassembled in January 2010. There were no comments in the January 2010 work order regarding flange misalignment by mechanical maintenance. Maintenance personnel interviewed by the licensee also stated they had not noted any misalignment of the lower flange in January 2010. The licensee concluded that the misaligned lower flange caused the loosening of the head-to-shell bolting and that had additional guidance been added to recheck bolt torques after all three flanged connections on the head had been made up, the event could have been precluded.
The inspector reviewed the work instructions used to assemble the joints on the stationary head of the lube oil cooler. The work instructions directed maintenance personnel to utilize procedure MA-MW-736-600, "Torquing and Tightening of Bolted Connections." The procedure and accompanying checklists for making up the bolted connections seemed detailed and incorporated guidance from the site training (sequentially increasing torque, patterned passes, final passes, etc.). Guidance on flange alignment, strategy for dealing with the multiple flanges off of the head, and direction for hot retorquing were not specified in the bolting procedure. The licensee cited these shortfalls in the procedure as a contributing cause. The inspector reviewed the EPRI document, "Assembling Bolted Connections Using Sheet Gaskets," from May 2001, which the licensee referenced in their root cause as containing the information that would have precluded the event. The document was a product of an EPRI Fluid Sealing Technology Working Group which conducted studies and experiments in the late 1990's to explore ways to minimize leakage from sheet gasketed joints. This document had not been reviewed by the licensee prior to the RCE. Specific recommendations from the document included guidance on alignment tolerances and the need to perform hot re-torques within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of assembly of critical sheet gasketed joints. While performing the RCE, the licensee also identified additional EPRI documents from the same series that had not been incorporated into plant procedures. The licensee concluded that had these recommendations been incorporated into their bolting procedures, the conditions leading to the oil leak would have been precluded.
The inspector interviewed maintenance personnel and reviewed mechanical maintenance training lesson plans that addressed bolted connections. The inspector could not identify a reference to a multi-flange connection strategy nor specific detailed guidance on alignment tolerances for flanges in the training. The inspector also reviewed the site's maintenance planning procedure and could not find guidance that would reasonably prompt additional actions to approach the cooler reassembly differently. American Society of Mechanical Engineers (ASME) documents reviewed by the inspector, which were referenced in the licensee's bolting procedure, also did not contain guidance on multi-flange connections. Additionally, the inspector reviewed vendor documentation for the lube oil coolers. The material was very general and did not provide detailed guidance on assembly of the bolted connections. Partially due to the lack of detail in the vendor documentation, the licensee had performed an engineering analysis on what acceptable bolt torques would be for the various connections on the coolers. The analysis specified torque values for the stationary end and the packed, or floating, end of the EDG lube oil coolers. However, a majority of the discussion focused on the floating end, as there had been some OE from another plant regarding loose bolting that was discovered during operation. The floating end differed from the stationary end in that it utilized packing and an elastomeric type gasket material and was designed to 'give' to accommodate thermal transients in the coolers. This OE and the subsequent engineering analysis were in the work package for maintenance performed on the cooler in January 2010. The inspector determined that given the focus on the other end of the cooler, it was not reasonable, based on this particular OE, to include more specific guidance on assembling the stationary head of the coolers. Based on the information reviewed, the inspector determined that the root and contributing causes determined by the licensee were reasonable for the event. c. Determine that the root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience. The inspector determined that the RCE included consideration of prior occurrences of the problem and knowledge of prior operating experience. The RCE examined prior occurrences on site of heat exchanger leakage associated with the EDGs through corrective action program database searches. Most of the issues identified described minor leaks and were associated with the floating (non-affected) head of the heat exchanger. Preventative maintenance activities and the site's leakage monitoring program were being used to monitor for these types of leaks. None of the leaks identified were significant. The licensee also performed industry database searches for similar issues. While this revealed numerous leakage issues from various sites, none were specific to the condition experienced by the licensee with regard to the severe loosening of the head-to-shell bolting. The most relevant OE documents discovered were part of the licensee's root cause and concerned an EPRI document issued before the licensee had established a formal program for incorporating such documents into plant procedures. This document was issued from an EPRI working group which sought to develop guidance for reducing leakage from bolted connections. As described in the RCE, the licensee did not have a formal structure in place to incorporate documents of this type, and as a result, missed an opportunity to incorporate the specific guidance which could have prevented the lube oil leak from occurring. Another EPRI document was published later, however, the site's OE procedure did not include EPRI documents as a potential source of OE. This document was also limited in that it did not include a discussion of hot re-torquing of critical sheet-gasketed joints, which would have alerted site personnel to the loose bolting.
The inspector reviewed the RCE and a common cause analysis performed at the site in 2011 which addressed some weaknesses in the mechanical maintenance department regarding bolted connection work. The inspector also reviewed a sample of issue reports referenced in the common cause analysis, which documented component leakage as a result of inadequate maintenance. While perhaps more documentation could have been outlined in the RCE to document the reasoning why there was not another contributing cause associated with maintenance work practices, the inspector determined that the supporting evidence provided for the documented root and contributing causes was sufficient to arrive at reasonable causes for the event. The inspector noted that although the RCE identified workmanship issues associated with the gasket and that they reasonably did not cause the joint failure for the EDG lube oil leak, that these workmanship issues were being addressed through various additional corrective actions stemming from the RCE. d. Determine that the root cause evaluation addressed extent of condition and the extent of cause of the problem. The inspector determined that the RCE adequately addressed the extent of condition of the problem. The inspector identified a finding of very low safety significance associated with the extent of cause review performed by the licensee.
For the extent of condition review, the licensee investigated the work history of the other seven EDG lube oil coolers on site. None had been worked recently, and the inspector verified this through a review of work orders completed on the EDG lube oil coolers. The licensee also performed torque checks on the lube oil coolers and validated that all were within specification. The licensee checked for similar work that had been performed on the EDG jacket water coolers, which were of similar size and design as the EDG lube oil coolers. No recent work had been performed on these coolers of a similar nature of that which led to the 2A EDG upper lube oil cooler leak.
The inspector reviewed the extent of cause review performed by the licensee. The licensee identified the root cause as the absence of a structured process to ensure EPRI documents were reviewed to capture good work practices. The inspector's discussions with the licensee revealed that a formal process for industry information gathering was not established until 2003. Hence, the 2001 EPRI report on bolted connections was never incorporated into plant work procedures. An additional EPRI document, which provided guidance on gasketed, flanged, bolted joints was also not incorporated into plant procedures. Although issued after 2001, discussions with the licensee and a review of the RCE indicated that the site's operating experience program did not list EPRI documents as a source of operating experience until recently.
Despite the shortfalls with the incorporation of EPRI best-practices, discussion of the extent of cause in the RCE focused on explaining why the formal process established in 2003 would reduce the likelihood of future missed opportunities. The extent of cause evaluation did not adequately explore what other relevant OE or best-practices could have been missed prior to the establishment of the formal industry knowledge transfer process in 2003 or with other ERPI documents outside the scope of the Leak Sealing and Technology series. The regulatory aspect of this issue is documented in Section 4OA4.02.02.f below. e. Determine that the root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in Inspection Manual Chapter (IMC) 0310. The inspector determined that, in general, the root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0310.
The inspector reviewed the RCE and validated the licensee had systematically considered each of the safety culture components. Through their RCE, the licensee identified weaknesses in several of the safety culture components. The inspector reviewed the identified weaknesses and found some were aligned with the root and contributing causes. The licensee identified other weaknesses through their investigation not directly related to the root or contributing causes. The inspector's review of the event did not identify other potential weaknesses in safety culture components.
f. Findings
Introduction:
A finding of very low safety significance (Green) was identified by the inspector when licensee personnel failed to perform an extent of cause review that conformed to the requirements of Licensee Procedure LS-AA-125-1001, "Root Cause Analysis Manual."
Description:
The licensee's RCE documented that the root cause for the White finding was that that there was no formal structured process in place to ensure that EPRI documents were reviewed to capture good work practices and that, as a result, specific EPRI guidance regarding bolted connections was not incorporated into licensee maintenance procedures. Specifically, EPRI issued relevant reports on bolted connections before the licensee had implemented a formal program in 2003 to review and incorporate such documents into licensee procedures, including maintenance procedures. The licensee, in the extent of cause review documented in the RCE, stated that although several EPRI documents within the Sealing Technology and Plant Leakage Reduction Series could have improved the procedure and prevented the loose bolts from being placed in service, it would not be cost effective to perform a detailed review of all EPRI records. Additionally, the inspector reviewed the licensee's Check-In Self Assessment Report that was performed to assess readiness for the NRC Supplemental Inspection. This report stated that based on the volume of EPRI-related documents, an extent of cause evaluation only included issues related to leak sealing technologies.
The licensee's root cause procedure, LS-AA-125-1001, "Root Cause Analysis Manual," directed that an extent of cause must be determined in a RCE and referenced LS-AA-125-1003, "Apparent Cause Evaluation Manual," for guidance. LS-AA-125-1003 defined extent of cause as the extent to which the causes of an identified problem had impacted other plant equipment, organizations, or processes in the same manner identified in the condition report. Contrary to this, the inspector determined that the extent of cause review documented in the RCE was excessively narrow in scope and did not satisfy the requirements of LS-AA-125-1001. Specifically, the extent of cause review performed as documented in the licensee's RCE was narrowly focused on only Sealing Technology and Plant Leakage Reduction Series EPRI documents and failed to more broadly determine whether any other site programs had also been impacted prior to 2003, when a formal program was established.
Analysis:
The failure to perform an adequate extent of cause review for the RCE regarding the 2A EDG lube oil cooler leak as required by LS-AA-125-1001, "Root Cause Analysis Manual," and LS-AA-125-1003, "Apparent Cause Evaluation Manual, was a performance deficiency warranting further evaluation in the significance determination process.
The issue was more than minor because if left uncorrected, it could become a more significant safety concern. Specifically, the licensee's stated root cause of not having a formal process in place to incorporate EPRI documents from the Sealing Technology and Plant Leakage Reduction Series, which led to an inoperable EDG, could also impact other programs or processes. However, the potential impact of the identified root cause on other programs or processes were not reviewed as part of the licensee's extent of cause review effort.
The finding was assessed to impact the Mitigating Systems Cornerstone, as this was the cornerstone impacted by the White finding which resulted in the RCE and subsequent NRC Supplemental Inspection. The inspector determined the finding could be evaluated using the SDP in accordance with IMC 0609, "Significance Determination Process," Attachment 4, "Phase I - Initial Screening and Characterization of Findings," Table 4a for the Mitigating Systems Cornerstone and answered "No" to the Mitigating Systems Cornerstone questions. Therefore, the finding screened as having very low safety significance (Green).
The finding had an associated cross-cutting aspect in the Self and Independent Assessments component of the Problem Identification and Resolution cross-cutting area, because the licensee's assessment on the readiness for the NRC Supplemental Inspection failed to recognize the weakness in the extent of cause review (P.3(a)).
Enforcement:
This finding does not involve enforcement action because no regulatory requirement violation was identified. Because this finding does not involve a violation and has very low safety significance, it is identified as FIN 05000455/2011016-01, Inadequate Extent of Cause for 2A EDG Lube Oil Leak. 02.03 Corrective Actions a. Determine that appropriate corrective actions are specified for each root and contributing cause or that the licensee has an adequate evaluation for why no corrective actions are necessary.
The inspector reviewed applicable corrective actions and the corrective actions to prevent recurrence (CAPR) and determined that the licensee specified reasonable and appropriate corrective actions for each root and contributing cause. The inspector also reviewed implementation of the corrective actions to verify completion status. The CAPR involved Exelon's Industry Participation Program. Two particular procedures within the program were identified in the RCE: CC-AA-1110, "Exelon Nuclear Industry Leadership Plan," and CC-AA-1111, "Exelon Nuclear Industry Participation Process Control." As stated in the RCE, the licensee identified the fact that not having a structured program in place to capture EPRI information allowed industry best-practices to escape incorporation into plant procedures. The two procedures listed above outlined major portions of the program that were not in place when EPRI promulgated guidance on bolted connections in 2001. The formal program established through these procedures was not implemented until about 2 years later. Additionally, the licensee's operating experience procedure, LS-AA-115, "Operating Experience Program," did not include specific guidance to incorporate EPRI documents as a potential source of operating experience until recently. The inspector reviewed the procedures and requested more information on the program. The site provided the inspector examples of industry data obtained through the program and outlined the resources involved in the program. Additionally, the inspector discussed with the corporate personnel the Industry Participation Program. Based on the information reviewed and discussions with the licensee, the inspector concluded that the CAPR was appropriate given the root cause. The contributing cause was identified as inadequate procedural direction on the assembly of multiple joint configurations. As stated before, the licensee first considered this to be the root cause. Many corrective actions were listed in the RCE to address the contributing cause and included the following: 1) Revise the bolting procedure used onsite to reflect EPRI guidance, 2) Update the EDG heat exchanger model work orders to require final torque checks after reassembly, and 3) Review Braidwood Station preventative maintenance work instructions for best work practices. Additionally, numerous corrective actions that addressed training in the mechanical maintenance department were listed. These included multiple flange connection training (to include mechanical maintenance planners), a case study review of the event with mechanical maintenance personnel and planners, dynamic learning activities on multiple flanged joints utilizing the revised bolting procedure, and EPRI training on bolted joints.
The procedure utilized by the licensee for bolted connections was an Exelon fleet procedure. The licensee had recommended a number of revisions to this fleet procedure, but in the event that bolted connection work was performed before the revisions were implemented, the licensee revised the fleet procedure for use on site based on the results of the RCE. The inspector identified one concern with a new step driven by the root cause corrective action to incorporate EPRI guidance. This new step required mechanical maintenance personnel to measure parallelism between flanges to identify a potentially "misaligned" joint. The inspector was concerned about what would constitute a "misaligned" joint before measurements were taken. The licensee generated an issue report to determine whether additional guidance to further enhance the procedure was warranted. b. Determine that corrective actions have been prioritized with consideration of the risk significance and regulatory compliance. The inspector concluded that the licensee adequately prioritized the corrective actions with consideration of the risk significance and regulatory compliance. The licensee's corrective actions appeared to be prioritized commensurate with the safety significance of the issues. Initial actions taken in response to the leak were to identify the cause and ensure that the heat exchanger would be repaired such that it could fulfill its design function. As described above, the licensee identified issues with how the stationary end of the heat exchanger had been installed, to include misalignment of flange faces. The licensee located EPRI guidance on bolted connections and incorporated the guidance into the repair efforts. The licensee performed an engineering change to reassemble the flanged connections on the 2A EDG lube oil cooler as it was necessary to machine some flange faces and modify some bolting to attain proper alignment. The inspector reviewed the engineering change, the EPRI guidance, and the work orders which reassembled the stationary head and found no issues. Additionally, the licensee performed a torque check on the 2A EDG lube oil cooler to validate torques in May 2011. The inspector reviewed the results of the associated work order, which stated no loosening had occurred. To ensure the necessary guidance was available for further bolted connection work should the need arise, the licensee revised the fleet procedure in the form of an onsite procedure that could be used by mechanical maintenance until a fleet procedure revision could be made. In summary, the licensee appropriately prioritized actions to repair the 2A EDG upper lube oil cooler leak and ensured that actions to prevent leaks for similar reasons were implemented. Other actions to address the root or contributing causes appeared to be appropriately scheduled or had already been completed.
c. Determine that a schedule has been established for implementing and completing the corrective actions. The inspector determined that the licensee adequately established a schedule for implementing and completing the corrective actions. The licensee assigned completion due dates that were commensurate with the safety significance of the issues being addressed as well as the level of effort required to complete the actions. Completion dates were being tracked in the corrective action program. d. Determine that quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence. The inspector determined that the licensee adequately developed quantitative or qualitative measures of success for determining effectiveness of the corrective actions to prevent recurrence.
The inspector reviewed the effectiveness review plan for the CAPR. As outlined in the RCE, the licensee planned to perform a focused area self assessment (FASA) of the CAPR. The CAPR was to validate that EPRI technical reports were being properly screened and dispositioned for potential applicability per Exelon procedures. The inspector also reviewed the procedure for performing FASAs. While the inspector agreed that a FASA would be an appropriate tool to assess the effectiveness of the CAPR, at the end of this supplemental inspection the licensee had not yet developed the specific details for conducting the FASA. The inspector discussed this issue with the licensee, and the licensee initiated an issue report to document the concern. The issue report included recommended metrics to be used when the FASA was conducted, and was available for inspector review before completion of this supplemental inspection.
Based on this review and the rigor involved in the FASA procedure, the inspector determined that the scheduled effectiveness review would be of appropriate scope for the CAPR and included appropriate measures of success for determining the effectiveness of the corrective actions. e. Determine that the corrective actions planned or taken adequately address the Notice of Violation that was the basis for the supplemental inspection. The inspector concluded that the corrective actions planned or taken adequately addressed the Notice of Violation.
The Notice of Violation associated with the White finding that was the subject of this IP 95001 inspection identified one violation of NRC requirements. In particular, a violation of 10 CFR 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," for the improper assembly of the 2A EDG upper lube oil cooler in January 2011 was identified. In the Notice of Violation, the NRC documented that the corrective actions taken and planned to be taken to correct the violation and prevent recurrence, and the date when full compliance was achieved, was already adequately addressed on the docket in Inspection Report 05000455/2011011. The inspector reviewed the referenced inspection report and determined there were no additional concerns with regard to addressing the Notice of Violation.
f. Findings
No findings were identified. 02.04 Evaluation of Inspection Manual Chapter 0305 Criteria for Treatment of Old Design Issues The licensee did not request credit for self-identification of an old design issue; therefore, the risk-significant issue was not evaluated against the IMC 0305 criteria for treatment of an old deisgn issue.
4OA5 Other
.01 (Closed) Violation 05000455/2011011-01, "Self-Revealing Failure of the 2A Diesel Generator Upper Lube Oil Cooler."
The inspector determined that the licensee's RCE was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event. The inspector also concluded that the licensee identified reasonable and appropriate corrective actions for each root and contributing cause and that the corrective actions appeared to be prioritized commensurate with the safety significance of the issues. No other instance of the violation was identified. This violation is closed.
4OA6 Meetings, Including Exit
.01 Exit Meeting Summary
The inspector presented the inspection results to Mr. T. Tulon and other members of the licensee management on August 26, 2011. Proprietary material received during the inspection was returned to the licensee and was not included in this report.
.02 Regulatory Performance Meeting
On August 26, 2011, the NRC met with the licensee to discuss its performance in accordance with IMC 0305, Section 10.02.b.4. During this meeting, the NRC and licensee discussed the issues related to the White finding that resulted in Byron Station, Unit 2, being placed in the Regulatory Response Column of the NRC's ROP Action Matrix. This discussion included the causes, corrective actions, extent of condition, extent of cause, and other planned licensee actions.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee
- T. Tulon, Site Vice President
- B. Adams, Plant Manager
- D. Gudger, Regulatory Assurance Manager
- B. Spahr, Maintenance Manager
- P. O'Neill, Program Specialist/Maintenance
- E. Hernandez, Engineering Director
- M. Eikstat, Nuclear Oversight
- B. Youman, Operations Director
- D. Coltman, Work Management
- S. West, Director, Division of Reactor Projects NRC
- E. Duncan, Branch Chief, Division of Reactor Projects, Branch 3
LIST OF ITEMS
OPENED, CLOSED AND DISCUSSED
Opened
- 05000455/2011016-01 FIN Inadequate Extent of Cause for 2A EDG Lube Oil Leak
Closed
- 05000455/2011011-01 VIO Self-Revealing Failure of the 2A Diesel Generator Upper Lube Oil Cooler
- 05000455/FIN-2011016-01 FIN Inadequate Extent of Cause for 2A EDG Lube Oil Leak
Discussed
None
Attachment
LIST OF DOCUMENTS REVIEWED
The following is a list of documents reviewed during the inspection. Inclusion on this list does not imply that the NRC inspector reviewed the documents in their entirety, but rather, that selected sections of portions of the documents were evaluated as part of the overall inspection effort. Inclusion of a document on this list does not imply NRC acceptance of the document or any part of it, unless this is stated in the body of the inspection report.
- IP 95001 - Inspection for One or Two White Inputs in a Strategic Performance Area - Root Cause Evaluation
- 1166626, 2A Diesel Generator Lube Oil Cooler Leak, July 20, 2011, Revision 1 - Check-In
- 1204502, Self Assessment for the NRC White Finding Inspection for the Failure of the 2A EDG Lube Oil Cooler -
- EC 382271, Evaluation of Turning Down Threads on the Studs for the 2A DG Upper Lube Oil - 1BOSR 8.1.2-2, Monthly Operability Surveillance -
- CC-AA-1110, Exelon Nuclear Industry Leadership Plan, Revision 2 -
- CC-AA-1111, Exelon Nuclear Industry Participation Process Control, Revision 3 -
- LS-AA-115, Operating Experience Program, Revision 17 -
- LS-AA-125-1001, Root Cause Analysis Manual, Revision 8 -
- LS-AA-125-1003, Apparent Cause Evaluation Manual, Revision 9 -
- LS-AA-125-1004, Effectiveness Review Manual, Revision 5 -
- LS-AA-126-1001, Focused Area Self Assessments, Revision 6 -
- MA-AA-716-010, Maintenance Planning, Revision 17 -
- MA-MW-736-600, Torquing and Tightening of Bolted Connections, Revision 3 -
- MA-BY-736-600, Torquing and Tightening of Bolted Connections, Revision 4a -
- LER 2011-001-01, Unit 2 Emergency Diesel Generator Inoperable for Longer than Allowed by Technical Specifications Due to Inadequate Work, June 22, 2011 - EPRI
- 1015337, Assembling Gasketed, Flanged Bolted Joints - ITT Fluid Technology Corporation Manual for 2A EDG Lube Oil Cooler - Mechanical Maintenance Q3-11 Human Performance Improvement Plan - Technical Evaluation 96-033, Compressed Asbestos Gasket Generic Guidelines, Revision 0 -
- IR 1239812, Cantera Requested Revision of
- RCR 1166626, July 14, 2011 -
- IR 1255464, During
- IP 95001 Inspection, EFR Criteria Not Adequate, August 25, 2011 -
- IR 1255650, Extent of Cause of
- RCR 1166626 Lacks Documentation, August 25, 2011 -
- IR 1204502, Provide Documented Review of Exelon Nuclear Industry Participation Program, August 11, 2011 -
- IR 1164479 Common Cause Analysis: Workmanship Issues with Respect to Bolted Connections on Safety Related Systems, February 14, 2011 -
- IR 1221643 Common Cause Analysis: Byron Station B1R17 Rework, June 24, 2011 -
- IR 1065038, Significant Lube Oil Leak during PMT Run of 2B AF Diesel, May 4, 2010 -
- IR 1164479, 1VA04SA Leaked during Leak Test, January 19, 2011 -
- IR 1190243, Evaluate Workmanship Issues Identified During Root Cause, March 21, 2011 -
- IR 1142078, Indications Found on Seating Surface of the Cooler Flange, November 18, 2011 - Training Module
- GC 40301, Bolted Connections, Revision 6 - Training Module
- MC 20605, Hydraulic Torque Equipment, Revision 4 - Training Module
- MC 20109, Gasket Fabrication, Revision 3 -
- WO 1206254, Clean Tube Side of Lube Oil Coolers -
- WO 1387717, Clean Tube Side of Lube Oil Coolers Attachment -
- WO 1438893, Proof Torque 2A EDG Upper LO Cooler Stationary Head Bolts Cooler Bottom Cooling Water Flange Attachment
LIST OF ACRONYMS
- USED [[]]
- M. [[Pacilio -2- reasonable and appropriate corrective actions for each root and contributing cause and that the corrective actions appeared to be prioritized commensurate with the safety significance of the issues. Several observations regarding specific aspects of your root cause evaluation and corrective actions that warrant additional consideration by your staff were also identified. Based on your overall acceptable performance in addressing the White finding that was the subject of this inspection, in accordance with the guidance in Inspection Manual Chapter (]]
- IMC ) 0305, "Operating Reactor Assessment Program," the White finding will only be considered in assessing plant performance for a total of four quarters (i.e. through the fourth quarter of 2011). Based on the results of this inspection, one
- NRC -identified finding of very low safety significance (Green) was identified. The finding did not involve a violation of
- NRC requirements. If you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region
- NRC Resident Inspectors at the Byron Station. The information you provide will be considered in accordance with
- NRC 's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any), will be available electronically for public inspection in the
- NRC Website at http://www.nrc.gov/readingrm/ adams.html (the Public Electronic Reading Room). Sincerely, /RA/ Eric
- NPF -66 Enclosure: Inspection Report 05000455/2011016 w/Attachment: Supplemental Information cc w/encl: Distribution via ListServ
- DOCUME [[]]
- NT [[]]
- DRPIII \BYRO\Byron 2011 016.docx Publicly Available Non-Publicly Available Sensitive Non-Sensitive To receive a copy of this document, indicate in the concurrence box "C" = Copy without attach/encl "E" = Copy with attach/encl "N" = No copy
- OFFICE [[]]
- RIII [[]]
- RIII [[]]
- NAME [[]]
- RECORD [[]]
- BYRON [[]]
- NRC [[]]
- INSPEC [[]]
- TION [[]]
- PROCED [[]]
- URE [[]]
- 95001 SUPPLE [[]]
- MENTAL [[]]
- INSPEC [[]]
- TION [[]]
- REPO [[]]