IR 05000318/2011008

IR 05000318-11-008, on 03/16/2011 - 03/18/2011, Calvert Cliffs Nuclear Power Plant, Unit 2, Supplemental Inspection - Inspection Procedure 95001

ML111190104
Person / Time
Site:	Calvert Cliffs
Issue date:	04/29/2011
From:	Glenn Dentel Reactor Projects Branch 1
To:	George Gellrich Constellation Energy Nuclear Group
References
IR-11-008
Download: ML111190104 (23)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION

REGION I

475 ALLENDALE ROAD KING OF PRUSSIA. PA 19406.1415

SUBJECT:

CALVERT CLIFFS NUCLEAR POWER PLANT, UNIT 2 - NRC INSPECTION PROCEDURE 95001 SUPPLEMENTAL INSPECTION REPORT 4500a318t201 1008

Dear Mr. Gellrich:

On March 18,2011, the U.S. Nuclear Regulatory Commission (NRC) staff completed a supplemental inspection in accordance with Inspection Procedure 95001, "Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area," at your Calvert Cliffs Nuclear Power Plant Unit 2. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on March 18,2011, with you and other members of your staff.

As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed because a finding of White safety significance was identified in the second quarter of 2010. This issue was documented previously in NRC Special Inspection Report 0500031712010006 and 0500031812010006. The NRC staff was informed on February 11, 2011 , of your staff's readiness for this inspection.

The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes for the risk-significant issues were understood; (2) the extent of condition and extent of cause of the issues were identified; and (3) corrective actions were or will be sufficient to address and preclude repetition of the root and contributing causes.

The inspection consisted of examination of activities conducted under your license as they related to safety, compliance with the Commission's rules and regulations, and the conditions of your operating license.

The inspectors determined that your staff performed a comprehensive evaluation of the White finding. Your actions in response to the White finding resulted in a root cause analysis that appropriately addressed the root and contributing causes, extent of condition and extent of cause, and corrective actions.

Your staff identified that a failure of the 28 emergency diesel generator (EDG) lube oil pressure monitoring circuit caused the 28 EDG to trip shortly after it started in response to the loss of power to the plant's emergency busses following the dual unit trip on February 18, 2010. The 28 EDG tripped due to the slow response of an engine lube oil pressure indicator and the early timeout of a time delay relay that were both associated with the EDG's lube oil pressure monitoring circuit. Your statf concluded that these equipment conditions were caused by an ineffective preventive maintenance program for the 28 EDG. The ineffective program was the result of the use of an inappropriate engineering change process to delete the requirement for periodic Agastat relay replacements and the use of poor work practices and inadequate proceduraiguidance when venting, flushing and calibrating pressure indications for the EDG lube oil system.

The corrective actions you completed to address these issues included revising the preventive maintenance strategy for the associated Agastat relays; implementing additional preventive maintenance requirements for EDG lube oil pressure sensing lines; revising procedures to improve work controls during maintenance and calibrations performed on EDG lube oil pressure indication components; and implementing a formal plan to monitor and detect degradation of lube oil pressure monitoring circuit components, including the Agastat relays. You also completed extent of condition actions to confirm similar equipment conditions did not exist on the other EDGs. Your extent of cause reviews are in progress and are intended to assess the adequacy of the preventive maintenance strategies for the plant's most risk significant systems, review the work controls for maintenance and calibrations for pressure indication components in those systems, and assess the adequacy of engineering change processes to ensure proposed changes are fully evaluated and that all processes and programs affected by the change are addressed. The results of these reviews will determine the need for more extensive reviews and corrective actions in these areas.

Based on the results of this inspection, no findings were identified.

Given your acceptable performance in addressing the loss of 28 EDG event and in accordance with thb guidance in lnspection Manual Chapter (lMC) 0305, "Operating Reactor Assessment Programl'the White finding will only be considered in assessing plant performance for a total of four quarters.

ln accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS).

ADAMS is accessible from the NRC Website at http://www.nrc.qov/readinqrrm/adams.html (the Public Electronic Reading Room).

lurilH

Sincerely, Glenn T. Dentel, Chief Projects Branch 1 Division of Reactor Projects Docket No.: 50-318 License No.: DPR-69

Enclosure:

Inspection Report 05000318/201 1008 w/ Attachment: Supplemental Information

REGION 1 Docket No.: 50-318 License No.: DPR.69 Report No.: 0500031 8i201 1 008 Licensee: Calvert Cliffs Nuclear Power Plant, LLC Facility: Calvert Cliffs Nuclear Power Plant, Unit 2 Location: Lusby, Maryland Dates: March 16,2011, through March 18,2011 lnspectors: L. Cline, Senior Project Engineer, Lead lnspector L. Casey, Resident lnsPector Approved by: Glenn T. Dentel, Chief Projects Branch 1 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

lnspection Report (lR) 0500031812011008; 0311612011 - 0311812011; Calvert Cliffs Nuclear

Power Plant, Unit 2; Supplemental lnspection - Inspection Procedure (lP) 95001 A senior project engineer and a resident inspector performed this inspection. The NRC's program ior overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Cornerstone: Mitiqatinq Svstems

The NRC staff performed this supplemental inspection in accordance with lP 95001,

"supplemental lnspection for One or Two White lnputs in a Strategic Performance Area," to assess Calvert Cliffs Nuclear Power Plant, LLC's (CCNPP's) evaluation of the failure of the 28 emergency diesel generator (EDG) during a demand start that occurred following the dual unit trip oiCatvert CliffJ Units 1 and 2, with the concurrent loss of supplied loads, on February 18,

ZOIO. The NRC staff previously characterized this issue as having low to moderate safety significance (White) as documented in NRC Special lR 0500031712010006 and OdOOOgtAIZCitOOOO. The inspectors concluded, based on the results of this inspection, that for the risk significant performance issues associated with the February 18, 2010,28 EDG failure,

CCNPP idlntified all root and contributing causes, appropriately addressed extent of condition and extent of cause, and assigned appropriate corrective actions to prevent recunence'

CCNP determined the cause of the 28 EDG trip was a faulted lube oil pressure monitoring circuit. CCNPP identified the primary causes for the faulted EDG lube oil pressure monitoring circuit to be implementation of an inappropriate change to the preventive maintenance strategy for the Agastat relays used in the Fairbanks Morse EDG lube oil pressure monitoring circuit and-controls inappropiiate work and inadequate preventive maintenance for Fairbanks Morse EDG lube oil pressure sensing lines used in the lube oil pressure monitoring circuit. Corrective actions ior these issues lncluded revising the preventive maintenance strategy for the associated Agastat relays; implementation of additional preventive maintenance requirements for EDG luOe oit pressuie sensing lines; procedure revisions that improve work controls during maintenance and calibrations performed on the EDG lube oil pressure indication components; and implementation of a formal plan to monitor and detect degradation of lube oil pressure monitoring circuit components, including the Agastat relays. CCNPP's extent of condition actions confirmed similar equipment conditions did not exist on the other EDGs. Extent of cause reviews will assess the'adequacy of the preventive maintenance strategies for the plant's most risk significant systems, review the work controls for maintenance and calibrations for pressure indication components in those systems, and assess the adequacy of engineering change processes to ensure proposed changes are fully evaluated and that all processes and progi"rs affected by the change are addressed. The results of these reviews will determine the need for more extensive reviews and corrective actions in these areas.

Given CCNPP's acceptable performance in addressing the performance issues revealed by the February 18,2010, 28 EDG failure, the White finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in NRC lnspection Manual Chapter 0305, Operating Reactor Assessment Program'

Findinos No findings were identified.

REPORT DETAILS

4OA4 Supplemental Inspection

.01 lnspection Scope

The NRC staff performed this supplemental inspection in accordance with lP 95001 to assess Calvert Cliffs Nuclear Power Plant, LLC's (CCNPP's) evaluation of a White 1nding that affected the Mitigating Systems cornerstone in the Reactor Safety strategic performance area.

The objectives for this inspection were to:

r Provide assurance that the root and contributing causes of risk-significant issues were understood; r Provide assurance that the extent of condition and extent of cause of risk-significant issues were identified; and r Provide assurance that CCNPP's corrective actions for risk-significant issues were or will be sufficient to address the root and contributing causes and to preclude rePetition.

Calvert Cliffs Unit 2 entered the Regulatory Response Column of the NRC's Action Matrix in the second quarter of 2010 as a result of one inspection finding of low to moderate safety significance (White). The finding was specifically associated with the failure of the Zfi eOC during a demand start that occurred following a dual unit trip of Catvert Cliffs Units 1 and 2 with the concurrent loss of supplied loads on February 18, 2010. The initiator for the event was water intrusion from the roof into the auxiliary building switchgear room that caused a fault in the switch gear cabinets that cascaded into trifs of both Unit 1 and 2 because of relay failures. The 28 EDG received a valid start signal due to under voltage on the 24 4kilovolt (kV) bus; the engine started and began to accept load, but then tripped. The cause of the EDG trip was a faulted low lubl oil pressure monitoring circuit. Sufficient oil pressure had not developed in the lube oil sensing lines and prevented the pressure switches from resetting at the nominal time; additionally, one of the circuits' time delay relays (T3A) timed out early_and armed the low lube oil pressure trip prematurely. This resulted in the trip of the 28 EDG. The finding was characterized as having a White safety significance based.on the results of a Phase 3 risk analysis with an exposure time of 323 days that resulted in a total calculated conditibnal core damage frequency of 7.1E-6. The finding and details of the preliminary risk assessment analysis are discussed in NRC Special lnspection Report bSOOOetZ72010006 and 05000318/201006. The risk significance determination for the finding was finalized by an August 3, 2010, NRC letter to CCNPP. This letter also issued Notice of Violation EA-10-080 that was associated with this finding.

CCNPP staff informed the NRC staff on February 11,2011, that they were ready for the supplemental inspection. ln preparation for the inspection, CCNPPLcerformed a caiegory 1 root cause analysis report (RCAR), Condition Report (CR)-2010-007157' Failure of tne 28 EDG during the Dual Unit Trip, Revision 2, to identify the equipment and organizational causes that led to the White finding.

The inspectors reviewed CCNPP's RCAR and other evaluations conducted to support the RCAR. The inspectors reviewed corrective actions that were taken or planned to address the identified causes. The inspectors also discussed with CCNPP personnelto ensure that the root and contributing causes and the contribution of safety culture components were understood, and that the corrective actions taken or planned were appropriate to address the causes and preclude repetition.

.02 EEluatjsl of the Inspection Requirements

02.01 Problem ldentification a.

lP 95001 requires that the inspection staff determine that the licensee's evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions under which the issue was identified.

CCNPP determined that the 28 EDG trip was caused by a lube oil pressure monitoring circuit relay (T3A) timing out earlier than normal and "sticky lubrication oil" that caused 28 EDG indicated engine lube pressure to be less than the relay setpoint when the T3A relay timed out. In accordance with NRC Inspection Manual Chapter 0612 definitions for licensee-identified, self-revealing, or NRC-identified, CCNPP's RCAR identified these equipment issues as self-revealing.

The NRC completed a specialteam inspection to review CCNPP's response, cause evaluation and corrective actions for the dual unit trip. The results of this inspection were documented in NRC Special Inspection Report 05000317/2010006 and 005000318/2010006. The special inspection team determined that the failed T3A relay was installed for 13.5 years, 3.5 years beyond the manufacturers recommended service life.

In 2001, CCNPP had approved extending the service life for these relays based on the implementation of an Agastat relay performance monitoring program that was not formally implemented. The special inspection team determined that this resulted in an "age-related" failure of the T3A relay that caused the 28 EDG trip on February 18,2010, and that this was a performance deficiency.

The NRC issued an NRC-identified Notice of Violation (NOV) for Technical Specification 5.4.1, which specifies that written procedures shall be established for activities listed in Regulatory Guide 1.33 Appendix A, including a replacement schedule for components with a specified service life. ln accordance with NRC lnspection Manual Chapter 0612 definitions for licensee-identified, self-revealing, or NRC-identified, CCNPP's RCAR identified this performance deficiency as NRC-identified.

The inspectors determined that CCNPP's RCAR for the event appropriately documented who identified the equipment issues and the performance deficiencies for this event and under what conditions the equipment issues and performance deficiencies were identified.

b.

lP 95001 requires that the inspection staff determine that the licensee's evaluation of the issue documents how long the issue existed and prior opportunities for identification.

CCNPP's evaluation was unable to determine exactly how long the two conditions - the sticky lubrication oil and the degraded Agastat relay - had existed to a degree that they impacted the ability of the 28 EDG to perform its safety function. As a result, the risk significance determination for the 2B EDG failure was based upon the last known salisfactory calibration results for the lube oil pressure monitoring circuit that was compfeted on May 13,2008.

CCNPP's evaluation determined that programs and processes in place during the period of time that led up to the 28 EDG trip were not capable of identifying that the lubricating oil in the system pressure sensing lines and the Agastat relay were degrading so that actions could be taken before the operability of the EDGs was impacted' CCNPP identified that, without performing additional monitoring of lube oil pressures and Agastat relay performance, only a cold fast EDG start could identify degradation of lube oil pressure monitoring circuit components. However, cold fast test starts were discontinued at Calvert Cliffs in September 1994, in accordance with a November 2, 1gg3, Calvert Cliff's license amendment request to eliminate the requirement to perform cold fast test starts.

CCNPP determined that this request was submitted based on NRC generic correspondence (Generic Letter 84-15) that suggested, due to the wear and tear caused by cold fast EDG starts, licensees should evaluate the need to perform them. CCNPP aiso determined that the NRC approved the request on February 24,1994; but that, at that time, CCNPP did not recognlze the need for or implement the additional monitoring necessary to identify degradation of the lube oil pressure monitoring circuit components.

CCNPP developed corrective actions to evaluate the testing and preventive_

maintenance programs for the EDGs. To address this identified gap in EDG system testing, CCNPP froceduralized additional monitoring for the Fairbanks Morse EDG lube oil pre-isure monitoring circuit components in order to detect degradation during the monthly EDG surveillance testing. CCNPP will also conduct an extent of condition review for the plant's most risk significant systems to identify potential vulnerabilities in the testing programs for these systems as well. The evaluations were scheduled to be completel in May 2011. The results of this review will determine the need for additional evaluation and corrective actions in this area.

The inspectors determined that, based on the information available, the RCAR for the 28 EDG trip appropriatety documented how long the performance deficiency existed and prior opportunities foiidentification. CCNPP also implemented appropriate changes to in" f"irbrnks Morse EDG preventive maintenance and testing programs and initiated an extent of condition review to evaluate the need for adjustments in this area for other risk significant systems. The inspectors reviewed the proceduralized monitoring program for th-e Fairbanis Morse EDG lube oil pressure monitoring circuit components and concluded that the new procedure should identify circuit degradation in time to allow corrective action to be completed before EDG operability is impacted.

c. lp 95001 requires that the inspection staff determine that the licensee's evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issue(s).

The NRC determined this issue was a White finding, as documented in NRC Special lnspection Report 0500031712010006 and 0500031812010006, and CCNPP's RCAR also acknowledged that the finding associated with this issue had White safety significance. CCNPP's RCAR documented that the consequences of the issue included:

complicating the operator's response to the February 18, 2410, dual unit trip; additional maintenance testing and repair costs and out of service time for mitigating systems equipment in order to correct the condition and fully identify the extent of condition and cause; and an increase in Unit 2's baseline core damage frequency by a factor of 3.68.

CCNPP also reviewed potential consequences for the failure had it occurred during a different initiating event, and determined that had the plant experienced a complete loss of offsite power with the 28 EDG failure and with the same equipment out of service for maintenance as on February 18,2010, Unit 2 would have met the criteria for an Alert emergency declaration. Therefore, on the day of the event, the potential existed for one additional equipment fault to place each Calvert Cliffs Unit in a plant condition that required an emergency declaration.

CCNPP also acknowledged the violation of Technical Specification 5.4.1 that required maintenance of written procedures for preventive maintenance items. CCNPP provided a response to the NRC Notice of Violation issued on August 3,2010, in a letter dated September 2,2010. CCNPP acknowledged the White finding and violation, discussed the causes, corrective actions to prevent recurrence, and stated that full compliance would be achieved on January 15, 2011.

The inspectors concluded that CCNPP appropriately documented the risk consequences and compliance concerns associated with the issue. The inspectors also confirmed, through the review of corrective actions completed, that full compliance with regulations was restored on January 13,2011.

d.

Findinos No findings were identified.

02.02 Root

Cause.

Extent of Condition. and Extent of Cause Evaluation a.

lP 95001 requires that the inspection staff determine that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.

The initial root cause for the dual unit trip was reviewed by the special inspection team

, that reviewed CCNPP's response to the February 18, 2010, dual unit trip. In response to the issues identified by the NRC inspectors during that inspection, CCNPP initiated a new root cause evaluation. The final RCAR for CR-2010-007157, which addressed the issues identified during the special inspection, was approved by management on January 19,2011. CCNPP used the following systematic methods to complete its evaluation: failure modes and effects analysis; events and causal factor charting; and Management Oversight and Risk Tree (MORT) methodology. The techniques used were applied in accordance with CCNPP procedure CNG-CA-1.01-1004, Root Cause Analysis, and the CCNPP handbook, CNG-CA-1.01-GL002, Causal Analysis Handbook.

The inspectors reviewed the RCAR, the CCNPP procedure and handbook. The inspectors also discussed the events and causal factors chart and the MORT methodology results with the members of the RCAR team. The inspectors' review confirmed that the results of the evaluation were based on rigorous application of these systematic evaluation methods.

b. lP 95001 requires that the inspection staff determine that the licensee's RCAR was conducted to a level of detail commensurate with the significance of the issue.

CCNPP's RCAR used systematic evaluation methods to identify the technical causes for the 28 EDG failure, but also appropriately used the MORT methodology to identify the organizational root and contributing causes that led to the technical causes resulting in the 28 EDG demand failure. The RCAR identified inadequacies in the preventive maintenance program and the work control process. Breakdowns in the corrective action program (CAP), engineering change processing and inconsistent application of standards and expectations for CCNPP's use and adherence to procedures and processes, in particular related to maintenance work practices, also contributed to the failures on a lower level. CCNPP also reviewed programmatic weaknesses in the lssue Response Team (lRT) process that led to the root cause evaluation weaknesses identified by the special inspection team.

CCNPP determined that the cause of the Agastat relay failure was a March 2001 decision to extend the service life of the Agastat E7000 series relays, which were used in the EDG lube oil pressure monitoring circuit, beyond the manufacturer specified 10 year service life. At that time, in the interest of efficiency, CCNPP implemented the change to the preventive maintenance program during a minor revision to E-406, the Electrical Design Standard. The minor revision was completed using the configuration document change (CDC) process in accordance with EN-1-101 , Design Change and Modification lmplementation. The change relocated certain information from the E-406 drawing 61406SEC234SH0001 to vendor technical manual 15167-001 and, at the same time, eliminated the requirement to replace the Agastat relay every 10 years.

The CDC process did not ensure that the impact of this change on the preventive maintenance program for the Agastat relays was fully evaluated. ln addition, using the CDC process did not ensure that this change received an appropriate level of independent review. As a result, the 10 year replacement requirement was eliminated based upon the implementation of an informal monitoring program not documented or tracked by the preventive maintenance program. Therefore, when the component engineer who performed the informal monitoring was moved to a new position in November 2002, the monitoring was no longer performed.

The CDC process is no longer used at Calvert Cliffs. All engineering change proposals are now processed through a single screening process defined in CCNPP procedures CM-1.01-1003, Design Engineering and Configuration Control, and CNG-FES-O07, Preparation of Design Inputs and Change lmpact Screen. CCNPP corrective actions confirmed that, had this process been used in 2001, it would have ensured that appropriate evaluations and reviews were completed before implementation of the change.

CCNPP determined that the cause of the entrapped air and contamination in the lube oil sensing lines was a lack of procedural guidance for pressure switch calibrations and sensing line flushing and refilling. The lack of guidance relative to testing medium, connection points for the calibration procedures, and the lack of periodic maintenance I

tasks to inspect, drain and refill the sensing lines, caused the build-up of entrapped air and contaminants that slowed the response time of the 28 EDG lube oil pressure indication. This slow response in indicated lube oil pressure caused the 28 EDG to trip during start-up on February 18,2010, after the T3A relay timed out early and armed the lube oil pressure trip circuit.

The inspectors discussed the events and casualfactors chart, and the MORT methodology results with the members of the RCAR team. The inspectors concluded that the results of the evaluation were based on rigorous application of these systematic evaluation methods and were conducted to the appropriate level of detail that ensured organizational weaknesses were identified and corrected.

c. lP 95001 requires that the inspection staff determine that the licensee's RCAR included a consideration of prior occurrences of the issue and knowledge of operating experience.

CCNPP's cause evaluation identified low level industry external operating experience items dealing with air entrapment in EDG lube oil pressure sensing lines and sticky lubrication oil in EDG lube oil pressure sensing lines, but did not identify external operating experience related to the specific Agastat relay failure. CCNPP concluded that the applicable low level items were missed opportunities, but also determined that the CCNPP operating experience program, as defined by procedure CNG-CA-1.01-1010, Use of Operating Experience, did not require actions to be taken in response to these items.

CCNPP's evaluation also identified a self-assessment of the CCNPP relay program, performed in September 2005, in response to a significant external operating experience item (as defined by Attachment 1 of CNG-CA-1.01-1004) that described the results of an industry review of relay related failures that contributed to automatic and manual scrams, The results of the self-assessment were of interest to this cause analysis because, although the external operating experience item that prompted the self-assessment did not directly relate to the issues associated with the February 19,2010, 28 EDG trip, the assessment recommended creation of a relay component health report and the assignment of a component engineer to track relay performance. CCNPP determined that these recommendations were not implemented because they were not entered into the site's CAP for tracking. CCNPP concluded that, had these two recommendations been implemented, CCNPP may have identified the existing weakness in the testing and preventive maintenance program for Agastat relays before it resulted in the 28 EDG trip on February 19,2010. To identify improvements in this area, CCNPP initiated CR-2010-012114 to review their performance relative to the self-assessment and operating experience programs and how these two programs interface with the CAP.

CCNPP's review of internal operating experience identified one auxiliary feedwater system relay failure in 2009 that caused excessive time delay (CR-2009-002150). lt is important to note that the relay that failed was not an Agastat relay. However, the apparent cause of the excessive time delay was determined to be age-related degradation because the subject relay had been installed for over 10 years. At the time of that failure, CCNPP determined that the failure mechanism did not have the potential to affect other systems. Had the cause analysis for this failure determined that all systems with time delay relays installed could have been affected, the degraded condition of the 28 EDG relay may have been identified. Corrective actions for this I

issue included training for the site's root cause evaluators and, on a temporary basis until training of onsite evaluators was completed, the use of root cause experts outside of CCNPP to ensure completeness and thoroughness of evaluations.

The inspectors reviewed the internal and external operating experience items that CCNPP identified as prior occurrences and missed opportunities and determined that the identified items, of which the most significant were discussed above, did not involve situations where CCNPP would have been required to take actions that would have prevented the 28 EDG failure on February 18,2010. In addition, the inspectors determined that the corrective actions CCNPP put in place to address the weaknesses that were identified in this area, would improve the depth of internal cause evaluations and the screening and processing of both internal and external operating experience items.

The inspectors concluded that CCNPP's RCAR included appropriate consideration of prior occurrences of the problem and knowledge of prior operating experience, and how the handling of these items may have impacted the outcome of the February 19,2010, 28 EDG failure.

d. lP 95001 requires that the inspection staff determine that the licensee's RCAR addresses the extent of condition and extent of cause of the issue(s).

CCNPP's evaluation considered the extent of condition associated with the thickened oil in the 28 EDG sensing lines and age-related degradation for Agastat relays installed in critical applications. For extent of cause, CCNPP assessed the adequacy of the preventive maintenance strategies for the plant's most risk significant systems (systems tracked by the mitigating systems performance index (MSPI)), reviewed the work controls for maintenance and calibrations for pressure indication components in those same systems, and assessed the adequacy of engineering change processes to ensure proposed changes were fully evaluated and that all processes and programs affected by the change were addressed.

After the event, CCNPP flushed the lube oil sensing lines on the 28 EDG and the other two Fairbanks Morse EDGs (2A and 1B). Additionally, CCNPP determined that the sensing lines of several other systems could also be susceptible to similar clogging and need flushing and/or fill and vent procedures to mitigate this potential. CCNPP did not limit its review to lube oil systems, but included an assessment of pressure indications for salt water and borated water systems as well. CCNPP initiated corrective actions to evaluate the need for flushing and/or fill and vent procedures for each susceptible system.

To address age-related degradation of Agastat relays, CCNPP replaced the T3A relay for the 28 EDG and calibrated or replaced all Agastat relays in the control logic for the three Fairbanks Morse EDGs. Relays were replaced if they were older than 10 years (from the date of manufacture) or could not meet the drift or contact resistance criteria in the calibration procedure. For extent of condition, CCNPP reviewed the maintenance history for Agastat relays used in other safety-related and critical applications at the site to identify relays that were beyond their 10 year replacement frequency or that exhibited excessive drift or contact resistance. CCNPP identified 55 critical relays that were older than 10 years and six of these relays also exhibited excessive drift. Operations performed an operability assessment for these six relays and determined that all the relays and the systems that they supported remained operable. CCNPP developed a replacement plan for these relays that prioritized the replacements based on risk significance and whether or not a failure of the relay would be immediately detectable.

All of the relays whose failure could have impacted the ability of a safety-related system to perform its design function have been replaced. CCNPP plans to replace all of the relays identified as being susceptible to a failure similar to the 28 EDG T3A failure by the end of 2011.

To rule out the existence of a manufacturing defect that could affect operation of more than one of the Agastat relays currently installed at Calvert Cliffs, CCNPP inspected the internals of 12 relays removed from service as part of the relay replacement plan. The retays were inspected for indications similar to what was identified during the contractor performed failure modes and effects analysis on the 28 EDG T3A relay. One of the relays inspected was purchased under the same purchase order as the failed relay and its serial number indicated that it was the next relay off the assembly line after the one that had failed. None of the 12 relays inspected presented conditions similar to what was found during the failure modes and effects analysis for the 28 T3A failed relay. This result provided CCNPP high confidence that any manufacturing defects that may have resulted in the 28 EDG T3A relay failure were limited to that relay' To address extent of cause relative to the adequacy of the site's preventive maintenance programs, CCNPP initiated a review of the testing methods for the sites most risk significant systems (as defined by the MSPI) to identify portions of those systems that were not adequately tested by the current testing methodology. The review will include benchmarking with other utilities and the results of this review will determine the need for additional evaluation and corrective actions in this area. CCNPP also reviewed all of the site's preventive maintenance program templates to verify that none of the templates took credit for monitoring as a substitute for a fixed replacement strategy, as had been done for Agastat relays. Finally, site calibration procedures for components associated with the site's most risk significant systems were reviewed to confirm that test connection points and testing medium were adequately specified' ln 2001, CCNPP inappropriately used the CDC process in accordance with EN-1-101 ,

Design Change and Modification lmplementation, to eliminate the requirement to replace Agastat E7000 series relays every 10 years. The CDC process is no longer used at Calvert Cliffs, all engineering change proposals are now processed through a single screening process defined in CCNPP procedures CM-1.01-1003, Design Engineering and Configuration Control, and CNG-FES-O07, Preparation of Design lnputs and Change lmpact Screen. To address the extent of this cause, CCNPP reviewed 104 of the 386 CDC engineering changes that were performed between 2Q01 and 2003 to verify that the changes implemented using this process were appropriately processed as a CDC. CCNPP did not identify other instances where significant changes not appropriate for processing using the CDC process were approved using the CDC process. CCNPP also reviewed current engineering change procedures, CM-1.01-1003 and CNG-FES-QQ7 to confirm that the current process would ensure that a change similar to the one implemented in 2001 would receive the appropriate level of review.

The inspectors reviewed a sample of CCNPP's assessments for risk significant system preventive maintenance strategies and CCNPP's review of the need for flushing and/or fill and vent procedures for salt water systems. The inspectors reviewed the replacement plan and related operability evaluations for safety-related Agastat relays in critical applications that were older than 10 years and the inspection results for the 12 Agastat relays removed from service to determine if a common failure mode was present. The inspectors also discussed the CDC process used to remove the Agastat relay 10-year replacement requirement and the current engineering change process, which CCNPP determined would have ensured the appropriate evaluations and reviews were completed, with design and system engineering personnel. Based on these reviews and interviews, the inspectors concluded that CCNPP's RCAR adequately addressed the extent of condition and the extent of cause for the performance deficiencies identified as a result of the 28 EDG failure.

lP 95001 requires that the inspection staff determine that the licensee's root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in Inspection Manual Chapter 0305.

CCNPP conducted a safety culture component assessment in accordance with site procedure CNG-CA-1

===.01 -1O04, "Root Cause

Analysis.

" The evaluation identified===

weaknesses, as defined in procedure CNG-CA-1

.01 -1004, in the human performance,

problem identification and resolution, and management performance cross cutting areas.

Specifically, the weaknesses were associated with the following safety culture components: work control, self-assessments and accountability, continuous learning, organizational change, and safety policies.

The inspectors reviewed CCNPP's safety culture component assessment and confirmed that the evaluation was performed in accordance with CCNPP's procedure. The inspectors also confirmed that CCNPP appropriately assigned corrective actions for weaknesses it had identified. The inspectors determined, based on these reviews, that CCNPP's RCAR properly considered weaknesses in safety culture components that were highlighted by the February 18,2010, 28 EDG failure.

f.

Findinqs No findings were identified.

02.03 Corrective Actions a.

lP 95001 requires that the inspection staff determine that:

(1) the licensee specified appropriate corrective actions for each root and/or contributing cause, or

(2) an evaluation that states no actions are necessary is adequate.

CCNPP immediately initiated corrective actions required to restore the 28 EDG to operable status. These actions included flushing the lube oil pressure sensing lines, replacing the 28 EDG failed T3A relay and performing a successful cold fast start of the 28 EDG. Within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of the 28 EDG failure, in accordance with technical specifications, CCNPP also confirmed that a common cause failure condition did not exist for the other two Fairbanks Morse EDGs.

To address the lube oil sensing line blockage that delayed the reset of the low lube oil pressure switches, CCNPP created periodic maintenance tasks to inspect, drain, and refillthe sensing lines on all three Fairbanks Morse EDGs. CCNPP also revised the calibration procedures for the lube oil pressure switches to specify the testing medium and connection point for the pressure source.

To address the failed Agastat relay, CCNPP developed new as-found acceptance criteria for the replacement of all safety-related and critical non safety-related Agastat relays. Procedure E-10, Testing and Adjustment of Agastat Relays, Rev. 00301 , which was eliminated around the time the 10 year replacement intervalwas eliminated in 2001, was reinstated and revised to specify the replacement criteria for critical safety-related relays. The new criteria stated that if during relay calibration the relay was identified to be greater than 8 years old (from the date of manufacture), or if the as-found calibration results for the relay exceeded the acceptance criteria for contact resistance or drift, the relay should be replaced.

As a starting point for relay replacements, CCNPP developed a relay replacement plan for all critical safety-related relays greater than 10 years old and for all non-critical safety-related and critical non safety-related relays greater than 18 years old. The replacement plan ranked each relay in terms of risk significance, relay performance during recent calibrations and whether or not a failure of the relay would be immediately detectable.

To ensure timely replacement of the subject Agastat relays in the future, in addition to reinstating the age based replacement criteria, CCNPP developed and implemented a relay monitoring program that included additional replacement criteria based on the trending of relay calibration results. These new criteria, that would be used to determine the need for replacement of safety-related and critical non safety-related Agastat relays, were developed based on CCNPP's review of all available historical Agastat relay calibration data. This new monitoring program, and its implementation, was described in thefollowing procedures: CNG-AM-1.01-1005, Engineering Rolesand Responsibilities, EN-1-136, CCNPP Relay Reliability Process, CNG-AM-1.01-1004, Equipment Reliability Reporting, E-10, Testing and Adjustment of Agastat Relays, and CSU-2, Agastat Relay Performance Monitoring.

To address the fact that the combined effect of the blockage in the lube oil sensing lines and the T3A relay drift on the ability of 28 EDG to successfully respond to a demand start could not be detected during periodic pre-lubricated surveillance testing, CCNPP implemented a test method to detect and monitor changes in the operating margin for the low lube oil pressure monitoring circuits for the Fairbanks Morse EDGs. CCNPP will monitor degradation by timing the actuation for one of the lube oil pressure switches on each Fairbanks Morse EDG during monthly surveillance testing. CCNPP also implemented a relay monitoring program that included a relay component engineer review and a relay component health report, which will trend performance for all safety-related and nonsafety-related Agastat relays based on calibration results. To provide additional margin to the low lube oil pressure trip setpoint, CCNPP increased the time delay setting of the T3A relay for all of the Fairbanks Morse EDGs.

To address CCNPP's inappropriate use of the CDC process to eliminate the requirement to replace Agastat E7000 series relays every 10 years, CCNPP reviewed current engineering change procedures, CM-1.01-1003 and CNG-FES-OO7, to confirm that no additional corrective actions were required because the current processes ensured that a change similar to the one implemented in 2001 would receive the appropriate level of review.

The inspectors reviewed CCNPP's corrective actions for each root and contributing cause as specified by the RCAR. The inspectors reviewed completed work orders for the replacement of the failed Agastat relay, and the filling and venting of lube oil sensing lines for all EDGs. The inspectors also reviewed the evaluation that CCNPP performed to increase the time delay relay for the lube oil pressure trip circuit and identified no concerns. The inspectors reviewed new periodic maintenance tasks that CCNPP created for the lube oil sensing lines and CCNPP's revisions to the Agastat relay calibration procedures. The inspectors walked down CCNPP's new venting and filling procedures for the Fairbanks Morse EDG lube oil systems with the system engineer and a maintenance department representative. The walk down confirmed the adequacy of these procedures. The inspectors also interviewed the relay component engineer, reviewed the relay system health report and procedures E-10, "Testing and Adjustment of Agastat Relays," Rev. 00301 and CSU-02, "Agastat Relay Performance Monitoring,"

Rev. 0 to verify that a relay monitoring program was in place and that as-found acceptance criteria for relay replacement were established. Based on these reviews and interviews, the inspectors concluded that the proposed and completed corrective actions assigned to address the root and contributing causes for the 28 EDG failure were appropriate.

b. lP 95001 requires that the inspection staff determine that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.

CCNPP took immediate corrective actions to restore the 28 EDG's operability by draining and refilling the lube oil sensing lines, replacing the failed T3A Agastat relay, and performing a successful cold fast start for the 28 EDG. In accordance with technical specifications, within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of the 28 EDG failure, CCNPP also confirmed that a common cause failure condition did not exist by confirming cold fast start capability for the other two Fairbanks Morse EDGs. Technical specification compliance for the emergency power system limiting condition for operation was restored on February 23, 2010, when the 28 EDG was declared operable.

To address the identified root causes and restore regulatory compliance relative to the performance deficiencies identified by the White finding and NOV, CCNPP completed its final revision to the root cause evaluation on January 19, 2011. CCNPP implemented corrective actions to address the causes of the White finding based on the results of this evaluation. CCNPP developed periodic maintenance tasks to inspect, drain and refill the sensing lines on the Fairbanks Morse EDGs; revised the calibration procedures for the EDG lube oil pressure switches to provide specific guidance for testing medium and test equipment connection points; and implemented a testing method to detect and monitor for degradation of the components in the Fairbanks Morse EDG lube oil pressure monitoring circuits not previously testing during cold fast EDG starts. CCNPP also developed and implemented a relay monitoring program and developed as-found acceptance criteria that controlled the replacement of all critical safety-related and critical non safety-related Agastat relays. Regulatory compliance was restored on January 13,2011, after CCNPP developed and implemented a performance monitoring program for the Agastat relays.

At the time of the supplemental inspection, all of the corrective actions described above were fully completed with the exception of the extent of condition Agastat relay replacements. The remaining relay replacements were scheduled based on CCNPP's relay replacement plan.

The inspectors reviewed CCNPP's relay replacement plan for the safety-related and critical relays in service that had exceeded the specifiedl0 year replacement requirement. CCNPP has replaced allAgastat relays whose failure could cause a safety-related system to be inoperable. CCNPP's replacement plan ranked each of the remaining relays in terms of risk significance, relay performance during recent calibrations and whether or not a failure of the subject relay was immediately detectable.

The remaining relays were scheduled for replacement during upcoming maintenance windows. CCNPP developed operability determinations for each in-service relay that had exceeded the 10 year age requirement and had previously exhibited excessive drift during testing. The inspectors reviewed these operability determinations and verified that they were adequate and that the relays remained operable. CCNPP is scheduled to finish the planned relay replacements before the end oI2O11.

The inspectors reviewed CCNPP procedure CNG-CA-1.01-1004, Root Cause Analysis, concerning guidance on prioritization and scheduling of corrective actions. The procedure required CCNPP to prioritize corrective actions with a due date based on risk significance. lt also required that compensatory actions be provided if permanent acltions could not be performed in a timely manner. The inspectors compared CCNPP's prioritization of corrective actions for the 28 EDG trip, as described above, to this guidance. Based on this review, the review of the relay replacement plan, and the operability determinations performed for relays that had exceeded both age and drift replacem-ent criteria, the inspectors determined that CCNPP's implementation of corrective actions was appropriately prioritized with consideration of risk significance and regulatory compliance.

c. lP 95001 requires that the inspection staff determine that the licensee established a schedule for implementing and completing the corrective actions.

CCNPP took immediate corrective actions to restore the 28 EDG's operability by draining and refilling the lube oil sensing lines, replacing the failed T3A Agastat relay, and performing a successful cold fast start for the 28 EDG. CCNPP also confirmed that a common cause failure condition did not exist by confirming cold fast start capability for the other two Fairbanks Morse EDGs.

At the time of the inspection CCNPP had completed several corrective actions to address the causes of the White finding based on the results of the RCAR. CCNPP developed periodic maintenance tasks to inspect, drain and refill the sensing lines on the Fairbanks Morse EDGs; revised the calibration procedures for the EDG lube oil pressure switches to provide specific guidance for testing medium and test equipment connection points; and implemented a testing method to detect and monitor for degradation of the components in the Fairbanks Morse EDG lube oil pressure monitoring circuits not previously

'a testing during cold fast EDG starts. CCNPP also developed and implemented relay monitoring program and developed as-found acceptance criteria that controlled the replacement of all tritical safety-related and critical non safety-related Agastat relays' CCNpP has several assigned corrective actions that have not yet been fully completed but are in progress. The three most significant remaining corrective actions include:

replacement of att Agastat relays that exceed the newly established replacement criteria requirements; a review of preventive maintenance strategies for a sample of safety-related system components to ensure the strategies complied with Regulatory Guide 1.33, Quality Assurance Program Requirements; and a review of testing procedures for the Calvert Cliffs site's most risk significant systems, to determine if testing measures would detect operating margin degradation before it impacted the systems capability to perform its safety function.

The inspectors determined that none of these remaining corrective actions were required to prevent recurrence of the significant condition adverse to quality that was identified by the February 18, 2010, 28 EDG failure. As previously discussed, all Agastat relays that were greater than 10 years old, whose failure could prevent a safety-related system from performing its designed safety function, were replaced and the relay monitoring program designed to control and track relay replacements was developed and implemented. The reviews of the preventive maintenance strategies and testing methods for risk significant systems, which are intended to verify that the programmatic or organizational conditions that resulted in the February 18, 2010, 28 EDG failure do not exist elsewhere at Calvert Cliffs, are in progress. The results of these reviews will determine whether actions are required to address the extent of cause in these areas.

The inspectors' reviews confirmed that regulatory compliance was restored on January 13,2011, and in accordance with CCNPP's current schedule for completion for the corrective action items discussed above and other less significant corrective actions for the 28 EDG trip on February 18,2Q10, all corrective actions will be completed before the end of 2011.

Based on this assessment and that all remaining corrective actions for the White finding are scheduled for completion before the end of 2Q11, the inspectors concluded that CCNPP established an acceptable schedule for corrective action completion that met CCNPP's corrective action timeliness requirements as provided in CCNPP procedure CNG-CA-1

===.01 -1 004, Root Cause

=

d. lP 95001 requires that the inspection staff determine that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition.

The root cause analysis for CR-2010-007157, documented performance of the following items to review the effectiveness of the corrective actions assigned to preclude repetition. Specifically, upon documented completion of all corrective action items for CR-2010-007157, the following items were assigned to be performed (Each item below was tracked by a separate corrective action item in the CCNPP CAP):

.

Confirm that EDG Agastat relay changes to the preventive maintenance strategy, testing methods, and replacement criteria were completed ;

.

Conduct a self-assessment of relay trending and maintenance results to evaluate the impact of Agastat relay trending on Agastat relay failure rates and the maintenance procedure impact on the identification of the need for relay replacement; and

.

Complete an effectiveness review in accordance with CNG-CA-1.01-1004, Root Cause Analysis, after all corrective actions and corrective actions to prevent recurrence have been completed to ensure the causes of the 28 EDG failure were identified and corrected. Specific items to be confirmed during the course of this review included: no Fairbanks Morse EDG trips due to low lube oil pressure, site wide preventive maintenance strategies meet the requirements of Regulatory Guide 1.33 with respect to replacement and inspection, and that an effective relay monitoring program has been implemented.

The effectiveness review in accordance with CCNPP procedure CNG-CA-1.01-1004 specified in the last item listed above required the following items to be completed:

Confirm all corrective actions and corrective actions to prevent recurrence were complete and met effectiveness review criteria as defined in the RCAR; Confirm the corrective actions and corrective actions to prevent recurrence were implemented as originally planned and were not modified, nor additional actions added without management review committee approval; Conduct interviews with affected personnelto confirm that the original problem no longer exists; Confirm that there have not been condition reports or key site performance indicator results that identify recent occurrences of this issue or an issue sufficiently similar in cause or consequence that indicates the problem still exists; and Confirm the corrective actions and corrective actions to prevent recurrence are still useful (i.e., procedure change works).

Upon completion of the review, CCNPP procedure CNG-CA-1.01-1004 required that the reviewer fully document the basis for the conclusions for each item and initiate CRs for effectiveness issues identified by the review.

The inspectors reviewed the effectiveness review corrective action items as documented in the RCAR and the effectiveness review process described in CCNPP procedure CNG-CA-1.01-1004 and determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition of the issues identified by the February 18,2010, 28 EDG trip.

e. lP 95001 requires that the inspection staff determine that the licensee's planned or taken corrective actions adequately address a NOV that was the basis for the supplemental inspection, if applicable.

The NRC issued an NOV of Technical Specification 5.4.1 to CCNPP on August 3,2010.

CCNPP provided the NRC a written response to the NOV on September 2,2Q10.

CCNPP's response described:

(1) corrective steps that have been taken and the results achieved;

(2) corrective steps which will be taken;

(3) the date when full compliance will be achieved; and

(4) the reasons for the violation. During this inspection, the inspectors confirmed that CCNPP's RCAR and its planned and completed corrective actions addressed the NOV and the associated performance deficiencies. The inspectors also confirmed that CCNPP restored full compliance with NRC requirements on January 13, 2011, when it developed and implemented a performance monitoring program for Agastat relays that complied with section 9.b of Appendix A to Regulatory Guide 1.33, as required by Calvert Cliffs Unit 2 Technical Specification 5.4.

Findinqs No findings were identified.

40A6 Exit Meetinq On March 18, 2011, the inspectors presented the inspection results to Mr. George Gellrich, Site Vice President, and other members of his staff, who acknowledged the inspection results. The inspectors asked if any of the material examined during the inspection should be considered proprietary. CCNPP did not identify any proprietary information.

Reoulatorv Performance Meetinq Following the March 18,2011, exit meeting, the NRC discussed with CCNPP its performance at Calvert Cliffs Nuclear Power Plant, Unit 2 in accordance with IMC 0305, Section 10.01.a. The meeting was attended by the Region I Division of Reactor Projects, Branch 1, Branch Chief, and other NRC staff and the CCNPP Site Vice President and other CCNPP staff. During this meeting, the NRC and CCNPP discussed the issues related to the White finding that resulted in Calvert Cliffs Nuclear Power Plant, Unit 2 being placed in the Regulatory Response Column of the Action Matrix. This discussion included the causes, corrective actions, extent of condition and extent of cause for the issues identified as a result of the February 18,2010, 28 EDG failure.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

G. Gellrich, Site Vice President

T. Trepanier, Plant General Manager

P. Amos, Performance lmprovement

H. Beasley, Principle Engineer

D. Lauver, Director Licensing

S. Loeper, Principle Engineer

C. Neyman, Licensing Engineer

T. Riti, General Supervisor, System Engineering

LIST OF ITEMS

OPENED. CLOSED. AND DISCUSSED Closed:

0500031 8/201 0006-02 vto Inadequate Preventive Maintenance Results in the Failure of the 28 Emergency Diesel Generator