IR 05000302/2006006
| ML060530042 | |
| Person / Time | |
|---|---|
| Site: | Crystal River  |
| Issue date: | 02/17/2006 |
| From: | Payne D NRC/RGN-II/DRS/EB2 |
| To: | Young D Florida Power Corp, Progress Energy Florida |
| References | |
| IR-06-006 | |
| Download: ML060530042 (16) | |
Text
February 17, 2006
SUBJECT:
CRYSTAL RIVER NUCLEAR PLANT - NRC SUPPLEMENTAL INSPECTION REPORT NO. 05000302/2006006
Dear Mr. Young:
On January 27, 2006, the U. S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure (IP) 95001 at your Crystal River Unit 3 facility. In a letter dated September 21, 2005, the NRC informed you of the final significance determination for a White inspection finding in the Mitigating Systems Cornerstone.
This finding, which was identified during the Triennial Fire Protection Inspection, involved unprotected post-fire safe shutdown cables and related non-feasible local operator actions.
The NRC Reactor Oversight Process Action Matrix requires that a supplemental inspection be conducted in accordance with IP 95001 for one White finding in a strategic performance area.
The NRC was informed of your readiness for the inspection on December 8, 2005.
The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.
Based on the results of this inspection no findings of significance were identified.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
\\\\RA\\\\
D. Charles Payne, Chief Engineering Branch 2 Division of Reactor Safety Docket Nos. 50-302 License Nos. DPR-72
Enclosure:
Inspection Report 05000302/2006006 w/Attachment: Supplemental Information
REGION II==
Docket No.:
50-302 License No.:
DPR-72 Report No.:
05000302/2006006 Licensee:
Progress Energy Florida (Florida Power Corporation)
Facility:
Crystal River Unit 3 Location:
15760 West Power Line Street Crystal River, FL 34428-6708 Dates:
January 23 - 27, 2006 Inspectors:
R. Fanner, Reactor Inspector N. Merriweather, Senior Reactor Inspector Approved by:
D. Charles Payne, Chief, Engineering Branch 2 Division of Reactor Safety
Enclosure Enclosure
SUMMARY OF FINDINGS
IR 05000302/2006006; 01/23/2006 - 01/27/2006; Crystal River Unit 3; Supplemental inspection for a White finding involving unprotected post-fire safe shutdown cables and related non-feasible local operator action.
This inspection was conducted by two regional inspectors from NRC Region II. No findings of significance were identified. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
A.
Results of Supplemental Inspection
Cornerstone: Mitigating Systems
The NRC performed this supplemental inspection to assess the licensees evaluation for a single failure vulnerability in the relay and metering circuit on Engineered Safeguards Buses 3A and 3B which, if damaged from fire, could result in a loss of all safety-related alternating current (AC) power; and a related local operator action which was intended to restore AC power that was judged to be non-feasible. This performance issue was previously characterized as having low to moderate risk significance (White) in NRC Final Significance Determination letter (IR 05000302/2005011), dated September 21, 2005.
During this supplemental inspection, performed in accordance with Inspection Procedure 95001, Inspection for One or Two White Inputs In a Strategic Performance Area, the NRC determined that the licensee performed a comprehensive evaluation to determine the root causes, extent of condition and extent of causes for the performance deficiency. The primary root causes for the single failure vulnerability were attributable to 1) a latent programmatic deficiency, in that, the design change processes and procedures did not require a failure modes and effects analysis, and 2) a lack of technical rigor was exercised during the design, verification, and licensee acceptance of the modification packages developed for the Offsite Power Transformer (OPT) and Backup Engineered Safeguards Transformer (BEST) by the architect engineer.
The causes for the non-feasible local operator action were attributable to inadequacies in the fire study resulting from omissions of relevant information, improper referencing or branching, and overconfidence due to previous or similar experience.
The inspectors concluded that the completed and proposed corrective actions, including actions to prevent recurrence, have adequately addressed the results of the root cause evaluations.
Given the licensees acceptable performance in addressing the event, the White finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in Inspection Manual Chapter 0305, Operating Reactor Assessment Program.
NRC-Identified and Self-Revealing Findings
No findings of significance were identified.
Licensee-Identified Violations
None
REPORT DETAILS
INSPECTION SCOPE
This supplemental inspection was performed to assess the licensees evaluation for a White finding involving a single failure vulnerability in the relay and metering circuit on Engineered Safeguards (ES) Buses 3A and 3B which, if damaged from fire, could result in a loss of all safety-related alternating current (AC) power; and a related local operator action which was intended to restore AC power that was judged to be non-feasible. The protection and metering circuits were not physically separated or protected from fire damage as required by 10 CFR 50, Appendix R,Section III.G.2.
Instead, an unapproved local operator action was used to restore AC power.
However, this local operator action was not feasible because the NRC determined that the fire response activities could cause the location for the operator action to be exposed to hot smoke, water mist, and water on the floor. The NRC also determined that the relay and metering circuits were vulnerable to non-fire random single active failures. The latter issue was related to the White finding and was also reviewed during this inspection. The White finding was in the Mitigating Systems Cornerstone in the reactor safety strategic performance area. The performance issue was previously characterized in NRC Inspection Report (IR)05000302/2005007, dated June 16, 2005, as preliminary White, and later characterized as White in the NRC Final Significance Determination letter (IR 05000302/2005011), dated September 21, 2005.
This inspection, which was conducted in accordance with the requirements of NRC Inspection Procedure (IP) 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, involved a review of the licensees problem identification, root cause analysis, and corrective actions associated with the White finding. Specifically, the inspectors assessed the adequacy of the licensees root cause analyses, determined if appropriate corrective actions were specified and scheduled commensurate with risk, and determined if the proposed actions were sufficient to prevent recurrence. The assessment included a review of the licensees action requests (ARs) or nonconformance reports, root cause analyses, completed and scheduled corrective actions, procedures, related documentation, and interviews with key licensee personnel. The inspectors also reviewed the related Licensee Event Report 05000302/2005-001-00, Design Change Creates Engineered Safeguards Bus Protective Relay Scheme Single Failure Vulnerability.
The following report details summarize the results of the inspection and are organized by the specific inspection requirements of IP 95001.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a. Determination of who (i.e., licensee, self-revealing, or NRC) identified the issues and under what conditions.
Enclosure The performance deficiency associated with the White finding was identified by NRC during the Triennial Fire Protection Inspection on January 27, 2005.
During the inspection, the licensee provided a response to NRC questions related to the Crystal River Unit 3 (CR-3) 10 CFR 50, Appendix R, Fire Study, Rev.12, dated May 13, 2004. Section 3 of the Fire Study, "Appendix R Circuits Listing," contained Notes 70 and 71 describing a design problem that could result from an Appendix R fire in the protective circuitry for incoming breakers 3211 and 3212 from the Offsite Power Transformer (OPT) to both 4160 volt (V)
ES buses and incoming breakers 3205 and 3206 from the Backup ES Transformer (BEST) to both 4160 V ES buses. During a review of the response to the questions, NRC inspectors saw that the response described an electrical protection and metering circuit which, if damaged from fire, could electrically lock out both 4160 V ES buses and prevent their re-energization both from offsite power sources (OPT and BEST) and from the emergency diesel generators (EDGs). The licensee had credited a manual action by a primary plant operator (PPO) in the 3B switchgear room to reset the EDG lockout relay so that emergency AC power could be restored to the dedicated safe shutdown train.
The NRC determined that the local manual action was non-feasible because an operator may have to enter the area while fire fighting is taking place in the 3A ES 4160 V Switchgear Room and may be subjected to smoke and water on the floor. This water from fire fighting activities from the 3A ES 4160 V Switchgear Room could flow unimpeded into the adjacent 3B ES 4160 V Switchgear Room.
The NRC also questioned if the problem described in Notes 70 and 71 could also occur if a single failure was taken during a 10 CFR 50, Appendix A event such as a loss of coolant accident (LOCA) coincident with a loss of offsite power (LOOP)event. Upon review of these NRC concerns, CR-3 Design Engineering staff determined that the extent of this design configuration had not been fully recognized nor had its consequences been previously evaluated for a potential single failure occurring during Appendix A events. The licensee then entered the nonconforming conditions into the corrective action program under ARs 00149507 and 00149509 on January 27, 2005. The licensee subsequently submitted to NRC on March 23, 2005, Licensee Event Report (LER) 50-302/2005-001-00, Design Change Creates Engineered Safeguards Bus Protective Relay Scheme Single Failure Vulnerability.
The inspectors concluded that the licensee had adequately documented these NRC identified issues.
b. Determination that the evaluation documents how long the issue existed, and prior opportunities for identification.
The licensees investigation concluded that the problem originated around the 1990 time frame when the OPT replaced the original CR-1/CR-2 start-up transformer. The original design for the CR-1/CR-2 start-up transformer had the metering circuits for over-current and residual ground over-current isolated from the safety related buses whereas this was removed when the licensee installed the OPT. Since the licensees design processes during the 1990 time period had no extensive guidance to perform a failure modes and effects analysis (FMEA),
Enclosure the flaw was repeated in the design of the BEST when it was installed around 1993. The licensee identified two missed opportunities to identify the issue earlier. For example, around December 1997 the licensee acknowledged incorporation of Interim Change (IC) No. 703 which directed licensee personnel to rewrite the 10 CFR 50, Appendix R, Fire Protection Study, Rev. 8, which resulted in the manual action to reset the lockout being subsequently incorporated into Operating Procedure (OP) 880. Appendix R Tunnel Vision was identified as a causal factor for missing this opportunity to identify the single failure vulnerability in 1997. An incomplete circuit analysis for a 1997 modification that added secondary protectors to the protective relay and metering circuits was the second missed opportunity to identify the single failure vulnerability. The review was limited to what effects the secondary protector could have on the circuit if it failed. The analysis did not look at the overall circuit for compliance with the single failure criterion for redundant power trains.
The NRC's review of information from the licensee's evaluation was conducted and no additional instances were identified by the inspectors that would have directed licensee personnel to this issue. The inspectors found the licensee's actions to be acceptable.
c.
Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues.
The licensee analyzed the core damage risk associated with the fire vulnerability using input from simulator runs, walk-downs, and Probabilistic Risk Assessment insights. In addition, the licensee performed a plant specific evaluation assessing the need for corrective actions that established compliance for the fire protection procedure performance deficiency. Based upon the current fire modeling practices, the availability of mitigating equipment and potential to recover power to the unaffected bus, the licensee concluded the fire risk to be low.
However, the NRC disagreed with the licensees risk assessment as discussed in NRC IR 05000302/2005011, dated September 21, 2005. The NRC concluded that the probability of failure to reset the EDG lockout relay was much greater than that assumed by the licensee due to the extreme environmental conditions produced by the fire. This was quantified by a Significance Determination Phase 2 Evaluation that produced a value of 2.4 E-5, low to moderate significance.
02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a. Evaluation of methods used to identify root causes and contributing causes.
The inspectors reviewed the methodology and results of the licensees root cause analyses as documented in AR 00149507, OPT and BEST Protective Relay Scheme Does Not Meet Single Failure. This AR was a Priority 1 nonconformance report requiring a root cause investigation.
The licensee formed a six-person root cause team which developed and documented an investigation strategy. The analysis used a combination of Enclosure Events and Causal Factors Charting, Barrier Analysis, and Interviewing to identify root and contributing causes. The licensees investigation concluded that the root causes for the single failure vulnerability were attributable to 1) a latent programmatic deficiency in the design change processes and procedures, in that, a FMEA was not required and 2) a lack of technical rigor exercised by the engineering staff during the design and verification, and owners acceptance of the modification packages. The licensee also identified Appendix R Tunnel Vision as a casual factor for missing an opportunity to identify the single failure vulnerability in 1997 during the review and approval of IC 703 to the Appendix R Fire Study. An incomplete circuit analysis performed for MAR 97-02-11-01, Current Transformer (CT) Secondary Protectors, resulted in a second missed opportunity to identify the single failure vulnerability in 1997. This resulted from the same causal factors discussed earlier.
The non-feasible manual action was documented in the corrective action program as AR 00149509, NRC Fire Protection Inspection Identified Environmental Concerns. This AR was a Priority 2 nonconformance report requiring only that an apparent cause determination be made by the licensee.
Therefore, no root cause investigation was performed for this issue. The apparent causes for the non-feasible manual action were determined to be due to inadequacies in the fire study resulting from omissions of relevant information, improper referencing or branching, and overconfidence due to previous or similar experience. In summary, the fire study did not adequately assess equipment and cables outside the fire area for smoke, heat, water and combustion products resulting from fire fighting activities.
The inspectors concluded that the licensee followed the corrective action program procedures for performing investigations of Priority 1 and 2 ARs.
b. Level of detail of the root cause evaluation.
The inspectors review of the licensee's root cause analyses determined that they had been performed to a level of depth commensurate with the significance of the issue and provided reasonable assurance that the root causes and contributing causes for the single failure vulnerability had been identified.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience.
The licensee reviewed numerous sources of information to identify other potential opportunities to have recognized this event. The documents reviewed included corrective action program documents, operating experience data from other licensees, and NRC Generic Communications.
The licensee performed a root cause evaluation for the single failure vulnerability which was conducted in accordance with the procedure for performing root cause evaluations. The licensees searches of external operating experience data identified no related data that would have led to identification of this issue.
The licensees search of internal data identified previous opportunities to identify the event and the inappropriate acts that contributed to this event.
Enclosure The inspectors review determined that the licensee conducted an adequate search of external and internal sources to determine if any similar problems had been previously identified.
d. Consideration of extent of cause and extent of condition of the problem.
The licensee performed an extent of a condition review for the single failure vulnerability. The licensee evaluated the protective relaying schemes for the following breakers to determine if single failure vulnerabilities existed:
4160 V ES bus offsite source breakers 3205, 3206, 3207, 3208, 3211,3212 4160 V EDG output breakers 3209, and 3210 4160 V/480V transformer 3220, 3221, and 3222 480 V beakers 3310 and 3311 480 V ES bus cross-tie breakers 3390 and 3391 The review concluded that no additional violations of the single failure criterion were identified. The licensees evaluation did produce a recommendation to improve the operating margin if the EDG is operated in parallel with one of the offsite power sources and that same source is powering both 4160 V ES buses, and the 4160 V ES bus on the opposite side for the running EDG faults to ground. The surveillance test procedures SP354A and SP-354B would be revised to prevent testing an EDG on either the OPT or BEST if the OPT or BEST is supplying both ES Busses.
The extent of condition review included:
- Reviewing all open Engineering Changes (i.e., mechanical, structural, electrical, and Instrumentation & Controls) to determine if a FMEA was required to be performed.
- Performing a FMEA safety-related protective relays and protective relays that can affect the operation of safety-related equipment and offsite circuits required to satisfy the Technical Specifications.
- Performing a FMEA on the relay circuits analyzed in MAR 97-02-11-01, Current Transformer Secondary Protection Installation, and a new Safety Assessment for the modification using the FMEA results.
Enclosure
- Reviewing the, Appendix R Fire Study for compliance with the single failure criterion.
- Evaluating the electrical safety-related system for cross-train dependencies and their compliance with single failure, redundancy, diversity, separation requirements, and failure effects.
- Re-evaluating the manual actions identified in the Appendix R Fire Study, to identify all non-feasible manual actions and establish compensatory measures including roving fire watches until corrective actions are completed.
The inspectors considered the licensees review for extent of cause and extent of condition to be appropriate and acceptable.
02.03 Corrective Actions a. Appropriateness of corrective actions.
[From Triennial Fire Protection IR 05000302/2004009] The licensee took prompt corrective actions to implement modifications to correct nonconforming conditions prior to the end of the Triennial Fire Protection Inspection. The inspectors verified at that time that the licensee implemented Engineering Changes (EC) 60150 and 60155 to disconnect CT circuits to the watt-hour meters and remove the common return path of both train's CT circuits through the watt-hour meters. The configuration changes restores electrical and physical separation between the two redundant trains. In addition, the separation removes the need for the non-feasible manual action to reset the 86B/5210 lock-out relay in the 3B ES 4160 V Switchgear Room.
The long term corrective actions addressed organizational and programmatic weaknesses that contributed to the failure to recognize the single failure vulnerability and non-feasible manual action. The actions to prevent recurrence included revising the EC procedures to require that a FMEA be performed or the reasons for not performing a FMEA be justified. In addition, design engineers were trained on the new requirements in the program. The licensee has also put in place roving fire watches as compensatory actions in those plant areas where operators are required to perform local manual actions while they pursue alternatives to resolving any non-feasible manual actions and/or transitioning to the performance based fire protection program in accordance with the National Fire Protection Association 805 Code.
Based on the above, the inspectors concluded that the completed and proposed corrective actions, including actions to prevent recurrence, have adequately addressed the results of the root cause evaluations. The licensee had initiated appropriate corrective actions for each of the root causes and other causal factors identified.
b. Prioritization of corrective actions.
Enclosure The inspectors determined that the priority assigned to the action requests was consistent with the requirements of the corrective action program. The inspectors determined that the immediate corrective actions taken by the licensee were appropriately prioritized based on risk significance of the issue and/or regulatory compliance (i.e. modifications implemented, root cause evaluation performed, EC procedures revised, design engineers trained, fire watches established, manual action feasibility study conducted, and etc.)
c.
Establishment of schedule for implementing and completing the corrective actions.
The inspectors determined that the corrective actions associated with the single failure vulnerability and non-feasible manual action were captured in the licensee's electronic corrective action program system with sufficient detail (i.e.
identification of responsible individuals, assigned tasks, and due dates established) to ensure that they are tracked and completed commensurate with their significance and priority.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence.
The licensee planned an effectiveness review of the single failure vulnerability to verify that the corrective actions to prevent recurrence were properly implemented in accordance with the corrective action plan. The review was scheduled to be completed by January 31, 2006. The scope of the review will include, but not be limited to, an examination of whether the corrective actions implemented were adequate and appropriate barriers are in place to prevent recurrence of the event.
The inspectors concluded that these measures were appropriate to determine the effectiveness of the corrective actions.
OTHER ACTIVITIES
.01 (Closed) LER 05000302/2005001-00: Design Change Creates Engineered
Safeguards Bus Protective Relay Scheme Single Failure Vulnerability.
The inspectors reviewed the subject LER and AR 00149507, which documented this event in the corrective action program, to assess the licensees root cause evaluation and corrective actions to address the single failure vulnerability. The problem described in this LER is an additional example of the White finding and associated Violation 05000302/2005011-01, Unprotected Post-Fire Safe Shutdown Cables and Related Non-Feasible Local Manual Operator Action.
The risk associated with this LER has been assessed and was included in the significance determination for the White finding.
Enclosure Both the LER and the Violation involved the design of the common electrical protection and metering circuit for the 3A and 3B safety-related ES 4160 V switchgear. One common protection and metering circuit served both of the switchgear, such that fire damage to the common circuit or a single failure in the common circuit could result in a loss of both the 3A and 3B 4160 V switchgear.
The issue was identified by the NRC in IR 05000302/2004009. The licensee entered this item into the corrective action program as AR 00149507 on January 27, 2005. The issue was previously characterized in NRC IR 05000302/2005007, dated June 16, 2005, as preliminary White, and later characterized as White in the NRC Final Significance Determination letter (IR 05000302/2005011), dated September 21, 2005. The licensee has conducted a thorough root cause analysis and implemented corrective action to prevent recurrence. Consequently, this LER is closed.
.02 (Closed) VIO 05000302/2005011-01: Unprotected Post-Fire Safe Shutdown
Cables and Related Non-feasible Local Manual Operator Action. Based on the satisfactory results of this supplemental inspection and the licensees established corrective actions, this violation was determined to be sufficiently addressed to close the associated open item. Given the licensees acceptable performance in addressing the event, the White finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in NRC Inspection Manual Chapter 0305, Operating Reactor Assessment Program.
MANAGEMENT MEETINGS
Exit Meeting Summary
The inspectors presented the results of the supplemental inspection to Mr.
D. Roderick and other members of licensee management and staff on January 27, 2006. The inspectors confirmed that proprietary information was not provided or examined during the inspection.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- M. Annacone, Engineering Manager
- S. Barkofski, Electrical and Instrumentation and Controls Design Supervisor
- M. Bishara, Engineering Services Superintendent
- G. Englert, Mechanical and Structural Design Supervisor
- J. Franke, Plant General Manager
- D. Roderick, Director Site Operations
NRC Personnel
- H. Christensen, Deputy Director, Division of Reactor Safety, Region II
- T. Morrisey, Senior Resident Inspector, Crystal River Unit 3
- R. Reyes, Resident Inspector, Crystal River Unit 3
ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
None.
Closed
- 05000302/2005-001-00 LER Design Change Creates Engineered Safeguards Bus Protective Relay Scheme Single Failure Vulnerability (Section 03.01)
- 05000302/2005011-01 VIO Unprotected Post-Fire Safe Shutdown Cables and Related Non-feasible Local Manual Operator Action (Section 03.02)
Discussed
None.