IR 05000324/2008010
| ML083470550 | |
| Person / Time | |
|---|---|
| Site: | Brunswick  |
| Issue date: | 12/12/2008 |
| From: | Randy Musser NRC/RGN-II/DRP/RPB4 |
| To: | Waldrep B Carolina Power & Light Co |
| References | |
| IR-08-010 | |
| Download: ML083470550 (21) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION December 12, 2008
SUBJECT:
BRUNSWICK STEAM ELECTRIC PLANT - NRC SPECIAL INSPECTION REPORT NO. 05000325/2008010 AND 05000324/2008010
Dear Mr. Waldrep:
On September 11, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed the onsite portion of a special inspection at your Brunswick Steam Electric Plant. The inspection reviewed the circumstances surrounding a failed surveillance which demonstrates the ability to locally control the Emergency Diesel Generators. A Special Inspection was warranted based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program. The determination that the inspection would be conducted was made by the NRC on August 22, 2008, and the inspection started on September 8, 2008. The preliminary inspection results were discussed with you and members of your staff on September 11, 2008. Subsequently, additional in-office reviews were conducted and the enclosed inspection report documents the inspection results, which were discussed by telephone with Mike Annacone and other members of your staff on November 13, 2008.
This inspection was performed in accordance with Inspection Procedure 93812, Special Inspection, and focused on the areas discussed in the inspection charter described in the report. The inspection examined activities conducted under your licenses as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your licenses. The team reviewed selected procedures and records, conducted field walk downs, observed activities, and interviewed personnel.
Based on the results of this inspection, no findings of significance were identified.
CP&L 2 In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Randall A. Musser, Chief Reactor Projects Branch 4 Division of Reactor Projects Docket Nos.: 50-325, 50-324 License Nos.: DPR-71, DPR-62
Enclosure:
Inspection Report 05000325/2008010 and 05000324/2008010 w/Attachment: Supplemental information
REGION II==
Docket Nos.: 50-325, 50-324 License Nos.: DPR-71, DPR-62 Report Nos.: 05000325/2008010 and 05000324/2008010 Licensee: Carolina Power & Light (CP&L)
Facility: Brunswick Steam Electric Plant, Units 1 & 2 Location: 8470 River Road SE Southport NC 28461 Dates: September 08 - November 13, 2008 Inspectors: J. Hickey, Hatch Senior Resident Inspector, Team Leader R. Rodriguez, Senior Reactor Inspector, Division of Reactor Safety C. Jones, Senior Construction Inspector, Center for Construction Inspection Approved by: Randall A. Musser, Chief Reactor Projects Branch 4 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000325/2008-010, 05000324/2008-010; 09/08/2008 - 11/13/2008; Brunswick Steam
Electric Plant, Units 1 and 2; Special Inspection.
This report documents special inspection activities performed onsite and in the Region II office by a senior resident inspector, a senior reactor inspector, and a senior construction inspector to review the circumstances surrounding a failed surveillance which demonstrates the ability to locally control the Emergency Diesel Generators. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649 Reactor Oversight Process, Revision 4, dated December 2006.
NRC - Identified and Self - Revealing Findings
None.
Licensee - Identified Violations
None.
REPORT DETAILS
Summary of Plant Events On August 18, 2008, the licensee was performing the Emergency Diesel Generator (EDG) DG4 Local Control Operability Test. EDGs are normally controlled from the control room. This test demonstrates the ability to locally control the EDG in the event the normal control location is not available. When the portion of the surveillance which locally resets the DG4 lockout was performed, the Lockout Control Relay (LOCR) would not reset. Without the LOCR reset, DG4 cannot be operated to supply electrical power to required equipment.
On August 19, 2008, the licensee determined there was no electrical power available at the LOCR with the Alternate Safe Shutdown (ASSD) key switch in the local position. Further investigation by the licensee determined that a wiring error occurred during the installation of a circuit modification to DG4 in June 2007. A prompt extent of condition review determined the remaining EDGs were also affected. The licensee established compensatory measures to detect any potential fire in the areas which could cause a control room evacuation and in the EDG building. The licensee proceeded to re-wire and test each affected EDG. The local control function was restored to all EDGs on August 21, 2008.
Inspection Scope Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a Special Inspection was initiated in accordance with NRC Inspection Procedure 93812, Special Inspection Team. An initial charter was developed on August 22, 2008. The inspection focus areas included the following special inspection charter items:
1. Develop a sequence of events, including applicable management decision points, from implementation of the relay modification (mid-2007) through restoration of the alternate safe shutdown function.
2. Review and assess the licensees corrective actions after the identification that the EDGs were not able to perform their alternate safe shutdown function, including compensatory measures the licensee put in place prior to completing corrective actions. This should also include a review of the probable causes and contributing factors to the event, as well as any common cause analysis performed by the licensee.
3. Review and assess the design change that was completed in mid-2007 that adversely impacted the lockout relay and the design change used to correct the initial discrepant condition.
4. Review and assess the post modification testing completed after the implementation of the design change to determine why the testing did not identify the inability of the EDGs to perform their alternate safe shutdown function.
5. Collect data necessary to support completion of the significance determination
process, as applicable.
6. Verify that applicable documents have been updated to reflect actual plant conditions.
7. Interview engineering and maintenance personnel involved in the modification which resulted in the EDGs being unable to perform their alternative safe shutdown function to gain additional information and insights regarding the deficiency.
8. Determine the extent of condition regarding the design change that incorrectly wired the lockout relays.
OTHER ACTIVITIES
4OA5 Other Activities - Special Inspection
.01 Develop a sequence of events, including applicable management decision points, from
implementation of the relay modification (mid-2007) through restoration of the alternate safe shutdown function (Charter Item 1).
a. Inspection Scope
The inspectors reviewed operating logs, corrective action documents, maintenance documents, engineering documents and interviewed licensee personnel to develop a sequence of events for this issue. Documents reviewed are listed in the Attachment.
b. Findings and Observations
On 2/19/2007, the Low Pressure Shutdown Control Relay on DG2 failed.
Management Decision: Preventive maintenance activities were developed to schedule replacement of all relays associated with the EDGs.
On 2/28/2007, Engineering Change Request (ECR) 7373 identifies obsolete LOCR relays in the EDG control circuit.
On 5/15/2007, Engineering Change (EC) 66274 issued to replace the obsolete EDG LOCR.
On 6/7/2007, DG1 LOCR was replaced.
On 6/15/2007, DG2 LOCR was replaced.
On 6/20/2007, DG3 LOCR was replaced.
On 6/27/2007, DG4 LOCR was replaced.
On 8/18/2008 at 9:52 pm, DG4 failed the Local Control Operability Test.
On 8/19/2008 at 11:10 am, Troubleshooting identifies the LOCR wiring error and the extent of condition determines all EDGs are affected.
Management Decision: Proceed with repairs for all EDGs by working all shifts.
On 8/19/2008 at 6:30 pm, compensatory measures are put in place in the areas which would cause a control room evacuation if a fire occurred.
On 8/19/2008 at 11:30 pm, EC 66274 is revised to correct the wiring error.
On 8/20/2008 at 5:02 am, DG4 is rewired and tested to demonstrate local control capability. The safe shutdown function was restored for DG4.
On 8/20/2008 at 2:15 pm, DG2 is rewired and tested to demonstrate local control capability. The safe shutdown function was restored for DG2.
On 8/21/2008 at 3:40 am, DG1 is rewired and tested to demonstrate local control capability. The safe shutdown function was restored for DG1.
On 8/21/2008 at 11:15pm, DG3 is rewired and tested to demonstrate local control capability. The safe shutdown function was restored for DG3. ASSD function is restored for all EDGs.
The timeline demonstrates that upon discovery the licensee proceeded in a deliberate manner to resolve the wiring error.
.02 Review and assess the licensees corrective actions after the identification that the
EDGs were not able to perform their alternate safe shutdown function, including compensatory measures the licensee put in place prior to completing corrective actions.
This should also include a review of the probable causes and contributing factors to the event, as well as any common cause analysis performed by the licensee (Charter Item 2).
a. Inspection Scope
The inspectors reviewed maintenance documents, engineering documents, corrective action documents, and interviewed licensee personnel to determine what actions the licensee took following the failure of the DG4 Local Control Operability Test. The inspectors performed a field walk down of the cabinet where the DG4 LOCR relays are located. The inspectors reviewed what compensatory measures the licensee put in place while the rewiring activities were ongoing. The inspectors also reviewed the licensees cause determination. Documents reviewed are listed in the Attachment.
b. Findings and Observations
The licensees response was deliberate and methodical. This is supported by the sequence of events (4OA5.01). From the time of discovery that all four EDGs were affected by this wiring error, approximately 60 hours6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> was required to restore the ASSD capability for all four EDGs.
Upon discovery that the EDGs were not able to perform their ASSD capability the licensee established compensatory measures to prevent and identify fires in the EDG building and locations which could cause a control room evacuation. The inspectors identified a weakness in the compensatory measures. Areas which would not result in a control room evacuation but could require operators to take local control of the EDG(s)were not considered by the licensee for compensation. Licensees procedure 0PLP-01.5 Alternative Shutdown Capability Controls step 6.1.3.3.c states that if both trains of ASSD equipment on a unit are inoperable, then implement appropriate compensatory measures within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. However, the licensee implemented repairs to the circuitry such that the ASSD function was restored within the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> procedure limit. The licensee accepted the inspectors comments and initiated corrective actions to re-evaluate affected areas in the plant. The licensee initiated NCR 294505.
The licensees investigation into the cause of the wiring error has determined two primary causes. The first cause relates to the wire number convention for the ASSD local-normal key switches. The wire segment number on either side of the ASSD local-normal key switch was the same. Therefore, the individual selecting the point for the wiring change chose the correct wire number but the wrong locations on the ASSD local-normal key switches. For the second cause, the licensee determined a lack of rigor was evident in the design change process. This relates to the human performance of the Engineer and Design Verifier in the control circuit design change process.
An extent of cause review is being performed by the licensee. The licensee will review all safety related circuit modifications for the previous two years. Two years was chosen by the licensee because the periodicity of most technical specification surveillances falls within this interval. The corrective action to address this issue is tracked by NCR 292232 CORR#5.
An extent of condition review is being performed by the licensee. The focus of this action is to identify and correct the licensees stated primary cause. A review of all ASSD switch modifications will be performed to verify the wire numbering convention across the switch is different. Additionally, a sample of other safety related switches will be performed to verify the wire numbering convention. The corrective action to address this issue is tracked by NCR 292232 CORR #6b.
.03 Review and assess the design change that was completed in mid-2007 that adversely
impacted the lockout relay and the design change used to correct the initial discrepant condition. (Charter Item 3)
a. Inspection Scope
The inspectors conducted direct observations of the as-built configuration at DG4, interviewed responsible engineers, and examined documentation created for the initial (mid-2007) design change and the subsequent change issued to correct the discrepant condition.
b. Findings and Observations
The initial design change was documented in EC 66274, revision 0, Evaluate Replacement Relays for the Diesel Generator LOCR, LTACR, and N2CR Relays, dated May 15, 2007. The change was issued to replace obsolete logic control relays in all four emergency diesel control systems. To accomplish this, the logic and wiring configuration for one group of relays, the LOCRs, had to be modified to accommodate differences in the operating characteristics of the replacement relays. The design solution featured the addition of an auxiliary lockout reset (LR) relay in the logic circuitry.
The change also corrected a legacy design problem that created a relay race between the LOCR relays and the emergency control relays (ECRs).
The inspectors reviewed the initial engineering change package and confirmed the critical design features of the replacement relays had been evaluated and measures were specified to establish system functionality equivalent to the original design.
Designers identified and dispositioned a number of design input requirements, including a general requirement for the replacement relays to operate under normal and accident conditions (Design Input 15). However, the design package did not identify an applicable requirement from Design Basis Document DBD-39, Emergency Diesel Generator System, which states the EDGs shall be considered a safe shutdown system and shall be capable of being controlled from both the Control Room and a remote location. Consequently, design specifications for acceptance testing did not require the functionality of the modified logic to be demonstrated while control was established at the remote location (i.e. the local control panel). Details from the inspection of design acceptance testing are provided under Section 4OA5.04 of this report.
The inspectors verified an error was committed by designers in translating design requirements into the sketches used to guide physical installations in the field. Although requirements depicted in higher-level schematic logic drawings were consistent with DBD-39, sketches provided for field installation of the changes specified incorrect termination points for linking control power to the lockout relay reset circuitry. The drawing errors were overlooked by the design engineer, a supervising engineer, and a design verifier. This had the unintended effect of removing control power from the circuitry whenever control of the associated emergency diesel was transferred from the control room to the local panel. As documented in corrective action report NCR 292232, the design output error was discovered in August 2008 when operators were unable to reset the emergency diesel generators while attempting to operate the systems from the local control panels. This issue is unresolved pending NRC review of the EDG control wire routing to verify what areas are affected and is designated as URI 05000325,324/2008010-001, Verify cable routing locations affected by a design change error which resulted in a loss of Emergency Diesel Generator local control function described in Section 4OA5.02 of this report.
Revision 5 to engineering change EC 66274 was issued to correct the wiring error. The inspectors review of the revision found the design input requirements were not updated.
The omission indicated a continuing lack of rigor in implementing the design control process; however, in this case the necessary information was separately captured in corrective action report NCR 292232. Inspectors found the installation sketches were revised with clear instructions for restoring full functionality of the lockout reset relays.
The changes to the design output products were processed, reviewed, and approved under controls commensurate with the original design. Inspector review of documentation contained in maintenance work order 01401989-03, dated 8/18/2008, determined the work records provided evidence that corrective installations were properly completed, peer verifications were conducted, and acceptance testing was completed for the required wiring changes.
.04 Review and assess the post modification testing completed after the implementation of
the design change to determine why the testing did not identify the inability of the EDGS to perform their alternate safe shutdown function. (Charter Item 4).
a. Inspection Scope
The inspectors reviewed post modification testing, control wiring diagrams, and the root cause evaluation for the EDGs relay replacement design change implemented in June 2007 to determine why the post modification test did not identify the inability of the EDGs to perform their alternate safe shutdown function. The review contrasted the test completed after the implementation of the design change in June 2007, and the subsequent performance of surveillance test 0PT-12.14.L, DG4 Local Control Operability Test, in August 2008.
b. Findings and Observations
The inspectors independently verified the licensees conclusion that the post modification testing was designed to validate the effectiveness of the design and circuit changes, but that it did not sufficiently test the LOCR to ensure that the new component was capable of performing all of its design functions. Specifically, the new relays were tested with the ASSD key-switch in the NORMAL position only, and were not tested with the ASSD key-switch in the LOCAL position. The licensee determined that while this was in compliance with their testing guidance procedure, EGR-NGGC-0155, it did not meet the intent of this procedure or Generic Letter 96-01. Corrective action was initiated to revise EGR-NGGC-0155 to 1) more fully discuss the concept of unintended consequences as it relates to control logic circuits, 2) discuss the need for additional care when dealing with daisy-chained circuits, and 3) more fully describe how to comprehensively determine the sphere of influence of a control circuit modification such that potential consequences can be adequately addressed. The inspectors concluded that the inadequate post modification testing was not a cause of the event, but a barrier that failed to detect the inadequate design change. A conceptual design error was transferred from the design modification package into the post modification testing process which resulted in the post modification test not identifying the wiring error. The inspectors concluded that the post modification testing issue did not constitute a separate performance deficiency and as previously stated, the inadequate design change will be assessed by URI 05000325,324/2008010-001. The licensee has entered the post modification testing aspects of this matter into the corrective action program as part of NCR 292232 CORR #4.
.05 Collect data necessary to support completion of the significance determination process,
as applicable (Charter Item 5)
a. Inspection Scope
The team reviewed plant procedures, corrective action documents, plant general arrangements drawings, plant fire area delineation drawings, and cable routing information to gather data necessary to develop and assess the safety significance of any identified findings. The team also walked down the affected fire areas to evaluate ignition sources and their potential impact to safe shutdown equipment.
b. Findings and Observations
No findings or observations of significance were indentified.
.06 Verify that applicable documents have been updated to reflect actual plant configuration
(Charter Item 6)
a. Inspection Scope
The inspectors reviewed Engineering Change (EC) 66274 as well as the Control Wire Drawings (CWD) and Control Panel Connection Diagrams (CPCD) associated with the EDGs to verify that applicable portions of the documents had been updated to reflect actual plant configuration.
b. Findings and Observations
No findings or observations of significance were indentified.
.07 Interview engineering and maintenance personnel involved in the modification which
resulted in the EDGs being unable to perform their alternative safe shutdown function to gain additional information and insights regarding the deficiency (Charter Item 7).
a. Inspection Scope
The inspectors interviewed several engineering, maintenance planning, and maintenance personnel to determine how these individual approach the design, planning and implementation of design changes.
b. Findings and Observations
The inspectors determined, based on the sample interviews, the individuals understood their responsibilities in the design change process.
.08 Determine the extent of condition regarding the design change that incorrectly wired the
lockout relays (Charter Item 8).
a. Inspection Scope
The inspectors reviewed 31 corrective action system reports related to quality of engineering change packages. The reports were selected from system history between 2006 and mid-2008. In addition, inspectors reviewed the licensees statement of extent of condition as described in Section 4OA5.02 of this report.
b. Findings and Observations
The inspectors verified the incorrect logic wiring extended to all four emergency diesel generators, affecting both generating units. No other instances were found in corrective action history where designers failed to properly connect logic circuitry involving mode selection switches. Also, none of the corrective action reports selected for review described a failure of a design change package to implement a requirement from a design basis document.
Six instances were identified where avoidable deficiencies were discovered after distribution of approved design packages. The following Action Requests (ARs) were identified to the licensee to include in their evaluation of AR 294505294505 Common Cause Implementation of ECs.
- AR 203701203701 dated 8/18/2006, reported that a design package assigned identification numbers to new terminal boards that were already assigned to pre-existing terminal boards in the same panel.
- AR 216386216386 dated 12/13/2006, identified that specifications for a design acceptance test provided inadequate set up of a turbine building ventilation flow instrument.
- AR 227854227854 dated 4/1/2007, identified a failure of designers to identify and control electromagnetic interference (EMI) from new model relays.
- AR 252046252046 dated 10/24/2007, reported that a new model relay would not physically fit into the location specified by design.
- AR 264459264459 dated 1/31/2008, reported design installation sketches specified an orientation for new RHR system valves that was opposite to the flow direction.
- AR 272275272275 dated 3/27/2008, reported a design package issued to accommodate a problem with field alignment of piping to a valve specified the wrong dimension for the alignment spacer.
4OA6 Meetings, Including Exit
On September 11, 2008, the special inspection team leader presented the preliminary inspection results to Mr. B. Waldrep, Brunswick Site Vice President, and members of his staff. Subsequently, additional in-office reviews were conducted and the final inspection results were discussed by telephone with Mr. Mike Annacone and members of his staff on November 13, 2008. The licensee acknowledged the inspection observations. No proprietary information is included in this inspection report.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- M. Annacone, Director - Site Operations
- B. Davis, Manager - Engineering
- S. Hardy, Fire Protection
- S. Howard, Manager - Operations
- R. Ivey, Recovery Manager
- A. Pope, Manager - Maintenance
- T. Sherrill, Engineer - Technical Support
- B. Waldrep, Site Vice President
NRC Personnel
- G. Kolcum, Resident Inspector, Brunswick, Region II
- J. Munday, Deputy Director (acting) Division of Reactor Projects, Region II
- R. Musser, Chief, Reactor Projects Branch 4, Division of Reactor Projects Region II
- P. OBryan, Senior Resident Inspector, Brunswick, Region II
LIST OF ITEMS
OPENED, CLOSED AND DISCUSSED
Opened
- 05000325,324/2008010-001 URI Verify cable routing locations affected by a design change error which resulted in a loss of Emergency Diesel Generator local control function
Closed
None
Discussed
None