ML083540244
| ML083540244 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 12/19/2008 |
| From: | Collins E NRC Region 4 |
| To: | Ridenoure R Southern California Edison Co |
| References | |
| EA-08-296, FOIA/PA-2011-0157 IR-08-013 | |
| Download: ML083540244 (64) | |
See also: IR 05000361/2008013
Text
UNITED STATES
NUC LE AR RE G UL AT O RY C O M M I S S I O N
R E GI ON I V
612 EAST LAMAR BLVD , SU I TE 400
AR LI N GTON , TEXAS 76011-4125
December 19, 2008
Ross T. Ridenoure,
Senior Vice President and
Chief Nuclear Officer
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
SUBJECT: FINAL SIGNIFICANCE DETERMINATION FOR A WHITE FINDING AND NOTICE
OF VIOLATION - SAN ONOFRE NUCLEAR GENERATING STATION - NRC
SPECIAL INSPECTION REPORT 05000361/2008013; 05000362/2008013
Dear Mr. Ridenoure:
On December 11, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed a special
inspection at your San Onofre Nuclear Generating Station facility. This inspection examined
activities associated with deficient electrical connections with the potential to adversely affect
the safety function of multiple safety systems used for accident mitigation. The NRC's initial
evaluation satisfied the criteria in NRC Management Directive 8.3, NRC Incident Investigation
Program, for conducting a special inspection. The basis for initiating this special inspection is
further discussed in the inspection charter, which is included in this report as Attachment 2. The
determination that the inspection would be conducted was made by the NRC on July 21, 2008,
and the inspection started on August 4, 2008.
The enclosed special inspection report documents the inspection results, which were discussed
on November 5 and December 11, 2008 with you and other members of your staff. The
inspection examined activities conducted under your license as they relate to safety and
compliance with the Commission's rules and regulations and with the conditions of your license.
The team reviewed selected procedures and records, observed activities, and interviewed
personnel.
The enclosed report documents one finding that was determined to be of low to moderate safety
significance (White). As described in Sections 2.1.5 and 3.4, of this report, the NRC concluded
that the failure to establish appropriate instructions in March 2004 for replacement of the Unit 2
safety-related Battery 2B008 output breaker resulted in the battery being inoperable between
March 2004 and March 25, 2008. Specifically, on March 25, 2008, following failure of a battery
voltage surveillance activity it was identified that loose electrical connections associated with the
battery output breaker were the cause of the failed surveillance. This finding does not represent
an immediate safety concern because of the corrective actions you have taken that involved
tightening the loose battery breaker connections and verifying all other battery output breaker
connections were tight following identification of the loose electrical connection. The safety
Southern California Edison -2-
significance of this finding was assessed on the basis of the best available information, including
influential assumptions, using the applicable Significance Determination Process and was
determined to be White (i.e., low to moderate safety significance). Attachment 3 of this report
provides a detailed description of the NRCs risk assessment.
This finding was determined to involve a violation of NRC requirements. You are required to
respond to this letter and should follow the instructions specified in the enclosed Notice when
preparing your response. In addition, we will use the NRC Action Matrix to determine the most
appropriate NRC response to this issue, and we will notify you by separate correspondence of
that determination.
Following a discussion of the preliminary safety significance of this finding during the exit
briefing on November 5, 2008, a phone call was held between Michael Hay, Branch Chief,
Division of Reactor Projects, and Ed Scherer, Manager, Nuclear Regulatory Affairs, on
November 13, 2008. During this call Mr. Scherer indicated that Southern California Edison does
not contest the characterization of the risk significance of this finding, and that you have
declined to further discuss this issue at a Regulatory Conference or provide a written response.
Accordingly, the NRC is issuing this final significance determination for the inspection finding.
This report also discusses seven NRC identified findings that were determined to be of very low
safety significance. Of concern is that these findings were identified by the NRC following your
review of the events prior to our announced special inspection indicating your evaluations
lacked the rigor necessary to identify these performance deficiencies. Your ability to effectively
identify and evaluate problems has been, and continues to be, a concern to the NRC. This
concern was documented in the past two NRC assessment letters dated March 3 and
September 2 of 2008. These seven findings will be assessed during our end of cycle
assessment along with other findings identified during calendar year 2008 to assess your
progress in addressing the substantive cross-cutting issue in problem identification and
resolution. The NRC will continue to focus our inspections in this area and evaluate if additional
actions are warranted until sustained improvements are recognized.
The seven NRC identified findings were determined to be of very low safety significance
(Green). The findings were determined to involve violations of NRC requirements. Because of
their very low safety significance and because they were entered into your corrective action
program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1
of the NRC Enforcement Policy. If you contest these noncited violations, you should provide a
response within 30 days of the date of this inspection report, with the basis for your denial, to
the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington
DC 20555-0001; with copies to the Regional Administrator, U.S. Nuclear Regulatory
Commission Region IV, 612 E. Lamar Boulevard, Suite 400, Arlington, Texas, 76011-4125; the
Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington,
DC 20555-0001; and the NRC Resident Inspector at the San Onofre Nuclear Generating
Station facility.
Southern California Edison -3-
In accordance with 10 CFR 2.390 of the NRC's Rules of Practice, a copy of this letter, its
enclosure, and your response (if any) will be made available electronically for public inspection
in the NRC Public Document Room or from the Publicly Available Records (PARS) component
of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at
http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Elmo E. Collins
Regional Administrator
Docket Nos. 50-361
50-362
License Nos. NPF-10
Enclosure 1: Notice of Violation
Enclosure 2: Inspection Report 05000361/2008013; 05000362/2008013
Attachment 1: Supplemental Information
Attachment 2: Special Inspection Charter
Attachment 3: Significance Determination Evaluation
cc w/Enclosure:
Chairman, Board of Supervisors Dr. David Spath, Chief
County of San Diego Division of Drinking Water and
1600 Pacific Highway, Room 335 Environmental Management
San Diego, CA 92101 California Department of Health Services
850 Marina Parkway, Bldg P, 2nd Floor
Gary L. Nolff Richmond, CA 94804
Assistant Director-Resources
City of Riverside Michael J. DeMarco
3900 Main Street San Onofre Liaison
Riverside, CA 92522 San Diego Gas & Electric Company
8315 Century Park Ct. CP21G
Mark L. Parsons San Diego, CA 92123-1548
Deputy City Attorney
City of Riverside Director, Radiological Health Branch
3900 Main Street State Department of Health Services
Riverside, CA 92522 P.O. Box 997414 (MS 7610)
Sacramento, CA 95899-7414
Southern California Edison -4-
Mayor
City of San Clemente
100 Avenida Presidio
San Clemente, CA 92672
James D. Boyd, Commissioner
California Energy Commission
1516 Ninth Street (MS 34)
Sacramento, CA 95814
Douglas K. Porter, Esq.
Southern California Edison Company
2244 Walnut Grove Avenue
Rosemead, CA 91770
Albert R. Hochevar
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92675
A. Edward Scherer
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
Mr. Steve Hsu
Department of Health Services
Radiologic Health Branch
MS 7610, P.O. Box 997414
Sacramento, CA 95899-7414
Mr. Mike Short
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
Chief, Radiological Emergency
Preparedness Section
National Preparedness Directorate
Technological Hazards Division
Department of Homeland Security
1111 Broadway, Suite 1200
Oakland, CA 94607-4052
Southern California Edison Company -5-
Electronic distribution by RIV:
RidsSecyMailCenter Resource RidsOcaMailCenter Resource
RidsOEMailCenter Resource RidsEdoMailCenter Resource
RidsOgcMailCenter Resource RidsNrrMailCenter Resource
RidsNroOd Resource RidsOpaMail Resource
RidsOiMailCenter Resource RidsOigMailCenter Resource
RidsOcfoMailCenter Resource RidsRgn1MailCenter Resource
RidsRgn3MailCenter Resource OEWEB Resource
RidsRgn2MailCenter Resource
Elmo.Collins@nrc.gov Dwight.Chamberlain@nrc.gov
Nick.Hilton@nrc.gov
Chuck.Casto@nrc.gov Anton.Vegel@nrc.gov June.Cai@nrc.gov
Karla.Fuller@nrc.gov Roy.Caniano@nrc.gov John.Wray@nrc.gov
WilliamJones@nrc.gov Troy.Pruett@nrc.gov MaryAnn.Ashley@nrc.gov
Mark.Haire@nrc.gov Dale.Powers@nrc.gov Russ.Barnes@nrc.gov
Christi.Maier@nrc.gov Shawn.Williams@nrc.gov
Bill.Maier@nrc.gov Dale.Powers@nrc.gov Alexander.Sapountzis@nrc.gov
Victor.Dricks@nrc.gov Michael.Hay@nrc.gov Doug.Starkey@nrc.gov
Marissa.Herrera@nrc.gov Don.Allen@nrc.gov Gerald.Gulla@nrc.gov
Greg.Warnick@nrc.gov John.Reynoso@nrc.gov
Heather.Hutchinson@nrc.gov Chuck.Paulk@nrc.gov Doug.Bollock@nrc.gov
Samuel.Graves@nrc.gov David.Loveless@nrc.gov Mica.Baquera@nrc.gov
ROPreports@nrc.gov
SUNSI Review Completed: mch ADAMS: x Yes No Initials: mch
x Publicly Available Non-Publicly Available Sensitive x Non-Sensitive
R:\_REACTORS\_SO\2008\SO2008-013RP-GGW.doc ML 083540244
RIV:DRS/PSB2 DRS/EB1 SRI:DRP/D SRA ACES
MBaquera SGraves GWarnick DLoveless MHaire
/RA/ /RA/ /RA - E/ /RA/ /RA/ by wbj
12/9/08 12/9/08 12/10/08 12/12/08 12/15/2008
MCHay KSFuller MAshley DChamberlain NHilton
/RA/ /RA/ /RA/ /RA/ /RA/
12/13/2008 12/10/2008 12/16/2008 12/15/2008 12/17/2008
EECollins
/RA/
Southern California Edison Company -6-
12/19/2008
OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax
NOTICE OF VIOLATION
Southern California Edison Company Docket No. 50-361
San Onofre Nuclear Generating Station License No. NPF-10
During an NRC inspection completed on December 11, 2008, a violation of NRC requirements
was identified. In accordance with the NRC Enforcement Policy, the violation is listed below:
10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings,
states, in part, that activities affecting quality shall be prescribed by documented
instructions, procedures, or drawings of a type appropriate to the circumstances and
shall be accomplished in accordance with these instructions, procedures, or drawings.
Instructions and procedures shall include appropriate quantitative or qualitative
acceptance criteria for determining that important activities have been satisfactorily
accomplished.
Contrary to the above, in March 2004, the licensee engaged in activities affecting quality
that were not prescribed by documented instructions or procedures of the type
appropriate to the circumstances. Specifically, maintenance and work control personnel
failed to develop appropriate instructions or procedures, and failed to include quantitative
or qualitative steps to ensure the maintenance activities on safety-related 125 Vdc
station battery Breaker 2D201 had been satisfactorily completed. The work plan
described in Maintenance Order 03100406000 was incomplete and lacked the steps
necessary to ensure that electrical connection fasteners on Breaker 2D201 upper stud to
bus bar connections were properly installed. This failure resulted in the Unit 2 safety-
related Battery 2B008 being inoperable between March 2004 and March 25, 2008.
This violation is associated with a White significance determination process finding.
Pursuant to the provisions of 10 CFR 2.201, Southern California Edison is hereby required to
submit a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN.:
Document Control Desk, Washington, DC 20555-0001 with a copy to the Regional
Administrator, Region IV, and a copy to the NRC Resident Inspector at the facility that is the
subject of this Notice, within 30 days of the date of the letter transmitting this Notice of Violation
(Notice). This reply should be clearly marked as a Reply to a Notice of Violation; EA-08-296,
and should include for each violation: (1) the reason for the violation, or, if contested, the basis
for disputing the violation or severity level; (2) the corrective steps that have been taken and the
results achieved; (3) the corrective steps that will be taken to avoid further violations and (4) the
date when full compliance will be achieved. Your response may reference or include previous
docketed correspondence, if the correspondence adequately addresses the required response.
If an adequate reply is not received within the time specified in this Notice, an order or a
Demand for Information may be issued as to why the license should not be modified,
suspended, or revoked, or why such other action as may be proper should not be taken. Where
good cause is shown, consideration will be given to extending the response time.
If you contest this enforcement action, you should also provide a copy of your response, with
the basis for your denial, to the Director, Office of Enforcement, United States Nuclear
Regulatory Commission, Washington, DC 20555-0001.
Because your response will be made available electronically for public inspection in the NRC
Public Document Room or from the NRCs document system (ADAMS), accessible from the
NRC Web site at http://www.nrc.gov/reading-rm/adams.html, to the extent possible, it should not
include any personal privacy, proprietary, or safeguards information so that it can be made
available to the public without redaction. If personal privacy or proprietary information is
necessary to provide an acceptable response, then please provide a bracketed copy of your
response that identifies the information that should be protected and a redacted copy of your
response that deletes such information. If you request withholding of such material, you must
specifically identify the portions of your response that you seek to have withheld and provide in
detail the bases for your claim of withholding (e.g., explain why the disclosure of information will
create an unwarranted invasion of personal privacy or provide the information required by
10 CFR 2.390(b) to support a request for withholding confidential commercial or financial
information). If safeguards information is necessary to provide an acceptable response, please
provide the level of protection described in 10 CFR 73.21.
Dated this 19 day of December 2008
-2- Enclosure 1
U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Docket: 50-361, 50-362
Report No.: 05000361/20078013; 05000362/2008013
Licensee: Southern California Edison Co. (SCE)
Facility: San Onofre Nuclear Generating Station, Units 2 and 3
Location: 5000 S. Pacific Coast Hwy.
San Clemente, California
Dates: August 4 through December 11, 2008
Team Leader: G.G. Warnick, Senior Resident Inspector, Project Branch D, DRP
Team: M.T. Baquera, Reactor Inspector, Plant Support Branch , DRS
S.T. Graves, Reactor Inspector, Engineering Branch 1, DRS
Accompanying G.B. Skinner, Electrical Contractor (Beckman)
Personnel:
Approved By: Elmo Collins, Regional Administrator
-1- Enclosure 2
SUMMARY OF FINDINGS
IR 05000361/2008013, 05000362/2008013; 08/04/2008 - 12/11/2008; San Onofre Nuclear
Generating Station, Units 2 and 3;
The report covered a 5-day period (August 4 - August 8, 2008) of onsite inspection, with in-
office review through December 11, 2008, by a special inspection team consisting of one senior
resident inspector, two reactor inspectors, and one electrical contractor. Eight findings were
identified. The significance of most findings is indicated by its color (Green, White, Yellow, or
Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings for
which the significance determination process does not apply may be Green or be assigned a
severity level after NRCs management review. The NRC's program for overseeing the safe
operation of commercial nuclear power reactors is described in NUREG-1649, Reactor
Oversight Process, Revision 3, dated July 2000.
Summary of Event
The NRC conducted a special inspection to better understand the circumstances surrounding
deficient electrical connections. In accordance with NRC Management Directive 8.3, NRC
Incident Investigation Program, it was determined that these deficient electrical connection
events potentially involved multiple failures in systems used to mitigate the effects of an actual
event, involved potential adverse generic implications, and had sufficient risk significance to
warrant a special inspection.
A. NRC-Identified and Self-Revealing Findings
Cornerstone: Initiating Events
- Green. The team identified a Green noncited violation of 10 CFR Part
50.65(a)(4) involving the failure to adequately assess the increase in risk and
effectively implement risk mitigation actions for emergent maintenance activities.
Specifically, on March 25 and March 26, 2008, the licensee failed to consider the
risk associated with the increased likelihood of an initiating event during
emergent work on energized safety-related 125 Vdc battery breakers. This issue
was entered into the licensees corrective action program as Nuclear Notification
200196248.
This finding is greater than minor because the licensees risk assessment failed
to consider that the maintenance activities on the 125 Vdc breakers could
increase the likelihood of initiating events. The finding is of very low safety
significance based on a senior reactor analyst bounding risk estimation that
assuming the performance deficiency resulted in operating the plant in an
elevated risk configuration during emergent maintenance activities for a 24-hour
period. The finding has a crosscutting aspect in the area of human performance
associated with resources for the failure to provide appropriate risk management
tools by maintaining complete, accurate, and up-to-date procedures H.2(c)
(Sections 2.1.4 and 3.4).
-2- Enclosure 2
Cornerstone: Mitigating Systems
- Green. The team identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure of an electrical maintenance supervisor to follow
procedures after notification that Battery 2B008 terminal voltage was less than
the TS required value of 129 Vdc. Specifically, the supervisor failed to notify the
control room shift supervisor after being informed of a failed battery surveillance
activity. The failure to follow procedures resulted in more than a two hour delay
in entering the required 2-hour technical specification action statement. This
issue was entered into the licensees corrective action program as Nuclear
Notification 200196248.
The finding is greater than minor because it is associated with the equipment
performance attribute of the mitigating systems cornerstone and affects the
associated cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. The finding is of very low safety significance based on a senior
reactor analyst risk estimation assuming the performance deficiency resulted in
operating the plant with an inoperable 125 Vdc battery for an additional period of
2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br />. The cause of the finding is related to the crosscutting element of
human performance associated with decision making because personnel did not
make safety significant decisions using a systematic process when faced with
uncertain and unexpected plant conditions to ensure safety was maintained.
This included the failure to formally define the authority and roles of the electrical
maintenance supervisors for decisions affecting nuclear safety H.1(a) (Sections
2.1.2 and 3.1).
- Green. The team identified a Green noncited violation of Technical Specification 5.5.1.1, for the failure of electrical maintenance personnel to follow Procedure
SO123-XX-1, Action Request/Maintenance Order Initiation and Processing,
Revision 20. Specifically, following identification of a failed 125 Vdc battery
surveillance, troubleshooting activities were performed without a maintenance
order and control room authorization. This issue was entered into the licensees
corrective action program as Nuclear Notification 200196248.
The finding is greater than minor because it would become a more significant
safety concern if left uncorrected in that more significant consequences could
occur if work control procedures are not followed when performing maintenance
on safety-related structures, systems, and components. The finding affected the
mitigating systems cornerstone. The finding is of very low safety significance
based on a senior reactor analyst estimation assuming the performance
deficiencies resulted in operating the plant with an inoperable 125 Vdc battery for
a period of 2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br /> while troubleshooting activities were conducted. The
finding has a crosscutting aspect in the area of human performance associated
with decision making because the electrical maintenance personnel did not make
safety significant decisions using a systematic process, especially when faced
with uncertain or unexpected plant conditions [H.1.(a)] (Sections 2.1.2 and 3.2).
-3- Enclosure 2
- Green. The team identified a Green noncited violation of Technical Specification 5.5.1.1, for the failure of electrical maintenance personnel to follow Procedure
SO123-XX-5, Work Authorizations, Revision 17. Specifically, work to correct
the identified degraded electrical condition was initiated prior to having an
appropriately authorized maintenance order. This issue was entered into the
licensees corrective action program as Nuclear Notification 200196248.
The finding is greater than minor because it would become a more significant
safety concern if left uncorrected in that more significant consequences would
occur if work control procedures are not followed when performing maintenance
on safety-related structures, systems, and components. The finding affected the
mitigating systems cornerstone. Using the Manual Chapter 0609, "Significance
Determination Process," Phase 1 Worksheets, the finding is determined to have
very low safety significance because it was not a design or qualification
deficiency, did not result in a loss of safety function, and did not screen as
potentially risk significant due to external events. The finding has a crosscutting
aspect in the area of human performance associated with work practices
because the licensee did not perform adequate pre-job briefings and did not
properly document the maintenance activities H.4(a) (Sections 2.1.3 and 3.3).
- White. The team identified a White violation of 10 CFR Part 50, Appendix B,
Criterion V, Instructions, Procedures, and Drawings, involving the failure to
establish appropriate instructions for performing maintenance activities on safety-
related 125 Vdc station battery Breaker 2D201. As a result, during replacement
of the breaker in March 2004 electrical connection integrity was not adequate to
ensure that the equipment would be able to perform its safety function. This
condition existed for approximately four years. This issue was entered into the
licensees corrective action program as Root Cause Evaluation 800121216.
The finding is greater than minor because it is associated with the equipment
performance attribute of the mitigating systems cornerstone and affects the
associated cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. The final significance determination performed by the senior
reactor analyst and approved by the NRC significance and enforcement review
panel determined the finding was of low to moderate safety significance (White).
This finding has a crosscutting aspect in the area of human performance
associated with resources because the licensee failed to establish adequate
procedures and programs related to electrical connection integrity H.2(c)
(Sections 2.1.5 and 3.5).
- SL-IV. The team identified a Severity Level IV noncited violation of 10 CFR Part
50.73 for the failure of the licensees regulatory compliance organization to
submit a required Licensee Event Report within 60 days after discovering an
event requiring a report. Specifically, compliance personnel failed to properly
assess the past operability of the safety-related 125 Vdc Battery 2B008, which
had been inoperable for greater than the technical specification allowed outage
time. This issue was entered into the licensees corrective action program as
Nuclear Notification 200059017.
-4- Enclosure 2
The finding was determined to be applicable to traditional enforcement because
the NRCs ability to perform its regulatory function was potentially impacted by
the licensees failure to report the events. The finding was determined to be a
Severity Level IV violation in accordance with Section D.4 of Supplement I of the
The finding has a crosscutting aspect in the area of problem identification and
resolution associated with CAP because the licensee failed to thoroughly
evaluate problems such that the resolutions address causes and extent of
conditions. This includes properly classifying, prioritizing, and evaluating for
operability and reportability conditions adverse to quality P.1(c) (Sections 2.1.6
and 3.6).
- Green. The team identified a Green noncited violation of 10 CFR Part 50,
Appendix B, Criterion XVI, Corrective Actions, for the licensees failure to
establish measures to assure that deficient electrical connections were promptly
identified and corrected. The licensees measures were not adequate to assure
that a long standing degraded electrical connection was identified for correction
during three inspection opportunities associated with safety-related Breaker
3BD21, Diesel Radiator Fan 3E550 Feeder Breaker, that occurred between
June 2005 and April 2008. This issue was entered into the licensees corrective
action program as Nuclear Notification 200047962.
The finding is greater than minor because it is associated with the equipment
performance attribute of the mitigating systems cornerstone and affects the
associated cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. Using the Manual Chapter 0609, Significance Determination
Process, Phase 1 Worksheets, the finding is determined to have very low safety
significance because the condition did not represent an actual loss of safety
function of a single train for greater than its technical specification allowed outage
time, and did not represent an actual loss of one or more risk-significant non-
technical specification trains of equipment for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. This finding
has a crosscutting aspect in the area of problem identification and resolution
associated with corrective action program because the licensee failed to
thoroughly evaluate problems such that the resolutions address causes and
extent of conditions. This includes properly classifying, prioritizing, and
evaluating for operability and reportability conditions adverse to quality. This also
includes, for significant problems, conducting effectiveness reviews of corrective
actions to ensure that the problems are resolved P.1(c) (Section 3.7).
- Green. The team identified a Green noncited violation of Technical Specification 5.5.1.1 for the failure to establish written procedures for a loss or degradation of
a safety-related electrical power source. Specifically, no procedural guidance
was provided to operations personnel to combat and recover from a loss or
degradation of a Class 1E 125 Vdc bus. This issue was entered into the
licensees corrective action program as Nuclear Notifications 20060584 and
200196248.
-5- Enclosure 2
The finding is greater than minor because it is associated with the procedure
quality attribute of the mitigating systems cornerstone and affects the associated
cornerstone objective to ensure the availability, reliability, and capability of
systems that respond to initiating events to prevent undesirable consequences.
Using the Manual Chapter 0609, "Significance Determination Process," Phase 1
Worksheets, the finding is determined to have very low safety significance
because it was not a design or qualification deficiency, did not result in a loss of
safety function, and did not screen as potentially risk significant due to external
events. This finding was reviewed for crosscutting aspects and none were
identified (Section 3.8).
B. Licensee-Identified Violations
None.
-6- Enclosure 2
REPORT DETAILS
1.0 SPECIAL INSPECTION SCOPE
The NRC conducted a special inspection at San Onofre Nuclear Generating Station
(SONGS) to better understand the circumstances surrounding deficient electrical
connections with the potential to adversely affect the safety function of multiple safety
systems used for accident mitigation.
The team used NRC Inspection Procedure 93812, Special Inspection Procedure, to
conduct the inspection. The special inspection team reviewed procedures, corrective
action documents, operator logs, design documentation, and maintenance records for
various deficient electrical connection issues. The team interviewed various station
personnel regarding one event, in particular, which occurred on March 25, 2008,
associated with a degraded 125 Vdc battery terminal voltage. The team reviewed the
licensees apparent and root cause evaluations (RCE), directed assessment reports
(DAR), past failure records, extent of condition evaluations, immediate and long term
corrective actions, and industry operating experience (OE). A list of specific documents
reviewed is provided in Attachment 1. The charter for the special inspection is included
as Attachment 2.
2.0 SPECIAL INSPECTION OBSERVATIONS
2.1 Battery Breaker Loose Connections
2.1.1 NRC Review of Licensee Evaluations
On March 25, 2008, electrical maintenance personnel identified that terminal voltage for
Battery 2B008 was at 121.29 Vdc, which was below the Technical Specification (TS)
limit of 129 Vdc. Troubleshooting discovered that loose bolts at the battery to breaker
terminal connection on Breaker 2D201 was the cause for the degraded battery voltage.
Operations personnel declared the battery inoperable and entered TS 3.8.4 Limiting
Condition for Operation (LCO), Condition A, which required restoration of the DC
electrical power subsystem within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. The licensee initiated repairs after TS LCO 3.8.4, Condition A, was entered. Since the degraded condition was not corrected within
2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />, the licensee entered TS LCO 3.8.4, Condition B, to commence a plant
shutdown. However, the plant shutdown was suspended 10 minutes after Condition B
was entered when all repairs on Breaker 2D201 were completed and Battery 2B008 was
declared operable. Over the next day, maintenance verified that other similar battery
breaker bus bolts were tight.
The licensee performed an apparent cause evaluation (ACE) assignment for Action
Request (AR) 080301117 to evaluate the March 25, 2008, events associated with the
failed surveillance. The ACE identified that the degraded battery voltage on Battery
2B008 was caused by a degraded electrical connection that had developed as a result of
the loose bolts on Breaker 2D201. The ACE also documented that the most probable
cause for the loose connections occurred during installation of a new thermal trip device
on the breaker in March 2004 using Maintenance Order (MO) 03100406000.
AR 080301117 also included a field support assignment, performed by engineering
personnel, to create OE for communication to the industry. The field support assignment
-7- Enclosure 2
stated the battery may not have been able to support its DC bus loads while the battery
breaker connection was degraded. Despite the statements documented in the ACE and
field assignments, regulatory compliance personnel concluded that there were no past
operability concerns with the degraded battery breaker connection since they
independently determined that the condition was failed when found.
In July 2008, the NRC resident inspectors performed an initial review of AR 080301117
and questioned the conclusions of regulatory compliance personnel since information in
the ACE and field support assignments provided information that contradicted the
conclusions of the reportability assessment. The inspectors challenged the failed when
found conclusion which prompted the licensee to reevaluate the potentially reportable
condition.
The inspectors observed that the purpose of ACE assignment for AR 080301117 was to
determine the cause of the loose bolts and implement corrective actions to minimize the
chance of recurrence. The evaluation identified that the cause of the loose connection
was an individual performance error during installation of a new thermal trip device on
the breaker in March 2004. Specifically, the evaluation determined that the electrician
did not demonstrate the competency expected of maintenance personnel, in that,
maintenance personnel are expected to correctly complete and accurately document all
aspects of the job. The evaluation did, however, identify that the MO work plan steps did
not specifically address torquing the breaker bolts and relied on skill-of-the-craft over
detail and defense in depth to ensure successful torquing of the breaker bolts. However,
no actions were taken to address the procedural inadequacies since it was concluded
that it was not a current problem because greater emphasis had been placed on the
identification and mitigation of critical steps in work plans since the 2003 timeframe that
the MO was planned. The evaluation focused on the human performance aspects and
determined that no current problem existed since the errors associated with
Breaker 2D201 that occurred in 2004 was prior to several initiatives in maintenance to
improve human performance. Consequently, the corrective actions identified consisted
only of individual coaching and training to reinforce human performance expectations.
Further, the inspectors observed that, in general, the licensee believed that the
organization performed well in responding promptly to the failed surveillance, initiating
the unit shutdown and immediate troubleshooting and corrective actions regarding the
failed surveillance once the condition was discovered on March 25, 2008.
The inspectors determined that the licensees evaluation of the condition in
AR 080301117 for the loose battery breaker bolts was inadequate in that it failed to
recognize the significance of the condition and address past operability and reportability.
The inspectors determined that the degraded battery breaker connection issue was
potentially safety significant. Additionally, the inspectors performed an extent of
condition review and identified additional examples of loose electrical terminations on
safety-related equipment. On July 21, 2008, the decision was made to perform a special
inspection as a result of the follow up inspections performed by the inspectors.
As a result of the inspectors identification of the inadequacies associated with
AR 080301117, and the decision to perform a special inspection, the licensee performed
RCE 800121216, Inadequate Maintenance Activity Results in Loose Battery Breaker
Connection in 2D201, and RCE 200059017, Deficiencies Associated with the 2D201
Breaker Connection Reportablity Assessment, just prior to the commencement of the
-8- Enclosure 2
special inspection. The RCEs were presented to the team for review at the beginning of
the special inspection. The team was told that the RCEs represented a comprehensive
and thorough evaluation of the events.
The team reviewed RCE 800121216 and observed that the licensee concluded that the
causes associated with the loose battery breaker bolts were more programmatic rather
than an individual performance error as previously identified in AR 080301117.
Specifically, the licensee identified that the event was caused by inadequate procedure
use, and inadequacies associated with work planning procedures and training when
MO 03100406000 was planned in 2003. The evaluation also concluded that the
underlying problems still exist presently as evidenced by recent events and evaluations,
in addition to the substantive crosscutting issue in the area of human performance for
failing to provide adequate procedures or work instructions described in NRC
assessment letters dated March 3, 2008, and September 2, 2008.
The evaluation performed in RCE 200059017 concluded that the event was reportable.
The evaluation was thorough with respect to deficiencies associated with the inadequate
reportability review for AR 080301117. However, the sequence of events presented in
the report was inaccurate (see timelines below). The evaluation also identified a
previous failure to submit a licensee event report (LER) when required. The previous
failure was identified by the NRC in 1997. The cause evaluation for the 1997 event
found many of the same weaknesses in the reportability review process that were
identified in RCE 200059017. However, the corrective actions from the 1997 event were
either not implemented or were ineffective over the long term. The failure to implement
corrective actions from the 1997 event contributed to the failure to adequately assess for
reportability the degraded battery voltage event that occurred on March 25, 2008.
Additionally, the licensee failed to identify corrective actions for some of the causes that
were identified in the evaluation. One noteworthy example involved the identification
that inadequate resources in the Compliance/Nuclear Regulatory Affairs Organization,
contributed to ineffective management of corrective action backlogs, and may have been
a potential underlying issue that resulted in the failure to perform an adequate
reportability assessment.
The team concluded the RCEs were too narrowly focused on the specific issues
associated with the failure to tighten the battery breaker bolts in 2004, and the
inadequate reportability review for AR 080301117. Consequently, the evaluations
lacked the rigor necessary to identify all performance deficiencies associated with the
event for development of adequate corrective actions to address all root and contributing
causes. The failure to thoroughly evaluate problems such that the resolutions address
causes and extent of conditions has been previously identified during past NRC special
inspections, and was the focus of a substantive crosscutting issue in the area of problem
identification and resolution described in NRC Assessment Letters dated March 3, 2008,
and September 2, 2008.
Timeline of Events Identified by Licensee
The licensee maintained that the organization performed well in responding to the
degraded battery voltage that was identified on March 25, 2008. This conclusion was
supported by the following sequence of the events as documented in the licensees
corrective action program (CAP):
-9- Enclosure 2
March 25, 2008
~0550 Electricians discovered low voltage at Battery 2D2 during surveillance testing
and reported the condition to the responsible supervisor.
~0610 The loose bolting connections were discovered during troubleshooting
activities.
~0615 The Manager of Electrical maintenance discussed the loose connection issue
with the Director of Operations.
~0630 The responsible electrical supervisor documented the adverse condition in AR
080301117.
0640 Operations Log noted: D2 battery declared inoperable as a result of
electricians finding loose connection on battery breaker (battery side) while
performing weekly battery checks. Per TS 3.8.4, Condition A, Unit 2 entered a
2 hour2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> action to restore battery to operable or be in Mode 3 in 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and
Mode 5 in 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. Notified Operations management.
0715 Electricians commenced troubleshooting and corrective maintenance. As
directed by supervision and Step 1 of MO 08031721000 removed protective
covers to access breaker bus connections. Discovered loose bolts on the
battery side of the breaker bus connection.
0840 Operations Log noted: Initiated MSR Cooldown per SO23-10-2, Attachment 5.
GOC notified. Entered 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> shutdown to Mode 3 per LCO 3.8.4, Action B.
0850 Operations Log noted: Exited LCO 3.8.4, Action B after tightening the loose
cable connection from 2D2 Battery (B008) to the 2D2 Bus battery breaker, and
completion of a satisfactory quarterly surveillance. Secured from MSR
Cooldown, GOC and Chemistry notified.
Timeline of Events Identified by the Team
The team evaluated the timeline of events for March 25, 2008, through a review of vital
area door access logs, control room log entries, MO records, and interviews with
personnel involved. The licensees documentation of the event was not consistent with
information that the team identified during their review. There were four periods of time
throughout this event that the team evaluated. This evaluation of the time periods was
performed to assess the effectiveness of licensees actions taken in response to the
electrical connection deficiencies. Based on this evaluation, the following timeline was
developed:
- 10 - Enclosure 2
March 25, 2008
INITIAL DISCOVERY OF DEGRADED CONDITON
0408 The electricians entered the battery equipment area on the 50 elevation of the
control building.
0410 The electricians began the weekly battery surveillance on Battery 2B008. The
electricians identified that the measured voltage was less than the acceptance
criteria, constituting an unsatisfactory (UNSAT) condition and failed
surveillance. The electricians validated the degraded battery voltage reading.
0415 Electrical maintenance supervisor notified of failed surveillance and the
condition. The supervisor instructed the electricians to discontinue the
surveillance on Battery 2B008, and continue the surveillance on the remaining
batteries.
0439 Electrical maintenance supervisor entered the battery equipment area.
TROUBLESHOOTING DEGRADED CONDITION
0445 The electrical supervisor verified the UNSAT readings on Battery 2B008. The
supervisor decided that his immediate supervisor, the dayshift electrical
maintenance General Foreman, should be notified prior to additional actions.
0500 The electrical supervisor called the General Foreman, described the condition,
and requested that he come to the battery equipment area.
0538 The General Foreman and other electrical maintenance supervisors arrived at
the battery equipment area to investigate the cause of the degraded battery
voltage.
0540 Electrical maintenance supervision, including the General Foreman, re-
validated the degraded voltage readings by performing measurements at
various points in the circuit to determine the cause of the degraded battery
voltage.
0550 The General Foreman took measurements inside of the cubicle for Breaker
2D201. Movement on a bolt was noted while placing a measuring probe on the
battery to breaker connection and the voltage reading returned to normal.
Coincident with this event, the 2D2 Trouble Alarm was received in the control
room.
0555 The control room operator dispatched to investigate the 2D2 Trouble Alarm
entered the battery equipment area and reported that an Army of Guys were
assembled in the area. The control room supervisor directed the General
Foreman to come to the control room.
- 11 - Enclosure 2
CONTROL ROOM NOTIFICATION
0603 The General Foreman entered the control room to describe the situation to the
control room supervisor.
0615 The control room supervisor contacted the shift manager and informed him that
there had been anomalous voltage readings taken on Battery 2B008.
0628 Condition documented on AR 0803001117.
0635 The Electrical Maintenance Manager discussed the situation, including the fact
that there were loose bolts on Breaker 2D201, with the shift manager. Actions
necessary to repair the degraded connection were discussed.
0640 Battery 2B008 was declared Inoperable as a result of electricians finding loose
connection on battery Breaker 2D201 and TS LCO 3.8.4, Condition A, was
entered.
CORRECTIVE MAINTENANCE
0700 Electricians were briefed on the emergent battery breaker maintenance and
were instructed to begin work to correct the condition. Eight bolts were found
loose at the top side of the Breaker 2D201 to Battery 2B008 connections.
0840 TS LCO 3.8.4 action time expired. TS LCO 3.8.4, Condition B, was entered
requiring a plant shutdown.
0850 Exited TS LCO 3.8.4, Condition B, after the loose bolts on the Breaker 2D201
to Battery 2B008 connections were tightened, and a quarterly battery
surveillance test was satisfactorily completed.
The teams evaluation of the event timeline identified additional observations (Sections
2.1.2 through 2.1.4) that were not identified by the licensees evaluations. The
inadequacies associated with the licensees evaluations for this event are similar to
inadequacies that the NRC has identified in their follow up of other events during past
special inspections. The team noted that the licensee's evaluation lacked the rigor
necessary to ensure an accurate assessment of their responses to the degraded battery
connections.
2.1.2 Discovery of Degraded Battery Condition
On March 25, 2008, electricians were in the progress of performing the weekly
surveillance on safety-related Battery 2B008 per Procedure SO123-I-2.2, 125 Vdc Pilot
Cell Battery Inspection, Revision 7. This surveillance satisfied the requirements of TS
Surveillance Requirement 3.8.4.1. The electricians measured battery bank terminal
voltage per Procedure SO123-I-2.2, Step 6.2, and identified that the measured voltage
was less than the acceptance criteria of 129 Vdc. The measured voltage was
121.29 Vdc, constituting an UNSAT condition and failed surveillance. The electricians
validated the degraded battery voltage reading and immediately notified their supervisor
as required by Procedure SO123-I-2.2. Procedure SO123-I-2.2, Step 6.2.1.2, stated
- 12 - Enclosure 2
that, This supervisor SHALL report a failed surveillance according to
Procedure SO123-I-1.3. Procedure SO123-I-1.3, Work Activity Guidelines, Revision
14, required that, A SUPERVISOR SHALL immediately provide written notification to
the shift supervisor for any surveillance found failed.
The team noted that the electrical maintenance supervisor notified of the UNSAT
condition by the electricians did not immediately inform the operations shift supervisor as
required by procedural guidance. The supervisor was acting in an upgrade capacity and
inappropriately understood that he was expected to notify the electrical maintenance
general foreman prior to taking further action. The team determined that the upgrade
supervisors inaction was, in part, a result of ineffective supervisor training and unclear
expectations. Instead of notifying the operations shift supervisor as required, electrical
maintenance supervision, which included the nightshift supervisor and dayshift general
foreman, performed unauthorized troubleshooting to more fully understand the cause of
the degraded terminal voltage. The team observed that the behaviors of the electrical
maintenance supervisors were such that an understanding of the cause, or explanation
for the UNSAT reading, was desired before reporting the condition outside of the
electrical maintenance organization.
As previously discussed operations personnel became aware of the degraded battery
condition when an alarm annunciated in the control room as a result of the unauthorized
troubleshooting. Following additional discussions between maintenance and operations
personnel to reach an understanding of the degraded voltage reading, Battery 2B008
was declared inoperable and TS 3.8.4, Condition A, was entered. This TS entry time
was approximately 2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br /> after the identification of the UNSAT condition.
The failure of the electrical supervisor to immediately provide written notification to the
shift supervisor after being informed of the failed surveillance was identified as a
violation of procedural requirements. Additionally, the night shift electrical supervisor
and dayshift general foreman performing unauthorized troubleshooting activities was
identified as a procedural violation of the work control process. Details for these
violations of Technical Specification 5.5.1.1, Procedures, are discussed in Sections 3.1
and 3.2 of this report.
2.1.3 Correction of the Loose Battery Breaker Connection
On March 25, 2008, electricians identified that the measured terminal voltage on
Battery 2B008 was less than the acceptance criterion of 129 Vdc during a weekly
surveillance. After verifying that the acceptance criterion was not met, the electricians
notified their responsible supervisor of the failed surveillance and the UNSAT condition.
The electrical maintenance supervisor told the electricians that he would come to the
battery equipment area to assess the situation. The supervisor gathered system
drawings, proceeded to the battery equipment area, and performed various
measurements to troubleshoot the cause of the failed surveillance. A while later, at the
beginning of dayshift, the electrical maintenance general foreman and another electrical
supervisor arrived at the battery equipment area. The electrical maintenance
supervisors continued troubleshooting activities to more fully understand the cause of
the degraded voltage condition.
During the troubleshooting activities, the general foreman opened panels on the
- 13 - Enclosure 2
associated breaker, which were labeled as being a Unit Trip Hazard, to investigate the
cause of the degraded voltage readings. While placing a probe on the energized bus
bar, a bolt moved, and the charger was observed to commence battery charging.
Battery 2B008 terminal voltage was re-verified and it was observed that the reading had
returned to normal. The unauthorized troubleshooting activities identified that loose
bolting on the Breaker 2D201 terminal connection was the cause for the degraded
battery voltage. Coincident with the movement of the bolt, the 2D2 Trouble Alarm was
received in the control room. Operations personnel were dispatched to investigate the
cause of the alarm. Upon arrival at the battery equipment area, operations personnel
observed numerous electrical maintenance personnel troubleshooting the degraded
equipment condition. At the request of the control room supervisor, the general foreman
returned with the operator to inform the control room of the situation. This was the first
time that operations personnel became aware that there was an issue with Battery
2B008.
The manager of electrical maintenance discussed the emergent equipment condition
with the shift manager, including actions necessary to repair the degraded connection on
Breaker 2D201. As a result of ineffective communications, work was not appropriately
authorized and an MO was not available prior to initiating work. The manager of
electrical maintenance believed that the corrective maintenance activities would be
performed per the Shift Manager Accelerated Maintenance (SSAM) process. However,
the team was unable to identify any evidence that the requirements associated with
using the SSAM process, contained in Procedure SO123-XX-5, were followed. For
example, the team determined that no shift managers log entry was made to document
implementation of SSAM, as required by procedural guidance, and an advance copy of
the MO was not available prior to initiating work. Further, during an interview the shift
manager did not recall authorizing the use of SSAM. The shift manager understood that
the paperwork required to perform the corrective maintenance was ready, and that he
was providing verbal authorization to the manager of electrical maintenance to
commence work. The team was unable to identify any evidence that the requirements
for verbal authorization, contained in Procedure SO123-XX-5, were followed since the
subject activities were beyond the scope of activities allowed to be performed by verbal
authorization.
The team determined that the repair activities associated with the degraded electrical
connection was identified as a procedural violation of the work control process. Details
associated with this violation of Technical Specification 5.5.1.1, Procedures, are
discussed in Section 3.3 of this report.
2.1.4 Extent of Condition Inspection
On March 25 and 26, 2008, MOs were implemented to verify that other connections
associated with the Units 2 and 3 safety-related battery breakers were properly
tightened. The decision was made to perform the work energized based on time
constraints and the inability to completely de-energize the breaker in the current mode of
operation. Performing the work on energized equipment introduced additional risk since
the area in which the work was performed was restrictive, difficult to access, and
included terminal connections in close proximity to each other. An error in the confined
area could have resulted in a loss of the 125 Vdc bus and a subsequent reactor trip.
- 14 - Enclosure 2
Procedure SO123-XX-10, Maintenance Rule Risk Management Program
Implementation, Revision 4, described the licensees process for implementation of the
requirements of 10 CFR 50.64(a)(4). Procedure SO123-O-A2, Operations Division
Personnel Responsibilities, Revision 9, described the shift technical advisors (STA)
responsibilities. One responsibility of the STA was to perform the maintenance rule risk
management program (MRRMP) once per shift and prior to changing the configuration
of equipment important to safety. The team determined that the MRRMP performed by
the STA on March 25 and 26, 2008, did not appropriately assess and manage the risk
associated with the emergent work activities. The team noted that only industrial safety
precautions were implemented which included the use of insulated tools and blankets for
performing the work. The team determined that these industrial safety measures
resulted in actions that incidentally helped to manage the likelihood of an error that could
have caused an initiating event.
The team concluded that the licensees program lacked specific guidance for
appropriately assessing and managing risk for emergent items that are non-routine, such
as the scope of work performed on March 25 and 26. Procedure SO123-XX-10, stated
that, The MRRMP assessment method may use quantitative approaches, qualitative
approaches, or blended methods. One qualitative item that the assessment should
consider is, The likelihood the maintenance activity will significantly increase the
frequency of a risk-significant initiating event. The team observed that the MRRMP
performed by the STA each shift, inappropriately focuses on the quantitative approach,
and does not incorporate qualitative approaches when conditions warrant.
The failure to assess and manage the risk associated with the increased likelihood of an
initiating event while working on energized safety-related reactor trip hazard equipment
was identified as a violation of 10 CFR Part 50.65(a)(4). Details associated with this
violation are discussed in Section 3.4 of this report.
2.1.5 Cause of the Loose Battery Breaker Connection
On March 25, 2008, while performing a weekly battery surveillance, the terminal voltage
of safety-related Battery 2B008 was measured at 121.29 Vdc. The TS minimum
terminal voltage for this battery is 129 Vdc. The safety function of the Battery 2B008 is
to provide power to the loads on 125 Vdc Bus 2D2 during three types of accident
scenarios: Safety Injection Actuation Signal (SIAS) with Loss of Voltage Signal,
Degraded Grid Voltage with SIAS Signal, and Station Blackout.
Following discovery of the inadequate terminal voltage, the battery was declared
inoperable and TS LCO 3.8.4, Condition A, was entered. Troubleshooting identified
eight loose fasteners on the Breaker 2D201 upper stud to bus bar connections. It was
determined that around March 21, 2008, a high resistance connection developed due to
the loose fasteners, resulting in the failure of the battery to meet the TS minimum
terminal voltage requirements.
Action Request 080301117 was initiated to correct the loose connections. The deficient
electrical connections were corrected and the battery bus was declared operable shortly
after the 2 hour2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> action statement had expired. An ACE was initiated to evaluate the
condition. The ACE determined that the most probable cause for the loose connections
occurred during installation of a new thermal trip device on the breaker in March 2004
- 15 - Enclosure 2
using MO 03100406000, Change the Short Time Delay Settings per Calc E4C-109 for
Breaker 2D201. Although the high resistance connection developed around
March 21, 2008, which resulted in the degraded voltage condition, the team concluded
that the safety-related battery was not maintained in a configuration capable of
performing its function during all design basis events during the four year period in which
the fasteners did not meet the design criteria for electrical connection integrity.
The team reviewed MO 03100406000 to determine the scope of the maintenance action,
and whether the MO had sufficient detail, instructions, and acceptance criteria to ensure
that activities affecting quality were satisfactorily accomplished. The team identified that
the Work Plan Detail section of the MO provided limited instructions on accomplishing
the task, relying on skill-of-the-craft over detail and defense in depth.Section I
required craft to obtain a replacement breaker and test in accordance with applicable
sections of Procedure SO123-I-4.7 (Molded Case Circuit Breakers). This procedure had
no quantitative steps to torque compression-type electrical connections. Additionally,
Section II had only two steps: a) Obtain work authorization; and b) Remove Breaker
2D201 and install the successfully tested replacement breaker. The MO did not have
steps to torque breaker connections during or after installation.
The failure to develop and implement an adequate procedure for installation of the
safety-related 125 Vdc station battery breaker 2D201 in March of 2004 was identified as
a violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and
Drawings. Details associated with this violation are discussed in Section 3.5 of this
report.
2.1.6 Reportability Review
The team reviewed the reportability assignment for AR 080301117. Procedure SO123-
XV-52, Functionality Assessments and Operability Determinations, Revision 7,
provided the requirements for performing reportability assessments. Procedure SO123-
XV-52, Attachment 14, described the process overview. Attachment 14 stated that the
responsible engineer and compliance engineer shall assess reportability. It also stated
that engineering input may be bypassed by regulatory compliance personnel when the
issue is obviously reportable. The team observed that the reportability assessment for
AR 080301117 bypassed engineering input, even though the issue was not obviously
reportable. Regulatory compliance personnel independently concluded that Battery
2B008 was failed when found based on their determination that there was no
compelling evidence of an earlier failure. In July 2008, the NRC resident inspector
performed an initial review of AR 080301117 and questioned the conclusions of
regulatory compliance personnel since information in the ACE and field support
assignments provided information that contradicted the conclusions of the reportability
assessment.
The team observed that the reportability assessment only focused on the aspects of the
initiation of a plant shutdown and failed to consider the degraded connections potential
impact on past operability. After a subsequent review, prompted by the NRC, the
licensee determined that Battery 2B008 was inoperable for greater than the TS allowed
outage time. Licensee Event Report 05000361/2008-006-00 was submitted to the NRC
on September 17, 2008, to report the event.
- 16 - Enclosure 2
The team performed a two month sampling of reportability assessments and identified
that engineering input was bypassed by regulatory compliance personnel for 95 percent
of the assessments that were not obviously reportable. The team also observed that
reportability assignment backlogs were inadequately managed, resulting in reportability
assessments that were less than adequate. Based on the programmatic issues
identified by the team, the licensee initiated an action to perform an extent of condition
review to assess the adequacy of reportability reviews performed for the identified
electrical connection issues associated with safety-related equipment.
The failure to report that a 125 Vdc battery was inoperable for approximately four years,
a condition prohibited by technical specifications, was identified as a violation of 10 CFR
Part 50.73, Licensee Event Report System. Details associated with this violation are
discussed in Section 3.6 of this report.
2.2 Failure to Maintain Design Control for Electrical Connections
Following NRC inspectors initial review of AR 080301117 that discussed the loose
electrical connections affecting the 125 Vdc battery breaker the inspectors questioned
whether other degraded electrical connection issues had been identified by the licensee.
Based on these questions additional examples were identified consisting of: (1) on June
25, 2005, emergency supply Fan 3A276 failed due to a loose wire, which resulted in the
inoperability of the emergency diesel generator (EDG) 3G003; (2) on September 17,
2007, loose electrical bolt connections were identified affecting 125 Vdc Bus 2D2; (3) in
2007, a loose electrical connection was identified affecting emergency chiller supply
Breaker E336; and (4) on July 9, 2008, a loose electrical connection was found affecting
EDG 3G002 cooling fan supply breaker.
Based on these examples having the generic potential to adversely affect the safety
function of multiple safety systems used for accident mitigation the NRC concluded that
a special inspection was warranted. The special inspection team performed a review of
plant corrective action documents, procedures, and work orders, associated with
deficient electrical connections to determine whether the existing processes for control of
electrical connection integrity were adequate.
From January 2005, to July 2008, the team noted that over 30 loose electrical
connection events occurred, with thirteen events occurring in equipment important to
safety. Loose electrical connections that were identified and evaluated included the
following:
Item Equipment Description Condition
1 3A276 EDG 3G003 Building Supply Fan Failed to start; Discovered
(3BH11) June 2005
2 3A277 EDG 3G002 Building Supply Fan 2 loose connections;
(3BH12) Discovered June 2005
3 E549 EDG 3G002 Radiator Fan Discovered June 2005
(3BH07)
4 2BY37 Fuel Handling Building Pump Failed to run; Discovered
Room Emergency Air March 2007
Conditioning Unit E441 Feeder
Breaker
- 17 - Enclosure 2
5 2BJ06 Safety Injection Tank 2T008 to Documented January
Reactor Coolant Loop 1A Valve 2006
2HV9340
- 18 - Enclosure 2
6 3BE06 Auxiliary Feedwater to Steam 3 loose connections;
Generator Control Valve Discovered August 2005
3HV4713
7 2BY30 Component Cooling Water Loose grounding wire in
Building Pump Room Emergency MCC bucket; Discovered
AC Unit E453 July 2005
8 2BE11 Safety Injection Tank T009 to 3 loose connections;
Reactor Coolant Loop 2A Valve Discovered January 2006
2HV9360
9 BS09 Control Building Control Room Loose connection in
Emergency Air Supply Fan A206 indicator circuit;
Discovered February
2006
10 2/3ME336 Emergency Chiller Supply Control panel power
Breaker E336 failure; Discovered June
2007
11 2B008 125 Vdc Battery 2D2 Loose connection on bus
bar; Discovered
September 2007
12 3RY7870 Condenser Air Ejector Wide Failed Surveillance;
Range Radiation Monitor Discovered June 2008
13 3BD21 Diesel Radiator Fan 3E550 Degraded connection;
Feeder Breaker Discovered July 2008
The team reviewed several procedures listed in the RCE associated with loose electrical
fasteners. Examples of identified weaknesses are discussed below and associated with
the following procedures:
- SO123-I-4.7, Molded Case Circuit Breakers, had no steps with quantitative values
for torques associated with electrical connections.
- SO123-I-4.59.6, 600V Power Cable Termination & Repair Guide, Attachment 4,
Maximum Recommended Torque Value for Electrical Terminations, listed values
for various bolt sizes and materials. The torque value units were listed as lb/in and
lb/ft. Torque values are generally listed in units of distance - force and not
force/distance (i.e. foot-pounds, inch-pounds, etc.).
Attachment 4 listed SCE Engineering Standards Electrical Construction Station,
Fittings - Bolted- Torque Data 31-85-10. This document was the reference
document from which torque values were taken. The licensee informed the team
that this document was no longer available and could not be located. No further
references for these torque values were provided.
- SO123-I-9.11, 480V Load Center and Transformer Inspection and Cleaning,
Attachment 4, Maximum Recommended Torque Values, Mechanical Bolting table,
showed a fastener size of 5/16 X 28, which differs from the threads-per-inch values
listed in reference Procedure SO123-I-4.59.6, which listed a size of 5/16 X 24.
- 19 - Enclosure 2
- SO123-I-9.13, 480VAC Linestarter Inspection, Coil and Power Contact
Replacement, Step 6.5.7, required line and load side connectors for molded case
circuit breakers to be tightened firmly. The step does not provide quantitative
values for torque of compression-type connectors on molded case circuit breakers.
In general, the team observed the following inadequacies for establishing adequate
electrical connections: (1) quantitative acceptance values in steps for torquing electrical
connectors in procedures were inadequate to ensure that these important activities have
been properly completed; (2) maintenance orders involving reestablishing connection
integrity were limited in scope and thoroughness; and (3) maintenance orders frequently
did not have quantitative steps or values for required torques.
The team reviewed documentation associated with training in the establishment and
maintenance of electrical connections. Documents describing the training program for
torquing mechanical bolted connections and instrumentation and control connections
were provided. While training programs existed for mechanical bolted connections,
formal training related to electrical connections was limited to instrumentation and
control connections. No training documents related to general electrical connection
integrity was provided. The team determined that formal training on torquing electrical
connections was not provided, and the reliance on skill of the craft, does not appear
adequate to ensure uniform application of proper techniques for making electrical
connections.
The team determined that these electrical deficiencies resulted in configurations where
structures, systems, or components, may not have been able to perform their design
function during a seismic event. The integrity of electrical connections is a key element
in the reasonable assurance of operability. The failure to ensure that appropriate
measures were maintained to assure that systems specified in the design basis were
maintained in a configuration which provided a reasonable assurance of operability
during design basis events is being considered an unresolved item pending further NRC
review: URI 05000361;05000362/2008013-07, Degraded Electrical Connections.
2.2.1 Actions to Identify and Correct Deficient Electrical Connections
The team reviewed the adequacy of licensees ability to identify, evaluate, and establish
corrective actions related to identified loose electrical connections. The team noted that
in June of 2005, EDG 3G003 Building Supply Fan 3BH11 failed to start during a
surveillance test. The failure was attributed to a loose electrical connection at the
thermal overload for the fan. Further investigation by the licensee revealed that similar
loose connections existed at the EDG 3G003 building supply fan and radiator fan. The
licensee performed RCE 050601315 to further understand the failure of these safety-
related components. The corrective actions identified by the RCE included the
development of a fastener trending program to more accurately capture data on the
looseness of electrical connection fasteners found during maintenance and inspections.
Trending of loose fasteners was implemented by the revision of several procedures to
incorporate acceptance criterion for fastener tightness, and a requirement to submit a
corrective action document when this criterion was not met. The intent of the corrective
action was to describe the loose fastener and its relative tightness.
- 20 - Enclosure 2
The ability to identify and correct degraded electrical connections must be a priority in
maintenance programs dealing with electrical equipment. The effectiveness of
maintenance programs depend, in part, upon establishing adequate criteria for
identification, trending, and repair of degraded conditions. The team noted that the
licensees acceptance criterion for trending and repair of loose electrical connections
was based on a condition identified as less than 1-turn loose criterion.
The team requested information that described the basis for the acceptance criterion
used to determine input to the fastener trending program. An email message from a
member of the licensee staff was provided to the team to document the origin of the 1-
turn loose criterion. The email message stated that bench testing was performed to
evaluate the impact to electrical connections with fasteners being less than fully
tightened. The devices used in the testing were identified as 49-auxiliary device
(thermal overload) and 42-auxiliary device (contact) assemblies. The testing consisted
of wiring the auxiliary contact assemblies with ring-tongue lugs commonly used in the
plant and fastened with screw type connectors. The connector was gradually loosened,
1/4-turn at a time and circuit continuity measured. The position of the fastener was noted
when continuity was lost. This test validated the connection geometry integrity for only
the 49 and 42-auxiliary devices, with ring tongue style terminal lugs. No other testing
was conducted to validate the 1-turn loose criterion for different types of electrical
connections.
The team noted several procedures listed in the RCE associated with loose electrical
fasteners that referenced use of the 1-turn loose criterion:
- SO123-I-4.7, Molded Case Circuit Breakers, had no steps with quantitative values
for torques associated with electrical connections, and several steps listed the 1-
turn loose criterion as the acceptance basis.
- SO23-I-2.27, Line Starter Thermal Overload Bypass Inspection, Step 6.2.1.2.10,
required craft to generate an AR to correct suspect [loose] connections and to verify
other connections were NOT loose. One bullet item stated that the AR include the
1-turn loose criterion as the required acceptance criterion.
- SO123-I-4.59.6, 600V Power Cable Termination & Repair Guide, Step 6.6.2,
stated the following, Where it is NOT physically possible to use ring tongue
connections use the same connection method supplied by the vendor. The 1-turn
loose criterion has only been validated using ring tongue connections.
The team concluded that the application of the 1-turn loose criterion to broad classes of
electrical connections, without analysis supporting the applicability, is a programmatic
weakness. Not all electrical fastener geometries will remain operable when the fastener
is not securely tightened. Further, the team observed that no guidance documents were
created to establish trending program guidelines, and no specific process existed for
disposition of fastener issues that met the 1-turn loose criterion. In fact, RCE
050601315, Assignment 98, to evaluate results of the trending program was
inappropriately closed approximately one year after the trending program was
implemented. Application of the less than 1-turn loose criterion in procedures for
inspecting or performing electrical connections for different types of connections was
- 21 - Enclosure 2
non-conservative in application, and inadequate to prevent loose electrical connections
in different fastener geometries.
The team determined the following event also illustrates the ineffectiveness of corrective
actions taken for the significant conditions evaluated in RCE 050601315. On July 9,
2008, safety-related Breaker 3BD21, Diesel Radiator Fan 3E550 Feeder Breaker, was
declared inoperable by an immediate operability assessment performed as part of
Nuclear Notification (NN) 200047962. The notification was generated following the
discovery of a stripped compression connector for the breaker B-phase conductor, with
visible signs of melting, and insulation degradation due to overheating.
Breaker 3BD21 was previously inspected, under AR 050601324 and MO 05062182000,
as part of the extent of condition review for RCE 050601315. MO 05062182000 was
written to check for loose connections in motor control center Panel 3BD. The MO
required the licensee to test the wires and connectors for loose connections by
performing a wiggle test, and tighten any loose connections found. The inspection of
Breaker 3BD21 was listed as completed on June 26, 2005, with no degraded conditions
identified. Additionally, on August 7, 2007, maintenance was performed on the line
starter for radiator Fan 3E550 per MO 05080446000 using Procedure SO123-I-9.13,
480 VAC Linestarter Inspection, Coil and Power Contact Replacement. Step 6.3.2 of
the procedure required inspection of internal wiring, including both line-side and load-
side breaker connections. The procedure step was marked as being satisfactory in the
MO, with no degraded conditions identified.
On April 14, 2008, a thermographic image was taken of Breaker 3BD21 while under
load. The team requested a copy of the thermal image, but was told no image was
available. Procedure SO23-V-2.14, Thermal Inspection of Plant Components, Section
6.3, Note 1, stated, in part, that thermal images should be taken of each inspection, as
this allows for trending and review of each thermographic inspection point. Procedure
SO23-V-2.14, Attachment 5, Unit 3 motor control center and Electrical Equipment
Inspection, Section 1.C(3), stated that, if an anomaly is found during an inspection,
obtain sufficient data to document a complete description of the thermal state of the
component. Section 1.D required generation of an AR for any identified equipment
problems, such as fasteners that need repair. Based on discussions with the licensee,
thermographic images are only stored when anomalies meeting licensee-established
severity criteria are exceeded and confirmed by the thermographer. Procedure SO23-V-
2.14, Section 7.0, stated, in part, that the thermal inspection program is not required for
licensing or regulatory compliance, therefore results of thermal inspections are not
required as part of permanent plant records.
In conclusion, the team noted that Breaker 3BD21 had been inspected as part of the
extent of condition review for RCE 050601315, and had been subsequently subjected to
thermography and a preventive maintenance inspection using the post RCE 050601315
maintenance programs and procedures. Evidence of a long standing degraded
connection was not identified for correction during three inspection opportunities. The
deficient electrical connection was only discovered by the licensee on July 9, 2008, while
performing work on adjacent equipment.
The team determined the licensee failed to establish measures to assure that deficient
electrical connections were promptly identified and corrected. This performance
- 22 - Enclosure 2
deficiency was also identified as a violation of 10 CFR Part 50, Appendix B, Criterion
XVI, Corrective Actions. Details associated with this violation are discussed in Section
3.8 of this report.
- 23 - Enclosure 2
2.2.2 Directed Assessment Report Evaluation
In July 2008, the licensee prepared a DAR titled, Loose Electrical Fastener
Assessment. The DAR was performed in response to NN 200066209 and Corrective
Action Order 800126624. The purpose of the DAR, as stated in the executive summary,
was to assess the extent and significance of loose electrical connections at the facility.
To accomplish this, the DAR defined seven objectives including data searches, an
assessment of corrective actions, and assessment of practices and experience relative
to industry peers. The time period examined was post-RCE 050601315 (late 2005 to the
present).
The team reviewed the DAR to determine whether it demonstrated that the licensee
understood the nature and extent of the issues associated with deficient electrical
connections. Since the DAR was not a formal corrective action document, the team also
reviewed the DAR to determine whether it identified any items that needed to be
documented in the corrective action program.
The team concluded that the seven objectives, as stated in the DAR, were not
sufficiently focused and complete to enable a thorough determination of the extent and
significance of loose electrical connections at the facility. In particular, the DAR was not
well focused on identifying whether corrective actions were actually effective in
identifying and correcting deficient electrical connections.
As part of Objective 1, the DAR provided a tabulation and graph of loose connections
found since the implementation of the trending program. The data showed an increasing
trend in the number of loose connections discovered in both safety-related and non-
safety-related equipment. The DAR remarked favorably on the effectiveness of station
practices to identify loose connections but did not address the apparent failure of the
trending program to reduce the number of loose connections being discovered. The
team noted that the increase in discovery would be expected immediately following the
implementation of new procedures in 2005, but the increasing trend has persisted to the
present. This trend was not noted or evaluated in the DAR. In addition, the team noted
that the threshold for documenting loose fasteners in the CAP was an as-found
acceptance criterion of one or more turns loose. The team concluded that this criteria
potentially excluded a large number of deficient connections since less than 1-turn loose
is typically enough to completely remove pressure from a wire or lug. Consequently, the
data documented in the DAR may have been considerably more optimistic than actual
field conditions.
A survey of other nuclear plants was also conducted as part of the data search under
Objective 1. The survey included two questions, the first regarding the incidence of
loose connections and the second regarding practices for discovery and correction. The
DAR concluded that the data showed that the practices at SONGS were comparable to
the industry peers. However, the team noted that, based on survey results, the
incidence of loose connections was much greater at SONGS than at most other plants.
Nonetheless, the DAR ignored this result and only discussed conclusions relative to
practices.
Objective 2 determined how many preventive maintenance activities had been
performed since the implementation of corrective actions for RCE 050601315 in order to
- 24 - Enclosure 2
assess the effectiveness of the actions. However, the DAR did not evaluate the
effectiveness of the preventive maintenance activities by identifying how many items
with loose connections discovered since 2005 had previously been inspected following
the implementation of the actions associated with RCE 050601315. As previously
discussed, the team identified examples where preventive maintenance inspection
activities were not effective in identifying electrical connection deficiencies.
Objective 3 was intended to perform an effectiveness review for corrective actions from
RCE 050601315. A key measure to determine effectiveness of corrective actions is
whether or not it prevents recurrence of the problem. This measure was not assessed
under Objective 3. Instead, the assessment of this topic was focused on process issues
rather than the fundamental problem of deficient electrical connections. The
assessment concluded that Assignment 98 from RCE 050601315 to perform a data
review had been inappropriately closed. It also identified that assignments had been
closed with no actions, and that there was no actual trending of loose connections being
performed. All of these items had been previously identified by the NRC. By contrast,
there was no discussion of the apparent continued occurrence of loose fastener
problems.
Although the DAR was of questionable effectiveness in accomplishing its stated
objectives, it did document several problems and recommendations for improvement.
However, the licensee did not enter these DAR findings and recommendations into the
CAP until prompted by the team during the special inspection. The licensee then
initiated NN 200089167 for the slow corrective action response, and NN 200066209 to
document the actual DAR issues.
2.2.3 Operating Experience Reviews
Personnel from the Operating Experience Branch of Nuclear Reactor Regulation
supported the team by performing searches of OE databases and other sources. The
intent was to identify OE reports of similar problems and other relevant information. The
team also performed searches of internal events at SONGS and reviewed the searches
performed by licensee personnel in support of their cause evaluations.
The licensee documented their review of OE for the March 25, 2008, events in ACE
080301117, and later in an RCE 800121216. Due to the narrow scope for the search
criteria used in both the ACE and RCE, the licensee missed relevant OE. Internal OE
existed, that documented significant failures of safety-related components due to loose
connections. For example, in 2003, a loose connection caused the failure of a high
pressure safety injection header isolation valve during a simulated safety injection
actuation signal. A noncited violation (NCV) was identified, NCV 05000361/2003002-06,
for the licensees failure to establish adequate maintenance procedures to assess the
condition of electrical terminations. This NCV is similar to the teams conclusion that
procedures were inadequate to properly install the Breaker 2D201 and terminate
electrical connections to Bus 2D2 (Section 3.5).
The team observed that the OE review for RCE 800121216 was not completed in
accordance with the requirements of Procedure SO123-XV-50.39, Cause Evaluation
Standards, Methods, and Instructions, Revision 8. The OE review for RCE 800121216
only referenced an OE review that was completed as part of the July 2008 DAR,
- 25 - Enclosure 2
performed to assess loose electrical fasteners. The OE review in the DAR only looked
at the two years prior to the March 25, 2008, event, which was contrary to guidance in
Procedure SO123-XV-50.39. Root cause evaluations require that the OE review covers,
at a minimum, the four year period leading up to the event.
3.0 SPECIAL INSPECTION FINDINGS
3.1 Untimely Entry Into Technical Specification Action Statement
The team identified a Green NCV of TS 5.5.1.1 for the failure of an electrical
maintenance supervisor to follow procedures after notification that Battery 2B008
terminal voltage was less than the TS required value of 129 Vdc. Specifically, the
supervisor failed to notify the control room shift supervisor after being informed of a
failed battery surveillance activity. The failure to follow procedures resulted in over a 2
hour delay in entering the required 2 hour2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> TS action statement. Details associated with
this finding are described in Section 2.1.2.
The failure to follow procedural requirements for notification of the operations shift
supervisor after being informed of a failed battery surveillance was a performance
deficiency. The finding is greater than minor because it is associated with the equipment
performance attribute of the mitigating systems cornerstone and affects the associated
cornerstone objective to ensure the availability, reliability, and capability of systems that
respond to initiating events to prevent undesirable consequences. In accordance with
NRC Inspection Manual Chapter 0609, Attachment 4, Phase 1 - Initial Screening and
Characterization of Findings, a Phase 2 estimation was required because the finding
resulted in the loss of safety function for the Unit 2 safety-related Battery 2B008 for
greater than the TS allowed outage time.
The team performed a Phase 2 estimation in accordance with NRC Inspection Manual
Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection
Findings for At-Power Situations." The team assumed that the performance deficiency
affected the risk of operating the plant for 2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br /> because the failure to follow plant
procedures resulted in delaying corrective action for this period of time. As a result, in
accordance with Appendix A, Attachment 1, Step 2.1.2 Determine the Appropriate
Exposure Time, the team selected an exposure period (EXP) of less than 3 days.
Using the Risk-Informed Inspection Notebook for SONGS Units 2 and 3, Revision 2.1a,
the team selected Battery of One Panel (bus) Fails, as the appropriate target for the
subject finding in the presolved table. The team utilized the presolved table to determine
that the finding was Green and that core damage frequency was the dominant
contributor. Therefore, no large-early release frequency analysis was required.
Because the result from the presolved table indicated that the result was greater than or
equal to 1 x 10-7, the team requested the senior reactor analyst to evaluate the potential
contribution to risk from external events. As documented in Attachment 3 to this
inspection report, the analyst determined that seismic events were the only external
initiators that significantly contributed to risk for this finding. The analyst calculated the
change in seismic-related core damage frequency (CDFSeismic) resulting from the
- 26 - Enclosure 2
improperly terminated Battery 2B008 to be 1.45 x 10-6/year. Therefore, the analyst
calculated the change over a 2.42-hour period (CDF2.42) as follows:
CDF2.42 = CDFSeismic ÷ 8769 hour0.101 days <br />2.436 hours <br />0.0145 weeks <br />0.00334 months <br />s/year * EXP
= 1.45 x 10-6/year ÷ 8769 hour0.101 days <br />2.436 hours <br />0.0145 weeks <br />0.00334 months <br />s/year * 2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br />
= 4.0 x 10-10
Based on the results of the Phase 2 estimation and the analysis of external events, the
finding is determined to have very low safety significance.
The team determined that this finding has a crosscutting aspect in the area of human
performance associated with decision making because maintenance personnel did not
make safety significant decisions using a systematic process when faced with uncertain
and unexpected plant conditions to ensure safety was maintained. This included the
failure to formally define the authority and roles of the electrical maintenance supervisors
for decisions affecting nuclear safety H.1(a).
Technical Specification 5.5.1.1 requires, in part, that written procedures be established,
implemented, and maintained covering the activities specified in Appendix A, Typical
Procedures for Pressurized Water Reactors and Boiling Water Reactors, of Regulatory
Guide 1.33, Quality Assurance Program Requirements (Operations), Dated February
1978. Appendix A, Section 8.b, requires procedures for the performance of surveillance
tests, inspections, and calibrations. Procedure SO123-I-2.2, 125 Vdc Pilot Cell Battery
Inspection, Revision 7, implemented the requirements of TS Surveillance Requirement 3.8.4.1. Contrary to the above, on March 25, 2008, following notification of a failed
surveillance identified by electricians, electrical maintenance supervisors failed to make
a timely notification as required by Procedure SO123-I-2.2. Specifically, electrical
maintenance supervisors failed to follow Procedure SO123-I-2.2, Step 6.2.1.2, which
required that, This supervisor SHALL report a failed surveillance according to
Procedure SO123-I-1.3. Procedure SO123-I-1.3, Work Activity Guidelines, Revision
14, required that, A SUPERVISOR SHALL immediately provide written notification to
the shift supervisor for any surveillance found failed. As a result of the untimely
notification, operations personnel only became fully aware of the degraded battery
condition 2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br /> after the degraded condition was discovered, and entered the
requirements of TS 3.8.4, Condition A, to perform actions within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to restore
Battery 2B008 to operable status. Because this finding is of very low safety significance
and has been entered into the licensee's CAP as NN 200196248, this violation is being
treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy:
NCV 05000361/2008013-01, "Failure to Follow Procedure Delays Entry Into Technical
Specification Condition."
3.2 Unauthorized Troubleshooting on Safety-Related Equipment
The team identified a Green NCV of TS 5.5.1.1, for the failure of electrical maintenance
personnel to follow Procedure SO123-XX-1, Action Request/Maintenance Order
Initiation and Processing, Revision 20. Specifically, electrical maintenance personnel
performed troubleshooting on safety-related equipment without an MO and control room
authorization. Details associated with this finding are described in Section 2.1.2.
- 27 - Enclosure 2
The failure of electrical maintenance personnel to follow work control procedures during
the events of March 25, 2008, was a performance deficiency. The finding is greater than
minor because it would become a more significant safety concern if left uncorrected in
that more significant consequences would occur if work control procedures are not
followed when performing maintenance on safety-related structures, systems, and
components. The finding affected the mitigating systems cornerstone. In accordance
with Manual Chapter 0609, Attachment 4, a Phase 2 estimation was required because
the finding resulted in the loss of safety function for the Unit 2 safety-related Battery
2B008 for greater than the TS allowed outage time.
The team performed a Phase 2 estimation in accordance with Manual Chapter 0609,
Appendix A. The team assumed that the performance deficiency affected the risk of
operating the plant for 2.42 hours4.861111e-4 days <br />0.0117 hours <br />6.944444e-5 weeks <br />1.5981e-5 months <br /> because maintenance personnel continued to work
outside the controls of plant procedures throughout this period of time. This
performance deficiency resulted in an equivalent risk impact to that evaluated for
NCV 05000361/2008013-01 documented in Section 3.1 of this inspection report.
Therefore, the finding is determined to have very low safety significance.
The team determined that the finding has a crosscutting aspect in the area of human
performance associated with decision making because electrical maintenance personnel
did not make safety significant decisions using a systematic process, especially when
faced with uncertain or unexpected plant conditions [H.1.(a)].
Technical Specification 5.5.1.1 requires, in part, that written procedures be established,
implemented, and maintained covering the activities specified in Appendix A, Typical
Procedures for Pressurized Water Reactors and Boiling Water Reactors, of Regulatory
Guide 1.33, Quality Assurance Program Requirements (Operations), Dated February
1978. Appendix A, Section 9.c, requires procedures for the repair or replacement of
equipment to be prepared prior to beginning work. Procedure SO123-XX-1, Action
Request/Maintenance Order Initiation and Processing, Revision 20, Attachment 2,
contains a listing of maintenance activities that may be completed without an MO.
Troubleshooting safety-related Class 1E electrical systems was not included within the
scope of activities outlined in this procedure. Contrary to the above, on March 25, 2008,
electrical maintenance personnel failed to obtain an MO and control room authorization
to perform troubleshooting to identify the cause of the degraded voltage on Battery
2B008. Because this finding is of very low safety significance and has been entered into
the licensees CAP as NN 200196248, this violation is being treated as an NCV,
consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000361/2008013-02, Failure to Follow the Work Control Process to Perform
Troubleshooting.
3.3 Failure to Follow the Work Control Process
The team identified a Green NCV of TS 5.5.1.1, for the failure of electrical maintenance
personnel to follow Procedure SO123-XX-5, Work Authorizations, Revision 17.
Specifically, work to correct the degraded battery condition was initiated prior to having
an appropriately authorized MO. Details associated with this finding are described in
Section 2.1.3.
- 28 - Enclosure 2
The failure of electrical maintenance and operations personnel to follow work control
procedures during the events of March 25, 2008, was a performance deficiency. The
finding is greater than minor because it would become a more significant safety concern
if left uncorrected in that more significant consequences would occur if work control
procedures are not followed when performing maintenance on safety-related structures,
systems, and components. The finding affected the mitigating systems cornerstone.
Using the Manual Chapter 0609, "Significance Determination Process," Phase 1
Worksheets, the finding is determined to have very low safety significance because it
was not a design or qualification deficiency, did not result in a loss of safety function, and
did not screen as potentially risk significant due to external events.
The team determined that the finding has a crosscutting aspect in the area of human
performance associated with work practices because the licensee did not perform
adequate pre-job briefings and did not properly document the maintenance activities
Technical Specification 5.5.1.1 requires, in part, that written procedures be established,
implemented, and maintained covering the activities specified in Appendix A, Typical
Procedures for Pressurized Water Reactors and Boiling Water Reactors, of Regulatory
Guide 1.33, Quality Assurance Program Requirements (Operations), Dated February
1978. Appendix A, Section 9.c, requires procedures for the repair or replacement of
equipment to be prepared prior to beginning work. Procedure SO123-XX-5, Work
Authorizations, Revision 17, requires for SSAM, that an entry be made into the shift
managers log and that there be an advance copy of the MO prior to initiating work.
Procedure SO123-XX-5, allows verbal authorization for work that does not require a TS
surveillance to return the equipment to operable status. Contrary to the above, on
March 25, 2008, electrical maintenance and operations personnel failed to follow the
appropriate work authorization process to obtain an MO to initiate work to correct the
loose bolt condition on Breaker 2D201. Specifically, the requirements for the use of
SSAM were not followed and verbal authorizations were not allowed for the scope of
work performed on Breaker 2D201. Therefore, an MO should have been present and
authorized prior to beginning work. Because this finding is of very low safety
significance and has been entered into the licensees CAP as NN 200196248, this
violation is being treated as an NCV, consistent with Section VI.A of the NRC
Enforcement Policy: NCV 05000361/2008013-03, Failure to Follow the Work Control
Process.
3.4 Failure to Properly Manage Risk for Tightening Battery Breaker Bolts on Live Equipment
The team identified a Green NCV of 10 CFR Part 50.65(a)(4) involving the failure to
adequately assess the increase in risk and effectively implement risk mitigation actions
for emergent maintenance activities on safety-related 125 Vdc battery breakers. Details
associated with this finding are described in Section 2.1.4.
The failure to adequately assess and manage the increase in risk associated with
emergent work activities was a performance deficiency. This finding is greater than
minor because the licensees risk assessment failed to consider that the maintenance
activities on the 125 Vdc breakers could increase the likelihood of initiating events. In
accordance with Inspection Manual Chapter 0609, Appendix K, Maintenance Risk
Assessment and Risk Management Significance Determination Process, Step 4.1.2, the
- 29 - Enclosure 2
team requested that the senior reactor analyst independently evaluate the risk because
there were notable limitations with the licensees configuration risk assessment tool for
work on vital dc components.
The analyst utilized the Standardized Plant Analysis Risk (SPAR) Model for SONGS
Units 2 and 3, Revision 3.45 to identify the highest risk direct current component at
SONGS. The component identified was the vital 125 Vdc Bus 2D. To bound the risk
related to these work configurations the analyst made the following assumptions:
- All the work completed on energized vital components presented the same risk
profile as if it had all been done in vital 125 Vdc Bus 2D.
- Throughout the time that work was being accomplished, it was 10 times more
likely that an inadvertent reactor trip would occur.
- Any human error, estimated at 2 x 10-2 probability, would result in a failure of the
bus. This assumption would tend to overestimate the risk of the configuration
because such a failure would likely be identified and corrected prior to an initiator
occurring.
- These configurations were in effect for the entire 24-hour period that terminations
and fasteners were being verified and/or tightened.
The analyst quantified the risk related to this plant configuration using the SPAR model.
The resulting incremental CDF was 2.6 x 10-5 /year. Given the 24-hour exposure period,
the incremental CDP was 7.1 x 10-8. Because the licensee had not performed a risk
assessment, the risk deficit is equal to the incremental CDP.
Based on the magnitude of the calculated incremental CDP deficit being less than
1 x 10-6, this finding is determined to have very low safety significance (Green).
The finding has a crosscutting aspect in the area of human performance associated with
resources for the failure to provide appropriate risk management tools by maintaining
complete, accurate, and up-to-date procedures H.2(c).
10 CFR Part 50.65(a)(4), states in part, that before performing maintenance activities
(including but not limited to surveillance, post-maintenance testing, and corrective and
preventive maintenance), the licensee shall assess and manage the increase in risk that
may result from the proposed maintenance activities. Contrary to this, on March 25 and
March 26, 2008, the licensee failed to adequately assess and manage the increase in
risk associated with emergent work activities. Specifically, the STA failed to perform an
adequate MRRMP for the work on safety-related 125 Vdc battery breakers and consider
the risk associated with the increased likelihood of an initiating event. Because this
finding is of very low safety significance and has been entered into the licensees CAP
as NN 200196248, this violation is being treated as an NCV, consistent with Section VI.A
of the NRC Enforcement Policy: NCV 05000361,05000362/2008013-04, Inadequate
Implementation of Risk Assessment and Risk Management Actions for Emergent Work
Activities.
- 30 - Enclosure 2
3.5 Inadequate Procedures and Instructions to Ensure Electrical Connection Integrity for
Safety-Related 125Vdc Battery Bank Supply Breaker 2D201
The team identified a violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions,
Procedures, and Drawings, for the failure of maintenance and work control personnel to
establish appropriate instructions for performing maintenance activities on safety-related
125 Vdc station battery Breaker 2D201. As a result, electrical connection integrity was
not adequate to ensure that the equipment would be able to perform its safety function.
This condition existed for approximately 4 years. Details associated with this finding are
described in Section 2.1.5.
The failure to provide adequate MO's and procedures related to the replacement of
safety-related Breaker 2D201 was a performance deficiency. The finding is greater than
minor because it is associated with the equipment performance attribute of the mitigating
systems cornerstone and affects the associated cornerstone objective to ensure the
availability, reliability, and capability of systems that respond to initiating events to
prevent undesirable consequences. The final significance determination performed by
the senior reactor analyst and approved by the Significance and Enforcement Review
Panel is documented in Attachment 3 to this inspection report. As documented in the
final significance determination, this finding has been determined to be of low to
moderate safety significance (White).
This finding has a crosscutting aspect in the area of human performance associated with
resources because the licensee failed to establish adequate procedures and programs
related to electrical connection integrity H.2(c).
10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings,
states, in part, that activities affecting quality shall be prescribed by documented
instructions, procedures, or drawings of a type appropriate to the circumstances and
shall be accomplished in accordance with these instructions, procedures, or drawings.
Instructions and procedures shall include appropriate quantitative or qualitative
acceptance criteria for determining that important activities have been satisfactorily
accomplished. Contrary to this, in March 2004, maintenance and work control personnel
failed to develop appropriate instructions or procedures, and failed to include quantitative
or qualitative steps to ensure the maintenance activities on safety-related 125 Vdc
station battery Breaker 2D201 had been satisfactorily completed. Specifically, the work
plan described in MO 03100406000 was incomplete and lacked the checks necessary to
ensure that fasteners on the Breaker 2D201 upper stud to bus bar connections were
properly installed. This failure resulted in the Unit 2 safety-related Battery 2B008 being
inoperable between March 2004 and March 25, 2008. This item has been entered into
the licensees CAP as RCE 800121216. This finding is identified as
VIO 05000361/2008013-05, Failure to Establish Appropriate Instructions.
3.6 Failure to Report Conditions Prohibited by Technical Specifications
The team identified a Severity Level IV NCV of 10 CFR Part 50.73 for the failure of the
licensees regulatory compliance organization to submit a required LER within 60 days
after discovering an event requiring a report. Specifically, compliance personnel failed to
- 31 - Enclosure 2
properly assess the past operability of the safety-related 125 Vdc Battery 2B008, which
had been inoperable for greater than the TS allowed outage time. Details associated
with this finding are described in Section 2.1.6.
The failure of licensees regulatory compliance organization to submit a required LER
within 60 days after discovering that a safety-related structure, system, or component
had been inoperable for greater than TS allowed outage time was a performance
deficiency. The finding was determined to be applicable to traditional enforcement
because the NRCs ability to perform its regulatory function was potentially impacted by
the licensees failure to report the events. The finding was determined to be a Severity
Level IV violation in accordance with Section D.4 of Supplement I of the NRC
The finding has a crosscutting aspect in the area of problem identification and resolution
associated with CAP because the licensee failed to thoroughly evaluate problems such
that the resolutions address causes and extent of conditions. This included properly
classifying, prioritizing, and evaluating for operability and reportability conditions adverse
to quality P.1(c).
10 CFR Part 50.73(a) requires, in part, that licensee shall submit an LER for any
operation or condition prohibited by TS within 60 days after the discovery of the event.
Contrary to this requirement, on May 22, 2008, licensees regulatory compliance
organization failed to submit a required LER within 60 days after discovering a condition
prohibited by TS. Specifically, on April 24, 2008, licensees regulatory compliance
organization incorrectly characterized the loose connection on the Breaker 2D201 failed
when found and closed the reportability assignment. Subsequent investigations
demonstrated that the Class 1E 125 Vdc Battery 2B008 was inoperable for greater than
the allowed TS outage time. Because this finding is of very low safety significance and
has been entered in the licensee's CAP as NN 200059017, this violation is being treated
as an NCV, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000361/2008013-06, Failure to Submit LER for Condition Prohibited by Technical
Specifications.
3.7 Failure to Promptly Identify and Correct a Condition Adverse to Quality
The team identified a Green NCV of 10 CFR Part 50, Appendix B, Criterion XVI,
Corrective Actions, for the licensees failure to establish measures to assure that
deficient electrical connections were promptly identified and corrected, and that
corrective actions taken for a significant condition evaluated in RCE 050601315 were
adequate to preclude repetition. Details associated with this finding are described in
Section 2.2.1.
The failure to identify deficient electrical connections and to correct the conditions during
inspection opportunities was a performance deficiency. The finding is greater than minor
because it is associated with the equipment performance attribute of the mitigating
systems cornerstone and affects the associated cornerstone objective to ensure the
availability, reliability, and capability of systems that respond to initiating events to
prevent undesirable consequences. Using the Manual Chapter 0609, Significance
Determination Process, Phase 1 Worksheets, the finding is determined to have very low
safety significance because the condition did not represent an actual loss of safety
- 32 - Enclosure 2
function of a single train for greater than its TS allowed outage time, and did not
represent an actual loss of one or more risk-significant non-TS trains of equipment for
greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />.
This finding has a crosscutting aspect in the area of problem identification and resolution
associated with CAP because the licensee failed to thoroughly evaluate problems such
that the resolutions address causes and extent of conditions. This includes properly
classifying, prioritizing, and evaluating for operability and reportability conditions adverse
to quality. This also includes, for significant problems, conducting effectiveness reviews
of corrective actions to ensure that the problems are resolved P.1(c).
10 CFR Part 50, Appendix B, Criterion XVI, Corrective Actions, states, in part, that
measures shall be established to assure that conditions adverse to quality are promptly
identified and corrected, and in the case of significant conditions adverse to quality, the
measures shall assure that the cause of the condition is determined and corrective
action taken to preclude repetition. Contrary to the above, between June 2005 and
August 2008, the licensee failed to ensure that a significant condition adverse to quality
was promptly identified and corrected. Specifically, the licensee failed to establish
measures to assure that deficient electrical connections were promptly identified and
corrected. These ineffective measures resulted in a long standing degraded electrical
connection that was not identified for correction during three inspection opportunities
associated with safety-related Breaker 3BD21, Diesel Radiator Fan 3E550 Feeder
Breaker, that occurred between June 2005 and April 2008. Because this finding is of
very low safety significance and has been entered in the licensee's CAP as NN
200047962, this violation is being treated as an NCV, consistent with Section VI.A of the
NRC Enforcement Policy: NCV 05000362/2008013-08, Failure to Promptly Identify and
Correct Condition Adverse to Quality.
3.8 Lack of Procedures to Respond to a Loss of a 125 Vdc Bus
The team identified a Green NCV of TS 5.5.1.1 for the failure to establish written
procedures for a loss or degradation of a safety-related electrical power source.
Specifically, no procedural guidance was provided to operations personnel to combat
and recover from a loss or degradation of a Class 1E 125 Vdc bus.
The Class 1E 125 Vdc Buses D1, D2, D3, and D4, are normally powered from Class 1E
480 VAC through battery chargers. The Class 1E buses provide 125 Vdc power for all
safety-related systems, including EDG control systems, switchgear control and tripping
functions for Trains A and B, and are the primary source of power for the vital bus power
supply system, which provides power for the plant protection system and the engineered
safety features actuation system. The 125 Vdc electrical power subsystems each
consists of a battery, a battery charger, and the corresponding control equipment and
interconnecting cabling within the train. The subsystems are required to be operable to
ensure the availability of the required power to shut down the reactor and maintain it in a
safe condition after an anticipated operational occurrence or a postulated design basis
accident.
Loss of a 125 Vdc bus was part of the NRC scenario development efforts in support of
the Component Design Basis Inspection pertaining to operator actions documented in
NRC Inspection Report 05000361; 05000362/2008010. The inspectors observed that
- 33 - Enclosure 2
operators demonstrated a lack of understanding of proper actions following a loss of a
125 Vdc bus. The inspectors observed that the lack of understanding was, in part, due
to the lack of formalized procedures to combat and recover from a loss of the safety-
related power source. This identified inadequacy was evaluated by the team due to its
relevance to the loose battery breaker bolting event discovered on March 25, 2008.
The failure to provide procedures for a loss or degradation of a safety-related electrical
power source was a performance deficiency. The finding is greater than minor because
it is associated with the procedure quality attribute of the mitigating systems cornerstone
and affects the associated cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. Using the Manual Chapter 0609, "Significance Determination Process,"
Phase 1 Worksheets, the finding is determined to have very low safety significance
because it was not a design or qualification deficiency, did not result in a loss of safety
function, and did not screen as potentially risk significant due to external events. This
finding was reviewed for crosscutting aspects and none were identified.
Technical Specifications 5.5.1.1, requires that written procedures be established,
implemented, and maintained for activities specified in Appendix A, Typical Procedures
for Pressurized Water Reactors and Boiling Water Reactors, of Regulatory Guide 1.33,
Quality Assurance Program Requirements (Operations), Dated February 1978.
Regulatory Guide 1.33, Appendix A, Section 6.c, recommends procedures for combating
emergencies and other significant events, including a loss of electrical power and/or
degraded power sources. Contrary to the above, between 1982 and October 2008, the
licensee failed to establish written procedures for a loss or degradation of a safety-
related electrical power source. Specifically, no procedural guidance was provided to
operations personnel to combat and recover from a loss or degradation of a Class 1E
125 Vdc bus. Because this finding is of very low safety significance and has been
entered into the licensees CAP as NNs 200060584 and 200196248, this violation is
being treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy:
NCV 05000361,05000362/2008013-09, "Lack of Written Procedures for a Loss of 125
Vdc Bus."
4.0 MEETINGS, INCLUDING EXIT
On August 21, 2008, the results of this inspection were presented to
Mr. Ross T. Ridenoure, Senior Vice President and Chief Nuclear Officer, and other
members of the licensees management staff who acknowledged the findings. On
November 5, 2008, the results of this inspection were presented to Mr. Ridenoure, and
other members of the licensees management staff who acknowledged the findings.
Additionally, on December 11, 2008, the final results of the inspection were presented to
Mr. Al Hochevar, and other members of the licensees management staff who
acknowledged the findings. The team confirmed that no proprietary material was
examined during the inspection.
ATTACHMENT 1: SUPPLEMENTAL INFORMATION
ATTACHMENT 2: SPECIAL INSPECTION CHARTER
ATTACHMENT 3: SIGNIFICANCE DETERMINATION EVALUATION
- 34 - Enclosure 2
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee
D. Axline, Technical Specialist, Nuclear Regulatory Affairs
J. Chang-Holt, Manager, Engineering Services
S. Genshaw, Manager, Maintenance/System Engineering
S. Gardner, Engineer, Nuclear Regulatory Affairs
A. Hochevar, Manager, Plant Operations
K. Johnson, Manager, Design Engineering
L. Kelly, Engineer, Nuclear Regulatory Affairs
D. Legere, Manager, Work Control
M. McBrearty, Technical Specialist, Nuclear Regulatory Affairs
R. Nielsen, Supervisor, Nuclear Oversight
C. Ryan, Manager, Electrical Maintenance
A. Scherer, Manager, Nuclear Regulatory Affairs
M. Short, Vice President, Engineering and Technical Services
R. St. Onge, Manager, Maintenance and Systems Engineering
T. Vogt, Manager, System Engineering
D. Wilcockson, Manager, Operations and Engineering Training
C. Williams, Manager, Compliance
T. Yackle, Manager, Operations
Nuclear Regulatory Commission
D. Loveless, Senior Reactor Analyst
M. Runyan, Senior Reactor Analyst
A1-1 Attachment 1
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
05000361/2008013-05 VIO Failure to Establish Appropriate Instructions
05000361, URI Degraded Electrical Connections05000362/2008013-07
Opened and Closed
05000361/2008013-01 NCV Failure to Follow Procedure Delays Entry Into Technical
Specification Condition
05000361/2008013-02 NCV Failure to Follow the Work Control Process to Perform
Troubleshooting
05000362/2008013-03 NCV Failure to Follow the Work Control Process
05000361, NCV Inadequate Implementation of Risk Assessment and Risk
05000362/2008013-04 Management Actions for Emergent Work Activities05000361/2008013-06 NCV Failure to Submit LER for Condition Prohibited by
Technical Specification
05000362/2008013-08 NCV Failure to Promptly Identify and Correct a Condition
Adverse to Quality
05000361, NCV Lack of Written Procedures for a Loss of 125 Vdc Bus05000362/2008013-09
A1-2 Attachment 1
LIST OF DOCUMENTS REVIEWED
Procedures
Number Title Revision
SO123-I-1.3 Work Activity Guidelines 14
SO123-I-2.2 125 Vdc Pilot Cell Battery Inspection 9
SO123-XX-1 Work Process Procedure 20
SO123-XX-5 Work Clearance Application/Work Clearance 18
Document/*Work Authorization Record (WCA/WCD/WAR)
SO123-I-4.59.6 Maximum Recommended Torque Value for Electrical 0
Terminations
TS 3.8.4 DC Sources - Operating
SO123-I-1.3 Notification of a Failed on Operable Equipment or Past Due 14
Surveillance
SO123-I-1.7 Work Order Preparation and Processing 20
SO123-I-1.45 Torque Manual 12
SO123-I-2.2 Perform Weekly 125V Battery Bank and Charger Operability 7
Verification Checks
SO123-I-2.3 Perform Quarterly 125V Battery Bank and Charger 7
Operability Verification Checks
SO123-I-2.5 Battery Service Test and Rapid Recharge 10
SO123-XII-2.7 Reporting of Quality Trends 3
SO23-V-2.14 Thermal Inspection of Plant Components 8
SO23-I-2.27 Line Starter Thermal Overload Bypass Inspection 10
SO23-I-2.47 Containment Penetration Molded Case Circuit Breaker 7
Inspection
SO23-I-2.52 Containment Penetration Circuit Breaker Overcurrent Test 15
SO123-I-4.7 Molded Case Circuit Breaker 7
SO123-I-4.59 Wire/Cable Inspection 4
SO123-I-4.59.1 Control and Instrument Cable Termination & Repair Guide 0
SO123-I-4.59.4 4kV/6.9kV Power Cable Termination & Repair 0
Guide
SO123-I-4.59.6 600V Power Cable Termination & Repair Guide 0
SO123-I-9.11 480V Load Center and Transformer Inspection and Cleaning 7
SO123-I-9.12 Motor Control Center Cleaning, Inspection and Megger 9
Testing
SO123-I-9.13 480 VAC Linestarter Inspection, Coil and Power Contact 9
Replacement
SO123-I-9.26 Miscellaneous Low Voltage Bus Panel Inspection, Cleaning 2
and Testing
SO23-XV-2 Troubleshooting Plant Equipment and Systems 2
SO123-XX-1 Action Request/Maintenance Order Initiation and Processing 21
SO123-I-1.7 Maintenance Order Preparation and Processing 19
SO123-I-1.3 Work Activity Guidelines 14
SO123-XX-5 Work Authorization 71
SO123-XX-3 "Fix It Now" Program 11
A1-3 Attachment 1
Number Title Revision
SO123-XX-4 SONGS Work Control 10
SO123-XV- Cause Evaluations Standards, Methods, and Instructions 8
50.39
SO123-XV-52 Functionality Assessments and Operability Determinations 7
SO123-XXX-3.4 Determination to Report Abnormal Occurrences and Events 7
or Adverse-To-Quality Conditions and Follow-Up Licensee
Event Reports (LER)
SO123-XXX-3.6 Accessing Events and Conditions for Reporting to the NRC 0
Notifications
200053004 800121216 200059017 200066209 200059004
200047962
Action Requests
080301117 050601324 080600666 070300033 050801627
080600579 021201414 080400575 080500248 080301404
080400541 070600347 050601315 070300033 050500051
080500060 080500549 080500551 080500642 080500932
080501003 080501287 080501290 080501340 080501345
080600023 080600105 080600206 080600214 080600219
080600275 080600313 080600350 080600351 080600479
080600509
Work Orders/Maintenance Work Orders
08031771000 08031772000 08031773000 08031775000 08031776000
08031777000 08031473000 08031721000 08001177000 05062182000
08031738000 03100406000 08031721000 05080446000 07060546000
08031721000 08031473000 06060103000 05050497000 08031729000
Drawings
Number Title Revision
30136 One Line Diagram 480V MCC 2BD (ESF) 18
30166 One Line Diagram 480V MCC 208/120VAC Heater Panels - ESF 45
31650 Wiring Diagram Control Building Panels 2/3L176, 177, 225 & 10
230, Sheet 1
32136 One Line Diagram 480V MCC 3BD (ESF) 19
32141 One Line Diagram 480V MCC 3BH (ESF) 19
31650 Wiring Diagram Control Building FNLS 2/3L176,177, 229 & 230 1
A1-4 Attachment 1
Miscellaneous Information
Door Logs
Organizational Charts
Licensee Event Report 2005-001
OSM-107
Maintenance Qualification Standard Signoff
Vital Area Door Logs for Individuals Responding to Battery 2B008 Event on March 25, 2008
Unit 2 Control Room Logs for March 25, 2008
Guidance for evaluating Operating Experience dated April 3, 2008
Directed Assessment Report, Loose Electrical Fastener Assessment, 7/2008
LIST OF ACRONYMS USED
ACE apparent cause evaluation
AR action request
CAP corrective action program
DAR directed assessment report
EDG emergency diesel generator
LCO limiting condition for operation
LER licensee event report
MO maintenance order
MRRMP maintenance rule risk management program
OE operating experience
NCV noncited violation
NN nuclear notification
NRC U.S. Nuclear Regulatory Commission
RCE root cause evaluation
SSAM shift manager accelerated maintenance
STA shift technical advisor
TS technical specification
UNSAT unsatisfactory
A1-5 Attachment 1
UNITED STATES
NUC LE AR RE G UL AT O RY C O M M I S S I O N
R E GI ON I V
612 EAST LAMAR BLVD , SU I TE 400
AR LI N GTON , TEXAS 76011-4125
July 21, 2008
MEMORANDUM TO: Greg Warnick, Senior Resident Inspector
San Onofre Nuclear Generating Station
Project Branch D, Division of Reactor Projects
Sam Graves, Reactor Inspector
Engineering Branch 1, Division of Reactor Safety
Mica Baquera, Reactor Inspector
Plant Support Branch 2, Division of Reactor Safety
FROM: Dwight Chamberlain, Director, Division of Reactor Projects /RA/
SUBJECT: SPECIAL INSPECTION CHARTER TO EVALUATE DEFICIENT
ELECTRICAL CONNECTIONS
A Special Inspection Team is being chartered in response to identification of deficient electrical
connections at the San Onofre Nuclear Generating Station with the potential to adversely affect
the safety function of multiple safety systems used for accident mitigation. You are hereby
designated as the Special Inspection Team members. Mr. Warnick is designated as the team
leader. The assigned senior reactor analyst (SRA) to support the team is David Loveless.
A. Basis
On March 25, 2008, maintenance personnel found the Unit 2, Train B, terminal voltage
of the battery at 121V dc; below the TS limit (129.17V dc). The operators declared the
battery inoperable and entered the 2-hour action, TS 3.8.4 condition A. Maintenance
discovered loose battery breaker bus bolts as the cause of the degraded battery voltage.
During followup inspection related to the extent of condition for loose electrical
terminations the following additional examples were identified.
1. On June 25, 2005, during a monthly surveillance of Unit 3 Train B EDG its
associated cooling fan failed due to a loose wire.
2. On September 17, 2007, loose electrical bolt connections were identified affecting
the 2D2 electrical DC bus. Specifically, loose bolts on a battery feeder cable and
loose intercell connectors were identified. This is the same DC bus that was
identified as degraded due to loose electrical connections in March of 2008.
3. In 2007 a loose electrical connection was identified affecting emergency chiller
supply Breaker E336.
A2-1 Attachment 2
4. On July 9, 2008, a loose electrical connection was found affecting Unit 3, Train A,
EDG cooling fan supply breaker.
This Special Inspection Team is chartered to review the circumstances related to
historical and present deficient electrical connection problems and assess the
effectiveness of the licensees actions for resolving these problems. The team will also
assess the effectiveness of the immediate actions taken by the licensee following
identification of these deficiencies.
B. Scope
The team is expected to address the following:
1. Develop an understanding of the electrical connection deficiencies and the impact
these deficiencies have related to the safety functions of affected systems.
2. Assess licensee effectiveness in identifying deficient electrical connection problems,
evaluating the cause of these problems, and implementation of corrective actions to
resolve identified problems.
3. Assess adequacy of licensee processes (procedures, maintenance instructions,
training, etc.) for maintaining proper electrical connections.
4. Assess the licensees RCE, the extent of condition, and the licensees common
mode evaluation for identified electrical connection deficiencies.
5. Evaluate pertinent industry OE and the effectiveness of licensee actions taken in
response to the OE.
6. Determine if there are any potential generic issues related to the electrical
connection deficiencies identified. Promptly communicate any potential generic
issues to Region IV management.
7. Determine if the Technical Specifications were met when the licensee identified the
associated electrical connection deficiencies.
8. Collect data as necessary to support a risk analysis.
C. Guidance
Inspection Procedure 93812, Special Inspection, provides additional guidance to be
used by the Special Inspection Team. Your duties will be as described in Inspection
Procedure 93812. The inspection should emphasize fact-finding in its review of the
circumstances surrounding the event. It is not the responsibility of the team to examine
the regulatory process. Safety concerns identified that are not directly related to the
event should be reported to the Region IV office for appropriate action.
The Team will report to the site, conduct an entrance, and begin inspection no later than
August 4, 2008. While on site, you will provide daily status briefings to Region IV
A2-2 Attachment 2
management, who will coordinate with the Office of Nuclear Reactor Regulation, to
ensure that all other parties are kept informed. A report documenting the results of the
inspection should be issued within 30 days of the completion of the inspection.
This Charter may be modified should the team develop significant new information that
warrants review. Should you have any questions concerning this Charter, contact me at
(817) 860-8173.
A2-3 Attachment 2
ATTACHMENT 3
FINAL SIGNIFICANCE DETERMINATION EVALUATION
San Onofre Nuclear Generating Station
Improper Vital dc Bus Bar Electrical Integrity
Significance Determination Basis
A. Statement of Performance Deficiency
Maintenance and work control personnel failed to establish appropriate instructions for
performing maintenance on safety-related 125 Vdc station battery Breaker 2D201. As a
result, electrical connection integrity was not adequate to ensure that the equipment
would be able to perform its safety function. This condition existed for approximately
4 years.
B. Significance Determination Basis
1. Phase 1 Screening Logic, Results and Assumptions
In accordance with NRC Inspection Manual Chapter 0612, Appendix B, "Issue
Screening," the analyst determined that the failure to properly tighten the bus bar
extension mounting bolts was a licensee performance deficiency. The issue was
more than minor because it was similar to Example 5.b in Manual Chapter 0612,
Appendix E, and it met the not minor if requirement because the system was
returned to service in the degraded configuration.
The analyst evaluated the issue using the Significance Determination Process
(SDP) Phase 1 Screening Worksheet for the Initiating Events, Mitigating
Systems, and Barriers Cornerstones provided in Manual Chapter 0609,
Attachment 4, "Phase 1 - Initial Screening and Characterization of Findings.
Although this finding affected multiple cornerstones, the analyst determined that
the Mitigating Systems Cornerstone best reflected the dominant risk of the
finding. The analyst determined that the finding represented an actual loss of
safety function of Battery 2B008 for longer than the technical specification
allowed outage time. Therefore, a Phase 2 estimation was conducted in
accordance with Manual Chapter 0609, Appendix A, Determining the
Significance of Reactor Inspection Findings for At-Power Situations.
2. Phase 2 Risk Estimation
In accordance with Manual Chapter 0609, Appendix A, Attachment 1, "User
Guidance for Phase 2 and Phase 3 Reactor Inspection Findings for At-Power
Situations," the Senior Reactor Analyst evaluated the subject finding using the
Risk-Informed Inspection Notebook for San Onofre Nuclear Generating Stations,
Units 2 and 3, Revision 2.1a. The following assumptions were made:
a. The identified performance deficiency occurred on March 17, 2004 when
Battery 2B008 was returned to service following the replacement of
Circuit Breaker 2D201 and continued to affect the plant until its discovery
A3-1 Attachment 3
on March 25, 2008.
b. In accordance with Manual Chapter 0609, Appendix A, Attachment 2,
Site Specific Risk-Informed Inspection Notebook Usage Rules, Rule 1.1,
Exposure Time, the analyst evaluated the time frame over which the
finding impacted the risk of plant operations. Because the performance
deficiency continued to affect plant risk for more than one assessment
period, the analyst determined that the appropriate exposure time was
one year. Therefore, the exposure time used to represent the time that
the performance deficiency affected plant risk in the Phase 2 estimation
was greater than 30 days.
c. In accordance with Appendix A, Attachment 1, Step 2.1.3, Find the
Appropriate Target for the Inspection Finding in the Pre-solved Table, the
analyst determined that the appropriate target for evaluating this
performance deficiency was Battery of One Panel (Bus) Fails.
Therefore, the analyst utilized the pre-solved table associated with the
SDP notebook to perform the estimation.
d. The analyst gave no operator action credit as discussed in Manual
Chapter 0609, Appendix A, Attachment 1, Table 4, "Remaining Mitigation
Capability Credit." The requirements to have procedures in place and to
have trained the operators in recovery under similar conditions for such
credit were not met.
The dominant sequences from the notebook were documented in Table 3-1
below:
TABLE A3-1
Failure of Vital Battery 2B008
Phase 2 Sequences
Initiating Event Sequence Mitigating Functions Results
Loss of Offsite Power 1 LOOP-AFW/RC 6
2 LOOP-REC-AFW 6
3 LOOP-EAC-HGEN-REC 6
4 LOOP-EAC-TDAFW-REC 6
6 LOOP-EAC-SEAL-HPR 9
7 LOOP-EAC-SEAL-EIHP 9
9 LOOP-EAC-SEAL-REC 8
Using the pre-solved worksheet, the result from this estimation indicated that the
finding was of moderate safety significance (YELLOW). However, the analyst
determined that this estimate did not include a full coverage of the risk related to
the failure identified, particularly because of the changing condition of the
A3-2 Attachment 3
connection over time and the affect that seismic events would have on the
specific condition. Therefore, a Phase 3 evaluation was conducted to better
assess the risk of the finding related to internal initiators and fully assess the risk
related to external initiators.
3. Phase 3 Risk Analysis
In accordance with Manual Chapter 0609, Appendix A, the analyst performed a
Phase 3 analysis using the Standardized Plant Analysis Risk (SPAR) Model for
San Onofre 2 & 3, Revision 3.45, dated September 2008, to simulate the failure
of Battery 2B008 and associated 125 Vdc Bus 2D2. Additionally, the analyst
conducted an assessment of the risk contributions from external initiators using
insights and/or values provided by the licensees Individual Plant Evaluation for
External Events (IPEEE).
Assumptions:
To evaluate the change in risk caused by this performance deficiency, the
analyst made the following assumptions:
a. The San Onofre SPAR model, Revision 3.45 represents an appropriate tool
for evaluation of the subject finding.
b. The bus bar extension mounting bolts for the Battery 2B008 feeder breaker to
Bus 2D2 were insufficiently tightened from March 17, 2004, when
Battery 2B008 was returned to service following the replacement of station
battery Breaker 2D201, until discovery on March 25, 2008.
c. There was sufficient continuity through the degraded connection to conduct
charging current (usually < 1 amp) at a very low differential voltage across
the connection from March 17, 2004 until commencement of spare charger
operation on March 17, 2008.
d. There was not sufficient continuity to conduct charging current commencing
sometime after March 17, 2008.
e. Once the open circuit developed, it exhibited sufficient resistance to prevent
the re-establishment of continuity for a gradual increase in voltage up to
10 Vdc.
f. Given Assumptions d and e, the battery would have failed to energize the
diesel generator starting circuitry from some time after March 17, 2008
through March 25, 2008. Additionally, the failure mode of the bus
connections, should a large load have been demanded of the battery during
this time, would likely have resulted in failure of Bus 2D2.
g. Given Assumption c, Battery 2B008 would have been capable of starting
Diesel Generator 2DG003 from March 17, 2004 until March 17, 2008.
However, the battery and/or connection to the bus would have failed prior to
A3-3 Attachment 3
completion of its station blackout mission time because of the high resistance
connection.
h. Given Assumption g, only accident sequences that demanded a major load
on the vital battery would have resulted in Battery 2B008 failure while the
connection was in the subject configuration.
i. The exposure time used for evaluating this finding should be determined in
accordance with Manual Chapter 0609, Appendix A, Attachment 2, Site
Specific Risk-Informed Inspection Notebook Usage Rules.
j. The appropriate exposure times (EXP), for use in this evaluation are as
documented below:
Case 1: Given Assumptions b, c and g, Battery 2B008 would have been
incapable of providing its station blackout function from March 17, 2004
though March 25, 2008. Therefore, an exposure period of one year,
representing the most recent assessment period was used for exposure
to this failure.
Case 2: The exact time at which Battery 2B008 became uncoupled from
the battery charger is unknown. However, we know that the battery was
appropriately charged on March 17, 2008 and that there was insufficient
charging current to the battery on March 25, 2008. Therefore, in
accordance with Assumptions f and i, Battery 2B008 would not have
started Diesel Generator 2DG003 upon demand for one half the period or
4 days.
k. Given the specific conditions of the buswork, the actual time required to
diagnose the problem upon identification of degraded battery voltage, and the
potential failure modes considered, operators would not have been able to
recover Battery 2B008 prior to core damage.
Internal Initiating Events:
The senior reactor analyst used the SPAR model for San Onofre Units 2 & 3 to
estimate the change in risk associated with internal initiators that was caused by
the finding. Average test and maintenance of modeled equipment was assumed,
and a cutset truncation of 1.0 x 10-13 was used. Two cases were evaluated
based on the indications observed.
Case 1: Failure of Battery and Bus for a 4-day period
Consistent with guidance in the Risk Assessment of Operational Events
Handbook, including NRC document, "Common-Cause Failure Analysis in Event
Assessment, (June 2007)," and Assumptions a, f, g, j and k, the senior reactor
analyst modeled the condition by adjusting the following basic events in the
SPAR model:
A3-4 Attachment 3
TABLE A3-2
Failure of Vital Battery 2B008
Case 1 SPAR Change Set
Basic Event Original Value Conditional Value
DCP-BAT-LP-B008 4.8 X 10-5 TRUE
DCP- BDC-LP-BUSD2 9.6 X 10-6 TRUE
In accordance with Assumption f, the analyst determined that the predominant
demands on Battery 2B008 are following a loss of offsite power (LOOP). The
analyst evaluated the potential losses of ac power that were not caused by a
LOOP. The potential for equipment losses that would put a demand on Battery
2B008 within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of an initiating event were determined to be at least two
orders of magnitude lower than the risks related to a LOOP. Therefore, the
analyst quantified only the LOOP sequences.
The SPAR baseline core damage frequency for LOOP sequences (CDFBASE) was
2.28 x 10-6/year. The evaluation case for the above change set resulted in a
conditional core damage frequency for the same sequences (CCDFSPAR) of 1.12
x 10-5/year.
The dominant core damage sequences were documented in Table A3-3:
TABLE A3-3
Failure of Vital Battery 2B008
Phase 3 Dominant Sequences
Initiating Event Sequence Preponderant Failures Frequency
Loss of Offsite 16-30 Station blackout with failure of 4.21 x 10-6/year
Power the turbine-driven auxiliary
feedwater system and failure
to recover offsite power or the
other diesel generator.
15 Failure of the auxiliary 4.34 x 10-6/year
feedwater system and failure
to recover offsite power.
16-21 Station blackout with failure to 2.13 x 10-6/year
control reactor subcooling
combined with failure to
recover offsite power or the
other diesel generator.
A3-5 Attachment 3
TABLE A3-3
Failure of Vital Battery 2B008
Phase 3 Dominant Sequences
Initiating Event Sequence Preponderant Failures Frequency
16-28-4 Station blackout with failure of 2.45 x 10-7/year
the turbine-driven auxiliary
feedwater system, followed by
recovery of offsite power, but
failure of high head injection.
The change in incremental conditional core damage frequency (ICCDF) was
calculated as follows:
ICCDF = CCDFSPAR - CDFBASE
= 1.12 x 10-5/year - 2.28 x 10-6/year
= 8.92 x 10-6/year
Given Assumption i, the exposure time, representing the time that the
performance deficiency impacted both the battery and the bus, for this analysis
was 4 days. Therefore, the change in core damage frequency for this case
(CDF1) caused by this finding, without applying any recovery to the subject
condition, and related to internal initiators was calculated as follows:
CDF1 = ICCDF * EXP
= 8.92 x 10-6/year * (4 days ÷ 365 days/year)
= 9.78 x 10-8
Case 2: Failure of Battery Following Start of Diesel Generator
In accordance with Assumptions a, b, g, j and k, the analyst evaluated the affect
of Battery 2B008 failing to perform its intended function while remaining capable
of starting Diesel Generator 2DG003. The analyst noted that this condition only
affected a station blackout and that it was unlikely to fail Bus 2D2. In accordance
with the SPAR, the LOOP initiation frequency for San Onofre is 3.59 x 10-2/year.
The analyst quantified the failure rate of both Unit 2 diesel generators using the
associated fault trees. The resulting probability was 3.00 x 10-3. Therefore, the
station blackout frequency (SBO) was calculated to be 1.08 x 10-4/year.
In accordance with Assumptions b and j, this condition existed for approximately
4 years. However, as documented in Manual Chapter 0609, Appendix A,
Attachment 1, Step 2.1.2, Determine the Appropriate Exposure Time, the
A3-6 Attachment 3
maximum exposure time used in the significance determination process is limited
to 1 year.
The analyst made the following adjustments in the SPAR model to determine the
baseline conditional core damage probability for a station blackout:
TABLE A3-4
Failure of Vital Battery 2B008
Case 2 SPAR Change Set
Basic Event Original Value Conditional Value
IE-LOOP 3.59 X 10-2 1.0
EPS-DGN-FS-2DG2 5.0 X 10-3 TRUE
EPS-DGN-FS-2DG3 5.0 X 10-3 TRUE
The resulting core damage probability for a baseline station blackout was 3.34 x
10-2. The analyst then set Basic Event DCP-BAT-LP-B008 to the house event
TRUE, indicating that the battery would fail to perform its intended function under
these conditions. The resulting conditional core damage probability for the
evaluated case was 3.48 x 10-2, making the change in core damage probability
(CCDPSBO) to be 1.40 x 10-3. The analyst calculated the change in core
damage frequency (CDF2) as follows:
CDF2 = SBO * CCDPSBO * EXP
= 1.08 x 10-4/year * 1.40 x 10-3 * 1 year
= 1.51 x 10-7
External Initiating Events:
Seismic
The analyst determined that, for the subject performance deficiency to affect the
core damage frequency, a seismic event must result in both a LOOP and the
failure of the Battery 2B008 connections.
As such, the analyst evaluated the subject performance deficiency by
determining each of the following parameters for any seismic event producing a
given range of median average spectral acceleration "a" [SE(a)]:
- The probability that a LOOP occurs during the event (PLOOP-SE(a));
- The probability that Bus 2D2 fails during the event (P BUS-SE(a)); and
- The conditional change in core damage probability (CCDPSE(a)).
A3-7 Attachment 3
The CDF for the acceleration range in question (CDFSE(a)) can then be
quantified as follows:
CDFSE(a) = SE(a) * PLOOP-SE(a) * P BUS-SE(a) * CCDPSE(a)
Given that each range a was selected by the analyst specifically to be
independent of all other ranges, the total increase in risk, CDF, can be
quantified by summing the CDFSE(a) for each range evaluated as follows:
6
CDF = CDFSE(a)
a=.03
over the range of SE(a).
Frequency of the Seismic Event
NRC research data indicated that seismic events of 0.05g or less have little to no
impact on internal plant equipment. As such, to ensure that the risk was
bounded, the analyst evaluated the risk of seismic events greater than 0.03g.
The analyst also assumed that seismic events greater than 6.0g lead to core
damage. The analyst, therefore, examined seismic events in the range of 0.03g
to 6.0g.
The analyst divided that range of seismic events into segments (called "bins"
hereafter); specifically, seismic events from 0.03g to 0.1g were binned by
hundredths, seismic events from 0.1g to 1.0g were binned by tenths, and seismic
events from 1.0g to 6.0g were binned by ones.
In order to determine the frequency of a seismic event for a specific range of
ground motion (g values), the analyst used the licensee's IPEEE and obtained
values for the frequency of the postulated seismic event that generates a level of
ground motion that exceeds the lower value in each of the bins. These values
were estimated in average spectral acceleration as used by the licensee as
opposed to peak ground acceleration used in the risk standardization handbook.
The analyst then calculated the difference in these "frequency of exceedance"
values to obtain the frequency of seismic events for each of the binned seismic
event ranges.
For example, according to the San Onofre IPEEE, the frequency of exceedance
for a 0.6g seismic event is estimated at 3 x 10-3/yr and a 0.7g seismic event
at 2 x 10-3/yr. The frequency of seismic events with median acceleration in the
range of 0.6g to 0.7g [SE(0.6-0.7)] equals the difference, or 1 x 10-3/yr.
Probability of a LOOP
The analyst assumed that a seismic event severe enough to break the ceramic
insulators on the transmission lines would cause an unrecoverable LOOP.
A3-8 Attachment 3
The analyst obtained data on switchyard components from the staffs evaluation
of the licensees IPEEE, dated September 29, 1999. Table 5.2 of this document
provided the major seismic fragilities for equipment at San Onofre. Additional
references utilized for generic fragility values were:
NUREG/CR-6544, Methodology for Analyzing Precursors to Earthquake-
Initiated and Fire-Initiated Accident Sequences," April 1998; and
NUREG/CR-4550, Volumes 3 and 4, Part 3, Analysis of Core Damage
Frequency: Surry / Peach Bottom, 1986.
The references describe the mean failure probability for various equipment using
the following equation:
Pfail(a) = [ ln(a/am) / (r2 + u2)1/2]
Where is the standard normal cumulative distribution function and
a = median acceleration level of the seismic event;
am = median of the component fragility;
r = logarithmic standard deviation representing random
uncertainty;
u = logarithmic standard deviation representing systematic or
modeling uncertainty.
In order to calculate the LOOP probability given a seismic event, the analyst
used the seismic fragility values listed for the San Onofre switchyard
components:
am = 0.74g
r = 0.20
u = 0.34
Using the above normal cumulative distribution function equation, the analyst
determined the conditional probability of a LOOP given a seismic event. For
each of the bins, the calculation was performed substituting for the variable "a"
the median average spectral acceleration level for that bin. The following table
shows the results of the calculation for various acceleration levels.
A3-9 Attachment 3
TABLE A3-5
Failure of Vital Battery 2B008
Seismic LOOP Probability
Spectral Acceleration Level/Probability of LOOP
0.03g 5.2 x 10-15 0.3g 2.9 x 10-2 2.0g 1.0
0.07g 3.3 x 10-9 0.7g 5.1 x 10-1
Probability That Bus 2D2 Fails
In order to calculate the probability that the bus bar extension vibrates enough
that it results in failure of Bus 2D2 through excessive variation in the supply of
direct current to bus relaying, the analyst used the used the seismic fragility
values listed for the San Onofre reserve auxiliary transformers. This assumed
that any movement large enough to fail an electrical component would be large
enough to fail the improperly terminated bus bar. The following values were
used:
am = 0.52g
r = 0.30
u = 0.45
Using the above standard normal cumulative distribution function equation, the
analyst determined the conditional probability that Bus 2D2 fails given a seismic
event for each of the bins. The calculation was performed substituting for the
variable "a" the median average spectral acceleration levels for that bin. The
following table shows the results of the calculation for various acceleration levels.
TABLE A3-6
Failure of Vital Battery 2B008
Seismic Bus Failure Probability
Spectral Acceleration Level/Probability of Bus Failure
0.03g 3.0 x 10-7 0.3g 2.3 x 10-1 2.0g 1.0
0.07g 1.7 x 10-4 0.7g 7.5 x 10-1
Conditional Change in Core Damage Probability
The analyst evaluated the spectrum of seismic initiators to determine the
resultant impact on the reliability and availability of mitigating systems affecting
the subject performance deficiency.
A3-10 Attachment 3
The analyst used the San Onofre 2 & 3 SPAR Model, Revision 3.45, to perform
the Phase 3 evaluation. The analyst first created a baseline case by setting the
initiating event probability for a LOOP to 1.0 and all other initiating event
frequencies in the SPAR model to the house event FALSE, indicating that these
events could not occur at the same time as a LOOP. Offsite power was
assumed to be non-recoverable following seismic events that break the ceramic
insulators (low fragility components) on the transmission lines. Therefore, the
analyst set the non-recovery probabilities for offsite power to 1.0. The SPAR
model showed the resultant core damage probability as 2.03 x 10-4, which
represented the baseline case that was used in the above equation.
The SPAR Model showed that loss of Battery 2B008 and Bus 2D2 during an
unrecoverable LOOP leads to a conditional core damage probability of
9.88 x 10-4. Therefore, the change in core damage probability was:
CCDPSE(a) = 9.88 x 10-4 - 2.03 x 10-4 = 7.85 x 10-4
Phase 3 Seismic Results
Considering the factors described above:
< The frequency of the seismic event;
< The probability that a LOOP occurs during the event;
< The probability that Bus 2D2 fails during the event; and
< The conditional change in core damage probability
The total increase in risk, CDF, can be quantified by summing the CDFSE(a) for
each bin as follows:
6
CDF = CDFSE(a)
a=.03
over the range of SE(a). This result was 1.45 x 10-6/year.
High Winds, Floods, and Other External Events
The analyst reviewed the IPEEE and determined that no other credible scenarios
initiated by high winds, floods, fire, and other external events could initiate a
LOOP and directly cause the perturbation of the bus bar extension connection
with the breaker stabs. Therefore, the analyst concluded that external events
other than seismic events were not significant contributors to risk for this finding.
Total Change in Core Damage Frequency
Given that each of the initiators in this analysis were treated to ensure that the
final probabilities were independent of each other, the analyst determined that
the total change in core damage frequency (CDF) could be calculated by taking
the
A3-11 Attachment 3
sum of each independent change. Therefore, the final Phase 3 result was
calculated as follows:
CDF = CDFInternal + CDFExternal
= CDF1 + CDF2 + CDFSEISMIC
= 9.87 x 10-8 + 1.51 x 10-7 + 1.45 x 10-6
= 1.70 x 10-6
This result indicated that the finding was of low to moderate significance to the
risk based on core damage frequency.
Risk Contribution from Large Early Release Frequency (LERF)
Using Manual Chapter 0609 Appendix H, Containment Integrity Significance
Determination Process, the analyst determined that this was a Type A finding
(i.e., LERF contributor) for a large dry containment. For pressurized water
reactor plants with large dry containments (like San Onofre), only findings related
to accident categories of intersystem loss of coolant accidents and steam
generator tube ruptures have the potential to impact LERF. In addition, an
important insight from the individual plant evaluation program and other
probabilistic risk assessment studies is that the conditional probability of early
containment failure is less than 0.1 for core damage scenarios that leave the
reactor coolant system at high pressure (>250 psi) at the time of reactor vessel
breach. The analyst noted that none of the cutsets were from steam generator
tube rupture or intersystem loss of coolant accident sequences. Therefore, the
analyst determined that the change in risk related to the subject performance
deficiency was insignificant with respect to LERF.
C. Final Significance Determination
As previously documented in this analysis, the Phase 3 result for total CDF was
1.70 x10-6 indicating that the finding was of low to moderate safety significance.
Additionally, the analyst determined that the change in risk related to the subject
performance deficiency was insignificant with respect to LERF. Therefore, in
accordance with Manual Chapter 0609, Appendix A, the finding is characterized as
being of low to moderate safety significance (White).
A3-12 Attachment 3