ML120100550
| ML120100550 | |
| Person / Time | |
|---|---|
| Site: | Palisades  |
| Issue date: | 01/05/2012 |
| From: | Entergy Nuclear Operations |
| To: | Office of Nuclear Reactor Regulation |
| Shared Package | |
| ML1200100495 | List: |
| References | |
| EA-11-241, EA-11-243, PNP 2012-06 CR-PLP-2011-4822 | |
| Download: ML120100550 (58) | |
Text
RC1 Revise EN-IS-123 to reduce the limit for electrical superintendent approval for work on Maint 04/28/12 energized equipment from 240 volts to 50 volts. (CA-29)
CC41 Extent Using the SAT process, initiate a TEAR and perform a needs analysis on the need for Training 01126/12 IOP2B I initial and continuing training in the use of EN-WM-1 04 for qualitative and quantitative OP5AD I risk assessment and their relationship to the work planning process described by EN-OP5AE I WM-104 and the use of prejob briefs described in EN-HU-102 and EN-HU-105.
OP5T Identify the correct population and establish schedule for training . If training is not recommended, develop a schedule of recommended actions and return to CARB for approval. (CA-30)
CC41 Extent Based on results of Training needs analysis for EN-WM-1 04 training , determine scope Maint 02/15/12 I OP5A I and schedule for effectiveness review. (CA-31)
OP5AE CC2/CC41 Provide information sharing to Maintenance supervisors and superintendents on the Maint 11/17/11 Extent I use of use of EN-WM-1 04 for qualitative and quantitative risk assessment and its OP2B relationship to EN-WM-105. (CA-32)
IOP5AE CC11 CC21 Revise EPS-E-10 to include all outage and on-line maintenance on these 125 volt DC Maint 02/28/12 CC4/0P4AI panels and breakers. Include critical steps and consideration of IPTE. Clarify Plant OP5AI impact to make clear the potential for train failure and Plant trip and remove Notes OP5D which provide instructions in lieu of actual procedure steps. Instructions should account for the fact that the physical layout and construction of the increases the chance of error during maintenance. (CA-33)
CC21 Revise or cancel other Maintenance EPS (Emergency Power System) procedures that Maint 03/31/12 Extent! have not been revised since the 2007 Entergy transition , which now contain outdated OP4A/OP5A references and which do not conform to Entergy standards of completeness. (CA-34)
IOP5D Page 44 of 93
CC21 Quarantine any Maintenance EPS procedures that have not been revised recently and Maint 11/17/11 Extent! are considered out of date. (CA-35)
OP4AI OP5D CC21 CC41 Develop a plan to prioritize and eliminate outdated references, workarounds, tribal Maint 11/28/11 Extent 1 knowledge and human performance traps in all Maintenance procedures and which OP5D includes plans to improve Maintenance worker ownership of procedure quality.(CA-36)
CC21 CC41 Complete the plan to eliminate outdated references , workarounds, tribal knowledge Maint 12/31/12 Extent 1 and human performance traps in all Maintenance procedures and which includes OP5D plans to improve Maintenance worker ownership of procedure quality. (CA-37)
CC41 Extent Reconsider the existence of an emerging trend in NRC violations with a cross cutting Licensing 01131/12 aspect in procedure compliance (H4b) based on situations that have been identified since the evaluation of CR-PLP-2011-2397. Issue a follow up condition report if required to evaluate a new recognized trend . (CA-38)
RC1/CC31 Verify that completed actions to address weaknesses in management oversight of Maint 02/28/12 LOW work activities assigned from the evaluation of CR-PLP-2011-4522 (CA-11 through CA-13) that include the development, instruction and use of WILL sheets and the review for formal training , have addressed concerns identified for this evaluation .
Develop follow up corrective actions and return to CARB for approval if CR-PLP-2011-4522 results are not satisfactory. (CA-39)
I Page 45 of 93
RC1/0P2B Submit a training evaluation action request (TEAR) to improve PLP staff knowledge of CA&A 12/29/11 the meaning of a Condition Adverse to Quality (CAQ). Specifically, provide direction to PLP staff on the definitions of "Nuclear Safety", "Industrial Safety", "Equipment Reliability" and "Procedure Adherence". If the training needs analysis results in a finding of no training is necessary, then arrange to have this accepted or overridden by the corrective action review board (CARB). This corrective action is to remain open until the completion of the training needs analysis.
(Refer to Recovery Plan WT-PLP-2011-366 CA199, and CA238) (CA 65)
RC1/0P2B Develop and implement WILL sheets on the 8 attributes of a strong safety culture. ISHP 12/18/11 (Close Recovery Plan WT-PLP-2011-366 CA401) (CA-50)
RC1 Coordinate a third party safety culture assessment, (Close Recovery Plan WT-PLP- NSA 2/28/12 2011-366 CA397) (CA-51)
RC1 Create and implement a WILL sheet specific to observing trip risk activities (Close Maint 12/8/11 Recovery Plan WT-PLP-2011-366 CA75) (CA-52)
RC1 Revise on-line scope add form to include statement that appropriate risk and risk PS&O 12/23/11 mitigators have been added to appropriate schedule. (Close Recovery Plan WT-PLP-2011-366 CA408) (CA-53) I RC1 Perform a causal analysis, which should include a review of previous 12 months CA&A 12/23/11 I RCEs, significant ACEs, NRC findings, Cross-cut issues, AFls, etc. Apply special focus on O&Ps. Use of 95-001/002/003 inspection manuals (Close Recovery Plan ,
WT-PLP-2011-366 CA90) (CA-54) :
RC1 CRG and DPICS to ensure that the keyword "Stop When Unsure" is applied all CR's GMPO 1/8/2012 written regarding failure to complete a task based the work being stopped because either: a. the workers are unsure, b. the task is not correct as written, c. the task cannot be completed as written, d. the replacement components are different, d. the scope of the work changes, e. the workers are directed to stop by Supervision/ManagemenUEngineering. CRG and DPICS to ensure appropriate follow-uQ actions are assigned. (CA-55)
Page 46 of 93
RC1 At the T-5 Critical Evolutions Meeting (CEM) and the T-2 meeting , Members validate GMPO 1/8/2012 the rigor and correctness of the Risk Analysis associated with evolutions identified by EN-FAP-WM-002. For work packages that do not have adequate Risk Analysis as determined by the members, a CR will be generated with the keyword "Risk Management" applied and appropriate actions created . (CA-56)
RC1 , CC3 , Create an annual CBT to re-affirm employee understanding of and commitment to Training 3/9/12 CC4 Entergy Accountability , Behavioral and Procedure Use standards. Create an accompanying Job Familiarization Guide for use with new employees. Ensure created materials reference this CR and require CARB approval for removal from site curriculum. (CA-57)
Extent Submit the LER for the identified conditions within the 60-day time period. (CA-02) Licensing 11/22/11 Enhance Request EN-IS-123 clarification of the EN-OP-102 requirement for a Shift manager Maint 11/30/11 ment face-to-face meeting with workers when work on energized equipment without tagging is involved. (CA-40)
Enhance Request EN-WM-104 clarification for disposition of completed risk assessments as Maint 11/30/11 ment records or incorporation into work order packages. (CA-41)
Other Perform Learning Opportunity Review per EN-QV-112 , "Learning Opportunity Review QA 12/15/11 Process" (CA-42)
Other Locate the actual root cause evaluation record for CPAL-97-1493 and initiate Maint 11/30/11 additional corrective actions to address issues not addressed by this corrective action plan. (CA-43)
Other Arrange with CA&A to have referenced actions from CR-PLP-2011-4835 and CR-PLP- Maint 11/30/11 2011-4522 appropriately linked in PCRS to this corrective action plan. (CA-44)
I Page 47 of 93
CC21 CC31 Verify completion of CERT Team Report 18 recommended actions as identified in LO- NSA 03/15/12 CC41 LOW WTPLP-2011-0366 which address procedure use, procedure quality and management oversight (CA-45)
EFR Obtain an LO document number and assign effectiveness review actions to verify that Maint 11/30/11 actions to preclude recurrence have been successful (CA-46)
J OE Prepare and issue internal Entergy fleet OE summary (CA-13) OE 10/26/11 OE Prepare and issue external OE summary (CA-16) Maint 10/26/11 Page 48 of 93
Effectiveness Review Plan LO-PLPLO-2011-00061 CAPRlI RC1 Reinforce and institutionalize Entergy Standards for Procedure Compliance, Accountability, and Unacceptable Behaviors via face to face communications from the COO through Individual Contributor Levels.
CAPR2/RC1 Implement, and ensure compliance with, Entergy Risk Management procedures EN-WM-104 and EN-FAP-WM-002.
Action Resp. Dept Due Date Focused self assessment by non-Method:
Palisades (PLP) personnel of Palisades procedure adherence with respect to promoting a sensitive risk culture Risk Management Attributes:
Work Practices Management Oversight 6/20/2012 Management Intrusiveness GMPO for first Non PLP personnel self assessment Success: assessment confirms that Palisades has properly institutionalized the applicable work oversight and methodology combined with institutionalized knowledge of work practice expectations Approximately every 6 months for 2 Timeliness:
years.
Page 49 of 93
References Documents reviewed Documents used in this evaluation which are not identified directly in the body of the report as they are used are identified as footnotes at the end of each report section.
Team Members Chuck Sherman - Sponsor/ RP Brad O'Donnell - Electrical Maintenance Mark Floyd - Training Brian Meredith - System Engineering Maggie Koyo - System Engineering Ted Berry - Training Paul Johnson - Hut IS George Stieber - Operations Richard Swanson - Contractor AI Baerren - Maintenance Attachments - Event Time Line from EA-PSA-SDP-Dll-2-11-07 - Failure Modes Analysis - Failure Modes Chart - Barrier Analysis - Change Analysis - Safety Culture Evaluation - O&P Evaluation Checklist - Maintenance Crew Statements - Why Staircase Page 50 of 93
~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-Dll-2 07 Rev. 0 Attachment 01 - Page 51 of 93 : Event Timeline Chart and Narrative This attachment contains the following :
- Event timeline in chart format (Table A01-1)
- Event timeline in narrative format Page 51 . of 93
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-DU-2-11-07 Attachment 01 - Page 52 of 93 Rev. 0 Table AO I-I: Event T imeline Chart Saturday 9/24 Sund ay 9/25 Sund ay 9/25 Frid ay 9/23 1607 Sund ay 9/25 1506 Sunday 9/25 1506 Sund ay 9/25 1506 Sund ay 9/25 1506 Sund ay 9/25 1506 2218 1109 1500 Electrical maintenance Battery chargers Temp mod 3 1973 Electrical While removing bus bar, MSIS (2/4 logic low SG Right channel SIAS (214 AF AS (2/4 logic low S/G Right channel CIS/CHR (2/4 restoring breaker 72- I 23 #1 ED-IS and #2 Installed. (Temp maintenance short occurred in dc panel pressure) due to loss of logic low PZR pressure) due level) due to loss of logic, RIAX-I BOS/RlAX-(Emergency Airlock ED- ED- I 6 initially in- power for breaker removed 4 de ED- I 1-2 preferred ac buses EY - I 0 to loss of preferred ac buses preferred ac buses EY - I 0 I B07) due to loss of 123) service 72-1 2 I (generator panel ED- I 1-2 and EY-30 EY- IO and EY-30 and EY-30 preferred ac buses EY -10 exciter field breakers (72- and EY -30. Left channel breaker control) 119,72-120,72- containment isolation valves from 72- I 27 (test 12 1,72- 123) closed due to loss of power cabinets))
Control room alarm: EK- FWPair Shunt trip breaker 72-0 I MSIYs CY-OSIO and CY- IE bus EA- 13 de-energized, Turbjne driven AFW pump PCP controlled bleedoff 0316GENFIELD compressor C- opened de-energizing dc OSO I and E-SOB MFRY CY- no power to C-903B FWP P-BB starts (CY-0522B valves CY-20B3 and CY-FORCING/OYER 903B cross-tied buses ED- lOR and ED-IOL 0703 closed on MSIS, and air compressor (was cross- failed open due to loss of 2099 close due to CHRIloss EXCITATION cycling supplyi ng plant E-SOA MFRY CY-070 1 tied supplying plant air). ED-II -I ). AFW flow of power, directing flow to on/off air system closed due to loss of power Closed MV-CA320 to control valves CY -0727 and primary system drain tank T-to EY-IOand EY-30 isolate FWP from instrument CY-0749 fail full-open. 74 in containment (S gpm) air. C-2A instrument air Flow imbalance develops compressor was in "sleep" between SGs due to mode and started differential in dome pressures (no flow indication available)
Multiple containment Dc panels ED-I I-I and ED- All ADYs CY-0779, CY- In service PZR level control AFW pump P-BC starts PCS unidentified leakage >
isolation va lves position I 1-2, and preferred ac buses 07BO , CY-07 BI, and CY- channel A fails, charging (AFAS) supplying 16S gpm I gpm for PCP controlled ,
indication lost EY-IO and EY-30 de- 07B2 fail closed/inoperable pumps P-55A and P-SSB in to each SG. bleedoff iso lation (LCO energized (qui ck open and normal service (93 gpm), and Loss of EY - 10 causes low 3.4.I3 .A. I, B.I , B.2) operation) due to loss of letdown orifices CY-2003 , suction pressure trip signal preferred ac panel EY-10 CY-2004,CY- 200S close (0 which prevents P-BA (LCO 3.7.4) letdown),PZR heaters de- operation.)
energize Entered ON P-7.1 (72-1 19 Preferred ac panel EY - 10 MSSYs lift on both SGs In service PZR pressure Inverter # I ED-06 input Right channel CHP alarm fai lure caused loss of inoperable LCO 3.B.9.B control channel A fails , breaker to EY-IO, 72-3 7 (2/410gic,PSX-IBOIIPSX-service air and CY -1221 (LCO 3.0.3) spray valves CY-IOS7 and tripped (LCO 3.B.7.A) I B03) due to loss ofEY-I O FWP building cross-tic to Preferred ac panel EY -30 CY-IOS9 fail closed, no and EY -30 panels, no fail open) inoperable LCO 3.B.9.B spray available actuation (actuation logic (LCO 3.0.3) requirements not met)
Reactor trip (2/4 logic RPS) Turbine trip (from reactor Operators enter EOP- I.O Battery charger # 1 ED-IS, Dc bus ED- lOR inoperable due to loss of preferred ac trip), generator breakers do Standard Post-Trip Actions output breaker closed but (LCO 3.B.9.6) buses EY-I O and EY-30 not open due to loss of dc charger not operating Dc bus ED-IOL inoperable panel D- II -I (LeO 3.B.9.6)
Page 52 of 93
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 Attachment 01 - Page 53 of 93 Rev. 0 Table AO I-I : Event T imclinc Chart Sunday 9/25 1515 Sunday 9/25 151 7 Sunday 9/25 1527 Sunday 9/251531 S und ay 9/25 1537 Sunday 9/25 1542 Sunday 9/25 1544 Sunday 9/25 1549 Sunday 9/25 1555 MSSVs open and then . Operator jumpered relay Enter EOP-9.0 Operator observed high Per EOP-9.0, enter ONP- Isolated RV-2006 NCO closed P-SC AFW Restored I E bus EA-1 3 Observed PZR level operate 487u (Y-p hase) to open Functional Recovcry E-50A level (90%). 24. 1 and ONP-24.3 due letdown relief by placing flow control valve CV- (lost on SIAS at 1506) >62.S% (LCO 3.4.9.A).
(throttle/close/open) to generator output Procedure (due to <3 out Order given to isolate to loss of preferred ae letdown orifice stop 0737 A to isolate flow to and reenergized Actual PZR level 78%
maintain SG pressure breakers 25F7 and 25H9 of 4 preferred ac buses CV-0522B (steam to buses EY- IO and EY-30 valves CV-2003 , CV- E-50A, continue associated PZR heaters PCS Tave 544' F available) AFW pump P-SB) per 2004, and CV -2005 to supplying 165 gpm to EOP Supplement 19 close E-50B via CV-0736A (LCO 3.7.5)
IA bus EA-2 1 de- PZR level 62% - 1530 Entered ONP-2.3 PZR pressure peaks high Charging 73 gpm, 0 gpm energized. Primary Loss of DC Power (time 2200 psig. letdown, 5 gpm PCP coolant pumps P-50A not verified) PZR level 7 1% controlled bleedoffto and P-50C stop, P-50B primary system drain and P-50D remain in tank T-74 service Realigned PZR pressure control to B channel to enable spray, pressure begins lowering Realigned PZR level control and heater control select switch to B channel. Letdown orifices open and RV-2006 (letdown heat exchanger inlet safety relict) lifts du e to CV-2009 (letdown containment isolation) being closed on CHRIloss of power. ID bus EA-1 2 PZR backup heaters reenergize Charging pumps P-55A and P-55B in service (73 gpm charging, I OS gpm letdown relieving to quench tank) 5 gpm PCP controlled bleedoff to PSDT --- --- --
Page 53 of 93
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-Dll-2-11-07 Attachment 01 - Page 54 of 93 Rev. 0 Table AO I- I: Event Tim.line Chart Sunday 9/25 1557 Sund ay 9/25 1602 Sunday 9/25 1603 Sunday 9/25 1609 Sunday 9/25 16 15 Sunday 9/25 162 1 Sund ay 9125 1630 Sunday 9/25 1639 Sund ay 9/25 1646 Electricians report no Charging pump P-55B Steam to P-SB turbine CV-0736A closed to SG E-50A MSSVs lift, Entered ONP-7. 1 "Loss Charging pump P-55B Restorcd AFW to E-50B Preferred ac bus EY -30 fau lts on dc buses ED- suction relief RV-2096 isolated by closing CV- isolate flow from AFW E-50B MSSVs throttle of Instrument Air (duc to suction and discharge from P-SC 150 gpm realigned from bypass 10L and ED- lOR. lifting to the equipment 0522B. 0 AFW flow to pump P-SC to E-50B, no open. MSSVs then loss of all instrument air valves closed to isolate regulator to #3 inverter Reenergized ED-I OL drain tank T-SO. The E-50A. Still supplying AFW flow to either SG operate compressors at 1557) suction relief RV-2096 ED-OS suppl y and ED- lOR by closing tank overfilled causing 165 gpm to E-50B via P- at this time (throttle/close/open) to leak breaker 72-0 I (ED-IOL floor drains to backup on SC and flow control maintain sa pressure and ED-lOR now the 590' Aux iliary Bldg val ve CV -0736A Tave 544"F operable) (order sent to isolate P-55B)
Generator ficld breaker Restored powcr to I E PCS Tave 529"F. PZR PZR level peaks high Preferred ac bus EY -10 34 1 opened when ED- bus EA- 13 and level S5% 101.5% placed on bypass 11-2 reenerg ized rce;ncrgizcd associated regulator. EY-I O pressurizer heaters operable Preferred ac bus EY -30 ADVs CV-0779, CV-powered via bypass 07S0, CV-07SI , and CV-regulator (EY -30 now 07S2 operable due to operable) EY-IO restored (H1C-Left channel safety 07S0A now powered),
injection actuated when started controlling heat EY -30 reenergized, remo val using ADVs.
resulting in loss of IE MSSVs close bus EA- 13 Tave 540"F Throttled safcty injection. Stopped charging pumps P-55A and P-55B. Charging flow 0, Ictdown flow 0, 5 gpm PCP controlled bleedoff to PSDT.
PZR level SO%
When dc restored, instrument air compressor C-2A tripped (reason unknown)
Control room manually started instmmcnt air compressors C-2B and C-2C Page 54 of 93
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-Dll-2-11-07 Attachment 01 - Page 55 of 93 Rev. 0 Table AO I-I : Event Timeline Chart Sund ay 9/25 1720 Sund ay 9/25 1746 Sund ay 9/25 1818 Sunday 9/25 1852 Sunday 9/25 1909 Sund ay 9/251911 Sund ay 9/25 1923 Sund ay 9/25 1933 Sunday 9/25 2100 Entered ONP-4. 1, Exited EOP-9 and Reset SIAS Restored P-SB steam Exited ONP-24.1, Loss Exited ONP-24.3, Loss ED-O I, main station #3 battery charger ED- P-910 (main condenser Containment Spurious entered GOP-8, Power supply CV-0522B to ofY-IO ofY-30 battery left channel, 17 in service supplying vacuum pump) in-Isolation, reset CHR Reduction and Plant AUTO (LCO 3.7.5) inoperable per 3.S.4.B ED-O I (battery chargers service Shutdown to Mode 2 or (no connected battery #2 and #3 now in Mode 3 ::: 525°F(AII 4 charger and surveillance service) preferred ac buses in requirement 3.S.4.1 not service) met)
- I battery charger ED-15 inoperable per LCO 3.S.4.A.2 Table AO I-I : Event Ti meli ne C hart Sund ay 9/25 2330 Sund ay 9/25 2348 Monday 9/26 0156 Mond ay 9/26 03 11 Monday 9/26 044 1 T uesday 9/27 1733 Test started instrument PZR level <62.S% (LCO Restored P-55B charging Placed #4 battery Main station battery ED- # I inverter ED-06 air compressor C-2A 3.4.9) pump to service charger ED-IS in-service oI left channel operable operable, supplying satisfactorily, and then (available) and removed #2 battery preferred ae bus EY -10 placed in AUTO (C-2B charger ED-16 from (LCO 3.S.7) still in-service, C-2C in service, #3 battery SLEEP mode) charger ED-I 7 and #4 battery charger ED- IS now in service Page 55 of 93
a . Entergy PSA Engineering EA-PSA-SDP-D11-2-11-07 Rev. 0
~Entergy Analysis Attachment 01 - Page 56 of 93 Event Timeline Narrative I. Initial Conditions (prior to event)
- 100% reactor power
- normal single charging and letdown lineup o Charging pump P-55A in service o Letdown orifice stop valve CV-2003 open o Primary coolant pump CBa returning to volume control tank T-54
- pressurizer T-72 pressure and level control channel A in service
- auxiliary feedwater system in normal standby lineup
- #1 battery charger ED-15 and #2 battery charger ED-16 in service
- feedwater purity air system cross-tied with and supplying the plant compressed air system II . Electrical Equipment Conditions Concurrent with the Reactor Trip at 1506
- dc buses ED-10L and ED-10R de-energized o shunt trip breaker 72-01 opened o #1 battery charger de-energized
- dc distribution panels ED-11-1 and ED-11-2 de-energized
- #1 battery charger ED-15 failed , not supplying associated buses ED-1 OL and ED-10R
- #1 inverter ED-06 and #3 inverter ED-OB de-energized (ED-06 internal breaker also tripped)
- preferred ac buses EY-10 and EY-30 de-energized
- 2400v 1E bus EA-13 de-energized III. Conditions Resulting from Loss of Power to Preferred AC Buses EY-10 and EY-30 Reactor Trip / Turbine Trip: main generator breakers 25F7 and 25H9 did not open due to loss of ED-11-2.
Main Steam Isolation Signal: both main steam isolation valves CV-0501 and CV-0510 closed and both main feedwater regulating valves CV-0701 and CV-0703 closed. CV-0701 closed as result Page 56 of 93
Entergy PSA EA-PSA-SDP-Dll-2-11-07 Rev. 0
~Entergy Engineering Analysis Attachment 01 - Page 57 of 93 of loss of EY-10 and EY-30; CV-0703 closed due to MSIS.
Auxiliary Feedwater Actuation Signal: P-8A did not receive a start signal due to loss of 0-11-1, but also did not run due to low suction pressure trip logic actuation on loss of EY-1 O. P-8C started and supplied 165 gpm to each steam generator (E-50A and E-50B). Steam driven AFW pump P-8B started due to loss of panel EO-11-2 and AFW flow control valves CV-0749 (E-50A) and CV-0727 (E-50B) failed full open. P-8B flow indication was not available. Flow distribution was dependent on SG pressures. E-50A is the steam source for P-8B, resulting in initially lower pressure, while E-50B had no steam removal path other than MSSVs.
Safety Injection Actuation Signal : Right channel SIAS only - resulted in de-energizing (load shedding) 2400V 1E bus EA-13, isolating non-critical service water header isolation valve CV-1359 and starting associated equipment including charging pump P-55B.
Containment High Radiation : Right channel CHR only - resulted in containment isolation valves closing, including letdown isolation valve CV-2009 and PCP controlled bleedoff valve C-2099.
Left channel containment isolation valves also closed due to the loss of dc to their control circuits.
Containment High Pressure: Logic inputs were not sufficient for system actuation , i.e. no initiation signal was generated, alarm only.
Pressurizer Pressure Control: In service pressurizer pressure controller PIC-01 01 (channel A) de-energized - resulted in pressurizer spray valves CV-1057 and CV-1059 failing closed , and all available heaters energizing .
Pressurizer Level Control : In service pressurizer level controller LlC-0101(channel A) de-energized - resulted in letdown orifices closing, charging pump P-55A running at maximum speed (53 gpm) and all pressurizer heaters de-energizing. P-55C did not start due to loss of breaker control power (EO-11-1 ).
2400V 1E Bus EA-13: de-energized - resulted in unavailability of associated PZR heaters and FWP air compressors. Plant air compressor C-2A automatically started to restore pressure .
Atmospheric Steam Dump Valves: all ASOVs CV-0779, CV-0780, CV-0781 and CV-0782 failed closed (both normal and 'quick open') due to loss of power to controller HIC-0780A (EY-10).
Generator Output Breakers: breakers 25F7 and 25H9 failed closed and all switchyard breaker indication lost due to loss of EO-11-1 . 1A bus EA-21 and 1F bus EA-23 did not transfer to startup power on turbine trip due to loss of EO-11-2. 1A bus EA-21 remained powered from #1-1 station power transformer EX-01 until operators opened the generator breakers using a jumper on relay 487u (Y phase) in control room panel EC-04. 1F bus EA-23 remained powered from #1-3 station power transformer until the generator breakers opened.
IV. Plant / Equipment Conditions and Operator Actions Following Event Initiation Notes:
- Due to the high activity level and unavailability of some plant computer data during this event, times recorded in the Operator Log are generally correct, but may not exactly match information from other sources.
-Post-Trip Behavior.
Page 57 of 93
Entergy PSA EA-PSA-SDP-Dll-2-11-07 Rev. 0
~Entergy Engineering Analysis Attachment 01 - Page 58 of 93 1506: Conditions noted in III above.
Main steam safety valves (MSSVs) on both steam generator headers opened and operated (throttled/closed/opened) to maintain SG pressures and lower PCS temperature and pressure.
(MSSVs opened due to ASDVs failing closed and MSIVs closing on MSIS.)
Operators entered EOP-1.0 Standard Post-Trip Actions.
1515: Due to there being no steaming path available, PCS temperature rose to 544°F resulting in MSSVs opening further and PCS temperature, pressure and level lowering. PCS temperature continued lowering primarily due to relatively cold (BrF) AFW being supplied to the steam generators (690 gpm total).
AFW pump P-BB flow control valves CV-0727 and CV-0749 failed full open. The flow delivered to each SG was dependent on piping losses and SG pressure differences . SG pressures were initially both -930 psig. However, E-50A's pressure lowered more than E-50B's (possibly due to varying MSSV characteristics), resulting in significantly more cool AFW flow to E-50A, which further lowered its pressure. By 1530 total AFW flows (P-BB +P-BC) to the SGs were 502 gpm to E-50A and 195 gpm to E-50B. This flow imbalance contributed to over-filling E-50A.
1517: Power Control verified main generator breakers 25F7 and 25H9 were closed (failed to open on turbine trip). Operators installed a jumper on relay 4B7u (Y phase) in control room panel EC-04 to open the breakers per EOP-1.0. Opening the generator breakers de-energized 4160v 1A bus EA-21 , stopping primary coolant pumps P-50A and P-50C. PCPs P-50B and P-50D remained in service, maintaining forced circulation with one operating pump in each PCS/SG loop.
1527: Operators entered EOP-9.0 Functional Recovery Procedure due to less than 3 preferred AC buses being available. (Pressurizer level 62%)
1531: Operator observed high SG E-50A water level (90%) and an NPO was directed to isolate steam to P-BB per EOP Supplement 19 Alternate Auxiliary Feedwater Methods, i.e. manually closing steam supply valve CV-0522B. Both SG levels had been observed approximately equal (35% - 40%) during EOP-1.0 verbal verifications (1515). Operators entered ONP-2.3 Loss of DC Power.
1537: Operators first addressed safety function MVAE-DC-1 due to it being jeopardized (acceptance criteria not being met). Per MVAE-DC-1 operators entered ONP 24.1 Loss of Preferred AC Bus Y-1 0 and ONP-24.3 Loss of Preferred AC Bus Y-30 to recover the buses.
Operator observed high PCS pressure (2200 psia) due to loss of power to pressurizer pressure controller channel A which failed spray valves CV-1 057 and CV-1059 closed. Operator placed pressurizer pressure control channel B in service, lowered pressure in manual mode and then placed the controller in auto mode. PZR spray valves then remained available for pressure control.
Operator also noted loss of power to pressurizer level controller channel A and placed channel B and pressurizer heater select channel B in service. This resulted in letdown orifice stop valves CV-2003, CV-2004 and CV-2005 opening and charging pump P-55A speed lowering from 53 gpm to 33 gpm, and restored bus 1D pressurizer heater availability. Opening the letdown orifice valves resulted in letdown relief valve RV-2006 opening, due to CV-2009 having closed on CHR.
RV-2006 directed letdown flow (1 OB gpm, 560 gal total) to quench tank T-73 in containment, and resulted in relief valve 2006 discharge high temperature annunciator EK-0702 alarming.
(Pressurizer level 71 %) .
Page 58 of 93
Entergy PSA EA-PSA-SDP-Dll-2-11-07 Rev. 0
~Entergy Engineering Analysis Attachment 01 - Page 59 of 93 1542: Operator closed letdown orifice stop valves CV-2003 , CV-2004 and CV-2005 to isolate letdown flow per ARP-4 Annunciator Response Procedure Primary System Volume Level Pressure Scheme EK-07 (C-12).
At this time charging flow was 73 gpm with 0 letdown and 5 gpm PCP bleedoff flow, resulting in 68 gpm PCS net inventory addition. When the density change from charging temperature to PCS temperature is considered this gives a 90 gpm effective charging rate or 1.36%/minute pressurizer level rise rate (90 gpm / 66.16 g/% = 1.36%/m). (Pressurizer volume gal / %
=
indicated level 66.16 g/% per surveillance procedure DWO-1 Operator's Daily/Weekly Items Modes 1, 2, 3, and 4 Rev 80.)
1544: Operator closed CV-0737 A, isolating P-8C AFW flow to steam generator E-50A. P-8C flow to E-50B continued at 165 gpm , and P-8B flow continued at 380 gpm to E-50A and 0 gpm to E-50B.
1549: Operators restored power to 2400v 1E bus EA-13 per SOP-30 Station Power and reenergized associated pressurizer heaters .
1555: Operator logged pressurizer level high (>62 .8%) (actual level 78%). Due to PCS temperature continuing to lower, the observed level rate of rise was less than would be observed if temperature was stable. Changing PCS temperature one degree has the effect of changing PCS water volume 74.43 gallons (per DWO-1). (Note: Per PZR pressure/level recorder LPIR-0101 B, pressurizer level exceeded 62.8% at 1528.)
1557: Operator aligned preferred ac bus EY-30 to be supplied from instrument ac bus EY-01 via the bypass regulator. Energizing EY-30 resulted in Left channel safety injection actuation which de-energized (load shed) 2400V 1E bus EA-13 and started associated equipment. P-55C did not start due to panel EO-11-1 being de-energized.
Operators verified SI throttling criteria met and stopped both operating charging pumps P-55A and P-55B to stop PCS inventory addition. Charging and letdown flows = 0, 5 gpm PCP bleedoff to primary system drain tank T-74 continues. (Pressurizer level 80%)
Electricians reported buses ED-10L and ED-10R fault free. Operator closed shunt trip breaker 72-01 reenergizing Left channel dc buses ED-10L, ED-10R, ED-11-1 , EO-11-2 from battery ED-01.
Generator field breaker 341 automatically opened when EO-11-2 was reenergized. Instrument air compressor C-2A tripped for an unknown reason when dc power was restored . Operator manually started compressors C-2B and C-2B. The brief loss of air compressor had no noticeable effect.
1602: NPO reported charging pump P-55B suction relief valve RV-2096 lifting and not reseating ,
equipment drain tank T-80 full and floor drains backing up on the auxiliary building 590 elevation.
Control room directed closing pump suction and discharge valves to isolate P-55B and its suction relief. Water discharged from the relief was from concentrated boric acid tanks T-53A and T-53B.
Operators restored power to 1E bus EA-13 and reenergized associated pressurizer heaters.
1603: Auxiliary operator reported steam supply valve to P-8B turbine CV-0522B manually closed per EOP Supplement 19. AFW flow to and steam flow from E-50A = O. AFW flow to E-50B continued at 165 gpm and steam flow from E-50B was controlled by associated MSSVs. PCS heat removal rate was reduced and PCS temperature stopped lowering and started rising. The Page 59 of 93
a
~Entergy Entergy PSA Engineering EA-PSA-SDP-Dll-2-11-07 Rev. 0 Analysis Attachment 01 - Page 60 of 93 PCS heatup rate was 1°F/m, resulting in PZR level rising 1.125%/m. (Tave 529°F, PZR level 85%)
1609: Operator dosed CV-0736A, isolating AFW flow to E-50B, slightly raising the PCS heatup rate. There was no AFW to either SG at this time and steam was only being removed from E-50B via MSSVs throttling.
1615: MSSVs on both steam generator headers opened due to PCS temperature rising to 544°F.
PZR level peaked at 101 .5% and then lowered as PCS temperature lowered. (Note: There was no PCS inventory addition since 1557. The PZR level rise was entirely due to PCS heatup from 529°F to 544°F.) After opening , the MSSVs remained partially open , effectively controlling PCS Tave 540°F until the ASOVs were placed in service.
1621 : Operators entered ONP-7.1 Loss of Instrument Air and placed C-2B and C-2C in service.
Compressor C-2A tripped during electrical bus restoration as previously noted.
1630: Charging pump P-55B suction and discharge valves reported dosed , isolating suction relief valve leakage.
1639: Operator restored 150 gpm AFW flow to E-50B using P-8C.
1646: After confirming no faults on preferred ac bus EY-10, #3 inverter EO-08 was aligned to supply EY-30 and EY-10 was powered from instrument ac bus EY-01 via the bypass regulator. All preferred ac buses were now available .
All 4 AOVs were available when EY-10 was restored , and operators began using them for PCS temperature control. MSSVs fully dosed . (Tave 539°F) 1720: Operators entered ONP-4.1 Containment Spurious Isolation and operator reset CHR.
1746: Operators exited EOP-9.0 and entered GOP-8 Power Reduction and Plant Shutdown to Mode 2 or Mode 3 ~ 525°F.
1818: Operators reset SIAS and restored non-critical service water per SOP-15 Service Water System.
1852: Operators restored AFW pump P-8B steam supply CV-0522B to AUTO per EOP Supplement 19.
1933: Placed #3 battery charger EO-17 in service supplying station battery EO-01 . #2 and #3 battery chargers EO-16 and EO-17 in service.
2348: Pressurizer level lowered to 62% and continued lowering due to PCP bleedoff.
09/26/11, 0311 : Placed #4 battery charger EO-18 in service supplying station battery EO-02.
Battery chargers #3 EO-17 and #4 EO-18 in service.
09/27/11 , 1733: Placed #1 inverter EO-06 in service supplying #1 preferred ac bus EY-10.
Page 60 of 93
a
~Entergy Entergy PSA Engineering EA-PSA-SDP-D11-2-11-07 Rev. 0 Analysis Attachment 01 - Page 61 of 93 Acronyms AFAS Auxiliary Feedwater Actuation Signal AFW Auxiliary Feedwater ASDV Atmospheric Steam Dump Valve CAS Compressed Air System CBO Controlled Bleedoff CHP Containment High Pressure CHR Containment High Radiation CIS Containment Isolation Signal CVCS Chemical and Volume Control System FWP Feedwater Purity MFRV Main Feedwater Regulating Valve MSIS Main Steam Isolation Signal MSIV Main Steam Isolation Valve MSSV Main Steam Safety Valve NCO Nuclear Control Operator NPO Nuclear Plant Operator PCP Primary Coolant Pump PCS Primary Coolant System PSDT Primary System Drain Tank PZR Pressurizer RPS Reactor Protective System SG Steam Generator SI Safety Injection SIS Safety Injection Actuation Signal VCT Volume Control Tan Page 61 of 93
Attachment 2 - Failure Modes Analysis Problem Statement: Electrical Power was lost to DC Bus ED-l OL and ED-lOR Additional Failure Modes Supporting Evidence Refuting Evidence Assumptions Conclusion Information Simultaneous Power to 125 VDC Bus There was no reported None None It is extremely unlikely these events manual opening of: EDIO-L & R was lost. observation of individuals could be coordinated to occur at the
[Bkr 72-18 (Battery opening these breakers. same time. Therefore this Failure No. I ED-I) or 72- The breakers would have to Mode is not considered to be 01 (Isolation Breaker have been opened simultaneous credible.
To DC Battery No. I with the shorting event ED-I)] observed during the ED-II-2 AND work.
Bkr 72-11 (Charger No.3 ED-17)
AND Bkr 72-15 (Charger No. I ED-15)
A (+) to (-) short Power to 125 VDC Bus Multiple individuals observing The short caused Bkr 72- Upon It is extremely unlikely that a short occurred in ED-08 ED I O-L & R was lost. the Electrical Maintenance 18 (Battery No. I ED-I) restoration, ED- would spontaneously occur in ED-(Inverter No.3) or in work being performed in DC to open and the short 08 (Inverter No. 08 (Inverter No.3) or it's connected its connected power Distribution Panel ED-I 1-2 caused Battery Chargers 3) was reported power feed cable simultaneously feed cabling. observed an electrical arc when ED-15 and ED-I 7 to to be operating with the electrical arc caused by the a piece of copper bar attached cease output operation. normally. loss of physical control of the to the aluminum bus bar swung copper bar being worked in DC and made contact with the Distribution Panel ED-I 1-2.
opposite polarity bus. There is Therefore this Failure Mode is not also photographic evidence of considered to be credible.
the results of an electrical arc.
A (+) to (-) short Power to 125 VDC Bus Multiple individuals observing The short caused Bkr 72- None It is extremely unlikely that a short occurred internally ED JO-L & R was lost. the Electrical Maintenance 18 (Battery No. I ED-J) would spontaneously occur in spare Page 62 of 93
to spare breaker 72- work being performed in DC to open and the short breaker 72-11 simultaneously with 11 (no connected Distribution Panel ED- I 1-2 caused Battery Chargers the electrical arc caused by the loss load or cabling) observed an electrical arc when ED-IS and ED-I7 to of physical control of the copper bar a piece of copper bar attached cease output operation. being worked in DC Distribution to the aluminum bus bar swung Panel ED- I 1-2. Therefore this and made contact with the Failure Mode is not considered to opposite polarity bus. There is be credible.
also photographic evidence of the results of an electrical arc .
A (+) to (-) short Power to 125 VDC Bus At the time of the event, ED-I7 The short caused Bkr 72- Upon It is extremely unlikely that a short occurred in ED- I7 ED! O-L & R was lost. (Battery Charger No. 3) was in 18 (Battery No . 1 ED- I) restoration, ED- would spontaneously occur in ED-(Battery Charger No. standby with it's breaker 72-12 to open and the short 17 (Battery 17 (Battery Charger No. 3) or it's
- 3) or in its connected (Charger NO . 3 ED- l7) OPEN. caused Battery Chargers Charger No. 3) connected power feed cable power feed cabling. Therefore, a short in ED-l7 ED- I S and ED-I 7 to was reported to simultaneously with the electrical could not have affected 125 cease output operation. be operating arc caused by the loss of physical VDC Bus ED! O-L & R as it normally. control of the copper bar being was not electrically connected. worked in DC Distribution Panel ED-II -2. Therefore this Failure Mode is not considered to be credible.
A (+) to (-) short Power to 125 VDC Bus Multiple individuals observing The short caused Bkr 72- None It is extremely unlikely that a short occurred in P-8IA ED!O-L & R was lost. the Electrical Maintenance 18 (Battery No. 1 ED-I ) would spontaneously occur in P-(DC Primary work being performed in DC to open and the short 8IA (DC Primary Coolant Pump Coolant Pump Oil Distribution Panel ED-I 1-2 caused Battery Chargers Oil Lift Pump) or it's connected I Lift Pump) or in its observed an electrical arc when ED- I S and ED-I 7 to power feed cable simultaneously I I
connected power a piece of copper bar attached cease output operation. with the electrical arc caused by the I feed cabling. to the aluminum bus bar swung loss of physical control of the and made contact with the copper bar being worked in DC opposite polarity bus. There is Distribution Panel ED-I 1-2.
also photographic evidence of Therefore this Failure Mode is not the results of an electrical arc . . considered to be credible.
A (+) to (-) short Power to 125 VDC Bus Multiple individuals observing The short caused Bkr 72- None It is extremely unlikely that a short occurred in P-81 C ED IO-L & R was lost. the Electrical Maintenance 18 (Battery No .1 ED- I ) would spontaneously occur in P-(DC Primary work being performed in DC to open and the short 81 C (DC Primary Coolant Pump Oil Coolant Pump Oil Distribution Panel ED- I 1-2 caused Battery Chargers Lift Pump) or in its connected Lift Pump) or in its observed an electrical arc when ED-I S and ED-l7 to power feed cable simultaneously connected power a piece of copper bar attached cease output operation. with the electrical arc caused by the Page 63 of 93
feed cable. to the aluminum bus bar swung loss of physical control of the and made contact with the copper bar being worked in DC opposite polarity bus. There is Distribution Panel ED-II-2.
also photographic evidence of Therefore this Failure Mode is not the results of an electrical arc . considered to be credible.
A (+) to (-) short Power to 125 VDC Bus Multiple individuals observing The short caused Bkr 72- None It is extremely unlikely that a short occurred in ED-ll-l EDI0-L & R was lost. the Electrical Maintenance 18 (Battery No.1 ED-I) would spontaneously occur in ED-(125 VDC Panel) or work being performed in DC to open and the short 11-1 (125 VDC Panel) or any of the any of the loads Distribution Panel ED-II-2 caused Battery Chargers loads connected to this panel connected to this observed an electrical arc when ED-15 and ED-17 to simultaneously with the electrical panel. a piece of copper bar attached cease output operation. arc caused by the loss of physical to the aluminum bus bar swung control of the copper bar being and made contact with the worked in DC Distribution Panel opposite polarity bus. There is ED-II-2. Therefore this Failure also photographic evidence of Mode is not considered to be the results of an electrical arc. credible.
A (+) to (-) short Power to 125 VDC Bus None The short caused Bkr 72- Based on This Failure Mode has a high occurred in ED-l1-2 ED 1O-L & R was lost. 18 (Battery No. 1 ED-I) inspection post probability as being the cause of the (125 VDC Panel) or to open and the short event, Bkr 72- loss of electrical Power to DC Bus any of the loads Multiple individuals caused Battery Chargers 18 (Battery No. ED-IOL and ED-lOR.
connected to this observing the Electrical ED-I5 and ED-I7 to 1 ED-I) was panel. Maintenance work cease output operation. found being performed in DC CLOSED, but Distribution Panel ED- After the event, Bkr 72- breaker 72-01 11-2 observed an 37 (Inverter No.1 ED-06 (Isolation electrical arc when a Power Supply Breaker) Breaker To Dc piece of copper bar was found tripped. Per Battery No. 1 attached to the the Inverter vendor, ED-I) was aluminum bus bar capacitors on the feed discovered swung and made side of the Inverter TRIPPED .
contact with the would have fed a opposite polarity bus. connected fault. The Per Dwg. E-8 There is also vendor concurred that it Sh. 1, Breaker photographic evidence was reasonable to 72-01 (Isolation of the results of an conclude that Bkr 72-37 Breaker To Dc electrical arc. could have tripped from Battery No . 1 the current produced by ED-I) is Page 64 of 93
the Inverter capacitors. identified as a Located between ED- Non-Automatic lOR (125 VDC Bus) and shunt trip ED-I 1-2 (125 VDC breaker, that is Panel) is fuse FUZ/D 11- this breaker 2 (Feeder Fuse To Relay would not In Panel ED- IIA) . This incorporate a fuse was found intact. magnetic or The assumption is that it thermal is reasonable to expect automatic trip that the fault current feature. As has contributed by ED-06 been discovered, (Inverter No. I) would this breaker trip Bkr 72-37 and not does in fact fuse FUZ/D11 -2. Note contain a that for the stated fai lure magnetic mode, all of the fault automatic trip current would have feature.
passed through fuse FUZ/D II -2, and this fuse was found intact.
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- Upon It is extremely unlikely that a short occurred in ED-IS Buss EDIO-L & R was the Electrical Maintenance 18 (Battery No. I ED-I) restoration, ED- would spontaneously occur in ED-(Battery Charger No. lost. work being performed in DC to open and the short IS (Battery IS (Battery Charger No. I) or it's
- 1) or in its connected Distribution Panel ED-I I -2 caused Battery Chargers Charger No . I) connected power feed cable power feed cabling. observed an electrical arc when ED-IS and ED-17 to was reported to simultaneously with the electrical a piece of copper bar attached cease output operation. be operating arc caused by the loss of physical to the aluminum bus bar swung normally. control of the copper bar being and made contact with the worked in DC Distribution Panel opposite polarity bus. There is ED-II -2. Therefore this Failure also photographic evidence of Mode is not considered to be the results of an electrical arc. credible.
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- None It is extremely unlikely that a short occurred internally Buss EDIO-L & R was the Electrical Maintenance 18 (Battery No. I ED-I ) would spontaneously occur in spare i
to spare breaker 72- lost. work being performed in DC to open and the short breaker 72-16 simultaneously with 16 (no connected Distribution Panel ED- II -2 caused Battery Chargers the electrical arc caused by the loss load or cabling) observed an electrical arc when ED-IS and ED- 17 to of physical control of the copper bar I Page 65 of93
a piece of copper bar attached cease output operation. being worked in DC Distribution to the aluminum bus bar swung Panel ED-I 1-2. Therefore this and made contact with the Failure Mode is not considered to opposite polarity bus. There is be credible.
also photographic evidence of the results of an electrical arc.
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- Upon It is extremely unlikely that a short occurred in ED-06 Buss EDlO-L & R was the Electrical Maintenance 18 (Battery No. I ED-I) restoration, ED- would spontaneously occur in ED-(Inverter No. I) or in lost. work being performed in DC to open and the short 06 (Inverter No. 06 (Inverter No. I) or it's connected its connected power Distribution Panel ED-I 1-2 caused Battery Chargers I) was reported power feed cable simultaneously feed cabling. observed an electrical arc when ED-15 and ED-I 7 to to be operating with the electrical arc caused by the a piece of copper bar attached cease output operation. normally. loss of physical control of the to the aluminum bus bar swung copper bar being worked in DC and made contact with the Distribution Panel ED-I 1-2.
opposite polarity bus. There is Therefore this Failure Mode is not also photographic evidence of considered to be credible.
the results of an electrical arc.
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- None It is extremely unlikely that a short occurred in EU-72 Buss ED lO-L & R was the Electrical Maintenance 18 (Battery No . I ED-I) would spontaneously occur in ED-(Public Address lost. work being performed in DC to open and the short 06 (Inverter No. I) or it's connected System Inverter) or Distribution Panel ED-I 1-2 caused Battery Chargers power feed cable simultaneously connected cabling. observed an electrical arc when ED-15 and ED-I 7 to with the electrical arc caused by the a piece of copper bar attached cease output operation. loss of physical control of the to the aluminum bus bar swung copper bar being worked in DC and made contact with the Distribution Panel ED-I 1-2.
opposite polarity bus. There is Therefore this Failure Mode is not also photographic evidence of considered to be credible.
the results of an electrical arc.
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- Upon It is extremely unlikely that a short occurred in the ED- Buss ED lO-L & R was the Electrical Maintenance 18 (Battery No . I ED-I) restoration, the would spontaneously occur in the lOL (125 VDC Bus) lost. work being performed in DC to open and the short ED-IOL (125 ED-IOL (125 VDC Bus) bus work bus work. Distribution Panel ED-I 1-2 caused Battery Chargers VDC Bus) bus simultaneously with the electrical observed an electrical arc when ED-I5 and ED-I 7 to provided power arc caused by the loss of physical a piece of copper bar attached cease output operation. to it's connected control of the copper bar being to the aluminum bus bar swung load without worked in DC Distribution Panel and made contact with the incident. ED-II -2. Therefore this Failure opposite polarity bus. There is Mode is not considered to be Page 66 of 93
also photographic evidence of credible.
the results of an electrical arc.
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- Upon It is extremely unlikely that a short occurred in the ED- Buss ED! O-L & R was the Electrical Maintenance 18 (Battery No. 1 ED-I) restoration, the would spontaneously occur in the lOR (125 VDC Bus) lost. work being performed in DC to open and the short ED-lOR (125 ED-lOR (125 VDC Bus) bus work bus work. Distribution Panel ED-l 1-2 caused Battery Chargers VDC Bus) bus simultaneously with the electrical observed an electrical arc when ED-15 and ED-l 7 to provided power arc caused by the loss of physical a piece of copper bar attached cease output operation. to it' s connected control of the copper bar being to the aluminum bus bar swung load without worked in DC Distribution Panel and made contact with the incident. ED-II-2. Therefore this Failure opposite polarity bus. There is Mode is not considered to be also photographic evidence of credible.
the results of an electrical arc .
A (+) to (-) short Power to 125 VDC Multiple individuals observing The short caused Bkr 72- It is extremely unlikely that a short occurred in the ED- Buss EDIO-L & R was the Electrical Maintenance 18 (Battery No. I ED-I) would spontaneously occur in the 13 (Metering Section lost. work being performed in DC to open and the short ED-13 (Metering Section WlIN MN Wl IN MN DC Dist Distribution Panel ED-II-2 caused Battery Chargers DC Dist Bus #1) ground detection Bus # I) ground observed an electrical arc when ED-15 and ED-17 to circuitry simultaneously with the detection circuitry. a piece of copper bar attached cease output operation. electrical arc caused by the loss of to the aluminum bus bar swung physical control of the copper bar and made contact with the being worked in DC Distribution opposite polarity bus. There is Panel ED-I 1-2. Therefore this also photographic evidence of Failure Mode is not considered to the results of an electrical arc. be credible.
Summary: The Failure Mode: A (+) to (-) short occurred in ED-I 1-2 (125 VDC Panel) or any of the loads connected to this panel, was considered to be the credible failure which resulted in loss of Electrical Power to DC Bus ED-I OL and ED-lOR. Specifically, the short was due to the contact of opposite polarity copper stand-offs, one of which was in the process of being removed by Electrical Maintenance when the fault occurred.
References:
Dwg. E-8 Sh. 1 and Operators Statements regarding the Plant Trip Notes: None Page 67 of 93
- Failure Modes Chart REACTOR TRIP 9 /25/11 @ 1506 I I I REACTOR @ POWER CHARGER 1 IN SERVICE RODS DROP
- 1 -- 1 I END END RPS CHANNELS A I C LOSE POWER I
EY-101 EY-30 DE-ENERGIZED I
Page 68 of 93
~~
I I I I 72-12 OPEN FUZ/D-11-2 DOES NOT 72-15 TRIPS 72-37 TRIPS 72-01 TRIPS 72-36 DOES NOT TRIP (CHARGER 3 OPEN (ED-01) (CHARGER 1) (INVERTER 1) (ED-01) (INVERTER 3)
IN STANDBY)
I 1 I 1 I I END END D10RVOLTS CAPACITOR I I D10R VOLTS DROP DROP DISCHARGE 72-01 TRIPS 72-15 TRIPS FASTER FASTER I I I 1 1 D10RI D10L FAULT D10RI D10L FAULT D10RI D10L FAULT 72-01 AUTO-TRIP END END I I
--------- I CONFLICT NOT FSARI DESIGN CORRECTED CONFLICT 1
END Page 69 of 93
MAINTENANCE ON SHORT CIRCUIT DURING ED-11-2 ED-11-2 MAINTENANCE OBSERVED DEFICIENCIES ORGANIZATION WORK ON ENERGIZED AND PROCESS PANEL WEAKNESSES END RECOGNIZED JOB SKILL BARRIER ANALYSIS CHANGE ANALYSIS END Page 70 of 93
- Barrier Analysis GOAL HAZARD BARRIER HOW BARRIER FAILED IMPACT ON EVENT Work order plan did not include Adequate details for removal of bus bar Contributing Cause 2 information with damaged threads and available to plan Work order planning insulation of energized to plan and components. Used notes for complete work instructions and was based on WOTask LTA outdated procedure.
Indicated there was Plant trip Ops impact review concern which but that it was Root Cause 1 slight.
I No Plant maintenance procedures included in WO plan Procedure quality but the procedure referenced was Contributing Cause 2 I Plant Maintenance outdated Procedures LTA EPS-E-10 has been in use but no Maintenance feedback evidence of DRNs Contributing Cause 2 HPER input that EN-IS-123 vs EN-OP-102 needs clarification Enhancement Procedure quality per shift manager face-to-face meetings on energized equipment Fleet Procedures LTA work Incorporation of previous No evidence of previous NA comments unincorporated com merits Page 71 of 93
Relevant OE30342 included in OE References LTA Planning development WO package NA Qualitative risk assessments at four points could have resulted in Root Cause 1; see change Risk assessment increased management attention analysis and better decision making Inappropriate work scheduling Managers were not intrusive, unaware and unable to control Management oversight Root Cause 11 LOW what was going on Crew indicated concerns Feedback to correct identified were reported and Work crew review NA previously known problems resolved Breaker inspection and Crew develops replacement was not turned over; Face to face review good plan to Shift turnover LTA continued with same crew after NA perform work temporary mod was installed Opportunity to review work Crew review had opportunity and order package on schedule NA identified concerns Crew review LTA Opportunity to resolve Concerns related to drawing comments on work order Root Cause 1 and Contributing availability not resolved but crew package Cause 4 continued with work Use of correct PJB form Based on interviews, EN-HU-102 Prejob brief LTA was not referenced and checklists not used; in addition, during HPER, workers indicated they Root Cause 1 and Contributing Correct performance of were not clear about how to use Cause 4 PJB EN-H U-1 02 flowchart for selecting prejob brief level Page 72 of 93
Supervisor, duty station manager and asst maintenance manager Manager and supervisor were present at brief and should Supports Contributing Cause 3 oversight have identified failure to use and Root Cause 1 checklists OE provided was relevant but not successfully used due to weak Use ofOE Contributing Cause 4 prejob brief During HPER discussions, workers were considered and Worker skills LTA Training and experience they considered themselves NA trained an experienced Worker decisions not to stop and regroup when deciding to remove bus connectors and to change Response to changes LT A Root Cause 1 and Contributing Worker regroup and rebrief from insulation sheets to Cause 4; see Change Analysis insulating gloves violated procedures and expectations Crew completes work as planned During HPER reviews, crew Worker and supervisor indicated there were no adverse Working conditions LTA NA recognition impacts from working conditions Weaknesses in job execution not recognized and corrected even though supervisor, duty station Oversight LTA Manager and supervisor Contributing Cause 3 and Root manager and asst maintenance oversight Cause 1 manager were present and able to recognize what was going on Page 73 of 93
- Change Analysis ATTRIBUTE PROBLEM PRIOR DIFFERENCES/ CHANGES IMPACT ON EVENT SITUATION SITUATION Work Decision is Only Replacement of breaker 72-123 Weak decision making and failure priority made to troubleshooting will involve work on energized bus to perform qualitative risk analysis decision replace 72- would be of panel ED-11-2 which has EOOS support Root Cause 1 123 performed orange risk factor Use of Insulating Prejob brief There is inadequate separation Failure to rebrief this change in insulating gloves are plan was to between opposing phase bus plans reduces chances for gloves used to sheets of connectors during repair work successful completion and remove insulating which increases the chance of supports Root Cause 1 damaged bus material short circuit connectors in panel ED-11-2 Work on Damaged bus Work order Removal reduces chances that Worker do not stop to change the damaged connectors plans indicate metal chips will fall into bus which instructions or rebrief the next bus are removed bus connectors is good; removal of connectors steps to be completed; supports connectors from panel will be repaired increases risk that errors will occur Root Cause 1 ED-11-2to in the panel. because it increases the repair threads Task does not complexity of the task include the word REMOVE. ----
Page 74 of 93
Attachment 6 - Safety Culture Evaluation TABLE 1 - SAFETY CULTURE COMPARISON CR-PLP-2011-4822- Plant Trip SAFETY CULTURE During Panel ED-11-2 DESCRIPTION COMPONENT Maintenance i
- 1. Decision-Making Licensee decisions demonstrate that nuclear safety is an overriding priority.
RC1
- 2. Resources The licensee ensures that personnel , equipment, procedures, and other RC1! CC2 resources are available and adequate to assure nuclear safety.
- 3. Work Control The licensee plans and coordinates work activities, consistent with nuclear safety RC1
- 4. Work Practices Personnel work practices support human performance .
RC1! CC2! CC3
- 5. Corrective Action The licensee ensures that issues potentially impacting nuclear safety are No safety culture weakness Program promptly identified , fully evaluated , and that actions are taken to address safety determined for any of the identified issues in a timely manner, commensurate with their significance. causes
- 6. Operating Experience The licensee uses operating experience (OE) information , including vendor No safety culture weakness recommendations and internally generated lessons learned, to support plant determined for any of the identified safety. causes
- 7. Self- and The licensee conducts self- and independent assessments of their activities and No safety culture weakness Independent practices, as appropriate, to assess performance and identify areas for determined for any of the identified Assessments improvement. causes
- - - - - - ~--
Page 75 of 93
CR-PLP-2011-4822- Plant Trip SAFETY CULTURE During Panel ED-11-2 DESCRIPTION COMPONENT Maintenance
- 8. Environment For An environment exists in which employees feel free to raise concerns both to No safety culture weakness Raising Concerns their management and/or the NRC without fear of retaliation and employees are determined for any of the identified encouraged to raise such concerns. causes
- 9. Preventing, Detecting A policy for prohibiting harassment and retaliation for raising nuclear safety No safety culture weakness and Mitigating concerns exists and is consistently enforced. determined for any of the identified Perceptions of causes Retaliation
- 10. Accountability Management defines the line of authority and responsibility for nuclear safety.
RClICC3 11 . Continuous The licensee ensures that a learning environment exists. RClICC3 Learning Environment
- 12. Organizational Management uses a systematic process for planning, coordinating, and No safety culture weakness Change Management evaluating the safety impacts of decisions related to major changes in determined for any of the identified organizational structures and functions, leadership, policies, programs, causes procedures, and resources . Management effectively communicates such changes to affected personnel.
- 13. Safety Policies Safety policies and related training establish and reinforce that nuclear safety is No safety culture weakness an overriding priority in that. determined for any of the identified causes NOTES
- 1. Decision Making - Indicated because personnel in all parts of the Plant organization did not recognize, account for or prepare for the industrial safety and Plant operational risk involved with the panel ED-11-2 breaker maintenance. (H1a). This was described by the examples identified for RC1 and will be addressed by the corrective actions assigned to that cause and its Extent.
Page 76 of 93
- 2. Resources - Indicated because management personnel exceeded established work hours rules and failed to follow prejob brief requirements (H2b) and because work orders used for breaker maintenance did not include details appropriate for energized, high critical equipment (H2c)
Concerns will be addressed by the corrective actions for RC1 and CC2.
- 3. Work Control - Indicated because all parts of the Plant organization did not recognize, account for or prepare for the industrial safety and Plant operational risk involved with the panel ED-11-2 breaker maintenance. (H3a) This was described by the examples identified for RC1 and will be addressed by the corrective actions assigned to that cause and its Extent.
- 4. Work Practices - Oversight by managers and supervisors did not result in identification and correction (H4b) of the errors and weaknesses on the part of the Maintenance work crew (H4a) involved with the breaker inspection and maintenance work that lead to this event. Based on the evidence, the Plant organization did not recognize and account for the industrial safety and Plant trip risk involved(H4c) This will be addressed by corrective actions assigned for RC1, CC2 and CC3.
- 5. Accountability - Managers and supervisors did not recognize the industrial safety and Plant trip risk involved with the breaker inspection and maintenance work that lead to this event and did not reinforce safety standards among themselves or with those doing the work (A 1b) . result in identification and correction (H4b) of the errors and weaknesses on the part of the Maintenance work crew (H4a) involved. This concern will be addressed by corrective actions assigned for RC1 and CC3.
- 6. Continuous Learning Environment - Regardless of supervisor and manager training based on SOER 10-2, none of those present at the prejob brief or at the work site, identified or corrected lapses in the conduct of the prejob brief, the lack of critical steps or the failure to stop and reconsider when the job plan changed (C2a). This concern will be addressed by corrective actions assigned for RC1 and CC3.
Page 77 of 93
TABLE 2 - DETAILED SAFETY CULTURE COMPONENT REVIEW CR-PLP-2011-4822- Plant Trip During Panel Description ED-1l-2 Maintenance
- 1. Decision-Making Licensee decisions demonstrate that nuclear safety is an overriding priority. Specifically (as applicable):
DM H.l(a) The licensee makes safety-significant or risk-significant dec isions using a systematic process, Applicable to RC1. See Note 1.
especially when faced with uncertain or unexpected plant conditions, to ensure safety is maintained. This includes fonnally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, and implementing these roles and authorities as designed and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions.
DM H. l (b) The licensee uses conservative assumptions in decision making and adopts a requirement to No safety culture weakness determined demonstrate that the proposed action is safe in order to proceed rather than a requirement to for any of the identified causes demonstrate that it is unsafe in order to disapprove the action. The licensee conducts effectiveness reviews of safety-significant decisions to verify the validi ty of the underlying assumptions, identify possible unintended consequences, and determine how to improve future decisions.
DM H. l (c) The licensee communicates decisions and the basis for decisions to personnel who have a need No safety culture weakness determined to know the infonnation in order to perform work safely, in a timely manner.
for any of the identified causes
- 2. Resources The licensee ensures that p ersonnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety. Specifically, those necessary for:
RES H.2(a) Maintaining long tenn plant safety by maintenance of design margins, minimization of long- No safety culture weakness determined standing equipment issues, minimizing preventative maintenance deferrals, and ensuring for any of the identified causes maintenance and engineering backlogs which are low enough to support safety.
RES H.2(b) Training of personnel and sufficient qualified personnel to maintain work hours within Applicable to RC1. See Note 2 working hours guidelines.
RES H.2(c) Complete, accurate and up-to-date design documentation, procedures, and work packages, and Applicable to CC2. See Note 2.
correct labeling of components.
RES H.2(d) Adequate and avai lable facilities and equipment, including physical improvements, simul ator No safety culture weakness determined fidelity and emergency fac ilities and equipment.
for any of the identified causes.
- 3. Work Control The licensee plans and coordinates work activities, consistent with nuclear safety. Specifically (as applicable):
we H.3(a) The licensee appropriately plans work activities by incorporating' risk insights;
- job site Applicable to RC1. See Note 3. ,
conditions, including environmental conditions which may impact human performance; plant structures, systems, and components; human-system interface; or radiological safety; and, the need for planned contingenc ies, compensatory actions, and abort criteria.
Page 78 of 93
CR-PLP-2011-4822- Plant Trip During Panel Description ED-1l-2 Maintenance WC H.3(b) The licensee appropriately coordinates work activities by incorporating actions to address: I
- No safety culture weakness determined the impact of changes to the work scope or activity on the plant and human performance.* the for any of the identified causes impact of the work on different job activities, and the need for work groups to maintain i interfaces with offsite organizations, and communicate, coordinate, and cooperate with each other during activities in which interdepartmental coordination is necessary to assure plant and human performance .
- The need to keep personnel apprised of work status, the operational impact of work activities, and plant conditions that may affect work activities .
- The licensee plans work activities to support long-term equipment reliability by limiting temporary modifications, operator work-arounds, safety systems unavailability, and reliance on manual actions. Maintenance scheduling is more preventive than reactive.
- 4. Work Practices Personnel work practices support human performance. Specifically (as applicable):
WP H.4(a) The licensee communicates human error prevention techniques, such as holding pre-job Applicable to CC3. See Note 4.
briefings, self and peer checking, and proper documentation of activities. These techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. Personnel are fit for duty. In addition, personnel do not proceed in the face of uncertainty or unexpected circumstances.
WP H.4(b) The licensee defines and effectively communicates expectations regarding procedural Applicable to CC2 and CC3. See Note 4.
compliance and personnel follow procedures WP H.4(c) The licensee ensures supervisory and management oversight of work activities, including Applicable to RC1. See Note 4.
contractors, such that nuclear safety is supported.
- 5. Corrective Action Program The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance. Specifically (as applicable):
CAP P.I(a) The licensee implements a corrective action program with a low threshold for identifying No safety culture weakness determined issues. The licensee identifies such issues completely, accurately, and in a timely manner for any of the identified causes commensurate with their safety significance.
CAP P.I(b) The licensee periodically trends and assesses information from the CAP and other assessments No safety culture weakness determined in the aggregate to identify programmatic and common cause problems. The licensee for any of the identified causes communicates the results of the trending to applicable personnel.
CAP P.I(c) The licensee thoroughly evaluates problems such that the resolutions address causes and extent No safety culture weakness determined of conditions, as necessary. This includes properly classifying, prioritizing, and evaluating for for any of the identified causes operability and reportability conditions adverse to quality. This also includes, for significant problems, conducting effectiveness reviews of corrective actions to ensure that the problems are resolved.
CAP P.I(d) The licensee takes appropriate corrective actions to address safety issues and adverse trends in No safety culture weakness determined a timely manner, commensurate with their safety significance and complexity.
for any of the identified causes CAP P.I(c) If an alternative process (i.e. , a process for raising concerns that is an alternate to the licensee's No safety culture weakness determined corrective action program or line management) for raising safety concerns exists, then it results for any of the identified causes in appropriate and timely resolutions of identified problems.
Page 79 of 93
CR-PLP-2011-4822- Plant Trip During Panel Description ED-1l-2 Maintenance
- 6. Operating Experience The licensee uses operating experience (OE) information, including vendor recommendations and internally generated lessons learned, to support plant safety. Specifically (as applicable):
OE P.2(a) The licensee systematically collects, evaluates, and communicates to affected internal No safety culture weakness determined stakeholders in a timely manner relevant internal and external OE.
for any of the identified causes OE P.2(b) The licensee implements and institutionalizes OE through changes to station processes, No safety culture weakness determined procedures, equipment, and training programs.
for any of the identified causes
- 7. Self- and Independent Assessments The licensee conducts self- and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement. Specifically (as applicable):
SA P.3(a) The licensee conducts self-assessments at an appropriate frequency; such assessments are of No safety culture weakness determined sufficient depth, are comprehensive, are appropriately objective, and are self-critical. The for any of the identified causes licensee periodically assesses the effectiveness of oversight groups and programs such as CAP, and policies.
SA P.3(b) The licensee tracks and trends safety indicators which provide an accurate representation of No safety culture weakness determined performance.
for any of the identified causes SA P.3(c) The licensee coordinates and communicates results from assessments to affected personnel, No safety culture weakness determined and takes corrective actions to address issues commensurate with their significance.
for any of the identified causes
- 8. Environment For Raising An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation and Concerns employees are encouraged to raise such concerns. Specifically ( as applicable):
ERC S.I(a) Behaviors and interactions encourage free flow of information related to raising nuclear safety No safety culture weakness determined issues, differing professional opinions, and identifYing issues in the CAP and through self assessments. Such behaviors include supervisors responding to employee safety concerns in an for any of the identified causes open, honest, and non-defensive manner and providing complete, accurate, and forthright information to oversight, audit, and regulatory organizations. Past behaviors, actions, or interactions that may reasonably discourage the raising of such issues are actively mitigated.
As a result, personnel free ly and openly communicate in a clear manner conditions or behaviors, such as fitness for duty issues that may impact safety and personnel raise nuclear safety issues without fear of retaliation.
ERC S.l(b) If alternative processes (i.e., a process for raising concerns or reso lving differing profess ional No safety culture weakness determined opinions that are alternates to the licensee's corrective action program or line management) for raising safety concerns or resolving differing professional opinions exists, then they are for any of the identified causes communicated, accessible, have an option to raise issues in confidence, and are independent, in the sense that the program does not report to line management (i .e., those who would in the normal course of activities be responsible for addressing the issue raised).
Page 80 of 93
CR-PL P-2011-4822- Pla nt T r ip During P a nel Description ED-1l-2 Maintenance
- 9. Preventing, Detecting, a nd A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced in that:
Miti!!:atin!!: Perceptions of Retaliation PDR S.2(a) All personnel are effectively trained that harassment and retaliation for raising safety concerns No safety culture weakness determined is a violation oflaw and policy and wi ll not be tolerated for any of the identified causes PDR S.2(b) Claims of discrimination are investigated consistent with the content of the regulations No safety culture weakness determined regarding employee protection and any necessary corrective actions are taken in a timely manner, including actions to mitigate any potential chi lling effect on others due to the for any of the identified causes personnel action under investigation.
PDR S.2(c) The potential chilling effects of disciplinary actions and other potentially adverse personnel No safety culture weakness determined actions (e.g., reductions, outsourcing, and reorganizations) are considered and compensatory for any of the identified causes actions are taken when appropriate.
- 10. Accountabilitv Ma na!!:ement defines the line of a uthor ity and r esponsibility for nuclear safety. Specificallv as applicable):
ACC A.l(a) (a) Accountab ility is maintained for important safety decisions in that the system of rewards No safety culture weakness determined and sanctions is aligned with nuclear safety policies and reinforces behaviors and outcomes which reflect safety as an overriding priority.
for any of the identified causes ACC A. l (b) (b) Management reinforces safety standards and displays behaviors that reflect safety as an Applicable to RC1 and CC3. See Note 5.
overriding priority.
ACC A.l (c) (c) The workforce demonstrates a proper safety focus and reinforces safety principles among No safety culture weakness determined their peers.
for any of the identified causes
- 11. Continuous learn in!!: environment T he licensee ensures th at a learnin\! environment exists. Specifically (as applicable):
C LE C.2(a) (a) The licensee provides adequate training and knowledge transfer to all personnel on site to Applicable to RC1 and CC3. See Note 6.
ensure techn ical competency.
CLE C.2(b) (b) Personnel continuously strive to improve their knowledge, skills, and safety performance No safety culture weakness determined through activities such as benchmarking, being receptive to feedback, and setting perforruance for any of the identified causes goals. The licensee effectively communicates information learned from internal and external sources about industry and plant issues.
- 12. O rganization al change mana\!ement OCM 12. O rganization al Management uses a systematic p rocess for planning, coordinating, and evaluating the No safety culture weakness determined change ma nagement safety impacts of decisions related to major changes in organizational structures and for any of the identified causes fun ctions, leadership, policies, programs, procedures, and resources. Management effectively com municates such changes to affected personnel.
- 13. Safetv policies Safetv policies and related trai nin!!: establish a nd r einforce that nuclear safety is an overridin!!: prioritv in that:
SP SP.4(a) (a) These polic ies require and reinforce that individuals have the right and responsibility to No safety culture weakness determined raise nuclear safety issues through available means, including avenues outside their for any of the identified causes organizational chain of command and to external agencies, and obtain feedback on the resolution of such issues.
SP SP.4(b) (b) Personnel are effectively trained on these policies. No safety culture weakness determined for any of the identified causes Page 81 of 93
CR-PLP-2011-4822- Plant Trip During Panel Description ED-Il-2 Maintenance SP SP.4(c) (c) Organizational decisions and actions at all levels of the organization are consistent with the No safety culture weakness determined policies. Production, cost and schedule goals are developed, communicated, and implemented for any of the identified causes in a manner that reinforces the importance of nuclear safety.
SP SP.4(d) (d) Senior managers and corporate personnel periodically communicate and reinforce nuclear No safety culture weakness determined safety such that personnel understand that safety is of the highest priority.
for any of the identified causes Page 82 of 93
Attachment 7 Evaluation for Organization and Programmatic Issues Include this Worksheet as an Attachment to the report. The questions are provided to promote consideration of like symptoms, not to define a specific failure mode. O&P causal factors are symptoms of the more basic causes of the event and are typically an action or condition that shaped the outcome of the situation.
For each causal factor block checked YES:
I. Ensure it is appropriately represented in the WHY Staircase as a cause or contributor.
- 2. In the BARRIER ANALYSIS, tie the O&P causal factors as appropriate to Barriers that fai led, were weak, missing, or ineffective.
- 3. Summarize in the O&P section of the report how the identified Organizational & Programmatic weaknesses caused or contributed to the event and identify the Barrier which should have prevented it.
O&P W or ksheet Contributed to or Potential O & P Failure Modes (Causal Factors) RC# AC#CC#
Caused Event?
YES NO I) OPI X - Organization to Organization Interface Weaknesses
- Inadequate interface a mong O r ganizations (Good organizational structure but organizations don't communicate).
- Excessive or lack of overlap in fun ctions (Overall structural design results in overlaps or ho les between organizations) a) OP I A - Does there appear to be evidence of inadequate interface among organizations?
Problems in this area surface in the form of a high human error rate in tasks requiring communication among different organizations. Usually this is caused by a lack of interface formality (tailgate meetings, forma l N interface documentation or agreements, etc.), inadequate teamwork or trust among organizations, or inadequate physical settings.
b) OP I B - Is there evidence of excessive or lack of overlap functions between organizations?
Negative performance in this block is usually caused by a lack of organizational planning resulting in an N inadequate definition of job functions between one or more organizations.
c) OPI C - Is there evidence that the required notifications were not made when the job was begun, interrupted or completed? N Describes either the fai lure to perform the verbal communication of status when required by the process or the failure to design the process to require the verbal communication of status when successful implementation of the process depended upon this communication.
d) OPID - Is there evidence that appropriate personnel and departmental interactions were not fully considered when new processes were created during the imp lementation phases of the change? N Changes to processes created new requirements for interaction between personnel or departments that were not considered in the implementation phase of the change.
e) OPIE - Is there evidence that planning was not coordinated with inputs from walk-downs and task analysis? N Job plan did not incorporate information gathered during field vi sits or task analysis concerning the steps and conditions required for successful comp letion of the task.
f) OP IF - Is there evidence that planning was not coordinated with all departments involved in task Interdepartmental communication and teamwork were not supported by the planned work flow. N
- 2) OP2X - Organization to Progra m Interface Weaknesses
- Lack of commitment to pr ogr am implementation (organization never gets program off the ground)
- Inadequ ate Program m onitoring or management (organization does not monitor or manage the program effectively)
- Lack of progr a m evaluation process (program survives but Organization does not evaluate program, so it goes in the wrong direction)
Lack of or ganizational a uthority for progr am implementation (organization starves to death because no one is protecting it) a) OP2A - Is there evidence of a lack of commitment to program implementation?
Usually evidenced by slow program implementation. The fai lure is generally due to excessive program y implementation requirements or a lack of management support of the program. RC11 CC4 b) OP2B - Is there evidence of inadequate program monitoring or inadequate management skills?
Indicated by a lack of program improvement over time. Usually it is caused by inadequate staffing or y RC11 LOW inadequate management skills.
Page 83 of 93
O&P Worksheet Contributed to or Potential O&P Failure Modes (Causal Factors) RC# AC#CC#
Caused Event?
YES NO c) OP2C - Is there evidence of a lack of a program evaluation process?
This area is more reactive, in that a program fai lure occurs before action is taken. However, the same items N contribute to negative performance, i.e., inadequate management practices, inadequate staffing for program implementation, or insufficient program design.
d) OP2D - Is there evidence of a lack of organizational authority for program implementation?
This code usually is associated w ith an insufficient budget for the program or fragmented responsibility y ReI and/or accountability for the program. Potential causes include a lack of organizational planning or a lack of management commitment to program implementation.
e) OP2E -Is there evidence of unclear or complex wording or grammar in program implementation documents? N Wording, grammar or symbols fai l to clearly and concisely specifY the required action; instructions provided for team of users fail to specifY roles of each user.
f) OP2F - Is there evidence of an omission of relevant information in program implementation y CCI, CC4, EOC documents that would have prevented an event from occurring (e.g. insufficient information in graphs, tables or illustration; lack of instructions or data sheet documentation requirements, etc.)
Over reliance on user's training, skill s or experience; lack of detail for infrequent, complex, crucial or error-prone tasks; insufficient information in graphs, tables or illustrations; lack of instructions for data sheet documentation requirements g) OP2G - Is there evidence of the lack of a procedure that should have been written but does not exist? N The process meets administrative requirements for having a procedure, but no procedure has been written.
h) OP2H - Is there evidence that policy guidance or management expectations were not well defined or understood by personnel involved in performing the task?
Personnel exhibited a lack of understanding of existing policy and/or expectations, or policy/expectations y
were not defined.
i) OP21- Is there evidence that job standards were not adequately defined or communicated?
Measurement of effectiveness cou ld not be performed for specific job functions due to lack of defined N standards.
j) OP21 - Is there evidence that personnel exhibited insufficient awareness of the impact of actions on safety and reliability?
Management failed to provide direction regarding safeguards against non-conservative actions by personnel y
concerning nuclear safety or reliability k) OP2K - Is there evidence that management follow-up or monitoring of activities was ineffective y in identifYing shortcomings in implementation?
I) Management's methods for monitoring the success of initiatives were ineffective in identifYing shortcomings in the implementation.
m) OP2L - Is there evidence that causes of a previous event or known problem were not determined? N Analysis methods fail ed to uncover the causal factors of consequential or non-consequential events.
n) OP2M - Is there evidence that the effects of changes on planned schedules were not adequately addressed prior to implementation? N Changes to processes which resulted in scheduled changes had effects on personnel or equipment that were not addressed in the change implementation.
- 0) OP2N - Is there evidence that the job scoping process did not properly identifY potential task interruptions or environment stress? N Work scoping process was not effective in detecting reasonable obstructions to work flow (e.g., shift changes) or the impact of environmental conditions.
p) OP20 - Is there evidence that the job scoping process did not identifY special circumstances or conditions that may be impacted or dependent on other circumstances or conditions? N Work scoping process was not effective in detecting work process elements having a dependency upon other circumstances or conditions.
q) OP2P - Is there evidence that the field walk down input to design was less than adequate?
Design change and/or field change requests as a result of inadequate field walk downs to verifY actual N configurations of plant components, structures and systems that interface with or affect the designs.
- 3) OP3X - Program to Program Interface Weaknesses
- Lack of interface requirements (no formal procedures to make sure two programs talk to each other)
- Conflicting program requirements (conflicting actions required by two different programs)
- Inadequate interface requirements (information required is available but program interfaces are inadequate to get it)
Page 84 of 93
O&P Worksheet Contributed to or Potential O&P Failure Modes (Causal Factors) RC# AC#CC#
Caused Event?
YES NO a) OP3A - Is there evidence of a lack of interface requirements between two or more programs that are required to interface in that details necessary to ensure a consistent standard are not adequately covered in programmatic implementing documents?
N Usually this is caused by inadequate program design or an inadequate work planning process.
b) OP3B - Is there evidence of conflicting program requirements where one program has different actions from another program for the same issue? N This codes is used when different actions are required by two or more programs for the same situation. As a result, staff efficiency and accountability is negatively impacted.
c) OP3C -Is there evidence of inadequate interface requirements in that one program specifies actions different from another program for the same issue? N This code is used when actions are required by one program belonging to another program that is inadequate in perform the actions. The cause is usually inadequate program design and/or inadequate work planning processes
- 4) OP4X - Programmatic Deficiencies
- Insufficient d etail (This is my first time doing this, how am I supposed to know what "use normal process" means?)
- Inadequate scope ("The procedure left out all the information on the electrical cables that need to be connected")
- Excessive implementation requirements (so many requirements that people give up and don ' t try to follow the procedure)
- Inadequate verification process ("We haven ' t really looked at our processes and given them a
'check up ' for over 15 years")
a) OP4A - Is there evidence that there are insufficient details in a procedure to perform the task?
When a program is vague regarding what is required in a particular situation, it is usually indicative of an y CC2 inadequate program design or insufficient feedback for individuals using the procedure.
b) OP4B - Is there evidence of inadequate job scope (omission of necessary functions) in an implementing procedure because of an inadequate program design or inadequate feedback from the field?
N Either inadequate program design or inadequate feedback from the field is usually taking place c) OP4C - Is there evidence of excessive implementation requirements that result in portions of the program being ignored by the staff due to overload?
N This can be caused by inadequate program design, lack of work prioritization, or inadequate staffing d) OP4D - Is there evidence of an inadequate verification process (single human error, high program failure rate, poor procedure quality or inadequate program design?
Program breakdown by a single human error; high program failure rate, poor procedure quality. Inadequate N program design e) OP4E - Is there evidence that there is a lack of responsibility by personnel because it is not well defined or personnel are not being held accountable? N Responsibility for process elements (procedures, engineering, training, etc.) was not placed with individuals or accountability for failures of those process elements was not placed with individuals.
f) OP4F - Is there evidence that a response to a known or repetitive problem was untimely?
Corrective action for known or recurring problems was not performed at or within the proper time. N g) OP4G - Is there evidence that needed changes to the plant were not approved or funded which resulted in a plant issue?
N Corrective actions for existing deficiencies that were previously identified were not approved or funded.
h) OP4H - Is there evidence that there was not a means or process to ensure procedures and documents were of adequate quality and up to date? N A process for changing procedures or other work documents to ensure quali ty and timeliness was nonexistent or inadequate i) OP4I - Is there evidence that duties were not well distributed among personnel that contributed to a problem?
N The work loading of individuals within a group or team did not adequately address training, experience, task frequency and duration, or other situational factors such that responsibility was inappropriately distributed.
j) OP4J - Is there evidence that too few workers are assigned to perform a task that contributed to an issue?
N Job planning did not allot a realistic number of man-hours based on the scope of work described .
Page 85 of 93
O&P Worksheet Contributed to or Potential O&P Failure Modes (Causal Factors) RC# AC#CC#
Caused Event?
YES NO k) OP4K - Is there evidence that an insufficient number of training or experienced workers were assigned to a task?
Though the overall number of personnel assigned matched the planned man-hour allotment, organization N methods fai led to identifY that personnel assigned did not have adequate experience or training to perform the work.
- 1) OP4L - Is there evidence that there is a problem in perform repetitive tasks and sub tasks which contributed to a problem?
Work flow plan repeated tasks or sub tasks to the detriment of successful completion of the evolution. N m) OP4M - Is there evidence that there was a less than adequate process for a configuration change to a design document? y CCI Documentation generated as a result of a design change which renders the as-left configuration of affected components, structures and systems indeterminate.
n) OP4N - Is there evidence that personnel exhibited insufficient awareness of the impact of actions y Nuclear Culture on safety reliability because management failed to provide direction regarding safeguards against Safety Assessment non-conservative actions by personnel concerning nuclear safety or reliability?
Management failed to provide direction regarding safeguards against non-conservative actions by personnel concerning nuclear safety or reliability.
- 0) 0P40 - Is there evidence that the planning process was not coordinated with inputs from walk downs and task analysis?
N Job plan did not incorporate information gathered during field visits or task analysis concerning the steps and conditions required for successful completion of the task.
p) OP4P - Is there evidence that previous industry or in-house operating experience was not effectively used to prevent problems and an event occurred because the information was not N
properly assimi lated by the organization (missed opportunity)?
Industry or in-house experience relating to a current problem existed previous to the problem, but was not assimilated by the organization (missed opportunity).
- 5) OP5X - O rganizational Weaknesses
- Inadequate function or st ructure (poor internal organizational design that is missing vital functions)
- Inadequate attentio n to emerging problems (organization doesn't pay attention to what is happening within it)
- Inadequate work prioritization (organization doesn ' t prioritize their workloads so they waste time on unimportant things)
- Inadequate communication within the organizatio n (commun ication does not get up and down the organizational ladder)
- Inadequate job skills, work practices, or decision making (organization had a problem because of its people) a) OP5A - Is there evidence of inadequate functions or structure which results in work not being performed due to a lack of organizational planning or inadequate staffing? y RC1 Specific issues or work are not performed or addressed, usually due to a lack of organizational planning or inadequate staffing.
b) OP5B - Is there evidence of inadequate attention to emerging problems? N Repetitive organizational crises in morale, work practice, or repeat events, etc. The causes are usually associated with a lack of strong self assessment, strategic planning, and root cause processes. Additionally, inadequate vertical information flow to the decision makers in the organization and inadequate prioritization of work can contribute to a breakdown in this area.
c) OP5C - Is there evidence of an inadequate work prioritization process?
Normally associated with staff work overload, over-run of the committed budget, and increasing backlog of work items. It is usually caused by inadequate work prioritization and inadequate vertical communication of y RClI CC4 the organization=s missions and goals.
d) OP5D - Is there evidence of inadequate communication within the organization?
Important issues are not being addressed or a breakdown of normal work processes has occurred. Low staff morale is usually the long term result. Common causes fo r this include an inadequate information flow path, y lack of a teamwork type of culture, or inadequate physical settings RCI e) OP5E - Is there evidence of inadequate job skills, work practices or decision making? N Generally evidenced by low morale and excessive human error rates. Causes usually include a punitive management style, inadequate supervision, training, staff qualification, or vertical communication, and conflicting or unreasonable organizational goals .
Page 86 of 93
O&P Worksheet Contributed to or Potential O&P Failure Modes (Causal Factors) RC# AC#CC#
Caused Event?
YES NO f) OP5F - Is there evidence that corrective actions for previously identified problems or event was N not adequate to prevent recurrence (fai led to take meaningful corrective actions for consequential or non-consequential events)?
Management fai led to take meaningful corrective action for consequential or non-consequential events.
g) OP5G - Is there evidence the supervisor was not properly notified of a suspected problem? N A problem requiring verbal communication with supervision arose, but was not verbally communicated to the supervisor.
h) OP5H - Is there evidence of that pertinent information is not being properly transmitted verbally between the transmitter and the listener and vice versa? N The sender failed to verbally transmit information to the listener. This requires a sender and listener to be present, and is regardless of either individual ' s position in management chain.
i) OP51 - ls there evidence that there are too many administrative duties assigned to supervisory N staff to properly perform supervisory activities?
The administrative load on immediate supervisors adversely affected their ability to supervise ongoing activities j) OP5J - Is there evidence that there is insufficient supervisory resources to provide the needed N supervision to plant personnel?
Supervisions resource is less than that required by task analysis considering the balance of procedures, supervision and training.
k) OP5K - Is there evidence that there is insufficient manpower to support the identified goals and N objectives of the plant?
Personnel are not available as required task analysis of goal/objective.
- 1) OP5L - Is there evidence that sufficient resources are not provided to ensure adequate training is N provided and maintained?
Training resources are not available as required by task analysis.
m) OP5M - Is there evidence that there is not adequate availability of appropriate materials and tools N to do the job?
A process for supplying personnel with appropriate materials or tools did not exist.
n) OP5N - Is there evidence that there is not a means provided for ensuring adequate equipment and N quality/reliability/operability for personnel equipment?
A process for ensuring personnel's equipment was satisfactory did not exist.
- 0) OP50 - Is there evidence that personnel selection did not ensure an appropriate match to ensure N a motivation for the worker?
Personnel selection processes fai led to determine a mismatch between motivation and job description prior to task.
p) OP5P - Is there evidence that tasks and individual accountabi lity were not made clear to the worker?
y Tasks (and the individua l accountability for the task) that were outside written guidance or training were not made clear to the worker.
q) OP5Q - Is there evidence that the progress and status of task is not adequately tracked by supervision? y RCI Supervision did not take the appropriate actions to monitor the task progress or status.
r) OP5R - Is there evidence that there is not an appropriate level of in-task supervision planned N prior to the task being performed?
Supervision did not adequately assess the task for points of supervisory interaction prior to assignment to workers.
s) OP5S - Is there evidence that direct supervisory involvement in the task interfered with the N overview role of supervision?
Supervision became so involved with the actual task steps that overall command and control were adversely affected t) OP5T - Is there evidence that emphasis on the schedule had an impact on doing a quality job and accepted standards were not met as a result of this emphasis? y RCI Accepted standards for methods were not met due to supervision's focus on comp leting the activity within a certain time frame.
u) OP5U - Is there evidence that job performance and self checking standards were not properly N Covered by OP5P communicated to the organization performing the work prior to the job being performed?
Supervision failed to adequately communicate how standards for job performance and self-checking could be applied to the actual job at hand Page 87 of 93
O&P Worksheet Contributed to or Potential O&P Failure Modes (Causal Factors) RC# AC#CC#
Caused Event?
YES NO v) OP5V - Is there evidence that too many concurrent tasks were assigned to the worker that were N beyond the individual's abilities?
Supervision fai led to detect that concurrent job assignments for an individual exceeded the individual's abilities.
w) OP5W - Is there evidence that there is frequent job or task shuffling without adequate time to N shift attention away from the previous task?
Supervision transferred a worker from one task to another without adequate time to shift attention away fro m previous task.
x) OP5X - Is there evidence that supervision did not consider the worker's need to use a higher N order of skills that consider the workers talents and strengths?
Supervision did not consider the worker's talents or innovative strengths that could be used to perform more challenging work .
y) OP5Y - Is there evidence that worker assignments did not consider the worker's previous task? N Supervision did not adequately assess the previous task's impact upon the worker's ability to implement the current task.
z) OP5Z - Is there evidence that a workers assignment did not consider the worker's ingrained work N patterns and necessary work patterns for successful completion of the current task?
Supervision failed to assess the incompatibility between worker's ingrained work patterns and necessary work patterns for successful completion of the current task.
aa) OP5AA - Is there evidence that there is too an infrequent contact with the workers to detect work N habit and attitude changes?
Supervision not aware of deviation from desired work habits/attitudes due to lack of interaction with personnel.
bb) OP5AB - Is there evidence that supervision provides feedback on negative performance of an N individual but not on positive performance?
Worker's performance adversely affected by supervision's focus on negative performance feedback.
cc) OP5AC - Is there evidence of a lack of teamwork as a result of inadequate training content? N Training content did not adequately address actions individuals must take in order for the crew or team as a whole to be successful.
dd) OP5AD - Is there evidence of a lack of evaluation of risk and consequences prior to making a change that would have an adverse impact as a result of the change?
Elements of the process change were not recognized as having adverse impact or increased risk of adverse y RCI impact prior to implementing the change.
ee) OP5AE - Is there evidence that personnel exhibited insufficient awareness of the impact of actions on safety and reliability? y RCI Management failed to provide direction regarding safeguards against non-conservative actions by personnel concerning nuclear safety or reliability.
ff) OP5AF - Is there evidence that causes of a previous event or known problem were not N determined?
Analysis methods fa iled to uncover the causal factors of consequential or non-consequential events.
gg) OP5AG - Is there evidence that a response to a known or repetitive problem was untimely? N Corrective action for known or recurring problems was not performed at or within the proper time.
hh) OP5AH - Is there evidence that needed changes were not approved or funded that resulted in a N plant problem?
Corrective actions for existing deficiencies that were previously identified were not approved or funded .
ii) OP5AI - Is there evidence that a means was not provided to ensure procedures and documents N are of adequate quality and up to date?
A process for changing procedures or other work documents to ensure quality and timeliness was nonexistent or inadequate.
jj) OP5AJ - Is there evidence that planning was not coordinated with inputs from walk downs and N task analysis?
Job plan did not incorporate information gathered during field visits or task analysis concerning the steps and conditions required for successful completion of the task.
Page 88 of 93
- Maintenance Crew Statements Statement #1 Date: October 5th, 2011 Event Date: September 25th, 2011 To Whom It May Concern:
I would like to start this statement by stating the fact that there have been 10 day's since the event and in that waiting period there have been lot's of speculation , misleading information , rumors , and emotional feeling 's that have now affected site personnel's thoughts as well as mine. I do feel that this 10 day waiting period has been too long to clearly recall the exact event without some slight influence from conversation's and meetings on what had transpired that day. Therefore this is how the day of the event unfolded to the best of my knowledge.
Started Sunday morning 9-25-11 with turnover from nightshift on temp mod package for 72-127 we had to wait until the supervisor arrived , after supervisor arrived a prejob brief was conducted , the brief went over work instructions, critical steps, human performance tools to be used, safety concerns and was the pre job brief form that the previous shift had filled out. We went over everything that was on the pre job brief form. The brief was held with the supervisor, engineering and acting maintenance manager present, there may have been others there but I can't recall and then had a briefing with ops. The temp mod was implemented and performed without incident.
After lunch on 9-25-11 a prejob brief was performed on the three work packages for inspect and replace breakers 72-119, 72-120, 72-121 and 72-123 , the prejob brief was conducted with repairmen, engineering, electrical supervisor, acting maintenance manager, the duty station manager and the NRC present. The briefing was conducted discussing detailed work instructions , safety concerns and the human performance tools to be used, however the brief was performed from memory without using a prejob briefing checklist. After the briefing everyone went out to the ED-il panel, we conducted a job site review with our yellow cards. Performed live dead live on breakers that we needed to remove sat. I determed breaker 72-120 and removed breaker, then engineering and supervisor and NRC inspected bus. I then removed breaker 72-i 19 and engineering and supervisor and NRC inspected cubicle and bus. I then removed 72-121 and engineering and supervisor and NRC inspected cubicle and bus. I then removed 72-123 and engineering and supervisor and NRC inspected cubicle and bus. 72-119 top bus bar had thread damage and the work instruction step 4.7 .7 stated as needed clean and tighten bus bar connections including chasing threads and replacing fasteners , we discussed with supervisor, acting maintenance manager and engineering how to chase threads on the live bus and we were told that our work instruction step allowed us to remove bus bar stab off of the live bus because we didn't want to chase threads on a live bus and the reasons were
- 1 the metal shavings could short out the bus #2 we don't have an insulated thread chaser. It was discussed by repairmen how to remove the bus bar and retain positive control of the Philips head bus bar retaining screw so that it wouldn't fall into the bus , the decision was to hold the screw with an insulated screw driver and the bus bar to be held Page 89 of 93
with an insulated gloved hand. I was on the left side of when he was in position to remove bus bar, he had the insulated screw driver on the retaining screw and his insulated gloved hand on the bus bar and with positive pressure on the screw to loosen, he began to loosen the screw when I saw an arc near the retaining screw and so did , he reacted and pulled his hands out of the panel with the thought that it was blowing up in his face ,
he was trying to protect his face and possibly his life, that is when the bus bars came in contact with each other and shorted out.
Statement #2 7:00AM 9-25-2011 Received a detailed turnover from C-shift of work to be performed, events that led up to this point.
Supervisor was scheduled to be in at 8:00AM so we waited until he arrived to brief the day's work.
Tried with help of engineering's help to find some kind of detail on panel construction, were not able to find any. Held brief with 3 electricians, supervisor and engineering on work for the day. Temp mod, breakers to be removed, engineering 's role , etc. Installed temp mod, finished before lunch.
Lunch break Briefly talked about breaker removal and the points at which engineering would step in to evaluate and take pics etc.
Removed 72-120 breaker, Eng. evaluated and took pics. Noted gap between the copper bus bar and breaker. With approval from supervisors and engineering checked breaker mounting screw to see if it was loose, it was not. Removed 72-119 breaker noted the threads to the copper bar in question needed to be chased. This isthe point when it was decided that the bar needed to be removed and taken to shop for repairs.
We continued removing removing breakers 72-121 and 72-123 at the request of our supervisor. As with the other 2 breakers engineering did there evaluating as we removed them.
At this point we (electricians) dicussed how to to safely remove piece of busbar decided on the method we used. I then donned my low voltage gloves and insulated screwdriver and when I started to loosen screw is when I saw arcing and reacted.
Statement #3 Statement of events on Sunday September 25th 2011 I arrived to work at 08:00 on Sunday, and received a face to face turnover with electrical maintenance superintendent, It was discussed that we would first install a temp mod to provide alternate power to the generator voltage regulator and then the sequence of the breaker removal and install of he the DC breakers located in panel 0-11-2 was discussed including the importance of insulating the positive and negative buss bar standoffs to prevent Page 90 of 93
an electrical short. The repairmen were briefed and ready to install the temp mod to provide alternate power to breaker 72-121 which provides control power to the generator voltage regulator. I then reviewed the TM to have an understanding of the work involved and had a short informal brief with the three repairmen to ensure we all understood the TM installation and what the repairmen's rolls and responsibilities would be. A briefing was then conducted in the control room with the shift manager, control room supervisor and the shift engineer to discuss our work and to establish communications with the control room in the event of an issue occurring during the work. The critical steps discussed were validating that the polarity of the DC power was the same between the temp power and the power currently feeding the voltage regulator. After the brief in the control room work commenced at the temp mod was installed successfully at 11 :00. At approximately 13:00, planning had completed the three work orders to remove and inspect the four breakers in panel 011-2. A pre-job brief was held with electrical maintenance, engineering, the acting maintenance manager, duty station manager, and the NRC to discuss the sequence of the work, safety and human performance tools to be used. It was discussed "what is the worst thing that can happen" and we discussed a short circuit event and that insulating the buss connection stabs would be the method to prevent this. After the completion of the brief, I asked around the table if anyone had any questions or concerns and no one had further input. Work shortly commenced at the 0-11-2 panel after an additional brief with the control room and the work orders were taken to working. The sequence of events was to first remove breaker 72-120 and to inspect the connection points between the line side stabs of breaker 72-119 and the positive and negative bus bars and then to continue down the panel and remove breaker 72-121 and then 72-123. Upon the removal of 72-120 and a 1/16 of an inch air gap was found between the positive line stab of 72-119 and the positive buss bar stand off, this was due to a cross threaded connection screw. Breakers 72-121 and 72-123 were then removed with no additional issues found. After discussion with engineering and the repairman the decision was made to remove the positive and negative standoffs to repair the threads on the positive connection stap and then to swap the positive and negative standoffs so that breaker 72- 119 would not have a repaired mounting hole. This meant breaker 72-120 would now have the repaired hole at the connection between the negative line side stab of the breaker, this was determined because 72-120 is a spare breaker. The work instruction note stated to support or insulate the buss bar stand offs to prevent an electrical short. While it was discussed in previous briefings that insulation would be used two repairmen felt they could retain positive control of the standoff and remove it safely. This action was not discussed with me prior to removing the positive bus bar. Standing behind the work boundary I was observing _ _
remove the positive bus bar when he lost control and the short occurred. At this point the plant tripped, shortly after we were asked to remove the fault from the panel 011-2 so operations could attempt to restore DC power. After the event conducted a fatigue assessment and I then escorted to the emergency room for fitness for dutytesting.
Statement #4 Plant tripped 9-25-11 statement date 10-5-11 Prejob brief attendance maintenance crew, supervisor, NRC, duty station manager, asst maint mgr, and two design engineers. talked about the job and what we were going to do, we removed 72-119 and found a cross threaded screw it was thought to be the smoking gun everybody that was in the brief came a crossed Page 91 of 93
the boundary to see even the shift manger, this went on for each breaker that was removed 4 total we didn't have any new buss stabs so we were going to swap the spare breaker stabs which had good holes with the breaker with the bad holes. The job was going well and the NRC left we looked at the work plan and it gave us direction to support the stab and remove it, the hot stab was then supported with a tested voltage gloves and the screw loosened an arc was seen and the stab was let go of, letting the positive stab fall into negative stab at no time did any body see the danger before us or we wouldn't have done it. My role in this job was handing tools to the teem handing tape anything that would make the job go as expected Page 92 of 93
, "Why Staircase" WHY STAIRCASE Problem statement: On September 25, 2011, at 1506, while performing maintenance activities on DC electric distribution panel 0-11-2, a short circuit condition occurred resulting in reactor trip .
Why? Reactor tripped due to loss of power to two-of-four RPS channels Why? Loss of left train DC Bus results loss of power to channels A & C Why?1 Shunt trip breaker 72-01 opened Why?1 Latent Design Issue Why?2 0-11-2 short resulted in Bus trip Why?2 Loose stab dropped causing short Why?2 No insulation protection for bus Why?2 Workers decided insulation was not needed Why?2 Workers and supervision did not stop and question Why?2.1 Workers rationalized "skill of craft" Why?2.1 Workers did not recognize risk to themselves or to the plant Why?2.1 Common practice to work on energized equipment Why?2.1 Work on energized equipment considered low risk Why?2.2 Supervision did not recognize risk and ensure insulation was in place Why?2.2 Work not characterized as risk to workers or plant Why?2.2 Plant EOOS assessment of risk incomplete Why?2.2 EOOS result not questioned by management Why?2.2 Management not always recognizing or managing risk Why?2.2 Management not sensitive to risk Why?2.2 Senior management has not ensured the plant culture is risk sensitive Page 93 of 93
Attachment 4 SOP Assessment of DC Panel 011-2 Fault EA-PSA-SOP-O 11-2-11-07, Revision 1 227 Pages Follow
EA-PSA-SOP-011-2-11-07 Revision: 1 Date: 01/05/2012 Entergy Number of Pages: 152
Title:
SOP Assessment of OC Panel 011-2 Fault Approval: See signature page.
Purpose This engineering analysis assesses the significance of the dc panel fault and subsequent plant trip that occurred on 09/25/2011 . Inadequate maintenance work instructions led to a short within dc panel ED 2. Contrary to intended design, the fuse between ED-11-2 and dc bus ED-1 OLlED-1 OR failed to provide adequate protection and did not isolate the panel from the bus. This resulted in the loss of ED-1 OLlED-10R and subsequent plant trip .
Conclusion Based on reviews of the event timeline , plant design and response , operator responses , plant-specific thermal-hydraulic analyses, potential human errors and logic model quantification, the following conclusions were reached:
- Plant risk during the event increased. The increase in the conditional core damage probability given the dc panel ED-11-2 fault and subsequent plant trip is evaluated to be 4.3E-6, and is considered WHITE.
- The risk increase is driven by scenarios in which the lost train of dc power is not recovered . When combined with other failures , this could result in a loss of secondary side cooling via the steam generators, failure to refill the condensate storage tank to provide long term cooling, failure to cool down and transition to shutdown cooling , and the failure of once-through-cooling as a last resort for decay heat removal, and ultimately core damage.
- The risk increase is also comprised of scenarios in which charging pumps are not isolated in time to prevent a challenge to pressurizer safety relief valves, resulting in a potential loss of coolant accident if one or more relief valves sticks open. Failures to mitigate this consequential event can then lead to core damage.
- A stuck open pressurizer safety relief valve is classified as an above core, vapor space LOCA.
For these scenarios, as long as secondary side cooling is available for decay heat removal the transient does not necessarily require high pressure safety injection to preclude core damage. If auxiliary feedwater remains available, the core survives the initial blowdown and inventory
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 Page 2 of 36 Rev. 1 makeup from charging is sufficient to maintain primary coolant system inventory and preclude core damage. Long term heat removal via the steam generators (or transition to shutdown cooling) then becomes a success path, even when a SRV sticks open - provided a nominal level of inventory makeup is available (e.g., via charging with SIRWT inventory conserved by terminating sprays, or via HPSI in recirculation mode once SIRWT inventory is depleted) .
- Realistic and justifiable human error probabilities were used for fault-related recoveries . Use of conservative human error probabilities increases the conditional core damage probability. The increase in delta conditional core damage probability is 6.0E-06 for the event, and is still considered WHITE.
- Steam generator overfill was precluded during this event by isolating steam to the turbine driven auxiliary feedwater pump and limiting flow from AFW pump P-8C via flow control valves. Failure to do so could have resulted in steam generator overfill and the loss of the turbine driven auxiliary feedwater pump. The failure to restore the pump if needed was considered and did not contribute significantly to the risk.
Note: This engineering analysis is not a 10 CFR §50.2 design basis analysis and the results and conclusions of this analysis do not supersede those of any design basis analyses of record. The biases and degree of conservatism embodied in the methods, inputs and assumptions of this analysis may not be appropriate to support all plant activities. An appropriate level of engineering rigor commensurate with the safety significance of the topic under consideration is ensured in this analysis by conformance with all applicable Entergy procedures.
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 Page 3 of 36 Rev. 1 Table of Contents 1.0 PURPOSE .......................................................................................................................................... 4
2.0 CONCLUSION
................................................................................................................................... 4
3.0 BACKGROUND
................................................................................................................................. 5 3.1 Event Summary ......... ............. ........ .. .... ........ .... ........... .. ........ ....... .. .... .... ................. ............... ........ 5 3.2 Maintenance Initiating Event Summary .. ........ .. .. ........ .......... .. ........... ...... .......... .... ...... ................... 5 3.3 Latent Coordination Issue Summary .... .. ........ .. ........ .... .. .. .......... .. .. ........ .... .. .............. ...... .... .......... 6 3.4 Evaluation Context. ..... ...... ........ ......... ....... ..... ...... .......... .. ......... ..... .. .. .... ....... ......... .... ... .. .... ... ......... 6 3.5 Key Factors Impacting Plant Response .. .. .... .......................................... ........ .......... ..................... 7 4.0 INPUT ................................................................................................................................................. 9 4.1 PRA Tools and Models Input.. .................................. .. ........ .. ............ ... ... .... ...... .......... .... .. .............. 9 4.2 Plant Configuration Input .. .. .................... .................. .. ................................................................. . 11 4.3 Plant Design and Operation Event-Specific Input .. .. ............ ...... ............ .. .... ...................... .... .... .. 11 5.0 ASSUMPTIONS ............................................................................................................................... 15 5.1 Major Assumptions ...................... .. ...... ...... .. .. ................... .... .............. .... .. ...... ...... .. ... ........ ..... ... ... 15 5.2 Minor Assumptions .. .... ...................... ......... ........ ........ .. ...... .. ........ .. .................... .. .. ....... .. ....... ... .. . 17 6.0 METHODOLOGy ............................................................................................................................. 18 6.1 Thermal-Hydraulic Model .... .. .. ...................... .. ...... ...... ......... ...... .... .............. .. .. ..... ............ ...... .. .. . 18 6.2 Logic Model .... ................. ....... .... ... .. .... ........... .............. ... ... ..... .... .... .. ......... .......... .... ... .. ...... ... ...... 18 6.3 Human Error Probabilities ............................................ .... ............... ........ ........ .. ........ .. .......... .. ..... 20 6.4 Pressurizer Safety Relief Valve Failure Probability .... ..... ........... .... ............ .......... .... .. ... ...... ......... 27 6.5 Significance Determination Color Criteria ...... ............................................ ........ .......... .............. .. 28 7.0 ANALYSIS ....................................................................................................................................... 29 7.1 Evaluation of Increased Plant Risk .................... .... ......... .... .... .. ................................. .. ................. 29 7.2 Sensitivity Studies ..... .......... ... .. .... ........... .... ....... ... ... .... ................ ... .. ...... ....... ..... ..... ... .... ....... .... ... 33
8.0 REFERENCES
................................................................................................................................. 35 9.0 ATTACHMENTS .............................................................................................................................. 36
~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 I Page 4 of 36 Rev. 1 1.0 PURPOSE This engineering analysis assesses the significance of the dc panel fault and subsequent plant trip that occurred on 09/25/2011 . Inadequate maintenance work instructions led to a short within dc panel ED 2. Contrary to intended design , the fuse between ED-11-2 and dc bus ED-1 OLlED-1 OR failed to provide adequate protection and did not isolate the panel from the bus. This resulted in the loss of ED-10LlED-1OR and subsequent plant trip.
Specifically, this analysis evaluates the conditional core damage probability given the fault event, fault propagation , impacted components, and potential recoveries . The conditional core damage probability includes consideration of additional random component failures and recovery actions that might have been unsuccessful. This analysis addresses the dc panel ED-11 -2 fault. Only the single (internal) initiating event under the conditions that occurred is evaluated . This analysis does not address accident initiators from other internal events, internal flooding , or external events (high winds , tornadoes , internal fires , etc).
2.0 CONCLUSION
Based on reviews of the event timeline , plant design and response , operator responses , plant-specific thermal-hydraulic analyses, potential human errors and logic model quantification , the following conclusions were reached :
- Plant risk during the event increased . The increase in the conditional core damage probability given the dc panel ED-11-2 fault and subsequent plant trip is evaluated to be 4.3E-6, and is considered WHITE.
- The risk increase is driven by scenarios in which the lost train of dc power is not recovered . When combined with other fa ilures, this could result in a loss of secondary side cooling via the steam generators, failure to refill the condensate storage tank to provide long term cooling , failure to cool down and transition to shutdown cooling , and the failure of once-through-cooling as a last resort for decay heat removal , and ultimately core damage.
- The risk increase is also comprised of scenarios in which charging pumps are not isolated in time to prevent a challenge to pressurizer safety relief valves , resulting in a potential loss of coolant accident if one or more relief valves sticks open. Failures to mitigate this consequential event can then lead to core damage.
- A stuck open pressurizer safety relief valve is classified as an above core, vapor space LOCA.
For these scenarios, as long as secondary side cooling is available for decay heat removal the transient does not necessarily require high pressure safety injection to preclude core damage. If
. auxiliary feedwater remains available , the core survives the initial blowdown and inventory makeup from charging is sufficient to maintain primary coolant system inventory and preclude core damage . Long term heat removal via the steam generators (or transition to shutdown cooling) then becomes a success path , even when a SRV sticks open - provided a nominal level of inventory makeup is available (e.g., via charging with SIRWT inventory conserved by terminating sprays, or via HPSI in recirculation mode once SIRWT inventory is depleted).
- Realistic and justifiable human error probabilities were used for fault-related recoveries . Use of conservative human error probabilities increases the conditional core damage probability. The increase in delta conditional core damage probability is 6.0E-06 for the event, and is still considered WHITE.
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 I Page 5 of 36 Rev. 1
- Steam generator overfill was precluded during this event by isolating steam to the turbine driven auxiliary feedwater pump and limiting flow from AFW pump P-8C via flow control valves. Failure to do so could have resulted in steam generator overfill and the loss of the turbine driven auxiliary feedwater pump. The failure to restore the pump if needed was considered and did not contribute significantly to the risk.
3.0 BACKGROUND
3.1 Event Summary On 09/25/2011 , Palisades experienced an automatic reactor trip due to loss of power to 2 of 4 reactor protection system channels due to loss of power to preferred ac buses EY-10 and EY-30. Loss of power to dc bus EO-10LlEO-10R and consequently preferred ac buses EY-10 and EY-30 was the result of maintenance activities in dc panel EO-11-2. The maintenance activities caused a short in EO-11-2 and actuation of shunt trip breaker 72-01 on over-current protection. The consequence of these events was loss of power to dc buses EO-10L and EO-1 OR and loss of power from preferred ac buses EY-10 and EY-30.
No actual safety consequences resulted from this event. System response was as expected given a loss of one train of dc power. Right channel safety injection initiated immediately. Left channel safety injection initiated when EY-30 was placed on the bypass regulator. High and low pressure safety injection operated but did not inject since primary coolant system pressure remained above shutoff head. The opposite train of dc power remained available throughout the event.
The significant grounding event on dc panel EO-11-2 disclosed a latent coordination issue: the shunt trip breaker 72-01 opened , disconnecting the battery from the dc bus. The event also caused an internal fault in in-service #1 charger EO-15. The combination of events de-energized the dc bus resulting in loss of power to dc panels EO-11-1, EO-11-2, #1 inverter EO-06 and #3 inverter EO-08. Loss of power to the inverters resulted in loss of power to two preferred ac panels (EY-1 0 and EY-30). Opening of breaker 72-01 was not expected as the design for this breaker required that breaker operation only be available via remote push button.
See Attachment 01 for a detailed event time line.
3.2 Maintenance Initiating Event Summary Breaker 72-120 was the first breaker removed from panel EO-11-2. Upon removal, a small air gap between the positive bus tie stab and the line side positive connection on breaker 72-119 was noted . An initial attempt was made to tighten the connection and close the identified air gap. The termination screw was found to be tight. The air gap was a result of a cross threaded screw, preventing the termination to be made tight. Following the removal of breakers 72-119, 72-121 , and 72-123 the decision was made to remove the positive and negative copper connection stabs used to connect breakers 72-119 and 72-120 to the vertical bus; and to re-tap the damaged threads located on the copper connection stab as a result of the cross threaded screw.
As the positive copper connection stab was being removed , the repairman perceived a small arc which startled him resulting in a loss of control to the positive copper connection stand stab. The positive copper connection stab rotated downward and contacted the negative copper connection stab creating a direct short of the positive and negative dc bus within the EO-11-2 panel. Subsequently the reactor tripped following a loss of power to EO-11-2 panel.
Figures 3-1 and 3-2 below show the configuration of dc panel EO-11-2 just prior to and following the event.
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 Page 6 of 36 Rev. 1 Since the event was the result of a maintenance activity, personnel qualified to determine the extent of condition with respect to the fault and electrical component failures were present to carry out the recovery actions . Buses EO-10L and EO-10R were re-energized from station battery EO-01 within about 50 minutes.
Figure 3-1: DC Panel ED-11 Just Prior to Event Figure 3-1: DC Panel ED-11 After Fault Event 3.3 Latent Coordination Issue Summary See Attachment 02 for a discussion of expected and actual dc breaker and fuse coordination .
3.4 Evaluation Context The 09/25/2011 event revealed two performance deficiencies: (1) inadequate work instructions that led to a maintenance-induced dc panel fault, and (2) inadequate breaker/fuse coordination between a dc panel and bus that led to propagation of the dc panel fault to the dc bus.
A human performance deficiency (inadequate work instructions) caused a fault of sufficient magnitude to expose the latent breaker coordination deficiency. The short circuit current at the dc panel was sufficient to actuate dc breaker 72-01 internal trip function . The breaker actuation is a coordination issue since the fuse from dc panel EO-11-2 should have isolated the fault condition from dc bus EO-10Ll01 O-R.
Actuation of breaker 72-01 removed the battery as one source of power to dc bus EO-1 OLlEO-1 OR and contributed to the total loss of power to the bus.
This analysis evaluates the risk incurred during the post-event response . The human performance deficiency created a condition in which breaker 72-01 opened. Therefore this analysis models breaker 72-01 as open (unless successfully restored). The evaluation models the reactor trip event as a direct consequence of the human performance event, by setting the transient event frequency to unity.
- ~Entergy Entergy PSA Engineering Analysis EA-PSA-SDP-D11-2-11-07 Page 7 of 36 Rev. 1 The consequence of the human performance deficiency alone (without the breaker coordination deficiency) would have been isolation of the dc panel from the dc bus with the bus and the remaining loads continuing to be energized. However, the active (latent) trip mechanism in breaker 72-01 resulted in disruption of the existing coordination.
Breaker 72-01 opened prior to fuse FUZ1D11-2 resulting in disconnection of battery ED-01 from bus ED-10R. Subsequently an internal fault in the in-service #1 battery charger ED-15 actuated to disconnect the panel fault from ac power supply MCC #1 . Additionally the panel fault also caused at least one breaker to open : breaker 72-37 to #1 inverter ED-06 supplying preferred ac bus EY-10 .
Recovery from this event required identification of the fault condition (obvious in this case) and removal of the fault and or isolation of the fault from the dc bus. Once the fault was isolated individual components (battery chargers and inverters) were assessed for operability to allow restoration of power to the dc bus.
Initially, preferred ac bus EY-30 was restored by aligning power to it from the bypass regulator (redundant to the inverter and supplied by instrument ac panel EY-01). Next, buses ED-1 OR & ED-1 OL were declared operable and re-energized from the battery by closing breaker 72-01 . Once the dc bus was energized ,
power to preferred ac bus EY-30 was transferred back to #3 inverter ED-08 being supplied by the dc bus and power was restored to preferred ac bus EY-10 by aligning it to the bypass regulator. At th is point the dc bus and both preferred ac buses were re-energized with portions of dc panel ED-11-2 not available.
3.5 Key Factors Impacting Plant Response Based on the plant response to the ED-11-2 fault event, a review of the following factors represents an opportunity for improved operations and engineering training . The plant response and sensitivities discussed below are considered to be within the knowledge base of operations and engineering.
However, the degree of sensitivity and the operational implications are worth noting here.
Identification of these factors was an indirect result of the risk assessment. Presentation here is for background purposes only. These factors underscore the complexity of the loss of dc event and provide a context for the successful operator actions during the event.
Note: all temperatures , pressures, levels and percentages are considered approximate in the discussions below.
3.5.1 Sensitivity of PZR level to PCS temperature changes PCS temperature changes significantly impact pressurizer level.
For example, based on a PCS volume of 81 ,500 gallons (10,900 ft3, FSAR Table 4-1) and the density change in water from 525°F to 544°F at 2060 psia (47 .1 Ibm/ft3 to 48.3 Ibm/ft ), PCS volume changes by 109 gallonsr F. Based on volumes of 809 ft3 and 593.7 ft3 at levels of 57%
and 42%, respectively [1], there are 107 gallons/%. This results in 1.02%rF .
During this event from 16:03 to 16:15, PCS temperature increased from 529°F to 544 OF. Even with charging and letdown isolated (charging was isolated at 15:57, with pressurizer level at
-80% ; controlled bleedoff at 5 gpm), pressurizer level increased from 85% to 101 .5% due to thermal expansion (see Attachment 01) .
The observed increase agrees reasonably well with a prediction based the rates calculated above (i.e., 1.02%rF and 107 gallons/%):
85% + (544°F - 529°F) * (1.02%r F) - 5 gpm
- 12 min / 107gal/% = 100%.